2022/23 — San José State University

— Preparing Courses for Quality Matters Certification

Proposal Summary: At San José State University, we proposed to provide training along with a faculty-mentor support structure for the faculty participants in the program. We were able to complete the goals of this project through a combination of Quality Matters Workshops, campus-based webinars/recordings, and faculty team leaders. The campus continues to expand our support with guidelines that faculty can use as they develop their online courses. The quality assurance program provides extensive training, faculty expertise, and guidance on the redesign of a course that takes into account best practices as identified in the Quality Matters (QM) rubric. This program focused on revising courses in preparation for Quality Matters Certification.

Campus Goal for Quality Assurance

The EOQA goals are in line with the CSU’s Graduation Initiative 2025 specifically addressing the measure of ensuring effective use of technology is part of every CSU student’s learning environment. Additionally, our approach follows guidelines proposed in SJSU’s Four Pillars of Student Success that promote the development of:

• richer and more readily accessible on-line supplemental study materials;
• more elaborate and interactive homework and self-check instructional materials;
• and more engaging in-class teaching strategies.

This proposal focused on developing a standard with which online and hybrid courses can use as a way to reflect upon their current course design and make the necessary revisions to reflect best practices. There are multiple needs of the campus to support Quality Assurance Efforts.

• 1: Develop materials and resources that faculty can access and use to guide course design.
• 2: Provide professional development opportunities to increase faculty awareness regarding quality assurance.
• 3: Build a group of faculty that can become experts in quality assurance and provide mentoring for new faculty.

Quality Assurance Lead(s)

• Jennifer Redd, Program Cooridinator
• Yingjie Liu, Project Facilitator
• Megan Chang, Faculty Reviewer
• Sulekha Anand, Student Quality Assurance Impact Researcher

Campus Commitment Toward Sustainability of QA Efforts

• Developed multiple Canvas course templates based upon Quality Matters Principles
• Encourage quality assurance principles in instructional design consultations

Summary of Previous QA  Accomplishments

This quality assurance program was the tenth iteration on campus. The previous cohort participated during the 2021-22 academic year. A variety of quality assurance efforts continue to expand on campus.

• Effort 1: A faculty cohort completed one or two Quality Matters trainings: Applying the Quality Matters Rubric and Improving Your Online Course. A previous cohort completed the Peer Reviewer Course. Additional information about last year's effort can be found in the Quality Assurance ePortfolio.
• Effort 2: Increase awareness through outreach activities. This includes posting resources on the eCampus website and through participation in informational webinars. It also includes promoting workshops and encouraging attendance through flyers and presentations at campus events.
• Effort 3: The rubric is provided as a resource for faculty in a password-protected Canvas course. Also, it serves as a guide when instructional designers consult with faculty members on course design.
• Effort 4: Encourage faculty and staff that have completed Peer Reviewer Training to become a Quality Matters Peer Reviewer.
• Effort 5: A Canvas course template that adheres to Quality Matters Standards is available to all faculty.

Course Review and Certifications

The program focused on providing training and support for faculty with online or hybrid courses and preparing those courses for Quality Matters Certification. The program itself was called Getting Ready for QM Certification (GRQC). The program was asynchronous and was broken down into six modules in a Canvas Course. For example, Start Here, QM Workshop information, All about Learning Objectives about Alignment, All about Activities and Assessments, All about Course Technology, Learner Support, and Accessibility and Usability, and QM Review. The faculty participants met with either the lead instructional designer and/or a faculty member for additional guidance to support the best practices learned in the Quality Matters workshop and within the Canvas course modules. They also received detailed feedback through an information course review to help assist them with submitting their courses for a formal review. The next stage involved submitting the course for a formal QM review and then making course revisions per the feedback received. 

Faculty Participants

Development of Campus QA Resources

Canvas Course Templates are available to SJSU faculty

Next Steps for QA Efforts 

• Expand the number of courses that are Quality Matters certified
• Provide guidance to faculty interested in having their courses certified
• Encourage faculty participation in Quality Matters Trainings

Training Completions

The following table summarizes all of the SJSU faculty and staff Quality Assurance training completions that occurred during the 2022/23 academic year.

The CSU QA Student Online Course Survey was distributed via Qualtrics to the classes taught by the 2022-2023 EOQA participants.   The survey was completed by 45 students in 2 lower division courses, 2 upper division courses, and 2 doctoral level courses.  For 56% of the respondents, the course was an elective.  Fifty-three percent of respondents were Asian, 16% were Hispanic or Latino, 11% were Caucasian, and the percentages were below 10% for each of the other ethnicities.  There were approximately equal percentages of male and female respondents, while 2% of respondents were transgender and 2% identified as “other.”  Most respondents were sophomores or seniors.  Fifty-eight percent of respondents had previously taken at least 4 or more online courses.  

In addition to the questions pertaining to course details and student demographics, there were 25 questions asking students to rate their agreement with a statement on a six-point scale from Strongly Disagree (1) to Strongly Agree (6).  There were 4 questions pertaining to Course Overview and Introduction, 5 questions pertaining to Assessment and Evaluation of Student Learning, 4 questions addressing Instructional Materials and Resources Utilized, 3 questions addressing Student Interaction and Community, 2 questions pertaining to Facilitation and Instruction, 2 questions pertaining to Technology for Teaching and Learning, 2 questions addressing Learner Support and Resources, and 3 questions addressing Inclusivity and Accessibility.  Descriptive statistics are presented in the Table.    

The average responses for the Course Introduction and Overview questions were in the Agree to Strongly Agree range (Figure 1).  The average responses for the Assessment and Evaluation of Student Learning questions were also in the Agree to Strongly Agree range (Figure 2).  The question with the lowest average response pertained to understanding how the learning activities helped the student achieve the learning objectives each week (mean = 5.36).   

Figure 1.  Mean responses to questions about course overview and introduction.  In all graphs, error bars depict the standard deviation. 

Figure 2.  Mean responses to questions about assessment and evaluation of student learning.For the questions on Instructional Materials and Resources Utilized, the average responses ranged were in the Agree to Strongly Agree range (Figure 3).  For the questions addressing student opinions on Student Interaction and Community, the average responses were also in the Agree to Strongly Agree range (Figure 4).   The lowest mean pertained to whether the learning activities helped the student understand the fundamental concepts and build skills that will be useful in the real world (mean = 5.27).

 

Figure 3.  Mean responses to questions about instructional materials and resources utilized.

Figure 4.  Mean responses to questions about student interaction and community.There were two questions on Facilitation and Instruction (Figure 5).  The average responses were 5.47 for clarity on how long it would take to receive feedback on assignments, and 5.27 for sending reminders of due dates.   The average ratings for the two questions pertaining to Technology for Teaching and Learning were 5.21 the use of a variety of technology tools to engage the class and encourage them to interact, and 5.49 for providing clear information on how to access/acquire the required technologies (Figure 6).  

 

Figure 5.  Mean responses to questions about facilitation and instruction.

Figure 6.  Mean responses to questions about technology for teaching and learning.The average responses for the questions on Learner Support and Resources were in the Agree to Strongly Agree range (Figure 7), as were the average ratings for the Inclusivity and Accessibility items (Figure 8).    

Figure 7.  Mean responses to questions about learner support and resources.

Figure 8.  Mean responses to questions about inclusivity and accessibility.

 

 

 

 

 

 

 

Seven of the eight faculty participants in the 2022-2023 EoQA program were interviewed upon completion of the program.  They were asked about the changes they had made to their courses or planned to make based on their EoQA training and specific modules in the EoQA program, namely the QM IYOC workshop, Learning Objectives module, Alignment module, and Course Technology/Learner Support/Accessibility & Usability module.  Participants incorporated numerous revisions, updates, and additions to their courses and Canvas shells as a result of what they learned in the EoQA program and QM IYOC workshop.  

The reported changes include: 

• Updates to the welcome or home page
• Incorporating pages and links for technology/learner support/accessibility & usability (the Start Here module)
• Organization of the Canvas shell and modules
• Adding a page to each module with learning activities and objectives aligned 
• Adding module-level objectives
• Mapping module learning objectives to course learning outcomes
• Mapping modules, objectives, and assignments
• Revised learning objectives
• Alignment of assignments with learning objectives or modules
• Updated or new assignments
• Changes to assessment
• Rubrics
• Videos
• Improving accessibility, Ally, PermaLink
• Improving usability
• Diversity statements
• Implementing QM standards and requirements

Some of the participants have submitted their course for QM certification or are almost ready to.  

All participants had positive comments about their experience in the EoQA program and would recommend it to colleagues.  One suggested that it be marketed to science and engineering faculty.  The program is a lot of work but participants spoke favorably about how much they learned and the improvements made to their courses. 

One suggestion is for updates to the Start Here module because there are redundancies that may not be useful for students (e.g. syllabus, course learning outcomes, other information appearing in multiple places) and some broken links.  Another suggestion is for a timeline for completion of work because the due dates weren’t always clear.