Quality Assurance for blended and online courses

2019/20 — San José State University

— Preparing for Quality Online Course Design

Proposal Summary: At San José State University, we proposed to provide training along with a faculty-mentor support structure for the faculty participants in the program. We were able to complete the goals of this project through a combination of Quality Matters Workshops, campus-based webinars, and faculty team leaders. The campus continues to expand our support with guidelines that faculty can use as they develop their online courses. The quality assurance program provides extensive training, faculty expertise, and guidance on the redesign of a course that takes into account best practices as identified in the Quality Matters (QM) rubric. Many faculty noted this program assisted with the transition to completely online that took place during the middle of the spring 2020 semester.

Campus QA Goals

Campus Goal for Quality Assurance

The EOQA goals are in line with the CSU’s Graduation Initiative 2025 specifically addressing the measure of ensuring effective use of technology is part of every CSU student’s learning environment. Additionally, our approach follows guidelines proposed in SJSU’s Four Pillars of Student Success that promote the development of:

  • richer and more readily accessible on-line supplemental study materials;
  • more elaborate and interactive homework and self-check instructional materials;
  • and more engaging in-class teaching strategies.

This proposal focused on developing a standard with which online and hybrid courses can use as a way to reflect upon their current course design and make the necessary revisions to reflect best practices. There are multiple needs of the campus to support Quality Assurance Efforts.

  • 1: Develop materials and resources that faculty can access and use to guide course design.
  • 2: Provide professional development opportunities to increase faculty awareness regarding quality assurance.
  • 3: Build a group of faculty that can become experts in quality assurance and provide mentoring for new faculty.

Quality Assurance Program Team and Participants

Quality Assurance Lead(s)

  • Jennifer Redd, Project Facilitator
  • Debbie Weissmann, Faculty Quality Assurance Team 1
  • Ravisha Mathur, Faculty Quality Assusrance Team 2

Supporting Campus Partners

  • Yingjie Liu, Lead Instructional Designer
  • Bethany Winslow, Instructional Designer
  • Emily Chan, Interim Associate Dean for Research and Scholarship
  • Ashour Benjamin, Course Reserves/Leganto Coordinator

Campus Commitment Toward Sustainability of QA Efforts

  • Developed multiple Canvas course templates based upon Quality Matters Principles
  • Encourage quality assurance principles in instructional design consultations

Summary of Previous QA  Accomplishments

This quality assurance program was the seventh iteration on campus. The previous cohort participated during the 2018-19 academic year. A variety of quality assurance efforts continue to expand on campus.

  • Effort 1: A faculty cohort completed one or two Quality Matters trainings: Applying the Quality Matters Rubric and Improving Your Online Course. A previous cohort completed the Peer Reviewer Course. Additional information about last year's effort can be found in the Quality Assurance ePortfolio.
  • Effort 2: Increase awareness through outreach activities. This includes posting resources on the eCampus website and through participation in informational webinars. It also includes promoting workshops and encouraging attendance through flyers and presentations at campus events.
  • Effort 3: The rubric is provided as a resource for faculty in a password-protected Canvas course. Also, it serves as a guide when instructional designers consult with faculty members on course design.
  • Effort 4: Encourage faculty and staff that have completed Peer Reviewer Training to become a Quality Matters Peer Reviewer.
  • Effort 5: A Canvas course template that adheres to Quality Matters Standards is available to all faculty.

Course Peer Review and Course Certifications

  1. Peer Review: Faculty members were assigned a partner. Using a rubric, they provided a peer review for each other's course.
  2. Faculty Lead Review: Faculty members were equally divided into two teams. The faculty lead for each team provided each faculty member with an informal course review.

Faculty Participants

NameCourse NumberCourse Name
Deborah BoldingOCTH 246Occupational Therapy with Young Adults
Michele Burns
EDCO 4Personal, Academic, and Career Exploration
Toni Campbell
CHAD 106Concepts of Childhood
San-hui Chuang
CHIN 25BIntermediate Chinese
Jimma CortesEDCO 4Personal, Academic, and Career Exploration
Susana GallardoWOMS 102The Global Study of Women
Tabitha Hart
COMM 154IEthnography for Comm Studies
Yinghua Huang
HSPM 121Hospitality Leadership and Management
Tina KoraniMCOM 284Advanced User Experience
Tanvi KothariBUS 187Global Dimensions of Business
Steven Lee
FS 160Special Topics in Forensic Science
Robin LoveCHAD 106Concepts of Childhood
Rosanne Macek
INFO 210Reference and Information Services
Bill MongelliINFO 282Seminar in Library Management
Wilson YuanJS 151Criminological Theory

Resources and Program Efforts

Accessibility/UDL Efforts

  1. One of the webinars during the yearlong program focused on accessibility. This included document preparation, presentations, and LMS features.
  2. One of the webinars during the year-long program focused on Universal Design for Learning. This included tips and examples.
  3. Workshops throughout the year that introduce faculty to new teaching methods as well as to the LMS includes time for discussions regarding developing accessible instructional materials.

Feedback was gathered following each of the webinars that were part of the year-long cohort. Participants answered two questions:

  1. Did the ________ webinar help you to identify something you would like to include, change, or develop in your course?
  2. Share 1 question or comment regarding the ________ webinar.

Development of Campus QA Resources

Canvas Course Templates are available to SJSU faculty

Next Steps for QA Efforts 

  • Expand the number of courses that are Quality Matters certified
  • Provide guidance to faculty interested in becoming a Peer Reviewer
  • Offer an on-campus workshop and/or online opportunities for faculty to attend Quality Matters Trainings
  • Provide a faculty-team lead Quality Assurance Training program that incorporates Quality Matters workshops, webinars, and peer-guidance/feedback

Quality Assurance Training Completions

Training Completions

The following table summarizes all of the SJSU faculty and staff Quality Assurance training completions that occurred during the 2019/20 academic year.

TrainingNumber of Completions
Applying the Quality Matters Rubric9

Applying the QM Rubric Online Facilitator Certification (AOFC)

2
Assessing Your Learners (AYL)

2
Connecting Learning Theories to Your Teaching Strategies (CLTTS)
1
Creating Presence in Your Online Course (CPOC)
1
Evaluating Your Course Design (EYCD)
2
Exploring Your Institution's Policies (EYIP)
2
Gauging Your Technology Skills (GYTS)
2
Improving Your Online Course (IYOC)15
Improving Your Online Course Online Facilitator Certification (IOFC)
1
Introduction to Teaching Online Using the QLT Instrument (QLT1)10
Introduction to Teaching Online Using the QLT Instrument for Summer (QLT1-S)
47
Master Reviewer Recertification 2020 (MRR)2
Orienting Your Online Learners (OYOL)
2
Peer Reviewer Course (PRC)1
Teaching Online Certificate
1


Student Quality Assurance Impact Research: Student Survey Results

The CSU QA Student Online Course Survey was distributed via Qualtrics to the classes taught by the seventeen 2019-2020 EOQA participants.  The survey was completed by 262 students in 10 classes, 5 of which were upper division and 3 of which were graduate courses.  Seventy-four percent of respondents were female and 26% were male.  The majority of respondents were juniors (37%) or graduate students (29%).  Forty-six percent of respondents reported having previously taken 4 or more online courses, while 18% reported that this was their first online course.  Twelve percent of respondents had taken one online course before and 24% had taken 2-3 online courses.  Thirty-eight percent of respondents were Asian, 29% were Caucasian, 18% were Hispanic or Latino, 7% were “Two or More Races,” and there were low percentages of African American (3%) and “Other” (5%) respondents.  No respondents self-reported as Alaska Native/Native American/Pacific Islander. 

In addition to the questions pertaining to course details and student demographics, there were 25 questions asking students to rate their agreement with a statement on a six-point scale from Strongly Disagree (1) to Strongly Agree (6).  There were 4 questions pertaining to Course Overview and Introduction, 5 questions pertaining to Assessment and Evaluation of Student Learning, 4 questions addressing Instructional Materials and Resources Utilized, 3 questions addressing Student Interaction and Community, 2 questions pertaining to Facilitation and Instruction, 2 questions pertaining to Technology for Teaching and Learning, 2 questions addressing Learner Support and Resources, and 3 questions addressing Inclusivity and Accessibility.  

Descriptive statistics are presented in the Table.  The average response for each question was greater than 5.0, falling between the ratings of Agree (5) and Strongly Agree (6) on the scale.  The average responses for the Course Introduction and Overview questions ranged from 5.47 to 5.66 (Figure 1).  The average responses for the Assessment and Evaluation of Student Learning questions ranged from 5.44 to 5.58 (Figure 2).  

Figure 1.  Mean responses to questions about course overview and introduction.  In all graphs, error bars depict the standard deviation. Figure 1. Mean responses to questions about course overview and introduction.  In all graphs, error bars depict the standard deviation. 

Figure 2.  Mean responses to questions about assessment and evaluation of student learning.Figure 2. Mean responses to questions about assessment and evaluation of student learning.

For the questions addressing Instructional Materials and Resources Utilized, the average responses ranged from 5.52 to 5.62 (Figure 3).

Figure 3.  Mean responses to questions about instructional materials and resources utilized.Figure 3. Mean responses to questions about instructional materials and resources utilized.

For the questions addressing student opinions on Student Interaction and Community, the averages ranged from 5.29 to 5.53 (Figure 4).

Figure 4.  Mean responses to questions about student interaction and community.Figure 4. Mean responses to questions about student interaction and community.

There were two questions on Facilitation and Instruction (Figure 5).  The average response was 5.25 for the question addressing the instructor’s clarity on how long feedback on assignments would take and whether feedback was provided in a timely fashion.  The average response was 5.41 for the question asking whether the instructor sent reminders of due dates and other information to help keep the student on task.

Figure 5.  Mean responses to questions about facilitation and instruction.Figure 5. Mean responses to questions about facilitation and instruction.

The average ratings for the two questions pertaining to Technology for Teaching and Learning were 5.27 for the use of a variety of technology tools to engage the class and encourage them to interact, and 5.35 for providing clear information on how to access/acquire the required technologies (Figure 6).  Regarding Learner Support and Resources (Figure 7), the average was 5.45 for both questions.

Figure 6.  Mean responses to questions about technology for teaching and learning.Figure 6. Mean responses to questions about technology for teaching and learning.

Figure 7.  Mean responses to questions about learner support and resources.Figure 7. Mean responses to questions about learner support and resources.

Finally, average ratings for the Inclusivity and Accessibility items ranged from 5.48 to 5.53 (Figure 8).

Figure 8.  Mean responses to questions about inclusivity and accessibility.Figure 8. Mean responses to questions about inclusivity and accessibility.


Student Quality Assurance Impact Research: Faculty Interview Summary

Ten of the seventeen faculty participants in the 2019-2020 EoQA program were interviewed upon completion of the program.  They were asked a series of questions pertaining to the changes they had made to their courses based on various components of the EoQA program.  

Overall, the program was viewed favorably, with participants enthusiastically reporting that they had made numerous positive changes to their courses as a result of the EoQA training as well as the summer QM courses, and that they are planning to make additional changes.Faculty indicated that they had learned very useful information, skills, principles, and a framework for online and hybrid teaching.  They were able to understand the student perspective during online learning.  It was often reported that because of the EoQA program, participants were well equipped for the transition from in-person to online instruction that was mandated due to the pandemic and they were very thankful for this.

Modifications to courses made thus far include:

  • Design and structure of a new course
  • Structure/organization, modules, navigation, links
  • Getting Started module, introductory information for class, welcome/introduction video
  • Putting Announcements on home page
  • Further development of home page
  • Links to documents in each announcement to save the students time 
  • Revising course learning outcomes (CLOs), mapping them (and GELOs) to learning activities
  • Module-level learning outcomes
  • Accessibility, headers, hierarchy, captioning
  • UDL and presenting information in different ways for different learners
  • Lecture capture, videos, incorporating other media 
  • Links to University support services, resources page in Modules
  • More on-campus and off-campus resources for students in syllabus and module in Canvas
  • Including the name of a website along with its URL
  • Active links for pictures and icons
  • Learning and applying QM standards, template

There were overall rave reviews for the Lecture Capture/Storytelling webinar.  The UDL/DI webinar was reported to be informative in that it emphasized different learning styles and modalities of instruction, but the information was not new for some participants.  Improving accessibility is a goal for most participants, and many already knew the information presented in the Accessibility webinar or were already using accessible materials but they did learn a few new tips.  Most participants felt that the Affordable Learning Solutions and Copyright webinars were useful but not completely new information; however, some participants feel that they need additional information about copyright restrictions.

The facilitators were reported to be responsive and accessible via email, helpful, and professional.  Peer reviews and facilitator reviews would have been very helpful but many were cancelled due to the pandemic.  It was reported that completing a peer review on a peer’s course was a very useful process.

Participants would and have highly recommended the program to their colleagues.