Quality Assurance for blended and online courses

2020-2021 — Fullerton — Quality Assurance ePortfolio for CSU Online Course Services

CSUF Quality Matters Tuffy elephant mascot

Proposal Summary: For AY 20-21, we engaged faculty from a wide variety of disciplines and partnered with the Online Education and Training (OET) department to strengthen and expand our Quality Matters (QM) community. Despite the challenges of COVID 19, we had a significantly large number of faculty members complete quality assurance training on our campus, had two campus members complete peer review or other specialized QM training, and three OET team members certified as DYOC facilitators and two OET team members certified as APPQMR facilitators.


Our campus continues to work toward addressing the four areas of campus need identified in 2014 for online teaching, assessment, and quality assurance: 

  1. Uneven level of expertise in online/blended pedagogy, design, and delivery
  2. Lack of understanding of faculty needs related to effective online/blended teaching
  3. Lack of coordination on strategies and choice of tools to ensure the quality of online and blended courses
  4. Lack of organization and community among departments to support quality online and blended teaching

Campus Quality Assurance Goals

Campus Goals for Quality Assurance

  • Goal 1: Expand the awareness and implementation of the QM process on campus 
  • Goal 2: Deepen QA/QM expertise on campus
  • Goal 3: Support expansion of QM course certification
  • Goal 4: Disseminate project success and lessons learned
  • Goal 5: Expand research on the impact of faculty QA/QM expertise on student learning outcomes and experiences

Quality Assurance Leads

  • Shelli Wynants, Ed.D., Faculty QM Lead, QMC/Campus Contact
  • Sinjini Mitra, Ph.D., SQuAIR Lead

Supporting Campus Partners

  • Online Education and Training
  • Faculty Support Services (Faculty Development Center, Academic Technology Center, Faculty Affairs and Records)
  • Office of Assessment and Educational Effectiveness
  • Information Technology
  • CSUF Colleges/Departments

Campus Commitment Toward Sustainability of QA Efforts

  • Host workshops and events for faculty on online teaching pedagogy and effective use of technology tools
  • Disseminate information on Quality Matters to the campus to promote online quality assurance
  • Support faculty and campus team members to attend professional development opportunities locally and nationally

Summary of Previous QA  Accomplishments

QM Certified Faculty Peer Reviewers

  • Manal Alatrash, Ph.D. RN, (2019) Professor, College of Health and Human Development
  • Tara Barnhart, Ph.D. NBCT, (2017) Professor, College of Education
  • Shana Charles, Ph.D., (2017) Assistant Professor, College of Health and Human Development
  • Catherine Dinh, MBA, (2015) Instructional Designer, Online Education & Training
  • Susan Gaitan, MA, (2014) Instructional Designer, Online Education & Training
  • Barbara Glaeser, Ph.D., (2016) Professor, College of Education
  • Sudarshan Kurdwarkar, Ph.D., (2016) Associate Professor, College of Engineering and Computer Science
  • Rachel McClanahan, DNP, RN, (2015) Associate Professor, College of Health and Human Development
  • Sinjini Mitra, Ph.D., (2016) Associate Professor, Mihaylo College of Business and Economics
  • Ken Moyer, M.A., (2019) Instructional Designer, Online Education & Training
  • Marsha Orr, MSN, RN (2014) Lecturer, College of Health and Human Development
  • Suzanne Robertson, Ph.D. RN, (2014) Professor, College of Health and Human Development
  • Chris Street, Ph.D., (2018) Professor, College of Education
  • Diana Tyler, MSN, RN (2018) Professor, College of Health and Human Development
  • Shelli Wynants, Ed.D., (2016) Lecturer, College of Health and Human Development
  • Whitney Youngren, M.A., (2017) College of Humanities and Social Sciences

QM Certified Courses

List of 11 CSUF QM certified courses


Development of Campus QA Resources

Creation of new resources to support faculty teaching in new LMS, Canvas:

  • Getting Help Using Canvas
    Multiple resources were culminated and created for this webpage to support faculty in teaching in our new Canvas LMS, including a self-paced Intro to Canvas course, a searchable Canvas Resource Center of 24/7 self-help guides, and a FDC/OET Canvas YouTube Channel

Continued updates and enhancements to helpful faculty resources:

  • Best Practices for Online Teaching Guide 
    The department of Online Education and Training (OET) created this guide to provide useful information about online instruction at CSUF to create opportunities for instructors to improve online teaching practices.

    Open Fullerton
    The goal of the OpenFullerton site, created by OET, is to empower faculty in supporting student success by providing them a variety of resources and professional development opportunities to help reduce textbook costs, and by supporting their efforts to explore, cultivate, create, and access high quality Open Educational Resources (OER).

Next Steps for QA Efforts 

By the end of AY21-22, CSUF hopes to achieve the following four deliverables:

  1. five QM online course reviews
  2. two or three CSUF-hosted DYOC+ learning community workshops taught by CSUF faculty/staff certified as facilitators
  3. at least 36 faculty completing all deliverables in the DYOC+ learning community workshops and earning their QM DYOC certificates
  4. At least three OET webinars to support faculty application of QM standards in their online/blended courses
  5. A research report analyzing study results of QM-expertise on student learning outcomes and experiences

Quality Assurance Results

Training Completions

The following table summarizes all of the CSUF faculty and staff Quality Matters and QLT training completions that occurred during the 20-21 academic year.

Training

Number of Completions

Applying the Quality Matters Rubric

68
Improving Your Online Course
60

Peer Reviewer Course

1
Introductory QLT 25
Advanced QLT
5
K-12 Reviewer Course for Higher Ed 1
APPQMR/DYOC Facilitator Course 5


New this academic year, OET created a 5-topic webinar series on Online Teaching Best Practices, which was offered in Fall 2020 and Spring 2021. The series consisted of the following webinar topics, each one hour long: (1) Applying the QM General Standard 1: Course Overview and Introduction; (2) Communication Strategies for Creating Community; (3) Breaking Course Activities into Bite-Sized Chunks; (4) Integrating Student Reflection to Promote Learning; and (5) QM Alignment: What is it and Why should I care? 

Webinar

Number of Attendees

QM Course Overview and Introduction 

61

Communication Strategies for Community

64

Bite-Sized Chunks for Activities

69
Integrating Student Reflection 53
QM Alignment 48


Faculty survey feedback on this webinar series (1050 responses across the webinars) indicated that 100% would sign up for more webinars like these, 96% found the pace of the webinars "just right," 98% reported learning at least one new strategy they would apply to their online courses, and 99% said they found the webinar a valuable use of their time.

Accessibility/UDL Efforts

  1. OET offers an online certificate program for Accessible Instructional Materials (AIM). Our AIM certificate program has 3 sequential modules; each 4-week module is designed to take 5 hours to complete and is a combination of instructional activities, collaborative discussions, and individual application.  In this program, faculty become more aware of the various disabilities of their students and the resultant issues, including legal and regulatory obligations, instructional needs, and campus resources. This program also includes an introduction to Universal Design for Learning and its application in creating accessible documents and captioning instructional videos.
  2. OET provides resources for faculty on accessibility and UDL on its website.
  3. CSUF purchased Ally for the LMS to help faculty.  

Student Quality Assurance Impact Research

In AY 20-21, because almost all courses at CSUF were offered in an online format, CSUF focused only on courses at the College of Business and Economics (CBE) for this analysis. This includes both undergraduate and graduate courses, spanning across 3 different online formats, namely, asynchronous, synchronous, and hybrid (asynchronous + synchronous).

 A total of 195 courses and 777 course sections were offered at CBE in Fall 2020 (all of them in the virtual format). Details regarding these online course offerings are included below.

Faculty Details and Background: A total of 306 faculty members taught online courses in CBE in Fall 2020. Out of these, 39 (12.7%) had received some QA-related training related to teaching online or hybrid classes. Details regarding the different types of trainings completed by faculty are shown in the table below (note that some faculty completed multiple trainings):

Out of these 52 different trainings completed by CBE faculty, 37 (71.1%) were completed after the pandemic-driven transition to virtual instruction (between May and December 2020). Currently there is no QM-certified online course at CBE. As to the number of QA trainings per instructor, 28 out of the 39 faculty members with QA-related experience had only one training, 10 had two different trainings, while 1 had four different trainings.


Course details: Of the 777 total course sections at CBE, 123 (15.8%) were taught by faculty members with some QA-related training while the rest 654 (84.2%) were taught by faculty with no QA-related training. The number of course sections taught by QA-trained and non-QA-trained faculty across different online modalities and course levels are shown in the following two tables. 38% of faculty member with QA-related training taught online asynchronous courses in Fall 2020 compared to 18% of non-QA-trained faculty. However, the proportion of QA-trained and non-QA-trained faculty members teaching different levels of courses are very similar (nearly 90% teaching undergraduate and 10% teaching graduate level courses).


Student outcomes data: Average percentage of repeatable grades obtained by all students in course sections taught by a faculty member with QA-related training is less than that obtained from course sections taught by a faculty member with no QA-related training (and also overall). When dividing students into different at-risk groups (underrepresented or UR students, first-generation students, and Pell grant recipients), a similar trend is found: less repeatable grades were observed for sections taught by QA-trained faculty than for those taught by faculty with no QA-related experience. These are shown in the following table. All of these differences, however, are not statistically significant.