Quality Assurance for blended and online courses

2018 2019 — Sacramento State— Quality Assurance

Proposal Summary In alignment with the University’s strategic plan goal #2: Foster innovative teaching, scholarship and research, our QA goal is to develop institutional capacity and infrastructure to support quality online teaching, learning, and student success. To do this we aim to:
I. Increase the number of faculty who are QA trained as practitioners and mentors.
II. Use action research, and data analytics to measure student learning, improve student success, and contribute to the Scholarship of Teaching and Learning (SoTL) in online learning environments
     III. Establish an institutional support structure for the development, redesign, review, and QA certification of online courses.

Campus QA Goals

Campus Goal for Quality Assurance

  • Goal 1: Complete 50 QA faculty training 
  • Goal 2: Recruit 5 new courses for coursematch
  • Goal 3: Create a QM internal review process
  • Goal 3: Submit 5 courses for formal QM review
  • Goal 4: SQuAIR report: Data Analytics report looking at correlations between QM training and DFW rates

Quality Assurance Lead(s)

  • Milica Marcovic, Professor of Engineering
  • Mark Rodriguez, Professor of Education

Supporting Campus Partners

Center for Teaching and Learning, Information Resource Technology Division, Instructional Designers, University Library, Faculty Associates from each of our Colleges: Arts and Letters, Health and Human Services, Natural Science and Mathematics, Social Science and Interdisciplinary Studies, Engineering, Education, and Continuing Education

Campus Commitment Toward Sustainability of QA Efforts

  • Establishment of internal review process and committee
  • Investment of $35,000 of matching funds for QA from Academic Affairs

Summary of Previous QA  Accomplishments

  • Sac State has now trained over 200 faculty and staff in Quality Assurance
  • An internal QM review process has been developed
  • 3 Faculty Learning Communities on hybrid and online learning have been conducted
  • Our SQuAIR project has created a data structure for QA research

Dissemination of QA Efforts— Conferences 

Exemplary resources and outcomes were presented at the QM annual conference and IEEE Annual Education Conference.

1. Poster Presentation at the QM, 10th annual Conference, Oct. 30-Nov. 2, 2018, St. Louis Missouri. “Open Source 6th Edition Quality Matters Template for Canvas, Kyle Cage, Instructional Designer & Milica Markovic, Professor of Engineering, https://www.qualitymatters.org/qa-resources/resource-center/conference-presentations/open-source-6th-edition-quality-matters

2. Zabihian, F., Achievement Gap in Online and Hybrid Courses Versus Face-To-Face Courses Among Engineering and Computer Science Students, 2018 IEEE Frontiers in Education Conference (FIE), San Jose, CA, Oct. 2018

3. As of Dec. 2018, the Sac State QM template created for Canvas has been downloaded over 200 times.

Quality Assurance Results

Training Completions82 QA trainings completed, exceeding the goal of 50 QA trainings

Course Peer Review and Course Certifications

Efforts were made by the Center for Teaching and Learning Quality Assurance faculty Mentor group to develop a campus level internal peer review process for blended/online courses. The group was comprised of faculty from each of the Colleges including the Library, the Office of Program Assessment, and the College of Continuing Education. The Internal Peer Review Process is aligned with the Quality Matters Peer Review process following guidelines established by Quality Matters. QM guidelines include that the Internal Peer Review “is to use QM tools and is conducted by a minimum of 2 reviewers who have completed either the Applying the Quality Matters Rubric (APPQMR) or the Improving Your Online Course (IYOC) workshop, at least one of whom is also a QM-Certified Peer Reviewer or Master Reviewer.” There are approximately 70 faculty or staff who are certified on the APPQMR course and 3 additional faculty who have received IYOC certification. There are 4 certified Master Reviewers and 4 additional certified QM Reviewers. There are also 2 certified Course Review Managers who would set up the Internal Course Reviews. The current core review team includes faculty Michelle Dang, Mark Rodriguez, Milica Markovic, Debra Welkley, Maureen Lojo, Russell DiSilvestro, and Ana Harper. Approximately 70 other faculty or staff are trained and available to call on for internal QM reviews. A process has been established to apply for reviews that includes a web site where information is provided and faculty may request their course to be reviewed. The QMC’s including Michelle Dang, Mark Rodriguez and Milica Markovic can approve courses and initiate the Internal Peer Review Process. The Course Review Manager directs the Internal Peer Review, which follows the established QM Peer Review Process. Two courses have gone through the Internal Peer Review process in the 2018-19 AY, including Nursing 170 which also is in the submission process for a QM Certified Review. Additionally the Phil 104 course is going through the Internal Peer Review Process. Four other courses are planned to go through the Internal Peer Review process this upcoming academic year.

Student Quality Assurance Impact Research 

Sample data showing comparison of average of course grades before and after instructor completion of the first QA course. In addition to this quantitative comparison faculty interviews, student surveys, and examination of the course structure are needed to determine the nature of the QA training impact and correlation to student success.  Case studies examining subsets of the data set are underway.

  • The SQuAIR research has begun with the establishment of a data base of faculty trained and 12 years of course and student success data between 2005 to 2017
  • Sample of Data Collected is shown above
  • Bulk descriptive analysis of data has revealed the need to examine smaller case studies prior to using the data set to make any generalizations about student success and QA training
  • Our SQuAIR faculty associate is a participant in the CSU SQuAIR online zoom meetings.

Development of Campus QA Resources


Accessibility/UDL Efforts

Accessibility efforts have been focused on piloting Blackboard Ally.

  • After the self and internal peer review NUR 170 was tested for accessibility using Blackboard Ally 
  • The areas of NUR 170 that were found to be inaccessible by Ally were revised by a team of instructional technologists in preparation for submitted the course to external QM review.

Next Steps for QA Efforts 

  • Institutionalize the internal QA review process
  • Expand the number of courses QM certified
  • Continue research on QA and student success, using institutional data sets