Quality Assurance for blended and online courses

A Tiered Approach to Quality Assurance Using QLT
AY 2020–2021
California State University, Northridge

Proposal Summary: At CSUN, we continued the tiered approach that we have implemented for the last three years of quality assurance efforts. We targeted 41 faculty members (plus 9 peer reviewers/mentors) who attended our Winter eLearning Institute.

Campus QA Goals

Campus Goal for Quality Assurance
This year we decided to continue our tiered approach to quality assurance. Our goal was to continue our training opportunities and accommodate faculty members with different levels of knowledge of QLT, and especially, reach out to those who had not yet been exposed to the instrument since its implementation on our campus. As we pivoted virtual instruction due to the COVID-19 emergency, our focus shifted entirely to getting as many faculty members as possible to go through our eLearning Institute with the allocated funds.

  • 41 faculty member were put through our Winter eLearning Institute. This entailed a full self-review of one of their courses by applying the entire QLT 57-objective rubric.
  • Each participant was paired with a peer reviewer/mentor (9 total), who did a full peer review by applying the entire QLT 57-objective rubric with scoring, comments, and one-on-one meetings.

Quality Assurance Lead

  • Kenneth V. Luna, Professor and Chair of Linguistics/TESL


Supporting Campus Partners
The QA Faculty Lead is joined by the Office of Faculty Development team of two instructional designers and other members, all of which are involved in training and are critical in support of the peer-to-peer professional development and the successful adoption of quality assurance methods to a growing number of online/hybrid courses and eLearning projects:

  • Hillary Kaplowitz, Senior Instructional Designer
  • Janett Silvers, Instructional Designer

The Director of Faculty Development and the Associate Vice President of Undergraduate Studies will be involved in all stages of the quality assurance implementation:

  • Whitney Scott, Director of Faculty Development
  • Elizabeth Adams, Associate Vice President of Undergraduate Studies

In addition, the Faculty Development works in close collaboration with the Faculty Technology Center within Academic Technology.


Campus Commitment Toward Sustainability of QA Efforts

  • CSUN matched funds directly related to this project by 142.5%.
  • In addition, the campus separately funds efforts for all the following programming:
    • Faculty Development:
      • Workshops on Demand (e.g. Learning-Centered Syllabus, Promoting Academic Honesty, departmental workshops)
      • eLearning Programs (e.g. Get Up to Speed with Online Teaching, eLearning Institute, Association of College and University Educators (ACUE)'s Effective Online Teaching Practices FLC)
      • eLearning Resources (e.g. Remote Teaching Community Canvas Site, eLearning Essentials Canvas Site, Digital Tools for Creating Content and Activities)
      • Faculty Learning Communities (e.g. Transforming STEM Teaching, Institute for Transformative Teaching and Learning)
      • Active Learning Ambassadors and eLearning Ambassadors
    • Faculty Technology Center:
      • Summer Bootcamps (Assignments, Quiz, Engagement, Lecture Recording, Virtual Meetings, Learning Analytics, and Accessibility)
      • Zero to Online


Summary of Previous QA Accomplishments

  • Faculty members applied to attend a eLearning Institute over the summer.
  • At the Institutes, QLT was introduced and applied through a think, pair, share activity.
  • A QLT page was created along with a video featuring CSUN's QLT award program nominees.
  • Exemplary redesigned courses were submitted to the annual QLT award program.
  • Emphasis on quality was also an integral part of our Tablet and eText initiatives.
  • During the 2014—2015 quality assurance efforts:
    • Twenty faculty members attended a CSU QLT workshop or 1-on-1 CSU QLT meeting, completed a CSU QLT self-assessment, completed an exit survey reflection.
    • We created a self-service module on the uses, application, and implementation of CSU QLT instrument.
    • Faculty Associate and two Instructional Designers attended the CSU QLT training at the Chancellor’s Office.
    • One Faculty Associate attended QM training at the Chancellor’s Office.
  • During the 2015—2016 quality assurance efforts:
    • We held two QLT workshops (tier 1) for a total of 29 faculty members introducing them to the QLT instrument. We used a QLT Moodle site–created as part of our 2014—2015 quality assurance efforts–for the workshops and as a self-service module.
    • Eight faculty members rated one of their courses with the potential for submission to the CSU QLT Awards and Recognition Program (tier 2). Six faculty members submitted to the recognition program.
    • We hosted a peer reviewer training, facilitated by Ashley Skylar from CSU Chancellor’s Office, Academic Technology Services (tier 3). A total of 17 faculty members, instructional designers, and instructional technologists attended the training.
    • Hillary Kaplowitz presented at the Association for Educational Communication and Technology (AECT) Conference how the use of knowledge surveys in our QLT trainings can be adapted for faculty development that focuses on redesign of face-to-face courses for online and/or hybrid delivery. Knowledge surveys are often used as a student course activity that employs confidence judgments to promote self-reflection, metacognition, and improve calibration.
  • During the 2016—2017 quality assurance efforts:
    • We held a QLT workshops (tier 1) for a total of 19 faculty members introducing them to the QLT instrument. We used a newly created QLT Canvas site for the workshops and as a self-service module. We gave priority to faculty members who teach courses with high DUF rates first and then opened it up to the rest of the campus community.
    • Eight faculty members rated one of their courses (tier 2). We gave priority to the top 10 courses with high DUF rates.
    • We sent 7 faculty members with prior experience with QLT to enroll in the Chancellor's Office Reviewing Courses Using the QLT Instrument asynchronous online course.
    • We held a 2-day retreat for department chairs to discuss, engage, and develop ideas related to campus QA efforts. Their feedback informed our proposal for the 2017-2018 Quality Assurance Program.
  • During the 2017—2018 quality assurance efforts:
    • We held a QLT workshops (tier 1) for a total of 11 faculty members introducing them to the QLT instrument. We used a QLT Canvas site for the workshops and as a self-service module. We gave priority to faculty members in courses that would impact student success in support of CSUN’s graduation initiative.
    • Five faculty members self-reviewed one of their courses (tier 2). We gave priority to bottleneck courses, especially those with high rates of non-passing grades and high enrollment. Faculty were required to fill out a reflection of their QLT score.

Quality Assurance Results

Training Completions

Course Peer Review and Course Certifications
As part of our Winter eLearning Institute, each participant was required to do a full self-review of one of their courses by applying the entire QLT 57-objective rubric. In addition, each participant was paired with a peer reviewer/mentor, who did a full peer review by applying the entire QLT 57-objective rubric with scoring, comments, and one-on-one meetings.

Participating Faculty Feedback

  • "This was an excellent opportunity to prepare my course to teach online with the guidance and mentorship setup in the program."
  • "I loved this program and think that it would be valuable for all faculty to be part of."
  • "The value was providing the space and time for reflection and review of my course as well as providing real life examples from other faculty.”
  • "I wanted help in improving my courses online and with the new zoom requirements as well as finding more ways to engage my classes online and this program did even more than I expected! The faculty development program has always been excellent, but this one has been extraordinary in the ways that it gave me ideas with examples within how this program was set up and run."
  • "I think this was an incredibly simple and effective program and honestly, should be mandatory for ALL faculty. As much as I appreciate the compensation, I think this training would have transformed my previous trials and errors in composing my courses. I wasn't too far away from the components of QLT Workbook, but I did pick up some ways to really streamline my course materials and the peer feedback was evidence of why it's important to have a second set of eyes to review your work. I think it was great!"
  • "I cannot praise this program--its organization, its guidance, its materials, and its leaders--highly enough."
  • "This program was incredibly valuable. I really appreciated being able to go step-by-step through the QLT and address how well I met those milestones (or didn't!). Then reviewing them to see where I could make them stronger with the peer review helped me take my page to the next level. The in-depth examples helped me see areas I could strengthen those core items and be more specific and also make my content more accessible."
  • "This program is invaluable. The access to well-trained facilitators and mentors provides the confidence necessary for pursue new challenges."
  • "This is the most helpful program I have ever attended at CSUN."


Development of Campus QA Resources

Successes

  • Increased faculty confidence and knowledge of QLT
  • Ability to scale much more than before, especially to those who had not yet been exposed to the instrument since its implementation on our campus
  • Positive reaction and feedback from participants, and their willingness to further their QLT training and participation despite the pressures and stress of COVID-19

Ideas/Lessons Learned

  • The pandemic tested our ability to adapt and respond quickly to changing priorities and multiple projects in a rapidly changing environment.
  • There is demand and "thirst" for this type of programming if there are direct benefits associated with it.
  • We decided that we wanted to develop the peer reviewer/mentor system and start developing a path for course reviews and certification.

Next Steps for QA Efforts 

  • Scale QLT training and implementation by faculty across all departments and disciplines. We will continue to include the QLT framework in our trainings, workshops, and annual summer eLearning Institute
  • Expand the number of faculty peer reviewers and QLT experts on our campus