2023 - 2024 — Cal Poly Pomona — The Impacts of Quality Assurance

Proposal Summary:  Our 2023-2024 project differed from our CHAI projects of the last three years.  Instead of focusing on training faculty, we investigated the results of previous work.  During the pandemic, dozens of faculty members participated in OCS trainings, but we had not assessed the effect on teaching practices and student success.  We surveyed faculty and interviewed academic leaders. We also initiated "sprint" reviews which were well received.

The Center for the Advancement of Faculty Excellence (CAFE) has completed its 5th year in support of the Cal Poly Pomona (CPP) Champions for Accessibility and Inclusivity (CHAI) project. 

CHAI serves as a model in CPP’s long-standing commitment to accessibility.  CHAI creates a faculty-to-faculty approach of promoting awareness of best practices and a focus on continuous improvement, with direct help from a team of expert instructional designers. CHAI continues to recognize accessibility heroes and to help other faculty become heroes. CHAI creates a network of empathy, both towards students who benefit from Universal Design for Learning and accessible course materials, and towards faculty who provide those materials.

The driving force of CHAI is creating excellent learning experiences for each student. We believe that the joy of learning, the freedom to grow, and the benefits of education are basic human rights at a university. CHAI embodies the values that accessibility and high-quality instruction are fundamental tools in making these rights a reality. 

In the 2023-2024 year, the objectives of CHAI were to:

1. Initiate a systematic campus conversation about the place of high-quality flexible format courses in CPP’s curricular landscape
2. Train CHAI staff in the use of the QLT rubric and perform informal “sprint” QLT reviews for courses participating in an enhancement project
3. Summarize campus involvement in OCS trainings since 2020 and assess the impact on teaching practices and student success.

Project Personnel:

• Faculty Lead: Zahra Sotoudeh, Aerospace Engineering 
• SQUaiR Fellow:  Maha Ghosn, International Business & Marketing
• Director of the Center for the Advancement of Faculty Excellence: Victoria Bhavsar
• Instructional Designers: 
  • April Dawn
  • Toni Davis
• Administrative Coordinator: Carla Tetreault

Campus Commitment 

We align with the California State University Online Course Services (CSU-OCS) goals to: 

• Apply principles of Universal Design for Learning, accessibility, and equity in online courses toward greater success of all students, including those with disabilities. 
• Create a network of faculty, staff, and administrators informed of the various quality assurance factors, tools, and resources that enable effective online teaching and learning.

CAFE assists faculty in making course materials accessible. Our goal is that every faculty member who works with us is introduced to quality assurance principles, receives support for using quality assurance resources for any course format, and is introduced and supported to use best practices for Universal Design for Learning, accessibility, and inclusivity in all instructional materials.

Staff training and “sprint” reviews in QLT

We tested a new concept that has succeeded well: “Sprint” reviews based on the Quality Learning & Teaching (QLT) rubric.

Sprint reviews focus only on the core QLT standards, plus one extra standard in QLT Section 5 (Facilitation and Instruction) focusing on inclusion and belonging that the faculty identify.  Two instructional designers or trained faculty reviewers divide up the core standards and spend no more than two hours reviewing the course.  Each standard receives only one review.  We do not score the standards as Met or Not Met, but give feedback regarding strengths and areas to work on.  Here is an example, identifying details redacted:

It works well to have the reviewers set a synchronous, and if possible face to face, time to complete the sprint reviews, rather than having a mere deadline to complete their part. Working together, they can compare and contrast findings, or draw each others’ attention to items. We then meet synchronously with the faculty member to share and talk through findings. Faculty can revise and request a new review if they want.

We sprint-reviewed a total of 10 courses: 6 were courses from a substantial project that focused on addressing long-standing student success issues. The other four were from a single College, based on a deans’ conversation described above. While 10 may not seem like many courses to review, it’s far more than we have ever reviewed in a single year previously.

At CPP, full QM and QLT reviews have not been productive. CPP has no fully online programs, no university requirement for training to teach flexible format courses, and only a few CourseMatch courses. Based on feedback from highly committed and interested faculty, we concluded that formal review is far too time-consuming and is not valued in the faculty reward system. In short, it doesn’t “pay” for CPP faculty to undergo full review and certification.

Sprint reviews, on the other hand, have been unanimously valued by the faculty who participated. Verbal responses during the post-review conversations, plus email thank-you’s, have been extremely positive. We have only had one faculty member formally request a new review for a revised course, but we are confident that this will pick up. We plan to offer sprint reviews broadly in 2024-2025.

We find that sprint reviews of Canvas sites are not adequate to fully reflect the reality of flexible format courses. We will suggest that faculty invite in-class visits along with sprint reviews. While such visits are time-consuming, they will provide much richer feedback to enhance the sprint reviews focused on the QLT rubric core standards, and will let our team get to know faculty and students better.

Goal 1:  Provide “faculty reciprocal peer coaching” (FRPC) for course improvement, using the QLT rubric as an improvement framework 

For the last two years, CAFE has offered “faculty to faculty reciprocal peer coaching.”  FRPC is the single most enthusiastically received program we have ever offered, with near 100% completion and positive faculty responses. FRPC involves structured partnerships between two or sometimes three faculty who offer collegial support, ideas, and constructive critique.  Faculty engage in training to provide effective peer-to-peer coaching and to select teaching strategies they want to apply. They then visit and observe each other’s class meetings or otherwise review course materials two to three times per semester, offering mutual feedback.  Pre-visit and post-visit conversations are included. Peer coaching is completely formative, i.e., non-evaluative. FRPC allows faculty to systematically work in partnership to use new teaching skills and/or hone basic skills into sophisticated praxis.   

Focusing on sections of 6 courses identified by campus leadership for intensive DFW and equity gap remediation, we will offer FRPC to 12 faculty. We will use the QLT rubric as a framework for identifying improved teaching strategies and other course improvements. These faculty will be offered access to their individual Course Equity Dashboards from the Chancellor’s Office. FRPC training will include a deep dive into the QLT rubric, an overview of the Course Equity Dashboards, training on strengths-based feedback, and support in scheduling visits.

Goal 2:  Perform more informal “sprint” QLT reviews  
We will focus on further sections of the 6 intensive DFW remediation courses, as in Goal 1.  The sprint format will focus on core QLT standards. We will ask instructors to identify at least one standard in QLT Section 5 (Facilitation and Instruction) as a focus area. The informal review team will work together on one course for 2 hours at a time, with a maximum number of 2 review sessions per course (4 hours total).  Sprint review teams will consist of CAFE staff who trained in QLT review in Summer 2023. 

We have had little success in enticing faculty to undergo full reviews due to the low pay and many hours required.  Sprint reviews help.  Although it might seem disappointing that in our 2023-2024 project only five faculty of about 40 (~12%) who are engaged in the high DFW remediation courses are willing to do sprint reviews, this rate of response is actually very high compared to a near-zero response rate for offers of full review. 

Goal 3:  Systematically assess faculty reciprocal peer coaching and sprint reviews 
The OCS faculty lead/SQuAIR fellow will assess the outcomes and effects of FRPC and sprint reviews: 1) Completion of planned activities and description of faculty participants, 2) Faculty valuation of and satisfaction with their experiences, and 3) Extent to which faculty intend future changes and/or do make current changes within the grant time period. Evaluation apart from assessment will identify ways to improve the program.

Via recordkeeping and faculty surveys or other qualitative feedback instruments, we will investigate faculty perceptions of the effectiveness of FRPC including initial training, pre/post conversations, and class visits or review of course material.  Other questions will investigate the utility of Course Equity Dashboards, and for those who participate, the benefits of sprint reviews.  We will ask faculty to describe the support they need.  In Fall 2024, faculty will recommend program improvements for Spring 2025, and in spring recommend improvements for future.  

Prior to the pandemic, CPP faculty vastly preferred fully face-to-face courses.  The pandemic demonstrated to many faculty that flexible course formats can work.  The campus has returned to face-to-face courses, but there are still many more flexible format courses offered (>25% of all campus courses, compared to <10% pre-pandemic).  

Figure 1 shows differences between course formats in Fall 2019 and Fall 2023.

Figure 1 conceals significant differences between academic Colleges in Fall 2023: The College of Business Administration had only 38.39% of courses in person, while the College of Engineering had 94.79% in person.

Students preferentially enroll in flexible format courses and many faculty, especially lecturers, also prefer them. Remote learning is not always conducive to student success, but it can certainly improve to be a valuable tool in student progress to degree. Remote learning must be carefully conceptualized for many disciplines to align with CPP’s paradigm of “hands-on learn-by-doing.”

Our campus is at an inflection point in our relationship with online and hybrid course formats. In quick succession, we converted to semesters (Fall 2018), entered the pandemic (Spring 2020), converted to Canvas (Fall 2021), and experienced the influence of generative AI (Spring 2023). The campus community is just now reaching a period of relative calm in which to carefully consider how flexible course formats contribute to CPP’s curricular landscape, especially in view of the ongoing crucial national conversations concerning race and equity, the nationwide drop in college enrollment and resulting reduced revenues, the recent action of the UC in banning fully online degrees, and other “climate changes” in education.

We held interviews with 7 of 8 of the CPP College deans regarding their vision for flexible format courses.  This was especially timely as CPP had four new deans as of Fall 2023.  We also interviewed the interim AVP for Academic Programs.  The interviews served the dual purpose of gathering input and informing academic leaders about the numerous factors, tools, and resources that enable effective online teaching and learning. 

Summary of Academic Leader Interviews

Overall, academic leaders would prefer to see a more general return to face to face encounters except in colleges such as Engineering where courses have returned to pre-pandemic levels of in-person learning.  However, academic leaders also recognize the critical importance of providing flexible format courses for the Cal Poly Pomona student body.  They struggle to balance the need for flexible courses with the difficulty of ensuring quality of those courses when training is not mandatory and course reviews are not robust.

Deans recognize that students prefer flexible-format courses, but academic leaders are concerned that lack of face to face classes significantly reduces students’ learning, particularly in “power skills” such as communication, problem-solving, teamwork, time management, etc. There is also concern that an over-representation of introductory and general education classes in remote formats undermines the perceived importance of these courses. Asynchronous classes are perceived as “easy” and then have reduced success rates.

Deans recognize that faculty preference for hybrid courses is strong, and they perceive that this preference is due to a desire to limit on-campus presence for reduced commuting and other benefits of remote work.  Deans are generally concerned that quality assurance for flexible format courses is lacking, with no in-depth peer review performed.  In some colleges in which research has been heavily incentivized, tenured faculty show little motivation for teaching quality; in others, faculty burnout and disengagement is a serious problem. In addition, deans are concerned that the preference for remote courses has negative follow-on effects for faculty engagement overall, placing faculty at risk for leaving Cal Poly Pomona due to lack of a satisfying academic community. Deans are searching for ways to provide motivation and incentive for faculty to deeply re-engage with course quality and with the campus community.  

Approximately 250 CPP faculty have engaged in OCS QM and QLT trainings since 2020, most during the summer of 2020 and 2021 because federal funding was available to pay generous stipends for participation.  Our SQuAIR fellow analyzed participation data, faculty satisfaction responses, level of persistent change to courses, and faculty motivation for continuous improvement.  We have not yet completed student outcomes comparisons.

Participation

• In 2020 and 2021, 241 individual faculty completed OCS trainings:
  52% tenure line faculty and 48% lecturer faculty, indicating a slight over-representation of tenure-line faculty as a proportion of faculty on campus 
• There were equal numbers of men and women faculty participants, indicating a slight over-representation of women as a proportion of faculty on campus 
• Participation from Letters, Arts, & Social Sciences and the College of Business Administration were slightly over-represented as a proportion of faculty on campus

Faculty satisfaction with OCS trainings

66 survey responses of N=241; response rate 27%, with 59% of responses from tenure line faculty.  Quality Assurance courses were satisfactory:

• 87% report that QA courses were relevant
• 86% report that QA courses were a good use of time
• 86% report that QA courses were beneficial to their own classes

Changes to courses

Motivating factors for continuous improvement