2023-2024 — Sonoma State University— OCS Grant Report

Proposal Summary: At Sonoma State University, we proposed to facilitate a QLT Faculty Learning Community with 10 faculty members to guide participants through course redesign and QLT External Certification efforts. We also proposed to conduct a HyFlex working group with 10 faculty members to increase campus exposure to the HyFlex modality and to increase faculty interactions with HyFlex capable classrooms on campus. Finally, as a part of the SQuAIR Research Program, we sought to investigate pre-QLT prepared courses using a selection of students and a survey that was designed by the Center for Teaching and Educational Technology (CTET). 

1. We successfully completed the QLT FLC with eight faculty members. Three of the courses have been QLT Certified, and five additional courses have been submitted for QLT Certification and are pending review.
2. We successfully completed the HyFlex Working Group with nine faculty members, and received recommendations from participants for additional campus investment in HyFlex course offerings and HyFlex support.
3. We successfully complete Phase 1 of the SQuAIR Research Project, entitled "QLT Instrument Validation based on Student Response Data" wherein we evaluated a series of faculty provided course sites with actual students using a modified 32-item version of the QLT instrument.  Preliminary data indicated that students viewed the importance of QLT indicators in ways that are likely different from faculty and the preparers of the QLT rubric, and secondly that students' perceptions may align with well established information processing framework from  Kahneman & Tverskey (1982) in terms of System I and System II thinking that address differences in heuristic vs. central processing judgments about information.  We intend to dive deeper into our preliminary findings in 2024-25 with a fuller replication study with a campus-wide student sample.

Campus Goals for Online Course Services

The focus of our proposal was to 1) support our diverse student body by increasing access to online, hybrid, and HyFlex course offerings, 2) in conjunction with increasing online, hybrid, and HyFlex course offerings, ensure that faculty members receive training and support in developing accessible, engaging, and high-quality online learning environments via QLT Certification, and 3) to investigate pre-QLT certified courses with the goal of operationalizing QLT indicators as measures of student importance and success via pilot study exploring student attitudes using an operationalized survey version of the QLT instrument with student raters.

1. Facilitate HyFlex Faculty Working Group.
2. Support 10 new faculty to receive QLT Certification.
   
3. SQuAIR Research Program.

Online Course Services Lead(s)

• Justin Lipp, PhD, Director, Center for Teaching and Educational Technology
• John Lynch, PhD, Assistant Director, Center for Teaching and Educational Technology
• Kristin Denver, Psy.D, Senior Instructional Designer, Center for Teaching and Educational Technology
• Kyle Falbo, LMS Adminstrator, Center for Teaching and Educational Technology
• Gina Baleria, PhD, Assistant Professor of Journalism

Supporting Campus Partners

Center for Teaching and Educational Technology, Information Technology, Disability Student Services, Office of the Provost, Academic Affairs. SSU Faculty and Staff subject matter experts.

Campus Commitment Toward Sustainability of OCS Efforts

• Ongoing support for Level 1: Canvas Design Foundations and Level 2: Online Facilitation Fundamentals programming
• Increase the number of QLT Certified courses
• SQuAIR Research Project
• Graduate Initiative support increasing accessible course materials via CTET's Accessible Content Partnership
• Regular training opportunities to support faculty members in using course design best practices, using innovative teaching tools, developing accessible course materials, gradebook and rubric support, UDL, inclusive teaching practices, and a variety of other topics

Summary of Previous OCS  Accomplishments

• Completed a HyFlex Pilot program in Summer/Fall 2022
• Contributed to the Academic Master Planning initiative involving a number of specific objectives related to online and hybrid education
• Completed cohorts of Level 1 and Level 2 in Fall 2022, and redesigned the self-paced support class as an ongoing resource for all faculty
• Developed the campus QLT Certification webpage

Dissemination of OCS Efforts— Conferences 

• We participated in the DET/CHE and Educause Annual Conferences this year 

Training Completions

Course Peer Review and Course Certifications

SSU has established a workflow and QLT website to support faculty efforts for those seeking QLT Self-Review and/or External Review. The Level 1: Canvas Design Foundations and Level 2: Online Facilitation Fundamentals programs are designed based on the QLT Rubric exemplary practices and serve as prerequisites for the QLT certification process. In these programs, faculty receive the benefit of peer and facilitator feedback during the course design/redesign phase.

SSU has successfully completed three additional QLT Certified courses this year, with six additional course reviews in the queue. This is a record high of nine potential QLT Certification for Sonoma State University within the 2023-2024 academic year. 

Completed:

• KIN 350: YoungMin Chun (February 20, 2024)
• SOCI 263: Kyla Walters-Doughty (March 19, 2024)
• Psy 302: Missy Garvin (April 12, 2024)
• CCJS 450: Caitlin Henry (Aug 8, 2024)
• BUS 365: InHaeng Jung (July 31, 2024)

Pending Launch:

• ARTS 105: Sena Clara Creston (launch July 1)
• ANTH 341: Alexis Boutin (launch July 1)

In Queue:

• PSY 325: Kristin Denver (application submitted 4/25/24)
  
• 
  
• COUN 540: Cecile Bhang (application submitted 5/6/24)

Student Online Quality Assurance Impact Research 
SSU's project began with a simple question: How do students view the indicators in QLT rubric in terms of utility, importance, and ability to identify within online courses.  As such, our SQuAIR research team began with a review of the QLT instrument, reducing from 50+ down to 32 indicators for students to evaluate.  Items that were considered as duplicative or which students would be unlikely to be able to rate were removed from the final survey version.  A group of 8 SSU faculty agreed for the research team to utilize their course sites as the basis for a student-conducted rating of course content.  A team of 4 student raters completed the 32-item survey for each course site, yielding a data set of repeated measures for the indicators.

Our analysis showed several interesting preliminary trends.  Our preliminary goal was to assess whether students’ ratings of the online courses were consistent in terms of understanding of the QLT items (construct validity). 

• Fleiss’ Kappa (Range: 0.447 to 0.668, across 31 items), indicating moderate agreement among raters (desirable as we need some discriminant variability to assess which items provide statistical explanatory power) .
• Kendall’s W (strength of relationship among multiple raters): =0.506, (Chi-Sq = 486.21, df = 30, p < .001), again indicating modest agreement.
  

Principal Components Analysis (PCA) & Exploratory Factor Analysis (EFA):
We used PCA and EFA techniques to  statistically examine the dataset for latent structure a to look for similar variability in ratings, and apply potentially a theoretical framework

• PCA: Suggested a preliminary 7-factor overall latent conceptual framework (Variance Explained 86.71%).  We were experiencing issues with multicollinearity in data, which required dropping additional items with high % agreement, to look for more discriminating factors that could still explain sufficiently high degree of variance.
  It is also necessary to reduce total size of instrument for Phase 2 when we plan to replicate this study on a much larger scale and need a parsimonious instrument to complete the data collection.

EFA: After many iterations, 2-factor solution determined (62.54 % of variance explained), 13 total items.  Factor 1 included: Course indicates what successful participation looks like, Syllabus tells students what to expect/learn, Technology required for course, Instructor stating how they will interact w/ students, Course materials organized chronologically, Course organizational consistency, How to contact instructor w/ questions, Syllabus indicating req vs. optional materials, Grading rubrics/explanations easy to understand.  Factor 2: Statement on campus policies about cheating/plagiarism, How soon students can expect feedback from instructors, Instructional Materials presented in multiple formats.

Interpretation and Results:

Our inductive review of the themes' statistical groupings tentatively suggests some alignment with the work of Kahneman & Tversky on heuristic (factor 1) versus stematic (Factor 2) information processing framework.  If the tentative theoretical explanation holds, it could  allow us to develop/test hypotheses in line with this well-validated information processing theory to further explore student responses to QLT ratings.  It is entirely possible that the data may represent an artifact of how the survey was constructed (will need future survey to rotate item ordering, to eliminate testing effects).
For Phase 2, we will continue the item analysis, refine and finalize a short-form survey, and develop experimental manipulations (e.g., 2x2 factorial design) to test the presence/absence of QLT indicators in a larger scale replication study.

Development of Campus Online Course Services Resources

• Level 1: Canvas Course Design Foundations
• Level 2: Online Facilitation Fundamentals
• Online Course Design Template - Canvas Commons
• All Instructors - Canvas Self-Paced course - Canvas Commons
• QLT Webpage
• Online and Hybrid Teaching webpage
• HyFlex Training Course (and Showcase video recordings)
• AI in Education resources webpage

Accessibility/UDL Efforts

• Accessible Content Partnership (Content Remediation)
• Canvas Accessibility Tools
• ALLY/Canvas integration
• UDL and Accessibility content in Level 1 and Level 2 programming
• UDL, Accessibility, and ALLY workshops offered by the Center for Teaching and Educational Technology
• QLT self-review and external certifications
• Partnership with DSS
• ATI/UDL group - SSU is continuing its annual commitment to universal accessibility in the LMS through a sizable campus effort toward remediation of instructional materials with a dedicated team to take the burden off  of faculty.
  

Next Steps for OCS Efforts 

• Facilitate QLT Faculty Learning Community with eight participants
• Facilitate Online Program Development Institute to prepare up to three programs to be fully aligned with QLT "CORE 24" objectives
• Complete Phase 2 SQuAIR research project