Quality Assurance for blended and online courses

2018-19 — San Diego State University — Measuring Student Engagement with PlayPosit (In-Video Quizzing) in a High Enrollment Cinema Course

Proposal Summary:  Throughout the past academic year, Cinema as Art (TFM160) moved from a face to face modality to both face to face and online contexts. This high enrollment, lower-division film course employs video content within the course design, and includes a focus on the development of critical thinking skills. Professor Voytilla piloted PlayPost, an in-video quizzing software, to increase students' comprehension of, and engagement with, the cinematic course content. In addition to introducing PlayPosit, Voytilla maximized the impact research by engaging in QA trainings, and received more than 800 responses to the student Quality Assurance and Perceptions of PlayPosit survey.

Campus Quality Assurance Goals and PlayPosit Project Introduction

Campus Goals for Quality Assurance

  • ITS continues to partner closely with faculty to assess and build quality instructional resources within all courses, specifically those participating in the QA training and programs. 
  • In its first year of the SQuAIR program, ITS departed from the larger BOLT faculty consultation model, opting instead for a close faculty focus on student engagement within a high enrollment, lower division course in the arts. This approach allowed for a thorough assessment of a new technology aimed at increasing student engagement in a course that is taught in both face to face and online modalities. 
  • San Diego State University is committed to improving graduation rates for all students in alignment with the CSU Graduation Initiative 2025 (GI 2025) by increasing retention, reducing barriers to graduation, incorporating technology into the learning environment, and focusing intently on student achievement and the fulfillment of student learning outcomes.
  • The demand for fully-online courses at SDSU continues to grow, with the campus understanding that offering courses has the greatest impact for students when the academic rigor of the course is maintained regardless of the course's modality.

Quality Assurance Faculty and Research Leads

  • Stuart Voytilla, TFM160 Faculty Member, ITS Faculty Fellow, Professional Studies and Fine Arts
  • Maureen A. Guarcello, Research, Analytics, and Communications Specialist

Supporting Campus Partners

  • James Frazee, Director, Instructional Technology Services
  • Jennifer Imazeki, Director, Center for Teaching & Learning
  • Mark Laumakis, ITS Faculty Fellow, College of Sciences
  • Division of Academic Engagement and Student Achievement
  • Analytic Studies and Institutional Research

Campus Commitment Toward Sustainability of QA Efforts

The SQuAIR team continued the BOLT program's collaborative efforts, working across ITS, and the campus to:

  • Promote online teaching and learning effective practices
  • Provide opportunities for student engagement and interaction
  • Support diverse learners of various skill levels and abilities
  • Establish a sense of student community and inclusion online
  • Achieve measurable student learning outcomes

Summary of Previous QA Accomplishments (2017-18)

  • SDSU trained a cohort of 41 faculty members on the QOLT rubric and ideal practices for course design (BOLT and QA)
  • Delivered accessible instructional materials workshops, small group trainings, and individual consultations
  • Improved materials accessibility in 10 large enrollment courses
  • Partnered with faculty to provide course-based UDL exemplars 
  • Conducted more than 50 consultations focused on QOLT principles
  • Certified 1 QOLT Reviewer (SDSU Instructional Designer)
  • Created accessible instructional materials guides 

What is PlayPosit?

PlayPosit is an in-video quizzing tool that sits on top of video content. Faculty upload their video content to the PlayPosit website and create a "bulb." Once the content is ready, faculty can create questions or point out important concepts, as the student is viewing the content. In Voytilla's course, he used video clips from movies explored in a longer format during class. These "Closer Look" exercises reinforced key concepts within the films. 

Each clip was between 2-5 minutes in length, with pauses and quiz questions at critical points throughout the diverse film content.  

In the case of Voytilla's Cinema as Art class, students watched 5 video clips from: 

  • One Man Band (4.5 minutes)
  • Citizen Kane (5 minutes)
  • Rear Window (3 minutes)
  • Silence of the Lambs (2.5 minutes)
  • Eternal Sunshine of the Spotless Mind (3.5 minutes)

In the image below, note the three elements; each was present in all of Votilla's course PlayPosit bulbs: left-hand question bar, captioning, and the progress bar marking where the checks for understanding occur throughout the video. 

In this clip, the question posed is, "When we approach the analysis of a shot, the first question we should ask is: (select one):

  1. What was the director's intention in this shot? 
  2. How was the shot filmed? 
  3. Was the film a commercial success? 
  4. What is happening in the plot at this moment of the shot? 

PlayPosit screen capture from Eternal Sunshine of the Spotless Mind - with incorrect answer selection and student guidance

PlayPosit screen capture from Eternal Sunshine of the Spotless Mind - with correct answer selection and notification

Accessibility/UDL Efforts

Each of the 5 PlayPosit films were captioned (see screen capture above), and Professor Voytilla has also begun investigating ways to make PlayPosit questions accessible using Google Translate and mobile devices. These recommendations are also heard by the PlayPosit developers. 

Development of Campus QA Resources

  • PlayPosit evaluation and research design
  • Instructional design recommendations for PlayPosit video placement within future courses
  • QA and PlayPosit surveys built within the Qualtrics environment
  • Report and assets from this 2018-19 SQuAIR project

Dissemination of QA Efforts - White Paper, Campus Communications, and Sustainable Research Design 

The SQuAIR project enabled ITS and Professor Voytilla to invest optimal time and resources to conduct a thorough investigation into the student impact of PlayPost in-video quizzing in TFM160. While these outcomes are not generalizable, the instructional design implications and research design will serve as a strong foundation for other faculty who are considering the use and evaluation of PlayPosit in their courses. 

Further, this project will inform a Scholarship of Teaching and Learning effort that Voytilla will remain engaged in, as well as a white paper, that ITS will co-author with PlayPosit, to ensure these findings and recommendations are disseminated to the greater education community. 

Quality Assurance and PlayPosit Pilot Results

Training Completion

Introduction to Teaching Online Using QLT (Q1)

Reviewing Courses Using the QLT Instrument (Q2)

Course Peer Review and Course Certifications

By introducing QA training and the Student Quality Assurance Impact Research approach to TFM160, SDSU has grown the existing QA community in terms of faculty, students, and by moving the focus to the College of Professional Studies and Fine Arts, increased the reach of QA and the opportunities for peer review and further engagement. 

Student Quality Assurance Impact Research (SQuAIR)

Professor Voytilla first utilized PlayPosit in-video quizzing during the summer of 2018. After some positive feedback from students, Voytilla furthered the use of PlayPosit in his fall and spring classes (18-19). 

As Voytilla built PlayPosit into the homework and recall assignments in his class, he posed the following questions and expressed an interest in how this new way of engaging students may impact the Quality Assurance elements within the course.

Research Questions: 

  1. Is there a measurable difference in students' understanding of course material based on how PlayPosit is used in TFM160?
  2. Does the instructors' use of PlayPosit facilitate critical thinking skills in students?
  3. Can a high enrollment lower division GE course cultivate active learning?

Student Quality Impact

Cinema as Art is a high enrollment course with more than 850 enrolled students in the face to face and online sections combined. Data collected for this study included: 

  • Student QA and PlayPosit Student Perception Survey feedback
  • Student course performance data
  • Course design information (syllabus, Blackboard course design, etc.)
  • Regular faculty consultations regarding nuanced course features (grading, timing, classroom culture, etc.)
  • PlayPosit video and quiz engagement patterns (time spent, attempts, performance, frequency, etc.)

Development of Campus QA Resources

  • PlayPosit evaluation and complementary QA research design
  • Instructional design recommendations for future PlayPosit video use 
  • QA and PlayPosit surveys built within the Qualtrics environment
  • Report and assets from this 2018-19 SQuAIR project

Student Perceptions of PlayPosit

Overall, students reported that PlayPosit was a valuable use of their time. An NVivo sentiment analysis of the 408 open response comments showed that 61% of student respondents had a "Very Positive" experience. Some of the student feedback included the following quotations: 

"I actually enjoyed how the movie would stop and ask a question. Even if I got the question wrong I was able to understand why because it provided reasoning behind the answers and you could also compare the image on the screen to the answers."

"I really enjoyed the PlayPosit exercises. I thought they were a very useful review tool for my understanding of the concepts and for the quizzes."

There were also some detailed recommendations for course design with PlayPosit: 

"The concept was great but there should be more explanation. The beginning should help teach the students what the video is going to be about and what questions there are. Therefore, the students are prepared for what is to come."

 Student comments in response to: "What did you think of
your PlayPosit experience?"

 

Student PlayPosit Engagement

Research observing student viewing behaviors indicate that optimal video content length resides near the 6 minute mark (Guo, P. J., Kim, J., & Rubin, R., 2014), however this cinema course requires longer engagement with the material. The TFM160 activity plot below demonstrates student viewing and quiz completion medians above 10 minutes, and in the case of the classic film Citizen Kane, more than 17 minutes. These times vary based upon the length of the video clip, as well as the number and rigor of the questions posed within the video.

This finding is exceptional, in that it demonstrates 2-3 times more engagement time with the video content (2-5 minute video clips), by making the student an active participant, rather than a passive viewer.

Cinema as Art PlayPosit video engagement

Student Course Performance

Students are generally successful in TFM160, though poor study habits and procrastination prove to be barriers for many first year undergraduates. Below are grade distributions for the Spring 2019 semester. 

Spring 2019 Cinema as Art grade distribution

Quality Assurance and PlayPosit Research Findings

In response to the three research questions at the outset of the SQuAIR project, we encountered a number of answers, and more questions to investigate further. 

Quality Assurance Research

Even with the introduction of a new instructional technology, Voytilla's (more than 800) Student QA survey respondent ratings yielded scores greater than 5 on the 6 point scale. The following two items were especially telling in terms of the delivery and reinforcement of the PlayPosit tool within the course design. This finding also reinforces the potential for faculty to implement new instructional technologies in their course designs without evaluative penalties from their students, when done well and with purpose. 

"The instructor provided specific, well-defined, and measurable learning objectives. I understood what I was supposed to accomplish both weekly and by the end of the course. For example, each week there were specific learning goals and I knew exactly what I was supposed to learn/accomplish (e.g, there were bulleted lists of activities to complete each week)." Student Rating: 5.23

"The instructor used a variety of online technology tools to engage me and encourage me to interact with others in the course and I felt the tools used supported the course objectives.  Examples include, but are not limited to, in video quizzing (e.g., PlayPosit), online collaboration tools (e.g., Google Docs), social media tools (e.g., Twitter)." Student Rating: 5.03

PlayPosit Research

QIs there a measurable difference in students' understanding of course material based on how PlayPosit is used in TFM160?

A. At face value, yes. Using students' quiz and exam scores as a measure of understanding the course material, we found that yes, students who engaged in the PlayPosit content also received higher scores on their exams. However, the attribution to PlayPosit cannot be confirmed until further study is conducted. This may include pre and post video quizzes, item level analyses, specific recall activities, controlling for already high performing students engaging with the videos, etc.

Q. Does the instructors' use of PlayPosit facilitate critical thinking skills in students?

A. According to students' survey responses, Yes. Many of the students discussed the value of the questions and prompts within the videos, but some of them addressed the exercises at a much deeper level. For example: 

"I thought it was interesting because I had seen this movie last semester in my philosophy class, so watching it first from a philosophical point of view, then in this class watching it from an analytical point of view was very interesting."

 Q. Can a high enrollment lower division General Education course cultivate active learning?

A. This is the hope, though it cannot be demonstrated through the outcomes of this study. PlayPosit, virtual and immersive reality, personalized learning courseware, authentic assessment, mobile-ready learning management systems, and catalysts for student dialogue and community pose tremendous opportunity for active learning in high enrollment courses. Quality Assurance research and student feedback loops help to inform progress and effectiveness. 

Next Steps for Quality Assurance Efforts 

In terms of the 2018-19 SQuAIR project, the PlayPosit instructional design implications are the most actionable. Going forward, we have the data to support specific instructional design recommendations for faculty who integrate PlayPosit into their course designs. These include tips on the placement of the videos within the course, optimal placement of questions and checks for understanding within the videos, and ways to reinforce the content and PlayPosit activities that took place outside of the class context, during class time. 

Looking ahead to the 2019-20 SQuAIR project, SDSU will launch a ~1,000 student, 22 faculty member Canvas pilot in Fall 2019. The year-long pilot affords faculty members the opportunity to reassess instructional design and the structure (or blueprint) of their courses. Specifically the ways in which faculty use data to inform student success interventions and communication. SDSU's SQuAIR efforts will be trained on this project, combining the expertise of Instructional Designers and the Academic Media teams.