Developing Novel Hardware Controllers for Hands-on Virtual and Augmented Reality Science Labs
Developing Novel Hardware Controllers for Hands-on Virtual and Augmented Reality Science Labs
Kambiz M. Hamadani
Associate Professor, Dept. of Chemistry and Biochemistry, CSU San Marcos
- General, Organic, and Biochemistry for Life (CHEM 105/105L): Covers the basic principles of general, organic, and biochemistry as applied to the biochemistry, pathophysiology, pharmacology and nutrition of human body systems. This course is intended for students pursuing degrees in health-related topics such as nursing or kinesiology.
- General Biochemistry (CHEM 341): Intended for biological science, biotechnology, and chemistry majors. The objective of this course is to introduce the student to the fundamental concepts and language of biochemistry, and to the principles that govern the structure and behaviour of water, amino acids, proteins, carbohydrates, lipids, and nucleotides. Also covered are basic enzyme kinetics, mechanisms of enzyme action, and enzyme regulation; the principles of bioenergetics and metabolism; the generation, use, and storage of metabolic energy; the synthesis and degradation of bio-molecules, and the regulation of metabolism.
- Biochemistry Lab (CHEM 351L): The goal of this laboratory course is to teach the following aspects of experimental biochemistry: experimental design and execution, and data recording, analysis, interpretation and presentation.
Project Abstract: The present project seeks to provide students with realistic tactile feedback in virtual, mixed, or augmented reality lab experiments. We are designing and testing a series of prototype hardware tools which augment the functionality of standard laboratory equipment and allow them to pass into the virtual environment while still allowing students to manipulate and sense them as they would in a traditonal hands-on lab. The novel hardware includes augmented/virtualized pipetteman, test-tubes, tube racks, tip racks, beakers, and graduated cylinders. We are also making two virtual reality lab modules using Unity3D to showcase the advantages of truly "hands-on" virtual reality labs.
Keywords/Tags: Virtual Reality, Mixed Reality, Virtual Labs, Augmented Reality, Chemistry, Biology, Education.
Instructional Delivery: These labs could be used as a supplement in lecture courses which don't have associated labs (CHEM 341), as visually-rich complements to existing take-home or in-class lab experiments which are limited to providing only macroscopic visualizations of chemical/physical phenomena (CHEM 105/105L), or as pre-labs for existing lab activities which are time-intensive and prone to errors (CHEM 351L).
Pedagogical Approaches: Augmented Reality, Virtual Reality Lab Experiences/Activities.
About the LIT Redesign
Background on the Redesign
My previous course redesigns of CHEM 105/105L (General, Organic, and Biochemistry) and CHEM 341 (General Biochemistry) using commercially available Labster virtual lab improved student learning/engagement (CHEM 341) and addressed resource constraints (CHEM 105/105L) that were creating bottlenecks for multiple health-science programs on my campus. However, a major issue with these virtual lab tools was their lack of authentic tactile sensory feedback to the user. Enabled by recent advanced in Virtual and Mixed reality (VR/MR) hardware and software (see below), here I expand on the insights I’ve gained from my past redesign efforts and begin to generally resolve this issue by creating and providing proof- of-principle for a novel mixed/virtual reality approach which offers an authentic tactile sensation of doing hands-on wet-lab biochemistry research by virtualizing various lab tools as the gaming industry has done with guns, ping pong paddles, guitars, and baseball bats. I will test this approach ultimately not only in CHEM 105/105L and CHEM 341 but also in an advanced Biochemistry lab course (CHEM 351L). This is a proof-of-principle study which examines the feasibility of a generalizable and novel approach to scalable, safe, low-cost and engaging hands-on science education which is well-suited to a wide range of learner types and student populations. We will build two VR Lab Modules: one on pipette calibration and the other on stock solution generation to illustrate the capabilities of the technology
About the Students and Instructor
- GOB Chemistry (CHEM 105/105L):
- Pre-nursing majors: GOB Chemistry is a requirement for CSUSM's pre-nursing majors. Pre-nursing students are typically highly motivated and do very well. Many Nursing programs nationwide have recognized the value of virtual or hybrid labs within their own clinical curricula yet they still also greatly value hands-on training experiences.
- Kinesiology majors: GOB Chemistry is a requirement for all kinesiology majors. About 50% of CSUSM kinesiology majors are pre-physical therapy. These students typically are highly motivated because they need to do well to enter physical therapy programs following graduation. PT graduate programs typically require two semesters of chemistry with lab. AT CSUSM the combination of CHEM 150/150L (General Chemistry for science majors) and CHEM 105/105L is used to satisfy this requirement for Pre-PT majors. For the other ~50% of Kinesiology majors that take CHEM 105/105L, a hands-on lab is actually not necessary and a virtual or hybrid lab would suffice. Roughly 45% of Kinesiology students are seniors that have already taken CHEM 150/150L while roughly 30% are a freshman who haven't. The diversity of student backgrounds makes this course a challenge to teach. Keeping students engaged with the content is a major barrier. The freshman have had minimal hands-on wet-lab experience prior to the course and are also generally not the best at math. Providing these students with authentic hands-on laboratory experiences which effectively illustrate the dense content presented in lecture, the application of math to chemistry, and how science gets done (creating hypotheses, designing experiments, collecting data, analyzing data, making conclusions, and presenting findings) are key learning goals.
- General (non-majors) Biochemistry (CHEM 341):
- Biological Sciences Majors: CHEM 341 is an upper-division elective for the molecular and cellular biology concentration for this major. Upon entry into CHEM 341, these students have often not had any general or organic chemistry in over 1.5 years. They have however had considerable exposure to advanced topics in biology. Unfortunately, because they haven't had biochemistry their understanding of biology is not fully grounded in chemical principles. Using advanced molecular visualization to show these students how chemical principles underlie biological processes and phenomena is a key learning goal.
- Biotechnology Majors: CHEM 341 is a required course for the B.S. in biotechnology. Again students often take general and organic chemistry early and wait until their senior year to take CHEM 341. This student population also takes numerous non-science business courses and often lacks the breadth of exposure to more advanced biological concepts that the biological science students have. Getting them more hands-on training is a key learning goal.
- Chemistry Majors: CHEM 341 is a required course for the B.S. in chemistry. This population of students has an extensive knowledge-base covering basic chemical principles and concepts as well as more advanced topics. However, as they are not required to take any basic molecular biology courses they lack the means for applying these principles to living systems in general and biomolecules in particular so they often do poorly in CHEM 341. The highly polarized difference in background knowledge for the chemistry and biology students represents and huge challange in this course. Providing the chemistry students with opportunities to exposue to biological concepts them with
- Majors Biochemistry Lab (CHEM 351L):
- Biochemistry Majors: Unfortunately, CHEM 351L is the only biochemistry lab course provided to biochemistry majors (all other labs are either chemistry or biology labs). There is thus significant pressure to pack in as much biochemistry as possible. Recently, changes were made which reduce the number of contact hours and the variety of labs that are offered exacerbating the problem. While Hands-on VR labs are certainly no substitute for hands-on training, as pre-lab activities they may provide an effective means for students to be able to do more with less face-to-face time while still absorbing the content effectively.
- Chemistry Minors: Many biology majors complete a minor in chemistry rather than majoring in biochemistry. These students also take CHEM 351L.
Impact of Student Learning Outcomes/Objectives (SLOs) on Course Redesign
- GOB Chemistry (CHEM 105/105L) SLOs... Students should be able to...
- recognize and differentiate between ionic and covalent compounds
- write the names and formulas of compounds commonly encountered in the field of health care
- define the various measures of concentration and be able to correctly use them in calculations
- describe the concepts of pH, buffers, the differences between acids and bases, and be able to calculate the pH or the hydrogen ion concentration of a solution given relevant information.
- describe the properties of gases, the relationships between gas pressure, volume, and temperature, and also be able to use them correctly in calculations
- correctly use significant figures in calculations.
- recognize the most common organic functional groups
- describe how properties of functional groups dictate the chemical and physical properties of organic compounds (such as drugs, and biological macromolecules) encountered in the field of healthcare
- identify the classes of organic molecules that play important roles in human health.
- recognize and describe the chemical and physical properties of carbohydrates, lipids, nucleic acids, and proteins
- describe the various biological functions of carbohydrates, lipids, nucleic acids, and proteins.
- Integrate their knowledge of general, organic, and biochemistry to describe and analyze relevant issues
in the field of health care.
- perform introductory laboratory techniques.
- document experimental observations, analyze experimental results, calculate relevant experimental variables, and write clear laboratory reports summarizing findings.
- describe the chemical principles on which chemical and biochemical experiments are based.
- describe how the structure and properties of matter at the sub-atomic, atomic, molecular, and macromolecular scales can be measured and used to report on issues relating to human health.
- General Biochemistry (CHEM 341) SLOs... students will demonstrate their understanding by recalling, explaining, applying, and integrating their knowledge of:
- the various interactions and forces that help maintain the structures of the four major classes of biological macromolecules (proteins, nucleic acids, carbohydrates, and lipids).
- the basic principles of thermodynamics and kinetics as they apply to biochemical processes.
the metabolic pathways used by biological systems to extract energy from their surroundings and how high energy molecules and electrochemical gradients are used to store this chemical potential energy for future use.
how conformational changes are used for biochemical sensing, response, signaling, regulation, and catalysis in living systems.
Major's Biochemistry Lab SLOs... student will be able to:
design biochemical experiments
- execute experimental biochemical procedures.
- acquire and record data using biochemical instruments.
- analyze experimental datasets using microsoft excel.
- interpret experimental datasets and make well-founded conclusions.
- generate written lab reports or presentations of experimental results.
Alignment of SLOs With LIT Redesign
- The SLOs underlined above will be targeted in the pipette calibration and stock solution generation VR Lab Modules. In CHEM 105/105L students do not currently use a pipetman or make their own stock solutions. The pipette calibration VR lab module will provide students with an opportunity to learn how to use a pipettman. Similarly, the Stock Solution Generation VR Lab Module will reinforce a wide range of content from both the lecture and lab.
- In CHEM 341, students often have trouble recognizing the practical applications of acid-base chemistry and chemical equilibria in biochemistry. Using both the Pipette Calibration and Stock Solution Generation VR Lab Modules we will reinforce these concepts in a concrete and practical manner by having students make buffers of defined pH and ionic strength and carry out calculations using the Henderson-Hasselbalch equation.
- Pipette calibration was recently removed from the CHEM 351L lab curriculum for lack of time. The Pipette Calibration VR Lab Module will reintroduce this activity in a virtual format. Similar to CHEM 105L the Stock Solution VR Lab Module will allow students to virtually make stock solutions that are provided to them in the actual CHEM 351L labs.
Assessments Used to Measure Students' Achievement of SLOs
- Outside of the VR Lab Modules, standardized multiple-choice assessments given to students in previous semesters (control data) will be given to experimental student populations within quizzes and exams to gauge the impact of the Hands-on VR Labs on SLOs.
- Within the pipette calibration VR Lab Module students will be tasked with determining whether any of three different pipetteman they are given are out of calibration and if so by how much and in what direction. They will also determine the precision of their pipetting.
- Within the Stock Solution Generation VR Lab Module students will be tasked with making a single or multi-component buffered solution within certain required specifications.
Accessibility, Affordability, and Diversity
- Accessibility: We will caption any audio content presented in the VR Modules for the hearing impaired to improve accessibility. This technology may also be useful for teaching sight-impaired students some of the more advanced practical skills that would be dangerous or even impossible for them to acquire in a real-life face-to-face lab format. Finally, the detail provided in each module and the tasks assigned will be tunable allowing each module to serve in an introductory (CHEM 105/105L), reinforcing (CHEM 341), or mastery-level (CHEM 351L) instructional capacity.
- Affordability: The approach we take focuses on providing cost-effective hands-on VR science labs for underresources institutions of higher learning by harnessing the power of now ubiquitous 3D printing technologies. Affordability and scalability of the technology being developed is thus a top priority.
- Diversity: Virtual Reality systems can uniquely and cost-effectively support student learning by harnessing cultural, ethnic, gender, socioeconomic and other diversity-related learning preferences first-generation status, allow scalable customization of the hands-on learning process in ways that to improve student motivation, engagement, and self-efficacy.
About the Instructor
- Academic Background and Research Interests: I recieved my B.A. in Molecular and Cell Biology with concentration in Biochemistry and a minor in Chemistry from the UC Berkeley, where I worked in various labs doing computational biology, electron microscopy, proteomics, and microbiology. I recieved my Ph.D. in Biochemistry and Molecular Biology from UCLA for my work on non-equilibrium single molecule protein folding. I conducted my post-doctoral training at UC Berkeley developing novel chemical biology and "synthetic biochemistry" approaches for studying protein folding during translation using single molecule fluorescence spectroscopy. Such work is highly interdisciplinary and makes use of concepts from a wide range fields including biology, chemistry, physics, and computer science.
- Teaching Philosophy: My teaching philosophy in centered on the idea that the primary goals of instruction should ultimately be to foster intellectual independence in students so they can effectively continue learning finer and finer-grain details of the content covered as they come to require it in their future studies/careers. I take a learning-centered approach to pedagogy and employ "backwards-design" principles in creating and aligning my course content.
- Faculty Page
- Teaching in Higher Ed PostCast Interview
- Interview on using Labster Virtual Labs
- Curriculum Vitae Kambiz Hamadani 2019
LIT Redesign Planning
Building Infrastructure for 3D Hands-on VR Labs
- Lab tool Virtualization: In previous implimentations I have use 2D virtual labs to reinforce chemistry and biochemistry content. In Spring 2019, I tested out a more immersive 3D virtual reality lab wiht 3 degrees of freedom (dof) mobility within the headset and another 3-dof within an inauthentic controller used to interact with the virtual world. Motion capture teachnology now enables both active and passive 6-dof (translational as well as rotational). tracking of hand-held objects with millisecond temporal and submillisecond spatial resolution. Plug-and-play active tracking systems such as the HTC Vive tracker are now widely used to rapidly virtualize various gaming accessories (e.g. paddles, rackets, bats, and guns --see video at left). However, the virtualization of laboratory tools for hands-on VR/MR remains unrealized. Larger lab tools such as beakers can be virtualized using Vive Trackers (below left) but smaller lab tools (test tubes, pipettemen, reagent bottles, etc) require lightweight passive tracking (below right). In the present project, we have virtualized large and small lab tools using active and passive tracking respectively. To do so we've designed and 3D printed marker adapters to hold either Vive Trackers or sets of IR retroreflective passive markers (i.e. "rigid bodies") onto actual lab tools in a reproducible manner. The geometry of each non-degenerate marker groups or rigid body encodes lab tool identity. We've developed an algorithm which can create ~200 rigid bodies (enough to virtualize most of the lab tools in chemistry and biochemistry labs). Using STEAMVR together with Optitrack Motive software we can easily track as many as 30 virtualized lab tools simultaneously in a given Hands-on VR Lab Module.
- Virtual Reality Hardware: Major hardware required for this project includes an HTC Vive Pro head-mount display, 2 controllers, 2-4 Valve Lighthouse 2.0 basestations, a gaming desktop/laptop computer, 2-4 Vive trackers, high-quality 3D printers (i.e. Stratasys SE Uprint, FormLabs Form2), and an Optitrack Camera system (here we use a V120:Trio) with associated markers.
- Virtual Reality Software: Working with my Computer Science Information Systems colleague Dr. Yuanyuan Jiang and her students, I have developed the first generation foundational software required to stream passive and active object positions and orientations from Motive and SteamVR into Unity. We are now developing the higher-level Lab Module software which defines the triggers for different types of feedback as well as the storyline and milestones that the student must achieve to complete the Lab Module.
- Computer Aided Drafting and Design: I recruited an excellent arts/ digital media design undergraduate who has helped a great deal with creating models of virtual and virtualized lab tools as well as with 3D printing of the adapters required for mounting markers onto the lab tools. We did much of the CAD work using TinkerCAD but some of the more involved models had to be made using more advanced CAD tools such as Blender, SketchUp, and Fusion360.
- Proof-of-Principle-- a Student Using Virtualized Lab Tools: In the real-world (left) and virtual world (right) images below a student is pictured transferring virtual liquid from an actively tracked 1L beaker into a 50mL conical tube.
- Class size and Scalability: With a single VR setup it is possible to asynchronously process a maximum of about 50 1-hour lab completions a week (more than enough for the current proof-of principle test). However, for future scaling of our project, we have worked extensively with our extremely supportive Instructional and Information Technology Services Dept and were able to secure additional hardware and space which will ultimately allow a throughput of 500 lab completions/wk without requiring that students check out hardware. If we devise a suitable system for asset tracking which allows students to check out and return the hardware, the current space resource limitations which restrict the CSU's ability to offer more science lab courses may be at least partially circumvented.
- Pre-redesign control tests: In Spring of 2019, I obtained an untethered 3-dof headset with a 3-dof controller (i.e. Lenovo Mirage Solo with Google Daydream) and had students complete a 3D immersive Labster VR lab. This allowed me to compare the 2D non-immersive Labster Labs to the 3D immersive variants (data below) and also identify the best ways to schedule and cycle students asynchronously through completion of the hands-on VR labs currently under development. In this and previous semesters, students completed numerous standardized in-class assessments. These datasets will be used as a control dataset for the experimental semester in which I impliment the Hands-on VR Labs (Fall 2019).
Professional development activities:
- In AY 2018-2019 I participated in monthly LIT cohort Zoom Meetings as well as AR/VR Common Interest Group Meetings. These were very helpful for me as I built up a network of potential future collaborators and colleagues who might help with this long-term project of developing Hands-on VR Science Labs.
Revised CHEM 341 Syllabus for Fall 2019
LIT Results and Findings
LIT Redesign Impact on Teaching and Learning
- Ove the past academic year, I obtained preliminary data for, submitted, and was awarded an NSF CyberLearning award for $750K. This was a direct extension of my LIT project and will serve to expand the scope of the redesigns and experiments wiht educational technology that I've started to conduct.
- In Spring of 2019, In addition to implimenting a more immersive Labster virtual lab, I also gamified my CHEM 341 course. Both components seem to have improved student engagement and interest in the course but these results are qualitative and prelminary.
Assessment Findings - (in progress)
Student Testimonials comparing 2D to 3D hands-off VR in Spring 2019 CHEM 341
- "I think this is a creative way to learn and take in material from class. Although, sometimes during the VR stimulation it felt almost the same as the 2d computer version and while using the hand remote felt more just like a mouse. If there were to be accessories shaped like lab instruments like a pipettes, tubes, and a rack to connect muscle memory to learning, then that could very interesting. Overall, it was a great experience and can allow class to have a lab component without having to be in the physical lab."
"This technology was stimulating in a way that it wasn’t just a go click and memorize way of teaching the metabolic pathways. It allowed a more hands-on experience that allowed for concepts to stick better with students learning these topics. It wasn’t complicated to learn the controls for the virtual headset and pointer which provided fun through the process. In addition, it really felt like I was there experimenting. I believe many more labs should be utilizing virtual reality to teach students on certain procedures."
"Even though working on a real lab would always be a better learning experience, I think doing virtual labs using Labster have multiple strong points that make it worth its application. One of the strong points of these simulations is that it allow us to use more critical thinking and problem solving skills in the lab. Instead of following just a list of instructions or steps without really thinking what is the reason behind it, in labster we can obtain the theoretical information behind each step and we are also constantly being assessed during the lab. I think this help us to really understand the principal behind each experiment. Another advantage of labster is that it allow us to use, play with things that would be outside our control in a real lab setting which I consider can be more engaging. Also, virtual labs allow us to make mistakes or fail without any negative consequence. In my experience, chemistry labs can be a little difficult and complex when you are a student, topics are new and you do not have much experience. It is very likely to make mistakes which can be dangerous and also can affect your entire experiment, making you to start over, waste time, or even not be able to do the experiment. The immersive virtual reality lab on cellular respiration using the virtual reality glasses headset was even more encouraging and interactive. While doing the lab I found myself more invested on the lab. It allowed me to concentrate more in the topic and the actual activity. I found it very interesting and was a great learning experience. The only negative point on the virtual reality is that the remote was very sensitive to the movement."
"My experience with the immersive version of the virtual lab great. It gave me a sense of being in a laboratory setting even if it is only virtually… the control is a bit off even after calibrating it… Setting up the system is what took a while… it fogs a little bit inside the goggles and obscured my view, making it a little bit blurry…The main advantage for is that by doing it in virtual immersive system, there are less distraction and I can finish my lab faster."
"I felt that the virtual reality felt more engaging. It really reduced any possible distractions because all your senses where being occupied. Especially my phone wasn't tempting to use, and i had earphones in blocking out any noise. The only downside i would say would be maybe video quality and the potential for a headache ( not really an issue) and secondly the remote was a little difficult to use and navigate. In the future I would actually prefer to use the Virtual realty version due to its more engaging and realistic feel."
"I liked that the lab itself felt more real, as if I was there. I think that it made it more memorable and fun to do the lab. I liked that I got to see everything in a more 3D reality and that I got to be more interactive during the lab…Overall, I think that the immersive lab is a cool experience to learn the material."
"Being able to walk around would also enable students to see the 3 dimensional structure of the phosphoenolpyruvate better as some of the atoms were hidden when looking at it from a single standing position. Another option to being able to see the 3D structure of the molecules would be to allow the structure to be rotated or present it as a rotating model instead of a static model. Besides these two points, I feel that the VR does not have much of a bigger advantage over the computer screen. I did enjoy the VR goggles though as this was my first time ever using a VR goggles. Thank you for the experience! I’m motivated to get a pair when I can afford it."
Challenges that Students Encountered
- Logging into the immersive Labster VR lab system was a major challenge and stumbling block.
Lessons Learned & Redesign Tips
- Please contact me to discuss if you are interested in developing Hands-on VR Science Labs. I'm confident that I can help you design a customized experience that will meet the needs of your students.
Course Redesign Obstacles
- One major obstacle for me this semester was simply ordering all the hardware we required for the project. This required authorization from many individuals on my campus.
- I was on family leave in the Spring semester as was thus unable to compelete the implimentation phase of the project and had to delay implimenting until the Fall 2019 semester.
- I was also undergoing Tenure and Promotion Review this semester and that also slowed down my progress on the LIT project for various reasons.
Strategies I Used to Increase Engagement
- The most important thing that promotes retention in STEM fields is a students level of engagement with the material in order to develop a sense of identity as a scientist. There is no better way than by putting on a labcoat and doing actual experiments to promote such a sense of identity. The Hands-on VR Labs developed here will make such opportunities for students to feel immersed in the world of scientific inquiry much more accessible and thus should contribute to addressing the issue to STEM retention rates.
- I will use the $750K in NSF funding I've recieved to sustain and continue to advance the technology development and assessment activities I have started in this LIT project.