Studying the Efficacy of Smartphone-Driven Inspirit VR Biology Content

At inspirit, our vision is to make learning immersive and engaging for learners
around the globe, across diverse formal and informal learning environments, and across diverse subjects and topics. Building upon virtual reality’s affordances for supporting experiential and immersive learning in the classroom, we wanted to explore how far we could push smartphone-based virtual reality (VR) use on a regular basis in the classroom in a way that is meaningful, engaging, and affordable for first-time VR users in public schools.

This report describes our key findings from a pilot experiment conducted at the Cobb County School District in the state of Georgia (USA) from March 1, 2018 through April 1, 2018. We had 9th- grade students enrolled in Freshman Biology classes in three high schools use our inspirit Biology VR content in their biology class for 5-10 minutes every day, for one week. We collected a combination of qualitative and quantitative data over the course of the study. This report summarizes key finds from the latter.

Why VR ?

In recent years, a growing body of research on virtual reality (VR) has shown that highly immersive virtual environments that can be accessed using specialized hardware like head-mounted displays have a number of benefits, including improved learning outcomes, increased confidence, and positive behavior change. However, most of these findings come from experimental and controlled
laboratory settings.

At the same time, the consumer market is increasingly being flooded with
affordable VR technologies such as low-cost viewers that work with one’s
smartphone. These developments make it increasingly viable for VR to be used in everyday instructional settings such as the classroom. Given the reality of limited budgets for public education, VR could be used to take students on tours of physical locations around the world of global significance without leaving the classroom or take students on tours of locations that are physically infeasible to visit today — such as a tour of a plant and animal cell or a tour of a planet outside the solar system.

Through our research and design work over the last one year, our mission at
inspirit has been to democratize access to immersive VR learning content for a variety of formal and informal learning environments, providing learners from diverse backgrounds with an engaging and meaningful learning experience. Using
a human- centered design approach, we –

● work with teachers in creating and curating VR learning content for their students as they see fit,
● engage learners with new and immersive VR experiences, and thus
● expand our repository of mobile VR learning content.

Our Proof of Concept

We designed and developed 50 minutes of animated biology virtual reality (VR) educational video content grounded in the biology curriculum standards of the state of Georgia (USA). Using quantitative and qualitative research methods, our
goal was to conduct a proof of concept with ~150 students, where we evaluated the potential learning benefits of our smartphone-driven VR biology content for high school students over existing digital teaching resources. Our objectives are
focused on taking a human- centered design approach to unpack the social and technical dimensions necessary to support the various learning affordances of VR, and subsequently integrate high fidelity VR educational content into a classroom environment.

Study Facts

• As of January 2018, this would be one of the largest VR control experiments conducted in high schools outside of a laboratory.
• We enrolled 121 students in total across 4 high schools. Three high schools (N=96) received the VR experience while the fourth high school (N=25) received the control condition.
• In the experimental condition, students watched the VR content on their personal smartphones during their regular biology class time for around ~5 minutes everyday for 5 days. We had all students participate in –
  – a 30 minute pre-assessment prior to their VR use, and
  – a 30 minute post-assessment after to their VR use.
• In the control condition, students participated in a normal classroom experience identical to the experimental condition, but without the VR experience. For the last 5 minutes of the class, these students were provided a digital handout ‘textbook’ of the textual content covered in the VR experience along with supporting 3D photographs of the VR environment and references to external digital content.
• In total, we enrolled a superset of 350 students across 6 middle and high schools to conduct a deeper contextual inquiry of the use of VR in the classroom.

Study Objectives

The goal of the study was to evaluate the effectiveness of smartphone- driven VR curriculum as a content reinforcement tool. Students used the VR experience AFTER they participated in the regular classroom learning activity, and we studied how VR reinforced and supported the understanding of these topics. We
conducted pre- and post-assessments to assess the –

● increase in factual and spatial knowledge
● increase in student confidence on the topics
● increase in engagement with and interest in the topics
● increase in critical engagement and inquiry-based learning

Please see our journal publication for all details on our study methods. In this
report, we will now present our findings.

Findings

We will now list some of our key findings below. In all graphs, Group
A is the experimental group that received the inspirit biology VR
experience, and group B is the control group.

Knowledge [N = 96 (Group A) + 25 (Group B)]

We found that overall knowledge levels of students were higher by 9.59% (see Graph 1). This is almost a full letter-grade increase in test scores of students who used VR to revise and reinforce topics or concepts they had already studied earlier. Here, we find that VR proves to be a powerful tool for content revision and reinforcement before a test.

Confidence [N = 96 (Group A) + 25 (Group B)]

We found that average student- reported confidence levels were higher in Group A than in the baseline and in Group B when averaged over all subject areas and participants (see Graph 2). When we look at confidence levels on the DNA/RNA chapters only (see Graph 3), we see a large overall confidence increase (18.6%) in Group A, while users in Group B suffer a drop in confidence levels. On average, we noticed that students in Group A saw a larger confidence boost on topics they
perceived to be harder than average, and students in group B suffered a drop on these topics. This finding supports for using VR on hard topics, as well as on topics that were best understood when context on shape, scale, and geometry was provided to the learners.

Engagement [N = 96 (Group A) + 25 (Group B)]

Overall student-reported levels of confidence higher in Group A than the baseline and the control (12.66% more) when averaged over all subject areas and participants (see Graph 4). When we look at confidence levels on the eukaryotic cell chapters (see
Graph 5) as well as on the DNA/ RNA chapters (see Graph 6), we see a large overall increase in Group A, while users in Group B suffer a drop in each case in their average confidence levels. The largest average increase for Group A is seen in Graph5(onthe eukaryotic cell chapters), while the largest average decrease for Group B is seen in Graph 6 (on the DNA/RNA chapters).

Interest/Involvement [N = 96 (Group A) + 25 (Group B)]

We compared student interest and involvement with each chapter’s content in our VR experience (provided to Group A) against analogous chapters in a digital ‘textbook’ handout (provided to Group B). We found that, throughout all the
chapters, student interest and involvement (self-reported) scores in Group A were consistently ~12% higher than the control (see Graph 7). This also supports prior laboratory work in VR that has found students to engage more deeply with underlying content when it is presented in an immersive VR format versus other digital and textual modes of presentation.

Attitudes [N = 96 (Group A) + 25 (Group B)]

We also examined attitudes of students in Group A, and asked them to self-report their feelings and emotional state in Likert-type surveys after they had engaged with the VR content (see Graph 8 on the next page). We found that negative attitudes such as tiredness, stress, nervousness, and feeling overwhelmed were low, while
positive attitudes such as engagement, curiosity, happiness, confidence, and enjoyment were high. We would like to note that tiredness was considerably higher than other negative parameters, and this is certainly a drawback of the VR
experience that we hope to address with future work.

Additionally, student self- perception and performance reflection was generally higher in Group A over Group B. We measured this by asking students to self-report their response to the following prompt, after they had engaged with the respective
treatments: “Overall, I consider myself to be academically good in biology.” (See Graph 9.)

Finally, we also measured student satisfaction with the digital content presented to them, and we had them answer the following prompt to measure this, after they had used their respective treatment: “I wish I had access to more digital learning content for biology.” (See Graph 10.)

Other Findings

To examine the validity and reliability of our data, we conducted regression
analyses and reliability testing on various factors. Of these findings, a simple linear regression found no significant difference (p=0.34) between the pretest knowledge score of students in the control and experimental group. This indicates that neither group had an overall prior advantage over the other in
knowledge scores. Similarly, we also found that all scores reported above were not affected by which school the student was enrolled in, nor were there any extreme average score values coming from any single school that was biasing our results. A full list of findings can be provided upon request.

Other findings that have not been discussed here (and are available in the full journal publication, upon request) include knowledge, confidence, engagement, and interest/involvement scores for each individual topic, subtopic, participating school, gender, and socioeconomic group represented in our participant pool.
Please feel free to contact the inspirit team if you would like to read the detailed study summary.

Acknowledgements

We would like to thank all participants and researchers who were involved in this massive study effort. This list includes, but is not limited to, all the students, teachers, and administrative staff of the Cobb County Schools District, members of the Communications Department at the College of Computing at Georgia Tech,
and members of the TanDEm Lab at the School of Interactive Computing at
Georgia Tech. We are very grateful for all the support we received through this study.

Note – This report summarizes a study conducted in collaboration with the Georgia Institute of Technology, Cobb County School District, and inspirit, which concluded on April 6, 2018. This report is meant to provide a high-level summary of all the findings from this study. All content featured in this report falls under the inspirit terms of service and privacy policy, both of which can be found online at www.inspiritvr.com. The copyright for all photographs in this report belong to Terence Rushin and we would like to thank him for these photos. Please do not distribute, share, modify, or replicate this report and distort its content in any way without receiving written permission from the authors and owners of this content. This report is not a complete list of findings from this study, nor is it a peer- reviewed publication of any kind. For full details on the overall study, including details on the journal and conference publications on this work, please reach out to the inspirit team. All content in this document has been presented with as much accuracy as possible, however, inspirit does not provide any guarantee on any facts or details presented here. Please direct all questions to the inspirit team.