Using for an Immersive Experience

in the Water Cycle

Nicole Chin Aakash Gupte ​ [email protected] [email protected] ​

John Nguyen Sabrina Sukhin Grace Wang ​ ​ mjrnguyen@.com [email protected] [email protected] ​

Joe Mirizio* ​ [email protected]

New Jersey Governor’s School of Engineering and Technology

21 July 2017

*Corresponding Author

Abstract—A common misconception regarding virtual reality II. BACKGROUND ​ ​ ​ (VR) technology is that it is simply a gaming platform. The main A. VR Technology purpose of this project is to refute this notion and demonstrate the applications of VR beyond entertainment. One such usage for In the past few years, VR has become widely used in the VR is in education. VR can be used to immerse the user in an creation of video games. However, the practical applications environment where he or she learns through interaction with the of VR technology can extend much farther than gaming. VR is surroundings. This concept led to the development of SplashSim, a platform that places the user in a virtual world with an a mobile-based VR application in which the user traverses interface that allows for realistic interaction. The user puts on through the stages of the water cycle in the perspective of a water a headset and can become fully engaged in a simulated droplet. Through this approach, SplashSim enables the user to experience the water cycle in a unique and engaging way. Within environment as if they are actually existing in it, providing a this immersive VR environment, users are not only able to more unique medium of immersion. A smartphone's accelerometer closely observe processes such as the water cycle, but can also and gyroscope bolster the VR experience with a sense of study other important mechanisms that are not readily observed motion and position. The user is also able to move their head in the real world. to the respective part of the scene or explore the environment as they please. There is also 3D sound that plays louder when the user is near the audio source. By inserting students into I. INTRODUCTION ​ normally inaccessible areas and allowing them to experience it The 21st century is commonly defined as the age of first hand, VR enriches problem solving and critical thinking technological advancements. With the rise of smartphones and skills. [1] There are many different types of VR such as other digital devices, many new ideas have been developed to Window on World, Immersion, and Telepresence. Window on accompany the latest innovations. One such concept is virtual World does not require a headset: it is simply a realistic ​ reality (VR), which allows users to immerse themselves in a simulation. Immersion is the Window on World system that ​ ​ computer-generated environment. The objective of this project uses a headset to further engage the user. SplashSim is an is to create a product that uses VR to improve the learning example of immersion virtual reality. Finally, Telepresence is experience. The resulting application, SplashSim, places the a form of VR controlled remotely by a user. It typically ​ ​ user in the water cycle and allows him/her to become more involves using a drone to transmit a VR image to the operator. engaged in the environment. This paves the way toward VR is a viable instrument for advancing education integrating VR in education, where students can learn through techniques because it is capable of holding students’ attention. interaction and visual reference. [2] SplashSim is an example of how VR can be used as a

1

versatile type of technology with broad applications to various fields. B. VR History The concept of virtual reality has been around since the 19th century. The first VR head mounted display was made in 1960 by Morton Heilig. Since then, VR has spread widely. 2016 can be considered the year of virtual reality because of the influx of new VR technology such as the Oculus Rift and ​ ​ the HTC Vive, both of which are high-performance headsets. ​ ​ C. Visual Learning and Immersion Some may argue that using VR for education is unnecessary since videos and shows are just as effective at explaining content. However, VR offers a level of immersion Fig. 2 Diagram of the water cycle [16] that is unparalleled by traditional mediums. With VR, users feel more connected to the content because they are able to D. The Water Cycle personally interact with and manipulate it. [3] Compared with The water cycle is the process by which water undergoes ​ television, where the experience is from an outside point of changes in states through evaporation, condensation, and ​ ​ ​ view, VR offers a realistic first person perspective. This key precipitation. Evaporation is the process through which water ​ aspect gives VR large potential for educational uses. changes from a liquid into a gaseous substance. During condensation, gaseous particles of water come together and form liquid droplets. Finally, during precipitation, the liquid droplets fall back to earth in the form of rain, hail, or snow, or sleet. Some of these processes are depicted and explained in SplashSim. E. VR as a Platform for Education Traditional educational methods are slowly becoming supplanted by more advanced techniques. Educators constantly seek to develop new and exciting methods to make education more interactive and effective for the students. The education experience is being reshaped for students and teachers alike through new innovative teaching methods, advancements in technology, and programs designed to make technology more readily available. [7] In particular, VR in Fig. 1 Pie chart depicting distribution of learner types [4] education helps students retain information better, as it creates a unique, individualized environment that cannot be replicated According to a study done at the University of Alabama in the classroom setting. School of Medicine, the majority of people are visual learners, meaning they learn best when looking at a visual F. representation of a concept. [5] SplashSim appeals to this Unity is a development platform that is used to create 2D wide body of visual learners by immersing them in an and 3D games. The editor allows users to design scenes, ​ ​ environment that they can see and explore. instances of the 3D landscape in Unity, and instantly observe the results of edits made. Unity also enables users to deploy their games to a myriad of platforms including TV, desktop, and mobile devices. Furthermore, Unity supports VR and (AR), which makes virtual objects appear in ​ the real world, and is used to create games for virtual reality ​ headsets. Most games in Unity include features from the Unity asset store, which also allows users to access artwork, models, and scripts, programs that accomplish tasks in Unity. Finally, ​ ​ Unity allows users to share their work with others: Unity ​ Teams allows for collaboration on projects while Unity ​ Connect allows users to share their work with potential ​ recruiters.

2

H. Blender and Assets In order to build an immersive world in SplashSim, artwork and models are moved from Blender, the software where they are created, into Unity where the movements are created. This artwork is called assets. They ​ ​ were essentially objects that populated the simulation. The 3D art is saved as an .fbx file, which allows the assets that are published in Blender to be manipulated and changed in Unity.

Fig. 3 The User Interface (UI) for Unity G. Cardboard Many VR devices are available to the general public, such as the HTC Vive, the Playstation VR, or the Oculus Rift; ​ ​ ​ ​ ​ ​ however, the is one of the most affordable ​ VR technologies on the market. It is a cardboard box with a slit that can fit Samsung and iPhones. This device is a popular alternative to other VR devices because it is cheap and simple to use. The interactive conductivity button, as seen on top of the device in figure 4, is one such piece that allows users to Fig. 5 Blender interface press a button that transfers an electrical signal and taps the I. Android SDK screen. Due to features such as this, VR simulations are able The Android Software Development Kit (SDK) was to be more immersive. While most other VR technologies cost ​ hundreds of dollars, the Google Cardboard is the perfect integral to the development of the project as it enabled instrument to mass produce for all people to use. It is an SplashSim to be deployed to a Samsung Galaxy Note 5. The important part of making VR an essential part of classrooms Android SDK integrates Unity-based programming with all over the world. Google estimated that about 1 million Blender artwork and allows it to be viewed with the Google ​ Cardboards had been distributed, and that was before the New Cardboard. There were many settings in Unity that were York Times recently distributed more than a million to its configured with Android SDK in conjunction with the Java ​ subscribers. [8] The Google Cardboard was used in the Development Kit that enabled SplashSim to be run on a development of the project, and is essential in making VR Samsung Galaxy Note 5. more mainstream and accessible for the general population. J. Github Repository Github is web-based platform that allows users to collaborate on code. As a version control system, a GitHub ​ repository was made and shared with all group members. GitHub desktop was used and enabled the ability to push and fetch updates and changes made by each group member almost instantly. GitHub Desktop made sharing the project simple and fast. GitHub’s user interface allows the any member to see every change made by other members and revert to old versions if deemed necessary. This feature was very useful throughout the entire process. K. Elements of Virtual Reality used in SplashSim 1) Low Poly Environment: Low poly is an artistic and ​ ​ 3D modeling style where the meshes, or framework of the models, contain a low number of polygons, hence the name “low poly”. 2) Rails: The rail consists of the path the camera takes ​ as it rises and falls throughout the course of the water cycle simulation. Fig. 4 The components of the Google Cardboard [19] 3) Animation: The rain was animated using particle ​

3

effects which allowed for the manipulation of properties such as size and amount. The stream animations were included in the purchased asset. Cloud animations were made by using keyframes, which are the start and end points of major actions. ​ 4) Subtitles: The text in SplashSim appears at a specific ​ time so that it corresponds to a particular stage of the water cycle. It is also attached to the camera so that it follows the user’s head movement. 5) Sound: SplashSim utilizes objects that contain ​ spatialized sound, allowing the user to experience different Fig. 7 This scene depicts condensation in SplashSim. sound effects as they progress through different regions of the Once the user arrives far up in the sky, they watch as virtual environment. clouds form around them and see the information about 6) 360° Movement: SplashSim incorporates the ability ​ condensation. Lastly they fall back to Earth along with rain for the user to look around their surroundings by employing droplets while reading information about precipitation. the phone’s accelerometer, a device that determines the ​ ​ motion of the phone, and gyroscope, a device that measures ​ ​ the user’s tilting and orientation. III. METHODS AND EXPERIMENTAL DESIGN ​ ​ ​ ​ ​ ​ ​ A. Simulation Design in SplashSim First, a storyboard was created, delineating the visuals and effects needed for each stage of the water cycle. Afterwards, the art assets and rail, which is a set path that the user would ​ ​ travel along, was created to transport the user through the stages of the water cycle and provide scientific information at Fig. 8 This picture shows the precipitation stage as depicted in SplashSim. the key steps. After that, additional art assets were brought into the environment. More trees, water, and mountains were At this stage, the view turns to a stream where the user added, as well as the sound of rain. flows back to their starting point. The simulation opens at the water’s surface, where information is displayed explaining evaporation.

Fig. 6 This scene from SplashSim depicts the evaporation stage of the water cycle The viewer is given time to look around the scene, where Fig. 9 Flowchart of simulation they see trees, a mountain, a sunset, and water flowing all B. Hardware Methodology around them. Then the camera rises to the clouds, where condensation is explained. Google Cardboard was the hardware device used for the project. It allows the user to experience VR by placing a smartphone in its compartment. The two lenses in the Cardboard merge the left and right images that are displayed on the screen of the smartphone in order to perceive the depth of the 3D environment. The Google Cardboard is an effective platform for the deployment of SplashSim because of its low

4

manufacturing cost, which makes it a popular and affordable specific part of the simulation. The Unity built-in animator VR head mounted display. was used to move the player through the simulation. The Samsung Galaxy Note 5 is a smartphone than runs the Inheritance was used for objects, elements that ​ ​ ​ ​ ​ Android operating system, a mobile operating system characteristics can be attributed to within a game, that needed developed by Google. Using this smartphone for development to be animated. Inheritance allows objects take on the made building and testing fast and simple because of its properties of other objects. Animated objects were made compatibility with Unity. children of other objects, meaning they inherited the attributes of their parent object. Animations were added to the parent C. 3D Art object and automatically applied to all of the children objects. The app required all the art assets to be made in Blender. E. Minimum Viable Product Models that were made include mountains, rivers, rocks, lakes, trees, and glaciers. A skybox, which is an infinite cube SplashSim was tested throughout the development process. ​ ​ A minimum viable product (MVP), a basic skeleton of the around the simulation that contains the sky texture, was also ​ ​ ​ environment needed for the simulation to function, was made using textures in Unity’s built-in editor. Another major ​ aspect of the program was animations, which were done in created as a way to test all of the mechanics of the VR Unity. All the assets were rendered in low poly. The shaders simulation. Through multiple tests, it was determined that the ​ are adjustable properties that manipulated how the lighting speed of camera animations had to be slowed down to reduce affected the materials and textures of the assets. the chance of the user experiencing motion sickness. The art assets and lighting were adjusted based on feedback from testers. There were many occasions where SplashSim encountered errors, but through rigorous testing, the final application runs smoothly.

Fig. 10 Example of a mountain asset

Fig. 12 The MVP in Blender F. Deployment and Final Product Over the course of the project, as various elements of the SplashSim environment were created, it became necessary for it to be seen through a VR device. These elements were deployed onto the Samsung Galaxy Note 5, which allowed it to be seen in VR through the use of the Google Cardboard. SplashSim was deployed onto an Android phone and turned into an app by building the scene as an Android Package Kit ​ ​ file (.apk), an executable file for Android phones. Fig. 11 Example of tree asset D. Programming All programming for this project was done in Unity using C#, a programming language and Microsoft Visual Studio, a ​ ​ ​ program that runs and organizes code. The GoogleVR asset ​ Fig. 13 The sunset hue of the skybox followed by the landscape was found on the Google developer website and contained the IV. RESULTS AND DISCUSSION scripts and assets necessary for creating a VR environment in ​ ​ ​ ​ ​ Unity. The GoogleVR asset came with scripts for 360° head A. Immersion in SplashSim movement and a reticle pointer, which is a small dot in the ​ ​ VR has long been criticized for being a niche technology. center of the screen through which the user can focus on a The most useful aspect of this technology is its immersive

5

capability, as it allows users to step into a simulation. VR is understand. The applications themselves are very user designed to keep its users invested in the simulation before friendly. they take off the headset and continue about their day. Not V. CONCLUSION only do these simulations allow the users to experience ​ environments that are very difficult to reach otherwise, but A. Summary they can also be used to help students visualize difficult In the past few years, VR has mainly been used as an concepts. VR adds a level of personal input which is hard to additional platform for playing games, from the HTC Vive to be recreated elsewhere. In SplashSim the immersion of being the Playstation VR. However, VR can be used in so many in the water cycle is what differentiates it from other mediums other ways. For example, it can be used in education to create for education such as charts and diagrams. It is much easier to a more immersive environment. The goal of SplashSim is to learn about something while actually experiencing it. have the user feel immersed in the water cycle. They are in the B. Improving the Learning Experience point of view of a droplet of water as it goes through all the stages of the cycle. Rather than using videos or charts that are VR is an effective education tool for youths in particular typically used to explain the water cycle, using VR allows the because its strengths lie where traditional education is lacking. user to feel more personally connected with their surroundings VR in education can connect with students and help them and helps them learn more effectively. As educators begin to retain information better, as it creates a unique, individualized look for more interactive methods to help their students learn, environment that cannot be replicated in the classroom setting. SplashSim aims to spread the benefits of VR technology to VR also demonstrates the applications of the content in the provide students with the proper amount of immersion and real world. Instead of showing a picture or a video or learning that they need to prepare for the future. describing an event, teachers can instantly bring their students on a virtual field trip so they can interact with the content B. Errors / Issues that Occurred firsthand and in full detail. By creating a more engaging and Throughout the course of the project, many technical stimulating environment, VR will improve the learning problems occurred. For example, Unity’s user interface had a experience. In SplashSim, learning about the water cycle is high learning curve, and it took several days to get familiar enhanced by the more immersive experience that comes with with it. There were many times when additional Unity using VR. Further applications may include allowing students tutorials had to be watched. The Android SDK had problems to virtually perform experiments that are not suitable for the integrating with Unity as well. Another problem was classroom, give presentations to overcome their fear of public importing the Unity app onto an Android phone. Due to the speaking, or participate in stories as their favorite characters. fact that there is no ability to deploy a Unity project on iOS using PC, there was no other option but to use Android which only one group member had access to. There were also issues with the GoogleVR asset and deploying the scene with the art assets that were designed in Blender. All of these problems were solved by troubleshooting and testing, as well as using the demo scene that was included with the GoogleVR asset. These issues that occurred slowed down the process of creating the app because it was difficult to test and debug the simulation. The project had to be broken down into several components that would be easily tested separately, and through the use of GitHub, the numerous components had to be assembled together to create SplashSim. The problems that arose distinctively shaped the way SplashSim had to be Fig. 14 Picture of a child amazed by VR [17] created. C. Mobile Application Benefits C. Future Improvements SplashSim is an educational mobile app that can be used in In the future as the application continues to be developed, and outside of the classroom. It effectively introduces further sections can be added to make SplashSim more students to the water cycle through a memorable and interactive and immersive for the user. For example, immersive experience. In the future, many apps similar to additional paths could be created which lead to other kinds of SplashSim can be created to target students of all ages. precipitation, such as hail and snow. This would allow the Difficult school subjects such as physics and chemistry can be user to choose a path that could lead to different results. taught through the use of VR simulations. The best part of Another way to make SplashSim more interactive would be to using a mobile application is that it is very easy for students to implement buttons that could be clicked by the user through download and learn. They do not have to go through a long the conductivity button and could allow the player to move or process to download, and do not have to watch tutorials to make text appear.

6

Additionally, improvements could be made in adding detail interact with the characters and affect the outcome of the to the environment of SplashSim. Small animals and sounds story. This simulation would not only be more interesting and could be added into the simulation as well as higher quality immersive than reading a book, but it would still retain the graphics. Additionally, the interactive conductivity button can education value of teaching a child basic language skills. have more usages with a pause and start button as well as a VI. ACKNOWLEDGEMENTS menu. Adding a user interface into SplashSim would also ​ make the application more welcoming to users. Lastly, SplashSim was achieved through the combination of many incorporating narration into SplashSim would add an people’s efforts. SplashSim would like to thank Joe Mirizio additional way to help users learn. and the other Lockheed Martin Engineers. Special thanks to New Jersey’s Governor’s School of Engineering and D. Future Applications and Discussion Technology for allowing this project to succeed. Many thanks In the future, the implementation of virtual technologies to Dr. Ilene Rosen and Dean Jean Patrick Antoine for all of will continue to expand. An additional example of this can be the amazing opportunities and for leading all of the students of seen through AR. It can be used as an extension to VR by GSET. Thanks to the Residential Teaching Assistant Ryan creating a more immersive experience by incorporating the Gamadia for being a great mentor throughout the course of the real world in simulations. As seen in figure 11, the Microsoft project. Lastly, much gratitude to all of the Governor’s School ​ Hololens, which are AR glasses, adds an extra layer into the staff, as well as Rutgers, Rutgers School of Engineering, the ​ ​ world through which additional information can be shown. State of New Jersey, Lockheed Martin, Google Cardboard, SplashSim may be further developed by expanding its social and Silverline Windows. impact and addressing important environmental issues REFERENCES concerning water. The simulation can discuss the implications ​ [1] R. Reynard, “The Impact of Virtual Reality on Learning,” Campus ​ ​ of water pollution and teach people how to conserve water, Technology, 26-Apr-2017. [Online]. Available: https://campustech ​ thus expanding the target audience to adults as well. nology.com/Articles/2017/04/26/The-Impact-of-Virtual-Reality- SplashSim already focuses on water, so incorporating on-Learning.aspx?Page=2. [Accessed: 15-Jul-2017]. pollution into the simulation could be simple to do, yet create [2] “VR Education: Why should we employ virtual reality in education?,” Unimersiv, 11-Apr-2016. [Online]. Available: ​ ​ a powerful message. https://unimersiv.com/vr-education-why-should-we-employ -virtual-reality-in-education/. [Accessed: 15-Jul-2017]. [3] A. Eisenberg, “How Virtual Reality Immersion Will Shape Our Future,” AppReal.[Online]. Available: https://appreal-vr.com/blog/vir ​ ​ tual-reality-immersion-what-is-it-and-how-it-works/. [Accessed: 15-Jul-2017]. [4] U. A. S. M., “Visual Aids,” Visual Aids, 2005. [Online]. Available: ​ ​ http://www.uab.edu/uasomume/fd2/visuals/page1.htm. [Acce ssed: 20-Jul-2017]. [5] R. Lindemann, “How Education is Changing,” ReadSpeaker, ​ ​ 07-Mar-2014. [Online]. Available: http://www.readspeaker.com/ how-education-is-changing/. [Accessed: 15-Jul-2017]. [6] R. Hof, “Google Cardboard Will Bring VR - And VR Ads - To The Masses,” Forbes, 23-Dec-2015. [Online]. Available: https://www. ​ ​ forbes.com/forbes/welcome/?toURL=https%3A%2F%2Fwww. forbes.com%2Fsites%2Froberthof%2F2015%2F12%2F23%2F -cardboard-will-bring-vr-and-vr-ads-to-the-masses%2F& refURL=&referrer=#42f6584f38a6. [Accessed: 15-Jul-2017]. [7] “Fun Water Cycle Facts for Kids,” Easy Science For Kids, ​ ​ 02-Dec-2016. [Online]. Available: http://easyscienceforkids.com/all- Fig. 15 Simulation of AR using Microsoft Hololens [11] about-the-water-cycle/. [Accessed: 15-Jul-2017]. [8] “Get Started with Google VR in Unity | Google VR | Google In addition to illustrating the water cycle, more simulations Developers,” GoogleVR. [Online]. Available: https://developers. ​ ​ can be added, to demonstrate other uses for VR in education. google.com/vr/unity/get-started#build_and_deploy_the_demo_sc For example, a simulation can be created that allows users to ene. [Accessed: 15-Jul-2017]. [9] “Blender Reference Manual¶,” Blender. [Online]. Available: https:// ​ ​ perform science experiments virtually. This would be helpful docs.blender.org/manual/en/dev/index.html. [Accessed: for students who do not have access to necessary laboratory 15-Jul-2017]. [10] “Building your Unity game to an Android device for testing,” Unity. materials, as they would no longer have to watch someone ​ ​ else perform a lab. Additionally, it would allow them to [Online]. Available: https://unity3d.com/learn/tutorials/topics/mobile -touch/building-your-unity-game-android-device-testing. [Accessed: perform important but dangerous experiments that are not 15-Jul-2017]. suitable for the classroom setting. [11] http://www.businessinsider.com/microsoft-hololens-augmented-reality- In addition to science, VR can also be extended to headset-roundup-2015-10?r=UK&IR=T. (2017). [image]. childhood literacy. A simulation could be created to insert the [12] K. Shah, “Secrets to Creating Low Poly Illustrations in Blender,” 3D & Motion Graphics Envato Tuts , 14-Jan-2014. [Online]. user in his/her favorite children’s story, allowing him/her to ​ Available: https://cgi.tutsplus.com/tutorials/secrets-to-creating-low-

7

poly-illustrations-in-blender--cg-31770. [Accessed: 15-Jul-2017]. [13] U. Technologies, “Unity User Manual (2017.1),” Unity - Manual: ​ Unity User Manual (2017.1). [Online]. Available: https://docs.unity ​ 3d.com/Manual/index.html. [Accessed: 15-Jul-2017]. [14] W., “Blender 3D: Noob to Pro/UV Map Basics,” Blender 3D: Noob ​ to Pro/UV Map Basics - Wikibooks, open books for an open world, ​ 26-Mar-2005. [Online]. Available: https://en.wikibooks.org/wiki/ Ble nder_3D:_Noob_to_Pro/UV_Map_Basics. [Accessed: 15-Jul-2017]. [15] http://www.uab.edu/uasomume/fd2/visuals/page1.htm. (2017). [image]. [16] https://www.youtube.com/watch?v=lLOeEj9klTU. (2017). [image] [17] http://www.trustedreviews.com/opinion/virtual-reality-in-education-293 1520. (2017)[image] [18] W., “Blender 3D: Noob to Pro/Build a skybox,” Blender 3D: Noob to ​ Pro/Build a skybox - Wikibooks, open books for an open world, ​ 08-Oct-2005. [Online]. Available: https://en.wikibooks.org/wiki/Blender_3D:_Noob_to_Pro/Build_a_s kybox. [Accessed: 15-Jul-2017]. [19] https://www.gadgets4geeks.com.au/google-cardboard-2-virtual-realit y-vr-headset-3rd-gen-for-mobile-phones

8