Virtual Reality Survival First Person Shooting Game

Shao-Hung Yu, Ya-Wen Hsu, Guan-Yan Chen and Jau-Woei Perng

Abstract— The aim of this study is to create a cheaper virtual connection between the Unity program and Arduino must first reality (VR) on the cellphone platform with equipment to gain be established. The client thread in C# is available to establish more external inputs rather than just using accelerator in the the client socket in the Unity program. In Arduino, the study cellphone to detect head rotation. The study used Android as a used ESP8266 to establish the TCP/IP protocol server. Thus, platform and the Unity game engine to design a VR game, with we can connect Unity and Arduino to transmit data from the Wi-Fi connection, using Arduino, gyroscopes, and Wii joysticks input equipment. As for the motion capture in VR, creating a to add external input controls to enrich the gaming experience. set of wearable sensors is a common solution to enable motion Keywords: VR, Unity, Arduino, Gyroscope, Wi-Fi capture [3]. The MPU-6050 GY-521 gyroscope module was chosen for capturing the user’s right arm motion. There are I. INTRODUCTION three gyroscopes placed on the right arm’s hand, forearm, and With the development of virtual reality (VR) technology arm joints, respectively, and one on the user’s chest. In order over the past few years, there are more and more VR systems to connect multiple I2C devices, the AD0 pin on the GY-521 available to consumers, such as Oculus Rift, PSVR, and HTC gyroscope module was designed to change its I2C address for Vive. Prices, however, remain high. Although there are communication on Arduino. As gimbal lock in the Unity inexpensive VR systems in the market, they usually lack more program is caused by Euler angles, using quaternion to rotate external controls since the only sensor used is a gyroscope or the game object in Unity was necessary to avoid such accelerator inside the cellphone for detecting the user's head phenomenon. Finally, the VR compatible mode was added to rotation. These kind of VR systems are called Gear VR. The Unity in order to develop VR games, because the regular Unity aim of this study is to create a less expensive VR system for edition only supports non-VR 3D or 2D games. Most VR- cellphones with Unity 3D and Arduino for more external related experiments or studies delve into task-oriented inputs. As table I shows, high-end VR platforms include more functions [4]. In this study, our goal is to present the value of equipment, thus providing a richer gaming experience. In this self- made system to the game industry. contrast, Gear VR is the cheapest VR platform—all you need is a headset and cellphone, but it lacks equipment to input more II. SYSTEM OVERVIEW controls. The whole system hardware is shown in figure 1. This VR Table I. VR platform comparison system includes a VR helmet, inside of which the Android phone is placed; a set of three gyroscopes attached to the right Platform PSVR Oculus HTC Vive Gear VR arm and another one on the chest; two joysticks on both hands; Rift a Wi-Fi module on the chest; and an Arduino board on the Experience Positional Positional Positional Seated chest as the computing center. There will be an external power Tracking, Tracking Tracking, VR supply for the Arduino board.

Seated or Seated or Room- Standing Standing scale VR, VR VR Seated or Standing VR Equipment Headset Headset Headset Headset

Processor Camera Controllers Unit Controllers Base Controllers Stations Price 400 USD 500 USD 800 USD 8~70USD Figure 1. System overview Integrating Arduino with Unity is used in many academic The system architecture is shown in figure 2. The studies, such as shape rendering by gestures [1] or managing gyroscopes will collect rotation data from the user’s right arm robot kinematics [2]. To create more inputs by Arduino, the joints and chest, while the joysticks are responsible for the digital and analog inputs. The data collected will be sent to the

S.H Yu, Y.W Hsu, G.Y Chen, J.W Perng are with the Department of Mechanical and Electro-Mechanical Engineering, National Sun Yat-sen University, Kaohsiung, Taiwan (phone: (+886) 934333437; e-mail: [email protected])

Arduino Due for formatting into a dedicated byte array, which rotation. will then be sent by Wi-Fi module to the Android phone, where the Unity program is running. Unity will analyze the byte array and store it as dedicated variables, which are later used in the game for player model control.

Figure 3. GY-521 with wearable design • Multiple Gyroscope Connection: A total of four gyroscopes were used to separately track the rotations of the hand, forearm, and arm joints of the right arm, and the chest rotation. However, communications between these four gyroscopes and Arduino Due was difficult. GY-521 uses I2C protocol to communicate with 0x68 address by default, AD0 pin with high ground will result it alter the address to 0x69. So the method to gather data from four GY-521 is changing the high or low on AD0 from GY-521. In this study, the GY-521 need to gather data will be low on AD0, the other three GY-521 will be high on AD0, thus only need to gather data from 0x68 address, and using for loop in Arduino program to alter the high/low on AD0 Figure 2. System architecture pin to control which GY-521 communicate to Arduino The system can be divided into two parts, hardware and is on 0x68 address, and the other gyroscopes don’t software, which will be discussed in detail in the next section. need at the time are on 0x69 address. • III. METHOD Gyroscope Data Filter: Now we can control which GY-521 can communicate with Arduino, but there is The method in this study attempts integrate two systems, still a problem. It seems that multiple I2C Arduino hardware and Unity program, in Android. Arduino is communications will cause some disturbance in the responsible for external inputs, while Unity is where the game FIFOBuffer, thus creating unexpected error data. To runs and processes data received from Arduino. eliminate this problem, after monitoring all the The method is divided into two parts for discussion: A. elements inside the FIFOBuffer array, we found that External Inputs and B. Internal System. the values of the 40 and 41 address inside the FIFOBuffer array were always 178 and 106, A. External Inputs respectively. Only when disturbances occur will the • Computing Center: The CPU for the Arduino Due values change. With this discovery, every time the has an ARM structure and the system is equipped with FIFOBuffer array received new data from GY-521, Atmel's ATSAM3X8E chip with Cortex-M3 for the we can use the Arduino program to monitor the values core. Arduino Due provides higher performance and of the 40 and 41 address in the FIFOBuffer array. If greater storage space with 84 MHz clock pulse, which these are not 178 and 106, then we know that a can totally handle data from the four gyroscopes. disturbance occurred; thus, we can discard this data and repeat the data gathering process. Gyroscope Module: MPU-6050 sensor’s GY-521 • module was chosen for the motion capture in VR. As Joystick Inputs: Aside from the gyroscopes, the shown in figure 3, it is wired on a small breadboard study also used two Wii secondary joysticks as with Velcro straps to make it wearable. The GY-521 external inputs, as shown in figure 4. The joysticks module is a six-axis gyroscope using I2C protocol to were re-wired in order to connect to Arduino. For the connect to Arduino. It can help capture the rotation right-hand joystick, we only used the trigger button data of objects. In this study, these are separately labeled “c” for firing the laser pistol in the game. For placed on the hand, forearm, and arm joints. In the the left-hand joystick, we used the analog joystick to early stage of the study, GY-521’s Yaw, Pitch, and control the player's movements inside the game, and Raw data was used to simulate the motions of the arm, "z" button is used for set arm rotation to default. but it was found that using Euler angles inside the Because rarely the gyroscopes will get the wrong computer program caused gimbal lock and resulted in rotation data due to poor connection or lagging. unexpected simulation. Thus, later quaternion data replaced Euler angle data inside GY-521 to track

Byte 15: A single negative sign indicates the sign of the number of the right arm’s GY-521 gyroscope module’s quaternion w element. Because quaternions can be calculated if you have at least three elements, the fourth element only needs a plus or negative sign. This pattern for the quaternion numbers of the GY-521 gyroscope module will continue on the forearm and hand rotation data until byte 64. Byte 65: For the trigger button c on the Wii joystick.

Byte 66~68: For the X analog direction button on the Figure 4. Wii joystick Wii joystick. • Wi-Fi Module: Now that all the external inputs Byte 69~71: For the Y analog direction button on the mentioned above are ready, the data is gathered and Wii joystick. stored inside Arduino Due, and is ready for transmission to the Android phone. In order to Byte 72: For the reset button z on the Wii joystick. establish communications between Arduino and the • Android phone, we used the ESP8266 Wi-Fi module VR Helmet: As the visual port for this VR game, the as the Wi-Fi transceiver. The ESP8266 Wi-Fi module study used the Android platform for the VR headset. is a chip that uses the TCP/IP protocol, which allows In the VR mode of the cellphone, the screen is divided any microcontroller to connect to the Wi-Fi network. into two sections as shown in figure 6, one for right Each ESP8266 module was preset with an AT eye and the other for left eye. After the cellphone is command, which means that you can simply combine placed inside the headset, the structure inside the the ESP8266 and Arduino devices to enable Wi-Fi on helmet can combine these two divided screen sections, the Arduino board as a Wi-Fi transceiver. thus providing a 3D visual effect for human eyes. Using the gyroscope inside the phone, the game can • Wi-Fi Data Receive and Analyze: Inside the Unity track rotation with the user’s head rotation. program, C# was used with the TCP/IP protocol to establish Wi-Fi communications with ESP8266. The data which needs to be transmitted are signals from the four gyroscopes and two Wii joysticks. The study used the substring in C# to analyze and convert the data into correct information, which was used by the Unity game. Thus, the data in Arduino Due must be in a fixed format. • Wi-Fi Data Structure: The data transmitted to Unity is a byte array in a fixed format, as shown in figure 5.

Figure 6. VR mode of a screen B. Internal System • Player: Each game must have at least one player. In this game, the player model used is shown in figure 7. It has a humanoid robot appearance and contains preset joint points to make it easier to animate or Figure 5. Transmitted data control. It also has a laser pistol in its right hand. The Wii joystick in the player’s right hand was used for We take number 4367 data as an example to illustrate firing the laser pistol, while the analog button of the the meaning of these byte array data. Wii joystick in the player's left hand controlled the From byte 0 to byte 4: The number -1972 contains the movement of the player model. In the player rotation number of the right arm’s GY-521 gyroscope simulation, the gyroscope on the player’s chest was module’s quaternion i element, which is -0.1972. responsible for the rotation of the player model. The other three gyroscopes on the player’s arm will From byte 5 to byte 9: The number -0729 contains the simulate the rotation of the hand, forearm, and arm, number of the right arm’s GY-521 gyroscope respectively. module’s quaternion j element, which is -0.0729. From byte 10 to byte 14: The number +0145 contains the number of the right arm’s GY-521 gyroscope module’s quaternion k element, which is +0.0145.

• Scene: A game must certainly contain at least one scene. In this game, we used a map found in the Asset Store in Unity. To manage walkable areas in the map, AI pathfinding, and AI target tracking, Unity provides a component called Navigation. It can bake a Navigation Mesh shown in figure 10 for the Navigation Agent to recognize which area is non- walkable, walkable (the blue part in figure 10), an obstacle, or how high it can climb. In addition, with the help of the tag system in Unity, the Navigation Agent can easily track and find the optimized path to arrive at the designated target.

Figure 7. Player model • Enemy: This game has three kinds of enemies that use the same model, as shown in figure 8, and the same animation controller to reduce their data size in the game. The animation tree is shown in figure 9. Although the model and animation controller are shared, three different kinds of enemies can still be created through randomly generated numbers and diversity of animation. The first enemy moves like a zombie, walking slowly toward the player model. The second will run, while the third will crawl fast. When the enemy model is close enough to the player model, it will shift to attack mode and start punching the player model. When they are hit by bullets from the Figure 10. Game scene laser pistol in the player’s hand, they will display • Game Mechanism: Every game must have at least an death animation and later remove their corpses and goal. In this game, it is simple. The goal of the game respawn into a new enemy from a random spawn point. is to survive the waves of hostile robots by using your laser pistol to destroy them.

IV. RESULTS AND DISCUSSION The study was executed by a player wearing the external input system and running the Unity program on an Android phone. The results can be divided into two parts for discussion: A. In-Game Program Result, and B. System Integration. A. In-Game Program Result As figure 11 shows, the game contains a UI, Enemy, Scene, Enemy Drop, Enemy Corpse, Bullet and Laser Pistol, each of which are discussed in turn.

• UI: The User Interface shows the first-person shooter Figure 8. Enemy model in the game and lets you know that how the game runs on the cellphone. Your life and ammo information are also shown beneath the screen. Once your life reach zero, the game is over. Once the ammo reaches zero, you can no longer fire a Bullet from the Laser Pistol. • Enemy Drop: Once the enemy is shot by the bullet you fired, they will drop supplies. There are two kind of supplies. The first is a medkit that can increase a player’s life points once it is collected by the player. The second is an ammo box that can increase the Laser Pistol’s ammo, so that the player can continue firing bullets. • Enemy: Once the enemy is spawned, it can tell which game object is the player by identifying its tag. The AI Figure 9. Enemy animation tree is illustrated in figure 12. It can find the shortest path

to the reach the player and tries to attack. There is also V. CONCLUSION collision detection in the enemy’s hand. Once you are This study designed a Unity VR game with the help of hit by the Enemy, your lives will decrease. Arduino and gyroscopes that successfully created more external inputs in the Android platform. It costs much less than other VR systems that have external inputs, and you do not need a console or PC to play it, only your phone. But so far, the game only simulates one arm. The self-made external input system is also required to play the game, which makes a little difficult to gear up, since it includes a vest, two joysticks, and a set of three gyroscopes. In contrast, the PlayStation VR platform only requires two joysticks and a Figure 12. AI cognitive ability headset. Moreover, this game has not improved many of the traditional limitations of VR, such as lagging caused by • Laser Pistol and Bullet: Every time the player shoots, hardware or minimal haptic feedback [5]. the ammo will decrease by one. The player must use The application in this study was created for the gaming this weapon to destroy the enemy marching toward industry, but other applications like interactive tours can be him relentlessly. Once the Bullet hits the Enemy, the considered, such as housing, museums, or campus VR enemy will display death animation and become an Enemy Corpse awaiting to respawn. walking tours. Furthermore, real-time field trips are also possible [6]. Future studies will focus on enhancing the • Scene: Scene is where the game takes place. Once the system and increasing external inputs; the simulation of both enemy becomes an Enemy Corpse, it will respawn at arms will definitely be considered. random spawn points on the map. ACKNOWLEDGMENT This work was supported in part by the National Science Council, Taiwan, R. O. C. under Grant No. 106-2813-C-110 -046 -E.

REFERENCES [1] Mario Covarrubias, Monica Bordegoni, “Immersive VR for natural interaction with a haptic interface for shape rendering”, 2015. [2] Kostyantyn Kolesnyk, Roman Panchak, Vitalii Pylypenko, Ismail Abliazizov, Oleksandr Fedosyeyev, Roman Ferens, “Managing Robot Kinematics Based on Arduino Controllers Using a Unity System”, 2017, pp. 44-45. Figure 11. In-game program result [3] Ernst A. Heinz, Kai S. Kunze, Matthias Gruber, David Bannach, Paul Lukowicz, “Using Wearable Sensors for Real-Time Recognition B. System Integration Tasks in Games of Martial Arts - An Initial Experiment”, 2006, pp. 99. As figure 13 shows, the gyroscope set can simulate the [4] Mirko Suznjevic, Matija Mandurov, Maja Matijasevic, “Performance motion of one’s arm. They track the rotations of the hand, and QoE Assessment of HTC Vive andOculus Rift for Pick-and-Place forearm, and arm joints separately. The ESP8266 sends this Tasks in VR”, IEEE, 2017. information to the Android phone, and then Unity analyzes [5] Denis Vajak, Cˇ aslav Livada, “Combining Photogrammetry, 3D Modeling and Real Time Information Gathering for Highly Immersive and uses the information in the game, thus achieving the part VR Experience”, IEEE, 2017, pp. 82. of VR in this study. [6] Jason W. Woodworth Christoph W. Borsty, “Design of a Practical TV Interface for Teacher-Guided VR Field Trips”, 2115, pp. 1-6.

Figure 13. System integration