LOW-COST POSITIONAL TRACKING FOR VIRTUAL REALITY APPLICATIONS A Degree Thesis Submitted to the Faculty of the Escola Tècnica d'Enginyeria de Telecomunicació de Barcelona Universitat Politècnica de Catalunya by Enric Moreu In partial fulfillment of the requirements for the degree in AUDIOVISUAL SYSTEMS ENGINEERING Advisor: Joan Llobera, Josep R. Casas Pla Barcelona, February 2018 Abstract The aim of the project is the implementation of a tracking system able to follow a ping-pong paddle and represent it precisely, and in real-time in a virtual reality environment. The data is obtained by several optical sensors and an inertial motion unit attached to the racket. Finally, to demonstrate the accuracy and immersion of the system, a virtual ping-pong game is implemented. 1 Resum L'objectiu del projecte és la implementació d'un sistema de seguiment capaç de seguir una pala de ping-pong i representar-la amb precisió, i en temps real, en un entorn de realitat virtual. Les dades són obtingudes per diversos sensors òptics i una unitat de moviment inercial units a la raqueta. Finalment, per demostrar la precisió i la immersió del sistema, s’implementa un joc de ping-pong virtual. 2 Resumen El objetivo del proyecto es la implementación de un sistema de seguimiento capaz de seguir una pala de ping-pong y representarlo con precisión, y en tiempo real en un entorno de realidad virtual. Los datos se obtienen mediante varios sensores ópticos y una unidad de movimiento inercial conectados a la raqueta. Finalmente, para demostrar la precisión e inmersión del sistema, se implementa un juego virtual de ping-pong. 3 Revision history and approval record Revision Date Purpose 0 21/11/2017 Document creation 1 21/01/2018 Document revision DOCUMENT DISTRIBUTION LIST Name e-mail Enric Moreu [email protected] Josep R. Casas Pla [email protected] ​ Joan Llobera Mahy [email protected] ​ Written by: Reviewed and approved by: Date 29/12/2017 Date 24/01/2018 Name Enric Moreu Name Joan Llobera Position Project Author Position Project Supervisor 4 Table of contents Abstract 1 Resum 2 Resumen 3 Revision history and approval record 4 Table of contents 5 List of Figures 6 List of Tables 8 Introduction 9 Statement of purpose 9 Requirements and specifications 9 Organization of this thesis 10 State of the art 12 Technologies for object tracking in 3D 12 Immersive Virtual Reality 12 Tools used: 13 HTC Vive 13 Steam VR 13 Steam Hardware Developer Kit 14 Hardware assembly 16 System overview 16 Sensors position optimization 17 Calibration of the controller 19 Haptics 20 VR Demonstration environment 22 Demonstration 22 Network implementation 24 Budget 26 Conclusions 27 Bibliography 28 Appendices 29 5 Annex A. Detailed guide to create a custom controller 29 Annex B. JSON file 42 6 List of Figures Figure 1. Gantt diagram. ...................................................................................................11 Figure 2. HTC Vive hardware. ..........................................................................................13 Figure 3. Watchman Core Module. ...................................................................................14 Figure 4. EVM Application Board. ....................................................................................14 Figure 5. Sensor Breakout Board. ....................................................................................14 Figure 6. Optical Sensors. ..................................................................................................14 ​ ​ Figure 7. Assembled HDK ................................................................................................15 Figure 8. System overview. ..............................................................................................16 Figure 9. Optimal sensors positions 3D representation....................................................18 Figure 10. Optimal sensors positions 2D representation .................................................18 Figure 11. Assembled paddle ...........................................................................................19 Figure 12. IMU calibration procedure. ..............................................................................20 Figure 13. EVM application board electronic scheme.......................................................21 Figure 14. Haptics output. …………………………………………………..………..………..21 Figure 15. 3G vibrator. ......................................................................................................22 Figure 16. Demonstration scheme. ..................................................................................23 Figure 17. Demonstration environment ............................................................................23 Figure 18. Network overview. ...........................................................................................24 7 8 List of Tables Table 1.Budget. ................................................................................................................26 9 1. Introduction The current revolution in hardware technology for virtual reality (VR) applications still presents one major bottleneck: full body tracking is difficult to achieve, imprecise in its quality, or beyond the reach of consumer budgets. In this context, the recent release and open-sourcing of the consumer-targeted HTC Vive VR equipment [1] has prompted the development of custom hardware [2] peripherals using the SteamVR tools suite [3]. 1.1. Statement of purpose The purpose of this project is to design, implement and validate a custom VR controller with high precision tracking, and to demonstrate how it works in a virtual reality demonstrator involving a physical simulation of a ping pong ball render in immersive virtual reality. The controller consists of a ping-pong racket with several sensors attached that can be used with the HTC Vive Head Mounted Display (HMD) and a 3D scenario in order to interact with the virtual ping pong ball, the game engine used in the simulation part will be Unity 3D [4] because of its GPU integration and the different platforms that can work it. The project is carried out in the i2CAT Foundation, in the Internet & Media department and supervised by professor Joan Llobera. 1.2. Requirements and specifications The requirements of the project remained the same during the thesis. However, the specifications changed due to internal and external factors. Project requirements ● Demonstrate the use of a tracking-enabled object in VR. ● Create a novel use case scenario involving the physical object, in a consistent virtual reality experience. ● Explore a use case scenario and perform user tests. ● Study to which extent such a scenario can contribute to the feeling of being there, and of being there together, within the VR environment. Project specifications The language used to develop the game is C# under Unity 3D. ● Create a 3D model of a real paddle using Blender [5]. ● Track a ping pong racket using sensors: This is going to be an electronic task, knowing the location of the sensors, the system should predict the position and orientation of the racket. ● Calibrate the racket sensors to get an accurate tracking. ● Integration in Unity 3D: Simulation of the physics of the racket, the ball, and the ping-pong table. 10 ● Connection with other players: The system should be able to sustain a ping-pong match with two persons. This will not consist in developing the network infrastructure, but rather using an existing one for this purpose ● Use a real ball that is simulated with precision by the virtual ball that the user will see. Make tests to confirm the precision of the simulation, combining different factors (physical and virtual ball combined or only virtual ball, custom controller or typical controller) ● Validate the usability of the proposed solution with real users 1.3. Motivation We are living in a society where the technology advances faster, it’s accessible for everybody, and it’s getting cheaper. In consequence, we find some problems due to his rapid evolution science: the technology isolates the people, reducing their social lives and the time that spend with family or friends. In this context, the Internet and Media department of the i2Cat, decided to make a solution using the last low budget virtual reality releases to engage people to spend more time socializing. The scope of this project is a virtual reality ping-pong game where people can play together via the internet, in order to participate in the VR Together european project, which is a non profit initiative that invest the improvement of the VR interactions between people. 1.4. Organization of this thesis This project is divided into four packages: 1. Assembly of the controller: In the first part of my thesis, I attached the optical sensors to the optimal positions of the paddle and verified the connectivity with the computer. Also, create a 3D model of the paddle. 2. Calibration of the sensors: Refine the position and orientation of the optical sensors and the IMU to improve the tracking. 3. Integration with the game engine: Display the paddle the Unity 3D with realistic physics. 4. Gameplay experience: Create an environment to play ping-pong with a second player over the network. 11 Figure 1. Gantt diagram 12 2. State of the art The virtual reality revolution still experiences a huge growth , and is expected to continue growing in the next few years. Several articles about VR were written in the last years [6]. To analyze the state of the