TALLINN UNIVERSITY OF TECHNOLOGY Faculty of Information Technology Department of Computer Control Andrei Mihnevitš Towards Oculus Rift based Augmented and Virtual Reality Bachelor’s Thesis Supervisor(s): Eduard Petlenkov, Aleksei Tepljakov Tallinn 2015 Declaration: I hereby declare that this Bachelor’s thesis, my original investigation and achieve- ment, submitted for the Bachelor’s degree at Tallinn University of Technology, has not been submitted for any degree or examination. Deklareerin, et käesolev bakalaureusetöö, mis on minu iseseisva töö tulemus, on esitatud Tallin- na Tehnikaülikooli bakalaureusekraadi taotlemiseks ja selle alusel ei ole varem taotletud aka- deemilist kraadi. Andrei Mihnevitš Date: . Signature: . Contents Abstract 3 Kokkuvõte 4 1 Introduction 7 1.1 State of the Art . .7 1.2 Objectives and Contributions . .8 1.3 Thesis Outline . .8 2 Overview of the Oculus Rift Development Kit 10 3 Selection of API for Rendering Computer Graphics 12 4 Software Environment for Application Development 14 4.1 Alternative Implementations . 14 4.2 Overview of the Qt Toolkit . 15 4.3 The Reasons to Choose Qt . 15 4.4 C++ as Programming Language for Software Environment . 16 5 Building the Code Base 17 5.1 Overview of the Architecture . 17 5.2 OpenGlApp Class . 18 5.2.1 Deriving from QWindow and QOpenGLFunctions Classes . 18 1 5.2.2 OpenGL Rendering Loop . 19 5.3 RiftApp Class . 22 5.3.1 A Subclass of OpenGlApp ....................... 22 5.3.2 Initializing Oculus Library and the HMD . 23 5.3.3 OVR Rendering Loop . 27 5.4 CustomApp class . 31 6 Developing a Custom Application using the Code Base 32 6.1 Stereo Video Stream . 32 6.2 The interconnection of the HMDs . 33 6.3 Implementation details . 34 6.4 The Working Application Results . 40 Conclusions 42 Bibliography 43 2 Abstract Towards Oculus Rift based Augmented and Virtual Reality The goal of this thesis was to create a software code base for developing Oculus Rift based vir- tual and augmented reality applications using Qt toolkit and Oculus SDK. In order to make the code base beneficial to a larger amount of developers, only cross-platform tools were used in this work. During the development process, several important topics were investigated, includ- ing rendering computer graphics with OpenGL and understanding the inner workings of Oculus SDK. A software solution was developed for building custom applications focused on virtual and augmented realities. Resulting working applications showed, that the goal of the thesis was successfully achieved. 3 Kokkuvõte Oculus Riftil põhinev täiendatud ja virtuaalne reaalsus Käesolev bakalaureuse töö käsitleb tarkvarateegi loomist Oculus Rift-il põhinevate virtuaalse ja täiendatud reaalsuse rakenduste arendamiseks Qt paketi ning Oculus SDK baasil. Välja paku- tud lahendus on universaalne, kuna põhineb ainult platvormiülestel lahendustel. Töö uurib ka arvutigraafika genereerimise võimalusi OpenGL teegi abil. Välja töötatud tarkvaraline lahendus loob aluse virtuaalse ja täiendatud reaalsuse rakenduste arendamiseks. 4 List of Figures 2.1 Oculus Rift DK2 . 11 5.1 Architecture Class Diagram . 17 5.2 Rendering steps in the Rift . 26 5.3 Rift eye textures and viewports . 27 6.1 Webcams mounted on the HMD . 33 6.2 HMDs interconnection concept . 34 6.3 UML class diagram for master’s reality application . 35 6.4 The Master’s Reality application multithreading scheme . 39 6.5 A pair of stereo images . 41 5 Nomenclature API Application Programming Interface DK2 Development Kit 2 FBO FrameBuffer Object FOV Field Of View GUI Graphical User Interface HMD Head-Mounted Display IT Information Technology OVR Oculus Virtual Reality SDK Software Development Kit UDP User Datagram Protocol UI User Interface UML Unified Modeling Language VR Virtual Reality XML EXtensible Markup Language 6 Chapter 1 Introduction 1.1 State of the Art With the advent of the first Oculus Rift Development Kit—the DK1—in 2012, the interest in virtual reality has experienced an enormous boost. The Oculus Rift has influenced the creation of a great amount of virtual reality content. Although much of the effort has been devoted to bringing virtual reality into the gaming industry, there are also other interesting applications of the experience provided by the Oculus Rift [1], [2], [3], [4]. It can be seen, that while game developers primarily focus on virtual reality—an artificial world with computer-generated envi- ronment, developers and researchers from other fields of IT tend towards augmented reality—an enhanced version of reality created by overlaying computer-generated objects on an image of real world viewed through a video capturing device, e.g. a webcam. The speech of Michael Abrash [5]—Chief Scientist at Oculus—can be considered a very great source of inspiration to embark on VR research. During his speech he gave a vision of the future of VR: Imagine anyone on the face of the Earth being able to be anywhere with anyone doing anything whenever they want. Following the idea of this quote, one can imagine futuristic scenarios where a person wearing a VR headset controls a robot—the avatar—on Mars fully immersing into the robot’s Martian reality. Something more realistic regarding the near future would be to use such an avatar to explore places on Earth unavailable to a human. 7 1.2 Objectives and Contributions This work focuses on the starting point of creating virtual and augmented reality applications— the development of software code base. This code base has to be easily extensible in order to build customized applications on top of it. The solution in this work uses inheritance in object- oriented design of the code base. By deriving from the classes provided by the code base, a custom application gets all the required features to interact with the Rift and implement its own specific logic. One of the challenges faced during the work was to use only the tools provided by the most recent Qt5 toolkit in conjunction with Oculus SDK, because the code samples available on the internet either used totally different tools, or used older versions of Qt. From this point of view, the use of modern Qt5 introduced novelty to this work. In order to prove the results of this work, a custom software application is built using the created code base. This application has to show, that: • The ’Qt5 only’ approach actually works; • The performance of the application is satisfactory. 1.3 Thesis Outline Chapter 2 gives an overview of the Oculus Rift. Due to the fact, that this work uses DK2, the focus in the chapter is laid on this particular development kit. Chapter 3 deals with the selection of computer graphics API. OpenGL and Direct3D are first introduced to the reader and then compared from the perspective of current work. Finally, one of the APIs is chosen to be used in the work. Chapter 4 starts with an overview of applicable software frameworks, which suit the current task. After that, the attention gets focused on Qt framework and the advantages of using it in the current work. Chapter 5 focuses on the code base implementation details. The explanation starts with an overview of the code base architecture followed by highlighting the most important parts of the code. 8 In Chapter 6 the reader learns, how a custom application can be developed using the code base. An application called Master’s Reality was chosen as an example of such software solution development. 9 Chapter 2 Overview of the Oculus Rift Development Kit The development kits released by Oculus [6] are aimed at game developers and enthusiasts in order to allow them to integrate new amazing experiences into their games and applications. This approach ensures, that by the time the consumer version is released, there will already be a large community of software developers involved with the Rift. The latest development kit for the Oculus Rift—the DK2—was released in 2014. This devel- opment kit represents a headset, which allows the wearer of this device to immerse into virtual reality. Figure 2.1 presents the DK2 headset in a way it is advertised by Oculus. In order to get the feeling of what the Rift can deliver to the user, it is worth quoting Palmer Luckey—the founder of Oculus—as he said: The magic, that sets the Rift apart, is immersive stereoscopic 3-D rendering, a massive field of view and ultra low latency head-tracking. To follow his words one needs to be aware, that while other consumer head-mounted displays, e.g. [7], have a diagonal FOV of about 40 degrees, the Oculus Rift makes an impressive step by providing a 100 degrees in its DK2. As a consequence, a person wearing the Rift is not simply looking at the screen, but gets the impression of actually being inside of the world. Also, the latency is so far regarded to be the biggest problem of virtual reality. The higher the latency, the less realistic the virtual scene will appear. Moreover, latency issues can cause disorientation of the user in a purely virtual environment, resulting in motion sickness [8]. The developers at 10 Oculus find this problem so important, that they built a special latency testing system into the DK2. An increasing popularity of the Rift is also caused by the fact, that it has out-of-the-box inte- grations for a number of modern game engines, such as Unreal Engine 4 [9] and Unity 4 [10], which gives game developers a great advantage. Finally, the Oculus Rift comes with the publicly available Oculus SDK, which includes all the necessary—the source code, documentation and samples—to create Oculus Rift based software applications. Figure 2.1: Oculus Rift DK2 11 Chapter 3 Selection of API for Rendering Computer Graphics Basically there are two options to consider—Direct3D and OpenGL.