Low-Latency Head-Tracking for AR
Total Page:16
File Type:pdf, Size:1020Kb
Low-latency head-tracking for AR An exploration into open source head tracking Konrad Urdahl Halnum Thesis submitted for the degree of Master in Networks and Distributed Systems (ND) 60 credits Department of Informatics Faculty of mathematics and natural sciences UNIVERSITY OF OSLO Autumn 2020 Low-latency head-tracking for AR An exploration into open source head tracking Konrad Urdahl Halnum © 2020 Konrad Urdahl Halnum Low-latency head-tracking for AR http://www.duo.uio.no/ Printed: Reprosentralen, University of Oslo Abstract Augmented Reality (AR) and Virtual Reality (VR) are technologies gaining traction these days, where several actors are trying to find their niche and grab the market. The common denominator between many of these actors is proprietary software. There are also two main pathways within AR: mobile devices or headsets, where the latter is the focus of this thesis. Mobile AR is often in the manner of overlaying something rendered onto the screen of the mobile device, which is showing the surroundings recorded by its camera. AR headsets are in contrast often equipped with clear reflective elements that allow the user to see through them while the rendered augmentation is reflected onto these, or with a see-through screen. This thesis will explore the possibilities of developing Open Source Software (OSS) AR tracking software for such headsets. In order to do this, an Inertial Measurement Unit (IMU) has been employed for fast tracking of the users’ head, while a camera is required for enabling the tracking software to gain spatial awareness. These components have been mounted on an open-source 3D-printed AR headset, and a simple game level has been developed in a suitable game engine to test its performance. Also, two experiments have been conducted which look into the amount of delay there exists between said IMU and computer or game engine. This has been an interesting process, where several hurdles had to be overcome, such as finding compatible hardware and software, interoperability between the game engine and external APIs, and software development. In the end, this project hopes to be a solid bedrock for further development within this field, where headset, IMU, camera, and game engine have been married together and provided for further development on low-latency head-tracking for AR. i ii Contents I Introduction1 1 Motivation3 1.1 Purpose and goals.........................4 1.2 Outline...............................5 2 Background7 2.1 Rotations..............................7 2.1.1 Euler angels........................7 2.1.2 Quaternions........................8 2.2 Previous Work...........................9 2.3 Tracking..............................9 2.3.1 Inertial Measurement Unit (IMU)............9 2.3.2 Marker-based tracking.................. 10 2.3.3 Edge detection...................... 10 2.4 Rendering in AR......................... 11 2.4.1 Hand tracking....................... 11 2.4.2 Cockpit detection..................... 12 II Planning 15 3 Introduction 17 3.1 Project outline........................... 17 3.1.1 Goals............................ 17 3.1.2 Latency testing...................... 18 3.1.3 Rendering......................... 18 3.2 Process............................... 19 3.3 Equipment............................. 20 4 Augmented Reality (AR) headset 21 4.1 Project North Star Modular AR Headset............ 21 4.1.1 Hardware......................... 22 4.1.2 Software.......................... 24 4.1.3 Assembly......................... 25 4.2 Alternatives............................ 26 4.2.1 Microsoft HoloLens 1.................. 26 4.2.2 Microsoft HoloLens 2.................. 27 iii 4.2.3 Meta 2........................... 27 4.2.4 Magic Leap........................ 29 4.2.5 Aryzon........................... 29 5 Inertial Measurement Unit (IMU) 33 5.1 Tinkerforge IMU Brick 2.0.................... 33 5.1.1 Specifications....................... 34 5.1.2 Tinkerforge Brick Daemon................ 35 5.1.3 Tinkerforge Brick Viewer................ 35 5.1.4 Tinkerforge API...................... 36 5.2 Alternatives............................ 37 6 Camera 39 6.1 Camera............................... 39 6.1.1 Universal Serial Bus (USB) Camera........... 39 6.1.2 Power over Ethernet (POE) Camera.......... 41 7 Game Engines 43 7.1 Unity 3D.............................. 43 7.1.1 Overview......................... 44 7.1.2 Scripting.......................... 45 7.2 Unreal Engine........................... 47 8 Other Software 49 8.1 Visual Studio............................ 49 8.2 Kdenlive.............................. 49 8.3 Blender............................... 50 9 Integration 51 9.1 Mount............................... 51 9.2 USB hub.............................. 52 9.3 IMU................................. 53 9.4 Camera............................... 53 9.5 Cables............................... 54 9.6 Room for improvement...................... 54 III Development 57 10 Latency 59 10.1 Equipment............................. 59 10.1.1 Huawei P30 Pro...................... 59 10.1.2 Asus VG24QE....................... 60 10.1.3 Tripod........................... 60 10.1.4 Phone holder....................... 60 10.2 USB stack.............................. 60 10.2.1 Setup............................ 60 10.2.2 Software.......................... 62 10.2.3 Conducting the experiment............... 63 iv 10.2.4 Results........................... 64 10.3 Game engine............................ 65 10.3.1 Setup............................ 66 10.3.2 Software.......................... 67 10.3.3 Conducting the experiment............... 67 10.3.4 Results........................... 68 10.4 Concluding thoughts....................... 69 11 Tracking 71 11.1 Rotation.............................. 71 11.1.1 Introduction........................ 72 11.1.2 Transforming objects in Unity.............. 72 11.1.3 IMU input......................... 73 11.1.4 Calculating rotation................... 74 11.1.5 Result........................... 75 11.2 Movement............................. 76 11.2.1 Development....................... 77 11.3 Visual................................ 77 11.3.1 CCTags........................... 78 11.3.2 ARToolKit......................... 79 11.3.3 Vuforia........................... 79 11.3.4 OpenCV.......................... 80 11.3.5 OpenCVSharp....................... 80 11.3.6 EmguCV.......................... 81 11.3.7 OpenCV plus Unity................... 81 IV Conclusion 83 12 Results 85 13 Future work 87 Bibliography 89 A Source Code 99 A.1 Delay experiment code...................... 99 A.2 Unity assets............................ 99 A.3 Project North Star repository.................. 99 A.4 Videos & Unity package..................... 99 v vi List of Figures 2.1 Occlusion example........................ 13 4.1 North Star AR headset w/calibration stand [20]....... 21 4.2 North Star headset final assembly (front view)........ 26 4.3 Microsoft HoloLens 1 [67].................... 26 4.4 Microsoft HoloLens 2 [93].................... 27 4.5 Meta 2 [68]............................. 28 4.6 Magic Leap [66].......................... 29 4.7 Aryzon AR/MR [74]....................... 30 5.1 Tinkerforge IMU Brick 2.0 [18].................. 34 6.1 Universal Serial Bus (USB) Camera............... 40 6.2 Basler Power Over Ethernet (POE) Camera [5]........ 40 7.1 Unity 3D interface overview................... 44 9.1 North Star headset attachments................. 51 9.2 North Star headset (rear view).................. 52 10.1 USB latency test, an overview.................. 61 10.2 USB latency test, detail...................... 63 10.3 Delay measurements USB stack................. 66 10.4 Unity 3D latency test, detail................... 66 10.5 Delay measurements Unity 3D................. 69 11.1 Scene overview.......................... 72 11.2 Scene interior........................... 72 11.3 Marker 2×2............................. 77 vii viii List of Tables 4.1 BOE VS035ZSM-NW0 technical specifications [85]...... 23 4.2 Display Driver Board technical specifications [82]...... 23 4.3 Combiner lenses technical specifications [81]......... 24 4.4 HoloLens 1 technical specifications [67]............ 27 4.5 HoloLens 2 technical specifications [93]............ 28 4.6 Meta 2 technical specifications [68]............... 28 4.7 Magic Leap technical specifications [66]............ 29 4.8 Aryzon technical specifications [86]............... 30 5.1 Tinkerforge IMU Brick 2.0 technical specifications [108]... 35 5.2 CTiSensors CS-IM200 IMU technical specifications [26]... 37 5.3 Osmium MIMU22BL IMU technical specifications [75]... 38 6.1 Basler Ace acA2000-50gc technical specifications [5]..... 40 9.1 Power and bandwidth requirements headset attachments.. 53 10.1 Equipment overview....................... 61 10.2 Delay: USB-stack......................... 65 10.3 Delay: Unity 3D.......................... 68 11.1 Overview of sensor fusion modes, IMU [31].......... 73 ix x Glossary 3D-print Create a plastic object in 3D. i, 21, 52, 55 Asset All components of a Unity project, be it code, 3D-models, or imported plugins. 44 script What unity calls code in C#. 45, 46, 49, 67, 71, 77 xi xii Acronyms API Application Programming Interface. 18, 19, 34, 35, 36, 38, 41, 46, 48, 62, 63, 67, 73, 80 AR Augmented Reality. i, 3, 4, 5, 9, 18, 19, 20, 22, 26, 29, 35, 43, 51, 71, 72, 80, 85 CPU Central Processing Unit. 65 DDB Display Driver Board. 52, 53, 54, 55, 87 DOF Degrees of Freedom. 29, 33, 34 DRM Digital Rights Management. 23 FOV Field of View. 22, 27, 28, 29, 30 FPS