Edge Computing for Mixed Reality
Total Page：16
File Type：pdf, Size：1020Kb
Linköping University | Department of Computer and Information Science Master’s thesis, 30 ECTS | Information Technology 2019 | LIU-IDA/LITH-EX-A--19/086--SE Edge Computing for Mixed Reality Blandad virtuell verklighet med stöd av edge computing Johan Lindqvist Supervisor : Klervie Toczé Examiner : Simin Nadjm‐Tehrani Linköpings universitet SE–581 83 Linköping +46 13 28 10 00 , www.liu.se Upphovsrätt Detta dokument hålls tillgängligt på Internet ‐ eller dess framtida ersättare ‐ under 25 år frånpublicer‐ ingsdatum under förutsättning att inga extraordinära omständigheter uppstår. Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstakako‐ pior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för undervis‐ ning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta tillstånd. Allannan användning av dokumentet kräver upphovsmannens medgivande. För att garantera äktheten, säker‐ heten och tillgängligheten finns lösningar av teknisk och administrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sammanhang som är kränkande för upphovsman‐ nens litterära eller konstnärliga anseende eller egenart. För ytterligare information om Linköping University Electronic Press se förlagets hemsida http://www.ep.liu.se/. Copyright The publishers will keep this document online on the Internet ‐ or its possible replacement ‐ for a period of 25 years starting from the date of publication barring exceptional circumstances. The online availability of the document implies permanent permission for anyone to read, to down‐ load, or to print out single copies for his/hers own use and to use it unchanged for non‐commercial research and educational purpose. Subsequent transfers of copyright cannot revoke this permission. All other uses of the document are conditional upon the consent of the copyright owner. The publisher has taken technical and administrative measures to assure authenticity, security and accessibility. According to intellectual property law the author has the right to be mentioned when his/her work is accessed as described above and to be protected against infringement. For additional information about the Linköping University Electronic Press and its procedures for publication and for assurance of document integrity, please refer to its www homepage: http://www.ep.liu.se/. © Johan Lindqvist Abstract Mixed reality, or augmented reality, where the real and the virtual worlds are com- bined, has seen an increase in interest in recent years with the release of tools like Google ARCore and Apple ARkit. Edge computing, where the distributed computing resources are located near the end device at the edge of the network, is a paradigm that enables offloading of computing tasks with latency requirements to dedicated servers. This thesis studies how edge computing can be used to bring mixed reality capabilities to mobile end devices that lack native support for that. It presents a working prototype for delivering mixed reality, evaluates the different technologies in it in relation to stability, responsiveness and resource usage, and studies the requirements on the end and edge devices. The experimental evaluation revealed that transmission time is the most significant chunk of end-to-end latency for the developed application. Reducing that delay will have a significant impact on future deployments of such systems. The thesis also presents other bottlenecks and best practices found during the proto- type’s development, and how to proceed from here. Acknowledgments I want to give a very big thank you to my supervisor, Klervie Toczé, for all the help and feedback throughout the work on this thesis. The many discussions about the different aspects of the thesis improved it tremendously. I would like to thank my examiner, professor Simin Nadjm-Tehrani, for all the support, as well as the opportunity to do this thesis. I would also like to thank my opponent Oskar Gustafsson for valuable comments and suggestions for improvements. Finally, I must express my great gratitude to my parents and my wife, Tina, for providing me with support and encouragement throughout the work for this thesis. This thesis would not have been possible without them. iv Contents Abstract iii Acknowledgments iv Contents v List of Figures vii List of Tables viii 1 Introduction 1 1.1 Motivation .......................................... 1 1.2 Aim .............................................. 2 1.3 Problem definition ..................................... 2 1.4 Approach .......................................... 3 1.5 Structure ........................................... 3 2 Background 4 2.1 Platforms .......................................... 4 2.2 Edge computing ...................................... 5 2.3 Computer video encoding ................................. 6 2.4 Video streaming ...................................... 7 2.5 Computer vision ...................................... 9 2.6 Related works ........................................ 13 3 The Two Prototypes 14 3.1 Local prototype ....................................... 14 3.2 Edge prototype ....................................... 15 3.3 Network communication for the edge prototype .................... 19 3.4 Video streaming for the edge prototype ........................ 21 3.5 Mixed reality for the edge prototype .......................... 23 4 Evaluation 27 4.1 Overview ........................................... 27 4.2 Resource usage ....................................... 29 4.3 Responsiveness ....................................... 33 4.4 Stability ........................................... 38 5 Discussion 42 5.1 Method ............................................ 42 5.2 Resource usage ....................................... 43 5.3 Responsiveness ....................................... 43 5.4 Stability ........................................... 44 v 5.5 Development of the edge prototype ........................... 46 5.6 Image quality for mixed reality ............................. 46 5.7 The work in a wider context ............................... 46 6 Conclusion 47 6.1 Performance ......................................... 47 6.2 Resources at the edge ................................... 47 6.3 Bottlenecks and encountered issues ........................... 48 6.4 Future work ......................................... 48 Bibliography 49 A Implementation Details 54 A.1 End device .......................................... 54 A.2 Edge Device ......................................... 55 A.3 GStreamer .......................................... 55 A.4 Camera calibration ..................................... 56 A.5 Functional testing ..................................... 58 A.6 Modifications of ORB-SLAM2 .............................. 58 A.7 Graphics rendering ..................................... 59 A.8 Measurements ........................................ 60 vi List of Figures 2.1 GStreamer pipeline ....................................... 10 3.1 Image from the local prototype, displaying mixed reality content. 15 3.2 An overview of the edge prototype. ............................. 16 3.3 Image from the end device running the edge prototype, displaying mixed reality content .............................................. 17 3.4 The components on the edge device and how they interact. 17 3.5 The components on the end device, arrows denoting interaction between the com- ponents. ............................................. 18 3.6 The end device’s application’s start menu. ......................... 19 3.7 Example of communication in the Edge Prototype. ................... 20 3.8 Close-up view of the mixed reality graphics. ........................ 25 4.1 The reference video used for testing. ............................ 29 4.2 Average tracked point cloud features for different streaming methods and bitrates. 31 4.3 Processing power usage on the edge device for the different configurations, with and without the Mixed reality component enabled. ................... 33 4.4 Processing power usage on the end device for the two prototypes. 34 4.5 Time measured for adding virtual objects for the two prototypes. 35 4.6 Round-trip times for the two prototypes. .......................... 36 4.7 The edge device’s components involved in the processing of a video frame. 37 4.8 The observed processing times on the edge device .................... 38 4.9 Frames per second displayed on the end device running the edge prototype. 40 4.10 Frames per second displayed on the end device running the local prototype. 40 4.11 Frames per second received by the the edge device .................... 41 4.12 Percentage not processed, of the number of frames arrived at the edge device. 41 5.1 Example of where a broken keyframe has led to also subsequent video frames being incorrect. ............................................. 45 A.1 Edge prototype camera calibration as seen on the end device. 57 vii List of Tables 2.1 The seven layers of the OSI model. ............................. 8 3.1 The evaluated video streaming and playback libraries for the end device. 22 3.2 The evaluated SLAM frameworks. .............................. 24 4.1 The four evaluated edge prototype configurations. .................... 28 4.2 Required bitrates for H.264 transmission from the