Wright State University CORE Scholar Browse all Theses and Dissertations Theses and Dissertations 2018 Developing a Kinect based Holoportation System Soumya Chiday Wright State University Follow this and additional works at: https://corescholar.libraries.wright.edu/etd_all Part of the Computer Engineering Commons, and the Computer Sciences Commons Repository Citation Chiday, Soumya, "Developing a Kinect based Holoportation System" (2018). Browse all Theses and Dissertations. 2231. https://corescholar.libraries.wright.edu/etd_all/2231 This Thesis is brought to you for free and open access by the Theses and Dissertations at CORE Scholar. It has been accepted for inclusion in Browse all Theses and Dissertations by an authorized administrator of CORE Scholar. For more information, please contact [email protected]. DEVELOPING A KINECT BASED HOLOPORTATION SYSTEM A Thesis submitted in partial fulfillment of the requirements for the degree of Master of Science by SOUMYA CHIDAY B.Tech., Jawaharlal Nehru Technological University, India, 2016 2018 Wright State University WRIGHT STATE UNIVERSITY GRADUATE SCHOOL December 13, 2018 I HEREBY RECOMMEND THAT THE THESIS PREPARED UNDER MY SUPERVISION BY Soumya Chiday ENTITLED Developing A Kinect Based Holoportation System BE ACCEPTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF Master of Science. ________________________________ Yong Pei, Ph.D. Thesis Director ________________________________ Mateen M. Rizki, Ph.D. Chair, Department of Computer Science Committee on Final Examination ________________________________ Yong Pei, Ph.D. ________________________________ Mateen M. Rizki, Ph.D. (Co-Advisor) ________________________________ Paul Bender, Ph.D. ________________________________ Barry Milligan, Ph.D. Interim Dean of the Graduate School ABSTRACT Chiday, Soumya. M.S. Department of Computer Science and Engineering, Wright State University, 2018. Developing a Kinect based Holoportation System. Holographic communication and distributed collaboration offer great potential for empathic computing to help remove the cost, distance, language and expertise barriers in many social and economic activities. Recent advances in AR-enhanced communication as evident by Microsoft Holoportation technology demonstrate the progress toward fully immersive collaborations distributed and remotely. Current holoportation system requires the use of extensive camera-arrays and powerful server system due to the computation demand and sensory needs to capture and reconstruct the subject of interests. Thus, they suffer in mobility and applicability in real-world scenarios. In this thesis, we present an ultra-portable holoportation system design that requires only modest hardware supports that include a Kinect v2 sensor and PC/Laptop in addition to the HoloLens devices. Using Unity 3D engine on Universal Windows Platform and HoloToolKit sharing service, we have implemented the software for holoportating a 3D holograms of a person from a remote location to a local HoloLens device over the Internet. iii This low-cost and ultra-portable holoportation system could be potentially used in various industries for skill training, real-time assistance from subject expert. iv TABLE OF CONTENTS 1. INTRODUCTION ....................................................................................................................... 1 1.1. Telepresence ......................................................................................................................... 1 1.2. Augmented Reality ............................................................................................................... 1 1.3. Potential Applications in Different Fields ............................................................................ 2 2. MICROSOFT HOLOLENS ......................................................................................................... 4 2.1. What is HoloLens? ................................................................................................................ 4 2.2. Specifications ........................................................................................................................ 5 2.2.1. Cameras and Sensors: .................................................................................................... 6 2.2.2. Voice Commands: .......................................................................................................... 7 2.2.3. Hand Gestures: ............................................................................................................... 7 2.2.4. Spatial Mapping: ............................................................................................................ 8 2.3. Limitations ............................................................................................................................ 9 2.4. HoloLens and Kinect Collaboration ..................................................................................... 9 3. MICROSOFT KINECT ............................................................................................................. 11 3.1. What is Kinect? ................................................................................................................... 11 3.2. Kinect SDK ......................................................................................................................... 12 3.3. Cameras and Data Streams ................................................................................................. 12 3.3.1. IR Projector and Depth Camera: .................................................................................. 12 3.3.2. RGB Camera: ............................................................................................................... 13 3.3.3 Skeletal Data: ................................................................................................................ 14 3.3.4. Construct 3D holographic video: ................................................................................. 14 4. HOLOPORTATION .................................................................................................................. 15 4.1. What is Holoportation? ....................................................................................................... 16 4.1.1. Cave Camera Set-Up:................................................................................................... 17 4.2. Mobile Holoportation .......................................................................................................... 18 4.3. Limitations .......................................................................................................................... 19 5. IMPLEMENTATION OF KINECT-BASED HOLOPORTATION PROTOTYPE SYSTEM 20 5.1. Unity 3D Engine ................................................................................................................. 20 v 5.2. HoloToolKit ........................................................................................................................ 22 5.3. HoloToolKit Sharing Service ............................................................................................. 23 5.4. Components of Hologram ................................................................................................... 24 5.4.1. Mesh:............................................................................................................................ 24 5.4.2. Texture: ........................................................................................................................ 24 5.5. Server Design ...................................................................................................................... 25 5.6. Client Design ...................................................................................................................... 27 5.7. Holographic Emulation ....................................................................................................... 28 5.8. Building an UWP Application ............................................................................................ 30 5.8.1. Player Settings in Unity: .............................................................................................. 30 5.8.2. Build Settings in VS2017: ........................................................................................... 30 5.8.3. Deploy: ......................................................................................................................... 31 5.9. Implementation Results ...................................................................................................... 31 5.10. Rear End of Hologram ...................................................................................................... 32 5.10.1. Scanned 3D Model for Rear End: .............................................................................. 34 6. CONCLUSION AND FUTURE WORK .................................................................................. 35 REFERENCES .............................................................................................................................. 36 vi LIST OF FIGURES Figure Page Figure 1. An expert helping customer remotely ...…………………………………. 3 Figure 2.1. Components of HoloLens …. ……………………………...………...... 5 Figure 2.2. Bloom Gesture give this Home Page on HoloLens … …...……...……. 8 Figure 2.3 Spatial Mapping of Living Room ….…………………………………... 9 Figure 3.1. Components of Kinect ….……………………………… …………….