AUGMENT HOLOLENS’ BODY RECOGNITION AND TRACKING CAPABILITIES USING KINECT A thesis submitted in partial fulfillment of the requirement for the degree of Master of Science By KRISHNA CHAITHANYA MATHI B.Tech, Jawaharlal Nehru Technological University,2012 2016 Wright State University WRIGHT STATE UNIVERSITY GRADUATE SCHOOL th January 12 , 2017 I HEREBY RECOMMEND THAT THE THESIS PREPARED UNDER MY SUPERVISION BY Krishna Chaithanya Mathi ENTITLED Augment HoloLens‟ Body Recognition and Tracking Capabilities Using Kinect BE ACCEPTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF Master of Science. _____________________________________ Yong Pei, Ph.D. Thesis Director _____________________________________ Mateen M. Rizki, Ph.D. Chair, Department of Computer Science and Engineering Committee on Final Examination _____________________________________ Yong Pei, Ph.D. _____________________________________ Mateen M. Rizki, Ph.D. _____________________________________ Paul Bender, Ph.D. _____________________________________ Robert E.W. Fyffe, Ph.D. Vice President for Research and Dean of the Graduate School Abstract Mathi, Krishna Chaithanya. M.S. Department of Computer Science and Engineering, Wright State University, 2016. AUGMENT HOLOLENS‟ BODY RECOGNITION AND TRACKING CAPABILITIES USING KINECT. In this thesis, we are primarily interested in exploring the HoloLens technologies for medical practices. Particularly, we will address the limitation of HoloLens‟ capability in human body sensing, recognition and tracking. We will then introduce and demonstrate the use of Kinect to augment HoloLens‟ sensory and processing capabilities in order to produce time and space-synchronized immersive environment with both virtual body and real body of the same patient for supporting distributed medical collaborations. Specifically, we are looking at a distributed solution in which we are collecting the patient body data from Kinect, followed by body recognition and position/motion tracking processing at a server; Then we will display and align the virtual body object, composed based on the Kinect-obtained patient body or body part position information, with the real patient body in the doctor‟s augmented reality environment using HoloLens. We have implemented a prototype system to illustrate the distributed solution. Our experimental results using the prototype system demonstrate successful data flow and workflows. This results in effective time and space-synchronizations between the virtual and real bodies in the augmented reality environment. III Through the prototype system, we can track movement, e.g., an arm, in all directions by the Kinect and the coordinates will be sent to HoloLens through which we will augment the real body with a virtual meshed body composed from Kinect-collected data. We have demonstrated that the solution and prototype can effectively address related limitations of HoloLens, and successfully present in real-time the movement of a recognized subject in the form of both virtual and real world human body, and aligned with each other. We believe this work has helped lay a solid foundation for future works to build complete virtual human atlas into the AR views, which may potentially assist surgeons with context-aware guidance by displaying him/her the instructions and feeding him with only the closely associated information through the course of operation. This distributed collaborative solution is of particular advantage for doctors to master faster and more safely advanced and complicated surgical techniques, like using advanced radiation therapy system for cancer treatments by eliminating the physical barriers such as distance, space, resource. IV CONTENTS Chapter 1 INTRODUCTION ......................................................................................................................................... 1 1.1 OVERVIEW .......................................................................................................................................................... 1 1.2 History of Augmented Reality................................................................................................................................ 3 1.3 Difference and Similarities between AR and VR ................................................................................................... 5 1.4 AR Application Domains ....................................................................................................................................... 8 1.4.1 Medical ........................................................................................................................................................... 8 1.4.2 Engineering Design ........................................................................................................................................ 9 1.4.3 Sports and Entertainment ............................................................................................................................... 9 1.4.4 Military Training .......................................................................................................................................... 10 1.4.5 Augmented Mapping .................................................................................................................................... 11 Chapter 2 OVERVIEW OF HOLOLENS .................................................................................................................. 12 2.1 Basic Knowledge about HoloLens ....................................................................................................................... 12 2.2 HoloLens Specifics .............................................................................................................................................. 14 2.3 Shell and Mixed Reality ....................................................................................................................................... 14 2.4 Configuring HoloLens .......................................................................................................................................... 16 2.5 Developing Application for HoloLens ................................................................................................................. 17 2.6 Understanding of HoloLens Application Blocks .................................................................................................. 20 2.6.1 HoloLens Spatial Coordinates ...................................................................................................................... 20 2.6.2 Interact with Holograms in HoloLens .......................................................................................................... 21 2.6.3 Spatial Mapping and Environment Understanding ....................................................................................... 24 Chapter 3 KINECT-BASED HUMAN BODY RECOGNITION AND POSITION/MOTION TRACKING ....... 27 3.1 Capabilities to Meet Our Need ............................................................................................................................. 28 3.2 Body Tracking Using Kinect ................................................................................................................................ 29 Chapter 4 DEVELOMENT OF THE PROTOTYPE SYSTEM ............................................................................... 31 4.1 Design of a Basic HoloLens Project ..................................................................................................................... 31 4.2 Our Prototype System Implementation ................................................................................................................ 34 4.2.1 Client Server Architecture ............................................................................................................................ 35 4.2.2 Deploying process of the application............................................................................................................ 39 4.3 Implementation Results ........................................................................................................................................ 40 Chapter 5 CONCLUSION AND FUTURE WORK ................................................................................................... 45 5.1 Conclusion ........................................................................................................................................................... 45 5.2 Future Work ......................................................................................................................................................... 46 REFERENCES .............................................................................................................................................................. 47 V LIST OF FIGURES Figure 1 Example of Augmented Reality on a Globe Image [2] ..................................................... 2 Figure 2 augmented reality and virtual reality [10] ......................................................................... 5 Figure 3 Virtual reality using Oculus Rift device [10] .................................................................... 6 Figure 4 Augmented reality in medical field [9] .............................................................................. 8 Figure 5 Augmented reality in sports [8] ......................................................................................... 9 Figure 6 HoloLens