Survey of Virtual and Augmented Reality Implementations for Development of Prototype for Practical Technician Training
Total Page:16
File Type:pdf, Size:1020Kb
Computer Science, Degree Project, Advanced Course, 15 Credits Survey of Virtual and Augmented Reality Implementations for Development of Prototype for Practical Technician Training Tobias Lindvall & Özgun Mirtchev Computer Engineering Programme, 180 Credits Örebro, Sweden, Spring 2017 Examiner: Franziska Klügl Örebro Universitet Örebro University Institutionen för School of Science and Technology Naturvetenskap och Teknik SE-701 82 Örebro, Sweden 701 82 Örebro Abstract Virtual training is a vital way of educating technicians to make them prepared for real maintenance work. Technicians that are educated to perform maintenance work on JAS 39 Gripen, complete parts of their training through the application Virtual Maintenance Trainer (VMT), which can provide a detailed simulation of the aircraft during operation. The technicians are able to complete courses and lessons with specific procedures such as debugging internal computers and parts of the aircraft and performing maintenance work to fix errors. However, the application is desktop-based and to make the education even more effective, there is a desire to explore the possibilities in virtual and augmented reality. This report explores the alternatives of education tools in virtual reality and augmented reality through a survey. In the survey, the advantages and disadvantages of current implementations are examined to provide an optimal system which could work to give technicians a realistic practical training simulation experience. Based on the results of the survey and by using the game engine Unity, a prototype application is built which can simulate technician training procedures on a model of JAS 39 Gripen. HTC Vive and Leap Motion were used to immerse the user into the simulation world and to enable realistic interaction. A technician may be able to learn through completing different training procedures in the simulation by walking around and interacting with a full-scaled Gripen aircraft. Sammanfattning Virtuell träning är ett viktigt sätt att utbilda tekniker för att förbereda dem för underhållsarbete i verkligheten. Tekniker som utbildas för att utföra underhållsarbete på JAS 39 Gripen, genomför delar av utbildningen genom programmet Virtual Maintenance Trainer (VMT), som kan återge en detaljerad simulering av flygplanet under drift. Teknikerna kan delta i lektioner med specifika uppgifter som inkluderar att felsöka interna datorer och delar av flygplanet samt utföra underhållsarbete för att åtgärda fel. Programmet är dock skrivbordsbaserat och för att göra utbildningen mer effektiv, finns det en önskan om att utforska möjligheterna i virtual och augmented reality. Denna rapport undersöker alternativen för utbildningsverktyg i virtual reality och augmented reality genom en teoretisk undersökning. I undersökningen vägs fördelar och nackdelar för nuvarande implementeringar för att tillhandahålla ett optimalt system som kan fungera för att ge tekniker praktisk erfarenhet i en realitisk träningssimulering. Baserat på resultaten från undersökningen och genom att använda spelmotorn Unity, har en prototypsapplikation skapats som kan simulera teknikerutbildning på en modell av JAS 39 Gripen. HTC Vive och Leap Motion användes för att låta användaren kliva in i simuleringsvärlden och för att möjliggöra realistisk interaktion. En tekniker kan lära sig att utföra underhållsåtgärder genom att genomföra olika träningsförfaranden i simuleringen genom att gå runt och interagera med ett fullskaligt Gripen-flygplan. Preface A big thank you to our supervisor at Saab, Johan Gustafsson, for all the help, encouragement and support during the exam work. A special thanks to Linus Lindberg at Saab, who helped us by providing valuable technical knowledge regarding 3D modelling. Many thanks to our supervisor at the university, Andrey Kiselev, for giving feedback for writing the report and for enabling us to use available hardware for the prototype. Also thank you to our examiner, Franziska Klügl, for help with writing the report. i Table of Contents List of Figures iv Abbreviations v 1 Introduction 1 1.1 Background . 1 1.1.1 Saab . 1 1.1.2 Virtual Maintenance Trainer . 1 1.2 Project . 3 1.3 Objective . 3 1.4 Requirements . 4 1.5 Division of Labour . 4 2 State-of-the-Art Survey 5 2.1 Introduction . 5 2.2 Clarification of Virtual and Augmented Reality . 5 2.3 Guidelines for Training in VR and AR . 6 2.4 Architecture . 6 2.5 Virtual Reality . 7 2.5.1 Introduction . 7 2.5.2 Hardware and Software . 7 2.5.3 Related Studies and Implementations . 12 2.6 Augmented Reality . 17 2.6.1 Introduction . 17 2.6.2 Hardware and Software . 17 2.6.3 Related Studies and Implementations . 20 2.7 Result . 24 2.7.1 Multi-user Capabilities . 24 2.7.2 Graphics and System Performance . 25 2.7.3 User Interactions . 29 2.7.4 Miscellaneous . 30 2.8 Conclusions . 31 2.8.1 Result Evaluation . 31 2.8.2 Summary . 32 3 Methods and Tools 33 3.1 Methods . 33 3.1.1 Development Planning . 33 3.1.2 Event-Driven Implementation . 33 3.2 Tools . 35 3.2.1 Hardware . 35 ii 3.2.2 Software . 36 3.2.3 Development Frameworks . 36 3.3 Other Resources . 37 4 Prototype Development 38 4.1 Refuel Procedure Training . 38 4.1.1 Ground Crew Panel . 38 4.1.2 Refuelling Access Panel . 38 4.1.3 Refuel Equipment . 39 4.1.4 Instructions . 39 4.2 Result . 40 4.2.1 Virtual Model Representations . 40 4.2.2 Interaction Design . 43 4.3 Discussion of Implementation . 48 4.3.1 Expenditure . 48 4.3.2 Evaluation of Used Hardware . 48 4.3.3 Development Potential and Future Developments . 49 5 Discussion 50 5.1 Compliance with the Project Requirements . 50 5.1.1 Summary of the Requirements . 50 5.1.2 Fulfilled Requirements . 50 5.2 Impact on Society . 51 5.3 Project Development . 52 5.4 Reflection on Own Learning . 52 5.4.1 Knowledge and Comprehension . 52 5.4.2 Proficiency and Ability . 52 Bibliography 53 Appendix A Setting up Unity with HTC Vive and Leap Motion 62 Appendix B Prototype System Diagram 64 Appendix C Demonstration of Implementation 65 C.1 Menu Interaction . 65 C.2 Refuel Procedure . 65 iii List of Figures 1.1.1 An overview of the VMT system . 2 2.2.1 The RV Continuum . 5 2.4.1 Basic VR and AR System Overview . 7 2.5.1 Oculus Rift with Touch . 8 2.5.2 HTC Vive with controllers and Lighthouse basestations . 9 2.5.3 Samsung Gear VR . 10 2.5.4 Plugged Leap Motion controller . 10 2.5.5 Microsoft Kinect V1 & V2 . 11 2.6.1 Microsoft HoloLens . 18 2.6.2 Meta Company’s Meta 2 . 18 2.6.3 Google Glass . 19 2.6.4 Google Cardboard . 19 2.7.1 Image illustrating the FoV of HoloLens for displaying virtual objects . 28 3.0.1 A top-down view of the tracked area by using the Room Overview Window in SteamVR . 34 3.2.1 Basic Prototype System Overview . 35 3.2.2 HTC Vive with a front-attached Leap Motion . 35 4.1.1 Panels . 39 4.2.1 JAS 39C Gripen . 41 4.2.2 Refuel equipment . 41 4.2.3 Hand models . ..