Survey of Virtual and Augmented Reality Implementations for Development of Prototype for Practical Technician Training

Computer Science, Degree Project, Advanced Course, 15 Credits

Tobias Lindvall & Özgun Mirtchev

Computer Engineering Programme, 180 Credits

Örebro, Sweden, Spring 2017

Examiner: Franziska Klügl

Örebro Universitet Örebro University Institutionen för School of Science and Technology Naturvetenskap och Teknik SE-701 82 Örebro, Sweden 701 82 Örebro Abstract

Virtual training is a vital way of educating technicians to make them prepared for real maintenance work. Technicians that are educated to perform maintenance work on JAS 39 Gripen, complete parts of their training through the application Virtual Maintenance Trainer (VMT), which can provide a detailed simulation of the aircraft during operation. The technicians are able to complete courses and lessons with specific procedures such as debugging internal computers and parts of the aircraft and performing maintenance work to fix errors. However, the application is desktop-based and to make the education even more effective, there is a desire to explore the possibilities in virtual and augmented reality. This report explores the alternatives of education tools in virtual reality and augmented reality through a survey. In the survey, the advantages and disadvantages of current implementations are examined to provide an optimal system which could work to give technicians a realistic practical training simulation experience. Based on the results of the survey and by using the game engine Unity, a prototype application is built which can simulate technician training procedures on a model of JAS 39 Gripen. HTC Vive and Leap Motion were used to immerse the user into the simulation world and to enable realistic interaction. A technician may be able to learn through completing different training procedures in the simulation by walking around and interacting with a full-scaled Gripen aircraft.

Sammanfattning

Virtuell träning är ett viktigt sätt att utbilda tekniker för att förbereda dem för underhållsarbete i verkligheten. Tekniker som utbildas för att utföra underhållsarbete på JAS 39 Gripen, genomför delar av utbildningen genom programmet Virtual Maintenance Trainer (VMT), som kan återge en detaljerad simulering av flygplanet under drift. Teknikerna kan delta i lektioner med specifika uppgifter som inkluderar att felsöka interna datorer och delar av flygplanet samt utföra underhållsarbete för att åtgärda fel. Programmet är dock skrivbordsbaserat och för att göra utbildningen mer effektiv, finns det en önskan om att utforska möjligheterna i virtual och augmented reality. Denna rapport undersöker alternativen för utbildningsverktyg i virtual reality och augmented reality genom en teoretisk undersökning. I undersökningen vägs fördelar och nackdelar för nuvarande implementeringar för att tillhandahålla ett optimalt system som kan fungera för att ge tekniker praktisk erfarenhet i en realitisk träningssimulering. Baserat på resultaten från undersökningen och genom att använda spelmotorn Unity, har en prototypsapplikation skapats som kan simulera teknikerutbildning på en modell av JAS 39 Gripen. HTC Vive och Leap Motion användes för att låta användaren kliva in i simuleringsvärlden och för att möjliggöra realistisk interaktion. En tekniker kan lära sig att utföra underhållsåtgärder genom att genomföra olika träningsförfaranden i simuleringen genom att gå runt och interagera med ett fullskaligt Gripen-flygplan. Preface

A big thank you to our supervisor at Saab, Johan Gustafsson, for all the help, encouragement and support during the exam work. A special thanks to Linus Lindberg at Saab, who helped us by providing valuable technical knowledge regarding 3D modelling.

Many thanks to our supervisor at the university, Andrey Kiselev, for giving feedback for writing the report and for enabling us to use available hardware for the prototype. Also thank you to our examiner, Franziska Klügl, for help with writing the report.

i Table of Contents

List of Figures iv

Abbreviations v

1 Introduction 1 1.1 Background ...... 1 1.1.1 Saab ...... 1 1.1.2 Virtual Maintenance Trainer ...... 1 1.2 Project ...... 3 1.3 Objective ...... 3 1.4 Requirements ...... 4 1.5 Division of Labour ...... 4

2 State-of-the-Art Survey 5 2.1 Introduction ...... 5 2.2 Clariﬁcation of Virtual and Augmented Reality ...... 5 2.3 Guidelines for Training in VR and AR ...... 6 2.4 Architecture ...... 6 2.5 Virtual Reality ...... 7 2.5.1 Introduction ...... 7 2.5.2 Hardware and Software ...... 7 2.5.3 Related Studies and Implementations ...... 12 2.6 Augmented Reality ...... 17 2.6.1 Introduction ...... 17 2.6.2 Hardware and Software ...... 17 2.6.3 Related Studies and Implementations ...... 20 2.7 Result ...... 24 2.7.1 Multi-user Capabilities ...... 24 2.7.2 Graphics and System Performance ...... 25 2.7.3 User Interactions ...... 29 2.7.4 Miscellaneous ...... 30 2.8 Conclusions ...... 31 2.8.1 Result Evaluation ...... 31 2.8.2 Summary ...... 32

3 Methods and Tools 33 3.1 Methods ...... 33 3.1.1 Development Planning ...... 33 3.1.2 Event-Driven Implementation ...... 33 3.2 Tools ...... 35 3.2.1 Hardware ...... 35

ii 3.2.2 Software ...... 36 3.2.3 Development Frameworks ...... 36 3.3 Other Resources ...... 37

4 Prototype Development 38 4.1 Refuel Procedure Training ...... 38 4.1.1 Ground Crew Panel ...... 38 4.1.2 Refuelling Access Panel ...... 38 4.1.3 Refuel Equipment ...... 39 4.1.4 Instructions ...... 39 4.2 Result ...... 40 4.2.1 Virtual Model Representations ...... 40 4.2.2 Interaction Design ...... 43 4.3 Discussion of Implementation ...... 48 4.3.1 Expenditure ...... 48 4.3.2 Evaluation of Used Hardware ...... 48 4.3.3 Development Potential and Future Developments ...... 49

5 Discussion 50 5.1 Compliance with the Project Requirements ...... 50 5.1.1 Summary of the Requirements ...... 50 5.1.2 Fulfilled Requirements ...... 50 5.2 Impact on Society ...... 51 5.3 Project Development ...... 52 5.4 Reflection on Own Learning ...... 52 5.4.1 Knowledge and Comprehension ...... 52 5.4.2 Proficiency and Ability ...... 52

Bibliography 53

Appendix A Setting up Unity with HTC Vive and Leap Motion 62

Appendix B Prototype System Diagram 64

Appendix C Demonstration of Implementation 65 C.1 Menu Interaction ...... 65 C.2 Refuel Procedure ...... 65

iii List of Figures

1.1.1 An overview of the VMT system ...... 2

2.2.1 The RV Continuum ...... 5 2.4.1 Basic VR and AR System Overview ...... 7 2.5.1 Oculus Rift with Touch ...... 8 2.5.2 HTC Vive with controllers and Lighthouse basestations ...... 9 2.5.3 Samsung Gear VR ...... 10 2.5.4 Plugged Leap Motion controller ...... 10 2.5.5 Microsoft Kinect V1 & V2 ...... 11 2.6.1 Microsoft HoloLens ...... 18 2.6.2 Meta Company’s Meta 2 ...... 18 2.6.3 Google Glass ...... 19 2.6.4 Google Cardboard ...... 19 2.7.1 Image illustrating the FoV of HoloLens for displaying virtual objects ...... 28

3.0.1 A top-down view of the tracked area by using the Room Overview Window in SteamVR . 34 3.2.1 Basic Prototype System Overview ...... 35 3.2.2 HTC Vive with a front-attached Leap Motion ...... 35

4.1.1 Panels ...... 39 4.2.1 JAS 39C Gripen ...... 41 4.2.2 Refuel equipment ...... 41 4.2.3 Hand models ...... 42 4.2.4 Picked up refuel hose nozzle ...... 43 4.2.5 Rotating the refuel valve protection cap ...... 44 4.2.6 Button interaction ...... 44 4.2.7 Menu ...... 45 4.2.8 Laser pointer ...... 45 4.2.9 Navigation sequence ...... 46 4.2.10 Refuel manual ...... 46 4.2.11 X-Ray mode ...... 47 4.2.12 Multiple users ...... 47

A.1 View of ﬁnished project setup of VR camera and Leap Motion modules ...... 63

B.1 Diagram showing an overview of the used devices and the process of the prototype . . . . 64

iv Abbreviations

AR Augmented Reality. 3–6, 13, 17–25, 27, 29–33, 50–52

AV Augmented Virtuality. 5, 6

CAVE Cave Automatic Virtual Environment. 12–14, 24, 25, 31, 32

COTS Commercial oﬀ-the-shelf. 12, 24, 26, 31, 32

DOF Degrees of Freedom. 8, 10, 14, 15, 17, 18, 23, 27, 28, 31

FOV Field of View. 8–10, 15–19, 21, 22, 24, 26–28, 31, 48, 49

HMD Head-Mounted Display. 7–11, 13–20, 24–33, 35, 48

IMU Inertial Measurement Unit. 8, 26

SDK Software Development Kit. 10, 15, 20, 24, 33

VMT Virtual Maintenance Trainer. 1–3, 12

VR Virtual Reality. 3–8, 10–17, 20, 24–27, 29–33, 36, 49–52

v 1 Introduction

Educating technicians in practical training procedures often requires the use of expensive hardware which also might have a limited availability. A training procedure may include the technicians to do certain practical tasks in order to learn about maintenance and inspection work of a particular hardware such as a vehicle or other machines.

With the swift development of virtual reality and augmented reality, a desire among many companies has emerged to ﬁnd out how the technologies could be utilised for practical training procedures as accessible and cost-eﬀective educational tools.

This thesis includes a survey of the state of the art in virtual and augmented reality technologies. The survey focused on existing implementations for practical training and was conducted using a qualitative method. Using certain keywords related to the topic, data was congregated and then compiled to extract important information by considering diﬀerent parameters relating to the objective.

A prototype was created (based on the results of the survey) to evaluate the technical capabilities and to show the concept of a practical procedure training in virtual or augmented reality.

1.1 Background

1.1.1 Saab

This thesis has surveyed the current technologies in virtual and augmented reality. Based on the result of the survey, a suggestion was made of how Saab could develop their technician training tool.

Saab AB is a military vehicle manufacturer, founded 1937, by the Swedish government with the purpose to secure the production of Swedish fighters. One fighter in particular called JAS 39 Gripen, one of the most known fighters, started to be produced 1988 and has since then undergone four generations (A, B, C and D) and the latest in development is the E generation.

To maintain a Gripen ﬁghter, extensive knowledge about the aircraft is required by educated technicians, provided by the specialised maintenance technician training application for Gripen, Virtual Maintenance Trainer (VMT) 39, which is developed by the business area Saab Aeronautics. It allows the training of technicians who will specialise in inspecting and maintaining the Gripen ﬁghter.

1.1.2 Virtual Maintenance Trainer

VMT 39 is a desktop-based education application developed by Saab Aeronautics that runs on the VMT platform, which supports other VMT training applications for other vehicles. The platform, also developed by Saab, can be run on the most common operating systems, such as Windows, Linux and UNIX. The VMT 39 application can be used to cost-eﬀectively train Gripen technicians or pilots. It covers virtual based maintenance training through oﬀering education of the aircraft system, procedure

1 of 52 Introduction Background

Figure 1.1.1: An overview of the VMT system training, system diagnostics and fault localisation. Tailored training sessions can be created using videos, sounds, charts, interactive simulations, 3D animations, etc. The system can be used partly by a technician student for self-education but also by a teacher under instructional training course. The system runs on a dedicated workstation, with three screens and a joystick. A diagrammatical overview of the system may be seen in ﬁgure 1.1.1.

1.1.2.1 Users

Trainees and instructors constitute the types of users that can use VMT 39. Trainees connect to the application server using the Java-based client where they are allowed to practice certain tasks. Instructors use the same client program but have the rights to create lesson setups for trainees to participate in.

1.1.2.2 Course Creation

The VMT platform is using the Shareable Content Object Reference Model (SCORM) [1] standard to make e-learning content and Learning Management System (LMS) work together. This enables creation of learning material which can be used in other SCORM conformant systems.

All the created courses in an application are backed up to the server. When an instructor has created a course in a VMT application, it is stored on the server enabling the selected trainees to access the same course through client workstations. In the next version of VMT, the Tin Can API [2], will be implemented instead of SCORM, which allows for more LMS functionalities.

During the use of the VMT 39 application as an instructor, it is possible to create new courses and custom lessons. For each course, diﬀerent learning modules can be created with diﬀerent components.

2 of 52 Project Introduction

Course Modules deﬁne a lesson as a training, examination, guidance type etc. Diﬀerent modules only shows relevant components that can be added to that type of module.

Course Components are added to a chosen course module. Diﬀerent course components can in real- time display cockpit panels, show 3D models of the aircraft or show diagrams and panels to monitor electric, hydraulic and fuel ﬂows. From the components the instructor is also able to decide if there are going to be any errors during run-time for the trainee to handle.

1.2 Project

VMT uses computer displays to visualise the maintenance work during a procedure for a technician student. To make this area of the training more eﬃcient and realistic, a speciﬁc education environment was needed to provide realistic interactions with virtual models. This was achieved by using available techniques in Virtual Reality (VR) and Augmented Reality (AR) to be an auxiliary component in VMT to create a practical virtual education environment.

This work included doing a survey of diﬀerent approaches and implementations of already existing solutions for using VR and AR for educational purposes. The result of this survey would reveal whether VR or AR is appropriate to use as an educational tool with the current technology, through comparisons showing the diﬀerent advantages and disadvantages. Possible hardware recommendations will also be proposed and a prototype will be developed showing the possibility for technician training by using appropriate application development tools.

The application prototype would enable a user to execute a procedure on the aircraft by interacting with virtual models through the use of controllers or their hands. The kind of hardware or software that would be used were dictated by the survey. This application will not be integrated with VMT but will instead be a proof of concept to show that practical education in virtual or augmented reality could work as a complement for VMT. It will be developed as a standalone application but could perhaps in future VMT systems, be integrated as a separate lesson or conﬁguration.

1.3 Objective

The objective of the project was to investigate the possibilities of providing a realistic practical procedure training concept for aircraft technicians. The investigation would cover VR and AR technology and be based on three diﬀerent training situations:

a) A teacher should be able to demonstrate a procedure on a virtual engine, while the students stand around this object, all wearing head-mounted displays. The teacher should be able to rotate and turn the virtual/superimposed object. The students will in turn be able to observe the model in more detail.

b) An inspection procedure should be done by the student alone, e.g; be able to inspect the aircraft before take-oﬀ, such as controlling air-pressure in tires, refuelling the aircraft or troubleshooting cockpit errors. Diﬀerent parts of the aircraft should also be able to react to interactions, such as opening a door when making a gesture etc.

3 of 52 Introduction Requirements

c) The system should provide the tools to complete a maintenance task. The student should receive a set of instructions from the teacher and fulﬁl the requirements to complete the task. This needs to be done by using the instructions manual together with virtual tools such as a screwdriver etc.

1.4 Requirements

The project should have completed an in-depth survey of the current situation of VR and AR technologies, with a documentation of advantages and disadvantages. From the survey, investigations should indicate which appropriate technology is suggested for a practical technician training application.

A prototype application should be created to demonstrate the capabilities of the recommended technology concept and attempt to implement the three scenarios mentioned in the 1.3 Objective section.

1.5 Division of Labour

Doing the survey, the workload was shared equally where each of the authors, on their own, collected and summarised the relevant data which then was discussed. The major part of the work that had to be done for planning and implementing the prototype was accomplished in an agile way using pair programming. Occasionally the labour was separated where the authors shared the work equally with manipulation of 3D models and producing code for the prototype application.

4 of 52 2 State-of-the-Art Survey

2.1 Introduction

This survey used a qualitative research method to find appropriate data for the topic, by searching for related papers using backward reference searching. Keywords that were used are related to virtual maintenance training. Expansion of the research field was enabled by the backward reference search, to find new areas to analyse where fields such as medical and other areas were included.

This chapter starts with a brief introduction of the definition of Virtual Reality (VR) and Augmented Reality (AR) in Section 2.2 with a follow-up of the guidelines for implementing an application in both fields in Section 2.3. The survey is divided as such that in Section 2.5 and Section 2.6, virtual reality and augmented reality are introduced with an explanation of available hardware before presenting available implementations regarding practical virtual training. It is recommended to read through both of these sections in order to get a better view of the available technology in both fields before continuing on to the results of the survey. Section 2.7 is devoted to the results of the presented implementations and how virtual or augmented reality may perform with regards to different factors in practical procedure training. Finally, the conclusion of the study is presented in Section 2.8.

2.2 Clariﬁcation of Virtual and Augmented Reality

The following section will further describe the meaning of VR and AR and where it ends up in the Reality-Virtuality Continuum. The presented continuum (Figure 2.2.1) is a prescribed system to elucidate the meaning of the extremes that are included. Four classiﬁcations of environments and realities are shown, where mixed reality includes environments between the real and the virtual environments called AR and Augmented Virtuality (AV) [3].

The real environment is the natural environment one can see with their own eyes, consisting of only real objects, devoid of any artiﬁcial ampliﬁcations. VR is described as an environment consisting entirely of virtual objects, which can be observed through a display [3].

Between the real and the virtual environments, there are environments where the virtual objects are in coalescence with the real objects, oﬀering the ability to, in real-time, use real objects as virtual

Figure 2.2.1: The RV Continuum Source: [4]

5 of 52 State-of-the-Art Survey Guidelines for Training in VR and AR objects and vice versa. These environments are called AR and AV. There is no definite agreement on the definitions of these two environments, however, three factors have been defined to enable a decisive explanation of how one may differentiate between them. The three factors are divided as:

Extent of the World Knowledge (EWK) The recognition of the world and complementary objects

Reproduction Fidelity (RF) The visual quality of the reproduced real and virtual objects

Extent of Presence Metaphor (EPM) The amount of presence or immersion the observer should feel

From this information, it is possible to deﬁne AR as a virtual environment for the observer, allowing the possibility of displaying virtual objects in a real environment - making it closer to the real environment side in the Reality-Virtuality continuum graph. In contrast, Augmented Virtuality enables the observation of objects or data, from the real environment, in a virtual environment setting. Depending on the weight of each previously mentioned factors, it either brings one closer to the real environment or the virtual environment [3].

While there are studies in speciﬁc areas of mixed reality for AV, this chapter will focus on VR and AR from a ﬁrst-person perspective, exploring how the technologies work in industrial, educational and other similar contexts.

2.3 Guidelines for Training in VR and AR

Implementing a system in VR and AR requires several factors to be fulfilled to offer a realistic experience fit for technician training environments. Gavish et al. [5] has introduced four different guidelines to follow when designing VR or AR applications for maintenance training. By following the guidelines it is possible to increase the training efficiency, enhance skills acquisition and enable a development of a useful model of the task for the users. The four guidelines include:

• Proper integration of observational learning

• Enactive learning by combining physical and cognitive ﬁdelity

• Providing guidance aids

• Providing suﬃcient information about the task

Having these guidelines as well as guidelines from Oculus [6] and Leap Motion [7], it will provide a proper way of analysing the diﬀerent related implementations and help with design decisions for the prototype.

2.4 Architecture

VR and AR implementations that are observed, have a similar architecture which takes input data and computes the data in a simulation, producing some output for the user. As an example, the user may input data by moving their hands or their position. Based on this input, the simulation will calculate the new state of the virtual entities and output a new representation of the virtual environment to the user. A diagrammatical overview of the inputs and outputs between a user and the devices (Figure 2.4.1) may give a better understanding of how a typical VR or AR system work.

6 of 52 Virtual Reality State-of-the-Art Survey

Figure 2.4.1: Basic VR and AR System Overview

2.5 Virtual Reality

2.5.1 Introduction

There have been multiple technological developments and experiments in the recent years in the world of Virtual Reality (VR). New innovations are continuously emerging to bring the development forward. The VR technology has found its way into many areas of application, such as education, health care and maintenance training, to mention a few. Which speciﬁc technologies to use, ultimately depends on what task is at hand [8].

An important aspect of VR is the visual perception, enabled by different visualisation devices which may offer the user a visual experience of the virtual environment. Another aspect is user interaction, achieved by several devices today, providing different levels of immersion. For instance, using glove- or camera-based tracking techniques, the user is able to interact with a simulation by performing hand gestures or remotely controlling a virtual hand [9].

Haptic feedback is also used to increase the immersion in a virtual reality setting. A way to achieve this is through built-in actuators in hand controllers that respond to certain events or through special vests which can produce body contact simulations by electrical impulses [10].

2.5.2 Hardware and Software

This section will present hardware and software used in the world of VR today. For hardware, the description of the most prevalent devices today in the market will be included, together with any tracking capabilities they may have. The software subsection will include the description of most used engines and development frameworks to create VR applications.

2.5.2.1 Device deﬁnitions

Wired Head-Mounted Display (HMD) in VR, available in the current market are dominated by Oculus Rift and HTC Vive. The HMD’s are often used in a spacious room, connected to an external workstation with camera trackers in the corners of the room. Positional tracking is enabled by the

7 of 52 State-of-the-Art Survey Virtual Reality cameras so that the user may walk around within artiﬁcial borders and experience a six degree of freedom positioning. The artiﬁcial borders aid the user from walking too far from the boundaries of the virtual environment [10].

Non-wired HMD’s are the second type of HMD’s. Google Cardboard, Google Daydream and Samsung Gear VR are all non-wired HMD’s and require a smartphone. Compared to wired devices, these devices (without additional external tracking devices) only provide three Degrees of Freedom (DOF) rotational tracking.

Tracking and Input Devices are essential input methods in VR since it enables the virtual recreation of a user’s physical actions. Technologies today enable tracking of the head, hands or the entire body of the user to represent their actions in a responsive way. Besides tracking there are other primordial non-tracked input devices, which will be mentioned towards the end of this subsection.

2.5.2.2 Hardware

Oculus Rift is the third iteration HMD from Oculus and was released 2016. It has a combined resolution of 2160x1200 with a refresh rate of 90 Hz and a Field of View (FOV) at 110° [10].

A built-in gyroscope, magnetometer and accelerometer in the Oculus Rift HMD enables tracking of the user’s head orientation in three DOF. Through a method called sensor fusion [11], which uses inputs from the three mentioned sensors, the user’s in-game head orientation (yaw, pitch and roll) is determined. Positional tracking of the user’s head is achieved through the Constellation system developed by Oculus. This system uses micro-LED on-board the HMD, which are tracked by at least one external IR-camera (Oculus Sensor), enabling optical tracking of the user in six DOF [6, 12].

In late 2016, Oculus released Oculus Touch, which is a native handheld input device of the Oculus Rift. In a virtual environment, Oculus Touch can represent the user’s hands accurately and in an intuitive way. Positional tracking is achieved by the built-in LED’s and utilisation of the same Constellation system, used for the Oculus Rift HMD. The Oculus Sensor can be used for both Oculus Rift and the Touch. Rotational tracking for the Oculus Touch is performed using a built-in Inertial Measurement Unit (IMU), similar to the HMD, enabling tracking of the user’s hands in six DOF. The device also has touch triggers, push buttons and an analogue stick enabling user interactions e.g. gesture recognition, user orientation, grabbing virtual objects etc. [13]. An image of the HMD and the Touch controllers can be seen in Figure 2.5.1.

Figure 2.5.1: Oculus Rift with Touch Source: [14, 15]

8 of 52 Virtual Reality State-of-the-Art Survey

HTC Vive was released in 2016 and is the main competitor to Oculus Rift. Similar to Oculus Rift it has a combined resolution of 2160x1200 with a refresh rate of 90 Hz. The FOV is at 110° [16, 10]. The distinct diﬀerence to the Oculus Rift is the used tracking technique.

The HTC Vive HMD uses a technique for positional tracking called ’Lighthouse’, developed by Valve. This technique works similar to how it is used in maritime navigation, where ships navigate near the shore by counting the time between ﬂashes from nearby lighthouses. Similar to that, Lighthouse requires the use of so called base stations (acting like lighthouses), which are accompanied with HTC Vive. The base stations, attached to each corner of the wall of an open room, tracks peripherals by performing an omnidirectional ﬂash. Each base station then transmits alternating sweeps of horizontal and vertical IR-lasers of the room to trigger the photo-sensors on the peripherals. By comparing the timing of when the sensors are activated, the exact position and orientation can be calculated with high accuracy and low latency [17, 18].

Communication with the workstation is done through a Linkbox, which powers the HMD and connects through USB and HDMI. The Linkbox also communicates with the basestations through bluetooth to activate or deactivate the devices since they only requires power to operate.

Vive Tracker is an upcoming device which can be used to track anything of choice. It may be attached to a speciﬁc item or to the body and provides an easy way for integrating full body experience into the virtual environment. However, for tracking the entire body, it requires multiple Vive Trackers to be able to give an accurate estimation of the body in the virtual environment. It uses the same ’Lighthouse’ technique for positional tracking as the HTC Vive and the controllers [19].

The HTC Vive controllers was released together with the HTC Vive HMD in 2016 and includes 24 sensors, a multi-function trackpad, a dual-stage trigger and haptic feedback. It is used for interaction with the virtual environment using the hands but gives a visual representation of the controllers instead of the hands [20]. Figure 2.5.2 displays the Lighthouse basestations together with the HMD and the controllers.

A new implementation of a wireless HTC Vive HMD has been developed in cooperation with Intel, which will be released in 2018. This relieves the need of a workstation and opens up further possibilities for developing an eﬃcient training system with this device.

Figure 2.5.2: HTC Vive with controllers and Lighthouse basestations Source: [21]

9 of 52 State-of-the-Art Survey Virtual Reality

Samsung Gear VR was developed in cooperation with Oculus and Samsung. It is a mobile, tetherless HMD that requires a connected Samsung smartphone to run. Only novel Samsung smartphones (Galaxy S6 or later) are supported. The HMD provides the user with a horizontal FOV of 101° [22].

By utilising the built-in gyroscope/accelerometer and proximity sensor in the connected smartphone, Gear VR enables rotational tracking of the user’s head (three DOF).

The cooperation between Samsung and Oculus has also resulted in a VR controller, intended for the Gear VR. This controller uses the same tracking technique as the Oculus Rift through sensor fusion. Additionally, the Gear VR has a built-in touchpad on the side to interact with the virtual environment and communicates with the smartphone through bluetooth. Figure 2.5.3 shows the GearVR HMD.

Figure 2.5.3: Samsung Gear VR Source: [23]

Leap Motion is a pocket-sized, rectangular shaped and a USB-connective device that is marketed for being an accurate hand-tracking tool in a VR or desktop context (Figure 2.5.4). This device in conjunction with the associated API, can in real-time determine the position of a users hands and ﬁngers in a Cartesian space. The Leap Motion controller has three IR-LED’s and two IR-cameras, providing stereo vision, which enables optical tracking in six DOF [24]. The captured data is of a grayscale stereo image, which is streamed through the USB-connector to the computer where the Orion Software Development Kit (SDK) takes over and performs mathematical calculations with tracking algorithms to reconstruct a 3D representation of what the device sees [25].

Figure 2.5.4: Plugged Leap Motion controller Source: [26]

10 of 52 Virtual Reality State-of-the-Art Survey

Microsoft Kinect V1 enables tracking of the user’s entire body through an RGB-D camera (Figure 2.5.5a). The captured data is computed to a depth map through structured light which infers the body position through machine learning. The later released model, V2 (Figure 2.5.5b), uses a built-in Time- of-Flight camera, which emits light and detects the reﬂected light. The time between emission and detection is then measured to determine the position of predeﬁned objects [27].

(a) V1 (b) V2

Figure 2.5.5: Microsoft Kinect V1 & V2 Source: [28, 29]

2.5.2.3 Software and Development Frameworks

There are several software applications which are widely used when designing for VR. The prominent game engines today are Unity [30] and Unreal Engine 4 [31], which are cross-platform engines used for game development. Unity and Unreal Engine supports plugins and packages to be imported to make it easy to create a VR application for diﬀerent HMD’s.

Software Development Kits (SDKs) Short description of a few selected SDKs and used APIs.

• OpenVR [32] is developed and controlled by Valve to support communication with HTC Vive and other virtual reality headset devices to run the same VR application. Essentially OpenVR consists of two APIs and acts as a mediator between other software and other devices. The First API, OpenVR API, is to allow communication between application and a software which communicates with the drivers of an HMD. The second API, OpenVR Device API, allows communication between the HMD software, such as SteamVR, and the drivers of the HMD.

• Open Source Virtual Reality (OSVR) [33] is developed by Razer and Sensics and is similar to OpenVR to enable diﬀerent devices to work with the same game but with the main diﬀerence being that it is open source.

• Oculus SDK [34] is used to enable installation and communication of drivers for using Oculus Rift. Oculus have one API which communicates with Oculus Runtime, which in turn directly communicates with the HMD and the tracking system.

• SteamVR [35] is a closed-course software by Valve and is used to enable communication with drivers of diﬀerent HMDs, but primarily for HTC Vive, by using the OpenVR API. Communication between the HMD device, HTC Vive, and SteamVR does not require the use of the OpenVR Device API however, since the drivers for the HMD and tracking devices already are implemented into SteamVR.

11 of 52 State-of-the-Art Survey Virtual Reality

The mentioned SDKs allows ﬂexible integrability with diﬀerent software such as Unity and Unreal Engine.

2.5.3 Related Studies and Implementations

The following section will describe several studies in the ﬁeld of VR in the recent years and what conclusions they came to with their implementations.

Before presenting the different studies a clarification of mentioned platforms may be useful. Often when talking about virtual systems there are two types that are mentioned: Commercial off-the-shelf (COTS) and Cave Automatic Virtual Environment (CAVE). COTS platforms often describe a system that is able to be purchased commercially for a low cost, and often used by only one user. CAVE platforms is a projection-based system where a virtual environment is projected on surrounding walls by projectors, where several users may experience the same virtual environment together. These acronyms will be used frequently throughout the text but focus will mostly be on COTS platforms.

2.5.3.1 Maintenance and Inspection Operations

A practical training environment called Virtual Teaching and Training System (VTTS) was developed by Quan et al. [36], to teach aircraft maintenance. VTTS is a computer based on VR technology and real-time education, similar to a CAVE. This particular VTTS uses a projective display and an LCD display with workstations and an aircraft simulator. The VTTS configuration has the possibility to realise numerous teaching contents and subjects. Similar to Virtual Maintenance Trainer (VMT), it possess the ability to implement a system with various routine operations, emergency situations and examinations. It is also possible to render a virtual scene similar to the real world operations and work conditions. By using this system the authors have come to the conclusion that this kind of application for maintenance training, reduces man-made errors, aircraft damage and accidents. VTTS enables to use virtual objects instead of real aviation equipment which lowers the cost, while essentially maintaining the same study efficiency. This type of learning environment will also increase the study efficiency compared to traditional methods e.g. text and multimedia.

For maintenance, Chang et al. [37] has suggested a solution for training in a virtual environment. In their implementation of a practical procedure training system for substation maintenance, they present a concept closely similar to VMT. Their system simulates models of equipment, failures that may occur and procedures to be completed by the student. A training and examination management tool is built into the system to help the teacher to keep track of each student. A mouse and keyboard are used as input devices for the platform. The platform communicates with a server, running on a Linux operating system, where all the computing is done. The scenario data (models, faults or tests) used by the server is stored in a database. The clients (teachers and students) are using workstations with the Windows operating system. The system provides many tools for the teacher to monitor training procedures and manage training archives and teaching plans. Compared to the teachers, students have an interface with limited access, but with the modules for training or examination of a procedure. A virtual toolbox (pliers, screwdrivers, duct-tape etc.) is also provided during the simulation which enables the student to use the

12 of 52 Virtual Reality State-of-the-Art Survey appropriate tool during a certain procedure. In this current implementation, the information is relayed on a monitor but may very well be to an HMD instead.

A recent study of an immersive VR maintenance training conducted by McNamara et al. [38] used Oculus Rift (Development Kit 2) and Kinect V2, and the game engine Unity. The study focuses on tracking a human body by using the Kinect V2 and examining the positional accuracy and system delay during use. Positional accuracy offers realistic representations of corresponding user actions. System delay is the time between a physical user action and the time for the action to be displayed on the HMD. It is an important factor to avoid simulator sickness to prevent the user from becoming dizzy [39]. Understanding the problems with system delay may aid in the understanding of the limitations of a VR system. Ten participants interacted with a virtual model depicting an Arresting Gear engine (an engine to decelerate an aircraft when it lands). Interactions included walking around the model and to try different body positions to test the capabilities of the Kinect V2. Results showed that positional accuracy of the Kinect V2 performed well, for a realistic movement in the virtual environment, on tracking a single user with the sensor in front. However specific body positions such as crouching, turning away from the sensor or blocking the view of the sensor would cause the Kinect to lose its tracking capabilities. Furthermore, it was unable to detect small gestures, which made it improper for the use of training where high accuracy was required. On the other hand, a considerable amount of the system delay was caused, between a performed action by the user and the rendering of the action on the HMD. Kinect V2 has a data transition rate at 30 Hz compared to the HMD at 75 Hz. As a consequence, user-performed actions, with this type of system, may take longer to update and cause disorientation which may cause simulator-sickness [6]. Moreover, a contributing factor is the stereoscopic rendering on the Oculus Dk2, which requires the computer to render a scene twice per frame, causing an excessive demand on the CPU and GPU. To alleviate this, the authors suggests using a computer with better specifications, with emphasis on better performing CPU, GPU and RAM. The time spent to prepare the image for the HMD would then be reduced significantly. Additionally, the system performed better on a computer running the operating system Windows 8.1 and a dedicated graphics card, due to the different compatibilities with the drivers between the hardware at the time. This type of training system may be suitable for when the user will need to become familiar with an equipment in a virtual environment by walking around it, or by interacting with it by using large distinct motions.

Continuing the topic of maintenance Borsci et al. [40] made a study of a train car service procedure with an implemented virtual-based training tool called LAugmented Reality (AR)TE-VBT. It is based on the HoloVis game engine (InMo) and enables users to visualise and interact with CAD models in diﬀerent devices. The devices in this study compared between a CAVE platform, Oculus Rift and a tabletop display. In the CAVE system, there were four walls with each wall having two projectors which projected the virtual environment. Nine workstations were used to visualise the virtual world. The user had to use polarised glasses with built-in trackers for interaction. The tabletop display was a device named zSpace, a 24 inch LCD running at 120Hz. A stylus device was used as a controller and similar to the CAVE, polarised glasses were used as a tracking device. In the case of Oculus Rift, it was used together with desktop controllers and joysticks. 60 participants were divided into three groups and were given the task to do random maintenance operations on parts of a train car. One group would use the CAVE

13 of 52 State-of-the-Art Survey Virtual Reality system, second group the tabletop display and the third group the Oculus Rift. The conclusion of the study presents all three of the devices as good learning experiences with the largest diﬀerence coming down to price. While the CAVE is a largely expensive alternative (>50k €), the tabletop display has a more reasonable price (< 6k €) and the Oculus Rift with a workstation at the cheapest price point (< 2k €). The authors also mention the importance of acceptance of the end-users when using VR systems for learning since the success of a system like this depends on how good the user believes the virtual training is comparable to the real world.

Cabral et al. [41] presents a simulator system for maintenance of power grids distribution lines which follows most of the guidelines in section 2.3 well. Oculus Rift is used to visualising the environment. Interaction with the environment is enabled by tracking real tools, which receives a virtual depiction, to give the user a realistic feeling during learning. The tracking system is based on an extremely accurate infrared camera system called OptiTrack. The body of the student is also tracked and is depicted as an avatar in the virtual environment allowing the teacher to assess if the maintenance is done in a correct way. Both the teacher and the student has their own program interface. The teacher can through their perspective control of the training allow different situations to occur to allow different challenges on the maintenance lessons. Challenges may include doing the maintenance at different weather situations, structures being on fire, experiencing electrical arcs and short-circuits and other obstacles such as trees, cars etc. From that, the teacher is also able to see how the student reacts to and solves the problem. The student performing the maintenance training uses the Oculus Rift to see the virtual world and do the operations required. Other students may, through an external screen, see what the training student is doing, to learn from each others mistakes.

DAssault Systèmes is a worldwide aerospace company, producing both passenger and ﬁghter jets. Recently they presented a virtual simulator developed for virtual maintenance training for their Falcon jet, by using their own 3DExperience platform. It is designed to be an inexpensive alternative for aircraft maintenance training to their already implemented CAVE system. It uses CATIA CAD data to provide an immersive training experience for engineers and technicians. The system uses Oculus Rift HMD together with a front-attached Leap Motion, to provide hand tracking capabilities, enabling real-time interaction for the users. Constellation tracking is also used for the Oculus Rift to provide six DOF virtual experience. Multiple instructors or trainees are allowed in one session, each represented by an avatar in the virtual world. The Leap Motion tracker also allows the user to see their own hand and the hands of others, useful when the teacher needs to point at a certain object, or when a trainee points at an object while asking questions about it. Interaction in the world is enabled by a menu, spawned by facing your left-hand palm towards the Leap Motion controller. From the menu, mentioning only a few things, the user is able to (i) set transparency of the virtual model, (ii) select diﬀerent data to show, (iii) recolour models to make some parts more distinct from others, (iv) bring the trainees to the instructor position, (v) spawn a pointer steered by the HMD motion to point or select an object further away from the hand. Movement in the virtual world is done by the arrow-buttons on a traditional keyboard [42].

2.5.3.2 Educational Beneﬁts

A proposal for a virtual education environment was put forward by Crespo et al. [43] by testing the beneﬁts of a VR simulator. The simulator allows engineering students to perform an interactive

14 of 52 Virtual Reality State-of-the-Art Survey operation on an industrial robot. Used hardware was Oculus Rift HMD and Razer Hydra Joysticks. The application was designed using the game engine Unity. Oculus SDK was used for configuring the Oculus Rift while Sixense SDK was used to allow programming for six DOF motion tracking of position and orientation with the Razer Hydra controllers. Students that tested the system executed a number of interacting tasks of the virtual robot with the controllers, by moving the robot around in the virtual environment or making it perform certain tasks, such as moving items. It was concluded that this system has promising possibilities of becoming a reliable engineering learning tool. The used software and hardware enable the system to be highly flexible by working with different platforms, operating systems and controllers. Furthermore, it allows for mistakes to be made and makes it possible for the student to experience what will happen if something goes wrong. Through the immersive interaction, it enabled a swift learning of specific routines’ and reduced the need to use real hardware.

Alhalabi [44] also made an educational study with 48 students, by comparing different VR systems. The VR systems were provided from Worldviz, a worldwide provider of VR systems. It included The Corner Cave System (CCS, a projector based VR system) and an Oculus Rift DK2. The CCS system requires the users to wear special shutter glasses equipped with precision position tracking ‘eyes’ to see and interact with the virtual world through the projected screens. The students were divided into groups with (1) no VR, (2) HMD with tracking, (3) HMD without tracking and (4) CCS with tracking. The tracking system was set up with eight cameras for user tracking in a spacious room. The students were asked to study different engineering materials and learn about new concepts. The results showed that the learning efficiency increased significantly by using VR systems. The CCS system would be an appropriate alternative for multiple users at a time, however, the HMD system is more inexpensive, with the additional advantage of providing a more immersive environment for the user.

Additional virtual training implementations has included Oculus Rift together with Leap Motion (and Kinect V2) to provide an effective way of training for situations in construction and fire emergencies. [45] used Unreal Engine 4 in their implementation of an inexpensive virtual environment for an engineering and construction setting. [46] used OpenGL to implement a web-based training system for fire wardens to offer a safer alternative for training the cognitive decisions during fire in a building. Both of the projects used Industry Foundation Classes (IFC) to define CAD-graphic data, related to architecture and construction, as 3D objects.

2.5.3.3 Input Practices

In terms of user control, Khundam [47] implemented a system using the Oculus Rift DK2 together with Leap Motion to provide a way of controlling the movement inside a virtual environment with hand palm gestures. Using Unity, an application was created to provide a virtual space for the user to move in freely. 11 participants tested the system with two different scenes for each of the input methods. In the experiment, the subjects’ mission was to move around the virtual world in a fixed order. The evaluation showed that using the Leap Motion controller the users were able to complete the missions faster with the palm control gesture, enabling finer control of the movement speed, than the gamepad. However, due to the smaller FOV of Leap Motion, the users had to face the HMD towards their hands when performing the specific gestures.

Consequently, Alshaal et al. [48] presents their own platform, SmartVR, as an alternative to the

15 of 52 State-of-the-Art Survey Virtual Reality limitations of current available input methods and to expensive wearable devices. In particular, the FOV limitations of Leap Motion was mentioned to be less eﬀective. The SmartVR platform was used together with the Microsoft Band 2 wristband, a VR workstation and Oculus Rift DK2. A Bluetooth transceiver was used for communication between the workstation and the wristband. Unity was used to develop the application of a virtual store shop, where the user would be able to pick two modes, switching between them by performing a distinct gesture. First mode is a selection mode, which lets users select virtual items, and a navigation mode, which enables movement in the virtual environment. Second mode is a navigation mode, where the user may perform a swiping gesture to move around, moving towards the direction of the swiped action. In the selection mode, the user is able to select the virtual item with a spawned cursor representing the hand, while the camera is static. A menu was provided to manipulate the object, such as changing colour or size. Using a wearable device in this way, such as the Microsoft Band 2, enables the hands to be tracked regardless of the orientation of the HMD and allows the user to be less limited when performing the gestures.

16 of 52 Augmented Reality State-of-the-Art Survey

2.6 Augmented Reality

2.6.1 Introduction

Augmented Reality (AR) is a technique which augments what the user can see, feel and hear. Usually, an AR application involves visualisation of virtual objects superimposed on the real world, to mainly supplement the real world with useful information [49, 50].

The occurrence of virtual objects, displayed in an AR application, together with their pose and position, can be based on camera-recorded ﬁducial markers which operate as positional reference points [51, 50]. A marker may consist of a picture or pattern, which is recognisable by the particular AR application, such as QR-codes (Quick Response Codes), BIDI codes (BIDimensional) or bar codes [52].

A virtual object’s appearance can also be based on marker-less tracking. For instance, external stationary cameras can be used for recognition and tracking of real-world objects. Based on detected objects in the video recording, the occurrence, position and pose of a virtual object can be determined [51]. The marker-less positioning of virtual objects can also be applied through the usage of accelerometers, GPS-technique, compasses etc [50].

The possibility for user interactions, in an AR application, can be implemented in similar ways as with Virtual Reality (VR). Such as gesture tracking, voice commands, hand controllers, touch devices etc. [53].

2.6.2 Hardware and Software

In this section, modern AR-related hardware and diﬀerent development frameworks will be presented.

2.6.2.1 Device Deﬁnitions

Head-Mounted Display (HMD) intended for AR, can display the real world, augmented with additional virtual objects for the user. There are both portable and tethered models for data transfer and/or power. An HMD can be classiﬁed as either an optical-see-through or video-see-through device [53].

2.6.2.2 Hardware

Microsoft HoloLens is an optical-see-through device with a visor, on which virtual objects may be projected in a maximum resolution of 1268 × 720 pixels [54]. By utilising four built-in spatial-mapping cameras and a depth camera, a 3D-model of the surrounding environment is created. The geometry of this 3D-model is used to determine the position and pose of virtual objects (also called holograms in HoloLens context) and for positional head tracking. Rotational head tracking can be achieved by using the built-in gyroscope, magnetometer and accelerometer. Both of these tracking techniques enables six Degrees of Freedom (DOF) movement. The horizontal Field of View (FOV), in which the 3D-objects may be projected, is 30° wide. Figure 2.6.1 depicts the HoloLens HMD.

17 of 52 State-of-the-Art Survey Augmented Reality

Figure 2.6.1: Microsoft HoloLens Source: [55]

User interactions can be performed by looking in a particular direction while performing a certain ﬁnger gesture (recorded by the built-in camera) or through voice commands in Cortana, Microsoft Windows’ voice-activated assistant [56, 57]. Microsoft’s HoloLens is completely tetherless and probably the most known of currently few available AR HMDs. HoloLens can be bought in a development kit, ﬁt for an individual developer. For enterprises a similar kit can be purchased, but with a warranty and features like higher security etc. [58]. HoloToolkit, which is a set of APIs towards the HoloLens utilities, enables development of applications. These APIs can be integrated into Unity.

Meta 2 is similar to HoloLens; an optical-see-through device with common properties and a maximum resolution of 2560 × 1440 pixels distributed over a 90° wide FOV [59]. One diﬀerence is that Meta 2 is not tetherless when running and requires a connection to a workstation with Windows 8 or 10. The real world environment features are captured using a built-in computer-vision-camera. This camera enables virtual object positioning through a modiﬁed simultaneous localisation and mapping (SLAM) algorithm. The movement of the wearer is tracked by using a built-in inertial measurement unit. Totally this provides six DOF [60]. Figure 2.6.2 shows the Meta 2 HMD with the connection cable.

Figure 2.6.2: Meta Company’s Meta 2 Source: [61]

18 of 52 Augmented Reality State-of-the-Art Survey

Google Glass is an optical-see-through HMD that appears like a regular pair of glasses which can provide an image of virtual objects, displayed on the built-in 640 × 360 pixel prism-projector screen just above the user’s natural line of sight (Figure 2.6.3) [62]. To the user, the built-in screen appears comparable to a 25-inch high deﬁnition screen viewed from a distance of 2.4 meters [63]. Similar to Google Cardboard, Google Glass captures video in the direction the user is facing with a built-in camera. The recorded video can then be used for object recognition, video conversations etc. User interactions can be performed by voice commands or through a built-in touchpad located on the side of the spectacle frame [53]. Using the Android API also enables rotational head tracking by utilising a built-in gyroscope, accelerometer and magnetometer [64].

Figure 2.6.3: Google Glass Source: [65]

Google Cardboard is a primitive video-see-through HMD (Figure 2.6.4) that requires an attached smartphone to be used for AR. Video of the real world can be captured by the camera of the smartphone and displayed on the screen in real-time, augmented with additional virtual objects. The experienced horizontal FOV can alternate between 90° to 120°, depending on the attached smartphone [53].

Figure 2.6.4: Google Cardboard Source: [66]

19 of 52 State-of-the-Art Survey Augmented Reality

Tablets and smartphones also oﬀer the possibility to be used for AR applications. The user observes the screen which displays a video of the environment; captured by the built-in camera [67]. A tablet or smartphone device itself can be utilised for user interactions, which may be performed using the touchscreen or the microphone by voice commands, depending on the used AR software [53].

The Leap Motion controller can, similar to VR, be used for achieving natural user interactions in AR applications [24, 9].

2.6.2.3 Software and Development Frameworks

Vuforia is a proprietary AR Software Development Kit (SDK) that uses image recognition through computer vision technology for creating virtual objects. A Vuforia mobile app utilises the built-in camera to capture the environment. It can recognise a 2D-image (normally a photo, drawing, label etc.) or 3D-objects like cylinders and cuboids by reaching a cloud database, where predeﬁned 2D- or 3D- data for recognition is stored. The program uses the data as a reference point to determine the position and pose of the virtual object that will be imposed on the real environment. Vuforia supports native development for iOS and Android but also enables integration with the Unity editor for targeting both platforms [50].

ARToolKit is a multi-platform, open-source (LGPLv3) library for development of marker-based AR applications. Similar to Vuforia, ARToolKit uses image recognition for determining occurrence, position and pose of virtual objects, but can only recognise 2D images, not 3D-objects. The library allows for integration into the Unity editor, providing a convenient way of developing AR applications for any platform [50].

D’Fusion is a proprietary, multi-platform AR engine established by Total Immersion. The engine supports both marker-less and marker-based tracking. One of the strengths is that the engine has well- developed human face recognition and tracking. To develop D’Fusion-based applications, the GUI- based tool D’Fusion Studio is used [68, 50].

2.6.3 Related Studies and Implementations

2.6.3.1 Usages in Assembly, Maintenance and Inspection Work

Aircraft maintenance training activities are closely related to maintenance and inspection work tasks in e g. the automotive or aviation industry. Ramakrishna et al. [53] propose a framework for inspection work using AR-technique. The framework consists of an HMD or tablet that streams video images to a back-end server, using UDP-connection. The server runs the images through a Convolutional Neural Network (CNN) to detect certain objects for inspection. When an object is recognised by the server, modelling data is sent to the HMD/tablet providing a guiding graphical overlay for the user.

An experimental study including this framework was performed, by the same authors, where three diﬀerent AR platforms were used: Google Glass, Google Cardboard and an Android tablet. The experiment was carried through on 20 subjects (engineers and research staﬀ from an industry research

20 of 52 Augmented Reality State-of-the-Art Survey lab). Their mission was to perform an inspection task on a 3D-printer. To create an inspection guiding graphic overlay, the user initially scanned a QR code, placed on the printer, which provided user manual data for the specific printer model. The user then had to inspect different parts of the printer and answer a set of questions through platform-specific interactions. The built-in touch-pad was used for Google Glass, voice commands for Google Cardboard and the touch screen for the Android tablet [69, 53].

Observations during this experiment show that stress was reduced when using AR with graphic overlays containing the needed information compared to looking through extensive paper- or PDF- manuals for completing complex tasks. The Android tablet performed best of the three platforms since the users found it the most intuitive. Google Cardboard, on the other hand, caused eye-sickness, gave the user a claustrophobic feeling and the low graphics resolution aggravated the user’s ability to read printed text on the printer parts. A low FOV gives the user a cramped vision which can reduce the awareness and be dangerous in rough industrial environments. The voice command interactions also work poorly in crowded/noisy areas. The Google Glass was hard to ﬁt over regular safety glasses and the swipe patterns were diﬃcult to remember. It is a more expensive platform than the others in this experiment but could be convenient for users in situations where both hands are needed for work tasks. In summary, the tablet was the best candidate in this experiment due to simplicity and measured inspection turnaround time [53].

Industrial assembly work, like previously mentioned inspection work, can be related to maintenance training. Loch et al. [70] analyse a training tool that provides step-by-step guidance for assembly tasks in an AR-setting. It consists of a computer screen, placed in front of the user, which displays a real-time camera recording of the workspace for the assembly task with an additional virtual graphic overlay for assembly guidance. The camera is mounted above the workspace, pointing downwards. A study was conducted to evaluate and compare AR-based to video-based guidance for assembly tasks. 17 undergraduate students from social science and technical disciplines, without any previous experience in AR-guidance, participated in the study. Before the experiment, preparation training was performed to become more familiar with the AR-guidance system. The experiment involved a practical part, where the participants were to assemble two diﬀerent models using Lego blocks.

Measurements in the study showed that the average time to complete an assembly task was longer for the video-based instructions compared to the AR-guidance and the average number of errors per assembly task was over ten times(!) higher. Evidently, using AR-guidance increased the accuracy and task performance due to the lower number of occurred errors, compared to using the video-based instructions. The author draws the conclusion that, for assembly tasks, experts in a certain area are the most eﬀective guides, but that AR-based solutions work better in several important aspects than video- based instructions or printed manuals [70].

Boeing Research and Technology (BR&T) performed several comprehensive studies that explore the usage of AR solutions in satellite and aircraft part manufacturing. For developing virtual work scenarios, BR&T used D’Fusion Studio. The choice of visualisation AR-platform fell upon high-end ruggedized PC tablets, snapped into hands-free holders to enable work with both hands simultaneously. A set of eight external Vicon RGB cameras mounted to the inner-roof of a dedicated room, were used for accurate marker-less tracking [50]. For passing the captured camera input to the D’Fusion application, a custom-made Lua script was developed. Microsoft Kinect was used for low-resolution 3D modelling of objects for tracking and recognition, but also allows existing CAD ﬁles to be utilised.

21 of 52 State-of-the-Art Survey Augmented Reality

The system was tested by simulating an assemblage of an aircraft wing. According to the BR&T studies, utilising AR techniques in ideal engineered environments can- provide more eﬀective manufacturing processes in terms of task completion time and failure rate. Although, more development will be needed for the technology to be extensively applicable in mainline manufacturing industries. BR&T proposed an AR solution that an average manufacturing work site today can suﬀer from camera line of sight problems when working in narrow spaces or behind occluding vehicle parts. Due to this problem, external cameras or sensors should not be preferred as the tracking solution. For display, tablet PCs works but does not give the user a deep level of immersion. Other more immersive display technologies, that can provide a wide FOV and high safety, would be preferred in manufacturing processes [51].

Caterpillar Inc. is the world’s leading mining equipment manufacturer at the time of this report. During 2015 Caterpillar, in cooperation with ScopeAR, developed an AR application for maintenance and inspection work assistance, using marker-less tracking; that can run on a smartphone, tablet or AR glasses [71]. The application provides a graphic overlay which guides the user through a certain maintenance or inspection task. The overlay consists of arrows, ﬂoating text and 3D models of machine parts involved in the task. Work instructions are given step-by-step and the application can evaluate if the user has completed a certain step in a correct way [72, 73, 74].

2.6.3.2 Educational Beneﬁts

Ibáñez et al. [49] conducted an empirical study, including 40 ninth grade students, for the purpose of exploring student behaviours, learning effectiveness and motivation while using a marker-based AR- simulation tool called AR-SaBEr. The tool can be used for learning basic electrical principles and developed by using Vuforia and Unity [50]. The experiment used Android tablets and was divided into three topics: electric current, voltage and resistance. Each topic had one reading activity and two to three experimental activities where instructions were given step-by-step in a highly guided way called scaffolding. The students were given the task to build basic virtual electrical circuits from components like batteries, fans and switches which were visualised on marked physical blocks. By moving the blocks into the right place, while observing the augmented scene on the tablet screen, a circuit could be constructed. AR-SaBEr and similar tools can be useful as interactive learning environments and are especially important for achieving desirable results in given tasks. To help students keep the focus on their goal, it is necessary to provide free experimentation but also additional scaffolding for completing tasks.

Using the AR-SaBEr, Ibáñez et al. [67] also made a case study, investigating the attitude of learners towards education with AR as a technical device. The subjects in this study were physics undergraduate students with no previous experience in usage of AR. Their task was to understand and solve an electromagnetic problem which included a visual 3D cube with a force to be determined and a simulation of the trajectory of a particle. Two single-choice questions with four alternatives were then answered by the subjects. Before the experiments, the subjects were given some AR preparation training with a smartphone application which lets the user determine the position of a point in a 3D-space. The result from the study concluded that AR helped students to visualise the electromagnetic problem. The subjects also enjoyed it, which positively aﬀects the attitude towards AR-learning that in turn can motivate students to perform learning activities.

22 of 52 Augmented Reality State-of-the-Art Survey

2.6.3.3 Object Tracking Investigations

User interactions is a vital part of an educational AR application. Natural user interactions can bring immersion to the application and depth sensors can be used for achieving this. For detecting body or hand gestures with depth sensors, Microsoft Kinect and Leap Motion Controller are today’s most used devices.

Kim et al. [9] analyses a set of approaches for user interactions in a partially marker-based AR application. The implementation was done with the Vuforia SDK and Unity [50]. 20 persons, whereof 11 who had previous experience of augmented reality, participated in the experiment. A Samsung Galaxy Tab S 10.5 was used for running the application and a Leap Motion Controller to capture user interactions. Two different approaches were analysed without using the Leap Motion Controller. The first, vision-based approach records the user hands with the built-in camera, where the program interpreted the different hand gestures. The second approach uses the screen of the Android device for touch-based interactions. These cases were compared to an immersive approach where the Leap Motion Controller is used for tracking the hands of the user and virtually recreated as 3D hands in the application for natural user interactions. Users performed three experimental tasks by using the mentioned interaction approaches: The first task consisted of transforming a virtual 3D bunny into a goal position and pose. The second task involved picking up and hanging a virtual 3D torus on a pin. The last task consisted of a simple assemblage of virtual engine parts; the user was to put a piston into a conrod, insert a lock pin into a hole and then to screw the pin into the hole. The mean value of measured completion time for the first task was 45% longer for the touch and vision-based methods compared to using the immersive method. No measurable differences were observed while measuring failure rate.The second task did not produce any measurable differences in performance between the interaction methods, but a wide-angle lens attached to the camera helped to grant vision of the fiducial markers while performing the task. In the final task, the average measured completion time for the touch-based, vision-based and immersive methods were 225, 190 and 120 seconds and the mean failure rate were 55%, 40%, and 5% respectively. The results show that the immersive implementation outperforms touch and vision-based methods. By using this or a similar method, the user can easily, in a natural and intuitive way, manipulate virtual 3D objects in six DOF.

Garon et al. [75] investigates the HoloLens and concludes that there are limitations in the device that can not be neglected in situations where high precision is important. It was found that shortage of high- quality depth data is one limitation that can result in poor tracking and thus, bad positioning of virtual objects. When developing for the HoloLens, a programmer is only permitted to use Microsoft’s own API and cannot reach the raw data from the built-in sensors. This prevents implementation of other, better-purposed tracking functions and algorithms. To bypass this restriction, Garon et. al. mounted an external Intel RealSense RGBD (Red-Green-Blue-Depth) camera on a HoloLens. The camera captured video that was processed on a connected stick PC and transferred via Wi-Fi, to the HoloLens. The mentioned processing on the stick PC consists of a program with a sophisticated object detection implementation that detects and estimates the pose and position of a scanned object. When testing this custom-built system, a great improvement in detection of objects measuring close to 20 × 20 × 20 cm could be observed.

23 of 52 State-of-the-Art Survey Result

2.7 Result

This section will contain comparisons between diﬀerent system depending on the objectives. While Virtual Reality (VR) and Augmented Reality (AR) both can satisfy similar needs, there are both advantages and disadvantages with using techniques for both concepts.

From the given requirements, separate areas were extracted for evaluation. Since the requirements state that the teacher should be able to demonstrate a virtual model together with a group of students, using AR or VR technology to, one requirement would be to have a system which supports multiple users simultaneously. Both a teacher and a student should be able to interact with the virtual environment. Therefore making the manipulation of virtual models necessary, requiring support for user interactions. Being able to accurately observe a highly detailed virtual model, adequate graphics and performance are desirable for both VR and AR [6]. A high performing system for this matter should deliver low system delay, suﬃciently high frame rate and stability in the placement of virtual objects [76]. These topics will present advantages and disadvantages of various VR and AR systems for the particular area.

2.7.1 Multi-user Capabilities

Multi-user in VR

There have been many VR systems utilising the Cave Automatic Virtual Environment (CAVE) platform. The main advantage is inherited by its multi-user capabilities and by oﬀering an immersive and close-to-reality environment. Compared to VR Commercial oﬀ-the-shelf (COTS) systems, this platform can enable a wider Field of View (FOV) and allow multiple users to physically stand in the CAVE-area to feel present within the virtual environment. However, it only allows one user to be tracked at one time to control the simulation.

From the related studies and implementations, there is only one implementation which uses a COTS platform with added multi-user capabilities. DAssault Systèmes, used their 3DExperience platform, to create a multi-user virtual environment where an instructor guides trainees around a CAD-based 3D-model of a Falcon jet aircraft. Since this system is using Oculus Rift which is a tethered Head- Mounted Display (HMD), each user must have their own workstation. They use a network system, where the trainees join a session guided by an instructor. Since this is a novel implementation, there are no studies regarding how well the multi-user usability of this system works and if there are any possible drawbacks. However, there are videos [77, 78] which may give an observation of how the system works in general.

Though there are currently a limited amount of presented solutions for multi-user environments for VR, there are multiple Software Development Kit (SDK)’s and third-party applications being continuously developed. Using Unity 5 also easily enables the creation of multiplayer environments through using their networking API. Most SDK’s include some form of networking capabilities and are too many to mention all of them but most prominent ones are the built-in HLAPI of Unity and the Photon Engine [79] among others. The support is mainly for the more popular HMDs such as Oculus Rift and HTC Vive. The implementations are similar to how standard multiplayer networking works, for instance peer-to-peer, client-server, client-side prediction etc. [80].

24 of 52 Result State-of-the-Art Survey

Multi-user in AR

Microsoft HoloLens offers the ability to share virtual objects amongst multiple users through networking solutions. HoloToolkit includes a specific network library intended for implementing multi- user support into applications. Running such an application, a common object can appear with the same position and pose for all connected participants. Users can appear as avatars (virtual representations of the users) to each other. In a multi-user session, participants can be in the same or different physical rooms. To enable synchronisation of virtual objects and avatars between participants, a virtual anchor (a spatial reference point) is placed and shared between the users. This ability of object-sharing in real-time can be used while multiple users collaborate e. g. designing, manipulating or just observing 3D-objects [81]. Video conversation programs like Skype etc. may be used for communication, collaborating and guiding between HoloLens, PCs, smartphones and tablets [82]. In a similar way to the HoloLens, Meta 2 HMD can also be used for collaboration [83].

Mobile devices can be utilised for AR as well with multiple users in the same virtual environment. A marker-based application developed using Vuforia or ARToolKit, may run on smartphones and/or tablets. Multiple users can observe the same virtual object by turning the tablet/smartphone camera towards the same marker. Pose and position transformations of the virtual object are performed by physically moving or rotating the marker. Other manipulations and interactions with an object are possible by using one of the standard multiplayer network solutions; allowing connected users to see and interact with shared virtual objects in real-time [84].

The Google Glass does not allow the same type of flexibility in positioning of virtual 3D-objects due to the fixed built-in display. Instead, a multi-user situation could consist of e.g. a video conversation between a desktop user and a Glass user. In this case, the desktop user would receive a point-of- view video feed, captured by the built-in camera of Google Glass. The Glass user is then able to simultaneously perform a certain work task while receiving guidance from the desktop user through audio and video. Google Glass can also be utilised for streaming a first-person point-of-view video while performing a work task for instructional usage.

2.7.2 Graphics and System Performance

This section will cover details about devices used in VR and AR and their graphical and systematical capabilities.

Performance of VR Devices

A CAVE system can consist of one or multiple walls, with each wall having two projectors displaying the projections. The user needs to wear 3D glasses, for displaying stereoscopic images, to experience VR. Stereographic rendering renders the same scene twice, meaning a great deal of computer power is required. Tracking is enabled by using diﬀerent systems, such as camera-based tracking, magnetic tracking and optical tracking - each one of these may work well depending on the situation but comes with their own problems. The graphics engine possibilities for a CAVE system are limited and usually have expensive license costs. There are other alternatives besides fully professional CAVE’s, with lower

25 of 52 State-of-the-Art Survey Result prices [85] though with lower rendering quality. A system like this requires a high-grade graphics card and high-quality projector which requires regular maintenance and calibrations [86, 87].

In the area of COTS platforms, the most relevant HMDs are Oculus Rift and HTC Vive. Both of these are similar to each other but still have their internal diﬀerences. Both have a screen resolution at 2160 × 1200, refresh rate at 90 Hz, a FOV at 110 °, oﬀers six rotational and spatial positioning, making it adequate to give a full VR experience and reduce simulator sickness [6, 39]. One advantage HTC Vive has to Oculus Rift is a larger tracking area, 4.6 × 4.6 meters with Lighthouse compared to 1.5 × 3.3 meters with Constellation [10].

HTC Vive uses Lighthouse and Oculus Rift uses Constellation, both of these are known as positional tracking systems and formally called drift correction systems. The positional and rotational tracking is managed by a gyroscope, an accelerometer and an on-board Inertial Measurement Unit (IMU). A critical issue with using IMU as positional tracking is that it is prone to generate drift errors due to gravity and other noises [75]. Since the sampling rate of the IMU is up to 1000 Hz, the errors may add up quickly and cause the tracked object to move along undesired directions. However, the issue may be alleviated by using reference points. Lighthouse generates reference points by registering the order of activated photodiodes by the base station lasers, providing 60 updates per second from each axis of the tracked object. Constellation generates its reference points by pulsing 10-bit patterns in the LEDs of the HMD, which is detected by the global shutter of the cameras, also providing 60 updates per second. The data of the IMU and the photodiodes is then computed to provide an accurate calculation of where the tracked object is in the 3D space and helps with reducing latency [17, 88].

Currently, the biggest problem with outside-in tracking, such as Constellation and Lighthouse, is occlusion. If something is blocking the view of the sensor from the LEDs of the tracked object, it will not be detected, thus reducing the tracking capabilities of the object. This problem may be alleviated by using more sensors, but at the same time, the price becomes higher which makes it undesirable as a long-term solution. Oculus has tried to mitigate this problem by patching the tracking software, which has helped the majority of consumers, but the underlying problem with occlusion still remains with these types of tracking devices.

When regarding the potential of GearVR and its capabilities of wireless virtual experience, it looks promising to use for different applications in the future. The HMD offers a FOV at 101 ° and tracked rotational orientation. Rest of the specifications depend on which smartphone is used e.g. using it together with a Samsung Galaxy S7 provides a screen resolution at 2560 × 1440 and refresh rate at 60 Hz. No studies were found to have used this device in maintenance or inspection work, probably because of lack of positional tracking. However, if the rotational tracking is good enough, to only observe a virtual environment from a static position, there are options to develop custom applications for the GearVR by using the Oculus SDK. Common problems that usually affects this system is mostly caused by the used smartphone. These problems relate to a limited amount of battery time and overheating issues, which may be ameliorated by better performing smartphones in the future.

Regarding optimal speciﬁcations of VR HMDs, having a wide FOV, display refresh rate and low system delay gives a better experience and immersion for the user. Failing to achieve an optimal value of these requirements may give a bad experience for the user and cause motion sickness. The recommended refresh rate is at 90 Hz for HMDs to reduce the motion-to-photon latency to tolerable levels. Motion-to-photon latency deﬁnes the time it takes from the movement of a users’ head, until the simulation is updated to represent the new position on the HMD. The motion-to-photon latency

26 of 52 Result State-of-the-Art Survey is affected by the input latency, GPU processing time and the display latency. Failing to meet these requirements, may cause the user to experience an unsynchronised 3D simulation where the actions and the physical movements of the user are not properly mirrored in the simulation environment. Having an adequate FOV is also important in reducing motion sickness inducing factors and is a great factor for immersion and giving the user an effect of presence. To get an immersive VR environment, it is recommended to have a horizontal FOV at 120° and a vertial FOV at 135°, depending on the position of the viewers eyes relative to the HMDs lenses. When using complex 3D models or graphics it is even more important to make sure the system is capable of displaying the simulation environment at an adequate framerate which is possible to achieve by applying different computing tasks to improve rendering quality such as using occlusion culling or remeshing the complex models, which in turn may require a powerful workstation to handle the required computations [6, 39, 89, 90]. See Table 2.1 for an overview of the technical details.

Table 2.1: Technical details of selected VR HMDs

Oculus Rift HTC Vive Samsung Gear VR

(Combined) Resolution 2160 × 1200 2160 × 1200 Smartphone-based Refresh Rate 90Hz 90Hz 60Hz FOV 110° 110° 101° Positional Tracking Yes Yes No Constellation Tracking System Lighthouse — (Optical)

Room-scale Boundaries 1.5m × 3.3m 4.6m × 4.6m — DOF 6 6 3 Built-in Camera No Yes Smartphone-based Wireless No No Yes Software Windows, Oculus Windows, SteamVR Android, Oculus Display Type OLED AMOLED AMOLED Price (inc. controllers) $800 $900 $130

Performance of AR Devices

When using an AR HMD, the FOV is a vital part to give visual awareness of the surrounding virtual objects. Microsoft advertises the HoloLens to be able to display all the virtual objects within the user’s eye-span. However, developers and hardware critics who have tested the system, are reporting that the FOV is relatively narrow, limiting the user to only an area of a rectangle for seeing virtual objects (see Figure 2.7.1) [91, 92, 93]. The geometric tracking in HoloLens is more accurate than most of the competitors, however, without any external aid it still has issues with superimposing virtual objects to ﬁt properly with the real environment. HoloLens is restricted only to Microsoft’s own API for utilising the on-board sensors which limits the ways for a developer to implement better tracking techniques

27 of 52 State-of-the-Art Survey Result

[75]. Compared to the HoloLens, the Meta 2 HMD connected to an up-to-date computer oﬀers a wider FOV to project virtual objects, but at a slightly lower screen resolution. However, with the same case as HoloLens, users have perceived the virtual object positioning of Meta 2 as unstable and jittery [60, 94].

Figure 2.7.1: Image illustrating the FoV of HoloLens for displaying virtual objects Source: [93]

There is also a possibility for using an immersive, primitive video-see-through device such as Google Cardboard. Compared to more sophisticated HMDs like HTC Vive and Oculus Rift, this device has a low screen resolution (how low, depends on the connected smartphone) and a narrow FOV. Studies that involve inspection work tasks, has shown that these problems can impair the user’s sight which aggravates the work task and can even be dangerous in certain environments [53].

User performance was shown to be badly aﬀected when the latency of a system was too high than if it had a lower refresh rate. To alleviate this, instead of increasing the refresh rate, it is advised to instead reduce system delay as much as possible through improvement of object target tracking [95]. See Table 2.2 for an overview of the technical details.

Table 2.2: Technical details of selected AR HMDs

Google HoloLens Meta 2 Google Glass Cardboard Smartphone- Resolution 1268 × 720 2560 × 1440 640 × 360 based FOV 30° 90° — 90° - 120° Positional Tracking Yes Yes — No DOF 6 6 — 3 Wireless Yes No Yes Yes Windows, Windows, Meta Windows, Linux, Android, iOS, Software HoloToolkit 2 SDK Glass SDK Google VR SDK Price $3000 - $5000 $900 $1500 $15

28 of 52 Result State-of-the-Art Survey

2.7.3 User Interactions

Interactions with virtual models is a key property of an application developed to train aircraft technicians and maintenance in other ﬁelds. When using a VR or AR application there are many alternatives of input devices to choose from and the option to combine freely is available through the use of diﬀerent frameworks. However, in the end, the importance of intuitive user interactions is essential for a realistic experience in the virtual environment.

2.7.3.1 Natural User Interactions

There are a lot of alternatives for choosing a system for user interaction. An intuitive choice when choosing a system, for interactivity in a virtual reality setting, is for it be realistic and natural; using the hands without holding any hardware devices. There exist a few devices today which enable this type of interaction. The prevalent devices in the market include Leap Motion and Microsoft Kinect.

In a recent study [96], user interaction was analysed to see whether the natural user interactions devices, Leap Motion and Microsoft Kinect, oﬀers a desirable and interactive user experience. The users preferred the Leap Motion for its precision and Kinect for larger movements. In a similar research [38] the Kinect was likewise deemed to be unﬁt for precise interactions but more appropriate for tracking of distinct body motions.

Leap Motion has been proven to possess a sub-millimetre accuracy and can be applied as a robust hand tracking device in both an AR and VR setting [24]. If the controller is mounted to an HMD, it will also follow the direction the user is facing and keep track of the hands while using the hands and turning the head, fulﬁlling the guideline for enactive interactions in section 2.3.

Microsoft Kinect has been used in several related studies [51, 9, 38, 45, 46] and was considered by most to be an inexpensive body tracking device, while Leap Motion was more considered for precise small hand tracking. However, both Leap Motion and Microsoft Kinect have trouble with tracking caused by occlusion and by too much surrounding light since they are based on infrared cameras.

Tracking the eyes of a user is a technology which might help in interaction but also in graphical processing power with foveated rendering, by rendering the 3D scene in high-quality at the area of where the eye is centrally looking and reducing the render quality in the peripheral vision, to save computing power. In interaction speciﬁcally eye-tracking may enable the user to interact with the virtual environment by looking at objects directly and selecting by pressing buttons or making gestures. Tobii is the leader in eye-tracking tech development and has implemented working systems in VR technology [97, 98].

2.7.3.2 Gestures

Gesture tracking is closely related to natural user interactions, used both in AR and VR with different devices. In AR there usually is a machine learning algorithm scanning the visual screen for recognisable gestures. For instance, Microsoft HoloLens can pick up a predefined distinct finger gesture to raise an event, activating an action. This is the reason gesture-based interactions are not counted as natural user interactions since they usually need to be quite distinct for the program to verify as a gesture thereby making the interaction less realistic.

29 of 52 State-of-the-Art Survey Result

It has also been noted that it is diﬃcult for users to understand how to properly perform a particular gesture which is understandable by the system. In the case of Google Glass, swipe patterns were shown to be hard to remember without a proper brieﬁng. Similar cases have been noted in VR where the user performs gestures based on virtual hands from Leap Motion or Kinect [38, 47].

2.7.3.3 Voice

Voice commands can be problematic in crowded or noisy areas. Voice input for AR applications running on smartphones and tablets, can be an intuitive way of user interaction due to the extensive use of smartphones today. However, it is not as immersive as techniques that allows natural user interactions, e.g. Leap Motion.

2.7.3.4 Controllers

Using controllers or other handheld devices does not provide the same immersion as using natural interactions, however, it may give a more responsive and precise experience when handling virtual objects.

Oculus Touch uses the same Constellation tracking as its HMD; Oculus Rift. To work optimally it requires at least two cameras to not lose tracking, making it more ﬁt to use in a room dedicated for this type of activities. The same applies to the HTC Vive Steam VR controllers, which also uses the Lighthouse tracking. Both controllers work well in gaming and have been applied to maintenance and inspection. By aid from software, certain virtual objects are able to attach to the virtual hands of the user to reduce the requirement of extreme preciseness and other minor problems that may occur during handling.

Attachable devices may give more freedom in handling virtual objects and are close to provide natural user interactions. Microsoft band was used in a study [48] to navigate around by using only the palms of the hands, Vive Tracker is an upcoming input device which enables the user to track the hands precisely by attaching it to the armwrist and wearing special tracking gloves, enabling the user to interact with virtual objects by open palms, which is not possible with handheld controllers.

For GearVR, a newly released VR controller will enable some interaction with the virtual environment when using the HMD, to complement the limited use of its touchpad. Other input methods include keyboards, mouse and joystick. Such as the case of the training simulation made by DAssault, where they use the keyboard to navigate the position in the virtual 3D world.

2.7.4 Miscellaneous

Measurements from studies involving students from diﬀerent areas, show that guidance through an AR application, displaying superimposed objects on a workspace in real-time, can provide shorter task completion time and lower rate of errors compared to other instruction methods. Conducting a simple assembly task while getting instructions through a video feed, were shown to produce 10.8 times more errors and took 22% more time than using a guiding AR application through the assemblage. AR-guidance for industrial manufacturing tasks can perform very well but requires an extensively

30 of 52 Conclusions State-of-the-Art Survey engineered environment. While tracking objects in an AR application, distractions can occur, not least in crowded manufacturing environments [70].

AR based solutions have proven to be a useful technical tool for education activities. Today’s novelty of the technique can interest and motivate students to engage themselves in AR-learning activities. Basic educational AR applications running on e. g. smartphones can help students to visualise a given problem [67, 49]. AR frameworks for inspection tasks, utilized in applications running on various platforms, can also reduce stress, compared to using paper- or PDF manuals while performing work [53].

Different companies have implemented VR training simulations using HTC Vive and Oculus Rift with Oculus Touch [99, 100, 101, 102]. The training takes place in a virtual environment specific for the chosen training with objects reacting to different interactions by the user. Maintenance training has been implemented with training on an engine in an assembly workshop where the user first watches a tutorial of how the procedure is done in a 3D modelled animation and then do the same procedure by using virtual tools available in the workshop. Another maintenance task is done at a railway track where the user is expected to repair a railroad switch in a realistic setting. The manufacturing training simulation offers the user to operate a lathe and inspecting products. Interaction with buttons and different parts are possible by the precision of the Oculus Touch and HTC Vive controllers.

2.8 Conclusions

In this survey, several studies and experiments involving VR and AR solutions have been presented. Some of the presented implementations are applicable to the stated problems in the requirements of this report. The result section was divided into diﬀerent subsections to get an overview of how particular implementations performs in the most important aspects of a virtual training application.

2.8.1 Result Evaluation

For the multi-user aspect, compared to AR, VR offers better support for real-time simulations with multiple users that can interact with the virtual environment. VR CAVE platforms enable multi-user environments however with a limit to the number of active users and typically with only one user allowed to be tracked at a time. COTS platforms include a plethora of tools to enable different HMDs for multi-user collaboration and allow as many users as desired depending on hardware capabilities. Same principles are found in AR which also offers multi-user and object-sharing capabilities through Microsoft HoloLens, Meta 2 and other smartphone or tablet implementations, however, programmers are restricted in the choice of development frameworks, compared to VR.

Concerning the graphics and system performance aspect, AR oﬀers tetherless implementations with up to six DOF which beneﬁts user mobility. However, wireless solutions limit available computing power and can therefore not provide the best graphics. Furthermore, most AR HMDs have a restricted FOV (compared to VR HMDs) and a minimum distance, in which virtual objects may appear. This impairs the use of AR in this aspect, especially because of the importance to be able to fully observe high-detail models. Additionally, AR-solutions often show instabilities of the appearance of virtual objects because of unstable tracking methods.

31 of 52 State-of-the-Art Survey Conclusions

In VR, it is recommended to use one of the more developed tethered HMDs connected to a powerful workstation to get the best VR experience and to avoid motion sickness. To allow positional tracking, however, it is important to understand how the used tracking system works to avoid potential problems and limitations, such as occlusion. CAVE platforms often require a lot of space for the speciﬁc components and powerful workstations to run smoothly causing the quality of the system to be highly related to available capital. Thus it is advocated to use COTS hardware platforms instead to develop interactive training simulations.

As for user interactions, AR and VR implementations offer various types of user input options and most of the mentioned stand-alone devices may be used in both environments. To fulfil the requirements, selecting a favourable input device depends on convenience, intuitive and stability. AR generally utilises touch, voice commands, and predefined hand gestures while certain VR solutions include controllers which imitate the motion and properties of the human hands but also enables the use of more traditional video game controllers, keyboards and pointing devices. Devices that are closer to natural user interactions or provide virtual hands, contributes to a sense of responsive control and a more realistic experience. Naturally, this sort of interaction devices are recommended for virtual training systems.

2.8.2 Summary

To determine whether an AR or VR-based implementation is suited for a technician training application, it is of great importance to examine the existing technological advances for both concepts, which was done in this survey.

It is found that AR requires further development before it can be properly implemented for a virtual technician training simulation. Currently, it pertains more to guidance by graphical overlays in maintenance, inspection or assembly work tasks, rather than the virtual technician training scenario.

At the current stage, VR has seen more implementations than AR in maintenance training and similar applications. In the context of COTS platforms, besides the low implementation cost, there are also many options to customise diﬀerent setups with existing frameworks making it very ﬂexible. VR also allows to simulate any environment and by using the right tools, this enables developers to conveniently create an immersive virtual learning platform for practical training purposes.

32 of 52 3 Methods and Tools

From the survey, it was concluded that VR was the route to take regarding implementation of practical technician training. While VR has some limitations compared to currently available technology in AR, technologies in VR had far more advantages in terms of current tracking technology and capabilities in display and interaction devices.

It was noted that most of the implementations in VR used similar hardware and development frameworks. Even though the Oculus Rift HMD was used more than HTC Vive, for the prototype implementation, HTC Vive was chosen for the reason that it provides a larger movement area than what the Constellation technology of Oculus Rift oﬀers. A larger movement area gives a better training environment for the trainee by allowing the ability to physically walk around more freely which makes the training more realistic. Figure 3.0.1 illustrates a top-down view of the tracked room that was used for the operation.

As an interaction and input device, Leap Motion was chosen as it performed well in the studies that used it and because it would provide better practical learning for the technician during training. Because of its high accuracy tracking of the ﬁngers and the hands it gives a realistic experience when the technician enacts the tasks physically by using their own hands.

The considered game engines was Unity or Unreal Engine, however, Unity was the ultimate choice because of its good platform and external hardware compatibility especially with HTC Vive and Leap Motion. HTC Vive requires SteamVR API to work which has OpenVR SDK in it, which supports other HMD’s as well. Leap Motion requires the Orion SDK to work and has diﬀerent modules which works with Unity, to enable interaction with the virtual models. The Unity project, together with the mentioned packages, was set up according to the instructions in Appendix A and a more detailed diagram of the system can be found in Appendix B.

3.1 Methods

3.1.1 Development Planning

The prototype was developed by using extreme programming (XP). The planning of which feature should be implemented for each week was done in a separate document. To learn about developing in a 3D game engine such as Unity, the FAQ on their web-page helped out.

3.1.2 Event-Driven Implementation

The simulation built in Unity was developed in an event-driven way where events, such as button clicks and user hand gestures trigger listener function calls. This could enable communication between two or more objects without the need of having a reference of each object in every object.

33 of 52 Methods and Tools Methods

Figure 3.0.1: A top-down view of the tracked area by using the Room Overview Window in SteamVR On the top-right and bottom-left are the basestations (Lighthouses) which scans the room in arcs of 120°, both horizontally and vertically. They were placed, no more than 5 meters apart from each other, at a high ground pointing downwards at a 30° - 45° angle towards a focal point of the movement area, marked by ’+’ in the ﬁgure. The real-time tracked HMD is displayed slightly above the centre. The surrounding blue lines are the boundary-marks (called Chaperone boundary) which are setup by the user to limit tracking when outside the FoV of the Lighthouses. The light blue rectangle is created automatically within the Chaperone boundary and is the play movement area for the user. When nearing the boundaries of this rectangle the user is warned to prevent moving outside the safe area for movement and activity. The Chaperone boundary in the ﬁgure measures approximately 3.6m × 4.6m, creating a play area averaging 3.2m × 4.0m. Further setup details can be found in [103].

34 of 52 Tools Methods and Tools

Figure 3.2.1: Basic Prototype System Overview

3.2 Tools

The selected tools may be seen diagrammatically in Figure 3.2.1 and will be further described in the following subsections.

3.2.1 Hardware

The HTC Vive HMD and Leap Motion controller has been described earlier in section 2.5.2.2). The Leap Motion was attached to the front of the HTC Vive HMD, so that the hands of the user are tracked in front of the head (see Figure 3.2.2). The Leap Motion and the HTC Vive were connected through USB and HDMI to a workstation running the simulation on Unity with the operating system Windows 10. The technical speciﬁcations of this workstation included an eight-core AMD FX-8320 CPU, 32GB RAM and an AMD Radeon R9 380 GPU.

Figure 3.2.2: HTC Vive with a front-attached Leap Motion

35 of 52 Methods and Tools Tools

3.2.2 Software

Development of a prototype application for demonstrating the concept requires speciﬁc tools, such as development frameworks and libraries to make the hardware work together.

3.2.2.1 Unity

Unity was used to develop and build the training simulation for the user. It is a multi-platform game engine developed by Unity Technologies. Mainly, it is used for developing games for personal computers, consoles, mobile devices and web pages. The development environment that comes with Unity enables a visual representation of the project. Game logic is scripted using C# or JavaScript. There is a broad set of asset packages that are used for extending a Unity project. Asset packages can contain 3D models, APIs for certain hardware or algorithms etc. [104].

The game engine is also mostly used in VR application development because of the facilitation of packages which enable an easy way to handle 3D models and to apply interaction between them and controllers. For this reason, it is used for this prototype development. Even though there are more recent versions, Unity 5.5.0 was utilised due to compatibility with currently available Leap Motion packages.

3.2.2.2 Blender

Blender is an open-source 3D modelling suite that was used in the project to manipulate the geometry of the imported models used in the simulation. The imported models usually didn’t have the correct scale or all the needed attributes, but this was solved by using the available set of tools in Blender [105].

3.2.3 Development Frameworks

3.2.3.1 SteamVR

To enable communication with the HTC Vive the SteamVR software was needed. The prototype application communicated with SteamVR through the OpenVR API and SteamVR communicated with the HTC Vive.

3.2.3.2 Orion SDK and Leap Motion Modules

Implementing interaction ability with the Leap Motion required the Orion SDK, which provided diﬀerent modules for Unity. The module Leap Motion Core Assets, enabled a 3D modelling of the hands. Using the Leap Motion Attachments Module enabled attachment of game object to the hands, so users could pick things up. Leap Motion UI Input Module included buttons, toggles and other UI elements to interact with the hands.

36 of 52 Other Resources Methods and Tools

3.3 Other Resources

• The aircraft manual for the Gripen ﬁghter plane was provided to show in detail how a procedure is performed by a technician. A procedure was then selected to be implemented in the prototype application.

• Virtual models by Saab were of CATIA 3D models and all the parts for a full model aircraft were not available. Instead, a free model of Gripen from Flight Simulator X, representing the outer shell of the aircraft was retrieved from a freeware site [106] providing aircraft models used in Flight Simulator. Other models in the simulation were retrieved from websites providing free 3D models for general use.

37 of 52 4 Prototype Development

4.1 Refuel Procedure Training

From the aircraft manual a procedure was selected that was considered appropriate as an initial practical training implementation for the prototype. The considered factors included the remaining time of the project and how well the interactions would ﬁt with Leap Motion. The procedure includes refuelling the aircraft by operating the Ground Crew Panel and the Refuelling Access Panel, which both are located to the starboard side of the hull on the centre of the aircraft. Other needed equipment includes a refuel hose, a ground cable and a pressure refuelling unit.

4.1.1 Ground Crew Panel

The Ground Crew Panel (GCP) is located at the starboard side of the hull to the centre of the aircraft. It is a relatively small panel concealed by a hatch. The hatch is able to be opened by pressing the ﬁngers on two quick-release lock buttons that are located on the bottom of the hatch. When the hatch is unlocked, the technician opens the hatch by pulling it towards oneself. The hatch is hinged unto the hull and rotates until it is perpendicular from the closed state.

The panel contains a battery switch, to provide power for the panel with its internal computers and several control buttons which are used to run various test on the aircraft before and after aircraft operation. From this panel, the technician is also able to communicate with the pilot in the cockpit by attaching a pair of speciﬁc headphones. For the refuelling procedure there is a dedicated fuel section on the panel, enabling the technician to run diagnostic tests and display the current fuel on a fuel indicator.

The fuel indicator displays three diﬀerent numbers, as either fuel level or diagnostic test errors. If a test has completed without any errors, the fuel indicator display will show three eights ("888") while pushing the test fuel button. Relatively close to the test fuel button is a fuel selector where the technician chooses the level of fuel the refuel equipment will ﬁll the aircraft with.

The real ground crew panel on Gripen has a lot more components than what is described here. Further disclosure is prevented by classiﬁed information. For this reason, the GCP panel in this prototype contains only a simpliﬁed implementation which only includes buttons that are relevant for the refuel procedure (Figure 4.1.1a).

4.1.2 Refuelling Access Panel

The Refuelling Access Panel (RAP) is similar to the Ground Crew Panel (GCP) and is located a few centimetres below the GCP hatch. The RAP hatch operates similarly to the GCP hatch by using the same lock buttons to open, however, it needs to be pulled downwards towards oneself instead of upwards and may be opened with a larger angle than 90° from the closed state. On the panel there is a refuel valve which has a protection cap which needs to be removed before the refuelling

38 of 52 Refuel Procedure Training Prototype Development

(a) Open ground crew panel (b) Open refuel access panel

Figure 4.1.1: Panels operation commences. The hatch needs to be opened as much as possible so that the refuel hose, between the refuel nozzle and the pressure refuelling unit, does not weigh on the hatch. A simpliﬁed implementation of this panel only includes the refuelling valve without any detailed parts (Figure 4.1.1b).

4.1.3 Refuel Equipment

The Pressure Refuelling Unit (PRC) and the ground cable are needed to refuel the aircraft. The PRC is similar to a compressor and provides sufficient pressure for the specific chemical fuel during the automatic refuel of the aircraft. A hose is connected between the fuel tank and the PRC and another hose from the PRC to the refuel valve on the aircraft. The most basic functionality of this device is to turn the pressure coupling to enable a flow of fuel at the appropriate pressure.

The ground cable is connected to the ground point of the aircraft, located to the right of the GCP panel hatch, and the PRC.

The virtual representations of these devices may be seen in Figure 4.2.2.

4.1.4 Instructions

The exact steps cannot be outlined publicly because of conﬁdential information in the manual, however the instructions (Table 4.1) has been brieﬂy broken down to steps that are manageable by a technician student and interaction by the Leap Motion controller, while still remaining closely realistic to the real procedure.

39 of 52 Prototype Development Result

Table 4.1: Simpliﬁed Refuel Procedure Instructions from the Aircraft-Manual

Step Instruction 1 Connect the ground cable to the aircraft ground point 2 Open the GCP and RAP panels. On GCP, switch the battery to ON 3 Remove the cap from the refuel valve and attach the refuel hose to the refuelling valve 4 Select the quantity of fuel to be filled and run a test by pressing the test button until the fuel indicator flashes between the test complete ("888") and the selected fuel quantity mode (e.g. "001" for 40%) → (888-001-888-001) 5 Set PRC to ON and open the pressure coupling 6 Refuel until it automatically stops and the fuel indicator displays the chosen fuel quantity 7 Set PRC to off and remove the coupling 8 Install the protective cap to the refuel valve and close the RAP hatch 9 Disconnect the ground cable from the aircraft 10 On GCP, set battery switch to OFF and close the hatch

4.2 Result

4.2.1 Virtual Model Representations

Developing a maintenance practice scenario, using models with a certain level of detail were of importance.

Gripen was downloaded from a website [106], which offers free Flight Simulator X models. This particular model came in the file format MDL, since this file format is not fully supported by the majority of 3D applications, a third party model converter was utilised, called ModelConverterX [107]. This application allows for conversion of objects between different file formats, and in this case, it was converted to OBJ, to enable import into Blender for further customisations. The model was found to have an extreme amount of detailed parts which had to be organised for easier selection.

The gears of the aircraft were up as well so to give the user a more realistic experience of the aircraft when standing in front of it, the gears had to be manually pulled out of the model using Blender. Access panels for the chosen procedure were also added to the aircraft hull. To the ﬁnal model an extra texture, created in Paint.NET, was applied to the hull to give it more roughness and a more natural appearance (Figure 4.2.1).

Refuel Equipment models were mainly downloaded from Unity Asset Store. Although, some parts were retrieved from [108] most commonly in SKP ﬁle format. Blender was used to handle conversion from SKP to FBX which is Unity-compatible. The Pressure Refueling Unit (Figure 4.2.2a) has a mounted handle that needed to be movable. To achieve this, the parts were separated using Blender. The ground plug (Figure 4.2.2b) and refuel hose nozzle (Figure 4.2.2c) models are originally designed as rocket engines but closely matches the look of the desired objects. The refuel wagon (Figure 4.2.2d) is a graphical representation of the fuel source and has no practical function.

40 of 52 Result Prototype Development

Figure 4.2.1: JAS 39C Gripen

(a) Pressure refueling unit (PRC) (b) Ground plug

Figure 4.2.2: Refuel equipment

Backdrops, skybox and asphalt assets were downloaded from Unity Asset Store and the website 3DWarehouse [108].

41 of 52 Prototype Development Result

Leap Motion Hand Models

The Capsule hand models are the default hand models one starts with after installing Leap Motion for the first time. The fingers have the appearance of cylinders which are attached to the joints of the hands and the fingers, which appears as spheres. It also has has attached arms (Figure 4.2.3a). This model was mostly used at the beginning until at a later stage where more human-like hand models were searched for to test whether it would be better or not. While human-like hands was found, it did not feel good when interacting with objects neither represented the real hands very well.

The Poly hand models was found from the Leap Motion Hand Assets package and looks slightly different with five thinly rectangular-shaped fingers (Figure 4.2.3b). The Poly hand models felt best when interacting with the virtual objects, probably because of the good fit between graphical and physical representations. Even though it does not have any attached arms, it was decided to let it remain that way since the arms could cause inconveniency when interacting with objects as they sometimes could dislocate themselves when the hands were affected by bad tracking conditions.

(a) Capsule hands

(b) Poly hands

Figure 4.2.3: Hand models

42 of 52 Result Prototype Development

4.2.2 Interaction Design

Diﬀerent factors were considered which would be of importance in implementing interaction. The interactions that would be most prevalent in the application prototype implementation of procedures would be removing and attaching objects from other objects, opening and closing hatches, activating buttons and turning handles.

To make the operation be close to how the real world operation is executed, it was important to make the actions in the real-world to be represented as identical as possible with Leap Motion. Doing this may give the trainee a sense of how a procedure is done in real life and opens up possibilities for muscle memory training, however to conﬁrm the eﬀectiveness of this possibility, further research will need to be done in this aspect together with Leap Motion.

4.2.2.1 User Interfaces

Traditional ﬁrst-person games often oﬀer user interfaces where text and other communicative medium are represented two-dimensionally on the view camera, this type of user interface representation is called non-diegetic. The opposite representation is called diegetic and is represented in the real game world instead of being shown as an overlay [109]. To avoid the risk of breaking immersion for the user, it is best to use diegetic representations of information.

In the case of the GCP panel, when pressing the battery button, the refuel indicator represents whether the battery is on or not, which is a form of a diegetic representation.

4.2.2.2 Distinct Procedure Actions

Picking Up Objects Being able to pick up the certain objects using the user’s right remote hand was implemented with help from Leap Motion Attachments package and some custom scripting. By detecting if the ring and middle ﬁnger is suﬃciently close to the hand palm, it is determined whether the hand is in a gripping pose or not. It is also determined if a graspable object is close to the hand palm. If the object is close enough to the right hand, the object locks to the hand for carrying and placing for as long as the hand of the user is in a gripping mode. Figure 4.2.4 depicts an example of the user picking up the refuel nozzle to bring it to the refuelling valve on the refuel panel.

Figure 4.2.4: Picked up refuel hose nozzle

43 of 52 Prototype Development Result

Rotating objects The ability to loosen certain objects through rotation was implemented, by utilising the Leap Motion Attachments package and installing custom scripting. The lid that covers the refuel valve in the Refuel Access Panel could this way be removed, rotating the right hand in a gripping state (Figure 4.2.5).

Figure 4.2.5: Rotating the refuel valve protection cap

Pressing Buttons To provide a realistic impression of pressing a button for the user, it was important to provide feedback when executing the actions. In the case with buttons the UI library in Unity, visual as well as audio feedback was provided. When the user is pushing a button with a Leap Motion ﬁnger, a visual cue is represented by fading the button to a darker colour while at the same time an audio clip is played resembling the sound of a initial click. When the ﬁnger is released away from the button, the button colour is reverted to the original light shade colour as it was previously, before it was pushed, and another audio clip is played, resembling a retracting button click. Figure 4.2.6 depicts a button interaction with the fuel test button on the ground crew panel.

(a) Pressing button (b) Releasing button

Figure 4.2.6: Button interaction

44 of 52 Result Prototype Development

Figure 4.2.7: Menu

4.2.2.3 Menu Design

Using the basic Unity UI assets in combination with the Leap Motion UIInput module, a hand-operated menu (Figure 4.2.7) was built to enable four diﬀerent menu choices:

• Usage of tools

• Navigation in close proximity around the Gripen model

• A readable manual with task instructions

• An X-ray mode, which adds transparency to certain parts of the Gripen model

When the user directs the left palm towards the face, a detection algorithm from the Leap Motion Core Assets, detects the orientation of the hand and spawns a menu to the right of the open palm.

Tools In the current implementation there is only one tool, but it is possible to easily add any form of tool to the list. The only existing tool at the moment is a laser pointer. When selected, this tool is placed on the right index ﬁnger of the users Leap Motion hands and can be used to point at diﬀerent objects of the aircraft to get a small description of the name of the aimed object. Figure 4.2.8 shows an example where the object has changed colour to a green shade to give a representation of the shape.

Figure 4.2.8: Laser pointer

45 of 52 Prototype Development Result

Navigation The ability for user locomotion in the virtual world was solved by translocating the user. From the research it was recommended to not include movement that could induce simulator- sickness to the player, caused by unwanted movement for the player, which is why translocation works well in limited cases. It is important to let the user feel as if they are in a realistic world, which led to the decision that positional tracking would be enough to physically walk around small sectors of the aircraft and using translocation would be an ability to move to a diﬀerent sector of the aircraft. The navigation menu has four buttons (right, left, front and rear) from the UnityUI library, each translocating the user to the corresponding side of the aircraft.

Figure 4.2.9 demonstrates an image sequence of the navigation process, where the user is standing at the right side of the aircraft and presses the front button of the navigation menu. A fading transition is then enacted and the user is translocated to the front of the aircraft. The fading transition was implemented to reduce the risk of disorientation when the entire environment instantly changes for the user.

(a) Pressing navigate front (b) Fading transition (c) Translocation completed

Figure 4.2.9: Navigation sequence

Manual The instructions was created by applying text to a Canvas, which is a UI element from the UnityUI library. The entire planning included the precondition that the user knows which steps to execute during the simulation, by looking at the manual beforehand or during simulation. In the prototype, the manual only consists of the refuel procedure instructions. However, since the real manual is vastly more complicated, this would not be a desirable solution if intending to implement the entire manual into the simulation. Instead some sort of PDF renderer would need to be created. But because of limited time and other reasons involving areas out of the scope of this project, this option was not implemented into the prototype. Providing a manual inside the simulation gives a high sense of realism and also reduces the risk of simulator-sickness.

Figure 4.2.10: Refuel manual

46 of 52 Result Prototype Development

X-Ray A transparency manipulation mode was implemented to set the transparency level of the aircraft. Changing the transparency makes the user able to see electrical wires and the internal parts of the aircraft. This could also be developed to only change the transparency of selected parts of the aircraft. A UI slider from the Leap Motion UI Input package was added and programmed to alter the transparency level (Figure 4.2.11).

(a) Normal opaque mode (b) Transparent mode

Figure 4.2.11: X-Ray mode

4.2.2.4 Multi-user Capabilities

At the time of this report, Leap Motion does not provide any packages for Unity with complete multi- user functionality for networking. Due to this, multi-user support was not fully implemented into the prototype. Instead, another stand-alone Unity application for desktop was developed using Unity’s built-in network High Level API (HLAPI), Leap Motion Core Assets and Attachments packages with supplementary self-written scripts. This enabled remote transformation of hand representations over network.

Figure 4.2.12 shows an example of two clients connected together. The perspective is from a first- person view and the users right hand is shown at the bottom of the figure. The hand of the other used is near the engine and is pointing at different parts for demonstration. The hands are represented by a sphere which imitates the hand palm and five cuboids which represents the fingers. This could of course be further implemented to include different joints, until Leap Motion comes with their own solution.

Figure 4.2.12: Multiple users

47 of 52 Prototype Development Discussion of Implementation

4.3 Discussion of Implementation

Starting from the beginning with setting up the project, getting used to Unity and the other tools, the implemented procedure took about 200 hours of work divided over two persons to complete. To implement a procedure with similar steps as the refuelling procedure, would probably take half of the time. The reason for this is the possibility to reuse functions that took time to implement and having more experience now helps to outline the needed steps to solve a problem.

It was challenging at ﬁrst to implement this prototype yet it was very educational for us to solve compatibility issues and combining diﬀerent packages and libraries together. Adding interaction to objects in Unity was not as challenging but rather required more time to make it more polished when interacting with Leap Motion. While there are some minor immersive-breaking situations, it still gives the user a general feeling of actually standing in front of a real aircraft. Links to videos showing the refuel procedure operation and the menu interaction can be found in Appendix C.

4.3.1 Expenditure

Örebro University robotics lab had a HTC Vive HMD and Leap Motion controller which we used for debugging. Though, since we also had to be able to debug from home or at Saab on our private laptops, we bought our own Leap Motion controller which costs about $80.

Total costs for the hardware components needed to run the Unity application at the time of the report:

Workstation $1000 HTC Vive $900 Leap Motion $80 Total $ 1980 Being able to develop Unity projects commercially comes with a cost of $125 per month if the revenue is more than $100.000 a year. Otherwise it can be used for free.

4.3.2 Evaluation of Used Hardware

Developing with HTC Vive was surprisingly easy together with SteamVR. The HTC Vive performed well and it felt very comfortable to equip it during simulation. If it did not ﬁt well enough, there were some issues with light leakage but it was alleviated by ﬁtting it more properly to the head.

At times the simulation was not able to keep up with the required rendering so it would cause the simulation to stutter and lag. While this was a minor nuisance and could cause simulator-sickness, it can be alleviated by simply acquiring a more powerful workstation.

Leap Motion worked well during simulation, however, it could lose detection of the ﬁngers when there would be too much light in the room, or if the other hand crossed the other causing occlusion. This was one of the important reasons the menu was not placed directly on the palm.

Another thing that which has been mentioned before and can be conﬁrmed by the prototype implementation is that Leap Motion requires the user to use their hands in the FOV of the Leap Motion

48 of 52 Discussion of Implementation Prototype Development controller. While the FOV is great, it is not possible to track the hands if they are beyond the tracking limitation of the controller.

The Native HTC controllers could probably be a plausible tool for interactions in a simulation similar to this prototype.

4.3.3 Development Potential and Future Developments

The potential for development is great. As this is only a prototype, there are a lot of things that could be improved to provide a better experience, such as better models for better immersion, in particular the aircraft model to provide more detailed objects for the user.

A better menu with more options would also be a welcoming feature, the manual in particular could be improved and more tools to use with the aircraft. Another improvement for tools could be by using the Vive Tracker attached to a real tool, which is depicted virtually in the virtual environment, so that the trainee may train with real tools, in the virtual training world. Oculus is also working on the same thing by adding the possibility to track a third Touch controller. Further research will have to show whether there is a signiﬁcant diﬀerence in learning between using real tools in virtual training versus using only virtual tools.

Multi-user capabilities are already at a point where its possible to add several players into the application, with more time the functionality would have been implemented.

Further development with interface representations and better feedback when interacting with the objects and the menu would be a necessary addition to make the application even more interactive and immersive. At this point the user needs to select menu options by looking at the menu and touching with a Leap Motion ﬁnger. To make interactions better, it would be a beneﬁt to follow the VR design guide made by Leap Motion [110]. Since Leap Motion is a rapidly developing technique, the software surrounding this device might change. Further support would be required to keep it up to date. At the time of this report there are packages which are used in the prototype that recently became deprecated and replaced by one combined package.

Also, in the future, eye-tracking may remove the need to interact with a menu physically entirely and instead let the user select with their eyes to translocate by just looking at a speciﬁc area or bring up information by looking at objects and so on. However, it comes down to implementation decisions to decide whether it is the correct path to take for the developed application.

The development possibilities goes as far as our imagination. Hopefully further technological developments will battle the current limitations of technology. Development of VR applications will become even easier and allow for more possibilities than what already exists today to give a truly immersive and exuberant alternate world.

49 of 52 5 Discussion

5.1 Compliance with the Project Requirements

5.1.1 Summary of the Requirements

The primary requirements, as stated in section 1.4, indicate that an in-depth survey of VR and AR had to be completed to decide an appropriate technology for practical technician training application. A prototype application would be developed to show the capabilities of the particular technology. There were three secondary requirements to show the capabilities of the selected technology. The following list will give a better overview of the requirements:

• An in-depth survey of VR and AR

– Will produce a documentation of the advantages and disadvantages

– Investigations will indicate which technology is suggested for developing a practical technician training

• A prototype application should be developed to show some capabilities of the particular technology. With three secondary requirements:

– A teacher should be able to demonstrate a part of the aircraft for the trainee

– A trainee should be able to do a procedure training by themselves

– A trainee should be able to do a procedure training by the aid of teacher guidance

5.1.2 Fulﬁlled Requirements

A substantial part of this report contains the survey of VR and AR containing advantages and disadvantages for each ﬁeld and a conclusion section which argues which technology is the best to use for practical technician training. In this case, it was concluded that VR, at the current time, is the most appropriate technology to use.

An in-depth survey of VR and AR

Will produce a documentation of the advantages and disadvantages

Investigations will indicate which technology is suggested for developing a practical technician training

To fulfil the second primary requirements a prototype application was implemented to show the capabilities of VR and guidelines from the survey was followed to ensure an implementation which would give a realistic experience for the trainee. Two of three of the secondary requirements were fulfilled with the prototype implementation. The remaining unfulfilled requirement was the multi-user feature, where the teacher would have been able to demonstrate an engine or a part of the aircraft to the

50 of 52 Impact on Society Discussion trainee in the same virtual environment. However, a standalone application was created to prove that it is possible to do with a bit more time on hand.

• A prototype application should be developed to show some capabilities of the particular technology. With three secondary requirements:

– A teacher should be able to demonstrate a part of the aircraft for the trainee

A trainee should be able to do a procedure training by themselves

A trainee should be able to do a procedure training by the aid of teacher guidance

5.2 Impact on Society

The development of VR applications for practical training can provide realistic training environments for many areas such as healthcare, car manufacturing etc. Enabling this type of training requires investments in the form of VR hardware and development of tailored applications. In some contexts, assets for the applications already exist, such as 3D-models, CAD-drawings which can alleviate the application development process.

If more companies would consider to adopt this way of training, this could in some ways be in favour for the society, since it is all about virtual content. Using virtual content means that less hardware has to be manufactured just for the sake of training, which could reduce the need of using natural resources for this purpose.

If Leap Motion is used more in training it may help with the development of this technology and will hopefully further increase the tracking accuracy of the hands of the user, without using any type of attached devices to the hands.

While VR and AR technology offers a wide form of advantages in the educational aspect, it requires a lot of time to develop a good functioning application or device. This may not be a good thing for enterprises that wishes to invest into this technology, since time is considered as money in the business world. This makes the balance between making realistic applications and production time a bit difficult to define. While consumers want a very realistic environment to spend their time in, the producers do not wish to spend several years to create an application and also do not want to force users to buy expensive powerful computers. Though, as computers become more powerful and cheaper, and the development of VR and AR will further increase in the coming years, it will become more accessible to more users, making these kind of applications invaluable for education.

Once the AR technology has developed a bit more, it may be possible to train people while at the same time letting them perform the work. It will make it easier to get people into work and may reduce the unemployment rate. Of course, it depends on which kind of work one talks about but as an example perhaps a business needs a cashier to work one evening. The AR technology may in that case help with showing overlays on the cash register for a person which has no previous experience.

Using this technology opens up a lot of doors to experience things one cannot do otherwise. It may help people with simulating the future, it may help with healthcare and overall make people smarter. Simulating the future may help with people who would like to train their ability to talk in front of people, get a visual representation of a future project and more. In healthcare, VR and AR

51 of 52 Discussion Project Development may help doctors with diagnosing patients by visually representing data about the patient, about which medications or which kind of illnesses one has. It may also help with therapy by letting a patient live through their fears or by aiding in surgery. Using this technology in education, will make people generally smarter. Instead of reading a book about an historic event, one may be able to relive through the event by using VR or AR. Using multi-sensory learning has been proven to increase the eﬀectiveness of learning and by using technology which enables us to do just that, it will revolutionise the education system.

5.3 Project Development

The survey in this project was purely based on articles and papers. This could have been supplemented with input from end-users (technicians and instructors) using some questionnaires. The prototype demonstrates some of the important abilities of a VR application pertained for practical procedure training. As the authors barely had any previous experience in Unity, this implementation can reveal how convenient it is to work with a tool such as Unity, rather than drawing the foundation of a VR application design pattern.

If the project would be evolved any further, developers could consider to use this prototype as a pointer while creating a new application project using well established development patterns. If the program would instead include more authentic and advanced 3D models, the program structure would become an important part of the design. Besides the increase in size of the application, it would be diﬃcult to navigate to desired model parts of the aircraft unless one would use an appropriate interface.

5.4 Reﬂection on Own Learning

5.4.1 Knowledge and Comprehension

After having completed the survey, it has been a significant knowledge booster in the field of VR and AR. We do feel that we know a lot more about where the technology in both fields are at the current stage and which problems each field needs to solve to provide a better solution than what exists today. The implementation work has increased our understanding for development using a tool such as Unity. We learned more about how to exploit the freedom of combining different code libraries to create the desired VR application.

5.4.2 Proﬁciency and Ability

By doing an extensive research on the particular fields we have learned to define and extract problems from the requirements. Using different methods have aided to process and analyse results of technical and scientific studies. From the studies we have been able to produce a report which is well structured and relevant to the project. We have also presented the project orally to our fellow course students and Saab Aeronautics employees through the use of illustrations and videos of the implementation.

52 of 52 Bibliography

[1] Rustici Software. Scorm explained, 2017. URL https://scorm.com/scorm-explained/. Accessed: 2017-05-22.

[2] Rustici Software. Tin can api tech overview, 2017. URL http://tincanapi.com/overview/. Accessed: 2017-05-22.

[3] Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino. Augmented reality: A class of displays on the reality-virtuality continuum. In Photonics for industrial applications, pages 282–292. International Society for Optics and Photonics, 1995.

[4] Wikimedia Commons (Russel Freeman). Virtuality continuum, 2007. URL https:// en.wikipedia.org/wiki/File:Virtuality_Continuum_2.jpg. [Digital image] File: Virtuality_Continuum_2.jpg. Retrieved: 2017-05-22.

[5] Nirit Gavish, Teresa Gutierrez, Sabine Webel, Jorge Rodriguez, and Franco Tecchia. Design guidelines for the development of virtual reality and augmented reality training systems for maintenance and assembly tasks. BIO Web of Conferences, 1:00029, 2011. doi: 10.1051/bioconf/ 20110100029.

[6] Oculus. Oculus best practices, 2015. URL http://static.oculus.com/documentation/ pdfs/intro-vr/latest/bp.pdf. Accessed: 2017-04-12.

[7] Leap Motion. Vr best practices guidelines, June 2015. URL https://developer-archive. leapmotion.com/assets/Leap%20Motion%20VR%20Best%20Practices%20Guidelines. pdf. Accessed: 2017-06-06.

[8] Tom DeFanti. Vr: past, present, and future [virtual reality]. In Image Processing, 1998. ICIP 98. Proceedings. 1998 International Conference on, pages 1–vol. IEEE, 1998.

[9] Minseok Kim and Jae Yeol Lee. Touch and hand gesture-based interactions for directly manipulating 3d virtual objects in mobile augmented reality. Multimedia Tools and Applications, feb 2016. doi: 10.1007/s11042-016-3355-9.

[10] Christoph Anthes, Rubén Jesús García-Hernández, Markus Wiedemann, and Dieter Kranzlmüller. State of the art of virtual reality technology. In Aerospace Conference, 2016 IEEE, pages 1–19. IEEE, 2016.

[11] Gabriele Bleser and Didier Stricker. Advanced tracking through eﬃcient image processing and visual–inertial sensor fusion. Computers & Graphics, 33(1):59–72, 2009.

[12] XinReality. Constellation, 2016. URL https://xinreality.com/wiki/Constellation. Accessed: 2017-04-15.

[13] XinReality. Oculus touch, 2016. URL https://xinreality.com/wiki/Oculus_Touch. Accessed: 2017-04-15.

53 [14] Wikimedia Commons (Samwalton9). The oculus rift cv1 headset, 2017. URL https: //commons.wikimedia.org/wiki/File:Oculus_Consumer_Version_1.jpg. [Digital image] File: Oculus_Consumer_Version_1.jpg. Retrieved: 2017-05-24.

[15] Wikimedia Commons (Samwalton9). Oculus touch controllers for the oculus rift cv1, 2017. URL https://commons.wikimedia.org/wiki/File:Oculus_Touch_Controllers.jpg. [Digital image] File: Oculus_Touch_Controllers.jpg. Retrieved: 2017-05-24.

[16] P. Dempsey. The teardown: HTC vive virtual reality headset. Engineering & Technology, 11(7): 80–81, aug 2016. doi: 10.1049/et.2016.0731.

[17] Oliver Kreylos. Lighthouse tracking examined, 2016. URL http://doc-ok.org/?p=1478. Accessed: 2017-04-15.

[18] rvdm88. Htc vive lighthouse chaperone tracking system explained, August 2015. URL https: //www.youtube.com/watch?v=J54dotTt7k0. YouTube [Video ﬁle, 1:26] Accessed: 2017- 04-27.

[19] Vive. Vive tracker, 2017. URL https://www.vive.com/us/vive-tracker/. Accessed: 2017-04-20.

[20] Vive. Vive controller, 2017. URL https://www.vive.com/us/accessory/controller/. Accessed: 2017-04-20.

[21] ﬂickr. Htc vive now up for pre-order, 2016. URL https://www.flickr.com/photos/ bagogames/25845851080/. [Digital image]. Retrieved: 2017-05-24.

[22] Samsung. Samsung gear vr speciﬁcations, 2017. URL http://www.samsung.com/global/ galaxy/gear-vr/specs/. Accessed: 2017-04-20.

[23] Wikimedia Commons (CS104group72015). Image of the samsung gear vr ﬁnal version, 2015. URL https://commons.wikimedia.org/wiki/File:Samsung_Gear_VR_V1.png. [Digital image] File: Samsung_Gear_VR_V1.png. Retrieved: 2017-05-24.

[24] Frank Weichert, Daniel Bachmann, Bartholomäus Rudak, and Denis Fisseler. Analysis of the accuracy and robustness of the leap motion controller. Sensors, 13(5):6380–6393, 2013.

[25] How does the leap motion controller work?, 2014. URL http://blog.leapmotion.com/ hardware-to-software-how-does-the-leap-motion-controller-work/. Accessed: 2017-05-28.

[26] Wikimedia Commons (SkywalkerPL). Leap motion orion controller connected, 2016. URL https://commons.wikimedia.org/wiki/File:Leap_Motion_Orion_Controller_ Plugged.jpg. [Digital image] File: Leap_Motion_Orion_Controller_Plugged.jpg. Retrieved: 2017-05-24.

[27] Oliver Wasenmüller and Didier Stricker. Comparison of kinect v1 and v2 depth images in terms of accuracy and precision. In Asian Conference on Computer Vision Workshop (ACCV workshop), Springer, 2016.

54 [28] Wikimedia Commons (litheon). Kinect sensor as shown at the 2010 electronic entertainment expo, 2010. URL https://commons.wikimedia.org/wiki/File:KinectSensor.png. [Digital image] File: KinectSensor.png. Retrieved: 2017-05-24.

[29] Wikimedia Commons (Evan-Amos). The xbox one’s kinect, 2014. URL https:// commons.wikimedia.org/wiki/File:Xbox-One-Kinect.jpg. [Digital image] File: Xbox-One-Kinect.jpg. Retrieved: 2017-05-24.

[30] Unity. Unity 3d, 2017. URL https://unity3d.com/. Accessed: 2017-04-15.

[31] Unreal Engine. Unreal engine, 2017. URL https://www.unrealengine.com/. Accessed: 2017-04-20.

[32] Valve. Openvr, 2017. URL https://github.com/ValveSoftware/openvr. Accessed: 2017-04-20.

[33] What is osvr?, 2016. URL http://www.osvr.org/what-is-osvr.html. Accessed: 2017- 05-21.

[34] Oculus. Oculus sdk, 2017. URL https://developer.oculus.com/licenses/sdk-3.4.1/. Accessed: 2017-04-20.

[35] Valve. Steamvr, 2017. URL http://store.steampowered.com/steamvr. Accessed: 2017- 04-20.

[36] Xiaolin Quan, Feng Feng, Limin Qiao, Shaochun Zhong, and Shusen Shan. The research and application of aircraft maintenance virtual teaching training system. In 2011 International Conference on Mechatronic Science, Electric Engineering and Computer (MEC). IEEE, aug 2011. doi: 10.1109/mec.2011.6025387.

[37] Zhengwei Chang, Yu Fang, Yun Zhang, and Can Hu. A training simulation system for substation equipments maintenance. In 2010 International Conference on Machine Vision and Human- machine Interface. IEEE, 2010. doi: 10.1109/mvhi.2010.31.

[38] Courtney McNamara, Matthew Proetsch, and Nelson Lerma. Investigating low-cost virtual reality technologies in the context of an immersive maintenance training application. In Lecture Notes in Computer Science, pages 621–632. Springer International Publishing, 2016. doi: 10.1007/978-3-319-39907-2_59.

[39] David B. Kaber, Yingjie Li, Michael Clamann, and Yuan-Shin Lee. Investigating human performance in a virtual reality haptic simulator as inﬂuenced by ﬁdelity and system latency. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 42(6): 1562–1566, nov 2012. doi: 10.1109/tsmca.2012.2201466.

[40] Simone Borsci, Glyn Lawson, Bhavna Jha, Mark Burges, and Davide Salanitri. Eﬀectiveness of a multidevice 3d virtual environment application to train car service maintenance procedures. Virtual Reality, 20(1):41–55, jan 2016. doi: 10.1007/s10055-015-0281-5.

[41] Marcio Cabral, Olavo Belloc, Andre Montes, Eduardo Zilles Borba, and Marcelo Knorich Zuﬀo. VR THOR — virtual reality training with hotstick on operations risks. In 2016 IEEE Virtual Reality (VR). IEEE, mar 2016. doi: 10.1109/vr.2016.7504786.

55 [42] DAssault Systems. Immersive maintenance training, 2017. URL http: //www.dassaultfalcon.com/en/MediaCenter/Newsd/Pages/PR%202017/ Immersive-maintenance-training.aspx. Accessed: 2017-04-06.

[43] Raul Crespo, Rene Garcia, and Samuel Quiroz. Virtual reality simulator for robotics learning. In 2015 International Conference on Interactive Collaborative and Blended Learning (ICBL). IEEE, dec 2015. doi: 10.1109/icbl.2015.7387635.

[44] Wadee S. Alhalabi. Virtual reality systems enhance students’ achievements in engineering education. Behaviour & Information Technology, 35(11):919–925, jul 2016. doi: 10.1080/ 0144929x.2016.1212931.

[45] Thomas Hilfert and Markus K?nig. Low-cost virtual reality environment for engineering and construction. Visualization in Engineering, 4(1), jan 2016. doi: 10.1186/s40327-015-0031-5.

[46] Helen V. Diez, Sara García, Andoni Mujika, Aitor Moreno, and David Oyarzun. Virtual training of ﬁre wardens through immersive 3d environments. In Proceedings of the 21st International Conference on Web3D Technology - Web3D '16. ACM Press, 2016. doi: 10.1145/2945292. 2945296.

[47] Chaowanan Khundam. First person movement control with palm normal and hand gesture interaction in virtual reality. In 2015 12th International Joint Conference on Computer Science and Software Engineering (JCSSE). IEEE, jul 2015. doi: 10.1109/jcsse.2015.7219818.

[48] Salah Eddin Alshaal, Stylianos Michael, Andreas Pamporis, Herodotos Herodotou, George Samaras, and Panayiotis Andreou. Enhancing virtual reality systems with smart wearable devices. In 2016 17th IEEE International Conference on Mobile Data Management (MDM). IEEE, jun 2016. doi: 10.1109/mdm.2016.60.

[49] Maria-Blanca Ibanez, Angela Di-Serio, Diego Villaran-Molina, and Carlos Delgado-Kloos. Augmented reality-based simulators as discovery learning tools: An empirical study. IEEE Transactions on Education, 58(3):208–213, aug 2015. doi: 10.1109/te.2014.2379712.

[50] Dhiraj Amin and Sharvari Govilkar. Comparative study of augmented reality sdk's. International Journal on Computational Science & Applications, 5(1):11–26, feb 2015. doi: 10.5121/ijcsa. 2015.5102.

[51] Paul Davies and David Lee. Augmented reality in manufacturing at the boeing company - lessons learned and future directions, Dec 2014. URL http://thearea.org/wpfb-file/ augmented_reality_at_boeing_-_lessons_learned-pdf/.

[52] David Fonseca, Nuria Martí, Isidro Navarro, Ernest Redondo, and Albert Sanchez. Using augmented reality and education platform in architectural visualization: Evaluation of usability and student’s level of sastisfaction. In Computers in Education (SIIE), 2012 International Symposium on, pages 1–6. IEEE, 2012.

[53] Perla Ramakrishna, Ehtesham Hassan, Ramya Hebbalaguppe, Monika Sharma, Gaurav Gupta, Lovekesh Vig, Geetika Sharma, and Gautam Shroﬀ. An AR inspection framework: Feasibility study with multiple AR devices. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). IEEE, sep 2016. doi: 10.1109/ismar-adjunct.2016.0080.

56 [54] Microsoft. Mixed-reality rendering, 2017. URL https://developer.microsoft.com/ en-us/windows/mixed-reality/rendering. Accessed: 2017-04-16.

[55] Wikimedia Commons (Ramadhanakbr). Microsoft hololens, 2016. URL https://commons. wikimedia.org/wiki/File:Ramahololens.jpg. [Digital image] File: Ramahololens.jpg. Retrieved: 2017-05-25.

[56] Rod Furlan. The future of augmented reality: Hololens - microsoft's AR headset shines despite rough edges [resources_tools and toys]. IEEE Spectrum, 53(6):21–21, jun 2016. doi: 10.1109/ mspec.2016.7473143.

[57] Microsoft. Cortana, 2017. URL https://www.microsoft.com/en-us/mobile/ experiences/cortana/. Accessed: 2017-04-16.

[58] Microsoft. Hololens purchase, 2017. URL https://www.microsoft.com/en-us/hololens/ buy. Accessed: 2017-04-16.

[59] MetaVision. Meta 2 speciﬁcations, 2017. URL https://buy.metavision.com/products/ meta2. Accessed: 2017-04-16.

[60] Patrick Renner and Thies Pfeiﬀer. Attention guiding techniques using peripheral vision and eye tracking for feedback in augmented-reality-based assistance systems. In 2017 IEEE Symposium on 3D User Interfaces (3DUI). IEEE, 2017. doi: 10.1109/3dui.2017.7893338.

[61] Wikimedia Commons (MetaMarket). Meta 2 headset, 2016. URL https://commons. wikimedia.org/wiki/File:Meta_2.jpg. [Digital image] File: Meta_2.jpg. Retrieved: 2017-05-25.

[62] A. Ho, C. Maritan, J. Sullivan, E. Cheng, and S. Cao. Measuring glance legibility of wearable heads-up display interfaces using an adaptive staircase procedure: A study with google glass. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 60(1):2073–2077, sep 2016. doi: 10.1177/1541931213601470.

[63] Johnny Yau Cheung Chang, Lok Yee Tsui, Keith Siu Kay Yeung, Stefanie Wai Ying Yip, and Gilberto Ka Kit Leung. Surgical vision. Surgical Innovation, 23(4):422–426, aug 2016. doi: 10.1177/1553350616646477.

[64] Google. Google glass locations and sensors, 2015. URL https://developers.google.com/ glass/develop/gdk/location-sensors. Accessed: 2017-04-16.

[65] Wikimedia Commons (Tim.Reckmann). Google glass - view of the mini-computer, 2014. URL https://commons.wikimedia.org/wiki/File:Google_Glass_Main.jpg. [Digital image] File: Google_Glass_Main.jpg. Retrieved: 2017-05-25.

[66] Wikimedia Commons (Evan-Amos). Google cardboard headset, 2015. URL https:// commons.wikimedia.org/wiki/File:Google-Cardboard.jpg. [Digital image] File: Google-Cardboard.jpg. Retrieved: 2017-05-25.

[67] Maria Blanca Ibanez, Angela Di Serio, Diego Villaran, and Carlos Delgado-Kloos. The acceptance of learning augmented reality environments: A case study. In 2016 IEEE 16th International Conference on Advanced Learning Technologies (ICALT). IEEE, jul 2016. doi: 10.1109/icalt.2016.124.

57 [68] Total Immersion. D’fusion studio, 2015. URL http://www.t-immersion.com/products/ dfusion-suite/dfusion-studio. Accessed: 2017-04-17.

[69] Ramakrishna Perla, Gaurav Gupta, Ramya Hebbalaguppe, and Ehtesham Hassan. InspectAR: An augmented reality inspection framework for industry. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). IEEE, sep 2016. doi: 10.1109/ismar-adjunct. 2016.0119.

[70] Frieder Loch, Fabian Quint, and Iuliia Brishtel. Comparing video and augmented reality assistance in manual assembly. In 2016 12th International Conference on Intelligent Environments (IE). IEEE, sep 2016. doi: 10.1109/ie.2016.31.

[71] ScopeAR. Scopear: An augmented reality company, 2017. URL http://www.scopear.com/ about/. Accessed: 2017-04-18.

[72] Caterpillar. A whole new reality, 2015. URL http://www.caterpillar.com/en/news/ caterpillarNews/innovation/a-whole-new-reality.html. Accessed: 2017-04-18.

[73] PR Newswire. Scope ar and caterpillar deliver ﬁrst augmented reality-based remote support platform for heavy industry. http://www.prnewswire.com/news-releases/scope-ar-and- caterpillar-deliver-ﬁrst-augmented-reality-based-remote-support-platform-for-heavy-industry- 300359664.html, 2016. Accessed: 2017-04-18.

[74] EquipmentWorld. Caterpillar augmented reality inspection demo, October 2015. URL https: //www.youtube.com/watch?v=S8jMgBimuxg. YouTube [Video ﬁle, 5:22] Accessed: 2017- 04-18.

[75] Mathieu Garon, Pierre-Olivier Boulet, Jean-Philippe Doironz, Luc Beaulieu, and Jean-Francois Lalonde. Real-time high resolution 3d data on the HoloLens. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). IEEE, sep 2016. doi: 10.1109/ ismar-adjunct.2016.0073.

[76] David Kanter. Graphics processing requirements for enabling immersive vr. AMD White Paper, 2015.

[77] Dassault Falcon. 3d maintenance magic - virtual reality for immersive training, February 2017. URL https://www.youtube.com/watch?v=LzqR7JsVVoY. YouTube [Video ﬁle, 4:16] Accessed: 2017-04-20.

[78] Dassault Falcon. How virtual reality tools train dassault falcon aircraft mechanics ? aintv, February 2017. URL https://www.youtube.com/watch?v=Yb4-ASQX1AQ. YouTube [Video ﬁle, 9:28] Accessed: 2017-04-20.

[79] Photon. We make multiplayer simple., 2017. URL https://www.photonengine.com/en-US/ Photon. Accessed: 2017-05-23.

[80] Gaﬀer On Games. What every programmer needs to know about game networking, 2010. URL http://gafferongames.com/networking-for-game-programmers/ what-every-programmer-needs-to-know-about-game-networking/. Accessed: 2017- 04-21.

58 [81] Microsoft. Shared holographic experiences, 2017. URL https://developer.microsoft. com/en-us/windows/mixed-reality/shared_holographic_experiences. Acessed: 2017-04-24.

[82] Henry Chen, Austin S. Lee, Mark Swift, and John C. Tang. 3d collaboration method over HoloLens™ and skype™ end points. In Proceedings of the 3rd International Workshop on Immersive Media Experiences - ImmersiveME '15. ACM Press, 2015. doi: 10.1145/2814347. 2814350.

[83] Road to VR. Meta 2 ar glasses - collaboration demonstration, March 2016. URL https://www. youtube.com/watch?v=7PZN-r_zGek. YouTube [Video ﬁle, 9:28] Accessed: 2017-04-24.

[84] Danakorn Nincarean Eh Phon, Mohamad Bilal Ali, and Noor Dayana Abd Halim. Collaborative augmented reality in education: A review. In 2014 International Conference on Teaching and Learning in Computing and Engineering. IEEE, apr 2014. doi: 10.1109/latice.2014.23.

[85] Samuel A. Miller, Noah J. Misch, and Aaron J. Dalton. Low-cost, portable, multi-wall virtual reality. In Proceedings of the 11th Eurographics Conference on Virtual Environments, EGVE’05, pages 9–14, Aire-la-Ville, Switzerland, Switzerland, 2005. Eurographics Association. ISBN 3-905673-21-5. doi: 10.2312/EGVE/IPT_EGVE2005/009-014.

[86] Muhanna A. Muhanna. Virtual reality and the CAVE: Taxonomy, interaction challenges and research directions. Journal of King Saud University - Computer and Information Sciences, 27 (3):344–361, jul 2015. doi: 10.1016/j.jksuci.2014.03.023.

[87] Siddhesh Manjrekar, Shubhrika Sandilya, Deesha Bhosale, Sravanthi Kanchi, Adwait Pitkar, and Mayur Gondhalekar. CAVE: An emerging immersive technology – a review. In 2014 UKSim- AMSS 16th International Conference on Computer Modelling and Simulation. IEEE, mar 2014. doi: 10.1109/uksim.2014.20.

[88] Oliver Kreylos. Hacking the oculus rift dk2, part ii, 2014. URL http://doc-ok.org/?p= 1124. Accessed: 2017-04-25.

[89] AMD Advanced Micro Devices. A path to truly immersive virtual reality (vr), July 2015. URL http://developer.amd.com/wordpress/media/2015/07/ A-path-to-truly-immersive-VR_Final_legally_approved.jpg. Accessed: 2017-06- 06.

[90] Oliver Kreylos. Optical properties of current vr hmds, 2016. URL http://doc-ok.org/?p= 1414. Accessed: 2017-06-16.

[91] Techradar. This is how microsoft’s hololens will address its biggest ﬂaw, 2016. URL http://www.techradar.com/news/wearables/ this-is-how-microsoft-s-hololens-will-address-its-biggest-flaw-1322596. Accessed: 2017-04-24.

[92] VBandi. Hololens vs meta 2, 2016. URL https://vbandi.net/2016/03/04/ hololens-vs-meta-2/. Accessed: 2017-04-24.

[93] Oliver Kreylos. Hololens and ﬁeld of view in augmented reality, 2015. URL http://doc-ok. org/?p=1274. Accessed: 2017-04-26.

59 [94] Wareable. Meta 2 ﬁrst impressions: Ar feels closer than ever, 2016. URL https://www. wareable.com/ar/meta-2-review. Accessed: 2017-04-24.

[95] Ming Li, Katrin Arning, Luisa Vervier, Martina Zieﬂe, and Leif Kobbelt. Inﬂuence of temporal delay and display update rate in an augmented reality application scenario. In Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia - MUM '15. ACM Press, 2015. doi: 10.1145/2836041.2836070.

[96] Takayuki Miura, Akihito Yoshii, and Tatsuo Nakajima. Designing aﬀordances for virtual reality-based services with natural user interaction. In Design, User Experience, and Usability: Technological Contexts, pages 266–277. Springer International Publishing, 2016. doi: 10.1007/ 978-3-319-40406-6_25.

[97] Tobii Technology. Towards immersive virtual reality - why eye tracking is the natural next step for vr, 2016. URL https://www.tobii.com/siteassets/tobii-tech/vr/ tobii-whitepaper-eye-tracking-next-natural-step-for-vr.pdf/?v=1. Accessed: 2017-06-09.

[98] Tobii Technology. Tobii eyetracking - an introduction to eye tracking and tobii eye trackers, 2010. URL http://www.acuity-ets.com/downloads/Tobii%20Eye%20Tracking% 20Introduction%20Whitepaper.pdf. Accessed: 2017-06-09.

[99] Heartwood3d. Virtual reality (vr) operations & maintenance training, November 2016. URL https://www.youtube.com/watch?v=4NCo9rRUniU. YouTube [Video ﬁle, 3:41] Accessed: 2017-04-21.

[100] MechaTraining LLC. Virtual reality lathe by mechatraining llc, March 2017. URL https: //www.youtube.com/watch?v=HRBuSwkiv7o. YouTube [Video ﬁle, 3:10] Accessed: 2017- 04-22.

[101] MechaTraining LLC. Virtual reality motor maintenance by mechatraining llc, February 2017. URL https://www.youtube.com/watch?v=dq2RSlslQcU. YouTube [Video ﬁle, 4:18] Accessed: 2017-04-22.

[102] Tengo Interactive. Tengo interactive vr training solutions for russian railways, October 2016. URL https://www.youtube.com/watch?v=_10mU0NES5U. YouTube [Video ﬁle, 3:23] Accessed: 2017-05-21.

[103] Hewlett Packard (HP). Vr room-scale setup – htc vive, 2017. URL http://www8.hp.com/ h20195/v2/GetPDF.aspx/4AA6-9648ENW.pdf. Accessed: 2017-06-16.

[104] Unity. Unity user manual (5.5), 2017. URL https://docs.unity3d.com/550/ Documentation/Manual/UnityManual.html. Accessed: 2017-05-22.

[105] Blender, 2017. URL https://www.blender.org/. Accessed: 2017-06-02.

[106] fs-freeware.net, 2017. URL https://www.fs-freeware.net/. Accessed: 2017-05-25.

[107] SceneryDesign.org. Model converter x, 2017. URL https://www.scenerydesign.org/ modelconverterx/. Accessed: 2017-05-25.

60 [108] 3DWarehouse. 3dwarehouse webpage, 2017. URL https://3dwarehouse.sketchup.com/. Accessed: 2017-05-25.

[109] Erik Fagerholt and Magnus Lorentzon. Beyond the hud-user interfaces for increased player immersion in fps games, 2009.

[110] Explorations in vr, 2017. URL https://developer.leapmotion.com/explorations#110. Accessed: 2017-06-10.

61 A Setting up Unity with HTC Vive and Leap Motion

Windows

1. Download

• Unity 5.5.0 from http://unity3d.com/get-unity/download/archive

• Leap Motion Orion from http://developer.leapmotion.com/get-started

• Leap Motion Unity Core Assets from http://developer.leapmotion.com/unity

• Leap Motion Attachments Module

• Leap Motion UI Input Module

2. Install Unity and Leap Motion Orion

3. Start Unity, create a new 3D project

4. Import the downloaded Leap Motion packages

5. In Asset Store, download and import SteamVR plugin

6. In Project window, navigate to Assets/LeapMotion/ and drag Leap_Hands_Demo_VR into the scene

7. Delete the default Untitled Scene

8. In the scene, select object LMHeadMountedRig → CenterEyeAnchor, go to the Inspector window and set the following component values:

• Camera component

– Clipping Planes: (Near: 0.05 - Far: 1000)

– Depth: -1

– Field of View: 60

9. Additionally on the CenterEyeAnchor object, add the script SteamVR_Camera

10. In the scene, select LMHeadMountedRig → HandModels. In Project window, navigate to Assets/LeapMotionModules/Attachments/Prefabs/

11. Drag the HandAttachments prefabs to the Handmodels object in the scene

12. In the scene, select LMHeadMountedRig → CenterEyeAnchor → LeapSpace → LeapHandController and, in the inspector window, increase the Hand Pool Model size by 1

13. Change the new Group Name that appeared to "Attachments_Hands" or something else more appropriate

14. Additionally, assign the new HandAttachments hands into the Left and Right model references

15. In Project window, navigate to Assets/LeapMotionModules/UIInput/Prefabs/ and drag LeapEventSystem into the scene

62 16. In the scene, select LeapEventSystem and in the Inspector window, under the Leap Input Module component, assign the only LeapHandController that exists in the project

17. Save the scene as a new scene and the setup is done. The project should now look like ﬁgure A.1 shows.

Figure A.1: View of ﬁnished project setup of VR camera and Leap Motion modules

63 B Prototype System Diagram Figure B.1: Diagram showing an overview of the used devices and the process of the prototype

64 C Demonstration of Implementation

C.1 Menu Interaction

Clicking on the image in this section will open a web browser playing a video. The video shows the interaction with the menu in real-time while using the HTC Vive and Leap Motion. To the top-left of the video, is another video showing one of the authors performing the actions. Further information about each feature is described in section 4.2.2.3 on page 45. (Keep in mind when viewing the video, that watching on a ﬂat screen gives a diﬀerent experience than using an HMD)

C.2 Refuel Procedure

The image in this section opens a web browser playing a video of the refuel procedure training by using HTC Vive and Leap Motion. The steps are described in detail in Table 4.1 on page 40. To the top-left of the video, one of the authors performs the actions that are needed for the steps and to the top-right, there is a top-down view of the room and the tracked object in it, which shows the HMD, to see how it moves around as the author does. (Keep in mind when viewing the video, that watching on a ﬂat screen gives a diﬀerent experience than using an HMD)