Standard Operating Procedure for Augmented Reality Operations Using Microsoft HoloLens Prepared for: Ohio UAS Center Prepared by: University of Cincinnati Date: March 24, 2020 Version Number: 2.0 Table of Contents List of Figures ..................................................................................................................................3 1.0 Introduction ................................................................................................................................4 2.0 What is Microsoft HoloLens? ....................................................................................................5 3.0 Input Methods on HoloLens ......................................................................................................6 3.1 Gaze .......................................................................................................................................6 3.2 Gesture ...................................................................................................................................7 3.3 Voice ......................................................................................................................................8 4.0 Prerequisites ...............................................................................................................................8 5.0 How to change the 3D model in the scene? .............................................................................10 6.0 Common Operating Platform ...................................................................................................14 7.0 How does HoloLens work with Common Operating Platform? ..............................................15 8.0 How to deploy your application from Unity to HoloLens? .....................................................19 9.0 How to create Shared Session between two or more HoloLens? ............................................23 References ......................................................................................................................................26 2 | Page List of Figures Figure 1 – A three-step process for 3D model generation .............................................................. 4 Figure 2 – Exploded view of HoloLens[1] ..................................................................................... 5 Figure 3 – Example showing cursor hugging a 3D model .............................................................. 6 Figure 4 – Air tap gesture [2] .......................................................................................................... 7 Figure 5 – Bloom Gesture[3] .......................................................................................................... 7 Figure 6 – Bloom gesture opens the system menu as shown in the figure below. [4] .................... 8 Figure 7 – Adding 3D model to Unity Scene ............................................................................... 10 Figure 8 – Newly added 3D model preview in the Unity scene ................................................... 11 Figure 9 – Model without mesh colliders ..................................................................................... 12 Figure 10 – 3D model with mesh colliders ................................................................................... 12 Figure 11 – Adding the name of the model in the scripts for markers to work and for HoloLens connection on server. .................................................................................................................... 13 Figure 12 – Placeholder to add a model in the script for UI buttons and slider to work. ............. 13 Figure 13 – Common Operating Platform .................................................................................... 14 Figure 14 – HoloLens connectivity with Common Operating Platform....................................... 15 Figure 15 – 3D model request from HoloLens to Common Operating Platform ......................... 16 Figure 16 – 3D model request and response between HoloLens and Common Operating Platform......................................................................................................................................... 16 Figure 17 – Representation of shared session connectivity between HoloLens ........................... 17 Figure 18 – User interface for interacting with the 3D model ...................................................... 18 Figure 19 – User’s avatar in the virtual space of a shared session ............................................... 18 Figure 20 – Setting IP address of Server ...................................................................................... 19 Figure 21 – Steps for HoloLens application deployment ............................................................. 19 Figure 22 – Build application in Unity ......................................................................................... 20 Figure 23 – Application build status in Unity ............................................................................... 21 Figure 24 – Deployment settings in visual studio ......................................................................... 22 Figure 25 – Wi-Fi connectivity check on HoloLens ..................................................................... 23 Figure 26 – Sharing service execution – new session initiation ................................................... 24 Figure 27 – Sharing service – connected users preview ............................................................... 24 3 | Page 1.0 Introduction Since the beginning of content creation, the visualization platform has evolved. Artists created 3D data but was viewed on a 2D paper and then on a 2D screen. In the last two decades, virtual reality has gained huge attention from major IT giants like Microsoft, Facebook, Apple and has become the dominant tool for exploring and visualizing 3D content. After a thorough literature review on existing AR/VR technologies, it was decided to use HoloLens for the project as it is far ahead of the competition and unlike HoloLens, others are not completely wireless, do not run autonomously on an operating system, need continuous HDMI connection, remote controllers for inputs, have poor battery backup. We primarily use HoloLens as a visualization tool for the 3D model. And the way these models are created is a 3-step process. Figure 1 – A three-step process for 3D model generation Step 1: A UAV flies a planned mission over a site and captures overlapping images from multiple angles. These images are uploaded to the UCII common operating platform (web server) Step 2: These images are then stitched together to generate the 3D model using professional photogrammetry software. Step 3: The 3D model is then available on web server. A 3D model is downloaded from webserver and then deployed on HoloLens at compile-time. The users could create the shared user experience, where the user may or may not be in same physical location but will always be in same virtual space and be able to interact with the model. During the shared session, users will be able to talk to each other over voice chat using VoIP technology, look at each other’s avatar while being able to manipulate the 3D model with respect to scale and position, and the synchronize these changes among everyone in the 3D conference with a single button. Users can interact by dropping notes, markers on the models which will be visible to other users at runtime. Users also have provision to mute their mic and toggle on and off for dropping markers. 4 | Page 2.0 What is Microsoft HoloLens? HoloLens is the flagship augmented reality headset from Microsoft. It was launched in early 2016 and has a heads-up display. This headset is way ahead of other competitors in this field. Unlike few of the other headsets which need their sensors mounted in the corners of the room, HoloLens has all the sensors mounted on the device as seen in the Figure 2 below. Figure 2 – Exploded view of HoloLens[1] • Four Microphones are used for giving voice commands to Cortana and for dictation. It can also be used to create custom voice commands in applications for any type of action. • Ambient light sensor helps in auto adjusting the brightness of the display. When the room is brightly lit, the brightness of holograms will be high, so the user can see it clearly and when the room is dim (poorly lit), the brightness of holograms will be low to reduce stress on eyes. • IMU (Inertia Measurement Unit) helps in tracking the user’s head movement. This helps in rendering user’s avatar in shared session which will rotate and reposition with user’s movement in the virtual space. • The environment understanding cameras and the depth camera work together to understand user’s surrounding and generate a 3D map of it, so it can simultaneously locate and map user’s location. • The front camera is a simple 2-megapixel camera capable of capturing mixed reality, so it sees what a user is seeing. We can take pictures, videos and even live stream what the user is looking at. • HoloLens runs on the Windows 10 operating system and does not need HDMI connection a computer system for functioning. This means that the HoloLens can run almost all the Universal Windows Platform (UWP) apps in the Store as 2D apps. UWP is a developer’s 5 | Page platform introduced with Windows 10 that runs applications on all devices like Desktop, Mobile, Xbox, and HoloLens as well as future Windows devices Moreover, HoloLens has