Naval Shipbuilding & Advanced

Manufacturing Center

Safe Employment of Augmented Reality in a Production Environment Market Research Deliverable #2

Q2804 N00014-14-D-0377

Prepared by: Scott Truitt, Project Manager, ATI Submitted by: ______06/13/2019 ______Signature Date Name, Title, Organization

Page 1 of 45

Table of Contents Introduction ...... 5 Purpose...... 5 1. Wearables & Mobile Devices for Augmented Reality (Lead: General Dynamics- Electric Boat) ...... 5 Background ...... 5 AR Registration Defined ...... 6 2D Markers ...... 6 3D Markers ...... 8 Planar Registration ...... 9 Control Point Registration ...... 10 AR Registration for Shipbuilding ...... 12 Registration Support ...... 12 Analysis ...... 13 Table Tabs ...... 14 Manufacturer Data Collection ...... 14 Hardware Data Collection ...... 15 Software Data Collection ...... 16 Wearable & Mobile Devices for Augment Reality Conclusion ...... 17 2. Safety Requirements (Lead: Boeing) ...... 18 Background ...... 18 Situational Awareness ...... 18 Standardization of Risk Assessment ...... 19 Fitness Evaluation ...... 20 Down selection Process ...... 21 Hardware Considerations ...... 21 Microsoft Hololens ...... 22 Google Glass ...... 22 Tablets ...... 22 Smartphones ...... 22 Software Considerations ...... 22 3. Security Considerations (Lead: Huntington-Ingalls Industries, Ingalls Shipbuilding) 23 Background ...... 23

Page 2 of 45

Bluetooth ...... 23 Bluetooth v3.0 Vulnerabilities ...... 23 Bluetooth LE & v4.0 ...... 24 Wi-Fi ...... 24 Data Interception ...... 24 Denial of Service (DoS) ...... 24 Rogue AP’s ...... 24 Wireless Intruders ...... 24 Misconfigured AP’s ...... 25 Ad Hocs and Soft AP’s ...... 25 Endpoint Attacks ...... 25 Evil Twin AP’s ...... 25 Wi-Fi 802.11ac ...... 25 Operating Systems ...... 26 Windows 10 ...... 26 Linux ...... 26 Blue Cat ...... 27 iOS ...... 28 Android ...... 28 Software Development Kits (SDK) ...... 29 DeepAR SDK ...... 29 EasyAR SDK ...... 30 ARCore SDK ...... 31 Onirix SDK ...... 32 Pikkart AR SDK ...... 33 Scope AR SDK ...... 34 4. User Interface (Lead: Huntington-Ingalls Industries, Newport News Shipbuilding) .. 35 Background ...... 35 Analysis ...... 37 Hololens, Microsoft ...... 37 Hololens 2, Microsoft ...... 38 Magic Leap One, Magic Leap ...... 39 Daqri Enterprise Smart Glasses, Daqri ...... 40

Page 3 of 45

Epson Moviero BT 2000 ...... 40 Leap Motion, Ultrahaptics LTD ...... 41 LAFORGE Optical ...... 41 Vuzix Blade AR Smartglasses, Vuzix ...... 41 Dreamworld Dreamglass, Dreamworld AR ...... 42 Google Glass Enterprise, Google ...... 42 References ...... 44

Page 4 of 45

Introduction The Office of Naval Research, Navy Manufacturing Technology (ONR Navy ManTech) program and its Naval Shipbuilding and Advanced Manufacturing Center of Excellence (NSAM) seek to collaborate with a cross-functional team of US Defense Contractors to perform a technology baseline study on the state of Augmented Reality (AR) technologies. The study will also determine the leading technology providers in the following technical areas:  Wearables & Mobile Devices for Augmented Reality  Wearables & Mobile Devices for Mixed Reality  Safety Requirements  Security Considerations  User Interfaces The Safe Employment of Augmented Reality in a Production Environment project will investigate each of these technology areas. This research will determine the current state of the market and determine how the technology fits within the context of production facilities in the Defense industry. The outcome of this analysis is to provide market research, hardware and software requirements, infrastructure assessment, and identify gaps between as-is and to-be. This analysis will also provide an implementation roadmap and identify the best practices to safely implement this technology and identify where this technology can potentially save cost and provide benefits to the Defense industry. Purpose The purpose of the Market Research effort was to research the current state of the market’s capabilities, requirements, security considerations, and user interface considerations. The focus is on commercial-off-the-shelf (COTS) technologies that will integrate into a production environment with little to no further development. 1. Wearables & Mobile Devices for Augmented Reality (Lead: General Dynamics-Electric Boat) Background The terms Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), and X- reality (XR) are encountered everywhere today. It is fitting perhaps to start with definitions. For our purposes the two important modes are AR and MR. Augmented Reality and Mixed Reality are often use interchangeably when describing both 2D/3D data displayed in front of the real world and 2D/3D data displayed and registered within the real world. AR and MR was broken up between two teams for research purposes but after some preliminary research and collaboration between to two teams, it was determined that most, if not all, AR devices support both in one form or another. As an example, user interfaces and menus to navigate an AR

Page 5 of 45

application will use 2D data displayed in front of the real world but it does not require it to be registered. Therefore, this would allow for the usage of the same device for either AR or MR solutions. For this reason, both market research efforts will merge into one section. AR Registration Defined The core feature of an AR application is its ability to track the motion of the camera as it moves through space. This movement tracking allows the ability to “register” or “mix” virtual objects to the real world. This section describes some current approaches to registration in augmented reality (AR) applications and where relevant it describes their applicability and their use to date on this effort. The position and orientation of the camera at a particular time, is its “pose” and the graphics rendering system uses it to draw holographic objects such that they appear in a consistent position relative to the real world video image. The pose is usually updated once per video frame (typically 1/30th of a second) so stationary holograms appear fixed in physical space. The pose at time tx, is defined as the position and orientation relative to some initial pose in the past, which may just be whatever the pose happened to at time t0. The pose at time t0 relative to the physical world is not important for rendering holograms that appear fixed in space but it is required to render the holograms at a particular place in the physical world. Registration in an AR application is the process of aligning the physical world as seen by the AR camera with the virtual world as defined in its own coordinate space. The goal of registration is to be able to render virtual holographic objects into the live video stream such that they appear in the appropriate place relative to real-world objects. Additionally these holograms must appear the correct size and orientation relative to the real world as viewed through the AR camera and rendered on the AR screen(s). The three components of the registration (rotation around each of the three axes, scaling, and translation) define five 4x4 matrices that are combined into a single 4x4 matrix which is used to convert between the coordinate system viewed by the AR system and the virtual coordinate system in which the holograms are defined. 10 0 0𝑐𝑜𝑠∅𝑦 0 𝑠𝑖𝑛∅𝑦 0 𝑐𝑜𝑠∅𝑧 𝑠𝑖𝑛∅𝑧 00𝑆𝑥 000100𝑇𝑥 0 𝑐𝑜𝑠∅𝑥 𝑠𝑖𝑛∅𝑥 0 0100𝑠𝑖𝑛∅𝑧 𝑐𝑜𝑠∅𝑧 00 0 𝑆𝑦 00010𝑇𝑦 0 𝑠𝑖𝑛∅𝑥 𝑐𝑜𝑠∅𝑥 0 𝑠𝑖𝑛∅𝑦 0 𝑐𝑜𝑠∅𝑦 0 001000𝑆𝑧 0 001𝑇𝑧 00 0 1 000100010001000 1 Figure 1 Transformation matrices for rotation about the X axis, rotation about the Y axis, rotation about the Z axis, scaling, and translation. The following sections describe some current approaches to computing the registration transformation matrix and their uses in recent work. 2D Markers A known image placed at a known location can be recognized by the visual processing software and used to compute the rotation, translation, and scaling of AR pose relative to the real-world coordinate system. Any image that is reliably recognized and is unambiguous in rotation can determine the AR pose. These

Page 6 of 45

markers are “fiducial” markers. The OpenCV computer vision library includes routines to detect and track special fiducial markers called “ArUco markers” which are specially designed to be robust.

Figure 2 ArUco Marker Examples The spARky NSRP project used this approach to align holographic wires and connectors within a physical wiring cabinet. So the rendered wires weren’t obscured the spARky ArUco fiducial markers were placed on the outside corners of the wiring cabinet.

Figure 3 spARky NSRP demo More recently, AR toolkits are now supporting arbitrary images as 2D fiducial markers, enabling application developers to integrate the marker into the graphic design of their product.

Page 7 of 45

Figure 4 Vuforia example showing a hologram rendered using a playing card as a 2D fiducial marker 2D marker-based registration is mature and robust and enables efficient and accurate registration but it requires the presence of the marker in any AR scene before the holograms render. In most cases, the marker must remain visible in the camera image or the holograms will disappear. 3D Markers Some AR platforms can use three-dimensional objects as registration markers and render the AR scene relative to the object in view. This capability is less mature and before used in a scene, it typically requires pre-processing the 3D model. Some of these pre-processing steps require uploading images of the object to be recognized to the vendor’s server and downloading the resulting digested tracking BLOB that can be loaded into the toolkit. This reliance on external cloud-based services makes that approach not applicable for DoD shipbuilding.

Page 8 of 45

Figure 5 Wikitude Object Recognition and Tracking Generating the object tracking model requires a set of images of the object taken from various angles. While, in theory, one can generate images directly from a 3D model using a rendering program like Blender, for example, in practice I have not been able to make that work. I have only had success using tracking data generated using actual pictures of physical objects. As with image markers, when the 3D model is no longer in view, the AR scene disappears. The AR toolkits that we have surveyed that support 3D object tracking are set up for tabletop-scale AR scenes. I have had no success using them for room-scale AR scenes. Planar Registration The ARgos team developed planar registration during the first phase of the ARgos NSRP project. It uses four planes in the CAD model and the four corresponding planes measured in the world to compute the rotation and then translation that overlays the CAD planes onto the world planes. Planar registration can be broken into two operations: 1) finding the association between corresponding planes in the CAD and in the world; and 2) solving for the rotation and translation needed to map the corresponding planes onto the same orientation. For ARgos phase 1, we chose to have the user make the correspondences between the CAD planes and the world planes. Then the software computes the transform between the two sets of planes. We explored ways to automatically determine the planar correspondences (step 1) but did not complete that work in ARgos.

Page 9 of 45

Figure 6 ARgos CAD Planes and Measured (World) Planes This registration method is less mature than image marker-based registration, but can be robust if the operator chooses planes that are conducive to registration, i.e.:  They are not coplanar  They are big enough to be accurately measured by the depth sensor (about 8 inches square for Tango)  They are offset from one another by as much as possible. An advantage of this approach is that it automatically corrects for compartments that are not plumb with respect to gravity. For example, a compartment that is on its side or upside down would register exactly the same way as one that is right side up. Disadvantages of this approach are:  It requires four suitable flat surfaces to be present in the space. In early construction, this may not be possible.  It requires a device that can accurately measure planar surfaces. Current monocular devices cannot accurately measure arbitrary planes, so a depth camera or a stereo camera is required. Control Point Registration This registration method (also known as the “Fast” registration method) uses accuracy control points that are measured into all work areas early in the construction process. Workers mark points measured from ships reference lines and write the offset of those points next to the points on bulkheads, floors, frame members, etc...

Page 10 of 45

Points are measured in three dimensions:  Forward/Aft is measured from nearby frames, which are at known offsets from the X dimension reference point – the front of the boat.  Port/Starboard is measured from the main axis which is the centerline of the boat.  Above Main Axis/Below Main Axis is measured from the main axis which is the centerline of the boat. This is often abbreviated “AMA” or “BMA.”

Figure 7 Example Accuracy Control Point Showing a Point Eight Feet Starboard of the Ship's Main Axis The registration happens in four steps. The first step, establishing the rotation, is done first to avoid mathematically correct but very counterintuitive behavior. The other three steps, setting the translation in each of the three dimensions, can be done in any order. 1) The user points the device and selects one of the cardinal horizontal axes (forward, aft, port, or starboard). This sets the rotational component of the transform. The Tango device orients its axes such that the Y axis is always aligned with geometry, so the software can assume that the rotation set in this step is only rotation around the Y axis. That is, the Euler angles of rotation is always of the form (0, θ, 0) for rotation θ around the Y axis. Once this step is complete, if reference geometry is shown, it will now be shown in its correct alignment. 2) The user positions the device to align a crosshair marker with the accuracy control point establishing the forward/aft offset. The user enters the offset from a nearby frame, selects “mark,” and the device records the offset in the ship’s X dimension. After this step, if reference geometry is shown, it will now appear in the correct forward/aft position. 3) The user positions the device to align a crosshair marker with the accuracy control point establishing the port/starboard offset. The user enters the offset port or starboard of the main axis, selects “mark,” and the device records the offset in the ship’s Y dimension. If reference geometry is shown, it will now appear in the correct port/starboard position.

Page 11 of 45

4) The user positions the device to align a crosshair marker with the accuracy control point establishing the AMA/BMA offset. The user enters the offset above or below the main axis, selects “mark,” and the device records the offset in the ship’s Z dimension. If reference geometry is shown, it will now appear in the correct vertical position. AR Registration for Shipbuilding Some shipbuilding companies use marker-based registration with fiducials. At Electric Boat (EB), the approach has always been toward marker-less tracking. Accordingly, EB developed the Fast Registration approach, which takes advantage of the fact that its ships are constructed in modular fashion. After the initial fabrication and assembly phases, most construction work moves to a heavy assembly or outfitting stage, in which the work revolves around marks that are precisely set to indicate the position of the three principal planes of the virtual ship’s grid. The Fast approach is semi-automated with the user aligning the AR device visually with each mark. An initial registration is calculated, but it can be ‘nudged’ into better alignment by the user. You can save the localization model to re-use in the same module for later AR sessions. This is handy, but, in fact, the Fast registration takes less than five minutes to do. The Fast registration aligns to about a one-inch accuracy, which is sufficient to support the so-called ‘approximate’ AR use cases: finding the part numbers of components (even studs) based on their location, identifying off-location items, and the like. Registration Support

Name Platforms License Support for Registration Vuforia Android, iOS, Commercial 2D Images, 3D Objects Hololens Diota Proprietary, Commercial No public technical details available Hololens, Others? Hololens Windows 10 Commercial Can perform spatial mapping which could be used Hololens with planar or object‐based registration. Supports registration persistence through spatial anchors. OpenCV Android, iOS, Open Object recognition through Haar Cascade Classifier. Windows, Source Algorithms for implementing surface matching and Linux plane detection. ORB‐SLAM2 Android, iOS, Open Includes OpenCV. Map relocalization supported Windows, Source with through a patch. Linux commercial license NNS System Presumably Proprietary 2D markers. Built on Vuforia. same as Vuforia

Page 12 of 45

ARToolkit IRIX, Linux, Open 2D markers. Last release was in 2007. Appears MacOS, source with unmaintained. Windows commercial license DAQRI Proprietary Commercial 2D markers. Possibly others. Public technical details helmet using limited Intel RealSense Wikitude Android, iOS, Commercial 2D markers, 3D object tracking, plane detection Windows UWP Inglobe Android, Commercial 2D and 3D markers. Plane detection. API reports Technologies OSX(soon), camera orientation as latitude, longitude, altitude and iOS, Windows pose. Not clear how they do that. Uses OpenCV (soon) feature detector. BuildAR Windows Commercial 2D markers and images (now Envisage AR) Catchoom Mobile and Commercial 2D images. Tailored for adding information to web apps shopping experience. NGRAIN Windows, Commercial Focused mostly on scanning for damage assessment. Android No technical information available. Recently acquired. EON Reality Hololens, Commercial 2D image markers, others? Android, iOS Bitstars Android, iOS Commercial Google street view for construction sites. Manual correspondence between blueprints and 360‐degree camera. Fraunhofer Windows, Commercial Last release mid‐2016. Appears unmaintained. IGD Instant OSX, Linux but free Reality Kudan SLAM C++14 Commercial Map relocalization (persistence). Planefinding. portable MaxST Windows, Commercial 2D Markers, 3D Object tracker Mac, Android, iOS ARKit iOS Commercial 2D Markers, 3D Object tracker, AR persistence but free ARCore Android, iOS Commercial Plane detection. Limited sharing of coordinate systems but free through cloud anchors. Placenote iOS Commercial AR Persistence. Built on ARKit. 6D iOS Commercial AR Persistence Table 1 Registration Support

Analysis The first action for the Augmented Reality (AR) team was to agree on a structure to guide research, and to record research findings. The agreed upon structure

Page 13 of 45

research documentation can be seen as Attachment (A) “AR_MATRIX_V1.9” (inserted below).

Attachment_A_AR_ MATRIX_V1.9.xlsx We agreed upon a simple spreadsheet with several tabs for focus areas. Given the dynamic and somewhat chaotic nature of the current AR market, the team opted for easily maintainable documentation as project members learn more over time. Table Tabs The AR_MATRIX has four tabs, three of which are active and the fourth is on reserve for potential future use. The first tab “Manufacturer Data” captures business and demographic data about manufacturers of AR hardware, software and supplemental sensor suites that is relevant to determining the state of the market. The second tab “Hardware Data” captures technical specifics describing AR devices including hand held and head mounted display and glasses form factors. The third tab “Software Data” captures relevant information about AR software offerings including Applications, Software Development Kits and Development Environments. Tabs 1 through 3 are active and Attachment (A) provides the data recorded. Tab 4 is “Performance Test Results” and is a placeholder to record the results of standardized benchmarking tests for AR Solutions. AR Solutions will capture combinations of hardware and software that comprise actual AR tools for the shipyard.

Manufacturer Data Collection The Manufacturer Data tab tracks a small set of information that is (A) easily obtained and (B) useful to determine the relative stability and pedigree of the producers of AR hardware and software. Every entry in the Hardware Data and Software Data tabs should have a corresponding entry in this tab recording information about the manufacturer. This information is relevant to determining the state of the market because many AR players are small start-ups that may not have a lasting presence, or be in countries that may represent a notably high security risk for Department of Defense work. Factors that were determined to be relevant to the research are:

 Place of Incorporation (Relevant to Security)  Headquarters Location (Relevant to Security)  Number of Employees (Relevant to Stability)  Date of Founding (Relevant to Stability)  Type of Company (Relevant to Stability)  Company Parent (Relevant to Stability)

Page 14 of 45

Hardware Data Collection The Hardware Data tab tracks a larger set of information that describes the important form factor and functional characteristics of available AR devices and supplemental sensor suites. Although more data could be harvested on the AR devices, the team felt that the factors identified in Attachment (A) are the minimally viable set that provides guidance to the shipbuilding community. Factors that were determined to be relevant to the research are listed below.

 Weight: Highly relevant to the ergonomic viability of wearable AR solutions.  Optics: There is a wide disparity in the type of optic technologies used, field of view and size. Any of these solutions may be a consideration on satisfying specific use cases.  Sensor Suite: The type and capability of the device sensors is a critical determinant for AR device registration, localization and tracking capability.  Human Interaction: Highly relevant to the user experience viability of wearable AR solutions.  Input/Output: Interoperability is a key factor in how well the AR device can be integrated with the larger industrial enterprise.  Battery Life: Highly relevant to the ergonomic viability of wearable AR solutions.  Processors: Processing power is a good indicator of device capability, but also a good indicator of how hard it might be to make the device ergonomically acceptable. Nobody wants to wear a super computer on their head.  Memory: May be important depending on how the device is connected to the enterprise infrastructure and whether the use case demands stand-alone capability.  Operating System: This is one of the current fault lines in selecting AR devices to ensure compatibility with the enterprise infrastructure as well as the workforce.  Safety Compliance: The AR market is largely defined by consumer market demand and is only recently considering industrial hardening and OSHA concerns.  Tethered Form Factor: An optional hardware architecture that may be of use to industrial and/or high power users.  Unit Cost: There are many reasons to be concerned about unit cost, not the least is the likelihood that wide usage of AR in an industrial environment will make the devices expendable.

Page 15 of 45

Software Data Collection The Software data tab tracks characteristics of AR software at three levels of the AR technology stack: Integrated Development Environments (IDE), Software Development Kits (SDK) and Applications. The boundaries between the three levels are fairly fluid but the team felt that is valuable to separate them. Factors that were determined to be relevant to the research are listed below. It should be noted that not all factors apply to all categories. For example, Distance Capture Range is characteristic of SDKs but not IDEs or Applications.

 Distance Capture Range: The distance that augmented reality SDKs can operate within. This is important for understanding what specific use cases this SDK will support.  2D Recognition: This SDK characteristic can be useful both for providing intelligent overlays on 2D objects in the scene, as well as providing additional tracking elements for registration and drift reduction.  3D Recognition: Similar to 2D Recognition, enhanced with recognition from multiple angles.  Geo-Location: Capability enables an AR app to understand specifically where it is in the world coordinate system vs a local coordinate system.  Cloud Recognition: Utilizing machine learning the SDK has the capability to store a large database of 2D and 3D images in the cloud to assist with localization and scene registration. This is a method of storing data to support 2D & 3D Recognition.  SLAM: Simultaneous Localization and Modeling is a localization methodology used by all spatially aware SDK’s.  Shared Spaces: The ability to share registered spaces between devices enabling multiple users/devices to operate in shared AR space.  Sales Model: Whether application software, in particular, is purchased, “leased” on a subscription basis or open source is an important business consideration.  Unit Cost: There are many reasons to be concerned about unit cost, particularly with application software that may multiply quickly as AR tools take hold.  Version: As with all software, the versioning is as important as identifying the software, itself.

In addition to the earlier recommendation of continuous maintenance of the Attachment (A) table, AR Team recommends expanding the research to include the factors:

 Plane Detection: The ability to detect planar surfaces in the scene. As of this writing a limitation of the common ARKit and ARcore SDKs is tracking of only

Page 16 of 45

horizontal and vertical surfaces. We anticipate that future upgrades and offerings will provide planar surface tracking at any angle.  Mesh Creation: This capability is an advanced feature that allows an SDK to generate a textured mesh that can be used to increase accuracy as well as 3D scan the existing session.  Advanced Lighting: This feature enables AR elements to appear more real. It is enabled by realistic physically-based materials, environment reflections, grounding shadows, cameras noise, motion blur and realistic lighting/shadows derived from current ambient light in the scene.  Occlusion: The ability for the SDK to naturally place holographic content into the scene and have it correctly appear in front of, or behind real objects.

Wearable & Mobile Devices for Augment Reality Conclusion A conclusion from the market research is confirmation that the market is still experiencing early stage development and significant churn. Though AR technologies have been emerging for quite a while, a dominant architecture for AR solutions has not yet emerged, particularly for industrial applications. AR software and hardware suppliers are developing solutions that still represent a broad diversity of options and approaches. Each approach offers important advantages over competitor solutions, but also brings inherent limitations. The market churn brings the probability of risk to any significant investment and deployment to the shipbuilding workforce. AR tools selected now will likely have a relatively short viable lifespan as the technology continues significant evolution. The payback for a substantial investment will have to be high to mitigate the potential risk of early obsolescence.

Another conclusion derived from the high rate of technology development is the current scarcity of turnkey AR solutions. Especially in the heavy industry space, early adopters will have to rely on custom software development, and possibly hardware. The next tasks in the Safe Employment of AR in Production Environment project will confirm, but the team speculates that ergonomics of hardware and accuracy of registration and tracking for software will require further development by implementing shipyards in the near future.

In addition to the two above conclusions, the AR Team offers three recommendations concerning the Attachment (A) table:

1. Maintenance: As stated earlier, given the high rate of development and churn in the AR Technology market, the team recommends that we find a way to maintain the data collected in the Task 2.1 research. We believe the reference is valuable to team members and will remain so if maintained to reflect market changes and updates.

Page 17 of 45

2. Expansion: Through the process of conducting the market survey we learned how to improve the results of data harvesting. Certainly the list of research topics, although representative is not definitive. The list can and should be expanded. Further, the number of attributes collected should be expanded to provide a more thorough summary of capabilities. Specific recommendations are listed in the Software data Collection section.

3. Conversion: Keeping the results of the market survey in a spreadsheet was a quick and flexible way to start the process of harvesting and recording information. Ultimately, if the table continues to expand in volume and complexity, we recommend a conversion from the current spreadsheet format to a simple database.

2. Safety Requirements (Lead: Boeing) Background The state-of-the-art in hardware and software for AR devices as it pertains to safety features will be covered in this document. However, as it relates to health and safety, much of the concern is more on the effects of prolonged usage of the devices, and less on the technical capabilities of the device. Three of the technology gaps that were identified by the industry partners as being primary obstacles to overcome are captured in this document. Situational Awareness Augmented reality devices can introduce the hazard of diminishing situational awareness of the operator. In production environments, it is crucial that an operator is mindful of trip hazards, fall hazards, and activities occurring around them such crane moves of large parts. In the case of AR wearables, projecting data onto an operator’s field of view can occlude this critical information. A current solution to this problem is to use the IMU, ego-SLAM or other sensors of the device to detect movement, and halt projection into the user’s field of view. While this may be an acceptable solution for some work cells, this may limit the use cases suitable for the device. Potential improvements is to use computer vision to detect hazards in the environment, and to halt projection of information in this area of the user’s field of view, and to highlight these risks, however this capability has not matured to a useable state to date. AR devices can also enhance situational awareness if done correctly. External sensors can stream data to the onboard device, which then analyzes the data for abnormal events, and relay this information to the operator. Some additional use cases are camera systems being used to capture poor ergonomic poses of the

Page 18 of 45

operator, detect fall and trip hazards, detect cut hazards, and then highlighting these areas for the operator even if they are not in the current field of view. With the idea of “smart factories” or smart manufacturing facilities becoming more of a reality, there introduces another avenue for augmented reality devices to enhance user situational awareness. For example, an operator can be notified when overhead work is occurring near his location so that they are aware. An operator can also be notified for any type of work cell specific events are occurring, or have this information projected into their view. An example of this within Boeing is a cell where operators work in conjunction with a robot drilling holes from the outside. As the workers cannot see the robot through the part, projecting where the robot is, and when it begins drilling allows the worker to operate more safely and more confidently with the outside robot. Standardization of Risk Assessment In order to scale across industry, it will be necessary to have a standardized method of performing risk assessment for AR usage. This assessment should be all- encompassing as to the types of hazards uniquely relevant to AR usage found in a work environment. From this risk assessment will be a mitigation plan, which will drive device requirements for the device manufacturer to meet. Hazard risk assessment must be done consistently and with the same criterion, it is just a matter of determining the right methodology for wearable technology. Standardized hazard risk assessment methodology allows for safety professionals to conduct risk ranking, evaluation and comparison using a predetermined set of parameters. In addition, standardized hazard risk assessment methodology helps reduce subjectivity, ambiguity and potential error, thus making the hazard risk assessment more accurate. A great example of a widely adopted risk assessment methodology is RIA TR R15.306-2016, which is one method for identifying and document task-based risks for automation safety requirements covered under ANSI/RIA R15.06-2012: Industrial Robot Safety.

Many private companies, including Boeing, have adopted “standard” methods such as these so that hazard risk assessments are done exactly the same regardless of the industry, geographical location or other potential factors. Standards are usually the product of leading edge industry work when unknown risks are identified, resolved and documented. It is only a matter of time before wearable technology will have identified a risk assessment methodology that is efficient, standardized and widely adopted.

A question that is worth asking is whether or not we have found the best hazard risk assessment methodology yet. To date, the hazard risk methodology of Boeing EHS professionals, called the HIDRA Method has been only semi-effective when applied to wearable technology. The HIDRA Method uses a two-factor multiplier system using Probability of Injury x Severity of Injury. These factors are rated on a scale of 1-5, with 5 being the highest rating for each factor. A score, on a scale of 1 to 25, is then provided for evaluation. A descriptive definition/criteria table is provided for

Page 19 of 45

EHS professionals to generally evaluate these numerical factors. A pre- and post- mitigation scoring is provided as well. This tool is great for a before and after shot of risk mitigation, however it does not do an effective job of tracking changing risk. In the trials that Boeing has done with wearable technology, EHS professionals had to go through an extensive trial-error-correction process to determine the appropriate risk mitigations until an acceptable risk level was achieved. It is recommended that future research look into how to track risk as it changes through the integration process into workplace environments such as the manufacturing facilities related to ONR and other government programs.

Fitness Evaluation Due to how recent augmented reality technology is, there is limited longitudinal data on the long-term effects of using these devices. From a health and safety perspective for manufacturing, this is critical as daily usage may exceed that typically seen in the commercial market, and may introduce liability to the company. In addition to understanding generally the long-term effects of prolonged use of AR devices is understanding the relationship between an individual’s unique state of health and the effects of AR devices. Similar to the topic of standardization of risk assessment, understanding and solving this problem will enable AR devices to efficiently scale across industry.

The Occupational Safety and Health Administration requires, under civilian labor law, medical screening and surveillance programs to protect employees from harmful risk in the work environment (OSHA). Medical screening is done to evaluate for job fitness, screen for occupational illness and provide individual treatment in cases where occupational illness is present. In addition, a comprehensive surveillance program allows for the collection and interpretation of data over time to facilitate the elimination and prevention of harmful risks through consultation with safety professionals. Well known harmful risk examples, where screening and prevention have led to improved occupational health practices, include asbestos, lead, and ergonomics.

These types of workplace programs will likely be necessary in the future because of the types of risks documented by hardware manufacturers. Hardware manufacturers, such as Microsoft, have minimally provided the necessary product health and safe use information for consumers of their technology (Microsoft Corporation, 2019). The severity of some of these risks is enough to warrant the typical safety professional to ask for more data when evaluating the risks posed to the individuals using them. These include acute risks such as electrical shock and interference with medical devices and chronic risks such as hearing loss and musculoskeletal disorders (Microsoft Corporation, 2019). While the information provided by hardware manufacturers notes the risks and necessary recommendations, it does not sufficiently provide guidance to medical practitioners or safety professionals for managing harmful risks. Medical screening and surveillance programs are a gold standard in the occupational health field as they help address both individual and population risks concurrently. It will only be a

Page 20 of 45

matter of time before medical screening and surveillance programs include wearable technology as addition.

With these ideas in mind, one challenge that is presented is how medical screening and surveillance will be conducted for wearable technology. Limited information from hardware manufacturers and the infancy of wearable technology adoption/adaptation for the workplace contributes to a limited research and data pool. The economic cost and appeal of wearable technology has already led to large scale use and adoption, however this has not allowed time for comprehensive research to be done on health risks on the individual and population scale. If you have a population of 30,000 employees, like Boeing does at its Everett, Washington plant, it is probable that trends will emerge for the overall population, with certain individuals in the population being more adversely affected as typically seen on an epidemiological bell curve. Partnership of private entities (ex. Boeing, ANSI) with public entities (ex. NIOSH, public universities) is one avenue in which to explore the development of medical screening and surveillance criteria. Collaboration to come up with the development of this criteria will not only help provide appropriate evaluation and risk mitigation information, but it will also help drive towards the development of effective labor regulations regarding wearable technology.

Down selection Process When evaluating resources to perform the technical work to bridge the technical gaps, several things should be considered.

First, the companies included should already be operating at scale. There is limited value in investing money to improve a product that isn’t widely used. For industry partners, it is equally important that there exists a wide variety of work environments so that the outcome of the work is truly generalizable to manufacturing. The second consideration is to pick a vendor that is enterprise focused. There exists several AR products that are commercially oriented. It’s important that the company supporting this effort understands how to work with and meet the requirements of EHS professionals from large corporations. Third, is that the product’s solution needs to tie in well with existing eye protection standards. This should either support puncture protection as part of the device, or allow room to comfortably wear eye protection under the device. Lastly, the companies selected should be able to support maintenance of the standards and technical solutions that are achieved, as part of a long-term vision of using AR devices in manufacturing. As such, these requirements largely lean against working with small capacity companies to perform this work.

Hardware Considerations Augmented Reality can be achieved on a number of different types of hardware. Many uses of the technology involve a Head Mounted display, but there is a growing user base that are interested in handheld form factors such as a tablet PC or a Smartphone, using the devices’ camera to capture the real world, before augmented in and presenting a combined view of the scene on the display.

Page 21 of 45

Handheld devices may avoid many of the safety concerns of wearables, such as not impacting the field of view of the user, and having a smaller impact on the user’s attention and focus. Some examples of types of hardware for AR are listed here.

Microsoft Hololens The most prevalent Augmented Reality HMD on the market today. The device is capable of tracking its environment by building a spatial map, and provides an excellent user experience. The peripheral vision on only partially blocked above, and slight distorted below by the edges of the display.

Google Glass One of the earlier HMD’s to come to market, and aimed more at “Informed Reality” or “Assisted Reality” where the device is not capable of tracking itself, and content on the display is not aligned with the real world. It is a small device, with very minimal impact to the user’s peripheral vision.

Tablets Tablets can run mobile operating systems, such as Android, or iOS, or a full operating system such as Windows 10. Users are generally much more familiar and comfortable using tablets than HMD’s, and they avoid many of the safety risks introduced by HMD’s. Currently there is a technology gap in the market with regard to accurate, reliable tracking on tablets. Android tablets will run applications using AR Core, but these platforms are limited usually by a single camera, which limits tracking accuracy. Windows 10 is very limited in tracking libraries.

Smartphones Smartphones for the most part run iOS or Android. Android AR capabilities are captured above in the Tablet section. iOS will run applications built with AR Kit, but suffer the same tracking limitations. Smartphones do offer a stable ecosystem for applications with a mature infrastructure and persistent data connection. Smartphones, when handheld also avoid many of the safety risks associated with HMD’s.

Software Considerations Applications for AR typically are developed in iOS, Android or as a UWP application for Windows devices, including the Hololens. There is a large potential to increase the safe use of AR applications in manufacturing through features in software. Generally, there are two ways to impact safety with regard to AR:

1. A class of application primarily designed to improve safety (such as illustrating hot surfaces or other hazardous elements in the environment).

2. A class of software (for example, blanking the display while the user is walking) that is built into other applications whose primary focus is something else (such as providing a work instruction). Future work on this project will

Page 22 of 45

likely focus on developing software features to improve safety in existing use cases.

3. Security Considerations (Lead: Huntington-Ingalls Industries, Ingalls Shipbuilding) Background The current mobile and wearable augmented reality technologies, that are available today, have some form of communication interface; i.e. direct connection, Wi-Fi, Bluetooth, etc… Each communication technology must address their own security issues. The common security risk they all share is information security. Malware, ransomware, DDos attacks, and so on, can affect AR systems and performing a risk assessment before implementing any AR technology can help to identify vulnerabilities. The team assigned to security considerations must research some of the most commonly used communication technologies and OS platforms the AR devices will communicate with. Bluetooth All versions of Bluetooth are vulnerable to adversaries who can view and potentially modify link keys if they are stored improperly. In addition, encryption keys may be small, which can make them vulnerable to attackers. The Bluetooth standard only includes device authentication; therefore, there is no user authentication. It is also important to note that a device can remain in discoverable or connectable mode for an indefinite period of time. The AR devices that use Bluetooth:  Hololens 1  Recon Jet Pro  Hololens 2  Sony Smart Glass  Daqri Smart Glasses  Vuzix M300XL Smart Glasses  Magic Leap One  Vuzix Blade AR Smart  Shadow Air Glasses  MIX  Optinvent ORA-2  Epson Moverio BT-200  ODG R-7 Smart Glasses  Epson Moverio BT-300  ODG R-9 Smart Glasses  Epson Moverio BT-2000  RealWear HMT-1Z1  Google Glass Enterprise

Bluetooth v3.0 Vulnerabilities If a Security Mode 4 device connects to a device using Bluetooth v3.0, it is possible for Security Mode 1, which offers no security, to be used on the device. This makes versions 2.1 and 3.0 more vulnerable to attacks. Also, SSP static keys are used in

Page 23 of 45

versions 2.1 and 3.0, which increases the device’s vulnerability to Man-in-the-Middle attacks. The AR devices that use Bluetooth v3.0:  Epson Moverio BT-200  Epson Moverio BT-2000  Sony Smart Glass Bluetooth LE & v4.0 Low energy (LE) pairing provides no eavesdropping protection. If successful, eavesdroppers can capture secret keys distributed during LE pairing. LE Security Mode 1 Level 1 does not require any security mechanisms, therefore, LE Security Mode 1 Level 3 (authenticated pairing and encryption) is recommended. Also, the Just Works pairing method provides no Man-in-the-Middle (MITM) protection. MITM attackers can capture and manipulate data transmitted between trusted devices. LE devices should be paired in a secure environment, and Just Works pairing should not be used. The AR devices that use Bluetooth LE and v4.0:  Hololens 1 (LE)  Optinvent ORA-2 (LE)  RealWear HMT-1Z1 (LE)  Recon Jet Pro (v4.0) Wi-Fi Some security threats that effect Wi-Fi are: Data Interception Data that is sent across a network is split into packets and sent by various routes. As the packets travel from one network to another, they run the risk of being intercepted, read, altered or deleted. Denial of Service (DoS) Typically accomplished by flooding the targeted machine or resource with superfluous requests in an attempt to overload systems and prevent some or all legitimate requests from being fulfilled Rogue AP’s Wireless Access Point that has been installed on a secure network without explicit authorization from a local network administrator. Wireless Intruders Connections to a Wireless Access Point without permission to do so

Page 24 of 45

Misconfigured AP’s Both human error and Access Points left on default settings will cause misconfigured AP attacks. Ad Hocs and Soft AP’s Ad Hoc and Soft AP attacks usually come in the form of a 3rd party user acting as a Man-in-the-middle between the Access Point and a device. Endpoint Attacks Laptops, tablets, mobile phones and other wireless devices connected to corporate networks creates attack paths for security threats Evil Twin AP’s Fraudulent Wireless Access Point that appears to be legitimate but is set up to eavesdrop on wireless communications. The AR devices that use Wi-Fi:  Hololens 1  Google Glass Enterprise  Hololens 2  Recon Jet Pro  Daqri Smart Glasses  Sony Smart Glass  Magic Leap One  Vuzix M300XL Smart Glasses  Shadow Air  Vuzix Blade AR Smart  MIX Glasses  Occipital Bridge  Optinvent ORA-2  Epson Moverio BT-200  ODG R-7 Smart Glasses  Epson Moverio BT-300  ODG R-9 Smart Glasses  Epson Moverio BT-2000  RealWear HMT-1Z1

Wi-Fi 802.11ac Also referred to as Wi-Fi 5, has had vulnerabilities and gaps in security. These vulnerabilities include:  SSID is broadcasted in  Backwards compatibility plaintext and can be known vulnerabilities even if broadcast is off  Passphrase issues  Default Administration  Hash collision attacks Passwords are known  MIC algorithm issues passwords  PSK vulnerable to various  MAC Address is sent in attacks plaintext and MAC Address  Backdoor ports are available filtering can be easily and ports are open by default defeated The AR devices that use this version of Wi-Fi:

Page 25 of 45

 Google Glass Enterprise  ODG R-7 Smart Glasses  Hololens 1  Daqri Smart Glasses  Magic Leap One

Operating Systems Windows 10 Windows 10 has suffered from multiple vulnerabilities. These vulnerabilities involve:  Win32k Elevation of Privilege  Microsoft Windows Journal Vulnerability Vulnerability  Windows 10 WiFi Sense  Internet Explorer Contact Sharing Vulnerabilities  Microsoft Font Driver  Microsoft Graphics Vulnerability Component Vulnerabilities  .NET Framework Escalation  Microsoft Edge Vulnerabilities of Privilege Vulnerability  Windows 10 Mount Manager  Re-Direct to SMB Vulnerability Vulnerability The most recent security vulnerability concerns a local privilege escalation (LPE) that can be used by a third-party to gain access to the entirety of a machine by granting them administrator access. According to SandboxEscaper, the vulnerability stems from the Windows Task Scheduler. If a hacker runs a specific .job file, it can grant their account administrative privileges. This means the third-party would have the power to change fundamental system settings and more. The AR devices that use the Windows 10 OS includes:  Hololens 1  Hololens 2  DreamGlass

Linux The National Vulnerability Database (NVD) is the U.S. government repository of standards based vulnerability management data represented using the Security Content Automation Protocol (SCAP). This data enables automation of vulnerability management, security measurement, and compliance. The NVD includes databases of security checklist references, security-related software flaws, misconfigurations, product names, and impact metrics. Below are vulnerabilities found in the current database:  CVE-2017-18017

Page 26 of 45

o This vulnerability impacted all Linux versions before 4.11, 4.9x and 4.9.36 o It can allow remote hackers to carry out a denial of service attack, used to shut down a machine or network, making it inaccessible to its intended users.  CVE-2017-18202 o This vulnerability can be exploited to trigger a copy_to_user command, cause a Denial of Service (DoS) attack, or force the component to handle gather files incorrectly.  CVE-2018-8822 o When exploited, hackers can cause memory corruption and DoS attacks through remote code executions.  CVE-2017-15126 o It can be exploited to execute an arbitrary code within the kernel, with failed exploitation attempts resulting in DoS.  CVE-2018-1000026 o The vulnerability can be most easily exploited using a large special packet that can launch a DoS attack on the network card, knocking it offline. The AR device that operates on Linux includes:  Daqri Smart Glasses Blue Cat Below are vulnerabilities found in the current NVD database:  CVE-2017-3143 o A malicious attacker with knowledge of a valid TSIG key name for a zone and service being targeted could able to send and receive messages to an authoritative DNS server to manipulate DNS service into accepting a dynamic update. If exploited, an attacker could manipulate zone content on a server that relies solely on TSIG or SIG(0) keys with no other address-based ACL protection.  CVE-2017-3142 o A vulnerability has been discovered in TSIG whereby an attacker that is able to send and receive messages to an authoritative DNS server might be able to circumvent TSIG authentication of AXFR (full zone transfer) requests through a carefully constructed request packet. If exploited, an attacker could view the entire contents of a zone from an unauthorized AXFR if the server provides an AXFR of a zone to an unauthorized recipient.  CVE-2017-3136 o A query with a specific set of characteristics could cause a server using DNS64 to encounter an assertion failure and terminate. An

Page 27 of 45

attacker could deliberately construct a query, enabling denial-of- service against a server if it was configured to use the DNS64 feature and other preconditions were met.

The AR device that operates using Blue Cat includes:  Shadow Air iOS Apple’s iOS has suffered from multiple vulnerabilities including:  iOS Trustjacking: Which can be exploited to gain persistent control over a victim’s device  Blueborne: Allows hackers to hijack and gain control of vulnerable devices running iOS systems before version 10  WebKit: Has a vulnerability that can crash a victim’s device using simple, crafted CSS code  Mobile Device Management: Vulnerability that allows hackers to bypass authentication to enroll potentially malicious devices in a network  Apple iOS VoiceOver: Has a bug that permits threat actors to perform a lock screen bypass and gain access to stored photos

The AR device that uses iOS includes:  Occipital Bridge Android Android’s OS has suffered from vulnerabilities that include:  RAMPage: Which is an Android exploit that allows hackers to access DRAM memory and cause information leaks.  Communication Management Programming: Vulnerability that allowed rogue, malicious apps to eavesdrop on broadcast information including Wi-Fi network names, BSSID, local IP addresses, DNS server data, and MAC addresses, and also permitted attackers to track smartphone users.  Man-in-The-Disk: attack that could be used to eavesdrop on user data, hijack sessions, and crash Android devices. The AR devices that use Android OS include:  Dreamglass  Sony Smart Glass  Epson Moverio BT-200  Vuzix Blade AR Smart  Epson Moverio BT-300 Glasses  Epson Moverio BT-2000  Optinvent ORA-2  Google Glass Enterprise  ODG R-7 Smart Glasses

Page 28 of 45

 ODG R-9 Smart Glasses  RealWear HMT-1Z1

Software Development Kits (SDK) A software development kit, or SDK, is nothing more than a set of tools that allow developers to create software or apps for a specific platform, operating system, computer system or device. For instance, developers interested in creating virtual reality software for the Oculus Rift need access to the software development kit for that device in order to do so. Simply put, it enables them to create software that will operate correctly on a particular platform or with a particular service. If the developer did not have access to the SDK, this would not be possible. In fact, it would most likely be impossible to create software that would operate at all in the environment in question. Software development kits can be used to develop augmented reality apps. DeepAR SDK Capabilities  DeepAR’s Augmented Reality SDK is used to add the highest quality Snapchat and Facebook style 3D face lenses, masks and special FX to iOS, Android, HTML5 and Unity applications.  The SDK offers realtime face detection that detects faces and facial features, using a variety of proprietary data models and advanced machine learning in 3D. It has fast and precise face detection and fast, robust eye, nose, and chin detection. It detects more than 68 facial feature points at up to 60 frames per second.  DeepAR has also developed a high performance, cross platform, rendering engine optimized for mobile devices and browsers applications. It delivers outstanding performance even on lower specification Android smartphones.  DeepAR Studio allows you to focus on creating incredible assets rather than focusing on building a complex Augmented Reality technology platform. The DeepAR Studio allows you to use existing software like Maya and Blender to create deformable and rigid objects, video filters, video effects, animated and morphing objects, as well as particle effects. It enables super-fast asset creation and testing. Requirements  A device that supports iOS, Android, HTML5, or Unity (device list is not specified)

Page 29 of 45

Case Studies  TEADS.tv, AirPush, Absolut, Armani, Sephora, and RayBan have all used DeepAR for AR advertising Safety Considerations  Input Protection  Data Protection  Threats to Gestures and other Active User Inputs Compatibility:  Occipital Bridge  Vuzix Blade AR Smart  Dreamglass Glasses  Epson Moverio BT-200  Optinvent ORA-2  Epson Moverio BT-300  ODG R-7 Smart Glasses  Epson Moverio BT-2000  ODG R-9 Smart Glasses  Google Glass Enterprise  RealWear HMT-1Z1  Sony Smart Glass

EasyAR SDK Capabilities  Video Playback Using H.264 Hardware Decoding  Transparent Video Playback  QR Code Scanning  Support for Unity3D 4.x, Unity3D 5.x, Unity3D 2017, and Unity3D 2018  3D engine plugin  Planar Image Tracking - Recognize and track planar images in real time, Runtime target generation from standard images, Unlimited recognition times, Multi-target simultaneous detection and tracking, Simultaneous QR Code scanning and target tracking, Up to 1000 offline targets, Cloud recognition support  Unlimited recognition times  Multi-target simultaneous detection and tracking  Up to 1000 offline targets  Cloud recognition support  SLAM - Monocular real-time 6 DOF camera pose tracking, instant initialization, fast relocalization, and robust to strong motion blur (optimized for mobile devices and low texture scenes).  3D Object Tracking - Recognize and track a common 3D object with textures in real time. The object may have different shapes and structures. Runtime target generation from standard wavefront obj model files, No strict limitations for the physical size of 3D objects, Recognize and track

Page 30 of 45

any 3D object after loading multiple objects, Recognize and track multiple 3D objects simultaneously  Multi type of target simultaneous detection and tracking  Screen Recording - Highly efficient and simple content recording solution. Native high performance with little resource usage, Standard H.264/AAC/MP4 output format, Multiple choices for standard resolutions and customizable parameters, Android/iOS support Requirements  EasyAR can possibly be used on all AR devices using Android, iOS, Windows, and Mac OS operating systems. Safety Considerations  Although cloud based support can save time by passing tasks onto the service provider instead of your IT department, cloud security has many vulnerabilities including data breaches, hacking and taking over accounts, insider theft, insecure API’s, etc.  Data Protection.  Output Protection Compatibility:  Android, iOS, Windows, and Mac OS  C API  C++11 API  Traditional C++ API  Java API for Android  Swift API for iOS  Objective-C API for iOS  Unity3D API ARCore SDK ARCore is Google’s platform for building augmented reality experiences. Using different APIs, ARCore enables your phone to sense its environment, understand the world and interact with information. Some of the APIs are available across Android and iOS to enable shared AR experiences. Capabilities  Motion Tracking - allows the phone to understand and track its position relative to the world.  ARCore's motion tracking technology uses the phone's camera to identify interesting points, called features, and tracks how those points move over time. With a combination of the movement of these points and readings from the phone's inertial sensors, ARCore determines both the position and orientation of the phone as it moves through space.

Page 31 of 45

 Environmental Understanding - allows the phone to detect the size and location of all type of surfaces: horizontal, vertical and angled surfaces like the ground, a coffee table or walls.  In addition to identifying key points, ARCore can detect flat surfaces, like a table or the floor, and can also estimate the average lighting in the area around it. These capabilities combine to enable ARCore to build its own understanding of the world around it.  ARCore's understanding of the real world lets you place objects, annotations, or other information in a way that integrates seamlessly with the real world. You can place a napping kitten on the corner of your coffee table, or annotate a painting with biographical information about the artist. Motion tracking means that you can move around and view these objects from any angle, and even if you turn around and leave the room, when you come back, the kitten or annotation will be right where you left it.  Light Estimation - allows the phone to estimate the environment's current lighting conditions.

Requirements  Android o Android 7.0 or later (some models require newer versions) o Device that originally shipped with the Google Play Store o Play Store requires internet access in order to update ARCore  iOS o An ARKit compatible device running iOS 11.0 or later Case Studies  Streem (Lifestyle), TendAR (Game), and Curate (Real-Estate) have all used ARCore to develop their apps Safety Considerations  This software uses cloud based sharing between devices, which has been vulnerable to data breaches and hacking in the past.  Threats to Passive Inputs  Data Protection Compatibility:  Android, Android NDK and Unity for Android  iOS and Unity for iOS  Unreal Onirix SDK According to Onirix, their SDK, “was built to interact with AR powered devices. It provides libraries and utilities for easy and rapid application development under Android, iOS and Unity. A complementary REST API allows you to easily import data into your apps.” They constantly update the documentation for all Onirix

Page 32 of 45

components and compatible devices. This includes Android, iOS, adaptations to the libraries ARKit and ARCore, Google Tango and much more. Requirements  Android o Android 7.0 or later (some models require newer versions) o Device that originally shipped with the Google Play Store o Play Store requires internet access in order to update ARCore  iOS o An ARKit compatible device running iOS 11.0 or later Safety Considerations  Input Protection  Data Protection  Output Protection  Threats to Data Processing Compatibility:  Onirix Target SDK: Mobile SDK for Android and iOS  Onirix Space SDK: Android only  Onirix Place SDK: Android and iOS  Onirix compatible with: o ARCore supported devices (Android 7.0 or later) o ARKit supported devices (iPhone 6s and above, iPad Pro, iPad 5th Gen and above) o Tango supported devices (Lenovo Phab 2 Pro, Asus Zenfone AR) o Any Android device with built-in gyroscope and magnetic sensor Pikkart AR SDK A software solution that enables applications to see, recognize and understand the surrounding world. It allows developers to easily add advanced computer vision functionality to any application, enabling it to recognize images, objects and retrieve their position and orientation w.r.t. the user device. Capabilities  PIKKART AR SDK: a fast, stable and computationally inexpensive on- device detection and tracking system of both detailed natural markers and low-detail logos. Every kind of image can be a target for our detection and tracking system that, in real time, connected to the device camera, do its magic and continuously compute the correct location and orientation of the image (3D Object detection and tracking coming soon). Unity 3D engine plugin also available.  PIKKART AR LOGO: AR Logo is a Pikkart proprietary technology that allows you to create multiple “versions” of the same image (marker) by adding an invisible code detectable by our SDK. This allows developers to

Page 33 of 45

create multiple augmented reality experiences on the same image. The ARLogo detection is on-device and thus does not need an internet connection.  PIKKART CRS (Cloud Recognition): a cloud-based image recognition and retrieval service that is fast, precise, inexpensive and easily scalable. Useful when your app needs to recognize one image out of hundreds of thousand markers or frequently updated databases.  PIKKART CRS API: a complete set of web service REST-like API that allows you to manage the Pikkart CRS image database efficiently and enables you to automate your workflows by direct integration into your content management systems. Comes together with a target manager web app. Requirements  iOS (ARM64 only) and Android (ARM and x86 devices both 32 and 64 bit) Safety Considerations  Cloud-based API  Input Protection  Data Protection  Ouput Protection Compatibility:  DreamGlass  Vuzix Blade AR Smart  Epson Moverio BT-200 Glasses  Epson Moverio BT-300  Optinvent ORA-2  Epson Moverio BT-2000  RealWear HMT-1Z1  Google Glass Enterprise  Occipital Bridge  Sony Smart Glass  Hololens 1  Hololens 2 Scope AR SDK ARCore is Google’s platform for building augmented reality experiences. Using different APIs, ARCore enables your phone to sense its environment, understand the world and interact with information. Some of the APIs are available across Android and iOS to enable shared AR experiences. Capabilities  WorkLink Assist  Image & Video  WorkLink Create Preparation  WorkLink Platform  Custom WorkLink  CAD Conversion &  Project Consulting Optimization  Advanced Training  3D Modelling & Adaptation

Page 34 of 45

 Project Based Services  ERP Integration AR Work Instructions  IoT Integration  Proof of Concept/Demo  Multi-User Scenarios  Onsite Consult/Support Requirements  Apple/iOS  Android  Windows Compatibility:  Hololens 1  Google Glass Enterprise  Hololens 2  Sony Smart Glass  Occipital Bridge  Vuzix Blade AR Smart  DreamGlass Glasses  Epson Moverio BT-200  Optinvent ORA-2  Epson Moverio BT-300  RealWear HMT-1Z1  Epson Moverio BT-2000

4. User Interface (Lead: Huntington-Ingalls Industries, Newport News Shipbuilding) Background User interfaces are the medium with which humans interact with a system and the method of interaction. The user inputs information that the system must understand and in turn provide an output to that the user can comprehend. Augmented Reality, or the overlay of digital information onto the real world, and mixed reality, stationary digital information placed into the real world, provide user interfaces that encompass the user inputs, the medium for interaction, and the system outputs that occur when a user interacts with an augmented reality system. User interface inputs include: gesture, 6 DOF controllers, speech, eye tracking, and keyboards. Gesture inputs are designed around the limitations of the software and typically require a discrete input. Well-designed systems should account for natural deviations in movement across users. Users often lack the necessary mental model to understand the limitations in the software’s ability and can become frustrated by unrecognized inputs without proper training. User frustration is compounded when the systems do not provide feedback for gesture correction to assist with error prevention. Furthermore, gestures can require deliberate unnatural movements that require cognitive effort to be learned. Gestures can be limited by the gloves, lack of haptic feedback, user embarrassment, space constraints, and require the user to remove their hands away from other tasks.

Page 35 of 45

A 6- DOF controller may be used instead of a gesture for AR user interface inputs. A 6- DOF controller allows for user control in the forward, back, left, right, up, down, pitch, yaw, and roll directions. These controllers have limitations in their ability to provide feedback to the user, particularly on the magnitude of user input. The user is also no longer hands-free to perform other tasks in the environment. Speech inputs are a way for the user to remain hands-free and mitigate some of the limitations found with gestures and 6- DOF controllers. However, speech inputs can lead to false alarms when the system picks up extraneous dialogue in the environment. Speech inputs also require considerations with the over hearing of sensitive material, user accents, and the additional cognitive load for the user to remember specific system known vocabulary. Eye and head tracking are another way in which the user can interact with a system while remaining hands and voice free. The user can move their eyes or head to select an element within the interface. Eye tracking has benefits over head tracking by eliminating ergonomic neck concerns. Eye tracking also provides the system user state information such as work load and attention. Eye tracking inputs require consideration on dwell time for selection as well as an undo option for users. A final user input is keyboards or touch screens. Keyboards used in head mounted displays are cumbersome because they remove the muscle memory found in physical keyboards. Touch screens can be combined with HMD’s for text input, however, this leads to divided attention between the two devices. Virtual keyboards are also another consideration, but they lack the haptic feedback to find and press keys. Once the AR system receives a user input, the system provides an output through visual displays, auditory feedback, and haptic feedback. Visual displays for AR devices need to display enough information to be beneficial for the user without distracting from the task at hand. Too much information can lead to cognitive overload, while too little can confuse and frustrate the user. Placement of visual information is also an important consideration. Task related information should be placed physically next to each other to abide by the Proximity Compatibility Principle, however, particularly with HMD’s, placement of this information should not occlude important task or safety related information in the environment. Buttons are also an important consideration as they should be large enough for user selection through gesture or touch. Systems can also provide feedback through audition, however users in a production environment can be battling extraneous noise or be wearing hearing protection. Another consideration with auditory feedback is the nature of the voice. With the rise of systems like Alexa and Siri, users are accustomed to interacting with a human-like voice. Finally, AR systems can provide haptic feedback to users, however, considerations include the placement of the sensor, the strength of feedback, and the safety of the user.

Page 36 of 45

Production environments can vary in noise, brightness, and free space. A worker in a loud environment utilizing an AR device with only auditory inputs and outputs would experience frustration with false alarms (ie: the system picking up false speech inputs) and difficulty hearing the auditory outputs of the system. Workers in production environments might also be required to wear hearing protection making any auditory outputs useless. Conversely, workers at times need to wear gloves or work in tight spaces making gesture or touch screen inputs not feasible. Therefore, emphasis on the current market research was placed upon interfaces that support multimodal interactions. Analysis Specific devices and the company that make them are outlined below along with an evaluation of their user interface. Devices are divided into head mounted displays and hand-helds. Head Mounted Displays Hololens, Microsoft The Hololens is a mixed reality headset. Users can provide inputs to the system through gestures, a clicker, speech, and gaze. Basic allowable gestures are air tap and bloom. There are more gestures that can be programmed into applications known by the Hololens. Issue’s with the Hololens gestures include how to teach gestures to novice users, limited room for variability that the system will allow, and the gestures that do not match the user’s mental model for interacting with physical objects. The Hololens knows a specific set of voice commands as well those that require the user to learn what vocabulary the Hololens recognizes. Users focus on different elements of the interface through gaze which utilizes head tracking but not eye tracking. Utilizing head tracking can lead to muscular strain from frequent neck movement and looking in one direction for long periods of time. The Hololens outputs information through floating 3D canvases. These canvases are placed in the environment and require the user to remember their placement for future use. The Hololens also provides auditory responses to the user. The Hololens comes with certain voices such as Cortana or the developer can record responses for a more human-like interaction. The Hololens provides visual feedback to the user on where they are gazing through a gaze circle. This circle changes when the user selects a GUI element. The Hololens is limited by a very small field of view (FOV) with which to place contextual information. This can lead to divided attention as the user looks away from the task at hand to find parts of the interface. Conversely, overcrowding and cognitive overload can occur in an attempt to keep all task relevant information in the FOV. Shared experiences are capable with the Hololens. Multiple users can view the same 3D information through a software called Photon. The user sees changes the

Page 37 of 45

other users have done in 3D space. Research considerations should look at how to represent this information and how to represent collaborative users with the interface. Hololens can also connect to tablets for shared experiences. Without eye tracking or other physiological measures, it is difficult to do workload assessment. The Hololens may be programmed with other machine learning software to identify things in the environment that may contribute to increased workload however, this would require significant additional development. Additionally, high workload varies between users and without convergent physiological measures may not be accurate. Hololens 2, Microsoft The next version of Hololens will be released later this year and promises significant improvements to the user interface and an increased FOV. Microsoft labels user inputs as “Instinctual Interaction” and includes gestures, eye tracking, and speech inputs. The gestures recognize the full hand and calibrates to personal hand size allowing for user customization. The user can interact with holograms or 3D information in the same manner as they would with physical objects such as pressing a holographic button the same way they would press a real button. The system then provides physical feedback so that the user can see the button depress into 3D space. The system also accommodates speech input through natural language speech processing and according the Microsoft website uses a smart microphone to work in “industrial environments.” Users can also connect the application to their phone and input text information that then appears in the real world via the Hololens. This increases ease of interaction given the difficulty of typing on a HMD and when voice to text situations are inappropriate such as a loud production environment. The Hololens 2 outputs information using traditional GUI icons and floating 3D canvases. The user has the ability to pick up and move canvases as needed for the task using a grab and drop gesture similar to picking up and moving physical objects. The user can also have 3D canvases follow them with speech commands. GUI icons on the 3D canvas are traditional flat icons without labels. Application icons can be 3D holograms allowing for direct manipulation. When the Hololens detects your hand within the XYZ dimensions of interacting, the 3D canvas of the icon highlights and a box out line shows how big or small you are resizing the 3D model. This element helps to provide visual feedback for 3 dimensions. The start menu, however, is 2D and opens using a gesture similar to the “bloom” feature in Hololens 1. Users can have 3D canvases follow them into an application. The benefit would be when a user needs to pull information about the task from multiple areas and reduce the workload of switching between applications. The system also provides auditory feedback when pressing holograms. Videos for the Hololens 2 make reference to a “tactile sensation” when interacting with sliders. However, it is unclear

Page 38 of 45

how this is being simulated without a device for vibrotactile feedback. GUI lists are handled with horizontal carousels. The Hololens 2 will have a built in procedure training application. The application connects with a tablet for selection by an instructor. The application presents work instructions on a nearby 3D canvas and tethers canvas with a white dotted line to the part the instructions for that step are referring to. The feature utilizes holograms of the part to line up with the user’s goals. The Hololens 2 allows for user customization with Microsoft stating the system automatically recognizes who you are and logs you in. Shared Experiences are also possible with Dynamic 365 Remote Assist, which appears to work with a 3D canvas like a phone call sharing with the caller what the user sees through their Hololens. The device also uses holographic avatars to represent users you are communicating with elsewhere. As the Avatar user moves around in real space, the user sees the hologram move in the space in front of them, enabling collaboration on digital objects. Workload Assessments will also be possible with the use of eye tracking. Magic Leap One, Magic Leap The Magic Leap One is a mixed reality headset. User inputs include: head tracking, eye tracking, and 6-DOF controller. The 6-DOF does not require an external sensor and has a trigger and bumper button allowing for the clicker metaphor for GUI when needed. There is a home button directly on the controller. The trackpad allows for a combination of micro and macro interactions by moving the controller. Gesture interactions are available with the device, however, the hand recognition is limited to part of the hand. Eye tracking with the device records fixation point position, eye centers (pupils), and blinks. The device also allows voice inputs such as speech to text. While reports state that eye tracking and hand tracking capabilities are built into the Lumin OS, none of the applications native to the device utilize these forms of input. The Magic Leap One provides LRA haptic feedback within the controller. The system also provides visual on screen feedback of the hand detected, the gesture, and the “percent detected” through text. Issues with this feedback include dividing attention between object of focus and onscreen feedback. This issue may be compounded when doing multimodal interactions between gesture and eye tracking (ie: eye tracking for the “pre select” and gesture for “confirmation”). The standard GUI for the device is minimalistic with radial menus positioned in the center of the FOV. Icons highlight and grow in size to show what is about to be selected. Shared experiences with Magic Leap One are coming “soon” through Cast and Avatar Chat. Seeing an individual’s avatar in space differs greatly from gaining the perspective of the other user providing different experiences. The ability to view from a different user’s perspective is a capability not yet enabled according to Magic Leap and neither is the avatar feature.

Page 39 of 45

With eye track capabilities the device also has the capability to assess workload. This can be used to determine when the user is experiencing fatigue and mental over load in a production environment.

Daqri Enterprise Smart Glasses, Daqri Daqri Smart Glasses allow users to input information through head tracking and dwell for a hands- free interaction. However, it is not clear how long the “pause”/dwell time is for selection. The user must perform significant head movement to interact across the interface which can lead to fatigue and divided attention. A virtual keyboard also uses dwell to select letters for textual input. This leads to flow entry time because the user cannot use muscle memory from previously learned input devices. The speed of input is also limited to the minimum dwell time for the system to register selection. Daqri Smart Glasses have a main menu called the launcher that has traditional desktop and webpage GUI elements. Vertical lists are used in the launcher that are text heavy rather than icon based. Any 2D GUI icons present in the interface are small and located in menus to the top and left. The size of the icons may make it difficult for users to select particularly with gaze and dwell. The system displays a gaze reticle to provide feedback to the user on where they are gazing. This reticle then expands and flashes to provide feedback on selection. Daqri has an application for the work environment titled WorkSense. Videos of WorkSense user interactions show the user moving in and out of menus frequently which can lead to an increased workload. The application also supports shared experiences where the user can make a phone call and display their perspective to the receiver. The GUI appears to match that of a cell phone call and the receiver can view on a laptop. The receiving viewer can also mark on the screen and the input can be seen by the headset user. Epson Moviero BT 2000 The Epson Moviero BT 2000 is a HMD that allows for gesture, voice, head tracking, and controller inputs. The controller contains an enter key, A, B, X, Y, and 4 directional keys. The keys can be mapped by the developer, but require the user to learn the different inputs. The voice input works on a specific set of known vocabulary. The signal to noise ratio can be adjusted to avoid interference from loud background conditions. Other versions of this device have controllers with trackpads that use traditional touch screen user inputs. This devices is based on Android tablet 4.0 UI. The device can switch between 2D and 3D displays based on whether the info is displayed to both or one eye. The device allows for video streaming so the user can share their perspective with others through Moviero Mirror.

Page 40 of 45

Leap Motion, Ultrahaptics LTD Leap Motion is a gesture recognition device. The controller must be mounted to track the user’s hands. According to the company, the user’s hands will track in complex backgrounds and extreme lighting conditions. The user must be aware of their hands relation in reference to the device and there is a chance for occlusion of the environment. There are individual differences with gesture performance which should be a consideration for application development. The system displays a 3D model of the user hand mimicking the gesture that they have done. If the gestures developed are not easily detected by the system with a specific start and end motion the user can become frustrated by a lack of feedback. Gesture interactions are strongly dependent on visual feedback because the user does not have the benefit of tactile feedback from interacting with a physical device. The Leap Motion is paired with other devices, one of the common ones being the HTC Vive virtual reality headset although the Leap Motion could also be paired with AR devices. LAFORGE Optical LAFORGE Optical are smart glasses that allows the user to input information through buttons on the arm of the glasses and voice interaction. The system can project on one or both eyes. The glasses connect to the user’s smart phone and applications are downloaded through socialFLO. The device does not allow: full screen videos, web browsers, ads, e-readers, PDFs, spreadsheets, surveillance, and mirroring to other devices. Therefore, the devices ability to assist a worker in a production environment is extremely limited. Applications built for the device follow GUI Design standards including a gHost theme. The gHOST theme uses Avenir-Next font, black backgrounds, and hexagonal brackets. The user can dismiss with a red X in the bottom left hand corner, which goes against normal design standards. The company however, believes in focusing on not displaying a mobile application, but reinventing UI to provide value to the user. Applications on this device can support collaboration if both users have the same application open. The user can customize and control what applications are downloaded for the glasses. Vuzix Blade AR Smartglasses, Vuzix The Vuzix Blade AR Smartglasses allow the user to input with a touch pad, head motion, or an application on and Android or iPhone device. The touch pad swipe works like a phone but the orientation on the arm of the glasses is not representative to that of the device (ie you swipe backwards in space and the display moves left). The user can select by tapping on touch pad, 2 finger tap to move back, and a 2 finger hold takes the user home. There appear to be touch pads on left and right

Page 41 of 45

side to accommodate both handedness. The device also supports voice command inputs. The device provides output through haptic vibration. The GUI of the device utilizes flat icons, with some labels, and very minimalistic. The display projects on the right eye. The user can customize the UI to one of their settings to dark or light mode (transparent background or white background). The device will also automatically switch when moving from bright to dark environments. The user can also change what alerts they get on their glasses through their smart phone. Dreamworld Dreamglass, Dreamworld AR Dreamglass is a head mounted AR device with a 90-degree FOV. The device connects to an Android phone (iPhone in development) by a cord or cordless for mobile user input. The device has hand recognition, head tracking in 3-DOF, and a 6- DOF controller. The 6-DOF controller works with Nolo, and the user is able to use two controllers at once. The controller has buttons on top, a trigger, and a track pad for touch interaction. Dreamglass displays a virtual cursor to provide feedback to the user as to where they are looking. Secondary users are also able to input information on a mobile device for the primary user to see in the HMD. Google Glass Enterprise, Google With the Google Glass Enterprise, the user can use voice command inputs and a touch pad on the arm of the glasses. The system outputs information in the periphery and not in the line of sight of the user. This prevents occlusion of task important information in the environment. The glasses can be linked to phones and tablets for live video streaming and shared collaboration. Hand- held Devices  Tablets, Android (Galaxy Tab S3 and above, Samsung), iOS (iPad, Apple)  Smart Phones, Android (Galaxy S8 and above, Samsung, Pixel 3, Google), iOS (iPhone 6 and above, Apple) With tablet and smart phone devices users input information through the touch screen. These devices have the added benefit of the user being familiar with the interactions. Therefore, the user does not experience increased cognitive load from having to learn how to interact with the device. Android and iOS platforms have design standards for native applications however, the GUI for AR created applications on these devices would be up to the designer of the application. For iOS devices, AR Kit is used for the development of AR applications. The latest AR Kit allows for shared AR experiences across multiple devices. The Apple website contains pages of recommendations on how to create different GUI elements when designing an application using AR Kit. For Android devices and iOS devices, AR Core (Google) is used for the development of AR applications. AR Core also allows

Page 42 of 45

for shared experiences across users. Google’s website also contains a set of design guidelines when developing an AR application. Developers should design applications that are accessible and with appropriately sized 3D content given the environmental constraints. Users can tap to select and drag to move 3D content. The user can rotate the content using 1 or 2 finger manual rotation and scale with a pinch gesture. Google does not recommend haptic feedback for Android devices because the developer is unable to design for every device and the devices vary. Haptic feedback, or vibrations, can also cause tracking to be lost. However, audio outputs can be used for multimodal interactions. By design, handheld devices tend to be less immersive than head mounted displays. In order to increase immersion, Google recommends avoiding pop ups, full screen takeovers, notifications, or any other elements that may draw the users attention away from their current task. Given the novelty of AR, keeping the elements on the screen as minimal and simple as possible can help minimize additional cognitive load for the user.

Page 43 of 45

References Microsoft Corporation. (2019, March 6). Safety information. Retrieved from Microsoft Support: https://support.microsoft.com/en-us/help/4023454/safety-information NIOSH. (2018, March 28). Worker Health Surveillance. Retrieved from Centers for Disease Control and Prevention: https://www.cdc.gov/niosh/topics/surveillance/ OSHA. (n.d.). Medical Screening and Surveillance Overview. Retrieved from United States Department of Labor: https://www.osha.gov/SLTC/medicalsurveillance/ "[Infographic] Top 10 Windows 10 Vulnerabilities And How To Fix Them". Upguard.Com, 2019, https://www.upguard.com/articles/top-10-windows-10-security- vulnerabilities-and-how-to-fix-them. Accessed 31 May 2019. "Arcore Overview". Google Developers, 2019, https://developers.google.com/ar/discover/. Accessed 31 May 2019. "Developer Portal". Onirix, 2019, https://docs.onirix.com/onirix-sdk. Accessed 31 May 2019. "Dreamworld". Dreamworldvision.Com, 2019, http://www.dreamworldvision.com/ Accessed 31 May 2019. "Easy AR SDK". Easyar.Com, 2019, https://www.easyar.com/view/sdk.html. Accessed 31 May 2019. "Glass". Glass, 2019, https://www.google.com/glass/start/. Accessed 31 May 2019. "ORA-2". Optinvent, 2019, http://www.optinvent.com/our_products/ora-2/. Accessed 31 May 2019. “Pikkart AR SDK.” Pikkart AR SDK & Ecosystem, 2019, developer.pikkart.com/augmented-reality/products/sdk.aspx. "SDK | Deepar ~ Snapchat Face Filters And Lenses Augmented Reality SDK". Deepar, 2019, https://www.deepar.ai/augmented-reality-sdk. Accessed 31 May 2019. “Security & Vulnerability Updates.” BlueCat Networks, www.bluecatnetworks.com/support/security-vulnerability-updates/. “What Is a Software Development Kit (SDK) ?” GoodFirms, 26 July 2017, www.goodfirms.co/glossary/software-development-kit-sdk/. "5 Common Linux Security Vulnerabilities". Security Training, 2018, https://securitytraning.com/5-common-linux-security-vulnerabilities/. Accessed 31 May 2019. Carey, Joseph. "Windows 10 WARNING: Huge Vulnerability Discovered As Microsoft Releases May 2019 Update". Express.Co.Uk, 2019, https://www.express.co.uk/life-

Page 44 of 45

style/science-technology/1130677/Windows-10-warning-alert-new-zero-day-security- vulnerability-discovered. Accessed 31 May 2019. Farik, Mohammed, and Shawkat Ali. "Recurrent Security Gaps In 802.11Ac Routers". INTERNATIONAL JOURNAL OF SCIENTIFIC & TECHNOLOGY RESEARCH, vol 4, no. 09, 2015, p. 3., Accessed 31 May 2019. Guzman, Jaybie A. de, et al. “Security and Privacy Approaches in Mixed Reality: A Literature Survey.” CoRR, pp. 1–40., arxiv.org/abs/1802.05797. Lonzetta, Angela et al. "Security Vulnerabilities In Bluetooth Technology As Used In Iot". Sensor And Actuator Networks, 2018, p. 26., Accessed 29 May 2019. Osborne, Charlie. "Many Of 2018'S Most Dangerous Android And Ios Security Flaws Still Threaten Your Mobile Security 10 - Page 10 | Zdnet". Zdnet, 2018, https://www.zdnet.com/pictures/the-most-dangerous-vulnerabilities-security-flaws- found-in-google-android-apple-ios-over-2018/10/. Accessed 31 May 2019. Padgette, John et al. Guide To Bluetooth Security. Gaithersburg, MD, 2012. Phifer, Lisa. "Top Ten Wi-Fi Security Threats". Esecurity Planet, 2010, https://www.esecurityplanet.com/views/article.php/3869221/Top-Ten-WiFi-Security- Threats.htm. Accessed 31 May 2019. Hix, D. & Hartson, H. R. (1993). Developing user interfaces: Ensuring usability through product and process. New York, NY: John Wiley & Sons. LaViola, JR, J.J., Kruijff, E., McMahan, R.P., Bowman, D.A., & Poupyrev, I. (2017). 3D User Interfaces Theory and Practice Second Edition. Boston, MA: Addison- Wesley. Pereira, A., Wachs, J.P., Park, K., & Rempel, D. (2015). A user-developed 3-D hand gesture set for human-computer interaction. Human Factors, 57, 607-621. Wickens, C. D. & Carswell, C. M. (1995). The proximity compatibility principle: Its psychological foundation and relevance for display design. Human Factors, 37, 473 – 494.

Page 45 of 45