Assessing the Quantitative and Qualitative Effects of Using Mixed
Total Page:16
File Type:pdf, Size:1020Kb
23rd ICCRTS Info. Central, Pensacola, Florida 6 - 9 November 2018 Assessing the quantitative and qualitative effects of using mixed reality for operational decision making Mark Dennison1, Jerald Thomas2, Theron Trout1,3 and Evan Suma Rosenberg2 1U.S. Army Research Laboratory West, Playa Vista, CA 2University of Minnesota, Minneapolis, MN 3Stormfish Scientific, Chevy Chase, MD Abstract parate systems (physical objects, computers, paper documents, etc.) that require a significant amount The emergence of next generation VR and AR de- of resources and effort to bring into a unified space. vices like the Oculus Rift and Microsoft HoloLens Human interactions require shared cognitive mod- has increased interest in using mixed reality (MR) for els where interaction with systems must support and simulated training, enhancing command and control, maintain this shared representation, stored informa- and augmenting operator effectiveness at the tactical tion must persist the representation at a fundamen- edge. It is thought that virtualizing mission relevant tal data-level, and the underlying network must allow battlefield data, such as satellite imagery or body- these data and information to flow without hindrance worn sensor information, will allow commanders and between human collaborators and non-human agents analysts to retrieve, collaborate, and make decisions (1; 2). about such information more effectively than tradi- The emergence of Mixed Reality (MR) technolo- tional methods, which may have cognitive and spa- gies has provided the potential for new methods for tial constraints. However, there is currently little the Warfighter to access, consume, and interact with evidence in the scientific literature that using mod- battlefield information. MR may serve as a unified ern MR equipment provides any qualitative benefits platform for data ingestion, analysis, collaboration, or quantitative benefits, such as increased task en- and execution and also has the benefit of being cus- gagement or improved decision accuracy. There are tomized based on the mission needs and requirements also no validated metrics in the field for comparing of each operator (see Figure 1). MR lies in the across display devices and tasks. In this paper, we Reality-Virtuality Continuum between the physical surveyed potential metrics for assessing the usefulness and the digital world (3). Augmented Reality (AR) of MR technologies, discuss how these data might be and Virtual Reality (VR) exist at the extremes of acquired in experimental and tactical scenarios, and the MR spectrum, as shown in Figure 2. Where AR pose issues in multi-user communication and collabo- superimposes generated content over the real world, ration. We also introduce the Mixed Reality Tactical VR occludes the real world entirely to present some- Analysis Kit (MRTAK), which functions as an exper- thing entirely fabricated. Each of these immersive imental platform to perform these assessments during technologies has benefits and drawbacks, but there collaborative mission planning and execution. has been limited research exploring what these are beyond speculation. Importantly, MR ensures a vi- 1 Background sual connection to the physical world, while utilizing elements that may be superimposed on reality or oc- The modern battlefield and Army operational envi- cluding it completely with a purely virtual rendering ronment is becoming more varied and dynamic, with of information and objects. Thus, MR may serve as a greater reliance on the integration of information a medium to integrate data from sensors monitoring from intelligent things/devices, agents, and systems. the real world, with the ability to perceive and reason Information overload caused by multitasking and mis- on this information without many of the spatial and sion execution at standoff remain significant chal- physical constraints of currently used C3I systems. lenges in C3I scenarios. The integration of infor- The recent increase in the ease of access to mod- mation for decision making and other mission com- ern high-fidelity head-mounted displays has caused a mand tasks is often still done using discrete and dis- resurgence in interest for using immersive technolo- Page 1 Figure 1: Information flow between forward operators and analysts using immersive display devices. gies in the Private and public sectors. Consequently, 1.1 Evaluation of Immersive Tech- the “cool factor” associated with VR and AR tech- nologies nologies has become a common reason for their adop- tion, with little empirical data backing this up. Sim- Recently, this area of research has been referred to as ilar issues have been found in adding gamified ele- “Immersive Analytics” (5). Chandler and colleagues ments to training, which can be completely ineffec- suggest five major topic areas: 1) What paradigms tive or simply less effective than less engaging tradi- are enabled by immersive technologies and how do tional methods. With respect to command and con- we evaluate them over other traditional mediums and trol, prior work has shown that novel interfaces that each other?, 2) Do these technologies provide a more support decision making have traditionally been chal- holistic way of looking at data that contains 3D spa- lenging for users to understand and interpret (4). Ad- tial and abstract information?, 3) What are the best ditionally, there is little evidence in the literature of interface “tricks” and affordances that change a user’s the quantitative benefits of using immersive technolo- perspective from an allocentric to egocentric view of gies in operational decision-making, nor are there a the data?, 4) Do these technologies invalidate the set of tasks and validated measures for assessing opti- literature on 2D data interaction?, and 5) What is mality. Here, we review the limited current literature the typical work-flow for examining data across do- that have attempted to address this issue with re- mains and how do we develop generic platforms to spect to evaluation and communication in MR, and support immersive analytics? Although each of these discuss how the MRTAK project seeks to build upon questions is important, here we focus on items one this work as a sandbox for immersive C3I research. and two, which pose the more general question of how should we evaluate the effectiveness of immer- sive technologies and what data is necessary to per- form this assessment. Uses for MR Technology One area where immersive technologies have been used extensively is for simulation and training on real- world tasks. For example, a study by Donalek and colleagues (6) reported that in a way-point drawing task, subjects who viewed the environment in an Ocu- Figure 2: Mixed reality technology spectrum. lus Rift HMD performed with less distance and angle Adapted from Milgram (3) errors than those who viewed the environment on a 2D desk-top monitor. Moran and colleagues (7) cre- ated an immersive virtual environment where Twit- ter data was overlaid atop real geography to improve Page 2 the experience for analysts. The authors claimed that icantly and the authors felt that researchers focused this MR environment enhanced situational awareness, solely on assessing time and task errors that they cognition, and that pattern and visual analytics were failed to measure for critical effects such as motion more efficient than on traditional 2D displays. A sickness, decay and recall of skills after immersion, study by Dan & Reiner (8) measured performance the level of trust or acceptance of the device, and differences among subjects who had to complete a the users’ prior attitude, skills, and experience with paper folding task after viewing information on a 2D similar technologies. desktop monitor or through an augmented environ- Some studies have shown the importance of the is- ment. Subjects showed a higher cognitive load index sues brought up by Borsci. For example, it was found when learning in 2D vs 3D, as measured by the ratio that reported VR system usability was correlated of frontal theta power over parietal alpha power. This strongly with a user’s level of trust in that system indicated that information transfer was significantly (14; 15). These criteria were assessed through vali- easier when the data was viewed in an MR environ- dated metrics such as the System Usability Scale (16) ment. Other work has shown that the perception of and Trust in Technology questionnaires (17). Neuro- one’s virtual body and hands is also a critical feature physiological surveys have also been shown to corre- when performing cognitively demanding tasks, such late strongly with performance in immersive environ- as memorization, when done in a virtual environment ments. Davison (18) showed that Performance on the (9). This decreased cognitive load may be related to Trail-Making Task A (TMT-A) (19), a task consid- the fact that humans are “biologically optimized” to ered to assess motor speed, was found to be signifi- perceive in 3D (6). McIntire and colleagues (10; 11) cantly related to other measures which also assessed reported that use of a 3D stereoscopic display in- speed, such as the time taken to complete parking creased task performance by roughly 60% on average. simulator levels and the time taken to place virtual Recently, it was reported immersive AR was found to objects around a room. Measures of executive func- be better when manipulating data that required spa- tion, such as TMT B performance, was found to be tial perception and interactions with a high degree of significantly related to performance on both of these freedom (such as tangible user interfaces), but users spatial location tasks. Dennison and colleagues found were generally faster on the desktop if the task was that motion sickness caused by immersion in a virtual familiar (12). Generally, though, these studies pro- environment (VE) greatly impacted the duration to vide limited empirical evidence for which immersive which participants elected to remain in the VE and mediums (VR, AR, MR) are best for improving user complete decision making tasks (20; 21; 22). Collec- decision making across content domains, and many tively, these studies demonstrate the need to assess do not use similar or easily comparable metrics.