'HIHQFH5HVHDUFKDQG 5HFKHUFKHHWGpYHORSSHPHQW 'HYHORSPHQW&DQDGD SRXUODGpIHQVH&DQDGD

Assessment of Future Trends in Human- Computer Interaction for Military Intelligence

Denis Gouin Valérie Lavigne DRDC Valcartier

Defence Research and Development Canada – Valcartier Technical Report DRDC Valcartier TR 2012-425 December 2012

Assessment of Future Trends in Human- Computer Interaction for Military Intelligence

Denis Gouin Valérie Lavigne DRDC Valcartier

Defence Research and Development Canada – Valcartier Technical Report DRDC Valcartier TR 2012-425 December 2012 IMPORTANT INFORMATIVE STATEMENTS

© Her Majesty the Queen in Right of Canada, as represented by the Minister of National Defence, 2012 © Sa Majesté la Reine (en droit du Canada), telle que représentée par le ministre de la Défense nationale, 2012 Abstract ……..

Collective command and control civil-military operations impose a strong requirement for organizations and people to work collaboratively and to interact with information more efficiently. Human-Computer Interaction (HCI) technology has been evolving at a great pace during the last decade, and many novel HCI concepts and tools have become of interest to support not only analysts deployed in command centers but also warfighters deployed in the field. This will have an impact on how intelligence officers will operate in the future. In this regard, Defence R&D Canada has undertaken a major initiative to look forward into Future Intelligence Analysis Capabilities (FIAC). This study reports on HCI trends and applicability to support the FIAC and collective command and control in general. Topics of interest include ubiquitous computing, multimodal interaction, multi-touch tables/surfaces, large group displays, flexible displays, personal device assistants / smart phones, intelligent software assistants, biometry, brain/neural interfaces, collaboration tools / telepresence, smart room environments, head-mounted displays, mixed reality, and advanced HCI concepts and techniques.

Résumé ….....

Les opérations conjointes civiles-militaires de commandement et contrôle imposent aux organisations et au personnel de collaborer et d’interagir avec l’information de façon plus efficace. Les technologies d’interaction humain-machine (IHM) ont évolué à un rythme soutenu au cours de la dernière décennie et plusieurs nouveaux concepts et outils d’IHM sont devenus d’intérêt pour supporter non seulement les analystes de centres de commandement mais aussi les combattants déployés sur le champ de bataille. Ceci aura un impact sur la façon dont les officiers du renseignement vont opérer dans le futur. À cet effet, Recherche et développement pour la défense Canada a initié un projet dans le but d’explorer les capacités futures d’analyse du renseignement (CFAR). Ce mémorandum technique fournit un compte-rendu sur les tendances en IHM et leur application potentielle en support au CFAR et au commandement et contrôle conjoint en général. Les sujets d’intérêt incluent l’informatique omniprésente, les interactions multimodales, les tables ou surfaces multi-tactiles, les écrans de groupe de grande taille, les écrans flexibles, les assistants personnels / téléphones intelligents, les assistants logiciels intelligents, la biométrie, les interfaces neuronales, les outils de collaboration, les environnements intelligents, les casques de réalité virtuelle, la réalité mixte, et les concepts et les techniques avancés d’IHM.

DRDC Valcartier TR 2012-425 i Executive summary

Assessment of Future Trends in Human-Computer Interaction for Military Intelligence Denis Gouin; Valérie Lavigne; DRDC Valcartier TR 2012-425; Defence R&D Canada – Valcartier; December 2012.

Introduction or background:

The context of military operations has changed significantly since the Cold War era. New operations are characterized by a prevalence of asymmetric warfare such as in urban environments or when dealing with rogue states or terrorism; the emergence of instantaneous connections between the tactical and the strategic levels, and the increased linkages between the local, regional and global spaces; the complexity and diversification of human terrains and contexts; the conduct of Joint, Interagency, Multinational and Public (JIMP) operations; and the continued progression of Communication and Information Technology, which provides significant opportunities to better conduct military operations.

This new context has required organizations and people to work collaboratively and to interact with information more efficiently. A key enabler for this is the efficient use of Human-Computer Interaction (HCI) technology which has been evolving at a great pace during the last decade. Many novel HCI concepts and tools have become of interest to support not only military analysts deployed in command centers but also warfighters deployed in the field.

In order to address the new context of military operations, from an intelligence perspective, Defence R&D Canada (DRDC) has undertaken a major initiative to look forward into Future Intelligence Analysis Capabilities (FIAC). The core objective of the FIAC Program is to advance the state-of-the-art of intelligence methods and resources in order to adapt to and anticipate the new challenges of the Canadian Forces (CF). FIAC addresses information services from an all sources perspective, over the full intelligence cycle but with particular emphasis on advanced analysis capabilities. It seeks, in particular, to achieve synergy between human cognition and machine intelligence, fully exploiting collaborative interaction. This will have an impact on how intelligence officers will operate in the future.

Results:

This technical report is a state-of-the-art review of the HCI trends and the applicability to support the FIAC and collective command and control in general.

The primary HCI trend that is influencing new information systems is ubiquitous computing. Users are surrounded by computers or computational services embedded in various objects that have become smaller, more mobile and personalized. The addition of networked communications will allow many of these embedded computations to coordinate with each other and with the users.

ii DRDC Valcartier TR 2012-425 Interaction with computers is becoming more and more natural, in particular because of the growing provision of multimodal user interfaces, where a user can exploit touch, gesture and voice interaction, in addition to the traditional mouse and keyboard.

Multi-touch tables / surfaces will allow analysts to come together around the table to debate a military situation, in a way similar to the use of paper maps laid out on a table. The new technology would improve shared awareness by providing users with eye-to-eye interaction while providing them with information at their fingertips.

Display technology continuously evolves, providing displays with larger sizes, increased resolution and better readability. Display panels can even be assembled in a mosaic, in order to provide a knowledge wall. These are particularly suitable to command centers. At the same time, advances in display technology have allowed to produce small, light weight and flexible displays, very suitable for mobile operations.

In order to improve the work of military analysts, advanced workstations should be examined. Some of the features to be considered are an increased screen real estate, multi-touch interaction, and the use of a horizontal and a vertical surface.

Over the last several years, personal device assistants, smartphones and electronic tablet technology have also gone through significant evolution and wide user endorsement. In the United States, a brigade is being modernized with these devices and dedicated military applications are being developed. The devices are also being secured. A number of applications could be available for deployed intelligence collators and analysts, such as: location awareness, augmented reality, conferencing, culturally-assisted translation, information aggregation, live status tracking, dynamic route finder, biometry-based recognition, and virtual assistants.

Exemplified with the deployment of Siri on the Apple IPhone, advances are also made in terms of intelligent software assistants. In a military intelligence context, virtual assistants will be growing in importance in terms of reducing the human cognitive overload problem; they will provide assistance to the user by conducting a wide variety of tasks, including searching and organizing information, tracking people, managing schedules, assigning tasks, summarizing documents, mediating interactions, guiding and reminding the user, learning procedures and preferences, and suggesting alternatives.

Biometry has open up tangible possibilities to enhance security, in particular for authenticating users’ access to facilities or computers, and to identify / recognize belligerents.

Advances in biometry and brain / neural interfaces are showing the capability to recognize users’ emotions, stress level or cognitive overload, and this will be used to design adapted interfaces that accommodate capabilities and limitations in human information processing and decision making.

As military analysts will need to work in a collaborative manner, they should exploit both collaboration tools such as audio / video conferencing, chat / instant messaging, and electronic white board, but also exploit pervasive human sensing and telepresence. Users should also be supported by smart room environments which will sense, interpret and react to human activity in order to enable better collaboration, increased productivity, creative thinking and decision making.

DRDC Valcartier TR 2012-425 iii Head-mounted displays have diminished in size, even to the point of being available as a retinal display. They are devices of predilection for mixed reality applications. They also enable a private view of the information and reduce the risk of sensitive information being seen by other people.

Over the last decade, HCI technology has progressed significantly in terms of mixed reality. As part of it, augmented reality allows deployed military users to overlay tactical information onto information from the real world, and display it on the head mounted display or their personal device assistant. Immersive environments allow military staff to develop complex situation understanding or to conduct mission rehearsal before being sent in a hostile environment. Virtual world technology has grown as a good way to train and collaborate.

Intelligent and command and control analysts will also benefit from advances made in HCI concepts and techniques. On top of the list is information visualization and visual analytics which exploit the cognitive ability of the human to perceive and make a mental representation of information or a situation, and allow making sense of large quantity of information. The users will also benefit from concepts to better organize the information, interact with it or present it. This includes the use of configurable workspaces, adaptive user interfaces, advanced interface widgets, smart lenses, and storytelling.

Significance:

This report is of significant for people interested in the design, implementation or use of command, control and intelligence systems. HCI is a key aspect of such a system, as it provides the access layer. These systems must be designed and created in a way that the user will want to use them and will find them effective when used. It is important to understand the HCI trends and those that would be beneficial for a capability such as the FIAC.

Future plans:

This report is one contribution to the vision of the FIAC. In addition, a number of other documents have been produced, including a white paper “Towards a Cohesive R&D Program Definition of the FIAC”, the production of an illustration of a future multi-intelligence center, the writing of a fiction narrative of a future intelligence environment, and the development of Operating concepts for the FIAC. Following these efforts, a prioritization of the design and development activities will be performed and a roadmap will be produced. In terms of the HCI aspect of the FIAC, a number of the most promising techniques and concepts will be selected as building blocks for the first versions of the FIAC.

iv DRDC Valcartier TR 2012-425 Sommaire .....

Assessment of Future Trends in Human-Computer Interaction for Military Intelligence Denis Gouin; Valérie Lavigne; DRDC Valcartier TR 2012-425; R & D pour la défense Canada – Valcartier; décembre 2012.

Introduction ou contexte:

Le contexte des opérations militaires a changé de façon significative depuis l’ère de la Guerre froide. Les nouvelles opérations sont caractérisées par la prédominance des guerres asymétriques telles que celles ayant lieu dans des environnements urbains ou impliquant des états sans scrupules ou supportant le terrorisme; l’émergence de connexions entre les niveaux tactiques et stratégiques, et les liens accrus entre les dimensions locales, régionales et globales; la complexité et la diversification des terrains et contextes humains; la conduite d’opérations interarmées, inter- organisationnelles, multinationales et publiques; et l’évolution continue des technologies des communications et de l’information, permettant d’améliorer la conduite des opérations militaires.

Ce nouveau contexte a forcé les organisations et les gens à travailler de façon collaborative et à interagir avec l’information plus efficacement. Un facilitateur clé pour cela est l’utilisation efficace de la technologie d’interaction humain-machine (IHM) qui a évolué à un rythme soutenu au cours de la dernière décennie. Plusieurs nouveaux concepts et outils d’IHM sont devenus d’intérêt pour supporter non seulement les analystes de centres de commandement mais aussi les combattants déployés sur le champ de bataille.

À cet égard et dans une perspective du renseignement, Recherche et développement pour la défense, Canada (RDDC) a initié un projet dans le but d’explorer les capacités futures d’analyse du renseignement (CFAR). Le cœur du programme CFAR est d’améliorer les méthodes et ressources à la fine pointe de la technologie, dans le but de les adapter et d’anticiper les nouveaux défis se posant aux Forces canadiennes. Le CFAR traite des services d’information dans une perspective toute source, couvrant le cycle complet du renseignement avec une emphase sur les capacités d’analyse avancées. Il se préoccupe entre autres de réaliser une synergie entre la cognition humaine et l’intelligence artificielle, exploitant complètement l’interaction collaborative. Ceci aura un impact sur la façon dont les officiers du renseignement opéreront dans le futur.

Résultats:

Ce rapport technique fournit un compte-rendu sur les tendances en IHM et leur application potentielle en support au CFAR et au commandement et contrôle conjoint en général.

La tendance principale en IHM qui influence les nouveaux systèmes d’information est l’informatique omniprésente. Les usagers sont entourés d’ordinateurs ou de services informatiques intégrés dans divers objets qui sont devenus de plus en plus petits, mobiles et personnalisés. Les communications réseau vont permettre à plusieurs de ces services de se coordonner entre eux et avec les usagers.

DRDC Valcartier TR 2012-425 v L’interaction avec les ordinateurs est devenue de plus en plus naturelle, en particulier à cause du développement d’interfaces multimodales, permettant à un usager d’exploiter les interactions tactiles, gestuelles et vocales, en plus de s’appuyer sur l’utilisation traditionnelle de la souris et du clavier.

Les tables ou surfaces multi-tactiles permettront aux analystes de se réunir autour d’une table pour débattre une situation militaire, d’une façon similaire à l’utilisation de cartes de papier disposées sur une table. La nouvelle technologie améliorerait l’éveil situationnel partagé en permettant une interaction visuelle tout en donnant directement accès à l’information.

La technologie d’affichage se développe continuellement, fournissant des surfaces d’affichage de plus grandes tailles, de résolution accrue et de meilleure lisibilité. Les écrans peuvent même être assemblés en une mosaïque, pour fournir un mur de connaissance. Ceux-ci sont particulièrement appropriés pour les centres de commandement. En même temps, les avancés dans la technologie d'affichage ont permis de produire des périphériques d’affichage petits, légers et flexibles, très appropriés pour des opérations mobiles.

Pour améliorer le travail des analystes militaires, les postes de travail d’avant-garde devraient être examinés. Certaines des caractéristiques à considérer sont un espace de travail accru, une interaction multimodale et l'utilisation de surfaces de travail horizontales et verticales.

Au cours des dernières années, la technologie des assistants personnels, des téléphones intelligents et des tablettes numériques a évolué de façon significative et obtenu une vaste approbation de la part des utilisateurs. Aux États-Unis, une brigade a été modernisée avec ces périphériques et des applications militaires sont développées. Du travail est aussi fait pour sécuriser ces périphériques. Un certain nombre d'applications pourraient être disponibles pour du personnel déployé impliqué dans la cueillette ou l’analyse du renseignement, telles que : connaissance de localisations respectives, réalité augmentée, conférence à plusieurs, traduction culturellement aidée, assemblage d'information, surveillance de statut dynamique, calcul d’itinéraire dynamique, reconnaissance à base de biométrie et assistants virtuels.

Démontrées par le déploiement du logiciel Siri sur le téléphone intelligent IPhone d’Apple, des percées ont aussi été faites en ce qui concerne les assistants logiciels intelligents. Dans un contexte du renseignement, les assistants virtuels grandiront en importance en réduisant le problème de surcharge cognitive humaine; ils fourniront de l'aide à l'utilisateur en conduisant une grande variété de tâches, y compris la recherche et l'organisation d'information, le monitoring des gens, la gestion de calendriers, l'assignation de tâches, le résumé de documents, la médiation d'interactions, la direction et le rappel à l'utilisateur, l’apprentissage de procédures et des préférences et la suggestion d’alternatives.

La biométrie a permis l’apparition de possibilités tangibles d'améliorer la sécurité, en particulier pour authentifier l'accès des utilisateurs aux installations ou aux ordinateurs et pour identifier les belligérants.

Des avancées dans la biométrie et les interfaces neuronales démontrent la capacité de reconnaître les émotions des utilisateurs, leur niveau de stress ou leur surcharge cognitive et ceci sera utilisé pour concevoir des interfaces adaptées qui compenseront pour les limitations des usagers dans le traitement de l'information et la prise de décisions.

vi DRDC Valcartier TR 2012-425 Comme les analystes militaires devront travailler de façon collaborative, ils devraient exploiter des outils de collaboration comme la conférence audio / vidéo, le chat / la messagerie instantanée, les tableaux de collaboration numériques, mais exploiter aussi les collateurs omniprésents et la télé-présence. Les utilisateurs devraient aussi être supportés par les environnements intelligents qui sentiront, interpréteront et réagiront à l'activité humaine pour permettre une meilleure collaboration, une productivité accrue, une pensée créatrice et une meilleure prise de décisions.

Les casques de réalité virtuelle ont diminué de taille, au point d'être disponibles sous forme d’afficheur rétinien. Ils sont les dispositifs de prédilection pour les applications de réalité mixte. Ils permettent aussi une vue privée de l’information et réduisent le risque de compromettre l’information sensible.

Durant la dernière décennie, la technologie d’IHM a progressé significativement en termes de réalité mixte. Entre autres, la réalité augmentée permet aux utilisateurs militaires déployés de superposer des informations tactiques à des informations du monde réel, sur un casque, une lunette de réalité virtuelle ou un assistant personnel. Les environnements immersifs permettent au personnel militaire de développer la compréhension de situations complexes ou de répéter une mission avant être envoyé en milieu hostile. La technologie des mondes virtuels permet de s’entraîner et de collaborer.

Les analystes du renseignement et du commandement et contrôle profiteront aussi des avancées faites dans les concepts et les techniques d’IHM. En premier lieu, se trouvent la visualisation de l'information et l'analyse visuelle qui exploitent la capacité cognitive de l'homme de percevoir et de se faire une représentation mentale d'information ou d’une situation et permet de saisir la signification de grande quantités d'information. Les utilisateurs profiteront aussi de concepts pour mieux organiser l’information, interagir avec elle ou la présenter. Ceci inclut l'utilisation d'espaces de travail configurables, des interfaces utilisateur adaptatives, des composantes d'interface avancées, des loupes d’information intelligentes et des techniques narratives.

Importance:

Ce rapport est d’intérêt pour les gens impliqués dans la conception, la mise en œuvre ou l'utilisation de systèmes de commandement, contrôle et renseignement. L’IHM est un aspect clé de tels systèmes, puisqu’elle fournit la couche d'accès. Ces systèmes doivent être conçus et implantés de sorte que l'utilisateur sera motivé à les utiliser et les trouvera efficaces. Il est important de comprendre les tendances d’IHM et celles qui seraient avantageuses pour une capacité comme le CFAR.

Perspectives:

Ce rapport est une contribution à la vision du CFAR. De plus, un certain nombre d'autres documents ont été produits, y compris un livre blanc sur la définition du programme de R&D du CFAR, la production d'une illustration d'un centre d’analyse multi-sources du renseignement, l'écriture d'un récit de fiction impliquant un environnement du renseignement futur et le développement de concepts d'exploitation pour le CFAR. Suivra, une priorisation de la conception et les activités de développement, et la production d’une feuille de route. En termes de l'aspect IHM du CFAR, un certain nombre des techniques et des concepts plus prometteurs seront choisis comme pierres d’assise pour les premières versions du CFAR.

DRDC Valcartier TR 2012-425 vii This page intentionally left blank.

viii DRDC Valcartier TR 2012-425 Table of contents

Abstract ……...... i Résumé …...... i Executive summary ...... ii Sommaire ...... v Table of contents ...... ix List of figures ...... xii List of tables ...... xvi Acknowledgements ...... xvii 1 Introduction...... 1 2 Ubiquitous Computing...... 7 3 Multimodal and Natural Interaction ...... 9 3.1 Touch and Multi-touch...... 10 3.2 Multi-touch Tables / Surface Computing ...... 10 3.3 Gesture Recognition ...... 11 3.3.1 Handheld Controllers...... 12 3.3.2 Wired Glove / Data Glove ...... 12 3.3.3 Camera-based Recognition...... 13 3.4 Capacitive Touch Sensing ...... 15 3.5 Gaze/Eye Tracking ...... 16 3.6 Digital Pens ...... 18 3.7 Speech Recognition and Translation ...... 19 3.8 Emotion Recognition / Affective Computing...... 19 3.9 Biometry...... 21 3.10 Brain Computer Interfaces...... 21 4 Display Technology...... 23 4.1 Large Group Displays...... 23 4.1.1 Technology ...... 23 4.1.2 Use of Large Group Displays ...... 24 4.2 Flexible Displays...... 24 4.2.1 Flexible E-paper...... 25 4.2.2 Flexible LED Displays ...... 26 4.2.3 Flexible LCD Displays ...... 26 4.2.4 OLED Displays...... 27 4.3 Pico Projectors...... 27 4.4 Printing Technology ...... 28 4.5 Head-Mounted Displays...... 28 4.6 Advanced Workstations...... 30

DRDC Valcartier TR 2012-425 ix 5 Tablets, Personal Device Assistants and Smartphones ...... 33 5.1 Military Use in the United States ...... 35 5.2 Apps for Military Intelligence ...... 36 5.3 Mobile Data Collection ...... 37 5.3.1 ArcPad ...... 38 5.3.2 Touch Inspect...... 38 5.4 Intelligent Software Assistants for Smart Phones ...... 39 5.4.1 Siri ...... 39 5.4.2 Speaktoit Assistant ...... 40 5.4.3 Google Assistant ...... 40 6 Collaboration Technologies and Smart Room Environments...... 41 6.1 Collaboration Technologies ...... 41 6.1.1 Computer Supported Cooperative Work ...... 41 6.1.2 Pervasive Human Sensing and Remote Assistance ...... 41 6.1.3 Telepresence / Teleimmersion ...... 41 6.2 Smart Room Environments ...... 42 6.2.1 LiveSpaces ...... 43 6.2.2 LightSpace ...... 44 6.2.3 Navy Command Center of the Future ...... 45 7 Mixed Reality ...... 47 7.1 Augmented Reality ...... 48 7.1.1 Augmented Reality Using Projectors ...... 48 7.1.2 Head Mounted Display Augmented Reality ...... 49 7.1.3 Smartphone Augmented Reality ...... 51 7.2 Cave Automatic Virtual Environment (CAVE) ...... 52 7.3 Virtual Worlds ...... 53 7.3.1 T2 Virtual PTSD ...... 54 7.3.2 A-SpaceX ...... 54 7.3.3 Collaboration and Training ...... 55 7.4 360-Degree Interactive Imagery and Video ...... 56 8 Biometry ...... 59 9 Neural / Brain Computer Interfaces ...... 63 9.1 Sensors Technology and Commercial Devices ...... 63 9.1.1 NeuroSky MindSet ...... 63 9.1.2 Emotiv Systems EPOC ...... 64 9.2 Augmented Cognition - DARPA ...... 64 9.3 Neurotechnology for Intelligence Analysts - DARPA ...... 66 9.4 Epidermal Electronics ...... 66 9.5 Reconstructing Images from Brain Scans ...... 67 9.6 Brainbow Neuroimaging ...... 68 9.7 Challenges and Limitations ...... 69

x DRDC Valcartier TR 2012-425 10 Virtual Assistant / Virtual Advisor ...... 71 10.1 CALO / PAL - What Can a Virtual Assistant Do?...... 71 10.2 User Interaction with the Virtual Assistant ...... 72 10.3 Virtual Assistant Interaction with the User ...... 72 10.4 Virtual Assistant Representation ...... 74 11 Advanced HCI Concepts & Techniques...... 77 11.1 Advanced Information Visualization / Visual Analytics...... 77 11.2 Configurable Workspaces ...... 78 11.2.1 SitScape ...... 78 11.2.2 Apple’s Spaces...... 79 11.3 Adaptive User Interfaces ...... 80 11.4 Advanced Interface Widgets ...... 80 11.5 Smart Lenses / Magic Lenses ...... 81 11.6 Storytelling ...... 82 11.7 Precision Information Environments...... 85 12 Conclusion...... 89 13 References...... 91 List of symbols/abbreviations/acronyms/initialisms ...... 107

DRDC Valcartier TR 2012-425 xi List of figures

Figure 1: Illustration of a future multi-intelligence center (Poussart, 2011)...... 2 Figure 2: Hype Cycle for Human-Computer Interaction, 2011 (Gartner, 2011)...... 3 Figure 3 - A comparison of HCI styles (from Rekimoto and Nagao, 1995)...... 4 Figure 4 – Smart / assisted interaction...... 4 Figure 5: Resizing a photo using two fingers (Immich, 2010)...... 10 Figure 6: a) Flat LCD b) Projection-based c) Dual-touch skin...... 11 Figure 7: WiiMote (Nintendo, 2006)...... 12 Figure 8: a) P5 Glove (VRealities, 2009) b) ShapeHand (2009)...... 1 Figure 9 – CybergloveII (CyberGlove Systems, 2012) ...... 13 Figure 10: Xbox Kinect controller (Digital Trends, 2010)...... 13 Figure 11: 20 body point recognition with the Kinect (Bot Bench, 2011)...... 14 Figure 12: Extracts from The Leap video (Extreme Tech, 2012)...... 14 Figure 13: G-Speak gesture interaction (Oblong, 2011)...... 15 Figure 14: TLD – Video stream gesture tracking (Kalal et al., 2011)...... 15 Figure 15: Examples of Touché's capacitive touch sensing (CMU, 2012)...... 16 Figure 16: a) Head-mounted b) Remote Eye Tracker c) Monitor-embedded Eye Tracker...... 17 Figure 17: Eye-tracking Analysis Tools from Tobii Studio a) Heat Map b) Gaze Opacity c) Timeline Display (Nilson, 2011)...... 17 Figure 18: Pulse Smartpen (Livescribe, 2011)...... 18 Figure 19: a) Digital Scribe (Iogear, 2012) b) DigiMemo A502 (ACECAD Entreprise, 2011). .. 18 Figure 20: Ekman emotion recognition test (Guardian, 2009)...... 20 Figure 21: a) BARCO OV-1015 100’’ SXGA+DLP Projection Module (Barco, 2011); b) Sharp PN-V601 60 Inch LCD Video Wall (Isignpak, 2011)...... 23 Figure 22: AFRL Interactive Data Wall (AFRL, 2001)...... 23 Figure 23: Multi-touch Twitter Wall (MultiTouch, 2010)...... 24 Figure 24: A flexible display showing a map that can be rolled up and easily stored (ASU, 2012)...... 25 Figure 25: a) LG black and white e-paper (Davies, 2012) b) Sony's color e-paper (McGlaun, 2011)...... 25 Figure 26: Examples of a Flexible LED Display (LED Marketing, 2012)...... 26 Figure 27: Flexible LCD display prototype from TRADIM (Nozawa, 2012)...... 26 Figure 28: Performing a zoom-in operation on a flexible LCD display (Horsey, 2010)...... 27

xii DRDC Valcartier TR 2012-425 Figure 29: a) OLED flexible displays (Pink Tentacle, 2007) b) Samsung transparent OLED display (CES, 2009)...... 27 Figure 30: Pico projector (Microvision, 2011)...... 28 Figure 31: Zcorp Zprinter 650 and Products (Zcorp, 2012; WDV, 2012)...... 28 Figure 32: Head-mounted displays a) (Nowak, 2007) b) (Park Service, 2011)...... 29 Figure 33: Contact lens enabled eyewear (Next Big Future, 2012)...... 29 Figure 34: Field of View versus bulk of head-mounted displays (Innovega, 2012)...... 29 Figure 35: a) Zenview Multi monitor display (High Impact PC, 2011) b) Braccetto collaboration workstation (NICTA, 2011)...... 30 Figure 36: Benddesk prototype workstation (Edwards, 2010)...... 30 Figure 37: Microsoft prototype workstation...... 31 Figure 38: Illustration of the EXOdesk...... 31 Figure 39: Behind the screen overlay interactions (Microsoft Research, 2012a)...... 32 Figure 40: Smartphones with augmented reality (Zonkio, 2009; Moor, 2009)...... 33 Figure 41: Senseg's haptic touchscreen technology (Geere, 2011)...... 34 Figure 42: Tactus technology with dynamic buttons that rise up (Tactus, 2012)...... 34 Figure 43: Samsung's W960 mobile phone comes with 3D video content, generated by Dynamic Digital Depth that can be viewed without special glasses (Newitz, 2010). . 35 Figure 44: Android Toughpad (Anthony, 2011)...... 36 Figure 45: Example of the Army’s App Store, as of April 2011 (Ackerman, 2011a)...... 36 Figure 46: a) ArcPad on PDAs b) ArcPad, Mobile Component of ArcGIS (ESRI, 2010 & 2011)...... 38 Figure 47: Example of a TouchInspect Interface to Conduct Inspection of Urban/Street Assets (Jewell, 2009)...... 39 Figure 48: Examples of Speaktoit Assistant interaction (Google, 2012a)...... 40 Figure 49: DVE Telepresence system (DVE, 2011)...... 42 Figure 50: Extracts from A Day Made of Glass showing smart home and office environments (Corning, 2011)...... 43 Figure 51: DSTO’s Intense Collaboration Space, using LiveSpaces environment (Phillips, 2008)...... 44 Figure 52: LightSpace through-body object transition between existing interactive surfaces (a-b) and interactions with an object in hand (c-d) (Wilson and Benko, 2010)...... 45 Figure 53: Beamatron Setup and illustration of controlling virtual objects (Microsoft Research, 2012b)...... 45 Figure 54: Pictures of the Navy Command Center of the Future (Terdiman, 2009)...... 46 Figure 55: Virtuality Continuum (Milgram and Kishino, 1994)...... 47

DRDC Valcartier TR 2012-425 xiii Figure 56: A range of virtual reality immersion (Bernier et al., 2010)...... 47 Figure 57: a) Army Augmented Reality (Contactum, 2011) b) Mirage Augmented Reality system (Arcane Technologies, 2011)...... 48 Figure 58: LuminAR projects an interactive interface on any surface (Linder, 2010)...... 49 Figure 59: The 6th Sense project prototype augments the physical world with projected digital media information (Mistry, 2010)...... 49 Figure 60: Storyboards of how HMD-AR could be used by military personnel (Juhnke, 2010). . 50 Figure 61: Diminished reality example (Herling and Broll, 2010)...... 50 Figure 62: Just-In-Time Command and Control Center visualization system (PARVAC, 2012)...... 50 Figure 63: Project Glass prototype and concept video (Google, 2012b; Lum, 2012)...... 51 Figure 64: SoldierEyes smartphone augmented reality application (OverWatch, 2012)...... 51 Figure 65: The Flex CAVE at DRDC Valcartier (Bernier et al., 2008)...... 52 Figure 66: Immersive environments are of interest to a range of defence and security applications (Bernier et al., 2010)...... 53 Figure 67: A virtual Afghan street featured in the T2 Virtual PTSD Experience (T2 Health, 2012)...... 54 Figure 68: Example of a potential A-SpaceX workstation (PSE, 2012)...... 55 Figure 69: The I-Zone Virtual Collaboration Environment in Second Life (Tate, 2011)...... 55 Figure 70: Various Virtual World environments for military training (US DoD, 2011)...... 56 Figure 71: 3 Frames of a 360-degree interactive video (Hogg, 2010)...... 57 Figure 72: Pictures of the panoramic ball camera and its use (Burns, 2011)...... 58 Figure 73: Common features used in Biometry (Pratt, 2008)...... 59 Figure 74: Vascular pattern (vein structure) recognition device and biometric process (O’Neill, 2008)...... 60 Figure 75: Accuracy and usability of popular biometry technologies (O’Neill, 2008)...... 60 Figure 76: Collecting biometrics samples (Hunt, 2008b; Pratt, 2008)...... 61 Figure 77: Population groups to which biometry can be applied (Pratt, 2008)...... 61 Figure 78: Some of the commonly-recognized frequencies generated by different types of brain activity (Zhang et al., 2010)...... 63 Figure 79: NeuroSky MindSet is a commercial brain-computer interface (NeuroSky, 2011)...... 64 Figure 80: Emotiv EPOC neuroheadset (Emotiv, 2011)...... 64 Figure 81: Extracts from the short film ‘The Future of Augmented Cognition’ (DARPA, 2007)...... 65 Figure 82: Neurotechnology for Intelligence Analysts (NIA) project (Kruse et al., 2006)...... 66

xiv DRDC Valcartier TR 2012-425 Figure 83: Epidermal electronics prototype with physical properties matched to the epidermis (Kim et al., 2011)...... 67 Figure 84: Visual stimulus reconstruction using fMRI (Miyawaki et al., 2008)...... 68 Figure 85: Images of the brain produced using the new Brainbow technique (Livet et al., 2007)...... 68 Figure 86: a) CALO functions b) Examples of CALO learning capabilities...... 72 Figure 87: a) A humanoid ISA avatar with a text-based dialog (Wikipedia 2012b); b) A ISA Avatar embedded in the display (DARPA, 2011b)...... 74 Figure 88: A Virtual Assistant used in FOCAL (Wark and Lambert, 2007)...... 74 Figure 89: a) Visual Analytics examples: Theme River visualization (Ribarsky, 2009) b) Galaxy visualization (Future Point, 2011)...... 78 Figure 90: Example of a maritime portal on a large display with SitScape...... 79 Figure 91: a) Apple's Spaces (Tatnell, 2010) b) Mission Control (Apple, 2012)...... 79 Figure 92: The SLAP Widgets set includes a) Keypads b) Knob c) Slider d) Keyboard, and all control details are provided through the display (Weiss, 2009)...... 80 Figure 93: Ergodex configurable keyboard (Disabled Online, 2012)...... 81 Figure 94: Examples of smart lenses from Idelix (Baar and Shoemaker, 2004)...... 81 Figure 95: 3D Magic Lenses in an augmented reality setup (Mendez et al., 2006)...... 82 Figure 96: A tablet device can be used as a magic lens over a surface computing display to show additional room information in this architecture application (nsquared, 2011)...... 82 Figure 97: Stories allow for short and memorable communications (Gershon and Page, 2001). . 83 Figure 98: Staged animated transitions between chart types (Segel and Heer, 2010; Gapminder, 2005)...... 83 Figure 99: Afghanistan: Behind the Front Line (Segel and Heer, 2010; Green et al., 2009)...... 84 Figure 100: The Raconteur user interface (Chi and Lieberman, 2011)...... 84 Figure 101: Kaitlin and Sam discussing the new movie Jumper in News at Seven (Nichols and Hammond, 2009)...... 85 Figure 102: Show Me alternatives proposed by Tableau software (Mackinlay et al., 2007)...... 85 Figure 103: Illustration of Precision Information Environments concepts (DHS/PNNL, 2010). . 86 Figure 104: Illustration of Precision Information Environments concepts (DHS/PNNL, 2010). . 87

DRDC Valcartier TR 2012-425 xv List of tables

Table 1: Context of new military operations (DLCD, 2009)...... 1 Table 2: Report chapters and HCI styles cross reference ...... 5 Table 3: Emotion recognition in multimodal systems (Caridakis et al., 2007) ...... 20

xvi DRDC Valcartier TR 2012-425 Acknowledgements

The authors are very thankful to Dr. Denis Poussart for his indefectible interest in monitoring and sharing advances in Science & Technology and on trends in Human-Computer Interaction. They also wish to thank colleagues from the Intelligence and Information Section and from DRDC Valcartier for sharing some of their technology watch, in particular, Mr. Roger Fortin, Dr. Bruce Forrester, Mr. Alexandre Bergeron-Guyard, Mr. Étienne Martineau, Dr Michel Gagnon, Mr. Pierre Michaud, Dr. François Bernier, Mr. Stéphane Paradis, Mr. Christian Carrier and Dr. Guy Vézina.

DRDC Valcartier TR 2012-425 xvii This page intentionally left blank.

xviii DRDC Valcartier TR 2012-425 1 Introduction

The level of operational complexity facing (military) intelligence has hugely increased in recent times (DLCD, 2009). Recent confrontations have been largely characterized by (Poussart, 2011): x A prevalence of asymmetric warfare such as in urban environments or when dealing with rogue states or terrorism. x The emergence of cyber warfare. x The emergence of instantaneous connections between the tactical and the strategic levels, and the increased linkages between the local, regional and global spaces. x The complexity and diversification of human terrains and contexts. x The conduct of Joint, Interagency, Multinational and Public (JIMP) operations. x The continued progression of communication and information technology, which provides significant opportunities to better conduct military operations.

Table 1 depicts a number of the elements of the new context of military operations in comparison to traditional operations conducted in the Cold War era.

Table 1: Context of new military operations (DLCD, 2009).

Cold War Context New Military Context Well defined strategic context (Cold War) Poorly defined strategic context (War on terror) Static theatre of operations Multiple theatre of operations Single spectrum operation Full spectrum engagement Well defined adversary Elusive and changing adversary Technologically predictable enemy Technologically innovative enemy Structured enemy forces Network enemy forces Corps construct Battlegroup construct Rigid and concentrated forces Adaptive and dispersed forces Long term evolution cycle Very short term evolution cycle Limited third party considerations Crowded JIMP environment Controlled infosphere Uncontrollable infosphere

The purpose of intelligence is to provide commanders and staffs with timely, relevant, accurate, predictive, and tailored intelligence about the enemy and other aspects of the area of operations. Intelligence supports the planning, preparing, execution, and assessment of operations. (Department of the Army, 2010).

Military analysts need to assess, understand and predict the development of a situation. To this end, they need to handle a continuously growing set of information of different primitive data types and of different sources. Sources include: communication and electronic signal information, geospatial intelligent, imagery, meteorological and oceanographic information, human intelligence, open-source intelligence, as well as information from other government departments and allied countries.

DRDC Valcartier TR 2012-425 1 The data sets to be analyzed “present challenges not only because of their data diversity, volume and dynamic nature but also because data contain errors and are ambiguous, incomplete, uncertain and potentially deceptive. Data of multiple types must often be analysed in concert to gain insight” (Robertson, 2009).

With an ever-expanding set of sensing modalities and sophisticated sensing platforms, the analysts are faced with an increased volume of data and information to collect and process, resulting in data and cognitive overload. There is a need for more efficient information management and analysis capabilities, as well as better Human-Computer Interaction (HCI) capabilities to support collaboration and interaction with information. These enhanced capabilities must be provided both for analysts in command centers as well as for deployed collators or analysts.

In order to address the new context of military operations, from an intelligence perspective, Defence R&D Canada (DRDC) has undertaken a major initiative to look forward into Future Intelligence Analysis Capabilities (FIAC). The core objective of the FIAC Program is to advance the state-of-the-art of intelligence methods and resources in order to adapt to and anticipate the new challenges of the Canadian Forces (CF). FIAC addresses information services from an all sources perspective, over the full intelligence cycle but with particular emphasis on advanced analysis capabilities. It seeks, in particular, to achieve synergy between human cognition and machine intelligence, fully exploiting collaborative interaction. FIAC seeks to address the demands of intelligence in the modern operating terrain and the expanding uncertainties that the CF are facing.

Over the last year, a number of initiatives have been conducted to set the stage for the FIAC. This includes the development of a white paper (Poussart, 2011), the production of an illustration of a future multi-intelligence center (Figure 1), the writing of a fiction narrative of a future intelligence environment, and the development of operating concepts. One of the key concepts resulting from these initiatives is that the FIAC “seeks, in particular, to exploit an optimized level of synergy between human cognition and machine intelligence, using advanced, collaborative interaction. Its architecture is designed to support distributed, not necessarily co-localized exploitation while maintaining a single, unified and coherent informational space” (Poussart, 2011).

Figure 1: Illustration of a future multi-intelligence center (Poussart, 2011).

This technical report provides an assessment of advances in HCI that could be exploited in future intelligence environments and the FIAC. These environments should support productive collaboration among analysts with various backgrounds and expertise (multi-intelligence), support collaboration under a JIMP framework, and support multiple parallel theatres of operations and deal with several intelligence requests.

2 DRDC Valcartier TR 2012-425 The design of HCI should take into account three interaction aspects: physical, cognitive, and affective. “The physical aspect determines the mechanics of interaction between human and computer while the cognitive aspect deals with ways that users can understand the system and interact with it. The affective aspect is a more recent issue and it tries not only to make the interaction a pleasurable experience for the user but also to affect the user in a way that make user continue to use the machine by changing attitudes and emotions toward the user” (Karray et al., 2008).

The field of human-machine interface continues to go through rapid changes with the introduction of new multi-sensory interfaces (speech, sound, haptics) and metaphors (gestures, avatars in augmented or virtual reality worlds, shared cognitive spaces). Large interactive displays, smart devices and embedded systems become more and more pervasive.

Figure 2 shows the status of HCI technology on the hype cycle, showing emerging technologies as well as indicating which technologies have gone through convincing focused experimentation (slope of enlightenment) and have found adoption (plateau of productivity) (Gartner, 2011).

Figure 2: Hype Cycle for Human-Computer Interaction, 2011 (Gartner, 2011).

HCI technologies not only support the interaction between users and the information held in a computer, but also the interaction between users through the use of computing applications. Moreover, a number of HCI technologies also allow users to interact with the real world, using computer augmented environments (Wellner et al., 1993).

Rekimoto and Nagao (1995) have illustrated the different HCI styles that support the various interactions (Figure 3). Figure 3a illustrates the Graphical User Interface (GUI) type between a user and a computer. This interaction is isolated from the user interaction with the real world. Figure 3b illustrates Virtual Reality (VR) where the computer surrounds the user completely disabling any interaction with the real world. Figure 3c illustrates ubiquitous computer environments, where the user interacts with the real world but can also interact with computers embodied in the real world. Finally, Figure 3d represents augmented interaction or Augmented DRDC Valcartier TR 2012-425 3 Reality (AR), where users are able to interact with the real world, but using computer augmented information.

Figure 3 - A comparison of HCI styles (from Rekimoto and Nagao, 1995).

Some user interactions are missing from these HCI styles. A proposed extension to the HCI styles is presented in Figure 4. This figure illustrates users supported by a smart room environment. They can collaborate directly human to human or through computing applications. As part of the environment is also the provision of virtual analyst or an avatar, that may behave like a human but is a computer application.

Figure 4 – Smart / assisted interaction.

All of these HCI styles can be put to contribution in future intelligence environments. The report is organized along the following themes: 4 DRDC Valcartier TR 2012-425 Table 2: Report chapters and HCI styles cross reference

Chapter HCI technology Ubiquitous Ubiquitous computing GUI Virtual Reality Augmented interaction / Smart assisted interaction 2 Ubiquitous computing x 3 Multimodal and Natural xx Interaction 4 Display technology x 5 Tablets, personal device xx x x assistants and smartphones 6 Collaboration technologies xx and smart room environments 7 Mixed reality x x 8 Biometry x x 9 Neural / Brain computer xx interfaces 10 Virtual assistant / Virtual x advisor 11 Advanced HCI concepts & xx techniques

DRDC Valcartier TR 2012-425 5 This page intentionally left blank.

6 DRDC Valcartier TR 2012-425 2 Ubiquitous Computing

Ubiquitous computing has been named the Third Wave of computing. The First Wave was the mainframe era, many people - one computer. Then it was the Second Wave, one person - one computer which was called PC era and now ubiquitous computing introduces many computers - one person era (Riva, 2005). People are able to interact with the technology that surrounds them in more accessible, intuitive and less restrictive ways.

In a growing ubiquitous computing world, “computers will communicate through high speed local networks, over wide-area networks, and via infrared, ultrasonic, cellular, and other technologies. Data and computational services will be portably accessible from many if not most locations to which a user travels... Computation will pass beyond desktop computers into every object for which uses can be found” (Wikipedia, 2010).

Closely related to ubiquitous computing, location-aware technology includes sensors and methods for detecting or calculating the geographical position of a person, a mobile device or other moving objects. The most common localisation technologies are Global Positioning System (GPS), Assisted GPS, Enhanced Observed Time Difference, and Enhanced GPS. High-accuracy location technologies for private WiFi networks also provide location services within buildings and hot spots (Gartner, 2011). Location-aware technology is key for situation awareness in the conduct of military operations.

DRDC Valcartier TR 2012-425 7 This page intentionally left blank.

8 DRDC Valcartier TR 2012-425 3 Multimodal and Natural Interaction

“A multimodal interface acts as a facilitator of human-computer interaction via two or more modes of input that go beyond the traditional keyboard and mouse. The exact number of supported input modes, their types and the way in which they work together may vary widely from one multimodal system to another. Multimodal interfaces incorporate different combinations of speech, gesture, gaze, facial expressions and other non-conventional modes of input” (Oviatt, 2003). Multimodal interaction also involves various modalities to provide feedback to the user in his interaction, such as visual or auditory cues.

Multimodal user interfaces support multiple modes of interaction that can be combined: x Input modes: keyboard/keypad, mouse, pointing devices, pens, touch, multi-touch, haptics, gesture, gaze/eye tracking, head and body movements, speech, emotion detection, biometry, neural/brain. x Output modes: displays (text, graphical, images, videos, animations, avatars), speech, audio (auditory icons, music), haptics, olfaction (smell).

Here are day-to-day examples of the use of multimodal user interfaces: x Using a cell phone, a user is allowed to enter information using a combination of keyboard and voice; the cell phone can display a photograph of the caller; x A GPS displays the map and spokes out the driving instructions to a certain destination; x In an arcade game, the user is immersed in a virtual reality environment, seeing a realistic display of a racing track, hearing songs, manipulating a pseudo motor bike, feeling vibrations and the stiffness of the handlebars, and having to respect laws of physics in his driving; x A user with severe impairments is able to operate a number of devices (e.g. television, computer, wheel chair), using various input modes, such as speech and gaze.

There are six main benefits to multimodal interfaces: x “Efficiency follows from using each modality for the task that is best suited for; x Redundancy increases the likelihood that communication proceeds smoothly because there are simultaneous references to the same issue; x Perceptibility increases when the tasks are facilitated in spatial context; x Naturalness follows from the free choice of modalities and may result in a human-computer communication that is close to human-human communication; x Accuracy increase when another modality can indicate an object more accurately than the main modality; x Synergy occurs when one channel of communication can help refine imprecision, modify the meaning, or resolve ambiguities in another channel.” (Maybury & Wahlster, 1998).

The following sections provide a state-of-the-art review of the most promising input and output modes.

DRDC Valcartier TR 2012-425 9 3.1 Touch and Multi-touch

Touch interaction refers to a technique where a user is able to interact with a touch screen (e.g. screen, table, wall) or a touchpad. The sense of touch is proximal: it senses objects that are in contact with the body, and it is bidirectional in that it supports both perception and acting on the environment (Wikipedia, 2009a). In general, the interaction is done using a finger or a pen. It is an efficient way to easily select menu items, interact with objects or draw simple graphics on a screen. In recent years, technology has moved to multi-touch interaction.

”Multitouch refers to a touchscreen interaction technique in which multiple simultaneous touchpoints and movements can be detected and used to control objects in a user interface or other application. A user may, for example, zoom into a picture or Web page by placing thumb and index finger on a touchscreen and then moving them apart. To zoom back out, the user would then move the same two fingers back together (Gartner, 2010)”. Multi-touch interaction is achieved through a variety of means, including but not limited to: heat, finger pressure, high capture rate cameras, infrared light, optic capture, tuned electromagnetic induction, ultrasonic receivers, transducer microphones, laser rangefinders, and shadow capture (Pennock et al., 2008).

“A multi-touch user interface is becoming a must-have feature of smartphones, media tablets and portable consumer electronics devices… Multi-touch will change use models and will have a significant impact on product design” (Gartner, 2010).

3.2 Multi-touch Tables / Surface Computing

“Surface computers are large-screen displays that support direct interaction via touch or gesture (as opposed to external devices, such as mice or keyboards). They are typically horizontal, often built into the furniture, such as a table top, but may be delivered as vertical wall-mounted or free- standing displays. The displays incorporate much of the style of interaction (such as rotate, pinch, zoom and flick movements) found in multi-touch devices but can typically recognize more than one set of touches at a time, enabling multiple users to interact or work collaboratively” (Gartner, 2011). As seen on Figure 5, a person uses two fingers on two opposite corners of a photo and spread them apart to enlarge a photo.

Figure 5: Resizing a photo using two fingers (Immich, 2010).

Figure 6 shows examples of multi-touch platforms: a 55-inch high definition Liquid Crystal Display (LCD) from Ideum which can support multiple simultaneous touch points (GestureWorks, 2012); a 47-inch HD multi-touch table from Evoluce which uses projection and allows gesture-control from up to half a meter from the device (Chawla, 2011); a dual-touch skin

10 DRDC Valcartier TR 2012-425 from Displax that can be overlaid over existing displays to provide a multi-touch capability (Displax, 2012).

Figure 6: a) Flat LCD b) Projection-based c) Dual-touch skin.

Some multi-touch tables “also have the capability to recognize physical objects marked with a special ID tag, allowing context-sensitive information to be provided when items are placed on the display” (Gartner, 2010). Various techniques are available to track which individual makes each gesture so that roles, responsibilities and privileges specific to individuals can be supported, Techniques include instrumenting the users (Dietz and Leigh, 2001), using pens (Scott et al., 2010), smart phones (Schmidt et al., 2010a), tracking users’ locations around the tabletop (Dohse, 2008), orientation of fingers (Dang et al., 2009) or hand contour (Schmidt et al., 2010b). Using the integrated infrared cameras, the system recognizes the forms of objects put down on the surface of the touch-sensitive screen. So by putting down simply his bank card, one could signify his payment of his consumptions in a bar (Mimi, 2009).

“The size of the multi-touch tables is constrained by the ability to physically reach across the surface. Larger displays may require a noncontact approach involving a gestural interface, where the user does not need to physically touch the surface” (Gartner, 2010).

In a military intelligence context multi-touch tables will likely be very valuable allowing various analysts to come together around the table to debate a military situation. Multi-touch tables provide a number of HCI benefits: users can gather around the display surface and eye-to-eye interaction is provided; at all times, all collaborators have equal opportunity to participate in the analysis, as control is neither centralized nor passed from person to person; information is at the users’ fingertips; multi-touch interaction can improve user efficiency for several tasks; it brings familiarity to the military users as the display of maps on the multi-touch table can provide some similarity to the laying out of paper maps on tables; there is a potential to ‘instrument’ the users and provide adapted interfaces; and globally, this can lead to improved shared awareness.

3.3 Gesture Recognition

“Gesture recognition involves determining the movement of a user's fingers, hands, arms, head or body in three dimensions through the use of a camera or via a device with embedded sensors that may be worn, held or body-mounted” (Gartner, 2011).

The primary application for gestural interfaces at present is in the gaming and home entertainment market. However, for the military, the potential of hands-free control of devices, and the ability for several people to interact with large datasets, opens up a wide range of business applications - including data visualization and analytics, the interaction with Large Group Displays (LGDs) (see Section 4.1) and the interaction within Smart Room Environments (see Section 6.2).

DRDC Valcartier TR 2012-425 11 3.3.1 Handheld Controllers

“The commercialization of gesture interfaces began with handheld devices that detect motion, such as the Nintendo Wii's 3D controller (Figure 7), 3D mice and high-end mobile phones with accelerometers” (Gartner, 2010). The Wii MotionPlus is an expansion device for more accurate complex motion capture. Sony has also revealed its alternative, called Move, which uses a handheld controller similar to the Nintendo Wii device, but claims greater resolution.

Figure 7: WiiMote (Nintendo, 2006).

3.3.2 Wired Glove / Data Glove

A wired glove is a glove-like input device for virtual reality environments. “Various sensor technologies are used to capture physical data such as bending of fingers. Often a motion tracker, such as a magnetic tracking device or inertial tracking device, is attached to capture the global position/rotation data of the glove. These movements are then interpreted by the software that accompanies the glove, so any one movement can mean any number of things” (Wikipedia, 2009b).

As examples, the P5 Glove (Figure 8a) is a glove-like peripheral device, based upon proprietary bend sensor and remote tracking technologies, that provides users total intuitive interaction with 3D and virtual environments, such as games, websites and educational software (VRealities, 2009).

ShapeHand (Figure 8b) from Measurand ShapeHand is a portable, lightweight hand mocap system of flexible ribbons that captures hand and finger motion. ShapeHand's flexible sensors are not physically built into a glove so it can attach to most hand sizes (ShapeHand, 2009).

Figure 8: a) P5 Glove (VRealities, 2009) b) ShapeHand (2009).

12 DRDC Valcartier TR 2012-425 Cyberglove II, from CyberGlove Systems, is a motion capture data glove that “is fully instrumented with up to 22 high-accuracy joint-angle measurements. It uses proprietary resistive bend-sensing technology to accurately transform hand and finger motions into real-time digital joint-angle data” (CyberGlove Systems, 2012)

Figure 9 – CybergloveII (CyberGlove Systems, 2012)

3.3.3 Camera-based Recognition

Camera-based systems are now entering the market. One of the most visible examples is the newly launched Microsoft Kinect gaming controller for the Xbox 360 (Figure 10). Using a webcam-style peripheral, it allows users to control and interact with the Xbox 360 through a natural user interface using gestures and spoken commands, and without the need to touch a game controller.

Figure 10: Xbox Kinect controller (Digital Trends, 2010).

Although it was intended for game play, developers saw the huge potential this low cost sensor could have for computer interaction and an open source PC driver was developed by hackers within days (hours) of its launch. Microsoft realized all this creativity and the potential for the Kinect as a motion controller for the PC. As of February 2012, Microsoft released the new Kinect for Windows hardware and accompanying software. They have chosen a hardware-only business model, meaning that the software development kit or the runtime will be available freely to developers and end-users (MSDN Blogs, 2012).

Kinect is capable of simultaneously tracking up to six people, including two active players for motion analysis with a feature extraction of 20 joints per player (Figure 11), focused on the head, hands, feet and torso. However, Kinect is capable of tracking more points than those 20 points. For instance, MIT’s Robot Locomotion Group and Learning Intelligent Systems Teams have shown that the Kinect can recognize ten fingers and some relatively minute gestures, similar to the interaction shown in the Minority Report movie (Nosowitz, 2010). Kinect also supports vocal interactions using key words (vocal interactions are discussed in Section 3.7).

DRDC Valcartier TR 2012-425 13 Figure 11: 20 body point recognition with the Kinect (Bot Bench, 2011).

LeapMotion has introduced a device similar to the Kinect, but able to track hand gestures down to an accuracy of 0.01 millimetres. It can also track objects, such as a pencil. The Leap is available to pre-order now for $70, and is expected to ship early in 2013 (Extreme Tech, 2012). Figure 12 illustrates some of its capabilities, using extracts from The Leap video.

Figure 12: Extracts from The Leap video (Extreme Tech, 2012).

A spin-off from the Minority Report movie, the G-Speak spatial computing operating system stands at the higher end of the market. It was developed by John Underkoffler at MIT Media Lab and now commercialized by Oblong Industries Inc. (Oblong, 2011). In addition to the camera- based gesture interaction, this system offers a management capability that allows data to be moved between different computing systems and displays.

14 DRDC Valcartier TR 2012-425 Figure 13: G-Speak gesture interaction (Oblong, 2011).

Kalal et al. (2011) have developed interesting computer tracking software to recognize moving objects but also gestures. They developed a real-time tracking algorithm, known as TLD (Tracking, Learning, Detection), which tracks unknown objects in video streams. The object of interest is defined by a bounding box in a single frame. As the acronym spells out, “TLD simultaneously Tracks the object, learns its appearance and detects it whenever it appears in the video. The result is a real-time tracking that typically improves over time” (Kalal, 2012). TLD can be used to discriminate different users and track their input. For example, a user can define the junction of three fingers as the object to be tracked (Figure 14).

Figure 14: TLD – Video stream gesture tracking (Kalal et al., 2011).

3.4 Capacitive Touch Sensing

A team formed of individuals from Disney Research, Pittsburgh and Carnegie Mellon University (CMU) has developed a new sensing technique, called Touché, based on swept frequency capacitive sensing, the same principle that underlines the types of touchscreens used in smartphones. “But instead of sensing electrical signals at a single frequency, like the typical touchscreen, Touché monitors capacitive signals across a broad range of frequencies” (CMU, 2012). This allows not only to detect a ‘touch event’ but also sense how the object is being touched or the body configuration of the person doing the touching.

They ran five experimental setups to show the various kinds of gesture recognition that could be achieved (CMU, 2012):

x Doorknob gesture. The system could differentiate the following kind of gestures: One finger touch, two finger pinch, a circle gesture, a grasp and no touch. This could be used for example to implement a ‘smart doorknob’ that could trigger a specific action, for example leaving a different message on an ambient display shown by the door, such as ‘Do not disturb’ or ‘Back in five minutes’.

x Table gesture sensing. The system could detect the body posture on a table (e.g. one hand, two hands, one elbow, two elbows, arms flat, and absent).

DRDC Valcartier TR 2012-425 15 x Enhancing touch screen interaction. The system could recognize the following pinching gesture set on a smartphone: thumb, one finger pinch, two finger pinch, all finger pinch, QRWRXFK7KLVFRXOGIRUH[DPSOHEHXVHGWRLPSOHPHQWޒAlt-&OLFNޓRQDPRELOHGHYLFH touch screen.

x Hand gesture sensing. Touché does not require metallic objects and can even add touch sensitivity to the human body. It could detect: one finger, five fingers, grasp, cover ears, no touch. This could be used by someone for example to control his mp3 player using hand gesture.

x Sensing gestures in liquids. Touch sensitivity to liquid can be added easily. It can detect for example three finger tips, one finger bottom, hand submerged, or away.

These experimental setups are illustrated in a video (CMU, 2012). Extracts for the video appear in Figure 15.

Figure 15: Examples of Touché's capacitive touch sensing (CMU, 2012).

3.5 Gaze/Eye Tracking

An eye tracker is a device that allows measuring eye positions and movements, in order to determine where one is looking (the point of gaze), how long he is looking and his eye activities. It can allow determining what is seen and what is not seen, if the user is focused and engaged, or confused. Eye trackers are used in a number of domains, including scientific research (e.g. cognition, psychology, vision, HCI), design, usability testing and training, and assistance for people with disabilities. They can also be used in conjunction with virtual assistants (see Section 10) and augmented cognition (see Section 9.2). There are a number of methods to implement eye trackers. “The most popular variant uses video images from which the eye position is extracted... Eye movements are typically divided into fixations and saccades, when the eye gaze pauses in a certain position, and when it moves to another position, respectively... Eye tracking setups vary greatly; some are head-mounted, some require the head to be stable (for example, with a chin

16 DRDC Valcartier TR 2012-425 rest), and some function remotely and automatically track the head during motion” (Wikipedia, 2009c).

Figure 16 provides examples of eye trackers: a head-mounted eye tracker (Li, 2006); a remote eye tracker (Nilson, 2011) and an eye tracker embedded in a monitor (Nilson, 2011). One advantage of the last two devices is that the subject is not required to wear headgear of any kind.

Figure 16: a) Head-mounted b) Remote Eye Tracker c) Monitor-embedded Eye Tracker.

Eye tracking shows high value when hands and voice are not an option for controlling a device (for disable persons for instance); it can also be of value as a complement in a multimodal interaction environment. Eye-tracking interfaces may be of value for some users “by speeding some tasks such as icon selection and windows zooming in graphical user interfaces or object selection in virtual reality” (Li, 2006).

In his thesis, Kumar has explored the use of gaze-based input for pointing and selection, application switching, password entry, scrolling, zooming, and document navigation. He has demonstrated that “gaze information can be used as a practical form of input” (Kumar, 2007). At the heart of the technology he developed is the EyePoint software. This allows for example a user to look at a web link and press a ‘hot key’ on the keyboard, causing the system to open the corresponding web page. This would correspond to the same action of moving the cursor to this web link and clicking on the link.

But more importantly, eye tracking shows value in the ability to capture and support the analysis of the user interaction with a system. It is been used in particular in the advertizing and interface design industries. To support the analysis, Tobii has developed a number of eye-tracking tools, as part of their Tobii Studio software. This includes, recording tools (e.g. event logging, keystrokes and mouse clicks), timeline display (e.g. gaze replay, scene segmentation, video clips), and visualization (e.g. gaze plots, heat maps, statistic charts). Figure 17 provides examples of the Tobii Studio environment (Nilson, 2011).

Figure 17: Eye-tracking Analysis Tools from Tobii Studio a) Heat Map b) Gaze Opacity c) Timeline Display (Nilson, 2011).

DRDC Valcartier TR 2012-425 17 In a military context, eye-tracking could therefore be used in support of computer interface design and validation. It could be used to see if the user is confused. Finally, it could be used as part of a multimodal interaction.

3.6 Digital Pens

A digital pen is an input device which captures the handwriting of a user and converts into digital data. The data can then be interpreted using Optical Character Recognition (OCR) software or kept as graphics, to be used in various applications. Digital pens comprise a number of electronic components and exhibit a number of features, including input buttons, touch sensitivity, built-in memory, microphone, digital camera, image-processing unit data transmission capabilities, and electronic erasers. Some contain an ink cartridge that allows the user to see what he has written or drawn. The digital pen looks and feels like an ordinary ballpoint pen and the user uses it in the same way (Wikipedia, 2011a).

Examples of digital pens are shown in the following figures: the Pulse Smartpen from Livescribe (2011) (Figure 18); The Digital Scribe from Iogear (Iogear, 2012) (Figure 19a); the DigiMemo A502, from ACECAD Entreprise (2011) (Figure 19b).

Figure 18: Pulse Smartpen (Livescribe, 2011).

Figure 19: a) Digital Scribe (Iogear, 2012) b) DigiMemo A502 (ACECAD Entreprise, 2011).

Some of the advantages of digital pens are the following: conversion of handwriting into text accurately; no more scanning; automatic electronic format archiving; data validation at the point of entry; speed up processing; low cost and fast implementation; high user acceptance (little or no training, it’s pen and paper); and connectivity to other systems (Bluetooth wireless and USB) (Magicomm, 2011). Still, “handwriting recognition, is still struggling to extend beyond relatively

18 DRDC Valcartier TR 2012-425 niche applications” (Gartner, 2010). In a military context, digital pens could be used by military deployed to rapidly gathering observations or filling information from the field into forms.

3.7 Speech Recognition and Translation

Speech recognition converts spoken words to machine-readable input while voice recognition is a system trained to a particular user, which recognizes their speech based on their unique vocal sound. Transcription is the conversion of a spoken-language source into written / typewritten. Automatic translation is the conversion of a spoken-language source into another language (Wikipedia, 2011b).

Speech Recognition technology is pretty mature, although accuracy can be highly variable depending on background noise, size of the recognized vocabulary, level of natural-language understanding attempted, clarity of the speaker's voice, quality of the microphone and processing power available.

In terms of speech-to-speech translation, a number of low cost applications have been commercialized for consumer and tourist applications, where approximate translations are often good enough. However, speech-to-speech translation should not be viewed “as a replacement for human translation but rather as a way to deliver approximate translations for limited dialogues where no human translation capability is available” (Gartner, 2011).

Speech recognition can have large applicability in a military intelligence setting where input from deployed observers and collators are digitized or as part of collaboration activities where automated transcripts of discussions are produced. Considering that intelligence analysts need to digest messages and documents written in various languages, automatic translation has also a significant applicability. It also has applicability to the deployed collators interacting with local populations, using low-costs apps developed for the smart phones.

3.8 Emotion Recognition / Affective Computing

People are able to perceive one’s emotional state based on their observations about one’s face, body, and voice. Research in multimodal systems have been conducted to allow the inferring of one’s emotional state based on various behavioral signals. A bimodal system based on fusing the facial recognition and acoustic information, provided an accurate classification of 89.1 percent in terms of emotion recognition of “sadness, anger, happiness, and neutral state” (Karray et al., 2008). Figure 20 shows Ekman emotion recognition test, in which nine different emotions are expressed, including slight enjoyment, masked anger, fear and apprehension.

DRDC Valcartier TR 2012-425 19 Figure 20: Ekman emotion recognition test (Guardian, 2009).

Table 3, reproduced from (Caridakis et al., 2007), describes (on the left-hand side) eight categories of emotions, their valence (the positive or negative character of an emotion), arousal (the degree of excitement of an emotion) and what typical gestures characterize those emotions. On the right-hand side, is the confusion matrix of the multimodal emotion recognition, using face, body and speech recognition together. We can observe that ‘Joy’ is recognized 93.33% of the time, but confused 6.67% of the time with ‘Pride’. “The diagonal components reveal that all the emotions, apart from despair, can be recognised with more than the 70 % of accuracy” (Caridakis et al., 2007). Table 3: Emotion recognition in multimodal systems (Caridakis et al., 2007)

Zeng et al. (2009) provides an interesting survey of the state of the research applied to emotion recognition: “an increasing number of efforts are reported toward multimodal fusion for human affect analysis, including audiovisual fusion, linguistic and paralinguistic fusion, and multicue visual fusion based on facial expressions, head movements, and body gestures”.

In a military intelligence context, emotion recognition may provide valuable clues about a person’s stress level, intent, or thrust level. This may be very important in a JIMP context where interaction takes place with people from various organizations, often through audio / video collaboration or in a context where interaction takes place with elusive and changing adversary. Moreover, using affective computing, the human-computing interfaces would be able to recognize, and adapt to the emotional states of the users.

20 DRDC Valcartier TR 2012-425 3.9 Biometry

Biometry is a form of input that deals with the identification and verification of people. This is particularly relevant in military intelligence. This type of interaction is described in a separate section (Section 8).

3.10 Brain Computer Interfaces

The most futuristic types of interaction are without doubt the brain-computer interfaces. A full section is thus devoted to this topic. It is covered in detail in Section 9.

DRDC Valcartier TR 2012-425 21 This page intentionally left blank.

22 DRDC Valcartier TR 2012-425 4Display Technology

4.1 Large Group Displays

LGDs are very common in intelligence centers.

4.1.1 Technology

Technology for LGDs is continuously evolving, resulting in the availability of larger and more capable displays at lower costs as well as new interaction and collaboration technologies, such as the use of multi-modal interaction and more comprehensive support for co-located and distributed collaboration. Figure 21 a) and b) show respectively a rear-projection display and a mosaic of LCD panels.

Figure 21: a) BARCO OV-1015 100’’ SXGA+DLP Projection Module (Barco, 2011); b) Sharp PN-V601 60 Inch LCD Video Wall (Isignpak, 2011).

The Interactive DataWall (Figure 22) developed at the Air Force Research Laboratory (AFRL) is a good example of how multi-modal interaction can apply to LGDs. It is built using three horizontally tiled video projectors that each displays 1280 x 1024 pixels for a combined resolution of 3840 x 1024 pixels across a 12' x 3' screen area. The system also features speaker- independent voice activation and a wireless pointing device using camera tracked laser pointers (AFRL, 2001).

Figure 22: AFRL Interactive Data Wall (AFRL, 2001).

DRDC Valcartier TR 2012-425 23 In their October 2010 report “The Era of Touch Interface is Upon Us” (GP Bullhound, 2010), a list of several key attributes for assessing different touch technologies is provided. This includes (MultiTouch, 2011): x Sensitivity to stylus, fingers and gloves x Multi-touch support x High levels of clarity and transparency x No need for calibration x Substrate independence x Low cost x Low power consumption x High durability x Flush (100% flat) surface x Narrow borders

Figure 23 shows multiple people interacting simultaneously with information on a multi-touch wall.

Figure 23: Multi-touch Twitter Wall (MultiTouch, 2010).

4.1.2 Use of Large Group Displays

For decades, command centers have been equipped with LGDs in order to provide shared situation awareness and support collaborative working. However, these technologies are not always deployed and used effectively. In many cases, these centers are put together within constrained timelines, equipment is procured but not sufficient time is devoted to the design of the LGDs. The important questions such as ‘Who are the LGDs intended for?’, ‘What information should be displayed? or ‘How is the information shown controlled?’ are not addressed. The LGDs do provide a ‘wow’ factor to the command centers but they end up being under-utilized.

Gouin et al. (2009) provide a number of human factor guidelines in terms of the design and use of LGDs. This included human engineering (e.g. organizational, team and cognitive considerations, legibility guidelines, layout considerations), interaction and collaboration (e.g. multi-modal interaction, alerts and notifications, local and distributed collaboration), organization of displayed information (e.g. content design heuristics) and content control.

4.2 Flexible Displays “Flexible displays are the next revolution in information technology that will enable lighter- weight, lower-power, more-rugged systems for portable and vehicle applications” said Brig. Gen.

24 DRDC Valcartier TR 2012-425 Roger Nadeau, commanding general of the Army’s Research, Development and Engineering Command, in 2004, at the opening of the Army Flexible Display Center, at the Arizona State University (ASU, 2012). The purpose of the Flexible Display Center is to spearhead the next revolution in information displays, allowing real-time information sharing through ubiquitous commercial and military application in everything from portable pocket-held and vehicle- mounted devices to permanent and temporary conferencing/command rooms.

Figure 24 illustrates a flexible display, based on OLED technology (see Section 4.2.4), that shows a rollable map. This section describes various flexible display technologies.

Figure 24: A flexible display showing a map that can be rolled up and easily stored (ASU, 2012).

4.2.1 Flexible E-paper

E-paper screens are made of a flexible plastic substrate. They are lightweight and can be bent without breaking. They are particularly suited to the production of mobile devices and to the world of eReaders. “The tech uses little power allowing the reader to run longer and it is very easy to read outside in direct sunlight. The tech also has some clear limitations too. It’s not capable of showing video and most of the screens are black and white only. There are some color offerings in the works from firms like Triton, but the colors are muted compared to an LCD” (McGlaun, 2011). Figure 25a shows LG Display’s 6-inch flexible e-paper, which provides a 1024 x 768 monochrome sheet, and can be bent up to 40-degrees without breaking. Figure 25b is a color e-paper product in development from Sony. Despite its full color and video motion limitations, flexible e-paper would still have some applicability in the military, in particular for mobile devices and wearable computing.

Figure 25: a) LG black and white e-paper (Davies, 2012) b) Sony's color e-paper (McGlaun, 2011). DRDC Valcartier TR 2012-425 25 4.2.2 Flexible LED Displays

Flexible LED Displays are composed of rubber strips in which the high illumination Light Emitting Diode (LED) is embedded. The resulting display is light weight, slim body and easy to install. It is soft and flexible, which is suitable to setup in any irregular place. It can be transparent, waterproof and wind-proof, so lighting and wind can go through the body of the screen. It provides high brightness and can be used in a sunny environment. Many small panels can be connected together vertically and horizontally to make a bigger LED wall (LED Alive Show, 2012). All of these characteristics are of interest to the military, as mobile command posts have to be deployed rapidly in a flexible manner. Figure 26 shows examples of a small flexible LED display being setup.

Figure 26: Examples of a Flexible LED Display (LED Marketing, 2012).

4.2.3 Flexible LCD Displays

LCDs are the display medium used in most smartphone, tablet and laptop screens, as well as many desktop monitors and high-definition (HD) TVs, today. LCDs use a backlight to force light through tiny openings in a switchable liquid crystal layer, a filter layer adds color. “A liquid crystal medium is not suitable for use in flexible displays because bending the LCD layer tends to distort the light patterns that make up the display image” (Mitchell, 2011).

However, the Technology Research Association for Advanced Display Materials (TRADIM) has been working on a flexible LCD display prototype, formed on a film substrate, by using a roll-to- roll method. The display can be bent to a curvature radius of about 15 cm (almost six inches) without breaking, snapping or cracking, even at high temperature and humidity rates. The prototype measures 8.9 cm (3.5-inches) in diagonal and 0.49 mm in thickness, featuring a pixel density of 114 ppi (Figure 27) (Nozawa, 2012).

Figure 27: Flexible LCD display prototype from TRADIM (Nozawa, 2012).

26 DRDC Valcartier TR 2012-425 Toshiba has demonstrated user input as an interesting use for flexible LCD technology (Horsey, 2010). By bending the display, the user is able to perform a zoom-in or zoom-out operation in the Google Earth application, as illustrated in Figure 28.

Figure 28: Performing a zoom-in operation on a flexible LCD display (Horsey, 2010).

4.2.4 OLED Displays

OLED (Organic Light Emitting Diode) is a flat display technology, made by placing a series of organic thin films between two conductors. OLEDs are called organic because they are made from carbon and hydrogen. When electrical current is applied, a bright light is emitted. Because OLEDs produce (emit) light they do not require a backlight.

Some key advantages for the military use of OLED displays over today's flat-panel technology (LCD or plasma) are that they can be ultra-thin, flexible (Figure 29a) and transparent (Figure 29b). They also have low power consumption, a greater brightness, a fuller viewing angle and can operate in a broader temperature range. This provides the potential for curved OLED displays, placed on non-flat surfaces; wearable OLEDs; and transparent OLEDs embedded in windows or truck windshields (OLED Info, 2012).

Figure 29: a) OLED flexible displays (Pink Tentacle, 2007) b) Samsung transparent OLED display (CES, 2009).

4.3 Pico Projectors

“Pico projectors are very small projector modules which can be integrated into mobile devices such as handsets or laptops, or used to create highly portable projector accessories for mobile workers” (Gartner, 2011). An example is shown in Figure 30. This type of device may be especially useful for sharing information that is displayed on a small smartphone or mobile device screen to a larger audience. It has also been used for augmented reality applications likes the 6th Sense project (section 7.1.1). However, a more private mode of information display may be preferred for mobile military use when not in a secured environment.

DRDC Valcartier TR 2012-425 27 Figure 30: Pico projector (Microvision, 2011).

4.4 Printing Technology

Commercial printers are now able to reproduce colour 3D models. For example, the Zprinter 650 from Zcorp (WDV, 2012), is able create physical color objects from Computer-Aided Design (CAD) data, by fusing layers of powder with a liquid binder (Figure 31a and b). These composite products can be produced in a matter of hours, with objects up to a size of 10" x 15" x 8" (254 x 381 x 203 mm) and a resolution of 600 x 540 dpi (Zcorp, 2012). The Zprinter 850 can produce objects up to 20 x 15 x 9 inches (508 x 381 x 229 mm) (Zcorp, 2012). The produced objects are generally used to assess the design of products. But in a military context, the printer can be used to produce 3D urban and terrain models (Figure 31c and d).

Figure 31: Zcorp Zprinter 650 and Products (Zcorp, 2012; WDV, 2012).

4.5 Head-Mounted Displays

“Head-Mounted Displays (HMDs) are small displays or projection technology integrated into eyeglasses or mounted onto a helmet or hat (Figure 32). Heads-up displays are a type of HMD that do not block the user's vision, but superimpose the image on the user's view of the real world. An emerging form of heads-up display is a retinal display that ‘paints’ a picture directly on the sensitive part of the user's retina. Although the image appears to be on a screen at the user's ideal viewing distance, there is no actual screen in front of the user, just special optics (for example, modified eyeglasses) that reflect the image back into the eye. Some HMDs incorporate inertial sensors to determine direction and movement (for example, to provide context-sensitive geographic information) or as the interface to an immersive virtual reality application” (Gartner, 2011). 28 DRDC Valcartier TR 2012-425 Figure 32: Head-mounted displays a) (Nowak, 2007) b) (Park Service, 2011).

Through US Defense Advanced Research Projects Agency (DARPA) funding, Innovega and its strategic partners have been designing contact lenses (Figure 33) that enhance normal vision with megapixel stereoscopic 3D images. The new system, iOptik, “consists of advanced contact lenses working in conjunction with lightweight eyewear. Normally, the human eye is limited in its ability to focus on objects placed very near it, but the contact lenses contain optics that focus images displayed on the eyewear onto the light-sensing retina in the back of the eye, allowing the wearer to see them properly and appearing as equivalent to a 240-inch television, viewed at a distance of 10 feet” (Stereoscopynews, 2012).

Figure 33: Contact lens enabled eyewear (Next Big Future, 2012).

“Traditional Head Mounted Display architectures based on macro optical components are bound by an unbreakable relationship between field of view (screen size) and eyewear bulk” (Innovega, 2012). Figure 34 shows the relationship between display “bulk” and image size or field of view, for different head mounted technologies: as the field of view increases, the size of the display increases geometrically. “As indicated in the chart, the iOptik display system breaks out from this relationship by delivering almost any required image size from a fixed and conventional eyeglasses form-factor” (Innovega, 2012).

Figure 34: Field of View versus bulk of head-mounted displays (Innovega, 2012).

In the military, head-mounted and related display technologies are devices of predilection for mixed reality applications (see Section 7). They also enable a private view of the information and reduce the risk of sensitive information being seen by other people.

DRDC Valcartier TR 2012-425 29 4.6 Advanced Workstations

Intelligence analysts need workstations where they can organize and interact with information efficiently. Large screen real-estate is necessary to display various information elements together. Vertical and horizontal planes are considered. Following are example of interesting workstations.

Zenview is commercializing a multi monitor workstation (Figure 35a). National Information and Communications Technology Australia (NICTA) has developed a collaborative workstation, known as Braccetto, offering a vertical and a semi-horizontal plane (Figure 35b).

Figure 35: a) Zenview Multi monitor display (High Impact PC, 2011) b) Braccetto collaboration workstation (NICTA, 2011).

The Benddesk workstation (Figure 36) is a prototype workstation combining a vertical display together with a horizontal display, allowing continuous multi-touch interaction between both areas (Edwards, 2010). Documents can be easily moved between the horizontal and the vertical surfaces. The area between the two surfaces is a docking area. When documents get docked, their corresponding icons get reduced.

Figure 36: Benddesk prototype workstation (Edwards, 2010).

In its Gadget Lab, Microsoft has also been experimenting with a workstation composed of a vertical and horizontal surface (see Figure 37), with touch screen interaction (CNN Money, 2010).

30 DRDC Valcartier TR 2012-425 Figure 37: Microsoft prototype workstation.

Commercial workstations exploiting multi-touch surfaces are getting on the market. For instance, EXOpc has developed an Interactive Desk known as EXOdesk. It is made of a tapletop computer (e.g. surface computing), with a 40 inch high-definition display and a multi-touch capability. They have put together a video that illustrates some of the capabilities, including the use of a virtual keyboard and multi-touch interaction. The device should be released sometime in 2012 (EXOpc, 2011).

Figure 38: Illustration of the EXOdesk.

Microsoft Applied Sciences Group is conducting research on a see-through 3D desktop, using transparent OLED display technology and depth cameras. The device allows a user to see the keyboard through the display but at the same time to use his hands to manipulate information and objects displayed on the screen (Figure 39).

DRDC Valcartier TR 2012-425 31 Figure 39: Behind the screen overlay interactions (Microsoft Research, 2012a).

32 DRDC Valcartier TR 2012-425 5 Tablets, Personal Device Assistants and Smartphones

Over the last years, electronic tablets have gained in popularity with the release of the iPad. Electronic tablets are of two kinds: convertible and slate. Convertible tablets are laptops where the screen can be swivelled, that is rotated and laid down over the keyboard. Slate tablets only include the screen surface and a virtual keyboard can be displayed on the screen, although peripheral such as a mouse and a keyboard can be connected. Tablets support the user to write handwritten notes, draw graphs, charts, and pictures, using a stylus (Smith, 2006). New tablets also support touch screen interaction.

Among the main advantages of tablets are the following: great mobility because of the small size and weight of the device (the slate) and its wireless network connectivity (WiFi and cellular); capability to easily manipulate, consult and annotate documents; capability to easily write notes and create drawings.

Personal Device Assistant (PDA) and smartphone technology has also gone through significant evolution and wide user endorsement. “Multi-touch, inertial sensors, accelerometers, location awareness, video analysis and even direction are all becoming standard functionalities for high- end smart phones, enhancing familiarity with these non-traditional sensory interfaces and encouraging the move toward useful augmented reality applications” (Gartner, 2010). As shown in Figure 40, smartphones are increasingly being used for augmented reality applications (see Section 7).

Figure 40: Smartphones with augmented reality (Zonkio, 2009; Moor, 2009).

In addition to their phone network connectivity (3G, 4G), smartphones features may include: Wi- Fi and Bluetooth connectivity; camera with video recording; 3-axis accelerometers and gyroscope, GPS, digital compass; multi-touch input; ambient light and proximity sensors; FM radio; physical keyboard; slots for additional storage capacity. Moreover, “tactile haptic feedback is becoming common in cellular devices... In most cases, this takes the form of vibration in response to touch” (Wikipedia, 2011c). Senseg has developed a haptic technology that allows a user to ‘feel’ textures such as bumps, ridges and rough areas displayed on a touchscreen surface when accessing applications, see Figure 41 (Geere, 2011).

DRDC Valcartier TR 2012-425 33 Figure 41: Senseg's haptic touchscreen technology (Geere, 2011).

In terms of touch input, Tactus has developed a screen with “completely transparent physical buttons that rise up from the touchscreen surface on demand” With the buttons enabled, the user can rest his fingers as with any physical button or keyboard. Otherwise, the buttons become invisible, providing a smooth, seamless flat touchscreen surface (Tactus, 2012).

Figure 42: Tactus technology with dynamic buttons that rise up (Tactus, 2012).

Researchers at the CMU’s Human-Computer Interaction Institute have demonstrated that by attaching a microphone to a touchscreen, or smartphone, it is possible to distinguish the tap of a fingertip, the pad of the finger, a fingernail and a knuckle. This technology, called Tapsense would allow enriched touchscreen interactions. “While typing on a virtual keyboard, for instance, users might capitalize letters simply by tapping with a fingernail instead of a finger tip, or might switch to numerals by using the pad of a finger... Another possible use would be a painting app that uses a variety of tapping modes and finger motions to control a pallet of colors, or switch between drawing and erasing without having to press buttons” (HCII, 2011).

Advances are also taking place in 3D mobile devices. For example, the Samsung W960 smartphone looks like a typical smart phone, but when the screen is moved from a vertical to a horizontal orientation, the image jumps from 2D to 3D representation (Figure 43). Mobile 3D displays can be expected to be commercialized in the next few years (Newitz, 2010).

34 DRDC Valcartier TR 2012-425 Figure 43: Samsung's W960 mobile phone comes with 3D video content, generated by Dynamic Digital Depth that can be viewed without special glasses (Newitz, 2010).

The use of location-aware applications has been gaining increasing attention with tablets, smartphones and personal device assistants. After e-commerce, businesses are now entering mobile commerce, also known as m-commerce (Rao, 2011). The offering of mobile apps is ever increasing and some are directly applicable to military operations.

5.1 Military Use in the United States

In the United States, smart phones are being adopted to support military operations. A brigade at Fort Bliss, Texas is being modernized through a range of electronic devices, including smartphones, tablet devices, e-readers and mini-projectors. Dedicated military applications will be developed, including one that allows soldiers to track colleague’s locations on the battlefield. Special card readers will allow secure access to data (Mobile, 2010).

In 2009, the US Army has initiated the Connecting Soldiers to Digital Applications project, leading the way for the Department of Defense's evolving smartphone and mobile device strategy. A key goal is “to determine if it's viable to put smartphone technology in the hands of every soldier in the Army” (Hoover, 2011). Among the research interests are the following: mobile device security, transport layer, use of the devices in the field (e.g. with gloves). Still, “the Army isn’t near close to settling on an operating system or a mobile device for its ultimate goal of requiring soldiers to carry a smartphone just as they carry a rifle” (Ackerman, 2011a). The smartphones also need to go through the process of having the National Institute of Standards and Technology certify them as secure-enough to host government data.

Until 2010, BlackBerry smartphones were the only devices widely used by the US Department of Defense (DoD). In October 2011, Dell Mobile Security for Android platform has been officially certified by the Defense Information Systems Agency (DISA) for information assurance and use on defence networks (Ayers, 2011).

In November 2011, Panasonic has unveiled the new 10.1-inch, Android 3.2-powered Toughpad A1 (Figure 44). This tablet weighs 966 g (2.13 pounds), the weight of a netbook, and is 17 mm thick (.67 in), but it is drop-, water-, and bomb-proof. It is “able to survive a swim in the briny, or being run over by a Hummer full of beefy American soldiers” (Anthony, 2011) DRDC Valcartier TR 2012-425 35 Figure 44: Android Toughpad (Anthony, 2011).

In March 2010, the US Army announced its first internal applications development challenge, to provide Army personnel the opportunity to demonstrate their software development skills and creativity. The Army provided application development teams with key resources such as a cross- platform, cloud based, secure development environment (US Army, 2011). As a result of the challenge, fifteen winners and ten honourable mentions were selected from the 53 web and mobile applications developed.

In the beginning of 2012, the US Army has launched its mobile application store, the Army Marketplace (US Army, 2012). A preview of this Marketplace, dated April 2011, is shown in Figure 45. This Marketplace provides a good catalogue of the existing apps; it also brings together developers and end-users to collaborate on mobile app solutions that fit specific military needs (Ackerman, 2011a).

Figure 45: Example of the Army’s App Store, as of April 2011 (Ackerman, 2011a).

5.2 Apps for Military Intelligence

One can think of a number of applications that will be available for deployed intelligence collators and analysts, such as: conferencing, culturally-assisted translation, information aggregator, live status tracking, dynamic route finder (taking into account mission objectives and

36 DRDC Valcartier TR 2012-425 threats), biometry-based (e.g. facial) recognition (see Section 8), and virtual assistants (see Section 10).

Applications being developed to work with tablets and smartphones are the following (Hamblen, 2011):

x Harris Corp is working on an app for Apple's iPad and other tablets that will allow a soldier on the ground to use touchscreen gestures to remotely move a camera aboard an Unmanned Aerial Vehicle (UAV) to find enemy weapons or troops, while watching what the camera sees on the tablet. The video information, combined with data about location and time, can be quickly transmitted using Harris video technology to a network manned by intelligence commanders around the globe who could make quick decisions about military targets.

x Intelligent Software Solutions is readying a field test for Android and iPhone smartphone apps that will tell a soldier arriving in a war zone what fighting and bombings have already occurred at that precise location. Geo-mapping on the smartphones would be super-imposed with historical data sent wirelessly from a command center, showing the locations and types of encounters - from shootings and bombings to arrests - to better prepare troops on the ground.

As part of the US Army applications development challenge, three of the top five winning apps are of interest for military intelligence. They are:

x “Telehealth Mood Tracker (Android/iOS) is a self-monitoring app that allows users to track their psychological health over a period of days, weeks and months using a visual analogue rating scale. Users can track experiences associated with deployment-related behavioral health issues.

x Disaster Relief (Android) is a web-based data survey, dissemination and analysis tool for searching, editing and creating maps viewable on Google Earth and Google Maps. The app assists Army personnel working in humanitarian relief and civilian affairs operations. Clients can be most mobile and handheld devices such as PDAs and smart phones.

x Movement Projection (Android) is a map-routing app for road navigation that allows soldiers to input obstacles and threats - in addition to stops, start and end points - and calculates the best and fastest route” (Media Roundtable, 2010).

5.3 Mobile Data Collection

One key activity in military intelligence operations is data collection done by individual soldiers, particularly in low-intensity conflicts, humanitarian and civil / military operations. Multiple pieces of information are gathered daily by intelligence collection teams and infantry and reconnaissance patrols. Using a PDA or a smartphone every soldier in the field can become a sensor. “The combination of GPS, distance meter, pitch and roll sensors, compass, and camera as measurement tools, as well as a high-performance computer that displays background maps and resulting measurements, puts a high degree of data collection potential in a soldier’s hands… The data collected can be quickly transmitted to a central analysis, interpretation, and dissemination point, then returned as processed information to the soldier along with further instructions regarding data collection or other battlefield activities” (Surveylab, 2011).

DRDC Valcartier TR 2012-425 37 Collection tasks can involve gathering information about people, vehicle / vessels / aircrafts, equipment, infrastructures. These can be geo-referenced and various attribute information can be collected, including photos and videos of the entities.

5.3.1 ArcPad

ESRI has developed ArcPad, a mobile component of their ArcGIS product. It runs on a variety of PDAs (Figure 46a) and expands the benefits of Geographic Information Systems (GIS) to workers in the field. Information from a geodatabase can be extracted and converted for use with ArcPad, then updated when edits made in ArcPad are reintegrated, as illustrated in Figure 46b. Custom input forms can be automatically created using domains and subtypes defined in the geodatabase to generate pick lists.

Figure 46: a) ArcPad on PDAs b) ArcPad, Mobile Component of ArcGIS (ESRI, 2010 & 2011).

In a military context, “a Capture Minefield button, for example, could create a polygonal area by targeting its boundaries with a laser range finder and filling out the attributes in ArcPad. When recording data about a building, the soldier could select from a list of about 50 different types of buildings, then get a customized form to collect additional data about that specific building type. For each feature, the soldier can collect all relevant attribute information: condition of the building, number of beds in a hospital, number of medical personnel on staff, and local contact person for that facility. Time, date, and other required metadata is automatically populated and identifies the user name that registered the GeoXT handheld for that collection period” (Trimble Navigation, 2011).

5.3.2 Touch Inspect

“Touch Inspect is an application for Windows Mobile, XP, Vista and Windows 7 devices, which provides enterprise class, media-enhanced, geospatially intelligent, asset-centric field data collection and worker decision support ” (MobileEpiphany, 2009). Some of the key features are the following: x Easily configurable interface and work flows with no programming; x Finger-touch based interface with large, color-coded graphics (see Figure 47);

38 DRDC Valcartier TR 2012-425 x Advanced GIS and GPS capabilities for capturing, editing, and displaying geographic information quickly and efficiently; x Capability to take photos of assets; x Capability to work off-line then synchronize the data; and x Ability to transmit data in real time and auto-generate reports in moments.

Figure 47: Example of a TouchInspect Interface to Conduct Inspection of Urban/Street Assets (Jewell, 2009).

As Don Jewell, former Commander at US Schriever Air Force Base and deputy chief scientist at Air Force Space Command, points it out: “Because warfighters can access the fully encrypted database without Internet access, they can create a new IED (Improvised Explosive Device) or compare IEDs from previous encounters in any location. Plus, since they can produce a new IED category almost instantly, they are not restricted as they normally would be by a flat file type database. By the same token, the program can force the new warfighter to look at aspects of an IED that they might not think of at the time” (Jewell, 2009).

5.4 Intelligent Software Assistants for Smart Phones

Section 10 describes in detail research undertaken on intelligent software assistants (ISA). This technology can be deployed in command centers but is also of interest for mobile devices such as smart phones.

5.4.1 Siri

A spin-off of the PAL (Personalized Assistant that Learns) project (see Section 10), Siri has developed and deployed ISA technology onto the iPhone 4s. Siri understands a variety of questions from the user, in natural language, related to his day-to-day activities, such as contacting relatives, holding meetings, looking for restaurants, and it can take appropriate actions. So the user can tell Siri things like “Text Ryan I’m on my way” or “Will it be sunny in Miami this weekend?” (Apple, 2011).

“But what if you’re in the military, and you want to take Siri back to its PAL roots? Best of luck to you. Obviously, there’s no PAL in usage” (Ackerman, 2011b). As seen in Section 5.2, smartphone and mobile device technology is making its way in the military. The PAL project would then provide a very good roadmap in terms of the provided capability.

DRDC Valcartier TR 2012-425 39 5.4.2 Speaktoit Assistant

Similar to Siri, Speaktoit Assistant is a free personal assistant running on Android devices but also available for the Apple smartphones. It uses a virtual buddy and natural language technology to “answer questions, find information, launch apps and connect you with various web services, such as Google, Facebook, Twitter, Foursquare, Evernote, and many others. The Assistant works with your smartphone to get maps, search for news and images, look up weather reports, convert currency and measurements, send email, and much more” (Google, 2012a).

Figure 48: Examples of Speaktoit Assistant interaction (Google, 2012a).

5.4.3 Google Assistant

Google is working on an application similar to Siri that will integrate its voice recognition software and a virtual assistant. The application would be called Google Assistant and would be deployed on the Android platform and be available by the fourth quarter of 2012. The project has three parts: Get the world’s knowledge into a format a computer can understand; create a personalization layer; build a mobile, voice-centered ‘Do engine’ that’s less about returning search results and more about accomplishing real-life goals (Tsotsis, 2012).

40 DRDC Valcartier TR 2012-425 6 Collaboration Technologies and Smart Room Environments

As indicated previously, military intelligent analysts need to collaborate amongst themselves, examining multi intelligence sources, and with other organizations, in a JIMP context. They can be supported by collaboration technologies and by smart room environments.

6.1 Collaboration Technologies

In a JIMP context, military operations are increasingly dynamic and complex and require significant collaboration and coordination between people from various military and civilian organizations. These collaboration technologies are often present in smart room environments.

6.1.1 Computer Supported Cooperative Work

Over the last several years, a diversity of tools has proliferated to support collaboration across distributed teams. This includes technologies such as audio / video conferencing, chat / instant messaging, electronic white board, application sharing, groupware, collaborative real-time editors, wikis, internet forums, blogs and microblogs, social network services (e.g. Facebook, Twitter), virtual worlds and crowd sourcing. Allies organizations are using these tools but belligerents are using them as well. Crowd sourcing has become a significant source of information when an emergency situation occurs and people report on it using cell phones video feeds and/or text messages.

6.1.2 Pervasive Human Sensing and Remote Assistance

Deployed military and civilian personnel could provide live data streams, using instrumentation such as head-mounted cameras, to a multi-intelligence center. These deployed personnel could also benefit from virtual assistance. Using HMDs collaboration could take place between deployed soldiers and a military specialist in a command centre. For example, using a bidirectional feed, the helpful advices of a translator, cultural expert, explosive device specialist, intelligence analyst or medical staff could be provided directly to a soldier in the field. These specialists could insert information directly into the HMD of the soldier. Local population could help soldiers on cultural and ethnic issues directly from a local command center and not be exposed as helping the forces.

6.1.3 Telepresence / Teleimmersion

Telepresence technology allows people to interact as if they were physically. In telepresence videoconferencing, a higher level of fidelity is provided in terms of image and audio quality. Figure 49 is an example of DVE Telepresence system (DVE, 2011).

DRDC Valcartier TR 2012-425 41 Figure 49: DVE Telepresence system (DVE, 2011).

In a future telepresence setting, we could think that intelligence analysts will also be able to exchange documents just by moving them towards the other participants, as if they were in the same room. Processes transparent to the users would then use the network to send these documents.

6.2 Smart Room Environments

A smart room or intelligent environment is a “space in which computation is seamlessly used to enhance ordinary activity” (Coen, 1998), or a space “that is able to acquire and apply knowledge about the environment and its inhabitants in order to improve their experience in that environment” (Cook and Das, 2007). “The generic goal of such environments is to support group interactions at real time and in an intelligent way” (Marreiros et al., 2008). In other words, smart room environments are physical spaces that are instrumented with various networked sensors and devices and support ubiquitous computing permitting to sense, interpret and react to human activity in order to enable better collaboration, increased productivity, creative thinking and decision making. Collaboration can take place within and across meeting rooms.

Closely related is a new science and technology area called Ambient Intelligence (AmI) that applies to the instrumentation and support of smart room environments. Ambient Intelligence is defined as “a digital environment that proactively, but sensibly, supports people in their daily lives” (Augusto, 2007). Ambient Intelligence “emphasizes or mainly deals with user-friendliness, more efficient services support, user empowerment, and support for human interactions. AmI builds on three recent key technologies: Ubiquitous Computing, Ubiquitous Communication and Intelligent User Interfaces” (Chandrasekhar et al., 2011).

Additionally, in order to develop awareness of what is taking place in a smart room environment, there is a need to identify the people in the room or suite of rooms, determining their location and tracking their activities. This can be achieved using techniques ranging from badges, Radio- Frequency Identification (RFID) tags, id-sensors (biometry) and computer vision. “Tag-based methodologies tend to be obtrusive, requiring the individual to continuously wear them, however small the tag maybe. Some of the biometric techniques, such as fingerprint and iris scans, require a ‘pause-and-declare’ interaction with the human. They are less natural than face, voice, height, and gait, which are less obtrusive and hence are better candidates for use in smart environments” (Menon et al., 2010). Audio/video recognition systems (using sensors such as the MS Kinect, see section 3.3.3) could be used to track what activity or task a user is conducting, such as talking to another person or writing on a white board. Multiple techniques can be used which would then require a multimodal fusion system.

The concepts of smart rooms have evolved and have been applied to offices, class rooms and home environments. In the video ‘A day made of glass’ Corning Inc. illustrates a vision of the future, where a family would be immersed in smart environments exploiting ubiquitous and 42 DRDC Valcartier TR 2012-425 advanced displays (Corning, 2011). Extracts from the video are shown in Figure 50. Corning has expanded this vision in a second video (Corning 2012a) and provides details on the maturity of the illustrated technologies (Corning, 2012b).

Figure 50: Extracts from A Day Made of Glass showing smart home and office environments (Corning, 2011).

In military settings, command centers will transform into smart room environments meant to support analysts and watch officers in their work. These environments will be equipped with a number of physical devices such as: LGDs, high-end analyst workstations, multi-touch tables, digital dash boards and ambient displays, video conferencing and telepresence equipment, biometric authentication, room and people sensors (many of these devices were described in sections 3 and 4). From a user experience and a capability perspective, smart room environments could provide them with: ubiquitous computing, contextual / adaptive user interfaces (see section 11.3), shared situation awareness, shared collaboration awareness (across meeting spaces), and tools to support the collaborative decision process.

The following three examples are of interest to military smart room environments.

6.2.1 LiveSpaces

The Defence Science & Technology Organization (DSTO) in Australia has developed an environment called LiveSpaces, seamlessly integrating various technologies, to support both co- located and distributed teams engaged in intense collaborative activities such as analysis, planning and decision making (Phillips, 2008). These types of tasks have high cognitive requirements and cognitive abilities which can be easily overloaded when too much attention needs to be applied to simply managing and interacting with the environment, the devices, applications and required information. LiveSpaces provides control of room hardware such as lights, speakers, white boards and video projection, as well as groupware applications. Services DRDC Valcartier TR 2012-425 43 are provided to support speech recognition / speech transcription, and scripted audio-video briefings.

Figure 51 shows an example of a LiveSpaces environment, where users have access to a range of shared displays including LGDs and a number of smaller displays used to provide ambient information and for activities such as video teleconferencing. The underlying LiveSpaces infrastructure supports various ways of managing, controlling, and interacting with the shared displays. There is minimum set-up time for equipment, devices and tools, and no need for a dedicated resource to operate the room. There is a control panel that can be easily operated by anyone. Users can easily share the information on their own screens with others. Individuals can also interact with the shared displays by simply moving the mouse cursor from their own screen onto the shared screens. The LiveSpaces infrastructure also supports the federation of distributed sets of LiveSpaces. LiveSpaces has been successfully deployed on to a number of Australian Defence Forces sites for advanced trials. The LiveSpaces software has been contributed to the open source community, on sourceforge.net.

Figure 51: DSTO’s Intense Collaboration Space, using LiveSpaces environment (Phillips, 2008).

6.2.2 LightSpace

LightSpace is a project conducted at Microsoft Research, to explore a variety of interactions and computational strategies related to the exploitation of interactive displays in a small room installation. Multiple depth cameras and projectors are used to emulate interactive displays on un- instrumented surfaces (e.g. standard table or office desk) and facilitate mid-air interactions between and around these displays. Doing so, users are able to transfer an object from one display to another by simultaneously touching the object and the destination display (Figure 52 a-b). They are also able to “pick up the object by sweeping it into their hand, see it sitting in their hand as they walk over to an interactive wall display, and drop the object onto the wall by touching it with their other hand” (Wilson and Benko, 2010) (Figure 52c-d). LightSpace also supports making a menu selection by a menu projected on the hand.

44 DRDC Valcartier TR 2012-425 Figure 52: LightSpace through-body object transition between existing interactive surfaces (a-b) and interactions with an object in hand (c-d) (Wilson and Benko, 2010).

In a related project at Microsoft Research, a new augmented reality concept coined as Beamatron, explores the use of a Kinect camera and a projector to sense and project graphics anywhere in a room. Using a 3D model of the room and the physics model of a virtual object, for example a little car, the system can project that virtual object on real objects (variously shaped volumes and oriented surfaces). The system can also control the projection of the object appropriately as it interact with the physical characteristics and dimensions of the room, such as falling down off a ramp or stopping when colliding with an obstacle (Heise, 2012). Figure 53 shows the Beamatron setting and an example of the control of a virtual object in the real environment.

Figure 53: Beamatron Setup and illustration of controlling virtual objects (Microsoft Research, 2012b).

6.2.3 Navy Command Center of the Future

SPAWAR Systems Center Pacific in San Diego has developed a prototype of a facility of a Command Center called the Navy Command Center of the Future (CCoF). It is equipped with glass walls with many small video screens; a large video screen that serves as the central focus of those in the room; a small, and a private sound-proof room with glass walls that can automatically be darkened in case classified information needs to be discussed (Figure 54). On both sides of the glass-walled room, several video displays can be used to show live video, documents loaded by an artificial-intelligence system, news from commercial media, and just about anything the people need to see. Glass surfaces will allow decision makers to manipulate data and other information simply by running their fingers over the glass.

The room is designed to bring together those who need to be involved in discussions surrounding specific military engagements, regardless of whether they're local or not. Indeed, the room's very mission statement is to make it possible to rely on video teleconferencing and artificial intelligence in such meetings. It has also been designed to allow senior officials to walk and talk and never lose sight of those they're communicating with. Moreover, the Center would have AI meant to discern what is being talked about during a teleconference and to know how to source up whatever documents are needed as they're needed. (Terdiman, 2009).

DRDC Valcartier TR 2012-425 45 Figure 54: Pictures of the Navy Command Center of the Future (Terdiman, 2009).

46 DRDC Valcartier TR 2012-425 7 Mixed Reality

Mixed Reality (MR) refers to the area between the two extremes of the real world and a fully virtual world. Milgram and Kishino (1994) have produced simplified representation of a reality – virtuality continuum (Figure 55). In mixed reality the merging of real and virtual worlds produces new environments and visualizations where physical and digital objects co-exist in real time. Augmented Reality (AR) and mixed reality are now sometimes used as synonyms.

Figure 55: Virtuality Continuum (Milgram and Kishino, 1994).

“Virtual reality provides a computer-generated 3D environment that surrounds a user and responds to that individual's actions in a natural way, usually through immersive head-mounted displays and head tracking. Gloves providing hand tracking and haptic feedback may be used as well. Room-based systems provide a 3D experience for multiple participants... The growing popularity of 3D entertainment using 3D glasses — and ultimately 3D screens and projections that do not require glasses — may relegate immersive virtual reality to permanently niche status” (Gartner, 2010). Figure 56 shows a range of virtual reality devices, with a growing degree of immersion and price.

Figure 56: A range of virtual reality immersion (Bernier et al., 2010).

Virtual Reality (VR) can provide value for intelligence collators by providing them with a platform to conduct a mission rehearsal in the environment in which they will operate, in particular in hostile urban settings.

AR or MR is a particular technology which allows adding sensations, images and information generated by a computer to the normally perceived reality (Figure 57a). Additional information

DRDC Valcartier TR 2012-425 47 on the real environment such as images, writings and virtual objects are provided to the user through special visors worn by the individual or projection in his immediate environment (Contactum, 2011).

“Augmented reality applications, which superimpose information on the user's view of the real world, rather than blocking out the real world, have overtaken virtual reality, based on rapidly growing activity in the mobile marketplace” (Gartner, 2011).

The Mirage AR System (Figure 57b) is a complete solution allowing you to create your own augmented reality experience by inserting virtual content to the real environment and using stereo video see-through OLED HMD (Arcane Technologies, 2011).

Figure 57: a) Army Augmented Reality (Contactum, 2011) b) Mirage Augmented Reality system (Arcane Technologies, 2011).

7.1 Augmented Reality

AR is a broad research field and there are multiples technologies that can be used to implement it. For AR that is shared between multiple users and applied in controlled environments, such as smart room environments (see Section 6.2), projectors can be used. AR that uses head-mounted displays would be more appropriate when the information has to be available to the user outside of command centers and operation headquarters, as well as for protecting the privacy of the information. Using smartphones may be a reasonable compromise that is not too expensive but still mobile. These three types of AR technologies are discussed in the following subsections.

7.1.1 Augmented Reality Using Projectors

Similar to LightSpace (Section 6.2.2), MIT’s LuminAR “transforms surfaces and objects into interactive spaces that blend digital media and information with the physical space” (Linder, 2010). The innovative aspect of this project is that it is a self-contained system that can be screwed into standard light fixtures everywhere. Figure 58 shows the LuminAR Lamp which is an articulated robotic arm that is designed to interface with the LuminAR Bulb.

48 DRDC Valcartier TR 2012-425 Figure 58: LuminAR projects an interactive interface on any surface (Linder, 2010).

Another MIT project called SixthSense (Figure 59) explored mobile AR. This wearable device is controlled by gestural interactions and can display digital information on any physical surface. The intent is to “make the entire world your computer” (Mistry, 2010). For example, the prototype can augment a newspaper with videos, display reviews of a product when shopping, take pictures with a simple gesture or let the user dial a number on his palm. Basically, it can perform the same tasks as a smartphone, but using the world around as a larger screen space with advanced gestural interactions.

Figure 59: The 6th Sense project prototype augments the physical world with projected digital media information (Mistry, 2010).

7.1.2 Head Mounted Display Augmented Reality

Tanagram Partners participated in a DARPA project for developing military-grade augmented reality. The company developed many illustrations showing how augmented reality could be used. Figure 60 shows concepts such as sharing a 3D map of the area, having a tight connection to the command center as the operations are being carried out, keeping in sight the state of blue forces, and highlighting the features that are pointed at by other soldiers (Juhnke, 2010).

DRDC Valcartier TR 2012-425 49 Figure 60: Storyboards of how HMD-AR could be used by military personnel (Juhnke, 2010).

Since a HMD offers control over what each individual can see, it could serve for showing different information to people according to their clearance level and work duties. The use of diminished reality (Herling and Broll, 2010) could even allow hiding information that someone is not allowed to see by removing it from his view (see Figure 61).

Figure 61: Diminished reality example (Herling and Broll, 2010).

The Just-In-Time Command and Control Center visualization system (see Figure 62) uses tangible interfaces to manipulate virtual objects. “Users of the system manipulate a set of AR Toolkit markers augmented by 3D maps with geotagged locations containing images, video, audio, and text. Additionally, the user can interact with the geotagged information via a pointing device (also tagged with an AR marker) which allows for manipulation by scooping, pointing, passing, and zooming” (PARVAC, 2012).

Figure 62: Just-In-Time Command and Control Center visualization system (PARVAC, 2012).

50 DRDC Valcartier TR 2012-425 Although they say that it is still years away, the entry of Google in the AR domain with its ‘project glass’ has the potential to bring AR head-mounted display to a broader audience in the next years. A demo and a concept video have been developed (see Figure 63) and they began testing in April 2012 (Forbes, 2012).

Figure 63: Project Glass prototype and concept video (Google, 2012b; Lum, 2012).

7.1.3 Smartphone Augmented Reality

AR does not necessarily require expensive dedicated visualization devices. With the rise of smartphones, some analysts predict that AR technology will be featured natively in every smartphone by 2014 (Metaio, 2011). Some, such as SoldierEyes from OverWatch System, are specifically targeted at military personnel and first responders. This application lets “soldiers see information overlaid on what their smart-phone’s camera sees” (Gould and Hoffman, 2010). For more on smartphones military applications, see Section 5.

Figure 64: SoldierEyes smartphone augmented reality application (OverWatch, 2012).

DRDC Valcartier TR 2012-425 51 7.2 Cave Automatic Virtual Environment (CAVE)

A CAVE is an immersive virtual reality environment, usually formed of a cube, providing a 3D realistic environment. High-resolution images of a scene are projected onto the walls and floor of the cube (in the best case, 4 walls, one floor and one ceiling). Wearing special glasses, the user can stand and move within this space as if he would be in the real environment (Cruz-Neira et al., 1992).

DRDC Valcartier has set up such an environment as part of its Virtual Immersive Facility (VIF). It can be used in three modes: the ‘Holodeck’ immersive mode; the immersive theatre mode, where the side walls are placed at a 45 degree angle; and the flat wall mode, which can be useful to provide a form of knowledge wall (Figure 65).

Figure 65: The Flex CAVE at DRDC Valcartier (Bernier et al., 2008).

The VIF and in particular, the CAVE, is used to explore the potential of advanced visualization, interactive simulation and immersive capabilities for the CF. “Immersive virtual environments can provide a realistic, interactive, engaging, and cognitively augmented setting for a range of applications” (Bernier et al., 2010).

Bernier et al. (2010) have identified a range of defence and security activities where immersive environments are or could be of benefit (see Figure 66): x Stress management training; x Medic in operational situation; x Immersive exploratory perception; x Individual battle task standard; x Sensitive site exploitation; x After action review; and x Complex situation understanding.

52 DRDC Valcartier TR 2012-425 Figure 66: Immersive environments are of interest to a range of defence and security applications (Bernier et al., 2010).

For the intelligence domain, immersive exploratory perception is particularly of interest, allowing military staff to develop complex situation understanding or to conduct mission rehearsal before being sent in a hostile environment.

7.3 Virtual Worlds

“A virtual world is an online community that takes the form of a computer-based simulated environment through which users can interact with one another and use and create objects. The term has become largely synonymous with interactive 3D virtual environments, where the users take the form of avatars visible to others. These avatars usually appear as textual, two- dimensional, or three-dimensional representations” (Wikipedia, 2011d). Second Life, which was launched on June 2003, is one of the most popular Virtual World (VW) environments.

A number of interesting thoughts gathered in (Onislam, 2009) are the following: x Co-creating and inhabiting a three-dimensional space that they can collaborate upon betters chatting in a two-dimensional space. They allow people to ‘be together’ despite geographical location, age, gender, ethnic or socio-political affiliation (from Rita J. King, CEO of Dancing Ink Productions, a strategic creative content development and research company). x King also says: “The first step is to not distinguish between ‘real’ and ‘virtual’ relationships, but rather to find new ways to bring value into our own lives and the lives of others through meaningful interactions”. x “Virtual worlds will become as normal as the internet is now” (from Nergiz Kern, an English language educator inside Second Life). DRDC Valcartier TR 2012-425 53 The US Department of Defense has already been using VW to support a number of applications. These are detailed hereafter.

7.3.1 T2 Virtual PTSD

For example, the T2 Virtual Post-Traumatic Stress Disorder (PTSD) experience, based on Second Life, provides “an immersive, interactive learning experience designed to educate visitors about combat-related post-traumatic stress disorder (PTSD)” (T2 Health, 2012) (see also US DoD (2012). Traumatic brain injury and post-traumatic stress disorder are the ‘signature wounds’ of the wars in Iraq and Afghanistan. Approximately 19% of all Service Members returning from combat theatre screen positive for psychological health problems, and of those that screen positive, slightly more than half seek psychological health services. The T2 Virtual PTSD environment improves “learning through doing rather than merely reading about or watching a video about post-deployment issues” (T2 Health, 2012). The users participate in the simulation of a combat-related trauma, helping them to understand the common reactions to trauma, and to learn how avoiding to think about or talk about their deployment experience. The environment is therefore of high interest to the service members coming back from an operation and to their families. Figure 67 shows a scene from T2 Virtual PTSD, representing a typical Afghan street. There is also an introduction video of the T2 Virtual PTSD Experience on the same site.

Figure 67: A virtual Afghan street featured in the T2 Virtual PTSD Experience (T2 Health, 2012).

7.3.2 A-SpaceX

In 2008, the Intelligence Advanced Research Projects Activity (IARPA) launched a project known as Analysis WorkSpace for Exploitation (A-SpaceX) to explore the use of VWs in manipulating time frames and mapping decision processes, in order to improve intelligence analysis. One concept would be “to create ‘MindSnaps’ that would allow analysts to track and map out their decision processes in a virtual world” (FCW, 2008). An example of potential A- SpaceX workstation is shown in Figure 68.

54 DRDC Valcartier TR 2012-425 . Figure 68: Example of a potential A-SpaceX workstation (PSE, 2012).

7.3.3 Collaboration and Training

VW technology is being used to support training, collaboration and meetings. For example, the I- Room 3D Virtual Space, based on VW technology such as Second Life and Opensim, provides a web-based collaboration portal, in order to support intelligent interaction (Figure 69). It is configured with multiple work zones, designed for collaborative and brain storming style meetings. It links up to external web services, collaboration systems and intelligent systems aids (Tate et al., 2010; Tate, 2011).

Figure 69: The I-Zone Virtual Collaboration Environment in Second Life (Tate,2011).

DRDC Valcartier TR 2012-425 55 Some of the VW benefits for collaboration and meetings are the following (Gigaom, 2012):

x Team meetings. VW technology provides a sense of ‘presence’ to conduct effective virtual team meetings. Audio communications, chat and the display of multimedia presentation material are easily supported. x Collaboration sessions. VW technology can provide a ‘sandbox’ to deliver a truly ‘out-of- the-box’ thinking within a team. x Training and learning events. Training in a virtual environment can be provided through the form of formal classroom or auditorium settings as well as simulated environments.

The US DoD is planning to have that expansive virtual world for soldier training up and running in the next five years (US DoD, 2011). “The world will provide various avatar-led combat training situations that will merge operations from various branches of the military and prepare warfighters for a variety of scenarios”. A number of projects are being implemented based on VWs and simulation systems. Figure 70 shows respectively, a) a section of MyBase islands in Second Life; b) a map of U.S. Military Islands, called MiLands, in Second Life; c) the virtual Naval Undersea Warfare Center campus in Second Life; d) a scene from Virtual Battlespace 2 (US DoD, 2011).

Figure 70: Various Virtual World environments for military training (US DoD, 2011).

7.4 360-Degree Interactive Imagery and Video

360-degree imagery is a technique consisting in taking photographs from all directions from a single point and assembling them into a unified product. Using this product, the user can then experience a location as if he was there, looking left, right, up and down, and even moving into a scene. 360-degree imagery has been popularized by the Google Street View application, where street-level imagery allows users to have a feel for various vicinities. It is even used commercially to show house, hotel or restaurant interiors (Google, 2011a).

56 DRDC Valcartier TR 2012-425 360-degree interactive video extends the capability of 360-degree imagery. The technique consists in taking videos in different directions, from a fixed position as well as in movement, and assembling them into a unified product. Using the product, the user feels this time as if he were immersed in an interactive movie, looking in different directions, seeing people passing by and conducting their activities, looking at urban traffic, construction activities, and this, as the movie is playing.

An example of 360-degree interactive video is available at (Hogg, 2010). This video is filmed from a truck driving through a street of a village in Haiti, after the January 2010 earthquake. As the video is replayed, the user can move the video, using the mouse, to change the angle, look left or right, and up and down. He can also pause and zoom in or out on specific content. Three frames from the video, one centered on the street, one to the left and one to the right, are shown in Figure 71 to illustrate the concept.

Figure 71: 3 Frames of a 360-degree interactive video (Hogg, 2010).

In a military context, 360-degree interactive imagery, such as street-view, is already used. It can provide enhanced situation awareness when an operation in an urban environment is planned. It can be used to support mission rehearsal or for briefing purposes. Street View data has been gathered for many countries, although not those in conflict with Canada (Google, 2011b). Still similar technology can and is used to gather street imagery, in countries where coalition forces are conducting peace support and reconstruction operations.

One can anticipate that 360-degree interactive video will gain momentum. It can provide military staff a greater capability in analysing a scene, understanding the behaviour of people in an environment, monitoring activities.

Professor Marc Alexa and the Computer Graphics Group at the Technische University in Berlin have developed the throwable panoramic ball camera. The purpose is to capture pictures from a location that a user is not able to access. The ball has a diameter of 19.3cm and has protective foam all around to protect it from a drop. When the ball is thrown up, it uses the information from its accelerometer to calculate its highest point, and then multiple photos are taken at the same time by the array of cameras around the ball. The photos can then be transferred from the ball to a computer via a USB port (Burns, 2011). Figure 72 shows pictures of the ball and its use. Although the use of throwable ball seems a little awkward in a military context, the concepts would be very valid.

DRDC Valcartier TR 2012-425 57 Figure 72: Pictures of the panoramic ball camera and its use (Burns, 2011).

58 DRDC Valcartier TR 2012-425 8Biometry

“Biometry refers to the automatic identification or identity verification of living persons using their enduring physical or behavioral characteristics” (EFF, 2003). As a process, it is also defined as “the ability to capture, transmit, store, share, retrieve, exploit, and display biometrics data from multiple target population groups for timely identification or verification” (Pratt, 2008). To be effective, a biometric system must compare captured biometric data to a biometric database. Among the features measured are: face, fingerprints, hand, handwriting, iris, retinal, vein, DNA and voice (Figure 73). Emerging biometric technologies include: odour, facial thermography, nail bed, ear shape, skin reflectivity, lip movement, blood pulse, and gait recognition (GBT, 2011) and (Global Security, 2011).

Figure 73: Common features used in Biometry (Pratt, 2008).

Biometry has several advantages over current identification methods, such as the utilization of passwords or Personal Identification Numbers (PINs): It “links the event to a particular individual (a password or token may be used by someone other than the authorized user), is convenient (nothing to carry or remember), accurate (it provides for positive authentication), can provide an audit trail and is becoming socially acceptable and cost effective” (Biometric Consortium, 2011). Face recognition is now ubiquitous with the wide spread of face recognition in photo management software. “Facebook rolled out its ‘Tag Suggest’ system, which recognises pictures of users' friends as they are uploaded; face.com claims it has scanned 25 billion images; and Google (which uses face recognition in its Picasa photo-management system) says that, once it gets the legals figured out, Google Goggles will be used to identify members of the public by pointing your smartphone at them. Rioters in Canada and the UK have been identified using the technology, and at the 2014 FIFA World Cup, Brazilian police will use face-scanning glasses to automatically identify ‘persons of interest’ at matches” (Cheshire, 2012).

In a pilot project conducted by PBA Engineering Ltd., in Canada, it has been demonstrated that vascular pattern (vein structure) recognition (Figure 74) has significant advantages over face, iris, hand geometry and fingerprint recognition, in terms of usability and accuracy in authentication systems (Figure 75). In particular, hand vascular scanners are less intrusive, cannot be easily copied, provides high recognition rate, is hygienic and usable by most people, is equally usable by persons of different heights and mobility, and provides a very small network traffic load with encryption and security included (O’Neill, 2008).

DRDC Valcartier TR 2012-425 59 Figure 74: Vascular pattern (vein structure) recognition device and biometric process (O’Neill, 2008).

Figure 75: Accuracy and usability of popular biometry technologies (O’Neill, 2008).

Some leading edge applications of biometry are the following: x “Fingerprint scanners (and the necessary software to store and compare fingerprints) have already been installed in laptop computers and PDAs like the iPad. x Sensors installed in automobiles can identify the driver, and adjust mirrors, seat positions and climate controls. x Surveillance cameras can search crowds for missing persons or criminal suspects. x Face recognition software can be modified to recognize gestures, leading to improved assistive technologies for quadriplegic patients” (Technovelgy, 2011).

In a military context, biometry has the following applicability: x Authentication: A one-to-one matching process. It applies to security management of users to a facility (e.g. command center) or a computer; security management of the privileges; profile management of user roles and preferences. x Identification / Recognition: A one-to-may matching process. It applies to identification of belligerents; support to forensics operations (e.g. scene investigation of an explosion).

DoD has recognized the important role that biometrics play in prosecuting the global war on terrorism, protecting troops, and securing national security interests and created the Biometrics Task Force. Its role is to “program, integrate, and synchronize biometric technologies and capabilities and to operate and maintain DoD’s authoritative biometric database to support the National Security Strategy” (Hunt. 2008a).

60 DRDC Valcartier TR 2012-425 The US Army Biometrics has already shown its value in detaining encountered people, denying access and tracking movement in Afghanistan (Hunt, 2008a). Figure 76 shows the collecting of biometrics in action.

Figure 76: Collecting biometrics samples (Hunt, 2008b; Pratt, 2008).

Biometry is applied not only to likely belligerents but to friendly forces as well. For example, in Afghanistan, as depicted in Figure 77, biometrics spanned across the following population groups: red, gray and blue forces (Pratt, 2008).

Figure 77: Population groups to which biometry can be applied (Pratt, 2008).

From a military intelligence perspective, there is a need to enhance biometric capabilities by supporting (Swan, 2008): x Stand-off collection: Increase the distance from the sensor to the targeted individual; x Covert collection: Collect a biometric sample without the targeted individual’s knowledge.

DRDC Valcartier TR 2012-425 61 This page intentionally left blank.

62 DRDC Valcartier TR 2012-425 9 Neural / Brain Computer Interfaces

In this section, we discuss the use of neurotechnology and Brain Computer Interfaces (BCIs). BCIs are systems that translate the electrophysiological activity or metabolic rate of the nervous system into signals that can be interpreted by a mechanical device. Although brain implants have allowed humans to control complex devices such as a robotic arm (Tkoranyi, 2011), their use is not expected soon because they require extensive surgery and are still facing major hurdles. Even if they still face many challenges, non-invasive interfaces are beginning to appear commercially and are described in this section.

9.1 Sensors Technology and Commercial Devices

There are two main technologies for sensing brain waves: Electroencephalography (EEG) and Functional Magnetic Resonance Imaging (fMRI). fMRI measures changes in blood flow related to neural activity and is used for brain mapping. It has a low temporal resolution but can record signal from all regions of the brain, unlike EEG which is biased towards the cortical surface.

On the other hand, EEG measures the brain’s electrical activity with a higher temporal resolution but lower spatial resolution. Different brain states are the result of different patterns of neural interaction. These patterns lead to waves characterized by different amplitudes and frequencies as listed in Figure 78. Higher numbers of sensors enable the gathering of more brainwave information and thus greater accuracy. Typical medical EEG devices use 19 sensors. They traditionally require the use of gel, paste or saline at the scalp terminals to improve conductivity, but dry technologies that rely on natural skin oils are now available (Sutherland, 2011).

Figure 78: Some of the commonly-recognized frequencies generated by different types of brain activity (Zhang et al., 2010).

Neural interfaces that exploit EEG are making their way into the gaming and PC users market and brainwave headsets are now available commercially off the shelves from two main competitors: NeuroSky and Emotiv. Another headset, the NeuroFocus Mynd claims to be a full-brain wireless EEG headset that can capture brainwave activity at 2000 times a second. However, it is only developed for neuromarketing purposes and not available to the general public (Vochin, 2011).

9.1.1 NeuroSky MindSet

The NeuroSky MindSet (Figure 79) uses proprietary algorithms to decipher the type of mental activity from EEG measurements using only one electrode. The raw electrical measurements, calculated brainwave levels, and interpreted mental states are converted into digital numbers for computers, software, and programs. NeuroSky sensors interpret 2 different mental states based on 4 brainwaves and can detect eye blinks. They offer developer application programming interfaces (APIs) and tutorials for a large number of platforms and programming languages as well as the NeuroSky App Store for smart phones apps distribution.

DRDC Valcartier TR 2012-425 63 Figure 79: NeuroSky MindSet is a commercial brain-computer interface (NeuroSky, 2011).

9.1.2 Emotiv Systems EPOC

Emotiv EPOC (Figure 80) has 14 electrodes and can detect 4 mental states based on brainwaves, 13 conscious thoughts, facial expressions and head movements (from a two-axis gyroscope). The drawback of this headset is that it uses terminals with pre-moistened saline pads that need to be rehydrated with saline fluid, but it is nevertheless very usable. The EPOC BCI can translate brain activity into various simple computer commands, but the headset must first be trained to recognize what kind of thought pattern equates to a certain action. In addition, facial expressions can also be exploited by applications, for example, they could be used by a virtual assistant application to better understand user context (see Section 10). Emotiv also offers a software development kit for researchers. The EmoKey software is also provided. It is a background process that converts detected events into any combination of keystrokes that can be used by computer applications.

Figure 80: Emotiv EPOC neuroheadset (Emotiv, 2011).

9.2 Augmented Cognition - DARPA

“Limitations in human cognition are due to intrinsic restrictions in the number of mental tasks that a person can execute at one time, and this capacity itself may fluctuate from moment to moment depending on a host of factors including mental fatigue, novelty, boredom and stress. As computational interfaces have become more prevalent in society and increasingly complex with regard to the volume and type of information presented, researchers have investigated novel ways to detect these bottlenecks and have devised and continue to determine strategies to aid users and improve their performance by effectively accommodating capabilities and limitations in human information processing and decision making” (ACIS, 2011).

64 DRDC Valcartier TR 2012-425 Augmented Cognition (AugCog) is an emerging field of science that has the explicit goal to extend a user's abilities via computational technologies, which are explicitly designed to address bottlenecks, limitations, and biases in cognition and to improve decision making capabilities. It proposes to do this through continual background sensing, learning, and inferences to understand trends, patterns, and situations relevant to a user’s context and goals (ACIS, 2011).

Augmentation Cognition has significant applicability to military intelligence, as analysts must frequently perform cognitively demanding tasks in stressful environments. DARPA, through its AugCog Program has developed technologies to mitigate sensory or cognitive overload and restore operational effectiveness by extending the information management capacity of the warfighter. This is accomplished through closed-loop computational systems strategies including (DARPA, 2011a): x Intelligent interruption to improve limited working memory; x Attention management to improve focus during complex tasks; x Cued memory retrieval to improve situational awareness and context recovery; x Modality switching (i.e., audio, visual) to increase information throughput.

The Future of Augmented Cognition short film tells the story of AugCog science and technology, with an eye to how it will mature and be used in the coming decades. “This short film takes place in 2030 in a command center that is tasked with monitoring cyberspace activities for anomalies that could threaten the global economy. Given the ever-increasing amount of data to be analyzed even in today’s world, the workers in 2030 are inundated with information from all sources. They have so much information to contend with that they are literally unable to process it all unaided. Fortunately, AugCog technologies have matured by this point and are commonly integrated into information-rich domains, including the featured command center” (DARPA, 2007). Figure 81 shows extracts from the short film. In particular, it shows the use of an individually calibrated headed mounted neuro-physiological monitor that tracks and measures cognitive and stressful activities occurring within one’s brain. The AugCog system helps to balance the information load, for example by maximizing the appropriate elements of information (e.g. scrolling text, pictorial or auditory) allowing focusing attention to the task at hand.

Figure 81: Extracts from the short film ‘The Future of Augmented Cognition’ (DARPA, 2007).

DRDC Valcartier TR 2012-425 65 9.3 Neurotechnology for Intelligence Analysts - DARPA

The Augmented Cognition efforts have led to the Neurotechnology for Intelligence Analysts (NIA) program (Figure 82). Rather than trying to consciously control the computer with their thoughts, the NIA intend to take advantage of unconscious outputs from the visual processing part of the brain, for example to increase triage speed for satellite imagery. Current computer-based target detection capabilities cannot process vast volumes of imagery with the speed, flexibility, and precision of the human visual system. For this project, the analyst's brain is treated as a sensor. “Electrical activity it produces is recorded from electrodes placed on the scalp, the same way electroencephalography (EEG) is used in hospitals to monitor brain activity. Then, when the analyst looks at one of the images flashing by, a scalp plot shows when there is increased brain activity.

Figure 82: Neurotechnology for Intelligence Analysts (NIA) project (Kruse et al., 2006).

Kruse says the NIA project found that sorting through 5-10 images per second is possible. In fact, the analyst can do the job 5-7 times faster using the triage system than unaided. This is because the triage system picks up brain waves showing recognition of a target even before the human analyst is cognizant he has spotted it” (Hughes, 2008). Now that reading brain waves has been shown to work, the aim is to optimize it for use by a large population of analysts and at an accuracy rate that meets the needs of the intelligence agencies.

9.4 Epidermal Electronics

Future interaction devices could take the form of electronic circuit mounted directly onto our skin like (temporary tattoos) that can wirelessly transmit electric signals that they sense from our nerves and muscles (Figure 83). Kim et al. (2011) were able to produce ultrathin electronics that can pick up electrical signals produced by the heart. They applied a prototype with a microphone to a person’s throat and were able to differentiate between four words: up, down, left and right. Possible applications of epidermal electronics would include sensors that monitor heart and brain activity and computers that operate via the subtlest voice commands or body movement.

66 DRDC Valcartier TR 2012-425 Figure 83: Epidermal electronics prototype with physical properties matched to the epidermis (Kim et al., 2011).

9.5 Reconstructing Images from Brain Scans

“Using functional Magnetic Resonance Imaging (fMRI) and computational models, UC Berkeley researchers have succeeded in decoding and reconstructing people’s dynamic visual experiences – in this case, watching Hollywood movie trailers. As yet, the technology can only reconstruct movie clips people have already viewed. However, the breakthrough paves the way for reproducing the movies inside our heads that no one else sees, such as dreams and memories, according to researchers” (Anwar, 2011). Before that, another research had allowed scientists to reconstruct static images from brain scans as shown in Figure 84 (Miyawaki et al., 2008). Eventually, this technology could allow a mental picture from an analyst’s mind to be directly transferred into the computer. Another application of this research would be to issue visual commands.

DRDC Valcartier TR 2012-425 67 Figure 84: Visual stimulus reconstruction using fMRI (Miyawaki et al., 2008).

9.6 Brainbow Neuroimaging

The Brainbow neuroimaging technique enables researchers to see how individual nerve cells in the brain are woven (Livet et al., 2007). It uses fluorescent proteins to distinguish individual neurons in the brain from neighboring neurons (Figure 85). It could lead to better understanding of the human brain although we are still far from being able to use this for BCIs.

Figure 85: Images of the brain produced using the new Brainbow technique (Livet et al., 2007).

68 DRDC Valcartier TR 2012-425 9.7 Challenges and Limitations

Although neural interfaces offer great promises, the current state of the technology is only usable for simple commands and is only beginning to be applied outside of laboratories. The nature itself of the technology also presents challenges that will be difficult to overcome. It is unrealistic to expect that we will be able to sense and understand the huge number of electric signals that are generated within a human brain.

Also, Christoff et al. (2009) showed in their research that mind wandering occupies a large proportion of our conscious though. “According to Jonathan Schooler of UC, Santa Barbara, we think our minds are wandering about 10% of the time, when it is actually much more. In normal every day activities our mind is wandering up to 30% of the time, and in some cases, for instance when driving on an uncrowded highway, it might be as high as 70%” (Weinschenk, 2010).

DRDC Valcartier TR 2012-425 69 This page intentionally left blank.

70 DRDC Valcartier TR 2012-425 10 Virtual Assistant / Virtual Advisor

A virtual assistant or advisor, also known as Intelligent Software Assistant (ISA), is a conversational, computer-generated character that simulates a conversation and that is capable of providing guidance, bringing updates, pointing to significant events, and generating automatic triggers and alerts as required. A virtual assistant incorporates natural-language understanding, dialogue control, domain knowledge and a visual or vocal representation that changes according to the content of the dialogue. “By tuning the characters’ appearance and combining facial gestures and emotional cues, the virtual advisers can also provide context and convey appropriate levels of trust” (Wark et al., 2005).

Many people are familiar with animated interface agents embedded in the Microsoft Office environment. These agents took the form of a paper clip, a little dog or a wizard with various anthropomorphic characteristics. An experiment conducted by Serenko et al. (2007) shown that people did not really perceive the usefulness nor the enjoyment of such characters, even if some of these people had animation predisposition, enjoyed computer games or had an orientation towards innovation in information technology. Their recommendation is the following: “in order to achieve commercial success with interface agents, designers should put more emphasis on value-added features, realize functionality that goes beyond traditional direct-manipulation interfaces, and respect individual differences and preferences by allowing users to opt-out from using this technology” (Serenko et al., 2007).

Fortunately, the deployment of Siri onto the IPhone 4s (5.4.1) has introduced virtual personal assistants to millions of consumers and made them a reality. As already seen, Google is also working on virtual assistants (Section 5.4.3). These assistants are not just for the smart phones, but will become available on a number of platforms. For example, Zypr is a free-to-use Web service platform that provides voice-controlled access to a range of services such as mapping, navigation, social media, calendars, weather (Bright, 2011).

In order to support a virtual assistant, various information technologies are put to bear. This includes artificial intelligence, knowledge exploitation, management and discovery, human- computer interaction, and natural language processing. On behalf of DRDC, Canada Institute for Scientific and Technical Information (CISTI) has conducted a comprehensive literature survey in terms of the technologies, research initiatives and available products (CISTI, 2011). MIT, in its 2009 Technology Review, has identified ISA as one of the most promising emerging technologies that can change the way we live (Naone, 2009).

In a military intelligence context, virtual assistants will be growing in importance in terms reducing the human cognitive overload problem; they will provide assistance to the user by conducting a wide variety of tasks, including searching and organizing information, tracking people, managing schedules, assigning tasks, summarizing documents, mediating interactions, guiding and reminding the user, learning procedures and preferences, and suggesting alternative courses of actions. An interesting example of this is the DARPA initiative Personalized Assistant that Learns (DARPA, 2011b).

10.1 CALO / PAL - What Can a Virtual Assistant Do?

During the period 2003 to 2008, DARPA, through the PAL (Personalized Assistant that Learns) program, investigated and integrated a number of AI technologies into CALO (Cognitive Assistant that Learns and Organizes). CALO provides a good example of what a Virtual Assistant can do. Figure 86a shows the main CALO functions. Figure 86b provides examples of CALO’s

DRDC Valcartier TR 2012-425 71 learning capabilities. The PAL video gives an interesting vision of the use of an ISA in a military context (DARPA, 2011b).

Figure 86: a) CALO functions b) Examples of CALO learning capabilities.

10.2 User Interaction with the Virtual Assistant

A virtual assistant should allow the users to interact with the system(s) using multimodal interaction (e.g. voice, pointing, writing, drawing, gesture, eye/gaze, neural/brain interfaces, emotion detection). Moreover, “full understanding requires identification of speakers and addressees, along with resolution of reference to other participants and objects, and integration of both verbal and non-verbal communication” (Purver et al., 2005). In a military context, the communication with an ISA will evolve around topics such as tasks, planning activities, Standard Operating Procedures (SOPs), briefing material, documents.

In many cases, as the user will be using natural language interaction combined with movement, there will be some ambiguities. For example, in the PAL video (DARPA, 2011b) the user says: “These are my priorities… I’m attending this meeting… I need you to setup my briefing package”, while pointing on the screen. The deictic references to ‘my priorities and this meeting’ cannot be solved by speech alone, and in some cases the location being pointed to cannot be resolved to a high-enough precision by vision alone.

The Siri running on the IPhone 4S uses ‘Active Ontologies’ to analyze the requests made by users and call the relevant partner APIs to get suggestions. “For example , a request like ‘suggest me some good italian place for dinner’ is interpreted using the domain ontology it has relating to restaurants, comprising a domain-specific vocabulary, rules of interaction, reviews from partner APIs , leveraging the user current location (if needed), to provide relevant suggestions... it’s not a search engine, but an answer engine meant to make interactions inherently semantic; hence more personal and meaningful” (Natarajan, 2011).

Within the CALO initiative, in order to support the understanding of the interaction taking place and resolve ambiguities, a unified multimodal discourse ontology and knowledge base was designed. This ontology is coupled with a dialogue-understanding framework which maintains and shares multiple hypotheses between discourse-understanding components (Purver et al., 2005).

10.3 Virtual Assistant Interaction with the User

The virtual assistant should be able to interact with the users by conducting several tasks: present tools and information; answer questions; ask for clarification; remind the user of some tasks or 72 DRDC Valcartier TR 2012-425 procedures, and propose alternative possibilities. The ISA may even give a briefing. This interaction is provided through voice output and/or information display. As illustrated in the PAL video (DARPA, 2011b), the ISA should be able to interact directly with the user’s screen, in: highlighting information elements already displayed; displaying information in a new window, organizing it as the user wants to see it; gathering a set of documents; and filtering information based on user requests.

The ISA could also provide tools to capture information about a meeting. For instance, the CALO Meeting Assistant provides for distributed meeting capture, annotation, automatic transcription and semantic analysis of multiparty meetings (Tur et al., 2008).

The interface should adapt to the role and current tasks of the user, as well as his preferences. The ISA should be proactive, observing what the user is doing, his emotional state (section 3.8) and his mental or cognitive state (section 9.2), suggesting sequences of events, or waiting for an appropriate moment to stepping in. If the user moves to a large screen display, the user should be recognized using biometry and the interface should adapt to the distance of the user to the display, in terms of font sizes, granularity and quantity of information presented.

ISA could exploit story telling technology to present information. Because of its richness, “storytelling is recognized as an effective mechanism for establishing shared context and transferring tacit knowledge throughout an organization” (Wark and Lambert, 2007). Analysts ultimately tell stories in their presentations, with the stories providing a way to organize evidence by events and by source documents (Bier et al., 2008). (Gershon and Page, 2001) and (Wojtkowski, W. and Wojtkowski, W.G., 2002) present a number of storytelling principles and techniques, such as use of comics metaphor, animating the events, setting mood and place in time, and intentional omission. (Baber et al., 2011) discuss the formalism of stories. Stories should be organized around the actions or events, identifying the actors, action type, modality of action, context, rationale, but most importantly identify the relationships between these. Section 11.6 provides a more comprehensive review of storytelling technology.

Last but not least, the ISA should, over time, behave and perform in such a way that it generates a high level of trust in the user. This is particularly important as in the intelligent domain, the problems addressed are often complex, the analysts are highly skilled people and they are imputable in their decisions. As with any collaboration with another person, the level of trust in the ISA will need to develop with time, supported by various strategies. Some of the strategies could be the following: the ISA should provide access to a traceability of the rationale behind any action taken or recommendation made (e.g. interpretation and argumentation, as well as the sources of information); the system should allow the user to associate confidence indicators for different types of tasks (e.g. the user may develop trust more rapidly in the weather prediction than in the recommendations for courses of action); appropriate training should be provided, in particularly, based on reinforcement learning; the ISA should exploit emotion recognition (section 3.8) and cognitive state recognition (section 9.2), in order to determine if its actions and recommendations bring confusion to the user.

Reeves (2004) states that “The presence of a character can increase and sustain trust”. A first factor is the mere presence, where people believe that social presence is useful and preferred, in particular in situations where errors are likely. People also believe that these characters can add specific information to a conversation or during the conduct of a task, which increases trust. It can be in the form of acknowledging a comment, gathering appropriate information, or providing credibility because of expertise in the related domain.

DRDC Valcartier TR 2012-425 73 10.4 Virtual Assistant Representation

The ISA can take the form of a speech output, a simple icon or a two- or three-dimensional representation of a character (avatar) or simply appear through interactions such as highlighting or pop-ups. Figure 87 shows an automated online assistant, having a text-based dialog system and a humanoid avatar. The ISA can be personified, in particular from a gender perspective (male, female, neutral). The avatars can also exhibit other characteristics (age, ethnic group, profession - civil vs. military) based on the users’ social-cultural context and preferences. Facial and voice features (serious / smiling, tone of the voice) could reflect the importance / urgency, or certainty of a message. Different avatars could be used to support different ISA tasks. For example, the avatar for the weather analyst might be different, in terms of gender, age and profession, than the one that recommends the course of action.

In some cases, the avatar would be transmitting information by talking. The voice and/or facial components are then the important elements. During ‘on-the-map planning or video conferencing activities’, it may be useful to have a full-size avatar embedded in the display as if it were another user. An example of this is in Figure 87.

Figure 87: a) A humanoid ISA avatar with a text-based dialog (Wikipedia 2012b); b) A ISA Avatar embedded in the display (DARPA, 2011b).

“Experiments conducted by DSTO and the University of Adelaide have shown that different facial characteristics displayed by a speaker will influence a user’s affective trust, implicitly imparting uncertainty to the user if required, which at times can be a vital form of qualification for any information delivered” (Australian Defence Science, 2008). “By tuning the characters’ appearance and combining facial gestures and emotional cues, the virtual advisers can also provide context and convey appropriate levels of trust” (Wark et al., 2005). Figure 88 is an example of an avatar taken from the Future Operations Centre Analysis Laboratory (FOCAL) initiative conducted by DSTO in Australia.

Figure 88: A Virtual Assistant used in FOCAL (Wark and Lambert, 2007). 74 DRDC Valcartier TR 2012-425 We must remain careful, however. Wark and Lambert (2007) cite the spectre of the Uncanny Valley, postulated by Mori (1970) where “as you make a simulacrum look more human, people will identify with it more strongly until a point is reached as the simulacrum approaches ‘human looking’ where their affinity will suddenly drop steeply, as the differences become more important than the similarities”. What level of realism should we implement in the avatar? Will Mori’s principle still hold as a new wave of people, grown up with 3D / virtual reality gaming experience, become military analysts? More experimentation with avatars will be needed to reassess their benefits and drawbacks.

“Evidence from other labs supports this phenomenon. Among the more interesting is work by Merola, Pena and Hancock of Cornell, who found that people playing avatars with dark clothes behaved more aggressively. Yet more support came out this week as Nowak found that people are uneasy interacting with avatars of uncertain gender, supporting Reeves and Nass’ past work that humans interact with unknown agents (robots, avatars, whatever) and seek to establish these things, in this order:

1) Is this thing human? 2) What is its gender? 3) Is it compatible with me intellectually and socially?”

Since people will interact socially with embodied agents even though they know they are controlled by a machine (Krämer et al., 2012), scientific literature regarding social interactions should also be reviewed to identify the behaviours that an ISA should display.

DRDC Valcartier TR 2012-425 75 This page intentionally left blank.

76 DRDC Valcartier TR 2012-425 11 Advanced HCI Concepts & Techniques

This section presents a numbers of miscellaneous advanced concepts and techniques in HCI.

11.1 Advanced Information Visualization / Visual Analytics

Military operations can be very complex. The multiple dimensions of the information and the quantity of linkages to understand may lead to a cognitive overload. In order to support military intelligence analysis, it is essential to fully exploit the power of information visualization. Information visualization is defined as: “the use of computer-supported, interactive, visual representations of abstract data to amplify cognition” (Card et al., 1999). This definition refers to two essential parts contributing to information visualization: the first is the cognitive ability of the human to perceive and make a mental representation of information or a situation; the second is the supporting technology that allows the presentation of this information or situation.

Information visualization has constituted a significant research area which studied various human cognitive characteristics, such as preattention and inattentional blindness, and explored and developed various techniques to represent information in a salient way and provide efficient interaction. “Information visualization promises to help us speed our understanding and action in a world of increasing information volumes” (Card, 2008).

More recently, Visual Analytics (VA) has emerged as a multidisciplinary field of research that leverages information visualization and a number of other disciplines. “Visual analytics is the science of analytical reasoning facilitated by interactive visual interfaces” (Thomas and Cook, 2005). “Visual analytics focuses on the following areas: x Analytical reasoning techniques that enable users to obtain deep insights that directly support assessment, planning, and decision making; x Visual representations and interaction techniques that take advantage of the human eye’s broad bandwidth pathway into the mind to allow users to see, explore, and understand large amounts of information at once; x Data representations and transformations that convert all types of conflicting and dynamic data in ways that support visualization and analysis; x Techniques to support production, presentation, and dissemination of the results of an analysis to communicate information in the appropriate context to a variety of audiences” (Thomas and Cook, 2005).

In other words, “the basic idea of visual analytics is to combine the strengths of automatic data analysis with the visual perception and analysis capabilities of the human user. It uses visualizations, user interaction and data analysis techniques to find insight from complex, conflicting and dynamic information. Visual analytics is especially focused on situations where the huge amount of data and the complexity of the problem make automatic reasoning impossible without human interaction” (Jaሷrvinen et al., 2009).

Figure 89 provides two examples of VA. The first one shows a Theme River™ where the main news feeds are depicted in time and allows understanding which events have drawn more attention and in some cases, spun off other topics of interest. The second example is a Galaxy view based on the extraction of concepts from a large set of documents and the regrouping of the documents into concept clusters.

DRDC Valcartier TR 2012-425 77 VA is going through a significant growth and a large number of research projects are aimed at providing tools for intelligence analysis. It is expected to become a technology of choice in the military intelligence world as analysts must digest vast amounts of collected data and make sense of them, identify patterns and trends. In particular, through the use of VA, intelligence analysts will be able to: x “Synthesize information and derive insight from massive, dynamic, ambiguous, and often conflicting data; x Detect the expected and discover the unexpected; x Provide timely, defensible, and understandable assessments; x Communicate assessment effectively for action” (Thomas and Cook, 2005).

Figure 89: a) Visual Analytics examples: Theme River visualization (Ribarsky, 2009) b) Galaxy visualization (Future Point, 2011).

11.2 Configurable Workspaces

Novel software environments allow users to easily configure their working environments through drag-and-drop operations. Software developers have been working on ways of extending working spaces on various devices

11.2.1 SitScape

SitScape (2012) is a software application that allows users to easily aggregate and visualize disparate applications and information sources into a collaborative User-Defined Operating Picture (UDOP), running onto a web browser for live monitoring, situational awareness, information sharing, and visual contextual collaboration. Various applications can easily be dragged and dropped onto the browser, and its different tabs, while maintaining a live feed connexion. Figure 90 shows a maritime portal displayed on a large screen using SitScape.

78 DRDC Valcartier TR 2012-425 Figure 90: Example of a maritime portal on a large display with SitScape.

11.2.2 Apple’s Spaces

In its operating system, Apple has implemented the concept of virtual desktops (‘Spaces’) that can be tailored to the user’s needs and work habits, running multiple applications in full screen without the workspace becoming too cluttered. For example he can run a word processor on one desktop and a web browser on another. The software allows the user to easily and efficiently jump from one space to another, using a bird’s eye view widget (Tatnell, 2010). Figure 91a shows a representation of four virtual desktops, using Spaces. In the new Mac Operating system (Lion), Spaces is now integrated into the Mission Control application (Figure 91b), together with the Dashboard and Exposé.

It appears that this capability of running multiple independent desktops on one computer might be brought to the iPad, extending virtually the real estate of the tablet (Gigaom, 2011).

Figure 91: a) Apple's Spaces (Tatnell, 2010) b) Mission Control (Apple, 2012).

DRDC Valcartier TR 2012-425 79 11.3 Adaptive User Interfaces

The concept of systems that are aware of the user’s context and react accordingly is a central aspect of many paradigms such as ambient intelligence and adaptive user interfaces. These systems and technologies have the following characteristics: “ x Embedded: Many invisible dedicated devices throughout the environment. x Personalized: The devices know who you are. x Adaptive: Change in response to you and to the environment. x Anticipatory: Anticipate your desires as far as possible without conscious mediation: PRE-sponsive, not responsive.” (Zelkha et al.,1998)

An adaptive user interface changes its layout and element of information based on the user roles, needs and preferences. This allows tailoring the interface to the task at hand. This strategy can help overcome the information overload problem by focussing on the required information and processes. Eye-tracking (section 3.4) can be used to observe where the user is looking at and customize accordingly the user interface. BCIs (Section 9) can also provide information about the state of mind of the user and allow the interface to be readjusted accordingly.

In a concept demonstration prototype related to the Precision Information Environments project (section 11.6), an adaptive interface was implemented using a combination of a data wall with a MS Kinect sensor to detect the nearest user. Face recognition is used to identify the user and the information is tailored to his work profile. The distance between the user and the display is also measured and the text size is adjusted to be readable at the user position.

11.4 Advanced Interface Widgets

A widget is an element of a GUI that displays an information arrangement changeable by the user, such as a window or a text box, and enabling direct manipulation of a given kind of data (Wikipedia, 2011e). Interface widgets can significantly facilitate the task of a user. Silicone ILluminated Active Peripherals, or SLAP widgets, are physical interface components that add movable, tactical controls to a device such as a multi-touch table (as shown in Figure 92).

Figure 92: The SLAP Widgets set includes a) Keypads b) Knob c) Slider d) Keyboard, and all control details are provided through the display (Weiss, 2009).

The transparency of these widgets makes them very flexible as the labels and graphics are provided by the software and can be modified at any time to adapt to user needs. These widgets are meant to allow users to interact with touch device without having to continuously look at the screen. “SLAP buttons, sliders, knobs and keyboards have the physical shape of real devices to provide the right haptic feedback, but are still easily relabeled using a tabletop rear projection” (Weiss, 2009).

Ergodex has developed its DX1 Input System. It is a highly customizable programmable keypad, where keys can be placed anywhere on the pad and can be moved at any time to different 80 DRDC Valcartier TR 2012-425 locations (Figure 93). “It also comes with a software suite that lets you program the DX1 keys to perform tasks (macros). Use the pre-built macros (for more than 60 software programs and games) that come with the DX1” (Disabled Online, 2012).

Figure 93: Ergodex configurable keyboard (Disabled Online, 2012).

11.5 Smart Lenses / Magic Lenses

As illustrated in Figure 94, smart lenses technology, such as the Pliable Display Technology from Idelix Systems, allow a user to handle vast amounts of data and accurately interact with critical details in context (Baar and Shoemaker, 2004).

Figure 94: Examples of smart lenses from Idelix (Baar and Shoemaker, 2004).

Lenses can be used to magnify the underlying information space, but they are even more powerful when they add another information layer to the detailed area. This is the idea behind the magic lenses technique (Fishkin and Stone, 1995). Magic lenses allow the application of an effect (additional information, filtering, alternative representation, etc) to the detailed part of the display. This concept has been extended to 3D and coupled with augmented reality. That allows a 3D scene to be affected dynamically given contextual information, for example, to support information filtering (Mendez et al., 2006). Figure 95 shows a lens that renders differently the vessel trees of a liver depending on their type and the lens intersection.

DRDC Valcartier TR 2012-425 81 Figure 95: 3D Magic Lenses in an augmented reality setup (Mendez et al., 2006).

This idea can also be applied to surface computing or wall display using smart phones or tablets devices as lenses. However, significant integration efforts are required to tie together multiple devices. To demonstrate their “seamless computing” framework, nsquared developed an architecture application example that shows the use of a smart phone, a tablet computer, a wall display, a MS Kinect sensor, and vocal recognition (shown in Figure 96). The demonstration goes beyond the concept of magic lenses and displays some potential applications of pervasive and ubiquitous environments.

Figure 96: A tablet device can be used as a magic lens over a surface computing display to show additional room information in this architecture application (nsquared, 2011).

11.6 Storytelling

For thousands of years, storytelling has been the mean used by humanity to transfer knowledge. More recently, the field of cognitive narratology (for an introduction, see Herman, 2000) has emerged to study what makes storytelling so well suited to human cognitive abilities. Stories have also been used in command training by the U.S. Army (Frame and Lussier, 2000).

Gershon and Page (2001) suggested that storytelling techniques would be extremely useful to the information visualization domain because stories convey effectively large amounts of implicit information and are easily memorable, as depicted in Figure 97.

82 DRDC Valcartier TR 2012-425 Figure 97: Stories allow for short and memorable communications (Gershon and Page, 2001).

Many researchers worked to create interactive visual narratives from data to convey information more effectively. Segel and Heer (2010) review many examples (such as Figure 98 and Figure 99) and propose a Design Space Analysis of narrative visualizations.

Figure 98: Staged animated transitions between chart types (Segel and Heer, 2010; Gapminder, 2005).

DRDC Valcartier TR 2012-425 83 Figure 99: Afghanistan: Behind the Front Line (Segel and Heer, 2010; Green et al., 2009).

Raconteur 2 is “a system for conversational storytelling that encourages people to make coherent points, by instantiating large-scale story patterns and suggesting illustrative media” (Chi and Lieberman, 2011). This system offers assistance to a user that is in the process of telling a story to another user. The Raconteur user interface (Figure 100), includes a chat box (a), a preview window (b) and Raconteur’s suggestion panel (c) for observing the story patterns and the multimedia repository (Chi and Lieberman, 2011).

Figure 100: The Raconteur user interface (Chi and Lieberman, 2011).

Nichols and Hammond (2009) built an automated system that can generate novel multimedia content. One application is “News at Seven”, an automatically generated news show (Figure 101). This kind of automation of information presentation has a high potential for delivering results obtained through processing performed by artificial reasoning systems. The “Show Me”

84 DRDC Valcartier TR 2012-425 capability offered in Tableau is a different example of automatic selection of visualization techniques (Mackinlay et al., 2007). This feature suggests visual representations that consider data characteristics and visualization design best practices (Figure 102).

Figure 101: Kaitlin and Sam discussing the new movie Jumper in News at Seven (Nichols and Hammond, 2009).

Figure 102: Show Me alternatives proposed by Tableau software (Mackinlay et al., 2007).

Military intelligence would benefit from storytelling technology, as it would provide clearer briefings and a better sharing of analysis.

11.7 Precision Information Environments

In order to appreciate how the evolution of HCI could benefit future intelligence analysis, it is very informative to examine the work carried out by the US Department of Homeland Security and Pacific Northwest National Laboratory to design and develop future work environments for the emergency management community. Called Precision Information Environments (PIEs), these environments are meant to provide planners and responders with precise, relevant information and with tools that aid collaboration, information sharing, and decision support (DHS/PNNL, 2010).

A video illustrating the concepts of a PIE environment to support fire fighters has been developed (DHS/PNNL, 2010) and exhibits many of the HCI trends described in the current report. Some of the key concepts are the following: x Pervasive Sensor Network. Live data streams from instrumentation in the field give emergency management personnel an intimate view of remote conditions.

DRDC Valcartier TR 2012-425 85 x Synthetic Environments. Data feeds from the field are synthesized into a navigable virtual environment through which collaborators can better understand an event and explore response strategies. x Adaptive User Interface. User models define the roles, responsibilities and perspectives of each person using a PIE. Software interfaces adapt their form to support each user's tasks and preferences (Figure 103a). x Live Status Tracking. The tasks of every PIE user are tracked so that a clear picture of a response activity is always available (Figure 103b). x Seamless Information Transfer. Through natural interaction, information can be moved from one interface to another; relevant information follows the user and adapts its form to the device on which it is shown (Figure 103c). x Multimodal Interaction. Use of surface computing is used for collaborative interaction towards shared situation awareness and decision making (Figure 103d).

Figure 103: Illustration of Precision Information Environments concepts (DHS/PNNL, 2010).

x Intuitive Collaboration. Distributed users can interact with each other and with PIE software as if they were co-located. In the illustration shown in Figure 104a, distributed collaborators interact easily with each other and with the information using a live wall. Remote-users are embedded in the image, in mirror mode in order to reflect when they look at the same information on live wall. x Advanced HCI Widgets. Users are provided with advanced graphical user interface widgets and natural interaction mechanisms (Figure 104b). x Seamless Communications. Information that is relevant to him given his role, mission and context is automatically presented, and in form appropriate to the device on which he is interacting. x Ubiquitous Displays. The PIE architecture allows information to be shared across devices easily (Figure 104c); to support more in-depth analysis, the view from a mobile device can be shared with a nearby display. As illustrated in Figure 104d, the truck's windshield, equipped with an embedded OLED display, is used as an alternate display and controlled by the wrist computer as an input device. 86 DRDC Valcartier TR 2012-425 x Decision Support in the Field. Personnel in the field have will have as much computational support as those in the operations center. By understanding each user's tasks, PIE offers customized data and recommendations. x Augmented Reality. New interfaces will give users a unique view of their world, populating it with context-sensitive, task-appropriate information.

Figure 104: Illustration of Precision Information Environments concepts (DHS/PNNL, 2010).

DRDC Valcartier TR 2012-425 87 This page intentionally left blank.

88 DRDC Valcartier TR 2012-425 12 Conclusion

The nature of military operations has changed significantly over the last decade. Asymmetric warfare, urban and human terrain settings have increased the complexity of these operations, Military intelligence analysts are faced with a huge amount of information, of various types, which they need to make sense of. They also need to collaborate with different military and civilian organizations within a JIMP context. Luckily, communication and information technology has progressed significantly to support them in their work. In particular, new and future HCI technologies provide significant opportunities to support deployed intelligence collators as well as analysts working in command centers.

This report has identified a number of HCI trends relevant to the FIAC and collective command and control in general: x Ubiquitous communications and computing will continue and people will be able to interact with the technology that surrounds them in more accessible, intuitive and less restrictive ways. x Multimodal and natural interaction will be the common interface of the future. Multi-touch tables / surfaces will allow analysts to come together around the table to debate a military situation, in a way similar to laying out paper maps on a table. x Smart phones / smart personal device assistants have gained a significant momentum and will be provided with a large variety of advanced intelligence applications, such as augmented reality and culturally-assisted translation. x Deployed personnel will also benefit from small, light weight and flexible displays. x Smart room environments will be equipped with large group displays and advanced workstations and collaboration tools to provide better shared multi-intelligence situation awareness and collaboration. x Biometry will enhance security, allowing authenticating users’ of facilities and computers, and identifying / recognizing belligerents. x Advances in biometry and brain / neural interfaces will the recognition of users’ emotions, stress level or cognitive overload, in order to provide adapted interfaces. x Virtual assistants will help the user in his management of information and activities, and help reduce his cognitive overload problem. x Users will have access to mixed reality capabilities, such as head-mounted displays, immersive environments, and augmented reality capabilities, to develop complex situation understanding, to conduct mission rehearsal before being sent in a hostile environment or to collaborate. x Systems will be designed with intelligent, adaptive interfaces, understanding the user roles and the situation at hand. x Advanced visual analytics approaches will allow to derive insight, identify patterns and trends. x The users will also benefit from advanced HCI concepts to better organize the information, to interact with it or to present it. This includes the use of configurable workspaces, adaptive user interfaces, advanced interface widgets, smart lenses, and storytelling.

DRDC Valcartier TR 2012-425 89 As indicated in the introduction, this report is only one contribution to the vision of the FIAC. In addition, a number of other documents have been produced, including a white paper “towards a Cohesive R&D Program Definition of the FIAC”, the production of an illustration of a future multi-intelligence center, the writing of a fiction narrative of a future intelligence environment, and the development of operating concepts for the FIAC. Following these efforts, a prioritization of the design and development activities will be performed and a roadmap will be produced. In terms of the HCI aspect of the FIAC, a number of the most promising techniques and concepts will be selected as building blocks for the first versions of the FIAC.

90 DRDC Valcartier TR 2012-425 13 References

ACECAD (2011), DigiMemo A502, http://www.acecad.com.tw/index.php?option=com_content&view=article&id=21&Itemid=66, retrieved November 2011.

ACIS (2011), Augmented Cognition International Society, What is Augmented Cognition? http://www.augmentedcognition.org/, retrieved January 2011.

Ackerman, S. (2011a), First Look: Inside the Army’s App Store for War, Wired, 27 April 2011, http://www.wired.com/dangerroom/2011/04/armys-app-store-for-war/, retrieved January 2012.

Ackerman, S. (2011b), The iPhone 4S’ Talking Assistant Is a Military Veteran, Wired, 5 October 2011, http://www.wired.com/dangerroom/2011/10/siri-darpa-iphone/, retrieved March 2012.

AFRL (2001), Air Force Research Laboratory, Interactive Data Wall, 12 June 2001, http://cartome.org/datawall.htm, retrieved January 2011.

Anthony, S. (2011), Panasonic Toughpad Tablet: Now the Army can Play Angry Birds Too, Extreme Tech, 7 November 2011, http://www.extremetech.com/computing/103979-panasonic- toughpad-tablet-now-the-army-can-play-angry-birds-too, retrieved November 2011.

Anwar, Y. (2011), Scientists Use Brain Imaging to Reveal the Movies in our Mind, 22 September 2011, http://newscenter.berkeley.edu/2011/09/22/brain-movies/, retrieved October 2011.

Apple (2011), Your Wish is its Command, http://www.apple.com/iphone/features/siri.html, retrieved November 2011.

Apple (2012), OS X Lion – Mission Control, http://www.apple.com/macosx/whats-new/mission- control.html, retrieved April 2012.

Arcane Technologies (2011), Military Applications, http://www.arcane- technologies.com/en/index.html?section=demo&sub=demo_military.htm, retrieved January 2011.

ASU (2012), Breakthroughs, Arizona State University - FDC, http://flexdisplay.asu.edu/breakthroughs, retrieved April 2012.

Augusto, J.C. (2007), Ambient Intelligence: the Confluence of Ubiquitous / Pervasive Computing and Artificial Intelligence, In Schuster, A. (ed.), Intelligent Computing Everywhere, Springer London, pp 213–234, 2007.

Australian Defence Science (2008), Clones that Counsel - Technical Sheet, Australian Defence Science, Volume 16, Number 2, 2008.

Ayers, Joe (2011), Direct2Dell - Dell Mobile Security for Android Officially Certified for Government Use by DISA, 28 October 2011, http://en.community.dell.com/dell- blogs/direct2dell/b/direct2dell/archive/2011/10/28/dell-mobile-security-for-android-officially- certified-for-government-use-by-disa.aspx, retrieved November 2011.

Baar, D. and Shoemaker, G. (2004), Pliable Display Technology for the Common Operational Picture, NATO Workshop IST-043/RWS-006: Visualisation and the Common Operating Picture (COP), 14-17 September 2004, Canadian Forces College Toronto, Canada.

DRDC Valcartier TR 2012-425 91 Baber, C., Andrews, D., Duffy, T. and McMaster, R. (2011), Sensemaking as Narrative: Visualization for Collaboration, Proceedings VAW2011, University London College, 7-8 Sept 2011, London, UK.

Barco (2011), BARCO, OV-1015 100" SXGA+ DLP Projection Module, http://www.barco.com/en/product/2018, retrieved January 2011.

Bernier, F., Boivin, É. and Mokhtari, M. (2008), Virtual Immersive Facility – Future Battle Commanders with Advanced Decision-Making Capabilities, Defence R&D Canada, 28 February 2008.

Bernier, F., Mokhtari, M., Boivin, É. and Drolet, F. (2010), Virtual Immersion Laboratory (VIL) & Serious Gaming Laboratory (SGL), 11 May 2010, DRDC-Valcartier-SL-2010-473.

Bier, E.A., Card, S.K. and Bodnar, J.W. (2008) Entity-based Collaboration Tools for Intelligence Analysis, Proceedings IEEE Symposium on Visual Analytics Science and Technology, 19 - 24 October 2008, Columbus, OH, 99-106.

Biometric Consortium (2011), An Introduction to Biometrics, http://www.biometrics.org/html/introduction.html, retrieved December 2011.

Bot Bench (2011), Kinect SDK Beta Refresh, http://botbench.com/blog/2011/08/02/kinect-sdk- refresh/, retrieved December 2011.

Bright, P. (2011), Siri for Everyone, with Pioneer’s Zypr API, Gear & Gadgets, November 2011, http://arstechnica.com/gadgets/news/2011/11/siri-for-everyone-with-pioneers-zypr- api.ars?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+arstechnica%2F index+%28Ars+Technica+-+Featured+Content%29&utm_content=Google+International, retrieved April 2012.

Burns, C. (2011), Throwable Panoramic Ball Camera Details Released, Still Seeks Investors, 15 December 2011, http://www.youtube.com/watch?v=Th5zlUe6gOE, retrieved May 2012.

Card, S. (2008), Information visualization, in A. Sears and J.A. Jacko (eds.), The Human- Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications, Lawrence Erlbaum Assoc Inc, 2008.

Card, S., Mackinlay, J. and Shneiderman, B. (1999), Readings in Information Visualization: Using Vision to Think, Morgan Kaufmann Publishers, 1999.

Caridakis, G., Castellano, G., Kessous, L., Raouzaiou, A., Malatesta, L., Asteriadis, S., Karpouzis, K. (2007), in IFIP International Federation for Information Processing, Volume 247, Artificial Intelligence and Innovations 2007: From Theory to Applications, pp. 375-388.

CES (2009), Samsung Transparent OLED Display, January 2009, http://www.youtube.com/watch?v=8G3wWmtkN88, retrieved January 2011.

Chandrasekhar, A., Kaimal, V.P., Bhamare. C. and Khosla, S. (2011), Ambient Intelligence: The Next Generation Technology, International Journal on Computer Science and Engineering (IJCSE), 2011, http://www.enggjournals.com/ijcse/doc/IJCSE11-03-06-172.pdf, retrieved March 2012.

92 DRDC Valcartier TR 2012-425 Chawla, A. (2011), 10 Touchscreen Tables for Multimedia Professionals, 5 September 2011, http://www.designbuzz.com/entry/10-touchscreen-tables-multimedia-professionals, retrieved April 2012.

Cheshire, T. (2012), 25 Big Ideas for 2012: Ubiquitous Face Recognition, Wirek,co,uk, 4 January 2012, http://www.wired.co.uk/magazine/archive/2012/01/features/ubiquitous-face-recognition, retrieved April 2012.

Chi, P.-Y. and Lieberman, H. (2011), Intelligent Assistance for Conversational Storytelling Using Story Patterns, Proceedings of the 16th international conference on Intelligent user interfaces (IUI’11), ACM, Palo Alto, CA, USA, February 2011.

Christoff, K., Gordon, A.M., Smallwood, J., Smith, R. and Schooler, J.W. (2009), Experience Sampling during fMRI Reveals Default Network and Executive System Contributions to Mind Wandering. Proceedings of the National Academy of the Sciences, 11 May 2009.

CISTI (2011), Intelligence Virtual Analyst Capability Literature Survey, STI Assessment 8102, Defence R&D Canada – Valcartier, Contract Report SDA-07-001-022, October 2011.

CMU (2012), Touché, Carnegie Mellon University, Next-Generation Computing, http://www.cmu.edu/homepage/computing/2012/spring/touche.shtml, retrieved May 2012.

CNN Money (2010), Techmate - Inside Microsoft's gadget lab, 15 June 2010, http://money.cnn.com/video/technology/2010/06/15/tm_msft_lab_tour.fortune/, retrieved December 2011.

Coen, M.H. (1998), Design Principles for Intelligent Environments, Proceedings of the 1998 National Conference on Artificial Intelligence (AAAI-98), Madison, 26-30 July1998.

Contactum (2011), Where Reality Meets the Virtual World, http://www.augmented- reality.com/technology.php, retrieved January 2011.

Cook, D.J. and Das. S.K. (2007), How Smart are Our Environments? An Updated Look at the State of the Art, in Journal Pervasive and Mobile Computing archive, Volume 3 Issue 2, March, 2007, Elsevier Science Publishers B. V. Amsterdam, The Netherlands.

Corning (2011), A Day Made of Glass... Made Possible by Corning, 7 February 2011. http://www.corning.com/news_center/features/A_Day_Made_of_Glass.aspx, retrieved March 2012.

Corning (2012a), A Day Made of Glass 2: Same Day. Expanded Corning Vision, 3 February 2012, http://www.youtube.com/watch?v=jZkHpNnXLB0&feature=relmfu, retrieved May 2012.

Corning (2012b), A Day Made of Glass 2: Unpacked. The Story behind Corning's Vision, 3 February 2012, http://www.youtube.com/watch?v=X-GXO_urMow&feature=relmfu, retrieved May 2012.

Cruze-Neira, C., Sandin, D.J., DeFanti, T.A., Kenyon, R.V., and Hart, J.CV. (1992), The CAVE: audio visual experience automatic virtual environment, Communications of the ACM, Volume 35 Issue 6, pp. 64 – 72, June 1992

CyberGlove Systems (2012), CyberGlove II, http://www.cyberglovesystems.com/?q=products/cyberglove-ii/overview, retrieved November 2012. DRDC Valcartier TR 2012-425 93 Dang, C.T., Straub, M. and André, E. (2009), Hand Distinction for Multi-touch Tabletop Interaction. Proceedings ITS 2009, 101–108.

DARPA (2007), The Future of Augmented Cognition, http://video.google.com/videoplay?docid=-253368229180844307#, retrieved March 2012.

DARPA (2011a), Improving Warfighter Information Intake under Stress (AugCog), http://www.darpa.mil/dso/archives/warfighter/index.htm, retrieved January 2011.

DARPA (2011b), Personalized Assistant that Learns, http://www.youtube.com/watch?v=BF- KNFlOocQ&feature=youtu.be, retrieved March 2012.

Davies, C. (2012), LG Flexible E-Paper Devices Promised for April Launch, Slash Gear, http://www.slashgear.com/lg-flexible-epaper-devices-promised-for-april-launch-29220487/, retrieved April 2012.

Department of the Army (2010), Field Manual (FM) 2-0: Intelligence, March 2010, http://www.fas.org/irp/doddir/army/fm2-0.pdf, retrieved January 2011.

DHS / PNNL (2010), Department of Homeland Security (DHS) and Pacific Northwest National Laboratory (PNNL), Precision Information Environments, http://precisioninformation.org/, retrieved January 2011.

Dietz, P.H. and Leigh. D. (2001), DiamondTouch: A Multi-User Touch Technology. Proceedings UIST 2001, 219–226.

Digital Trends (2010), Kinect Price and Bundle Officially Announced, 20 July 2010, http://www.digitaltrends.com/gaming/kinect-price-and-bundle-officially-announced/, retrieved January 2011.

Disabled Online (2012), Ergodex DX1 Input System, http://www.disabledonline.com/products/direct-products/keyboardsmice/alternative- keyboards/ergodex-dx1-input-system/, retrieved April 2012.

Displax (2012), Displax Interactive Systems, http://www.displax.com/en/products/skin- dualtouch.html, retrieved March 2012.

DLCD (2009), Toward Land Operations 2021 Studies in Support of the Army of Tomorrow Force Employment Concept, Directorate of Land Concepts and Design, Kingston, Ontario, 2009, http://www.army.forces.gc.ca/DLCD- DCSFT/pubs/landops2021/Toward_Land_Operations_2021.pdf, retrieved January 2011.

Dohse, K.C., Dohse, T., Still, J.D. and Parkhurst, D.J., (2008), Enhancing Multi-user Interaction with Multi-touch Tabletop Displays using Hand Tracking. Proceedings ACHI 2008, 297–302.

DVE (2011), Digital Video Enterprises, http://www.dvetelepresence.com/products/immersion_room.asp, retrieved January 2011.

Edwards, L. (2010), BendDesk Introduced: The Desk that is a Touch Screen, 14 December 2010, http://www.physorg.com/news/2010-12-benddesk-desk-screen-video.html, retrieved March 2012.

EFF, (2003), Biometrics: Who's Watching You?, Electronic Frontier Foundation, September 2003, https://www.eff.org/wp/biometrics-whos-watching-you, retrieved March 2012

94 DRDC Valcartier TR 2012-425 Emotiv (2011), http://www.emotiv.com, retrieved October 2011.

ESRI (2010), Manage Data Faster and Easier with ArcPad 10, in ArcWatch: Your e-Magazine for GIS News, Views, and Insights, August 2010, http://www.esri.com/news/arcwatch/0810/arcpad10.html, retrieved November 2011.

ESRI (2011), ArcPad - Mobile GIS Software for Field Mapping Applications, http://www.esri.com/library/brochures/pdfs/arcpadbro.pdf, retrieved November 2011.

EXOpc (2011), EXOPC Interactive Desk - EXOdesk, 20 November 2011, http://www.youtube.com/user/EXOPCTV, retrieved November 2011.

Extreme Tech (2012), The Leap: Gesture Control like Kinect, but Cheaper and Much Higher Resolution, 21 May 2012, http://www.extremetech.com/computing/129638-the-leap-gesture- control-like-kinect-but-cheaper-and-much-higher-resolution, retrieved May 2012.

FCW (2008), Taking Intelligence Analysis to the Virtual World, Federal Computer Week, 4 September 2008, http://fcw.com/articles/2008/09/04/taking-intelligence-analysis-to-the-virtual- world.aspx, retrieved March 2012.

Fishkin, K. and Stone, M.C. (1995), Enhanced Dynamic Queries via Movable Filters, Conference on Human Factors in Computing Systems (CHI '95), Denver, CO, USA, pp. 415-420, 1995.

Forbes (2012), Google Glasses Sound As Crazy As Smartphones And Tablets Once Did, 5 April 2012, http://www.forbes.com/sites/greatspeculations/2012/04/05/google-glasses-sound-as-crazy- as-smartphones-and-tablets-once-did/, retrieved April 2012.

Frame, A. and Lussier, J.W. Eds. (2000), 66 Stories of Battle Command, School for Command Preparation, Army Command and General Staff College Fort Leavenworth, 2000.

Future Point (2011), Future Point Systems Inc., Starlight, http://www.futurepointsystems.com/, retrieved January 2011.

Gapminder (2005), Gapminder Human Development, 2005. http://www.gapminder.org/downloads/human-development-trends-2005, retrieved November 2011.

Gartner (2010), Hype Cycle for Human-Computer Interaction, ID Number: G00200994, 4 August 2010.

Gartner (2011), Hype Cycle for Human-Computer Interaction, ID Number: G00215631, 28 July 2011.

GBT (2011), Global Biometrics Technology Market (2010-2015) – Market Forecast by Products, End-User Application and Geography, January 2011, http://www.marketsandmarkets.com/Market-Reports/biometric-market-278.html, retrieved December 2011.

Geere, D. (2011), Senseg’s Haptic Touchscreen Technology Lets you Feel your Apps, Wired.co.uk, 12 December 2011, http://www.wired.co.uk/news/archive/2011-12/12/senseg- preview, retrieved April 2012.

Gershon, N. and Page W. (2001), What Storytelling Can Do for Information Visualization, Communications of the ACM, Vol 44, No. 8, pp 31-37, August 2001. DRDC Valcartier TR 2012-425 95 GestureWorks (2012), Multitouch Hardware, http://gestureworks.com/features/supported- hardware, retrieved March 2012.

Gigaom (2011), Patent Describes OS X Spaces on iPad, Apr. 4, 2011, http://gigaom.com/apple/patent-describes-os-x-spaces-on-ipad/, retrieved April 2012.

Gigaom (2012), Virtual Worlds: Immersive Training, Collaboration and Meetings, 1 June 2011, http://gigaom.com/collaboration/virtual-environments-for-training-collaboration-and-meetings/, retrieved March 2012.

Global Security (2011), Emerging Biometric Technologies, Global Security.org, http://www.globalsecurity.org/security/systems/biometrics-emerging.htm, retrieved December 2011.

Google (2011a), Street View: Explore the World at Street Level, http://maps.google.com/intl/en/help/maps/streetview/#utm_campaign=en&utm_medium=van&ut m_source=en-van-na-us-gns-svn, retrieved December 2011.

Google (2011b), Where is Street View Available, http://maps.google.com/help/maps/streetview/learn/where-is-street-view.html, retrieved December 2011.

Google (2012a), Speaktoit Assistant, https://play.google.com/store/apps/details?id=com.speaktoit.assistant&hl=en, retrieved April 2012.

Google (2012b), Project Glass: One Day..., 4 April 2012, http://www.youtube.com/watch?v=9c6W4CCU9M4, retrieved April 2012.

Gouin, D., Wark, S. and Vernik, R. (2009), Considerations and Guidelines for the Use of Large Group Displays in Command and Control Environments, TTCP C3I TP2 (Command Information Interfaces), December 2009.

Gould, J. and Hoffman, M. (2010), Army sees smart phones playing important role, ArmyTimes, 12 Dec 2010, http://www.armytimes.com/news/2010/12/army-smart-phones-for-soldiers- 121210w/, retrieved April 2012.

GP Bullhound (2010), The Era of Touch Interface is Upon Us, http://gpbullhound.com/en/research/, retrieved November 2011.

Green, M., Warrell, H., Tarling, S., Bernard, S. and Formentini, M. (2009), Afghanistan: Behind the Front Line, Financial Times, 2009, http://www.ft.com/cms/s/0/663b649e-b7e6-11de-8ca9- 00144feab49a.html, retrieved July 2012.

Guardian (2009), Can You Decode These Emotions? The Guardian, 7 March 2009, http://www.guardian.co.uk/lifeandstyle/2009/mar/07/health-and-wellbeing-psychology 1, retrieved January 2012.

Hamblen, M. (2011), “iPads, Android Tablets and Smartphones Join the Military”, Computer World, 16 March 2011, http://www.computerworld.com/s/article/9214624/iPads_Android_tablets_and_smartphones_join _the_military, retrieved November 2011.

96 DRDC Valcartier TR 2012-425 HCII (2011), Carnegie Mellon Develops Touchscreen Technology That Distinguishes Taps by Different Parts of Finger, Carnegie Mellon’s Human-Computer Interaction Institute, 18 October 2011, http://www.hcii.cmu.edu/news/stories/carnegie-mellon-develops-touchscreen-technology- distinguishes-taps-different-parts-fing, retrieved April 2012.

Heise, R. (2012), Microsoft’s ‘Beamatron’ Combines a Projector with a Kinect to Augment Reality, The Verge, 16 March 2012, http://www.theverge.com/2012/3/16/2876238/microsoft- beamatron-kinect-augmented-reality, retrieved March 2012.

Herling, J. and Broll, W. (2010), Advanced Self-contained Object Removal for Realizing Real- time Diminished Reality in Unconstrained Environments, Proceedings of the International Symposium on Mixed and Augmented Reality 2010 (ISMAR' 2010), Seoul, South Korea, October 2010, p. 207-212.

Herman, D. (2000), Narratology as a Cognitive Science, Image (&) Narrative 1, Online Magazine of the Visual Narrative, 2000, http://www.imageandnarrative.be/inarchive/narratology/davidherman.htm, retrieved April 2012.

High Impact PC (2011), High End Display Setups, http://highimpactpc.com/, retrieved January 2011.

Hogg, C. (2010), Immersive Media Releases 360-degree Interactive Video from Haiti, Digital Journal, 25 January 2010, http://digitaljournal.com/article/286410, retrieved November 2011.

Hoover, J.N. (2011), Army Readies Mobile App Store, Information Week, Government, 14 October 2011, http://www.informationweek.com/news/government/mobile/231900854, retrieved November 2011.

Horsey, J. (2010), Toshiba Flexible LCD Technology, Geegy Gadgets, 2 June 2010, http://www.geeky-gadgets.com/toshiba-flexible-lcd-technology-02-06-2010/, retrieved April 2012.

Hughes, D. (2008), DARPA Pursues Neuroscience to Enhance Analyst, Soldier Performance, Aviation Week an dSpace Technology, 28 January 2008, http://www.aviationweek.com/aw/generic/story_channel.jsp?channel=defense&id=news/aw0128 08p1.xml, retrieved October 2011.

Hunt, B. (2008a), DoD Biometrics Task Force - From HQDA to the Soldier and Back in 420 Seconds, Proceedings Biometric Consortium 2008 Conference, 23-25 September 2008, http://biometrics.org/bc2008/profiles/brianhunt.html, retrieved December 2011.

Hunt, B. (2008b), DoD Biometrics Task Force - From HQDA to the Soldier and Back in 420 Seconds, Proceedings Biometric Consortium 2008 Conference, 23-25 Sept 2008, http://biometrics.org/bc2008/presentations/204.pdf, retrieved December 2011.

Immich, T. (2010), Resurrecting User Interface Prototypes (Without Creating Zombies), Smashing Magazine, 17 May 2010, http://uxdesign.smashingmagazine.com/2010/05/17/resurrecting-user-interface-prototypes- without-creating-zombies, retrieved March 2012.

Innovega (2012), Innovega – Visualizing the Digital World, http://innovega-inc.com/how-it- compares.php, retrieved April 2012.

DRDC Valcartier TR 2012-425 97 Iogear (2012), Iogear Mobile Digital Scribe, http://www.iogear.com/product/GPEN200N/, retrieved March 2012.

Jaሷrvinen, P., Puolamaሷki, K., Siltanen, P. and Ylikeraሷla, M. (2009), Visual Analytics Final Report: Data Analytics, Visualization, Applications, Tools, VTT Working Papers 117, VTT Technical Research Centre of Finland, 45pp, 2009, http://www.vtt.fi/inf/pdf/workingpapers/2009/W117.pdf, retrieved January 2011.

Jewell, D. (2009), Kenaz and Touch Inspect: Must-Have Products for the Warfighter, GPS World, 9 December 2009.

Juhnke, J. (2010), Precise Overlay Registration within Augmented Reality – A Glimpse into the Technology, May 24, 2010, http://spill.tanagram.com/2010/05/24/precise-overlay-registration- within-augmented-reality-a-glimpse-into-the-technology/, retrieved April 2012.

Kalal, Z., Matas, J. and Mikolajczyk, K. (2011), Predator: Camera that Learns, 1 January 2011, http://www.youtube.com/watch?v=1GhNXHCQGsM&lr=1, retrieved April 2012.

Kalal, Z. (2012), Dr. Zenek Kalal, , http://personal.ee.surrey.ac.uk/Personal/Z.Kalal/, retrieved April 2012.

Karray, F., Alemzadeh, M., Abou Saleh, J. and Arab, M.N. (2008), Human-Computer Interaction: Overview on State of the Art, International Journal of Smart Sensing and Intelligent Systems, Vol 1, No 1, March 2008.

Kim, D.-H., Lu, N., and Ma, R. (2011), Epidermal Electronics, Science Magazine, 12 August 201, Vol. 333, no. 6044, pp. 838-843.

Krämer, N.C. Pütten, A.v.d. and Eimler, S. (2012), Human-Agent and Human-Robot Interaction Theory: Similarities to and Differences from Human-Human Interaction , Human-Computer Interaction: The Agency Perspective, Studies in Computational Intelligence, 2012, Volume 396/2012, p. 215-240.

Kruse, A.A., Boyd, K.C., and Schulman, J.J. (2006), Neurotechnology for Intelligence Analysts, Chemical and Biological Sensing VII, Proceedings SPIE Vol. 6218, 621819.

Kumar, M. (2007), Gaze-enhanced User Interface Design, Standford University, May 2007.

LED Alive Show (2012), Transparent Flexible LED Display Brief Introduction, http://www.ledaliveshow.com/learning-center/brief-introduction-transparent-flexible-led- display.html, retrieved April 2012.

LED Marketing (2012), Flexible LED Screen / Soft LED Display, 24 February 2012, http://www.youtube.com/user/ledmarketing, retrieved April 2012.

Li, D. (2006), Low-Cost Eye-Tracking for Human-Computer Interaction, Iowa State University, http://thirtysixthspan.com/openEyes/MS-Dongheng-Li-2006.pdf, retrieved March 2012.

Linder, N (2010), LunimAR, Fluid Interfaces Group, MIT Media Lab, http://fluid.media.mit.edu/people/natan/current/luminar.html, retrieved April 2012.

Livescribe (2011), Pulse Smart Pen, http://www.livescribe.com/index.html, retrieved November 2011.

98 DRDC Valcartier TR 2012-425 Livet, J., Weissman, T., Kang, H., Draft, R., Lu, J., Bennis, R., Sanes, J., Lichtman, J. (2007), Transgenic Strategies for Combinatorial Expression of Fluorescent Proteins in the Nervous System, Nature, 450 (7166), pp. 56–62.

Lum R. (2012), Will Google’s Project Glass’ be a Huge Hit for Marketers? Creative Guerilla Marketing, 5 April 2012, http://www.creativeguerrillamarketing.com/advertising/will-googles- project-glass-be-a-huge-hit-for-marketers, retrieved April 2012.

Mackinlay, J.D., Hanrahan, P. and Stolte, C. (2007), Show Me: Automatic Presentation for Visual Analysis, IEEE Transactions on Visualization and Computer Graphics, Vol. 13, No. 6, November 2007.

Magicomm (2011), Magicomm G303 Digital Pen, http://www.magicomm.co.uk/products/g303- digital-pen, retrieved November 2011.

Marreiros, G., Santos, R., Freitas, C., Ramos, C., Neves, J. and Bulas-Cruz, J. (2008), LAID - a Smart Decision Room with Ambient Intelligence for Group Decision Making and Argumentation Support considering Emotional Aspects, International Journal of Smart Home, Vol. 2, No. 2, April 2008.

Maybury M.T. and Wahlster, W. (Eds.) (1998), Readings in Intelligent User Interfaces, Morgan Kaufman Publisher, San Francisco, CA, 1998.

McGlaun, S. (2011), Sony Shows Off Flexible Color E-paper Screen at SID 2011, Slash Gear, 19 May 2011, http://www.slashgear.com/sony-shows-off-flexible-color-e-paper-screen-at-sid-2011- 19153068/, retrieved April 2012.

Media Roundtable (2010), Top Five Apps for the Army Winners Recognized at LandWarNet Conference, 4 August 2010, http://architecture.army.mil/images/stories/A4A_Winners_28_July_Media_Advisory.pdf, retrieved November 2011.

Mendez, E., Kalkofen, D. and Schmalstieg, D. (2006), Interactive Context-Driven Visualization Tools for Augmented Reality, Proceedings ISMAR 2006, October 22-25, 2006, Santa Barbara, California, USA, 2006, pp. 209-216.

Menon, V., Jayaraman, B. and Govindaraju, V. (2010), Multimodal Identification in Smart Environments’, in Personal Ubiquitous Computing, 14:685-694, Springer-Verlag, London, UK, December 2010.

Metaio (2011), By 2014: Augmented Reality will be on Every Smartphone, http://www.metaio.com/press/press-release/2011/by-2014-augmented-reality-will-be-on-every- smartphone/, retrieved April 2012.

Microsoft Research (2012a), Applied Sciences Group: Interactive Displays: Behind the Screen Overlay Interactions, 27 February 2012, http://www.youtube.com/watch?v=oGa1Q7NvsI0, retrieved March 2012.

Microsoft Research (2012b), Beamatron, 13 March 2012, http://www.youtube.com/watch?v=L9yccRm3Zu8&feature=player_embedded, retrieved March 2012.

DRDC Valcartier TR 2012-425 99 Microvision (2011), Pico Projector, http://www.microvision.com/pico_projector_displays/application_gallery.html, retrieved January 2011.

Milgram P. and Kishino, A. F. (1994), Taxonomy of Mixed Reality Visual Displays IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994

Mimi (2009), Table with Microsoft Touch-Sensitive Screen Surfaces, http://mimi19851226.wordpress.com/2009/03, retrieved November 2011.

Mistry, P. (2010), SixthSense, Fluid Interfaces Group, MIT Media Lab, http://fluid.media.mit.edu/people/pranav/current/sixthsense.html, retrieved April 2012.

Mitchell, R. (2011), Flexible displays: What's the Holdup? Computer World, 29 March 2011, http://www.computerworld.com/s/article/9215053/Flexible_displays_What_s_the_holdup_?taxon omyId=128&pageNumber=2, retrieved May 2012.

Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M.-A., Morito, Y., Tanabe, H.C., Sadato, N. and Kamitani, Y. (2008), Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image Decoders, Neuron , 60 (5), pp. 915–929.

Mobile (2010), US Army Smartphone War Draws Closer, Mobile.Blorge, 14 Dec 2010, http://mobile.blorge.com/2010/12/14/us-army-smartphone-war-draws-closer, retrieved January 2011.

MobileEpiphany (2009), Mobile Epiphany Launches First-of-its-Kind Geospatially Intelligent Inspection Application, December 2009, http://www.mobileepiphany.com/launch, retrieved November 2011.

Moor, C. (2009), Talk Android, Layar – Mobile Augmented Reality Browser App, 16 June 2009, http://www.talkandroid.com/1264-layar-mobile-augmented-reality-app, retrieved January 2011.

Mori, M. (1970), Bukimi no tami (The Uncanny Valley), Energy, vol. 7, 1970, pp. 33-35.

MSDN Blogs (2012), Starting February 1, 2012: Use the Power of Kinect for Windows to Change the World, 9 January 2012, http://blogs.msdn.com/b/kinectforwindows/archive/2012/01/09/kinect-for-windows-commercial- program-announced.aspx, retrieved March 2012.

MultiTouch (2010), MultiTouch Twitter Wall AMS, 20 October 2010, http://www.youtube.com/watch?v=rZxCuwwbbRc, retrieved December 2011.

MultiTouch (2011), MultiTaction White Paper v1.1, 26 April 26 2011, http://www.multitaction.com/why-multitaction/, retrieved November 2011.

Naone, E. (2009), TR10:Intelligent Software Assistant, Technology Review, MIT, March/April 2009, http://www.technologyreview.com/read_article.aspx?ch=specialsections&sc=tr10&id=22117, retrieved March 2012.

Natarajan (2011), Getting Siri-ous Abour Ontologies, 23 October 2011, http://courses.ischool.berkeley.edu/i202/f11/node/554, retrieved April 2012.

NeuroSky (2011), http://www.neurosky.com, retrieved October 2011. 100 DRDC Valcartier TR 2012-425 Newitz, A. (2010), TR10: Mobile 3-D, Smart Phones will Take 3-D Mainstream, Technology Review, May/June 2010, http://www.technologyreview.com/communications/25081/, retrieved April 2012.

Next Big Future (2012), DARPA Working with Innovega for Virtual Reality Contact Lens and Eyeware Combination, January 31, 2012, Next http://nextbigfuture.com/2012/01/darpa-working- with-innovega-for-virtual.html, retrieved March 2012.

Nichols, N. and Hammond, K. (2009), Machine-Generated Multimedia Content, Proceedings of the Second International Conference on Advances in Computer-Human Interactions, 2009.

NICTA (2011), Braccetto, http://www.nicta.com.au/research/project_list/completed_projects/braccetto, retrieved January 2011.

Nilson, T.E. (2011), Use Your Eyes – A Tobii Technology Introduction and Presentation – and How Tobii Eye Tracking Could be used in Advertising, http://www.slideshare.net/AcuityETS/tobii-eye-tracking, retrieved November 2011.

Nintendo (2006), Nintendo Wii Remotes, http://www.nintendowiiremotes.com/, retrieved March 2012.

Nosowitz, D. (2010), Video: MIT's Kinect Hack Tracks All Ten Fingers Simultaneously, POPSCI, 9 Dec 2010, http://www.popsci.com/gadgets/article/2010-12/video-mits-finger- tracking-kinect-hack-amazing-insert-minority-report-reference-here, retrieved January 2011.

Nowak, P. (2007), Why Man Melding with Machine is a Good Thing, CBC News, 23 Nov 2007, http://www.cbc.ca/news/background/tech/cellphones/mann.html, retrieved January 2011.

Nozawa, T. (2012), TRADIM Develops Flexible LCD Display by Using Roll-to-roll Method, Tech On, 29 March 2012, http://techon.nikkeibp.co.jp/english/NEWS_EN/20120329/210375/, retrieved May 2012. nsquared (2011), nsquared Seamless Computing, August 28, 2011, http://nsquaredsolutions.com/news/, retrieved October 2011.

Oblong (2011), The G-Speak Platform, http://www.oblong.com, retrieved January 2011.

OLED Info (2012), OLED Introduction and Basic OLED Information, http://www.oled- info.com/introduction, retrieved April 2012.

Onislam (2009), The Future of Virtual Worlds – At the Crossroads between Real and Sci-Fi, Mohammed Yahia, 21 July 2009, http://www.onislam.net/english/health-and- science/technology/441006.html, retrieved March 2012.

O’Neill, S. (2008), VFPA Biometric Implementation Hand Vascular Scanners -A Case Study, Proceedings Biometric Consortium 2008 Conference, 23-25 Sept 2008, http://biometrics.org/bc2008/presentations/210.pdf, retrieved December 2011.

OverWatch (2012), SoldierEyes, http://www.overwatch.com/products/soldiereyes.php, retrieved April 2012.

DRDC Valcartier TR 2012-425 101 Oviatt, S. (2003), Multimodal Interfaces, in J.A. Jacko and A. Sears (eds), The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Application, Lawrence Erlbaum Associates, Mahwah, 2003.

Park Service (2011), Semi-Transparent HMD, http://parkservice.linus.oxi.net/indigofolder/visualcues/HMD/image_view_fullscreen, retrieved January 2011.

PARVAC (2012), Just-In-Time Command and Control Center (JITC3), Pacific Rim Visualization and Analytics Center, University of Washington, http://parvac.washington.edu/projects/view.php?shortname=jitc3, retrieved April 2012.

Pennock, J. and Tabrizi, M.H.N. (2008), A Survey of Input Sensing and Processing Techniques for Multi-Touch Systems, Proceedings IPCV08, Joensuu, Finland, 18-29 August 2008.

Phillips, M. (2008), Life Spaces Technical Overview, Defence Science and Technology Organisation, DSTO-TR-2188, October 2008.

Pink Tentacle (2007), Flexible, Full-color OLED Display, 24 May 2007, http://pinktentacle.com/2007/05/flexible-full-color-organic-el-display/, retrieved January 2011.

Poussart, D. (2011), Future Intelligence Analysis Capability White Paper: Towards a Cohesive R&D Program Definition, Defence R&D Canada – Valcartier, Technical Memorandum, 2011, to be published.

Pratt, T. (2008), Biometrics in the CENTCOM AOR, US CENTCOM, Proceedings Biometric Consortium 2008 Conference, 23-25 Sept 2008, http://biometrics.org/bc2008/presentations/201.pdf, retrieved December 2011.

PSE (2012), IARPA A-SpaceX Concept and Prototype, Pacific Science & Engineering Group, Inc., http://www.pacific-science.com/content/hsi/aspace.html, retrieved March 2012.

Purver, M., Niekrasz, J. and Peters, S. (2005), Ontology-Based Multi-Party Meeting Understanding, Proceedings CHI’05, Portland, OR, April 2005.

Rao, L. (2011), comScore: 38 Percent of Smartphone Owners Have Used a Mobile Device to Make a Purchase, 5 December 2011, http://techcrunch.com/2011/12/05/comscore-38-percent-of- smartphone-owners-have-used-a-mobile-device-to-make-a-purchase/, retrieved April 2012.

Reeves, B. (2004), The Benefits of Interactive Online Characters, Stanford University, http://noahx.com/resources/CSLI_Stanford_Study.pdf, retrieved April 2012.

Rekimoto, J. and Nagao, K. (1995), The World through the Computer: Computer Augmented Interaction with Real World Environments, Symposium on User Interface Software and Technology (UIST'95) ACM Press, November 1995.

Ribarsky, B. (2009), Multimedia Visual Analytics, VAC Views, May 09, p. 10-11.

Riva, G., Vatalaro, F., Davide, F. and Alaniz, M. (2005), Ambient Intelligence: The Evolution of Technology, Communication and Cognition towards the Future of HCI, IOS Press, Fairfax.

Robertson, G., Ebert, D., Eick, S. Keim, D and Joy, K. (2009), Scale and Complexity in Visual Analytics, Information Visualization, Vol. 8 4, P. 247-253, 4 December 2009.

102 DRDC Valcartier TR 2012-425 Schmidt, D., Chehimi, F., Rukzio, E. and Gellersen, H., (2010b), PhoneTouch: A Technique for Direct Phone Interaction on Surfaces, Proceedings UIST 2010, pp. 13-16.

Schmidt, D., Chong, M.K. and Hans, G. (2010a), HandsDown: Hand-Contour-Based User Identification for Interactive Surfaces, Proceedings NordiCHI 2010, 432-441.

Scott, S., Allavena, A., Cerar, K., Franck, G., Hazen, M., Shuter, T. and Colliver, C. (2010), Investigating Tabletop Interfaces to Support Collaborative Decision-making in Maritime Operations, Proceedings International Command and Control Research and Technology Symposium (ICCRTS), Santa Monica, California, 22 – 24 June 2010.

Segel E. and Heer, J. (2010), Narrative Visualization: Telling Stories with Data, IEEE Trans. on Visualization and Computer Graphics, Vol. 16, No. 6, November – December 2010.

Serenko, A., Bontis, N. And Detlor, B. (2007), End-user Adoption of Animated Interface Agents in Everyday Work Applications, Behaviour & Information Technology, Vol. 26, No. 2, March - April 2007, 119 – 132.

ShapeHand (2009), Shape Hand, http://www.shapehand.com/, retrieved December 2009.

Sitscape (2012), http://www.sitscape.com, retrieved April 2012.

Smith, N. (2006), Organized for a Living - What Electronic Tablets Can Do For You, March 2006, http://www.onlineorganizing.com/NewslettersArticle.asp?newsletter=ol&article=621, retrieved November 2011.

Stereoscopynews (2012), 3D Contact Lenses ? Only if you are US soldiers..., 4 February 2012, http://www.stereoscopynews.com/hotnews/3d-technology/various-a-strange-facts/2175-3d- contact-lenses-only-if-you-are-us-soldiers.html, retrieved March 2012.

Surveylab (2011), ike304 – A Rapid Data Capture Device for Mobile Mapping Geospatial Intelligence, http://www.fs.fed.us/t- d/programs/im/geospatial_camera/pdf_files/Military_Brochure_mobile_mapping.pdf, retrieved October 2011.

Sutherland, M. (2011), Using Our Brains: Neuromarketing, No-hands Gaming and the Arrival of the EEG Headset, Gizmag, 23 May 2011, http://www.gizmag.com/eeg-headsets-gaming- neuromarketing/18693, retrieved October 2011.

Swan, L. (2008), DoD Biometrics Science and Technology, Proceedings Biometric Consortium 2008 Conference, 23-25 Sept 2008, http://biometrics.org/bc2008/presentations/167.pdf, retrieved December 2011.

Tactus (2012), Tactus Technology, http://www.tactustechnology.com/, retrieved April 2012.

Tate, A., Chen-Burger, Y-H., Dalton, J., Potter, S., Richardson, D., Stader, J., Wickler, G., Bankier, I., Walton, C. and Williams, P.G. (2010), I-Room: A Virtual Space for Intelligent Interaction, IEEE Intelligent Systems, Vol. 25, No. 4, pp 62-71, July-August 2010, IEEE Computer Society.

Tate, A. (2011), I-Room: a Virtual Space for Intelligent Interaction, 28 February 2011, http://www.aiai.ed.ac.uk/~ai/resources/2011-02-28-CISA/2011-02-28-cisa-iroom.pdf, retrieved April 2012.

DRDC Valcartier TR 2012-425 103 Tatnell (2010), 1w65 Research Paper - How Can the Division of the Screen Enhance our Experience of Dimensionality? 1 December 2010, http://davidtatnell.wordpress.com/category/research-paper, retrieved April 2012.

Technovelgy (2011), Biometric Technology on the Leading Edge, http://www.technovelgy.com/ct/Technology-Article.asp?ArtNum=13, retrieved December 2011.

Terdiman, D. (2009), Inside the Navy Command Center of the Future, 29 September 2009, http://www.eugeneleeslover.com/VIDEOS/Command_center_future.html, retrieved January 2011.

Thomas, J.J. and Cook, K.A., eds. (2005), Illuminating the Path: The Research and Development Agenda for Visual Analytics, IEEE Computer Society Press, http://nvac.pnl.gov/agenda.stm#book, retrieved January 2011.

Tkoranyi (2011), Touching Moment: Palm to Palm Again by using Bionic Arm, Neurogadget, 11 October 2011, http://neurogadget.com/2011/10/11/bionic-arm-to-use-fingers-like-real-ones/2783, retrieved October 2011.

Trimble Navigation (2011), Mobile GIS Solution Streamlines Data Collection for Civil Affairs Efforts in Afghanistan, in Innovative GIS Technology in the Defense and Intelligence Communities, vol 1, pp 72-74, retrieved November 2011.

Tsotsis, A. (2012), Google’s Plan to Compete with Apple’s Multi-Platform Siri? Google ‘Assistant’, TechCrunch, 2 March 2012, http://techcrunch.com/2012/03/02/2011-was-the-year-of- social-for-google-2012-is-the-year-of-assistant/, retrieved March 2012.

Tur, G., Stolcke, A., Voss, L., Dowding, J., Favre, B., Fernandez, R., Frampton, M., Frandsen, M., Frederickson, C., Gracairena, M., Hakkani-Tür, D., Kintzing, D., Leveque, K., Mason, S., Niekrasz, J., Peters, S., Purver, M., Riedhammer, K., Shriberg, E., Tien, J., Vergyri, D. and Yang, F. (2008), The CALO Meeting Speech Recognition and Understanding System.

T2 Health (2012), T2 Virtual PTSD Experience, National Center for Telehealth and Technology, http://www.t2health.org/vwproj, retrieved March 2012.

US Army (2011), Architecture Community, CIO/G-6, http://architecture.army.mil/technical- view/applications/applications.html, retrieved November 2011.

US Army (2012), Army Launches Apps Marketplace Prototype, 23 March 2012, http://www.army.mil/article/75966/, retrieved April 2012.

US DoD (2011), Explore DoD’s Virtual Worlds, US Department of Defense, 9 May 2011, http://www.defense.gov/home/features/2011/0511_virtualworlds, retrieved March 2012.

US DoD (2012), ‘Virtual World’ Helps with Post-Traumatic Stress, 19 Jan 2011, http://www.defense.gov/news/newsarticle.aspx?id=62501, retrieved March 2012.

VRealities (2009), P5 Glove, http://www.vrealities.com/P5.html, retrieved December 2009.

Vochin, A. (2011), NeuroFocus Mynd is World’s First Wireless Full-Brain EEG Measurement Headset, Softpedia, 21 March 2011, http://news.softpedia.com/news/NeuroFocus-Mynd-is- World-s-First-Wireless-Full-Brain-EEG-Measurement-Headset-190641.shtml, retrieved October 2011.

104 DRDC Valcartier TR 2012-425 Wark, S., Broughton, M., Nowina-Krowicki, M., Zschorn, A., Coleman, M., Taplin, P. and Estival, D. (2005), The FOCAL Point - Multimodal Dialogue with Virtual Geospatial Displays, Proceedings SimTecT 2005, Sydney, AU, 9-12 May 2005.

Wark S. and Lambert, D. (2007), Presenting the Story Behind the Data - Enhancing Situational Awareness, Proceedings MILCOM 2007, Orlando, Fl, 29-31 October 2007.

WDV (2012), 3D Printing: ZPrinter 650, http://www.wdv- gmbh.de/english/products/ebzcorpZ650.shtml, retrieved April 2012.

Weinschenk, S. (2010), Your Mind Wanders 30% of the Time, 100 Things You Should Know About People: #39, August 3, 2010, http://www.whatmakesthemclick.net/2010/08/03/100-things- you-should-know-about-people-39-your-mind-wanders-30-of-the-time/, retrieved September 2011.

Weiss, M., Wagner, J., Jennings, R., Jansen, Y., Khoshabeh, R., Hollan, J.D. and Borchers, J. (2009), SLAPbook: Tangible Widgets on Multi-touch Tables in Groupware Environments. In TEI '09: Proceedings of the 3rd international conference on Tangible and embedded interaction, pages 297-300, New York, NY, USA, 2009, ACM.

Wellner, P,. Mackay, W. and Gold, R. (1993), Computer Augmented Environments: Back to the Real World. Communication of the ACM, Vol. 36, No. 7, August 1993.

Wikipedia (2009a), Multimodal Interaction, http://en.wikipedia.org/wiki/Multimodal_interaction, retrieved November 2009.

Wikipedia (2009b), Wired Glove, http://en.wikipedia.org/wiki/Wired_glove, retrieved November 2009.

Wikipedia (2009c), Eye Tracking, http://en.wikipedia.org/wiki/Eye_tracking, retrieved November 2011.

Wikipedia (2010), Human-Computer Interaction, http://en.wikipedia.org/wiki/Human%E2%80%93computer_interaction, retrieved November 2010.

Wikipedia (2011a), Digital Pen, http://en.wikipedia.org/wiki/Digital_pen, retrieved January 2011.

Wikipedia (2011b), Speech Recognition, http://en.wikipedia.org/wiki/Speech_recognition, retrieved January 2011.

Wikipedia (2011c), Haptic Technology, en.wikipedia.org/wiki/Haptic_technology, retrieved January 2011

Wikipedia (2011d), Virtual World, http://en.wikipedia.org/wiki/Virtual_world, retrieved December 2011.

Wikipedia (2011e), GUI Widget, http://en.wikipedia.org/wiki/GUI_widget, retrieved January 2011.

Wilson, A.D. and Benko, H. (2010), Combining Multiple Depth Cameras and Projectors for Interactions on, Above and Between Surfaces, in Proceedings of the 23nd annual ACM symposium on User interface software and technology (UIST’10), ACM, Ney-York.

DRDC Valcartier TR 2012-425 105 Wojtkowski, W. and Wojtkowski, W.G. (2002), Storytelling: its Role in Information Visualization, in Proc. ESSC, October 2002.

Zcorp (2012), 3D Systems, http://www.zcorp.com/en/Products/3D-Printers/ZPrinter- 850/spage.aspx, retrieved April 2012.

Zelkha, E., Epstein, B., Birrell, S. and Dodsworth, C. (1998), From Devices to ‘Ambient Intelligence’, Digital Living Room Conference, June 1998.

Zeng, Z., Pantic, M., Roisman, G.I. and Huang, T.S. (2009), A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 31, No. 1, January 2009.

Zhang, B., Wang, J. and Fuhlbrigge, T. (2010), A Review of the Commercial Brain-Computer Interface Technology from Perspective of Industrial Robotics, 2010 IEEE International Conference on Automation and Logistics (ICAL), 16-20 August 2010, pp. 379-384.

Zonkio (2009), Augmented Reality Technology and Communication, 10 July 2009, http://www.zonkio.com/augmented-reality-technology-and-communication_1700.html, retrieved January 2011.

106 DRDC Valcartier TR 2012-425 List of symbols/abbreviations/acronyms/initialisms

AFRL Air Force Research Laboratory AmI Ambient Intelligence API Application Programming Interface AR Augmented Reality AugCog Augmentation Cognition BCI Brain Computer Interface CAD Computer-Aided Design CALO Cognitive Assistant that Learns and Organizes CAVE Cave Automatic Virtual Environment CCoF Command Center of the Future CFAR Capacités futures d’analyse du renseignement CISTI Canada Institute for Scientific and Technical Information CF Canadian Forces CMU Carnegie Mellon University DARPA Defense Advanced Research Projects Agency DISA Defense Information Systems Agency DND Department of National Defence DoD Department of Defense DRDC Defence Research & Development Canada DSTO Defence Science & Technology Organization EEG Electroencephalography FIAC Future Intelligence Analysis Capabilities fMRI Functional Magnetic Resonance Imaging FOCAL Future Operations Centre Analysis Laboratory HCI Human-Computer Interaction HMD Head-Mounted Displays GIS Geographic Information System GPS Global Positioning System GUI Graphical User Interface HCI Human-Computer Interaction HD High Definition HMD Head-Mounted Display IARPA Intelligence Advanced Research Projects Activity ID Identification DRDC Valcartier TR 2012-425 107 IED Improvised Explosive Device IHM Interaction humain-machine ISA Intelligent Software Assistant JIMP Joint, Interagency, Multinational and Public LCD Liquid Crystal Display LED Light Emitting Diode LGD Large Group Display MR Mixed Reality NIA Neurotechnology for Intelligence Analysts NICTA National Information and Communications Technology Australia OCR Optical Character Recognition OLED Organic Light Emitting Diode PAL Personalized Assistant that Learns PDA Personal Device Assistant PIE Precision Information Environment PIN Personal Identification Number PTSD Post-Traumatic Stress Disorder RDDC Recherche et développement pour la défense, Canada RFID Radio-Frequency Identification R&D Research & Development SLAP Silicone iLluminated Active Peripherals SOP Standard Operating Procedure TLD Tracking, Learning, Detection TRADIM Technology Research Association for Advanced Display Materials UAV Unmanned Aerial Vehicle UDOP User-Defined Operating Pictures USB Universal Serial Bus VIF Virtual Immersive Facility VR Virtual Reality VW Virtual World

108 DRDC Valcartier TR 2012-425 DOCUMENT CONTROL DATA (Security markings for the title, abstract and indexing annotation must be entered when the document is Classified or Designated) 1. ORIGINATOR (The name and address of the organization preparing the document. 2a. SECURITY MARKING Organizations for whom the document was prepared, e.g. Centre sponsoring a (Overall security marking of the document including contractor's report, or tasking agency, are entered in section 8.) special supplemental markings if applicable.) Defence Research and Development Canada – Valcartier UNCLASSIFIED 2459 Pie-XI Blvd North Quebec (Quebec) G3J 1X5 Canada 2b. CONTROLLED GOODS (NON-CONTROLLED GOODS) DMC A REVIEW: GCEC JUNE 2010

3. TITLE (The complete document title as indicated on the title page. Its classification should be indicated by the appropriate abbreviation (S, C or U) in parentheses after the title.) Assessment of Future Trends in Human-Computer Interaction for Military Intelligence

4. AUTHORS (last name, followed by initials – ranks, titles, etc. not to be used) Gouin, D.; Lavigne, V.

5. DATE OF PUBLICATION 6a. NO. OF PAGES 6b. NO. OF REFS (Month and year of publication of document.) (Total containing information, (Total cited in document.) including Annexes, Appendices, etc.) December 2012 134 

7. DESCRIPTIVE NOTES (The category of the document, e.g. technical report, technical note or memorandum. If appropriate, enter the type of report, e.g. interim, progress, summary, annual or final. Give the inclusive dates when a specific reporting period is covered.) Technical Report

8. SPONSORING ACTIVITY (The name of the department project office or laboratory sponsoring the research and development – include address.) Defence Research and Development Canada – Valcartier 2459 Pie-XI Blvd North Quebec (Quebec) G3J 1X5 Canada

9a. PROJECT OR GRANT NO. (If appropriate, the applicable research 9b. CONTRACT NO. (If appropriate, the applicable number under and development project or grant number under which the document which the document was written.) was written. Please specify whether project or grant.)

10a. ORIGINATOR'S DOCUMENT NUMBER (The official document 10b. OTHER DOCUMENT NO(s). (Any other numbers which may be number by which the document is identified by the originating assigned this document either by the originator or by the sponsor.) activity. This number must be unique to this document.) DRDC Valcartier TR 2012-425

11. DOCUMENT AVAILABILITY (Any limitations on further dissemination of the document, other than those imposed by security classification.) Unlimited

12. DOCUMENT ANNOUNCEMENT (Any limitation to the bibliographic announcement of this document. This will normally correspond to the Document Availability (11). However, where further distribution (beyond the audience specified in (11) is possible, a wider announcement audience may be selected.)) Unlimited 13. ABSTRACT (A brief and factual summary of the document. It may also appear elsewhere in the body of the document itself. It is highly desirable that the abstract of classified documents be unclassified. Each paragraph of the abstract shall begin with an indication of the security classification of the information in the paragraph (unless the document itself is unclassified) represented as (S), (C), (R), or (U). It is not necessary to include here abstracts in both official languages unless the text is bilingual.) Collective command and control civil-military operations impose a strong requirement for organizations and people to work collaboratively and to interact with information more efficiently. Human-Computer Interaction (HCI) technology has been evolving at a great pace during the last decade, and many novel HCI concepts and tools have become of interest to support not only analysts deployed in command centers but also warfighters deployed in the field. This will have an impact on how intelligence officers will operate in the future. In this regard, Defence R&D Canada has undertaken a major initiative to look forward into Future Intelligence Analysis Capabilities (FIAC). This study reports on HCI trends and applicability to support the FIAC and collective command and control in general. Topics of interest include ubiquitous computing, multimodal interaction, multi-touch tables/surfaces, large group displays, flexible displays, personal device assistants / smart phones, intelligent software assistants, biometry, brain/neural interfaces, collaboration tools / telepresence, smart room environments, head-mounted displays, mixed reality, and advanced HCI concepts and techniques.

------

Les opérations conjointes civiles-militaires de commandement et contrôle imposent aux organisations et au personnel de collaborer et d’interagir avec l’information de façon plus efficace. Les technologies d’interaction humain-machine (IHM) ont évolué à un rythme soutenu au cours de la dernière décennie et plusieurs nouveaux concepts et outils d’IHM sont devenus d’intérêt pour supporter non seulement les analystes de centres de commandement mais aussi les combattants déployés sur le champ de bataille. Ceci aura un impact sur la façon dont les officiers du renseignement vont opérer dans le futur. À cet effet, Recherche et développement pour la défense Canada a initié un projet dans le but d’explorer les capacités futures d’analyse du renseignement (CFAR). Ce mémorandum technique fournit un compte-rendu sur les tendances en IHM et leur application potentielle en support au CFAR et au commandement et contrôle conjoint en général. Les sujets d’intérêt incluent l’informatique omniprésente, les interactions multimodales, les tables ou surfaces multi-tactiles, les écrans de groupe de grande taille, les écrans flexibles, les assistants personnels / téléphones intelligents, les assistants logiciels intelligents, la biométrie, les interfaces neuronales, les outils de collaboration, les environnements intelligents, les casques de réalité virtuelle, la réalité mixte, et les concepts et les techniques avancés d’IHM.

14. KEYWORDS, DESCRIPTORS or IDENTIFIERS (Technically meaningful terms or short phrases that characterize a document and could be helpful in cataloguing the document. They should be selected so that no security classification is required. Identifiers, such as equipment model designation, trade name, military project code name, geographic location may also be included. If possible keywords should be selected from a published thesaurus, e.g. Thesaurus of Engineering and Scientific Terms (TEST) and that thesaurus identified. If it is not possible to select indexing terms which are Unclassified, the classification of each should be indicated as with the title.) human-computer interaction; military intelligence; future intelligence analysis capabilities; multimodal interaction; natural interaction; smart room environments; virtual assistants; storytelling; mixed reality; biometry; virtual assistants

Defence R&D Canada R & D pour la défense Canada

Canada's Leader in Defence Chef de file au Canada en matière and National Security de science et de technologie pour Science and Technology la défense et la sécurité nationale

www.drdc-rddc.gc.ca