Validation of the Comprehensive Approach Toolkit Mobile Application in a Lab-Based Humanitarian Exercise

CAN UNCLASSIFIED

Joshua A. Granek Tara Holton Tonya Hendriks William Kozey Chris Apostoli Ryan Bae DRDC – Toronto Research Centre

The body of this CAN UNCLASSIFIED document does not contain the required security banners according to DND security standards. However, it must be treated as CAN UNCLASSIFIED and protected appropriately based on the terms and conditions specified on the covering page.

Defence Research and Development Canada Scientific Report DRDC-RDDC-2019-R217 April 2020

CAN UNCLASSIFIED

IMPORTANT INFORMATIVE STATEMENTS

This document was reviewed for Controlled Goods by Defence Research and Development Canada (DRDC) using the Schedule to the Defence Production Act.

Disclaimer: This publication was prepared by Defence Research and Development Canada an agency of the Department of National Defence. The information contained in this publication has been derived and determined through best practice and adherence to the highest standards of responsible conduct of scientific research. This information is intended for the use of the Department of National Defence, the Canadian Armed Forces (“Canada”) and Public Safety partners and, as permitted, may be shared with academia, industry, Canada’s allies, and the public (“Third Parties”). Any use by, or any reliance on or decisions made based on this publication by Third Parties, are done at their own risk and responsibility. Canada does not assume any liability for any damages or losses which may arise from any use of, or reliance on, the publication.

The data collected as part of this study was approved either by Defence Research and Development Canada’s Human Research Ethics Board or by the Director General Military Personnel Research & Analysis’ Social Science Research Review Board.

Endorsement statement: This publication has been peer-reviewed and published by the Editorial Office of Defence Research and Development Canada, an agency of the Department of National Defence of Canada. Inquiries can be sent to: [email protected].

Template in use: EO Publishing App for SR-RD-EC Eng 2018-12-19_v1 (new disclaimer).dotm

CAN UNCLASSIFIED

Abstract

The purpose of this study was to gain information pertaining to the CAF IA information operation process by observing IA Analysts use the CAT app during a scenario exercise that was held in the Command and Control Livespace Laboratory (C2L2) at DRDC – Toronto Research Centre—a collaborative workspace equipped with networked computers, collaborative software, and video teleconferencing (VTC) capability. Observational data, information operation output and focus group data were used to: a) identify strategies and resources analysts use to prepare an information operation plan and gather information required by the commander, b) determine how analysts collaborate and create shared situational understanding, c) determine how IA Analysts use the CAT app, and d) validate the CAT app for its usefulness and usability in the conduct of IA information operations.

The study ran over the course of one and a half days for a group of three IA Analysts; the first day included a half day of training on the CAT app while the second day included a full day disaster relief exercise. Specifically, IA Analysts used the CAT app during a disaster relief scenario in the C2L2 to conduct their activities. Following the exercise, the participants generated a final report. Data was gathered in the form of observational data, questionnaires regarding app use, tracking of app usage (e.g., number of clicks on a page, time spent on a page), and written and/or oral feedback at the end of the study. Qualitative data analysis was conducted to assess the extent to which the participants found the CAT app useful when completing their tasks.

Results indicated that two out of the three IA Analysts regularly interacted with the CAT app during the humanitarian exercise with relative ease after minimal training. The CAT app was found to be user-friendly and was well suited for generating mission reports and preparing for face-to-face interactions. Participants found the tool less relevant as a knowledge transfer tool in a headquarters setting with access to other tools such as a personal computer. Further, participants suggested that additional information, access to additional links (e.g., maps), as well as providing additional templates for interacting with key leaders and preparing planning processes would render the CAT app as an even more useful tool. Future research should be done to compare the CAT app relative to a control in order to validate and quantify the magnitude of the added value of adopting the use of the CAT app as part of doctrine for both training and in the theatre of operations.

DRDC-RDDC-2019-R217 i

Significance to defence and security

Benefits of this research included an opportunity for CAF participants to evaluate new software tools and allowed them to become familiarized with the C2L2 as a facility that is can be made available to them to conduct their exercises. This exercise provided valuable data during a realistic scenario that informed DRDC investigators how they can further develop processes or tools to facilitate the search and analysis done by teams of CAF IA Analysts. Further, the study allowed for observation of an IA team’s use of a prototype tool “in-situ,” which will enable the refinement of the tool and increase the likelihood that it will be used effectively and increase efficiency in a theatre of operations.

ii DRDC-RDDC-2019-R217

Résumé

La présente étude avait pour objectif de recueillir des données sur le processus lié aux opérations d’information du domaine des AI au sein des FAC. À cette fin, des chercheurs du Centre de recherche Toronto ont observé des analystes du domaine des AI se servir de l’application TAG au cours d’une simulation réalisée dans le Laboratoire de commandement et de contrôle LiveSpace (C2L2), un espace de travail collaboratif où l’on retrouve un système de vidéoconférence et des ordinateurs en réseau sur lesquels des logiciels de collaboration ont été installés. Les données, tirées de l’observation, des groupes de discussion et des résultats des opérations d’information, ont permis (a) de cibler les stratégies et les ressources qu’utilisent les analystes pour préparer leur plan d’opérations d’information et colliger les renseignements demandés par le commandant, (b) de déterminer la façon dont ils collaborent pour établir une même compréhension de la situation, (c) de cerner la façon dont ils ont utilisé l’application TAG et (d) de vérifier l’utilité et la convivialité de celle-ci dans le cadre d’opérations d’information relatives aux AI.

L’étude, à laquelle trois analystes du domaine des AI ont participé, s’est déroulée sur une période d’une journée et demie, soit une formation d’une demi-journée sur l’application TAG, suivie d’une journée complète consacrée à un exercice de secours aux sinistrés. Plus précisément, les analystes ont utilisé l’application TAG pour effectuer leurs tâches dans le cadre d’une simulation tenue au laboratoire C2L2 et produire ensuite un rapport final. Des données ont été recueillies par le truchement d’observations, de questionnaires sur l’application, du suivi de son utilisation (nombre de clics sur un page, temps passé sur une page, etc.) et de commentaires formulés par écrit ou de vive voix après l’étude. Elles ont fait l’objet d’une analyse qualitative pour évaluer, du point de vue des participants, le degré de pertinence de l’application TAG dans la réalisation de leurs tâches.

Les résultats de l’étude démontrent que deux des trois analystes ont utilisé régulièrement l’application TAG durant la simulation d’opérations d’aide humanitaire, et l’ont fait avec une certaine aisance malgré leur formation sommaire. Les participants ont reconnu que l’application était conviviale et adaptée à la production de rapports de mission et aux préparatifs en vue de rencontres en personne. Ils l’ont trouvé toutefois moins utile à titre de mécanisme de transfert de connaissances dans un milieu d’état-major où l’on a accès à d’autres outils, par exemple un ordinateur personnel. De plus, les analystes ont avancé que l’application gagnerait en utilité si elle renseignait davantage et donnait accès à plus de liens (p. ex., des cartes), tout en proposant plus de modèles pour interagir avec des dirigeants principaux et élaborer des processus de planification. D’autres recherches seront nécessaires pour comparer l’application TAG à un témoin afin d’en valider et d’en mesurer la valeur ajoutée si son adoption est prévue dans la doctrine, à la fois sur le plan de l’instruction et du théâtre d’opérations.

DRDC-RDDC-2019-R217 iii

Importance pour la défense et la sécurité

L’étude a donné l’occasion aux participants des FAC d’évaluer de nouveaux outils logiciels et de se familiariser avec le laboratoire C2L2 dont ils peuvent disposer pour réaliser leurs simulations. Le présent exercice a permis aux chercheurs de RDDC de recueillir de précieuses données en situation réelle pour développer davantage les processus et les outils qui simplifient le travail de recherche et d’analyse des analystes du domaine des AI au sein des FAC. L’étude a également permis aux chercheurs de voir une équipe du domaine des AI utiliser in situ cet outil expérimental, ce qui permettra de le perfectionner, de l’utiliser plus efficacement et d’améliorer son efficience dans un théâtre d’opérations.

iv DRDC-RDDC-2019-R217

Table of contents

Abstract ...... i Significance to defence and security ...... ii Résumé ...... iii Importance pour la défense et la sécurité ...... iv Table of contents ...... v List of figures ...... vii List of tables ...... viii Acknowledgements ...... ix 1 Introduction ...... 1 1.1 Comprehensive Approach Toolkit (CAT) Mobile Application: Research and Development . 1 1.1.1 Design, Research and Development of the CAT App ...... 2 1.2 Current Use Case: Humanitarian Scenario Exercise ...... 7 2 Method ...... 9 2.1 Participants ...... 9 2.2 Procedure ...... 9 2.2.1 Overview ...... 9 2.2.2 Tasks for IA Analysts ...... 10 2.2.2.1 Mission Vignettes ...... 11 2.2.3 Available Materials and Tools ...... 12 2.2.4 Data Collection ...... 13 2.3 Analyses ...... 14 3 Results ...... 16 3.1 Usability and Utility of the CAT App ...... 16 3.1.1 User Questionnaires ...... 16 3.1.2 Qualitative Feedback ...... 16 3.1.2.1 General Qualitative Feedback on the Usability of the CAT App and the Relevance of the Exercise ...... 16 3.1.2.2 Module-Specific Qualitative Feedback of the CAT App ...... 17 3.1.3 CAT App Usage Analytics ...... 18 3.1.3.1 CAT App Usage Analytics—Individual Differences ...... 19 3.1.3.2 CAT App Usage Analytics—Usage Estimations ...... 21 3.2 Utilization of Additional Tools Within the HQ Environment ...... 23 3.3 Final Mission Report Analysis ...... 26 4 Discussion ...... 27 4.1 Recommendations to Improve the Usability and Utility of the CAT App ...... 27 4.1.1 Potential Implications of User Feedback ...... 27 4.1.2 CAT App Usage Analytics ...... 28

DRDC-RDDC-2019-R217 v

4.1.3 Future Requirements for the CAT App ...... 28 4.2 Limitations of the Current Study and Future Directions ...... 29 4.3 Conclusions ...... 29 References ...... 31 Annex A CAT App Questionnaire ...... 33 List of symbols/abbreviations/acronyms/initialisms ...... 35

vi DRDC-RDDC-2019-R217

List of figures

Figure 1: Screenshots of the CAT app...... 1 Figure 2: Layout of the C2L2 with IA participants at their work stations (A) and when meeting DJ Newtz at the Radio Station (B)...... 9 Figure 3: Screenshots of the user login, informed consent, demographics and feedback provided within the CAT app...... 14 Figure 4: (A) Mean number of clicks, (B) frequency of usage and (C) ...... 19 Figure 5: (A) Implicit example during Vignette 1 when User 2 (Sgt) used the Acronyms and Terminology module to look up (B) the Performance Measurement Framework (PMF) acronym and then (C) ...... 22 Figure 6: (A) Ad-hoc example of when User 2 (Sgt) used the RBM vs. OPP module to train his team on the differences between the RBM and OPP processes. (B) Example clicks (represented by yellow dots) while navigating through the RBM vs. OPP module. .. 22 Figure 7: (A) Optional examples of when (B) User 1 used the Country Guide module to look up information on Guyana culture to prepare for an interview with a local DJ and (C) . 23 Figure 8: Initial planning processes prepared by IA Analysts on the white board provided to them...... 23

DRDC-RDDC-2019-R217 vii

List of tables

Table 1: Details of the CAT app...... 2 Table 2: Summary of the estimated usage of the CAT app modules...... 12 Table 3: Rubric used to evaluate the Final Mission Report...... 15 Table 4: Detailed usage analytics for each module and sub-module of the CAT app per user. . 19 Table 5: Assessment of the quality of the mission report generated by the IA Analysts. ... 26

viii DRDC-RDDC-2019-R217

Acknowledgements

The authors acknowledge Hamid Boland, James Newton and John Townsend for their technical contributions to the project and LCol Ron Bell (Retired) for his support of the CAT app. The authors also acknowledge the reviewers for their insight on this Report. The authors would like to thank all of the CAF personnel who volunteered their time to participate in the study.

DRDC-RDDC-2019-R217 ix

This page intentionally left blank.

x DRDC-RDDC-2019-R217

1 Introduction

1.1 Comprehensive Approach Toolkit (CAT) Mobile Application: Research and Development

Increasingly, the Canadian Armed Forces (CAF) is being called upon to work in multifaceted mission settings involving interagency, allied and public players while on deployment. This is most recently reflected in the government’s commitment to peace support operations which require a Whole of Government (WoG) response. Success in these missions is achieved by effectively collaborating and navigating working relationships with other organizations and populations in the human environment including the joint, interagency, multinational and public (JIMP) components of a comprehensive approach (CA) setting. This is often a challenging endeavor due to dissimilarities in culture, terminology, technology, policy, values, and other differences. Ongoing research at Defence Research and Development Canada (DRDC) has focused on identifying and addressing capability gaps with respect to how the CAF collaborate with other organizations in training and mission environments.

In support of this goal, the Comprehensive Approach Toolkit (CAT) was developed to create and consolidate a number of training, education and aide-memoire resources for use in WoG mission contexts. When subject matter experts (SMEs) requested an accessible means of taking the CAT app into theatre and training, DRDC – Toronto Research Centre developed a mobile application (CAT app) prototype to enhance collaboration in CA settings, an effort that is aligned with CAF efforts to modernize its individual training and education system.

Figure 1: Screenshots of the CAT app.

Technology provides the learner with access to a world of knowledge via smartphone, tablet, or computer. Mobile learning (m-learning) is an emerging field of educational research (Denham, Gonzalez-Sanchez, Chavez-Echeagaray & Atkinson, 2012) and can serve to deliver content and support real-time communication as a remote knowledge transfer tool (e.g., Wayne, Perez, Kaplan & Ritvo, 2016). In addition, m-learning provides a mobile test-bed to conduct research by leveraging the usage-tracking and data exporting capabilities of a smartphone (Granek, Nazarov, D’Agata, Bae, Boland et al., 2019). Ensuring that the design and the development of the mobile app abides by interaction design principles

DRDC-RDDC-2019-R217 1

(Tognazzini, 2003), usability heuristics (Nielsen, 1994), and mobile design guidelines (Hoober & Berkman, 2011) can improve the user experience and compliance to training. Further, usability testing and analytics usage-tracking and user-feedback can ensure that the development of the mobile app adheres to the needs and requirements of specific trades or organizational cultures. The iterative usability and utility testing is crucial to uncover the assumptions made about how the users want to use the mobile apps (Hall, 2013). 1.1.1 Design, Research and Development of the CAT App

By harnessing mobile application technology as a means of knowledge transfer, we developed the CAT app to provide the user with easy access to nine tools intended to facilitate interaction with their WoG partners (Table 1). These tools contained within the CAT app include seven components that address concerns and capability gaps previously identified by subject matter experts (SMEs; Holton, Thompson, Febbraro, Hendriks & Gill, 2016), as well as additional features including a country guide used to provide offline open-source information on the country of deployment and collaboration tools used to capture photos, video, audio and notes. These additional features were added to the CAT app following presentation of the prototype to the sponsor and stakeholders, as well in response to feedback from SMEs who had trialed the prototypes. See Table 1 for a full description of each feature within the app. The prototype is modular in nature, and will allow for mission-specific information to be updated depending on mission context through the creation of a content management system (CMS) with a Graphical User Interface. The CAT app is available for iOS, Android and Windows devices.

Table 1: Details of the CAT app. Function Design Background App Description The CAT app features a Provides all players in a The Comprehensive number of training, CA context with easy Approach Toolkit education and aide- access to eight training, Mobile application memoire resources for use education and aide- in WoG mission contexts. memoire resources tools intended to facilitate interaction with their WoG partners.

2 DRDC-RDDC-2019-R217

Function Design Background App Description Country Guide The Country Guide Allows users to view section provides information on the information on the country deployment country’s of deployment. history, culture, Information is from open politics, geography, source including Central economy and media Intelligence Agency (CIA) information. Factbook and Global Affairs Canada (GAC).

Acronyms and The Acronyms and Provides users with Terminology Terminology section is details about the term, designed to help users such as, the definition, look up acronyms and other terminology used by who uses the term, and various organizations in a what other terms are CA mission environment. related to it. Users can It includes an initial search acronyms or database of acronyms and words in the local terminology from CAF, other government database and get basic departments (OGDs), non- information about each governmental term. organizations (NGOs) and international organizations (IOs) and can be updated with new acronyms and terminology specific to mission.

DRDC-RDDC-2019-R217 3

Function Design Background App Description Scenario Development The Scenario By answering Development section is situational judgment designed to help users questions related to the improve their negotiation, different scenarios communication, and provided in this section, relationship-building users will be able to skills. It focuses on gain knowledge on how communication, to tackle problems in negotiation and different CA contexts relationship building while improving the needed in CA contexts. skills mentioned above.

RBM vs. OPP Quick The Results Based Learn about and Reference Guide Management (RBM) vs compare the OPP used Operational Planning by the CAF and the Process (OPP) Quick RBM process used by Reference Guide is GAC. It provides users designed to help users understand what RBM and with information to OPP are, as well as the allow them to organize differences and their own projects using similarities between the either the RBM or OPP two planning approaches. method.

4 DRDC-RDDC-2019-R217

Function Design Background App Description Training and Education The Training and Users can search a Database Education Resource database of tools that section is designed to help can be filtered based on users find courses, what they are looking workshops, and other tools for. Where applicable, available to help them each resource or tool learn more about working in a CA context. Look up links to the tool or CA training and education resource webpage tools, courses and directly from the app. literature available to Search a database for members of the CAF and courses and other tools to civilians working in CA relevant to topics the contexts. user is interested in. Specifically, users can filter the search by Primary Audience, Themes, and the Type of course, add courses of interest into the Favourites tab for quick reference in the future, and get details (location, description, etc.) for the courses as well as external links for more information. Lessons Learned The Lessons Learned (LL) Users can search a Database Resource section is database of LL and best designed to help users find practices that can be CA lessons and best filtered based on what practice recommendations they are looking for. based on previous operations. Users can look Specifically, users can up LL from Exercises and filter the search by the Operations involving the Operation and the CA. Includes unclassified Issues discussed in the LL obtained from the lessons, add lessons of Knowledge Management interest to the System and the Army Favourites tab for quick Lessons Learned Centre reference in the future, Tracker. and get details (descriptions, recommendations, etc.) for the lessons as well as references to a larger

DRDC-RDDC-2019-R217 5

database should the user want more information. Future updates will include ability to gather new LL and export them via email. Religious Leader The Religious Leader Provides information Engagement Engagement (RLE) about each of the section is designed to briefing tools, such as provide users, specifically the PPEAT (planning, Chaplains, with various preparation, execution, tools to help them with and assessment tool), organization and the JIMP Chart, Quad gathering of information Chart, and the for their projects. The ASCOPE (Areas, RLE section uses Structures, Capabilities, education and training Organizations, People resources developed by and Events) / PMESII military chaplains. (Political, Military, Economic, Social, Infrastructure and Information) chart,

along with details on how to use them. It also contains the Army Doctrine Note which provides the users the opportunity to review the current doctrine on the topic of RLE.

6 DRDC-RDDC-2019-R217

Function Design Background App Description Briefing Tools The Briefing Tools section Provides on-the-go allows users to fill in and fillable JIMP Chart, export briefing charts for Quad Chart, and the mission reporting. ASCOPE/PMESII chart that can be emailed out.

Collaboration Tools By harnessing the Allows user to take advanced features of the notes and photos, and smartphone, the record audio and video Collaboration Tools and email out the section allows users to attachments without gather information on-the- having to exit the CAT go, which can then be app. shared.

1.2 Current Use Case: Humanitarian Scenario Exercise

The CAT app has been through validation and testing in the lab as well as in theatre with psychological operations (PSYOPS) and civil-military co-operation (CIMIC) CAF members. Different iterations of the app have gone through usability analyses using concurrent think aloud interviews, a small CAF Exercise which took place in Georgina Ontario (Ex Georgina) and a trial for use in theatre in Latvia (OP Reassurance). While the Ex Georgina and Latvia Trial data collection opportunities did not allow for scientists to be present, data could be collected remotely and downloaded once the devices were/are returned to the investigators at DRDC. The context for which the users engaged with the CAT app, however, is not always clear in remote settings. Simulated environments offer a safe and controlled opportunity to evaluate a software tool in potentially dangerous or non-permissive environments (Rosas,

DRDC-RDDC-2019-R217 7

Hidalgo, Gil-Costa, Bonacic, Marin, et al., 2016). In order to thoroughly evaluate the usability and utility of the CAT app in a controlled setting, we used an amended version of the Hendriks and Kwantes (2016) simulated disaster-relief scenario based out of Georgetown, Guyana, which centred on real footage and materials from the flood in November, 2014. The details of the scenario are discussed in Holton, Granek, Kozey, Bae, Hendriks, Newton, Townsend, Fraser, & Kwantes, 2019) In both the original and the present study, the participants played the role of a three person team of CAF Influence Activities (IA) Analysts who were deployed to Georgetown, Guyana to work in the CIMIC/PSYOPS operations centre in support of an international disaster relief team.

The purpose of this study was to gain information pertaining to the CAF IA information operation process and to assess the CAT app’s efficacy in providing support to operations when integrated into their process. Observational data, Information Operations products, and participant feedback (written and/or oral) were collected to: a) identify strategies and resources analysts use to prepare an information operation plan and search for information required by the commander, b) determine how analysts collaborate and create shared situational understanding, c) determine how IA Analysts use the CAT app, and d) determine the usefulness and usability of the CAT app for the conduct of IA information operations.

8 DRDC-RDDC-2019-R217

2 Method

2.1 Participants

A team of three CAF IA members, who currently work in the areas of PSYOPS or CIMIC, participated in this study. All participants were current members of the CAF, between 18–60 years of age. Participants were recruited through official channels within DRDC – Toronto Research Centre: DRDC’s Operations Officer contacted the Operations Officer at 32 Brigade at Denison Armoury to request permission to advertise our recruitment poster to 32 IA Company. Supervisors of 32 IA Company were not involved with the study, aside from sharing the recruitment poster and granting permission to participate. 2.2 Procedure 2.2.1 Overview

The study took place in the C2L2 at DRDC – Toronto Research Centre. The C2L2 is depicted in Figure 2. It is a collaborative work environment equipped with networked computers, smart boards, and shared displays. It provides users the ability to video teleconference, chat, take control of a shared display, or take control of each other’s displays during collaborative activities.

Figure 2: Layout of the C2L2 with IA participants at their work stations (A) and when meeting DJ Newtz at the Radio Station (B).

The study reported here uses a small group of participants (three IA Analysts) over an extended period of time so that various aspects of the tool’s utility and usability can be assessed in depth. Because the study uses a small participant sample, most of the data we collected to assess the tool were qualitative in nature. The team of IA Analysts participated in the study over two consecutive days of testing. The first day consisted of a 4 hour training session. The second day consisted of a full eight-hour test session. Details for each of the sessions are as follows:

Day 1:

Participants read and signed the consent form before beginning the study. The first day of participation consisted of a half day of training. Upon their arrival, they were familiarized with the C2L2 facility and the available software they could use. Next, the participants received training on the CAT app whereby participants performed actions in a series of scenarios for each of the app’s modules (e.g., search

DRDC-RDDC-2019-R217 9

acronyms and get basic information about the terms) in order to highlight the main functionality of the CAT app and to familiarize participants with the relevance of the tool.

Following training, participants received an Operation Order (Op Order; see Holton et al., 2019) as read-in materials to review prior to Day 2.The Op Order consisted of a detailed overview of the scenario, written in standard CAF format by a former Intelligence Officer, and provided participants with all of the information needed to complete their task. The Op Order provided information relevant for a multitude of roles in the operation. In this instance, participants in this study would have focused their attention on the content related to the duties that would fall under the J9. For example, part of our participants’ responsibility in the scenario would be to communicate health issues that people in the affected area need to be aware of in order to prevent disease—a responsibility that would generally fall to the J9. Hence, while participants were not instructed to focus on the J9 section of the Op Order, as SMEs in their respective fields, that section would be the most relevant to the duties they would be asked to perform.

Day 2:

On the second day, participants partook in a full day scenario which took approximately eight hours. The Op Order provided participants with the details regarding a natural disaster in which a severe flood in Georgetown, Guyana, occurred a result of a tsunami that had hit the area three days earlier. As a result of the flood, telephone lines were down, power was out throughout most of the city, and many cellular phone towers were damaged. 2.2.2 Tasks for IA Analysts

The IA Analysts were tasked to use a the Intelligence Preparation of the Operational Environment (IPOE) found in Holton et al. (2019) and to use the CAT app to complete the task of gathering information on the Human Environment (HuE)1 to develop and plan a campaign in the context of the flood scenario (Hendriks & Kwantes, 2016). Specifically, participants were required to find out:

1. What are the health issues that people in the affected area need to be aware of?

2. What items do locals need to prevent disease?

3. What information needs to get to locals?

4. How will messages get delivered to locals?

5. What are the challenges to delivering messages?

6. Are there any the local influencers who might help or hinder the mission?

Although the scenario was fictitious, the city used in the scenario, Georgetown, Guyana, was real. Using a real place has the potential to greatly enhance the fidelity of the exercise, because unlike other scenarios where all of the data are fictional, analysts in the Georgetown scenario could search for real people and institutions in the Area of Operations to find local influencers and enablers.

1 HuE refers to any socio-cultural factor within an operational environment.

10 DRDC-RDDC-2019-R217

Each team had seven hours to gather information related to their duties. The mission included three planned vignettes, and their job was to create a Mission Report to be presented to the commander upon her arrival (note that the commander was fictional).

2.2.2.1 Mission Vignettes

Upon receiving the Op Order, the IA Analysts were invited to participate in three vignettes as described in Holton et al., (2019). Below, we briefly describe the purpose of each vignette. In Table 2, we summarize what we estimate how and when the app would be used in each vignette. To accurately assess how the app is used by participants, we felt it important to be able to discern when functionalities available in the app were used at the request of the experimenters versus at their own discretion. Hence, as part of the information we wanted to gather about the app’s use, we estimated the extent to which each of the app’s functions would be used when explicitly prompted by characters within the vignettes, or when participants would decide on their own to use them. We used three categories describing participants’ potential interactions with the CAT app’s functions:

Explicit: Represents an explicit request to open up a relevant module of the CAT app.

Optional: Represents an explicit request to perform an act that could be done with the CAT app but could also be done with a different application.

Implicit: Represents the situation wherein the task the participant needs to do can only be done using a particular function of the app. Put somewhat differently, the requirement to use the app’s function is implicit in the nature of the task being performed.

In the following three paragraphs, we outline the three vignettes introduced to participants during the exercise. In each vignette, we represent an event that occurs over the course of the day in Georgetown and a task that is set up in such a way to encourage the use of the CAT app to complete it.

Vignette 1—Teleconference: IA Analysts received joining instructions via an email for an Interdepartmental Task Force Meeting on the situation in Guyana. This meeting was chaired by a GAC representative for the Head of Mission (HoM), and included the Commanding Officer (CO) of the Disaster Assistance Response Team (DART) and an International Committee of the Red Cross (ICRC) representative. The teleconference included a mission update, details on the Canadian response for humanitarian assistance, information on communications with the Press and social media and the next steps for all players involved. The participants in the teleconference referred to several relevant acronyms, LL and planning processes—an aspect of the mission update that was designed to encouraging the use of the CAT app’s modules that would help them understand acronyms, terminology common to other organizations, and the current situation’s relevance to lessons learned in previous missions.

Vignette 2—Social Events Update: IA Analysts received an email from their Chief of Staff (COS) requesting information on upcoming social events that the CAF can use to reach large audiences and to provide an update to the COS about the events in the form of an ASCOPE/PMESII chart. The template of this chart is provided in the CAT app. IA Analysts were also required to develop a rapport with the disk jockey (DJ) at a local radio station in order to book an interview on air to be able to communicate with the public and to attempt to gain their trust.

DRDC-RDDC-2019-R217 11

Vignette 3—Interview at Radio Station: IA Analysts received email from their CO to provide an update on the meeting with the local DJ and to send videos and photos of the interview. The interview allowed the IA Analysts to describe their purpose and reason for being deployed in Guyana, as well as their expected timeline. The IA Analysts were also able to take calls from listeners to discuss some of the concerns of the public, including health and safety concerns and where they could access aid.

Table 2: Summary of the estimated usage of the CAT app modules. Scenario Event Description Time Category of App Estimated CAT app Stamp Interactions Usage (Small- Medium-Large) Op Order is presented to Pre Ex Implicit Large participants Invitation to Teleconference 0830 Implicit Small is sent to participants by email Vignette 1: Teleconference 0900 Implicit Medium with participants and role players representing CAF and GAC leadership: Prompts to interact with the following sections: 1a) Lessons Learned 1b) RBM vs OPP - Implicit Large 1c) Acronym List - Implicit Medium 1e) Lessons Learned - Implicit Medium 1f) RBM : Outcomes - Implicit Large 1g) RBM : PMF - Implicit Medium Vignette 2 : Social Events 1000 Explicit Large Update with local DJ 2a) Request for participants to email a completed ASCOPE/PMESII Chart 2b) Background on Guyana - Implicit Small Culture in Country Guide Vignette 3: Request to record 1230–1330 Optional Large interview at radio station using Collaboration Tools

2.2.3 Available Materials and Tools

In addition to the CAT app, the following tools were made available to assist the participants with the task:

1. C2L2. The C2L2 consists of five user stations that face a large screen located at the front of the room. Each analyst sat in front of an individual workstation.

12 DRDC-RDDC-2019-R217

2. Internet-enabled Search. Each analyst had access to the internet from their workstation.

3. Microsoft PowerPoint on a Shared Canvas. Each user station faced a large screen located at the front of the room. The left half of the main screen consisted of a shared canvas that allowed any user to modify the material on the screen by dragging their mouse up past the top of their workstation screen, into the shared canvas.

4. Local News and Cell phone Videos of Georgetown Floods. A plasma screen, located on the left wall displayed a looping video of news clips that had been broadcasted during a severe flood in Georgetown, Guyana in November 2014. This source of information was added to provide useful information (e.g., in one video, the mayor is interviewed) regarding the exercise as well as to increase the fidelity of the task.

5. Guyana Flood Update Injects. A series of injects (expanded from those used in Hendriks and Kwantes, 2016) were presented on the main screen (one inject presented every two minutes), which consisted of updates on how the flood was affecting the city. The injects were similar to a newsfeed that one would see on a news station such as CP24. The injects simulated real-time intelligence and media reporting from the affected area. They were presented in the form of a short statement comprising a one- or two-sentence update with a time stamp and a source. For example, “12:02pm: Four patients have been admitted to Davis Memorial Hospital.”

6. Google Map Pro. Google Map Pro was installed on the multi-user touch-screen device located at the front of the C2L2 room. The Google-Map screen was also repeated/displayed on one of the wall displays so that the participants using the mobile devices or desktop computers were able to view the map layers of the area.

7. Microsoft Office. Specifically, participants had access to Word, PowerPoint, and Excel.

8. White boards, pens and paper. 2.2.4 Data Collection

Upon completing their final report, each team was asked to provide written and oral feedback for up to 30 minutes to discuss the extent to which they found the CAT app and the scenario useful. Finally, participants completed usability questionnaires for the CAT app within the app itself (see Figure 3). We used the Standard Usability Scale (SUS) (Bangor, Kortum & Miller, 2008), as well as questions that we created ourselves which are included in Annex A. These questions were designed to obtain nuanced insights into the app’s strengths and weaknesses and to gauge the extent to which the modules were considered useful by participants.

DRDC-RDDC-2019-R217 13

Figure 3: Screenshots of the user login, informed consent, demographics and feedback provided within the CAT app.

Internet search activity was recorded on each computer. In addition, the entire exercise was video and audio recorded so that the research team could analyze the interactions, conversations, and gestures of the participants, as per participants’ consent.

During the two phases of the study we collected data to understand search, collaboration, and sense-making activities. Data collection took the form of 1) observational data (e.g., time needed to complete tasks and frequencies of communication); 2) examination of the products created during and following data collection; 3) questionnaire results from participants after the exercise that assessed the usability of the tools; 4) tracking app usage (i.e., number of clicks or swipes, pauses/resumes, transitions); and 5) focus group feedback data. 2.3 Analyses

Qualitative data analyses were conducted on the observational data, questionnaire data and focus group data to assess the extent to which the participants found the CAT app useful when completing their tasks. The SUS and usage analytics (e.g., timing, clicks) captured within the CAT app provided metrics, usability and utility of the tool in context with the scenario. As we discussed above, the expected usage of the specific modules of the CAT app are outlined in Table 2. This table served as a cross-reference for the discussion of the extent to which each module was actually used and if it was used as anticipated. Internet search logs were also captured and cross-referenced with the timeline of the scenario and the usage of the CAT app to offer insight into what other information could be included in the CAT app for future use cases.

Further, the final mission report was evaluated by a member of the research team who has served as an IA analysts. The purpose of assessing the report was strictly to ascertain the extent to which the participants were able to complete their tasks using the CAT app. The inclusion of the assessment was not meant to assess the participants’ abilities to properly format an assessment in writing, but instead to determine

14 DRDC-RDDC-2019-R217

whether they were able to generate a viable and error-free plan in the time allotted, and to assess whether or not the CAT app could aid IA Analysts in reporting information during a mission.

Table 3: Rubric used to evaluate the Final Mission Report. Description Included in Report? Number of Number of errors References (e.g., names, locations)

Is Vignette 1 discussed in report?

Is Vignette 2 discussed in report?

Is Vignette 3 discussed in report?

Is there an Assessment made?

Are images taken during the exercise included at appendices?

Are any analytical tools and/or additional pieces of structured information included as appendices?

DRDC-RDDC-2019-R217 15

3 Results

3.1 Usability and Utility of the CAT App

In this section, we evaluate the qualitative and quantitative usability and utility feedback following the interaction with the CAT app in the current exercise. Since we only evaluate three participants, we focus our analyses on the key insights derived from in-depth qualitative feedback provided by the participants over a day of usage. The quantitative isn’t as valuable in this case, but is presented to demonstrate the data-collection capabilities of the CAT app as well as a preliminary means of comparison with the qualitative feedback. 3.1.1 User Questionnaires

Following the training on Day 1 and the exercise on Day 2, participants completed a SUS, a generic usability scale with validated psychometric properties (Brooke, 1996), and answered a series of specific, custom questions on the functionality of each module of the CAT app.

For the SUS pooled across modules, mean responses were considered to have “excellent” usability (Bangor et al., 2008) at both time points (Day 1: = 97.5±1.4/100; Day 2 = 86.7±8.4/100).

When asked more specifically by our custom follow-up questionnaire relevant to the utility of the CAT app, participants scored the app at a mean of 65±5.8/100. Overall, the participants found the CAT app was well suited for CAF personnel and missions, and more specifically, they found that it was most relevant for generating mission reports and for preparing for face-to-face interactions. On the flipside, however, users found the CAT app the least useful as a knowledge transfer tool while in a headquarter (HQ) setting (i.e., with internet access) and they felt neutral about whether they could continue to perform their duties without noticing the absence of the tool. 3.1.2 Qualitative Feedback

3.1.2.1 General Qualitative Feedback on the Usability of the CAT App and the Relevance of the Exercise

Following amalgamation of written and oral feedback by the participants, here we summarize the common themes and examples of the usability and utility issues and report some of the recommended possible enhancements:  CAT app usability issues  The App could benefit from better use of space, such as using bigger buttons.  The English vs. French button should be moved to a different spot located further away from the back button to avoid mistakenly changing the language.  Make saving location of charts, pictures and other files more clear and accessible so that the user can easily share them as an attachment.

16 DRDC-RDDC-2019-R217

 Experimental trial feedback  Trials would be more useful in a field setting (or different versions of the app for both settings).  The inject regarding the request to export a completed ASCOPE/PMESII for the social event (e.g., concert) was inappropriate and does not reflect the proper use of this tool. . The ASCOPE/PMESII tool is meant for broader or more general aspects of a particular subject/society/sample.  Add a keyboard and a stylus for easier interaction with the app.  Provide access to a SharePoint for cloud collaboration (pending Wi-Fi availability / security requirements).  Proposed general enhancements for the CAT app  Add a universal highlight tool.  Add a calendar that has a scheduling and reminder feature that can be used for scheduling.  Add additional content from the aid memories found in the Army Collaborative Information Management System (ACIMS).  Make customizable quick links for “favourites” for quicker access to frequently used modules.

3.1.2.2 Module-Specific Qualitative Feedback of the CAT App

Here we summarize the common themes from the written and oral feedback of the participants for the specific usability and utility issues for each module:  Country Guide  Add hyperlinks to additional resources (e.g., maps, pending Wi-Fi availability / security requirements).  Include a list of slang terms, greetings, and other common terms specific to the country.  Add the ability to save pictures, maps, and other content from within the app so that it can be exported.  Include offline resources such as national anthem audio file for the country.  Training and Education  Not relevant for this trial.  RBM vs OPP  Would be beneficial to have additional in-depth information for training purposes for naïve users to properly integrate with other agencies.  Lessons Learned  When searching keywords in the LL search field, it would be good to get a preview of the information provided linked to the keywords.

DRDC-RDDC-2019-R217 17

 Religions Leader Engagement  Not obvious that the Quad Chart section is linked to the Briefing tools module on a new page.  Direct relevance is not clear for this trial.  More information would be useful on RLE in general.  Briefing Tools  ASCOPE/PMESII is not clearly different from RLE.  Create a tool for interacting with key leaders that has a template whereby users can modify a profile of a key leader, categorize key personalities, add a name and ethnicity, as well as attach a picture and additional files.  Provide templates in the app for PSYOPS/CIMIC reports (if feasible).  Collaboration Tools  Add a doodling app as part of notes.  When taking a picture, add the ability to catalogue it with descriptions of the 5 “Ws” (i.e., Who, What, Where, Why, When).  Don’t ask the user to save every time that a picture is taken; the picture should be saved automatically. 3.1.3 CAT App Usage Analytics

We built a functionality to record all user clicks, transitions, as well as pauses and continuations of interactions with the CAT app. Overall, six of the nine modules were accessed during the trial (see Figure 4). Of those six modules, the Collaboration Tools (e.g., notes, video) and the Acronym and Terminology modules were interacted with the most, followed by the RBM vs. OPP and Country Guide modules. It is important to consider, however, that although a user spent a lot of time on a current module, if they did not interact frequently with the module (i.e., number of clicks), they most likely kept the app open for future use.

18 DRDC-RDDC-2019-R217

Figure 4: (A) Mean number of clicks, (B) frequency of usage and (C) total usage time (minutes) by all participants per module used during the study. Error bars represent standard error.

3.1.3.1 CAT App Usage Analytics—Individual Differences

The team of three IA Analysts interacted with the CAT app differently depending on their distinct roles within the scenario. Table 4 presents the details of the CAT app analytics for each user. As mentioned above, some of the individual numbers for time spent on each module may be misleading (e.g., User 2 for Acronyms and Terminology), and as such, all variables (clicks, frequency and time) should be considered in order to accurately interpret the usage of the app. The Sergeant (Sgt; User 2) took on the leadership role and explicitly used the CAT app as a reference and data collection tool to guide his team and to complete requested tasks. User 1 took on the role of searching for information within the CAT app at the request of the Sgt and took some time exploring the different relevant modules. User 3 did not interact with the CAT app during the scenario, but was consistently tasked by the Sgt for requests for information (RFI) not contained within the CAT app. These additional RFIs have been captured by the internet search logs described in Section 3.2.

Table 4: Detailed usage analytics for each module and sub-module of the CAT app per user. User Module Sub-Module Number of Frequency Time Spent Clicks of Visits (seconds) 1 Country Guide Index Page 14 10 1355.054

Country 4 1 351.491 Insights Culture 2 5 489.562

Economy 1 1 4.083

Overview 1 1 9.142

DRDC-RDDC-2019-R217 19

User Module Sub-Module Number of Frequency Time Spent Clicks of Visits (seconds) Politics 3 1 8.35

Acronyms & Index Page 39 8 7295.81 Terminology RBM vs. OPP Index Page 3 1 6.955 Comparison 15 2 4.025 Questions 4 1 4.365 for Planners Stories 6 1 5.485 Tools - 5 1 11.899 Design Tools - 2 2 5.914 Index Tools - 2 2 197.398 Language Tools - 20 2 251.885 Planning Tools - RBM 3 3 47.962 Chain Scenario Index Page 2 2 6.274 Development Category - 1 1 7.998 Listing Briefing Tools Index Page 3 3 30.669 Collaboration Tools Index Page 91 67 380.368

Audio 43 2 180.142

Picture 13 1 6.426

Video 12 2 4.207

2 Acronyms & Index Page 5 2 20965.967 Terminology RBM vs. OPP Index Page 6 5 45.027 Comparison 41 11 14009.513 Questions 10 4 109.798 for Planners

20 DRDC-RDDC-2019-R217

User Module Sub-Module Number of Frequency Time Spent Clicks of Visits (seconds) Stories 6 2 20.102 Briefing Tools Index Page 3 1 4.772 Collaboration Tools Index Page 25 9 361.505

Video 3 2 45.201

3.1.3.2 CAT App Usage Analytics—Usage Estimations

We estimated the amount and type of usage of the CAT app based on the expected relevance of the modules to the scenario and the type of prompt (characterized as implicit, optional, or explicit; see Table 2). As expected, the IA Analysts interacted with several of the modules that were relevant to the scenario and that they were prompted to use. Overall, participants frequently interacted with the Country Guide, Acronym and Terminology, Collaboration Tools, and RBM vs. OPP modules. Participants also visited the Scenario Development module, but did so infrequently. Interestingly, the participants did not utilize the LL module, despite the references to previous LL in the vignettes. Participants also interacted with the CAT app in unexpected (i.e., ad-hoc) ways, such as when the Sgt was training his team on the RBM vs. OPP process.

Figures 5–7 provide visual representations of example screenshots depicting purposeful sequences of interactions with the CAT app (shown as yellow dots on the screenshots) at three different point of time within the exercise. At these points in time, we highlight examples of users successfully navigating the tool to complete tasks in three different contexts. Figure 5 depicts example screenshots from the situation in which the sequence of interactions with the CAT app occurred when participants were implicitly prompted to look up an unknown acronym (e.g., PMF; Performance Measurement Framework). After looking up the acronym and as alluded to above, the Sgt then, on his own accord, proceeded to the RBM vs. OPP module to review a comparison of the planning processes and found more information on PMF during the 4th stage of the RBM. Next, Figure 6 represents an example of a sequence of events when the Sgt continued to interact with the RBM vs. OPP module on his own accord (i.e., ad-hoc) in order to serve as a training aid for the rest of the IA team. Figure 7 provides examples of successful interactions with the CAT app’s functionality when the participants could have used other options. In this case, User 1 “optionally” looked up information on the background of Guyana’s culture within the Country Guide prior to meeting with the radio DJ in Vignette 3 and User 2 recorded parts of the interview. Finally, participants also interacted with the CAT app successfully when explicitly prompted such as when User 2 filled out the PMSEII/ASCOPE chart within the Briefing Tools module during Vignette 2 (see overall interactions in Table 4 above).

DRDC-RDDC-2019-R217 21

Figure 5: (A) Implicit example during Vignette 1 when User 2 (Sgt) used the Acronyms and Terminology module to look up (B) the Performance Measurement Framework (PMF) acronym and then (C) review how it relates to the RBM planning process within the RBM vs. OPP module.

Figure 6: (A) Ad-hoc example of when User 2 (Sgt) used the RBM vs. OPP module to train his team on the differences between the RBM and OPP processes. (B) Example clicks (represented by yellow dots) while navigating through the RBM vs. OPP module.

22 DRDC-RDDC-2019-R217

Figure 7: (A) Optional examples of when (B) User 1 used the Country Guide module to look up information on Guyana culture to prepare for an interview with a local DJ and (C) User 2 used the Collaboration Tools module to record parts of the interview to at the request of their CO.

3.2 Utilization of Additional Tools Within the HQ Environment

In order to gain insight into the complete requirements for IA Analysts during a humanitarian scenario, a suite of tools (e.g., computer, white board) in addition to the CAT app were made available to the participants. Before any interaction with the CAT app, participants chose to initially organize their plan using the white board made available to them in order to clearly identify the roles of all the players involved (see Figure 8).

Figure 8: Initial planning processes prepared by IA Analysts on the white board provided to them.

Further, we examined the internet searching history, as well as the summary of email content, both sent and received, for all users to offer insight into what types of information are required for the successful completion of this mission that are not currently available within the CAT app. Here we summarize the missing content required for the participants to complete the task, by module:

DRDC-RDDC-2019-R217 23

1. Disaster Assistance

The current version of the CAT app is missing critical information specifically related to disaster assistance and emergency response efforts. In our study, User 2 conducted four internet searches to access information about DART from the websites www.canada.ca and www.forces.gc.ca. In particular, he searched for information about DART in general, how DART responds to emergencies, and past DART deployments. While the CAT app does include a brief definition of DART in the Acronyms and Terminology module, it does not provide detailed information about how DART responds to emergencies or past deployments. Further exploration of the data revealed that the participants did not attempt to search for information on DART using the Acronyms and Terminology module.

In addition, User 2 conducted two internet searches using Google to search for information about boiling water, in particular, “boiling water and bubonic plague” and “boiling water and cholera.” The CAT app does not include information about disaster assistance relief. It is clear that this is a deficiency within the app for this type of humanitarian mission and that general information about disaster assistance relief should be included as an additional offline resource module within the app. If used for a similar humanitarian mission, this module could be populated using the built-in CMS. Within the disaster assistance relief module, information about different types of disasters, such as floods, hurricanes, and earthquakes, as well as information about the types of relief efforts specific to each type of disaster could be included. For example, with respect to flood relief, the inclusion of information about how to prevent the spread of disease of waterborne and airborne illnesses, how to purify water, the use of latrines, emergency medical care set up, displaced persons management, transportation, communication, etc., could be useful. Further analyses are required in conjunction with SMEs to decipher the extent of information related to disaster assistance.

2. Country Guide

Currently, our country guide provides a detailed overview of the history, geography, politics, country insights, information environment, economy, and cultural figures in Guyana. In our study, User 3 visited the Government of Canada website to find information about bilateral relations between Canada and Guyana, such as bilateral Canada-Guyana foreign policy, security, and trade, as well as current statistics about product trade and investment, such as Canadian imports from Guyana, Canadian exports to Guyana, foreign direct investment in Canada, and Canadian direct investment in Guyana, Gross Domestic Product (GDP), GDP per capita, GDP growth rate (%), inflation (%), and unemployment (%). It is clear that in this case our country guide is missing pertinent information on bilateral relations between Canada and Guyana. This information could be provided as an offline resource if needed for a specific mission using the built-in CMS.

3. Acronyms and Terminology

The CAT app includes an acronym and terminology database that provides the user with the full name and definition for over 175 acronyms. In our study, User 1 searched the internet for the terms “community relations group,” “IA production request form,” “PSYOPS,” “provence house,” “PSYOPS production,” and “PSYOPS production request form.” User 3, searched the internet for information about “CCIR,2 FFIR,3 and EEFI,4” a search that led him to a pdf document entitled Focusing Intelligence that provided

2 CCIR = Commander’s Critical Information Requirements 3 FFIR = Friendly Force Information Requirements 4 EEFI = Essential Elements of Friendly Information

24 DRDC-RDDC-2019-R217

him with an overview of the intelligence process, including definitions of the terms CCIR, FFIR, EEFI, as well as other terms relevant to the intelligence process (PIR,5 etc.). This finding suggests that IA analysts would benefit from the inclusion of additional information about the intelligence process in general, as well as more detailed information about PSYOPS.

A review of the acronyms listed in the acronym and terminology database revealed that PSYOPS and CCIR were indeed listed, however, the other items searched were not. Further investigation of the definition found within the acronyms and terminology database for the acronym “CCIR” revealed that the definition cited the terms “friendly force information” and “priority intelligence requirements,” however, these terms do not appear in the database.

4. RBM vs OPP

In our study, User 2 conducted two internet searches to obtain information about the OPP. First, a google search was conducted on the term: “operational planning process.” Second, a pdf document was opened entitled The Operational Planning Process Handbook by Canadian Land Force Staff College. Further review of User 2’s search behaviour showed that he searched the internet for information about the OPP 40 minutes before he navigated through the RBM vs OPP Quick Reference Guide found within the CAT app. At that point, after implicitly being prompted by Vignette 1 (see Figure 5), User 2 navigated through the RBM vs OPP Quick Reference Guide for over 90 minutes (note that the app was not used consistently throughout this time period as the participants attended to other tasks and took breaks as needed). He also revisited the RBM vs OPP Quick Reference Guide later in the day (unprompted usage, see Figure 6) and it was kept open for 90 minutes for repeated reference. User 2 interacted with the RBM vs OPP module when implicitly prompted as well as an ad-hoc high-level resource during the scenario, however, qualitative feedback provided by User 2 suggests that more in-depth information about the RBM vs OPP could be useful in a training environment. The level of in-depth information required for this module should be established further with SMEs for different use-cases.

5. Briefing Tools

A review of the information searched and downloaded from the internet, as well as an analysis of the information included in the emails sent and/or received by our participants, revealed that two tools missing in the CAT app that were needed by the participants to complete their task were the Information Operations Synchronization Matrix, which is a fillable table used to keep track of the key leaders, meeting locations and information required for allied missions, and the Mission Report Template, which is the formatted template used to input the information gathered during a required allied mission in order to generate the final draft of the mission report. In our study, User 1 emailed these templates to himself from his personal email and the participants completed each of these templates during the study as they completed their task. In addition, User 1 searched for information on the internet for two additional tools, namely the IA production request form and the PSYOPS production request form. These forms were not downloaded from the internet and were not used by the participants to assist with the completion of their task, and thus should not be included in the CAT app.

No other tools offered in the C2L2 (e.g., Google Map Pro) were extensively utilized during the trial.

5 PIR = Priority Intelligence Requirement

DRDC-RDDC-2019-R217 25

3.3 Final Mission Report Analysis

The final mission report was evaluated by a member of the research team who has served as an IA analysts. We examined the accuracy and completeness of the final mission report produced by the participants in the current study. Table 5 provides the data assessing the quality of the final mission report against the rubric described above, and provides metrics relating to the amount of information provided to participants, and their ability to capture and report it in the time allotted. Overall, the data captured in the mission report assessment provide preliminary evidence that the materials provided in the current study (including the CAT app) were sufficient for gathering information on each of the vignettes, as well as to provide relevant images and analytical tools in the mission report. Further work is required to understand why an assessment was not included in the current final mission report. Although the data were sparse for our scenario and therefore provide minimal information regarding the quality of the final report, the rubric will be used as a basis for comparing the quality of the reports created with or without the help of the CAT app in future evaluations.

Table 5: Assessment of the quality of the mission report generated by the IA Analysts. Description Included in Report? Number of Number of errors References (e.g., names, locations) Is Vignette 1 Yes 4 0 discussed in report?

Is Vignette 2 N/A N/A N/A discussed in report?

Is Vignette 3 Yes 4 0 discussed in report?

Is there an No N/A N/A Assessment made?

Are images taken Yes N/A N/A during the exercise included at appendices?

Are any analytical Yes N/A N/A tools and/or additional pieces of structured information included as appendices?

26 DRDC-RDDC-2019-R217

4 Discussion

This Report summarizes a controlled assessment of the usability and utility of a mobile application designed to aid CAF IA Analysts in their work. We evaluated a team of IA Analysts during a CAF-relevant and controlled humanitarian mission scenario to provide insight into the usability and utility of the CAT app in order to optimize the end-user experience, as well as to formulate recommendations in order to further configure and enhance the existing platform to enhance its value as a deployable tool. As discussed below, overall, the participants found the current tool useful for performing their operations, specifically for generating mission reports and preparing for face-to-face interactions. Based on their usage analytics and their feedback, the CAT app could be further configured and enhanced in order to increase compliance and ultimately provide the potential of positively streamlining their planning process, provide remote access to offline and searchable databases, and for reporting up their chain of command. 4.1 Recommendations to Improve the Usability and Utility of the CAT App 4.1.1 Potential Implications of User Feedback

The qualitative usability feedback provided by the participants provides insight into the nuances of how the CAT app could support IA Analysts during a humanitarian scenario. Occasionally, participants struggled to interact with the CAT app, and in addition to the recommendations provided by the users to use larger buttons and additional hardware (e.g., stylus, keyboard), we could also consider building a similar app using a native framework (e.g., Swift for iOS), as opposed to the current hybrid HTML5 framework (e.g., Ember JS), which could improve the responsiveness of a finalized version of the mobile app (see example trial in Granek et al., 2019).

Participants also proposed a series of minor enhancements, all of which should be considered for feasibility and trialed with CAF members as part of an iterative design process (Hall, 2013). Several of the proposed enhancements were to provide access to offline resources such as maps and audio files (e.g., national anthem). These resources could be added on an ad-hoc basis using the built in CMS, configured for each mission. The addition of the maps and other resources would provide the users with supplementary information and further encourage them to rely on the app for guidance. Additional suggested enhancements could also be used to enhance the usability of the CAT app as a quick reference guide. For the LL module, participants suggested that a preview of the information from different LL would be useful. This could potentially be implemented using machine learning algorithms in order to autofill content based on keyword searches and should be trialed further to ensure that this enhancement would increase the utility of this module. Further, participants suggested that the Collaboration Tools module could also offer additional functionality with a more streamlined process. The CAT app prototype could be expanded to host additional background information and simple tools (e.g., Mission Report Template) and the evaluation of their utility could be provided by explicit prompts to use them in a similar follow-up evaluation exercise and scenario.

Participants also suggested additional in-depth information in the RBM vs. OPP and Religious Leader Engagement modules. Specifically, participants suggested additional in-depth information when viewing the differences in planning process for military and NGOs, which would apply to the RBM vs. OPP module. Currently the module has been developed as a quick reference guide for CAF members with

DRDC-RDDC-2019-R217 27

previous experience working with NGOs, but additional information could be added for it to be useful as a more in-depth training tool for less experienced IA Analysts. Further discussions on the extent of the information will be needed to ensure that the correct amount of information is provided for different contexts, without deterring users from engaging with the module. We propose evaluation during a more detailed scenario whereby the IA Analysts are required to interact more with NGOs in a roundtable setting could provide more insight into how the CAT app can facilitate the use of acronyms, discussion of the planning processes, etc. Further, participants recommended to expand the current Religious Leader Engagement module to include more general information on engaging with all key leaders (e.g., Mayor), including religious leaders (Key Leader Engagement module). Finally, in order to allow for ongoing user feedback and to ensure that the information provided within the CAT app remains current, we recommend to incorporate a crowdsourcing ability whereby users can provide suggested updates and additions (e.g., new Lessons Learned), which can be synced to a central database and be disseminated to all users. 4.1.2 CAT App Usage Analytics

The usage analytics revealed different usage patterns depending on the role of the individual within the humanitarian scenario. Specifically, for the leader of the IA Analyst team (Sgt), the tool served to support looking up information (e.g., Acronyms & Terminology), and teaching team members (e.g., RBM vs. OPP), and as a data collection tool (e.g., Collaboration Tools). The CAT app also served to be useful to look up information such as acronyms and planning processes for other members of an IA Analyst team at the request of the team leader.

Although all modules were included in the CAT app as a result of SME input (e.g., Holton et al., 2016) and iterative design process, the usage analytics revealed that some of the modules were not accessed during the humanitarian scenario. For example, although we provided implicit prompts (Table 2) alluding to previous LL in Vignette 1 (see Holton et al., 2019), participants did not interact with the LL module during the humanitarian scenario trial to gather additional information based on previous lessons learned. As such, based on participant feedback, we recommend that the copy (i.e., language) of the instructions for the different modules should be updated to ensure that the user understands the purpose (i.e., “so-what factor”) of each of the modules. 4.1.3 Future Requirements for the CAT App

An analysis of the internet searching history of the participants, as well as a review of the content of the emails, sent to and received by the participants, revealed potentially critical missing content that could be included in future versions of the CAT app. Following discussions with SMEs, the first the inclusion of critical information related to disaster assistance and emergency response efforts should be considered. First, after confirmation with SMEs, we recommend that this information could be inputted using the CMS in future versions of the CAT app when preparing for a disaster assistance mission. The new information could include a new module specific to disaster assistance relief that could include detailed information about DART, different types of disasters (e.g., flood, earthquake), as well as details pertaining to the specific relief efforts required for each type of disaster (e.g., how to prevent the spread of disease of waterborne and airborne illnesses, how to purify water, emergency medical care set up). Second, the CAT app could benefit from the inclusion of additional content in each of the existing modules. For example, the country guide could be strengthened by a) the addition of more maps, b) the addition of slang terms, greetings, and other common terms specific to the country, and c) the inclusion of information about bilateral relations between Canada and the country of interest. An expanded list of

28 DRDC-RDDC-2019-R217

acronyms, more elaborate definitions of acronyms, and the addition of acronyms and terms relevant to the intelligence process per se would strengthen the acronyms and terminology database. Future versions of the CAT app could also include additional forms, templates, handbooks, and material related to doctrine, tasks, and procedures, relevant to both the intelligence and IA process. The modular nature of the CAT app will allow for mission-specific information to be updated quickly through the built-in CMS. 4.2 Limitations of the Current Study and Future Directions

Although our study provided a useful approach and assessment of the usability and utility of the CAT app in a CAF-relevant humanitarian scenario, we cannot quantify the magnitude of the effectiveness and impact of the tool. Further research is required to directly compare the CAT app in a similar scenario with traditional (e.g., paper and pencil) materials and equipment (i.e., control group). In addition, we propose that having additional vignettes in the exercise (e.g., roundtable, meeting key leaders) that are developed with the intent to implicitly and explicitly prompt the use of the CAT app would enrich the data and allow for a more detailed analysis of user behaviours and preferences. We suggest that the exercise should employ an internet-denied setting and restrict the use of other tools for the experimental conditions. At the end of the exercise, we propose that the participants could then return to the headquarters setting with network access and share the information gathered locally on the device throughout the exercise in order to generate the final report. This would further highlight the utility of the CAT app as a potentially important resource for information and for data collection.

Moreover, although we provided training to all participants on Day 1 of the trial, previous evidence suggests the requirement to integrate mobile app training and promotion into training doctrine in order to reinforce familiarity with the tool and requirement for use in relevant context (e.g., Granek, Martin, Jarmasz & Bae, 2017). Further, in order to encourage adaptation of the CAT app in such an exercise, we propose that the IPOE received on Day 1 be updated to include some of the preliminary planning processes that the participants mapped out on the white board (Figure 8) and have a copy of the IPOE displayed within the app in order to provide all resources in one place and encourage awareness of the app as a wide-reaching resource. Similarly, all instructions, information injects and meeting locations could be included within the app, and delivered as, say, calendar notifications in order to optimize its integration with other enablers available to CAF members during operations. 4.3 Conclusions

Overall, we found reasonable initial support for the usability and the utility of the CAT app in a humanitarian scenario exercise for IA Analysts. The CAT app was predominantly perceived as user-friendly and was considered to be optimal as an on-the-go tool for generating mission reports for the participants’ chain of command and preparing for face-to-face interactions with potential key leaders and other players within the CA. Feedback from the IA Analysts revealed that although the current version of the CAT app was less effective as a training and knowledge transfer tool in a headquarters setting that has access to other tools such as personal computers, it would be useful as an offline resource for access to searchable databases and for collection of pertinent information required in their reports. Further, participants in the study offered constructive suggestions for additional information that could be added to the database, as well as additional tools in order to effectively complement their job requirements as part of a disaster relief team. We propose that future research should be conducted to directly compare the CAT app relative to a control (e.g., paper and pencil) in order to validate the magnitude of impact the CAT app can have as a training tool and as an aide-memoire and data gathering device in order to complement the job

DRDC-RDDC-2019-R217 29

requirements for IA Analysts in the theatre of operations. Further, we suggest that future evaluations should also be conducted with SMEs to decipher the optimal amount and depth of information as well as the tools required for improved effectiveness in the field, which can be configured to different types of users. These future analyses would help highlight and reinforce the justification for implementing a complementary tool into training and doctrine for all players within the CA setting.

30 DRDC-RDDC-2019-R217

References

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. Intl. Journal of Human–Computer Interaction, 24(6), 574–594.

Brooke, J. (1996). SUS-A quick and dirty usability scale. Usability evaluation in industry, 189(194), 4–7.

Denham, A. R., Gonzalez-Sanchez, J., Chavez-Echeagaray, M. E., & Atkinson, R. K. (2012). Mobile applications as tools to support embodied learning: Current practice and future directions. International Journal of Cyber Behavior, Psychology and Learning, 2(4), 1–16.

Duncan, M., Eaton, D., Hendriks, T., Keefe, A., McLellan, T., Michas, R. D., & Thompson, M. M. (2008). DRDC Toronto guidelines for compensation of subjects participating in research studies. Technical Memorandum, Defence R&D Canada – Toronto, DRDC Toronto TM 2008–138.

Hall, E. (2013). Just Enough Research. A Book Apart, LLC.

Hendriks, T., & Kwantes, P. (2016). Collaborative Sensemaking in a Simulated Humanitarian Disaster Scenario. The Technical Co-Operation Panel Report Number: TR-C3I-2-2016.

Holton, T. L., Thompson, M., Febbraro, A., Hendriks, T., & Gill, R. (2016). Comprehensive Approach Training Toolkit: Closeout Report. Scientific Letter, Defence Research and Development Canada, DRDC-RDDC-2016-L027.

Holton, T.,* Granek, J.,* Kozey, W., Bae, R., Hendriks, T., Newton, J., Townsend, J., Fraser, B., & Kwantes, P. (2019). Creating a Lab-Based Humanitarian Exercise. Reference Document, Defence Research and Development Canada, DRDC-RDDC-2018-D157. *these authors contributed equally to this work.

Hoober, S., & Berkman, E. (2011). Designing Mobile Interfaces: Patterns for Interaction Design. Sebastopol, CA: O’Reilly Media.

Granek, J. A., Nazarov, A., D’Agata, M. T., Bae, J., Boland, H., Kirollos, R., Guest, K., & Bailey, S. (2019). Leveraging technologies to complement the Road to Mental Readiness (R2MR) training program, E19-0318-01735, Interim Approval. Paper presented at NATO HFM-302 Symposium, Berlin, Germany.

Granek, J. A., Martin, B., Jarmasz, J., & Bae, J. (2017). Implications of Integrating Mobile Learning within the Canadian Forces Military Police Academy (CFMPA) Training Program (Unclassified) to LCol Battista, Commandant CFMPA, CFB Borden. Scientific Letter, Defence Research and Development Canada, DRDC-RDDC-2017-L127.

Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen, and R. L. Mack (Eds.), Usability Inspection Methods. New York: John Wiley & Sons.

Rosas, E., Hidalgo, N., Gil-Costa, V., Bonacic, C., Marin, M., Senger, H., ... & Marin, O. (2016). Survey on simulation for mobile ad-hoc communication for disaster scenarios. Journal of Computer Science and Technology, 31(2), 326–349.

DRDC-RDDC-2019-R217 31

Tognazzini, B. (2003). First Principles of Interaction Design (Revised & Expanded). Retrieved 13 June, 2016, from http://asktog.com/atc/principles-of-interaction-design/.

Wayne, N., Perez, D. F., Kaplan, D. M., & Ritvo, P. (2015). Health Coaching Reduces HbA1c in Type 2 Diabetic Patients From a Lower-Socioeconomic Status Community: A Randomized Controlled Trial. Journal of Medical Internet Research, 17(10), e224.

United States Department of Defense. (2013). Joint Intelligence Publication 2-0. Retrieved 27 August, 2019 from http://www.dtic.mil/doctrine/new_pubs/jp2_0.pdf.

32 DRDC-RDDC-2019-R217

Annex A CAT App Questionnaire

1. Module name (please circle): To be filled out for each component A) Acronym and Terminology B) Scenario C) Lessons Learned D) RLE E) Training and Education F) RBM and OPP G) Briefing Tool H) Latvia

Part I: Please respond to the following statements by selecting the response that most closely matches your opinion Strongly Neither Agree Strongly Disagree Nor Disagree Agree 1. The app provided me with important knowledge or skills I need to conduct my mission duties. 1 2 3 4 5

2. If I did not have access to the app while I practiced my performed my duties, I would notice 1 2 3 4 5 its absence.

3. The app gave me access to information that would have been otherwise unavailable or difficult 1 2 3 4 5 to access at a critical moment.

4. The app is not well suited for CAF personnel and missions. 1 2 3 4 5

5. I would use this app before a deployment. 1 2 3 4 5

6. I found the app less and less relevant to WoG interactions as time went on. 1 2 3 4 5

7. The app helped me remember or provided a refresher on knowledge or skills I learned in 1 2 3 4 5 previous training.

8. The app was not easy to navigate. 1 2 3 4 5

9. I learned knowledge or skills from this app that I would be able to apply without consulting the app. 1 2 3 4 5 10. Using this app had no effect on how I would

DRDC-RDDC-2019-R217 33

perform my other duties. 1 2 3 4 5

Part II: Please answer the following questions with a few brief sentences:

A. Thinking about an app that you find easy to use, what would you change to make the CAT App more user friendly?

B. Thinking back on the app you just explored, what feature or experience stood out the most? Did it stand out in a good way or bad way? Why?

Thank you!

34 DRDC-RDDC-2019-R217

List of symbols/abbreviations/acronyms/initialisms

App Application ASCOPE Areas, Structures, Capabilities, Organizations, People and Events CA Comprehensive Approach CAF Canadian Armed Forces CAT Comprehensive Approach Toolkit CCIR Commander’s Critical Information Requirements CIA Central Intelligence Agency CIMIC Civil Military Cooperation CMS Content Management System CO Commanding Officer COS Chief of Staff C2L2 Command and Control Livespace Laboratory DART Disaster Assistance Response Team DJ Disk Jockey DND Department of National Defence DRDC Defence Research Development Canada EEFI Essential Elements of Friendly Information Ex Exercise FFIR Friendly Force Information Requirements GAC Global Affairs Canada GDP Gross Domestic Product HoM Head of Mission HQ Headquarters HuE Human Environment IA Influence Activities ICRC International Committee of the Red Cross IO International Organizations IPOE Intelligence Preparation of the Operational Environment JIMP Joint, Interagency, Multinational and Public JS JavaScript

DRDC-RDDC-2019-R217 35

LL Lessons Learned m-learning Mobile Learning N/A Not Applicable NGO Non-Governmental Organizations OGD Other Government Departments OP Operation OPP Operational Planning Process PIR Priority Intelligence Requirement PMESII Political, Military, Economic, Social, Infrastructure and Information PPEAT Planning, Preparation, Execution and Assessment Tool PSYOPS Psychological Operations PIR Priority Intelligence Requirement PMF Performance Measurement Framework R&D Research & Development RBM Results Based Management Process RFI Request for Information RLE Religious Leader Experience Sgt Sergeant SME Subject Matter Expert SUS Standard Usability Scale VTC Video Teleconferencing WoG Whole of Government

36 DRDC-RDDC-2019-R217

DOCUMENT CONTROL DATA *Security markings for the title, authors, abstract and keywords must be entered when the document is sensitive 1. ORIGINATOR (Name and address of the organization preparing the document. 2a. SECURITY MARKING A DRDC Centre sponsoring a contractor's report, or tasking agency, is entered (Overall security marking of the document including in Section 8.) special supplemental markings if applicable.)

DRDC – Toronto Research Centre CAN UNCLASSIFIED Defence Research and Development Canada 1133 Sheppard Avenue West Toronto, Ontario M3K 2C9 2b. CONTROLLED GOODS Canada NON-CONTROLLED GOODS DMC A

3. TITLE (The document title and sub-title as indicated on the title page.)

Validation of the Comprehensive Approach Toolkit Mobile Application in a Lab-Based Humanitarian Exercise

4. AUTHORS (Last name, followed by initials – ranks, titles, etc., not to be used)

Granek, J. A.; Holton, T.; Hendriks, T.; Kozey, W.; Apostoli, C.; Bae, R.

5. DATE OF PUBLICATION 6a. NO. OF PAGES 6b. NO. OF REFS (Month and year of publication of document.) (Total pages, including (Total references cited.) Annexes, excluding DCD, covering and verso pages.) April 2020 45 16

7. DOCUMENT CATEGORY (e.g., Scientific Report, Contract Report, Scientific Letter.)

Scientific Report

8. SPONSORING CENTRE (The name and address of the department project office or laboratory sponsoring the research and development.)

DRDC – Toronto Research Centre Defence Research and Development Canada 1133 Sheppard Avenue West Toronto, Ontario M3K 2C9 Canada

9a. PROJECT OR GRANT NO. (If appropriate, the applicable 9b. CONTRACT NO. (If appropriate, the applicable number under research and development project or grant number under which which the document was written.) the document was written. Please specify whether project or grant.)

04g

10a. DRDC PUBLICATION NUMBER (The official document number 10b. OTHER DOCUMENT NO(s). (Any other numbers which may be by which the document is identified by the originating assigned this document either by the originator or by the sponsor.) activity. This number must be unique to this document.)

DRDC-RDDC-2019-R217

11a. FUTURE DISTRIBUTION WITHIN CANADA (Approval for further dissemination of the document. Security classification must also be considered.)

Public release

11b. FUTURE DISTRIBUTION OUTSIDE CANADA (Approval for further dissemination of the document. Security classification must also be considered.)

12. KEYWORDS, DESCRIPTORS or IDENTIFIERS (Use semi-colon as a delimiter.)

Comprehensive Approach to Operations/Whole of Government Missions; Mobile App; Influence activities; Operational Aid

13. ABSTRACT (When available in the document, the French version of the abstract must be included here.)

As a result of a recent requirements analysis, researchers at Defence Research and Development Canada (DRDC) – Toronto Research Centre have developed a new tool to assist members of the Canadian Armed Forces (CAF) who work in the area of influence activities (IA) with their daily duties. The tool, named the Comprehensive Approach Toolkit (CAT) mobile application (app), was created to address the need for a deployable suite to tools to enhance collaboration in comprehensive approach (CA) settings. The purpose of this study was to gain information pertaining to the CAF IA information operation process by observing IA Analysts use the CAT app during a scenario exercise that was held in the Command and Control Livespace Laboratory (C2L2) at DRDC – Toronto Research Centre—a collaborative workspace equipped with networked computers, collaborative software, and VTC capability. Observational data, information operation output and focus group data were used to: a) identify strategies and resources analysts use to prepare an information operation plan and gather information required by the commander, b) determine how analysts collaborate and create shared situational understanding, c) determine how IA Analysts use the CAT app, and d) validate the CAT app for its usefulness and usability in the conduct of IA information operations. The study ran over the course of one and a half days for a group of three IA Analysts; the first day included a half day of training on the CAT app while the second day included a full day disaster relief exercise. Specifically, IA Analysts used the CAT app during a disaster relief scenario in the C2L2 to conduct their activities. Following the exercise, the participants generated a final report. Data was gathered in the form of observational data, questionnaires regarding app use, tracking of app usage (e.g., number of clicks on a page, time spent on a page), and written and/or oral feedback at the end of the study. Qualitative data analysis was conducted to assess the extent to which the participants found the CAT app useful when completing their tasks. Results indicated that two out of the three IA Analysts regularly interacted with the CAT app during the humanitarian exercise with relative ease after minimal training. The CAT app was found to be user-friendly and was well suited for generating mission reports and preparing for face-to-face interactions. Participants found the tool less relevant as a knowledge transfer tool in a headquarters setting with access to other tools such as a personal computer. Further, participants suggested that additional information, access to additional links (e.g., maps), as well as providing additional templates for interacting with key leaders and preparing planning processes would render the CAT app as an even more useful tool. Future research should be done to compare the CAT app relative to a control in order to validate and quantify the magnitude of the added value of adopting the use of the CAT app as part of doctrine for both training and in the theatre of operations.

Aux termes d’une récente analyse des besoins, les chercheurs du Centre de recherche Toronto de Recherche et développement pour la défense Canada (RDDC) ont mis au point un nouvel outil pour aider les membres des Forces armées canadiennes (FAC) du domaine des activités d’influence (AI) dans leurs tâches quotidiennes. Cet outil, baptisé Trousse de l’approche globale (TAG), est une application mobile conçue pour répondre au besoin d’une suite d’outils portative afin d’améliorer la collaboration dans un contexte d’approche globale. La présente étude avait pour objectif de recueillir des données sur le processus lié aux opérations d’information du domaine des AI au sein des FAC. À cette fin, des chercheurs du Centre de recherche Toronto ont observé des analystes du domaine des AI se servir de l’application TAG au cours d’une simulation réalisée dans le Laboratoire de commandement et de contrôle LiveSpace (C2L2), un espace de travail collaboratif où l’on retrouve un système de vidéoconférence et des ordinateurs en réseau sur lesquels des logiciels de collaboration ont été installés. Les données, tirées de l’observation, des groupes de discussion et des résultats des opérations d’information, ont permis (a) de cibler les stratégies et les ressources qu’utilisent les analystes pour préparer leur plan d’opérations d’information et colliger les renseignements demandés par le commandant, (b) de déterminer la façon dont ils collaborent pour établir une même compréhension de la situation, (c) de cerner la façon dont ils ont utilisé l’application TAG

et (d) de vérifier l’utilité et la convivialité de celle-ci dans le cadre d’opérations d’information relatives aux AI. L’étude, à laquelle trois analystes du domaine des AI ont participé, s’est déroulée sur une période d’une journée et demie, soit une formation d’une demi-journée sur l’application TAG, suivie d’une journée complète consacrée à un exercice de secours aux sinistrés. Plus précisément, les analystes ont utilisé l’application TAG pour effectuer leurs tâches dans le cadre d’une simulation tenue au laboratoire C2L2 et produire ensuite un rapport final. Des données ont été recueillies par le truchement d’observations, de questionnaires sur l’application, du suivi de son utilisation (nombre de clics sur un page, temps passé sur une page, etc.) et de commentaires formulés par écrit ou de vive voix après l’étude. Elles ont fait l’objet d’une analyse qualitative pour évaluer, du point de vue des participants, le degré de pertinence de l’application TAG dans la réalisation de leurs tâches. Les résultats de l’étude démontrent que deux des trois analystes ont utilisé régulièrement l’application TAG durant la simulation d’opérations d’aide humanitaire, et l’ont fait avec une certaine aisance malgré leur formation sommaire. Les participants ont reconnu que l’application était conviviale et adaptée à la production de rapports de mission et aux préparatifs en vue de rencontres en personne. Ils l’ont trouvé toutefois moins utile à titre de mécanisme de transfert de connaissances dans un milieu d’état-major où l’on a accès à d’autres outils, par exemple un ordinateur personnel. De plus, les analystes ont avancé que l’application gagnerait en utilité si elle renseignait davantage et donnait accès à plus de liens (p. ex., des cartes), tout en proposant plus de modèles pour interagir avec des dirigeants principaux et élaborer des processus de planification. D’autres recherches seront nécessaires pour comparer l’application TAG à un témoin afin d’en valider et d’en mesurer la valeur ajoutée si son adoption est prévue dans la doctrine, à la fois sur le plan de l’instruction et du théâtre d’opérations.