User-Centered Design Guidelines for Collaborative Software for Intelligence Analysis Jean Scholtz and Alex Endert Visual Analytics, National Security Division Pacific Northwest National Laboratory Richland, WA USA [email protected], [email protected] Abstract—In this position paper we discuss the necessity of government interested in the same information. using User-Centered Design (UCD) methods in order to design Everybody wants their organic intelligence elements collaborative software for the intelligence community. We discuss to replicate the same analysis many times over [1].” a number of studies of collaboration in the intelligence community and use this information to provide some guidelines Is adding more intelligence agencies into the mix and using for collaboration software. social media and the wisdom of the crowd a sensible way to Index Terms—Intelligence community, collaboration, user- approach this? The problem is that while it is estimated that centered design, metrics, evaluation over 80% of the relevant information is NOT classified, this varies widely by subject. “Game-changing” information in critical areas is usually small and the most problematic to I. INTRODUCTION locate and verify [1]. This information may also be on the Collaboration in todays’ intelligence community (IC) borderline between agency responsibilities and access and have presents some potential benefits and many difficult challenges. a very limited, sensitive distribution. Thus, collaboration is one Installing off the shelf collaborative software, in many approach to aiding in this challenge. instances, only serves to exacerbate problems given the specific requirements and structure by which the IC operates. In order to design software that overcomes many of the However, this domain exhibits characteristics that lend obstacles in collaboration, it is necessary to look at current themselves well (or even require) collaboration from many practices and workflows and produce guidelines for software users across many domains. Therefore, understanding the that preserves or enhances current facilitators while reducing challenges specific to this domain through a user-centered obstacles. It is also essential to realize that insertion and design methodology can enhance the community’s adoption of any collaborative software will eventually change understanding of how to design and build such capabilities. the workflow in an organization. Therefore the collaborative software needs to be flexible enough that analysts are able to Todays’ intelligence community has an incredible task as manage the change in a non-obtrusive fashion. our world is the most complicated in history due to globalization and the need to cooperate with many foreign Additionally it is essential to measure the impact of the countries, although under severe constraints at times. collaboration software (i.e., evaluating the effectiveness). The Challenges include: number of agencies doing intelligence analysis of some sort is in the double digits and their tasks and work practices are • “Our ongoing 20 year struggle with information extremely varied from quick turn- around asks to long, overload has been further complicated by the complex strategic analysis. Collaboration efforts may focus on challenges of dealing with foreign and domestic teams within an organization, organizational collaboration, or information. between agency collaboration. Therefore, how can various • Virtually all subjects of any significance are a agencies decide which collaboration tools would even be complicated mosaic of social, economic, political, feasible to try in their organization and how can they measure the success? Analysts must perceive a benefit in using the tool military, and technical components. in order to continue (or begin) using it. • Individual bad actors have it within their grasp to take actions that can have strategic consequences. In this paper, we first present lessons learned from studies Barriers to entry are low and, in effect, these of analysts and collaborative software tools. We use these individuals use globalization against us. studies to derive some high level guidelines for collaboration • And as the roles and responsibilities of our Executive software within the IC. We also present methods of measuring Branch Departments blur, we find seniors across the the impacts of collaborative software. Finally we present 978-1-4799-5158-1/14/$31.00 ©2014 IEEE 478 features for future collaborative software that can be investigated. II. LESSONS LEARNED FROM PRIOR WORK The lessons detailed in this portion of the paper are presented as results from prior studies, observations, and interviews with analysts. We organized them in terms of the lessons learned and then present the evidence along with the references to the studies. A. Collaboration Through Social Media Technologies A report by McKinsey Global Institute claims that business employees spend 28 hours per week answering, Figure 1. A pair of users collaborating on a writing or responding to email, tracking down information, and large, high-resolution display. collaborating with co-workers [2]. In addition, social media is becoming an acceptable (and at times preferred) method of collaborating. A 2013 Pew Research study showed that 72% of adults use social media, with 89% of those aged 18 – 29 and The idea of cooperation may be more acceptable in many 78% of those aged 30 – 49 [3]. As we replace the aging analyst agencies [4]. The difference between collaboration and population with younger analysts, there will be an expectation cooperation is subtle. Both rely on technology tools but in for the same computing resources and communication tools at cooperation each analyst would do her own work but would work as they have at home. share with other analysts the tools and other resources available. The IC started their Intellipedia effort in 2005. Based on the success of Wikipedia for the general public, this was an C. Merging Shared or Opposing Perspectives effort designed to allow individual analysts to create pages to share information within the IC. As of 2009 Intellipedia had When collaborating, each collaborator stems their insights some 100,000 user accounts [4]. In the IC not all users should regarding a topic from a potentially different domain have access to all content. While sometimes just knowing that experience and dataset. For example, this issue is illuminated the information is there is useful, collaborative tools need to within Intellipedia as a duplication of content, or more take into account security issues in accessing content. generally on Wikis when content is changed or overwritten by Currently three versions of Intellipedia exist based on the a group of authors with opposing opinions [cite, I think there’s classification of the information. As many agencies do not use some work in that area]. Intellipedia as an official conduit for information, analysts In co-located settings, prior studies found that roles emerge move content from Intellipedia to an official data repository in organically in collaborative analytic tasks performed their agency for use. This only increases the analysts’ work as synchronously on large displays [8], [9]. They found that a information is now duplicated on many sites. Analysts are difficulty in such organic emergences of roles is coming to a often apprehensive about contributing information on common perspective that both collaborators could agree upon. Intellipedia as this reduces the control they have over how However, the ability to reflect on these perspectives using a information is used. common, shared externalization of their process (a common spatial organization) helped illuminate moments of B. Synchronous vs. Asynchronous Approaches disagreement between the collaborators. Figure 1 shows a pair Understanding the IC is critical in understanding whether of users discussing a point of contention regarding the to design for synchronous or asynchronous collaboration tasks. placement of a specific document. Fink et al. found that For example, Kang and Stasko observed students at Mercy analysts collaborating on such a display can also use “physical Hurst College in intelligence courses. They found that teams brushing and linking”, where one user points to an area of a of students did collaborate but not so heavily on content as on display, while the other points to another to show the status. They termed this loose collaboration [5], [6]. Similarly, connection [10]. Scholtz and Steves observed analysts at the United States Army Intelligence and Security Command Center (INSCOM) Collaboration around a tabletop display can produce a working on RFI taskings using the chat facility of a similar effect. Isenberg et al. showed how co-located collaborative tool more than setting up shared spaces for collaboration using a tabletop can show implicit signs of collaborative work – which was the intended collaborative information ownership based on the rotation of the information functionality of the tool [6]. They had other mechanisms for [11]. Further, their work discusses how collaboration can be sharing content and as their tasks were often very quick turn- measured by the sharing of information both within the around, the overhead of setting up a shared space for working visualization as well as socially between collaborators. on the task was extremely burdensome. Additionally, Nolan found in her interviews of National Center for Counter D. Pull versus Push Terrorism (NCTC)