First Annual Conference on Intelligence Analysis Methods and Tools, May 2005 PNNL-SA-44274 Top Ten Needs for Intelligence Analysis Tool Development Richard V. Badalamente and Frank L. Greitzer Battelle—Pacific Northwest Division P.O. Box 999 Richland, WA, 99352, USA [email protected] [email protected] Keywords: Fusion, Information Sharing/Collaboration, Usability/Habitability. Abstract sure that research and development products provide ef- fective solutions to critical problems. A workshop was held to generate ideas about future enhancements to software systems de- signed to aid intelligence analysts in the analy- The Pacific Northwest National Laboratory (PNNL) has sis process. Workshop participants were work- for some time been involved in both the development of ing analysts, working members of the Pacific software tools meant to aid in intelligence analysis (IA), Northwest National Laboratory Field Intelli- and in actually performing intelligence analysis as part of gence Element. An analyst from the Counter- the Department of Energy’s Field Intelligence Element Intelligence Program also participated in the (FIE). In an attempt to better inform the research and de- workshop. This paper summarizes the results velopment (R&D) of IA tools, we conducted a workshop of the workshop and discusses implications for with intelligence analysts to generate ideas about how to intelligence analysis software tool develop- enhance software systems designed to aid in the analysis ment. process. The purpose of this paper is to report on the results of that workshop and to discuss implications for future IA tool development and/or enhancement. 1. Introduction and Background Intelligence analysis (IA) professionals are faced with 2. Approach daily challenges to meet high demands for rapid, accurate We conducted a one-day workshop with analysts em- assessments that require discovery and marshalling of ployed at PNNL. Most of the analysts worked in the evidence, integration and synthesis of data from disparate PNNL FIE and, for the most part, performed a class of sources, interpreting and evaluating information that is analysis known as scientific and technical intelligence. constantly changing, and providing documentation and One of the analysts worked in the PNNL counterintelli- recommendations (intelligence products) that meet the gence (CI) program and conducted CI and counter terror- customer’s reporting requirements. Ongoing research ism (CT) analysis. efforts seek technology-based solutions to reduce the ana- lyst’s workload and improve the throughput and quality of The workshop, which was conducted in July 2002, started IA products. Some of the motivation for tool development at 8:00 in the morning and lasted 5 and 1/2 hours. A fa- is based on informal “corporate knowledge and experi- cilitator guided workshop proceedings. Two human fac- ence.” For example, a particularly influential book by tors specialists participated as catalysts and scribes. Nine Heuer (1999) provides such a perspective, informed by working analysts participated in the workshop, eight from principles of psychology and human information process- the PNNL FIE, and one from the PNNL CI Program. ing. There is a limited amount of empirical research to guide tool development (e.g., Patterson, Roth, and The workshop commenced with a presentation of an Intel- Woods, 2001), and there is a growing body of work based ligence Analysis Process Model developed as a part of a on cognitive engineering efforts such as interviews, ob- PNNL internal R&D Project (Figure 1). This multistage servations, and cognitive task analysis (e.g., Patterson, process model was developed from interactions we had Woods, Tinapple, and Roth, 2001; Hutchins, Pirolli, and from practicing intelligence analysts in open source op- Card, in press). There is a continuing need to organize, erations as well as from the analysts working in the PNNL motivate, and better define requirements in order to en- FIE. Determine Frame the Develop Analysis Identify Info Information Problem Strategy/Approach Sources Needs Problem/ Threat Access Info Sources Intelligence Tasking Evaluate Retrieval Information Product Organize & Develop Intelligence Retrieve Information Store Product & Perform Analysis Information Figure 1. Intelligence Analysis Process Model used to provide a background and common reference point to the analysts participating in the workshop. The analysts involved in our workshop perform functions 3. Results in all the steps illustrated in Figure 1. This diagram there- The two groups approached their tasks very differently. fore reflects the process from the perspective of the ana- Group 1 followed the steps in the model and attempted to lysts we interviewed. It also served our research purposes identify enhancements related to each step. In this sense, well as a hypothetical construct to guide our attempts at they took a systems or programmatic approach. Group 2 capturing data from analysts across the full spectrum of focused immediately on data processing facets of the in- their analysis functions. telligence analysis task and the kinds of automation sup- port they felt would aid them in their job. Despite this, Our discussions with analysts reinforced our own experi- there was a great deal of commonality in the enhance- ence with regard to the task of describing and understand- ments identified. The top ten enhancements identified ing a process that is essentially cognitive in nature. That during the workshop are described below and listed in is, such processes are difficult to describe in words or Table 1, in the priority order decided upon by the two diagrams. The analyst’s thought processes as described to groups (highest priority first). us are much more fluid, fuzzy, and opaque (even to the analysts) then our diagram would imply. We have at- tempted to illustrate the iterative and decidedly nonlinear Seamless Data Access and Ingest [#1] nature of the process with the dotted lines from “Evaluate Analysts want the ability to access data seamlessly across Retrieved Information” to “Frame the Problem,” “De- multiple databases and bring (ingest) the merged data into velop Analysis Strategy/Approach,” and “Determine In- whatever analysis tool (software application) they may be formation Needs,” as well as the feedback loop embedded using. The term “seamless” refers to the ability of the user in the overall process. However, even this diagrammatic to move from one database to another without logging in feature is inadequate to fully convey the fuzzy or non- and out of the multiple databases, or having to learn dif- sequential nature of the process. ferent query/retrieval routines and different graphical user interfaces (GUIs). The data ingest process needs to be After the presentation, participants broke out into two transparent to the analyst/user. Invariably, software ap- groups with instructions to use the model as a point of plications require that data be in a particular format before departure for identifying future software system develop- it can be processed by the application—e.g., Starlight (a ments/enhancements that would aid analysts in doing visualization tool developed by PNNL) requires that text their job. The groups were given 2 hours to do this. Upon be converted from ACSI to XML. Whatever data conver- completion, they reassembled and each group briefed sions are required in order for an analyst to employ data their findings. The participants then voted on the top ten of interest need to be accomplished in the background developments and/or enhancements they would like to see (much like the printer driver and buffer work) and not occur. require analyst intervention – the process needs to be “transparent” to the analyst. First Annual Conference on Intelligence Analysis Methods and Tools, May 2005 PNNL-SA-44274 Template for Analysis Strategy [#5] Analysts want a software tool that coaches them in the Table 1. Top Ten System Enhancements development of a strategy or plan of attack for conducting # Enhancement more complex, longer-term analyses. Such a tool might provide a template that covers the various facets of analy- 1 Seamless Data Access and Ingest sis and provides automated linkages to potential methods 2 Diverse Data Ingest and Fusion and sources. 3 Shared Electronic Folders for Collabora- Electronic Skills Inventory [#6] tive Analysis Often, there is no easy way for analysts to identify staff members who have the skills, experience, and clearances 4 Hypothesis Generation & Tracking needed to assist in the analysis task. Access to some form 5 Template for Analysis Strategy of electronic skills inventory would aid the analyst in quickly identifying staff and assembling an analysis task 6 Electronic Skills Inventory team. This sort of tool would be a critical component of an effective collaboration environment. 7 Dynamic Data Processing and Visualiza- tion Dynamic Data Processing and Visualization [#7] 8 Intelligent Tutor for Intelligence Product Analysts want the ability to deal effectively with data Development streams as opposed to static data sets. As analysts develop 9 Imagery Data Resources ways to better comprehend the data space through user- controlled visualizations, they want their visualizations 10 Intelligence Analysis Knowledge Base automatically refreshed with new data on a schedule dic- tated by the analyst. Diverse Data Ingest and Fusion [#2] Intelligent Tutor for Intelligence Product Develop- Analysts want the ability to access and work effectively