Defense Intelligence Analysis in the Age of Big Data
Total Page:16
File Type:pdf, Size:1020Kb
Crew chief with 36th Aircraft Maintenance Unit, Osan Air Base, South Korea, checks computer data during Red Flag-Alaska 14-2, ensuring F-16 Fighting Falcon readiness (U.S. Air Force/Peter Reft) Defense Intelligence Analysis in the Age of Big Data By Paul B. Symon and Arzan Tarapore ver the past decade, the U.S. their greatest challenge will be to ing, and the “Internet of things”—and and Australian intelligence com- develop new capabilities to manage the advanced analytics used to process O munities have evolved rapidly and exploit big data.1 We use the term that data. Big data yields not simply a to perform new missions. They have big data to mean the exponentially quantitative increase in information, developed new capabilities and adapted increasing amount of digital informa- but a qualitative change in how we their business processes, especially in tion being created by new information create new knowledge and understand support of joint and complex military technologies (IT)—such as mobile the world. These data-related informa- operations. But in the coming decade, Internet, cloud storage, social network- tion technologies have already begun to revolutionize commerce and science, transforming the economy and acting as enablers for other game-changing Major General Paul B. Symon, Australian Army, served as Director of the Defence Intelligence technology trends, from next-genera- Organisation from 2011 to 2014. Arzan Tarapore is a Doctoral Student in the War Studies Department 2 at King’s College London. The authors thank Josh Kerbel and Peter Mattis for comments on an earlier tion genomics to energy exploration. version of this article. In defense intelligence communities, 4 Forum / Defense Intelligence and Big Data JFQ 79, 4th Quarter 2015 some of these technologies have been ing tools such as Wikis and Chat are big data are both enabling and driving adopted for tasks, including technical already being used to facilitate better the creation of IT solutions that are cus- collection and operational intelligence collaboration between analysts. Beyond tomized and intuitive for the user. Gone fusion—but big data’s impact on all- simple software acquisitions, however, are the days of hefty user manuals or ob- source intelligence analysis has scarcely disruptive information technologies scure text-based user interfaces. Specific been examined. have birthed a number of trends in how applications perform specific functions. This article offers a view on how these data are collected, moved, stored, and Even major platforms such as Palantir are disruptive information technologies could organized. Four of the most salient delivered with bespoke service support, transform defense intelligence analysis and prevailing concepts, which are already both in tailoring the product to customer the functions of the all-source enterprise. transforming the economy and society, requirements and in providing ongoing It is not a comprehensive study on trends could reshape all-source intelligence. software development support. Complex in technology or in the intelligence Everything Is Social, Mobile, and data-driven analysis demands a menu of profession, nor is it a deterministic Local. Much of the explosion of big data apps or even dedicated software develop- scenario of a high-tech future. has been driven by the fact that informa- ers integrated into analyst teams—as they Rather, here we seek to identify some tion is increasingly social (generated and already are in some parts of the IC. opportunities and risks of the disruptive transmitted by many users, rather than a The Internet Is Everywhere. The rate technologies at hand. First, we sketch a few big producers), mobile (collected by of increase in big data will only grow as background of the most important IT sensors on ubiquitous Internet-connected more devices join the Internet. These trends that are shaping today’s economy mobile devices), and local (geospatially devices not only provide an interface for and society. Second, we outline how big tagged). These trends have irreversibly users, but are also creating a growing data could transform intelligence analysis; transformed IT; mobile devices in partic- “Internet of things”—everything from it has the potential to unlock enormous ular have become the primary means of household appliances to industrial ro- productivity gains and effectiveness connecting to the Internet and have thus bots—that generate and use more data, by automating some currently labor- become the primary market for much IT in turn creating more potential knowl- intensive tasks, enabling new forms innovation. This has already created new edge and vulnerabilities. At the same of analysis and creating new forms of opportunities not only for collection, time, emerging technologies (such as presentation. Third, we argue big data but also for intelligence processing, ex- free-space optical communications, which cannot do it all; its utility in making ploitation, and dissemination (PED), and use lasers to transmit data through the at- sense of complex systems and addressing analysis. mosphere) are allowing users to bring the knowledge gaps is limited. Finally, we Data Are Useless Without Data Internet into austere communications en- outline how big data could transform the Science. The exponential creation of vironments in order to enable the wider wider assessment agency enterprise. We digital data holds enormous potential for military use of Internet-connected IT and argue that the explosion in data supply creating insight and knowledge through greater resilience to network failures. and demand will incentivize assessment PED and data analytics. The burgeoning These technology trends have been agencies to reposition their roles more field of data science—at the intersection driven by the commercial and scientific toward service-delivery functions and to of statistics, computer science, and other sectors, but they also have powerful rebalance their workforces. related fields—is increasingly being used implications for the IC; they are rapidly None of this is inevitable. In both by the private sector to realize the com- challenging long-held conceptions analytic operations and enterprise mercial potential of big data, often for of intelligence collection targets, management, much of how the scenario prosaic tasks such as tracking a person’s business processes, required IT tools, actually unfolds will be determined by the consumption patterns to better target and workforce skill sets. But the IC’s vision and agility of our leadership, our advertising campaigns. The IC’s routine capacity to adopt these technologies partners, and our adversaries. Defense work of collection, PED, and analysis is remains inadequate; fully exploiting these and intelligence community (IC) leaders still largely organized on the Cold War trends would require a deep revision must play an active but balanced role, model of seeking out sparse and secret of innovation policy and IT-acquisition exploiting big data’s potential, but information. Now, however, it must cope business models. To adequately exploit understanding its limitations. with the inverse challenge (and exploit these opportunities, the IC would the opportunities) of managing and need to incorporate a “technology Today’s Tech Trends analyzing massive quantities of data and, push” acquisition model alongside the The big data phenomenon presents in the process, compete with the lucrative customary “demand pull” model. In to- defense intelligence with a range of private sector to attract the highly special- day’s IT environment of faster innovation opportunities, from off-the-shelf tools ized skills of data scientists.3 and more disruptive and unpredictable to complex business-process reforms. IT Solutions Are Customized and technologies, where government lacks Some tools can be absorbed wholesale Intuitive. The accelerating pace of the speed or vision to lead innovation, by the IC; for example, social network- innovation and the need to best harness the IC’s best option may be to monitor JFQ 79, 4th Quarter 2015 Symon and Tarapore 5 and leverage incipient innovation instead and algorithms are set, the data could be collection, PED, and analysis, and cueing of attempting to drive it. Rather than managed—collected, moved, stored, and further collection. Vast quantities of dictating requirements to firms through a organized—with relatively little additional data—unprocessed and unseen by any byzantine acquisitions process (as in most effort. Applied to all-source intelligence, analyst—would be stored, available to defense procurement programs), the IC’s the exponential increase in data and ana- be mined later in the context of future greatest potential for IT adoption may lie lytics would render manual information data or requirements or to discover in injecting its “use cases” (and resources) retrieval impractical and unnecessary; the or recognize associations or trends. in the start-up or development phases of heavy lifting of data management could be Machine learning would allow this en- future technologies. And in a data-inten- largely automated. Already-existing tools tire process to improve with time. The sive information environment, assessment can create an automatic and persistent accumulation of data and the refinement agency leaders would need to recognize push of data to analysts, obviating the of algorithms would allow for dynamic that adaptive IT is integral to analytic labor-intensive requirement to manually and progressively