Cybersecurity Experimentation of the Future (CEF): Catalyzing a New Generation of Experimental Cybersecurity Research Community Plan and Roadmap to Develop Future Experimentation Infrastructure in Support of Cybersecurity Research FINAL REPORT July 31, 2015 David Balenson and Laura Tinnel, SRI International Terry Benzel, USC Information Sciences Institute This material is based upon work supported by the National Science Foundation under Grant No. ACI-1346277 and ACI-1346285. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. 1 CYBERSECURITY EXPERIMENTATION OF THE FUTURE (CEF) EXECUTIVE SUMMARY This report presents a strategic plan and enabling roadmap intended to catalyze generational advances in the field of experimental cybersecurity research. These results represent the conclusions of a study conducted under NSF auspices by SRI International and USC Information Sciences Institute throughout calendar year 2014. The study had broad participation by stakeholders representing the cybersecurity research, research sponsor, and customer communities. The report outlines the process and methodology of the project, presents key inputs, supporting evidence developed through the course of the study, and synthesized results, and then presents our final conclusions. Our overarching finding is that transformational progress in three distinct, yet synergistic, areas is required to achieve the desired objectives: 1) Fundamental and broad intellectual advances in the field of experimental methodologies and techniques, with particular focus on complex systems and human-computer interactions. 2) New approaches to rapid and effective sharing of data and knowledge and information synthesis that accelerate multi-discipline and cross-organizational knowledge generation and community building. 3) Advanced, accessible experimentation infrastructure capabilities. The central result of our study is a roadmap that presents requirements, objectives and goals in each of the areas outlined above over three, five and ten year phases. In some cases, the phases build upon each other, and in other cases, new fundamental research is required over a longer period of time to satisfy the objectives of the roadmap. Taken together, these areas, as embodied in the roadmap, paint a vision for a new generation of experimental cybersecurity research – one that offers powerful assistance towards helping researchers shift the asymmetric cyberspace context to one of greater planning, preparedness, and higher assurance fielded solutions. The capabilities identified in the roadmap take into account the current state of the art in experimental cybersecurity research and its supporting infrastructure, other types of research facilities, and existing cyber-domain “test and evaluation” capabilities. In addition to leveraging current and expected capabilities in cybersecurity and adjacent areas, the roadmap presumes advances in key computer science disciplines such as ontologies, metadata, libraries, and corresponding resource discovery. We emphasize that while this type of study would typically focus heavily on experimentation infrastructure (i.e., tools and testbeds), and while we did pay significant attention to this topic, our fundamental conclusion is that an emphasis on infrastructure alone will fall far short of achieving the transformational shift in the research, community, and experimentation required to address cybersecurity in the rapidly changing cyber environment. Our conclusion is that strong, coupled, and synergistic advances across each of the areas outlined above – fundamental methodological development, fostering and leveraging communities of researchers, and in the capabilities of the infrastructure supporting that research – will transform the field of cybersecurity. i CYBERSECURITY EXPERIMENTATION OF THE FUTURE (CEF) ii CYBERSECURITY EXPERIMENTATION OF THE FUTURE (CEF) TABLE OF CONTENTS Executive Summary .................................................................................................................. i Table of Contents .................................................................................................................... iii 1 Introduction ....................................................................................................................... 1 1.1 Roadmap Findings ................................................................................................................. 2 1.2 Top Five Recommendations .................................................................................................. 3 1.3 Definition of “Cybersecurity Experimentation Infrastructure” .............................................. 5 1.4 Where is Experimentation Applicable? ................................................................................. 6 1.5 Representative Cybersecurity Hard Problems ....................................................................... 7 1.6 Experimentation – It’s About the Real World ........................................................................ 8 1.7 Motivation: Why Are We Doing This? ................................................................................... 8 1.8 Audience of the Report .......................................................................................................... 9 1.9 Structure of the Report .......................................................................................................... 9 2 Study Description ............................................................................................................ 11 2.1 Overall Process and Approach ............................................................................................. 11 2.2 Investigate Existing Experimentation Infrastructure ........................................................... 12 2.3 Conduct Community-Based Study Groups ........................................................................... 12 2.4 Generate Strategic Plan and Roadmap ................................................................................ 13 3 Survey of Existing Infrastructure ...................................................................................... 15 3.1 Approach ............................................................................................................................. 15 3.2 Existing Testbeds ................................................................................................................. 15 3.3 Existing Tools ....................................................................................................................... 16 3.4 Summary .............................................................................................................................. 16 4 Roadmap for Executing the CEF Vision ............................................................................. 19 4.1 Domains of Applicability ...................................................................................................... 23 4.2 Modeling the Real World for Scientifically Sound Experiments .......................................... 29 4.3 Frameworks and Building Blocks for Extensibility ............................................................... 36 4.4 Experiment Design and Instantiation .................................................................................. 46 4.5 Interconnected Research Infrastructure .............................................................................. 54 4.6 Experiment Execution and Management ............................................................................ 62 4.7 Instrumentation and Experiment Analysis ........................................................................... 68 4.8 Meta-Properties ................................................................................................................... 75 5 Conclusions and Community Recommendations .............................................................. 85 5.1 Roadmap Findings ............................................................................................................... 85 5.2 Conclusion ........................................................................................................................... 87 6 Acknowledgements ......................................................................................................... 89 7 References ....................................................................................................................... 91 A Survey of Existing Experimentation Infrastructure ........................................................... 95 A.1 Air Force Research Laboratory Cyber Experimentation Environment (CEE) ....................... 97 A.2 USC-ISI DeterLab .................................................................................................................. 98 A.3 Department of Transportation Connected Vehicle Test bedS ........................................... 102 A.4 European Union FIRE Initiative .......................................................................................... 104 iii CYBERSECURITY EXPERIMENTATION OF THE FUTURE (CEF) A.5 National Science Foundation GENI .................................................................................... 107 A.6 NICT Japan StarBed3 / JGN2+ ............................................................................................