Analyzing the Return on Investment of Security Research

Detlof von Winterfeldt Director, CREATE University of Southern California 1150 S. Olive Street, Suite 1700 Los Angeles, CA 90015 (213) 740-3863 [email protected]

May 15, 2019

ADSA20 Boston, MA

This research was supported by the United States Department of Homeland Security (DHS) through the National Center for Risk and Economic Analysis of Terrorism Events (CREATE) at the University of Southern California (USC) under award number 2010-ST-061-RE0001. However, any opinions, findings, and conclusions or recommendations in this document are those of the authors and do not necessarily reflect views of the United States Department of Homeland Security, or the University of Southern California, or CREATE. Carl Crawford asks: So what? Who cares?

• The Problem. The Office of University Programs at DHS has been challenged to show the return on investment of the research conducted by its Centers of Excellence (COEs) • Who Cares? Congressional appropriations committees, OMB, DHS S&T leaders, OUP leaders, COE leaders, HS researchers • The Solution. Conduct selected benefit-cost analyses of about 20 OUP-funded research products (TTKPs=Tools, Technologies, and Knowledge Products) • The Results. – One research product (CT Datasets and ATR) produces over $100 million in benefits at a cost of $5 million, approximate ROI = 2,000% – 10 other research products (across multiple COEs) produce about $400 million in additional benefits, ROI = 856% – Two research products from CREATE produced enough benefits to pay for $45 million = 12 years of OUP funding of CREATE

2 2 Steps in the Benefit-Cost Analysis • Selection of research products • Baseline analysis • Cost analysis • Benefits analysis • Sensitivity and uncertainty analysis • Results: Net Benefits, BCR, and ROI

3 Application to 20 Research Products (Selected by USCG, TSA, and CBP)

1. The likelihood of a successful transition (transitioned and used = 1) 2. The beneficial impact on customer, if successfully transitioned

Note: Selections are not representative of all OUP-funded projects

4 Example: CT Datasets and Algorithms • Improve detection technology at TSA checkpoints • CT Datasets enable better detection algorithms • No need for classified or sensitive data • Reduced false (nuisance) alarms

5 Cost Calculations for CT Datasets

Amount in Amount in Cost Category Start End Source Real Dollars 2017 Dollars Pre-project costs (COE) Pre-project costs (other funding) Project costs (COE) 9/1/13 6/30/18 $ 7 40,000 $ 748,315 COE/OUP Project costs (university cost share) Oversight cost at the COE 9/21/10 6/30/18 $ 2 10,000 $ 216,987 COE/OUP Oversight cost at OUP 9/21/10 6/30/18 $ 1 60,000 $ 165,324 OUP Overhead Transition development cost 9/21/10 2/27/15 $ 3 ,555,000 $ 3,768,300 S&T, TSA Implementation start up cost Implementation cost (User) Implementation cost (COE) Implementation cost (Other users) TOTAL COST $ 4 ,665,000 $ 4,898,926

6 6 The Benefits of CT Datasets

• Reduces false alarms • Saves time of passengers • Saves time of TSA officers

National Academy of Science: Each 1% reduction in false alarms saved $25 million

7 7 Benefit Calculations for CT Datasets Input Variable Base Number of Travelers/Year 7 19,000,000 Current False Positive Rate 10.0% Decrease in False Positives 25% Time in Secondary (Min) 7.5 Cost of One Minute of Wait Time $0.79 Cost of One Minute of TSA Time $0.27 Percent attributable to CT Vendor 25% Percent attributable to ATR 30% Cost of a Successful Attack 2 5,000,000,000 Current Cost of False Positives/Year $ 5 71,605,000 3D Cost of False Positives/Year $ 4 28,703,750 Attributabe Benefits/Year $ 1 0,717,594 Discount Factor 0.030 Total Value of 3D CT Scanners for 10 Years $ 107,175,938 Total Net Benefits for Ten Years $ 102,277,012 8 8 Costs vs. Benefits of CT Datasets

Benefits (in 2017 $) $107,175,938

BCR = 22 ROI = 2,087%

Cost (in 2017 $) $4,898,926

9 Summary of 10 Additional BCAs

Tool, Technology, or Years of Use for Cost (in Thousands, Median Net Benefit Knowledge Product Net Benefit in 2017 $) (50th Percentile) (TTKP) Calculations Research Products with Past PROTECT Applications$710 $35,505 6 Past & 4 Future Years ARMOR $1,057 $28,969 10 Years, Past Use CgSARVA $803 $5,247 One Time (Sandy) Research Products with Potential Future ADCIRC Applications$36,893 $256,411 10 Years, Future Use Engineered Swabs $1,867 $22,528 10 Years, Future Use GeoXray $273 $18,404 10 Years, Future Use TraffiCop $1,413 $10,444 10 Years, Future Use HOAX Calls $183 $4,646 10 Years, Future Use BOARD $1,018 $2,435 10 Years, Future Use E-CAT $942 $806 10 Years, Future Use TOTAL $45,161 $386,395 ROI=856% 10 10 Conclusions

• Benefit-cost analyses can be successfully conducted on OUP-funded research products • Uncertainty about future benefits • Nevertheless: Benefits for a dozen selected research products exceed OUP funding by a wide margin • A few research products are the main contributors to benefits (ADCIRC, IRIS, CT Datasets, Enhanced Millimeter Wave Scanner) • Start considering success criteria and benefits and benefit-cost analysis early – at proposal stage • Nine benefits criteria and models developed by CREATE help (e.g., reducing false alarms, increasing detection, reducing threat, informing decision-making)

11 Thank you! For more on CREATE, go to create.usc.edu

12 Supplementary Slides

13 Sensitivity Analysis for CT Datasets

Current False Positive Rate 5.0% 20.0%

Time in Secondary (Min) 5 10

Percent attributable to ATR 25% 40%

Cost of One Minute of Wait Time $0.50 $1.00

Decrease in False Positives 20% 30%

Percent attributable to CT Vendor 20% 30%

Cost of One Minute of TSA Time $0.18 $0.35

Number of Travelers/Year 719,000,000

Number of Attack Attempts Per Year 0.14

Detection probability with current system 0.9

Cost of a Successful Attack 150,000,000,000

$20,000,000 $40,000,000 $60,000,000 $80,000,000 $100,000,000 $120,000,000 $140,000,000 $160,000,000 $180,000,000 $200,000,000 $220,000,000 $240,000,000 Net Benefits for 10 Years

14 14 Uncertainty Analysis for CT Datasets

• Using wide ranges for all uncertain inputs (min, mode, max) • Using triangular distributions for each input

2500

2000

1500

y

c

n

e

u

q

e r F 1000

500

0 $- $50,000,000 $100,000,000 $150,000,000 $200,000,000 $250,000,000 $300,000,000 $350,000,000 $400,000,000 Net Benefits for Ten Years

15 15 10 Additional Research Products • ADCIRC - ADvanced CIRCulation Next generation modeling suite to predict coastal flooding CHC - Coastal Hazards Center of Excellence

• ARMOR - Assistant for Randomized Monitoring Over Routes Software that randomizes schedules, plans, or actions for security agencies CREATE – Center for Risk and Economic Analysis of Terrorism Events

• BOARD - Bus Operator Awareness and Research Development Training program for bus operators to respond to security threats NTSCOE – National Transportation Security Center of Excellence

• CgSARVA – Coast Guard Search and Rescue Visual Analytics Interactive analysis of trends, patterns, anomalies, and distribution of Search and Rescue cases VACCINE – Visual Analytics for Command, Control and Interoperability Environments

• E-CAT – Economic Consequence Analysis Tool Provides rapid estimates of economic impact of threats CREATE – Center for Risk and Economic Analysis of Terrorism Events 16 16 • Engineered Detection Swab Swabs to improve detection of trace explosives on luggage and persons ALERT - Awareness and Localization of Explosives-Related Threats

• GeoXray Tool to display geographic area and overlay relevant emergency response CCICADA – Command, Control and Interoperability Center for Data Analysis

• HOAX Calls State-of-the-art tool for identifying hoax emergency calls CCICADA – Command, Control and Interoperability Center for Data Analysis

• PROTECT – Port Resilience Operational/Tactical Enforcement to Combat Terrorism Software to randomize schedules in a port environment CREATE – Center for Risk and Economic Analysis of Terrorism Events

• TraffiCop Analysis of Twitter and social media to detect human trafficking CCICADA – Command, Control and Interoperability Center for Data Analysis 17

ADCIRC - ADvanced CIRCulation Storm surge/inundation modeling COE: CHC - Coastal Hazards Center of Excellence Median Benefit (50th Percentile) $293,804,000

• Storm surge model that combines rain, atmospheric pressure, and wind forecasts Cost (in 2017 $)

• Predicts when, where, and $36,893,000 to what extent flooding will inundate a coastal community

18 18 E-CAT – Economic Consequence Analysis Tool Rapid estimates of the economic impact of threats COE: CREATE- Center for Risk and Economic Analysis of Terrorism Events

Median Benefit (50th Percentile) $1,746,000

Cost (in 2017 $) • Provides quick $942,000 estimates of the economic impact of terrorist attacks, natural disasters, and technological accidents

19 Summary of Costs and Benefits

Tool, Technology, or Years of Use for Cost (in Thousands, Median Net Benefit Knowledge Product Net Benefit in 2017 $) (50th Percentile) (TTKP) Calculations Research Products with Past PROTECT Applications$710 $35,505 6 Past & 4 Future Years ARMOR $1,057 $28,969 10 Years, Past Use CgSARVA $803 $5,247 One Time (Sandy) Research Products with Potential Future ADCIRC $36,893 Applications $256,411 10 Years, Future Use Engineered Swabs $1,867 $22,528 10 Years, Future Use GeoXray $273 $18,404 10 Years, Future Use TraffiCop $1,413 $10,444 10 Years, Future Use HOAX Calls $183 $4,646 10 Years, Future Use BOARD $1,018 $2,435 10 Years, Future Use E-CAT $942 $806 10 Years, Future Use TOTAL $45,161 $386,395 ROI=856% 20 20 Nine Benefits Criteria and Models

• Cost savings with the same risk level – Savings in operational costs (ARMOR, E-CAT) – Savings by stretching expenses (cgSARVA) • Increased security at same cost – Reduction of threat (HOAX Calls) – Reduction of vulnerability (PROTECT) – Reduction of negative consequences (TraffiCop) • Improved signal detection – Reduction of false alarm rates (Engineered Swabs) – Improvement of detection rates (3D CT Scans) • Value of information – Reduction of uncertainty (GeoXray) – Improvement of decision quality (ADCIRC)

21 21