Analyzing the Return on Investment of Homeland Security Research
Detlof von Winterfeldt Director, CREATE University of Southern California 1150 S. Olive Street, Suite 1700 Los Angeles, CA 90015 (213) 740-3863 [email protected]
May 15, 2019
ADSA20 Boston, MA
This research was supported by the United States Department of Homeland Security (DHS) through the National Center for Risk and Economic Analysis of Terrorism Events (CREATE) at the University of Southern California (USC) under award number 2010-ST-061-RE0001. However, any opinions, findings, and conclusions or recommendations in this document are those of the authors and do not necessarily reflect views of the United States Department of Homeland Security, or the University of Southern California, or CREATE. Carl Crawford asks: So what? Who cares?
• The Problem. The Office of University Programs at DHS has been challenged to show the return on investment of the research conducted by its Centers of Excellence (COEs) • Who Cares? Congressional appropriations committees, OMB, DHS S&T leaders, OUP leaders, COE leaders, HS researchers • The Solution. Conduct selected benefit-cost analyses of about 20 OUP-funded research products (TTKPs=Tools, Technologies, and Knowledge Products) • The Results. – One research product (CT Datasets and ATR) produces over $100 million in benefits at a cost of $5 million, approximate ROI = 2,000% – 10 other research products (across multiple COEs) produce about $400 million in additional benefits, ROI = 856% – Two research products from CREATE produced enough benefits to pay for $45 million = 12 years of OUP funding of CREATE
2 2 Steps in the Benefit-Cost Analysis • Selection of research products • Baseline analysis • Cost analysis • Benefits analysis • Sensitivity and uncertainty analysis • Results: Net Benefits, BCR, and ROI
3 Application to 20 Research Products (Selected by USCG, TSA, and CBP)
1. The likelihood of a successful transition (transitioned and used = 1) 2. The beneficial impact on customer, if successfully transitioned
Note: Selections are not representative of all OUP-funded projects
4 Example: CT Datasets and Algorithms • Improve detection technology at TSA checkpoints • CT Datasets enable better detection algorithms • No need for classified or sensitive data • Reduced false (nuisance) alarms
5 Cost Calculations for CT Datasets
Amount in Amount in Cost Category Start End Source Real Dollars 2017 Dollars Pre-project costs (COE) Pre-project costs (other funding) Project costs (COE) 9/1/13 6/30/18 $ 7 40,000 $ 748,315 COE/OUP Project costs (university cost share) Oversight cost at the COE 9/21/10 6/30/18 $ 2 10,000 $ 216,987 COE/OUP Oversight cost at OUP 9/21/10 6/30/18 $ 1 60,000 $ 165,324 OUP Overhead Transition development cost 9/21/10 2/27/15 $ 3 ,555,000 $ 3,768,300 S&T, TSA Implementation start up cost Implementation cost (User) Implementation cost (COE) Implementation cost (Other users) TOTAL COST $ 4 ,665,000 $ 4,898,926
6 6 The Benefits of CT Datasets
• Reduces false alarms • Saves time of passengers • Saves time of TSA officers
National Academy of Science: Each 1% reduction in false alarms saved $25 million
7 7 Benefit Calculations for CT Datasets Input Variable Base Number of Travelers/Year 7 19,000,000 Current False Positive Rate 10.0% Decrease in False Positives 25% Time in Secondary (Min) 7.5 Cost of One Minute of Wait Time $0.79 Cost of One Minute of TSA Time $0.27 Percent attributable to CT Vendor 25% Percent attributable to ATR 30% Cost of a Successful Attack 2 5,000,000,000 Current Cost of False Positives/Year $ 5 71,605,000 3D Cost of False Positives/Year $ 4 28,703,750 Attributabe Benefits/Year $ 1 0,717,594 Discount Factor 0.030 Total Value of 3D CT Scanners for 10 Years $ 107,175,938 Total Net Benefits for Ten Years $ 102,277,012 8 8 Costs vs. Benefits of CT Datasets
Benefits (in 2017 $) $107,175,938
BCR = 22 ROI = 2,087%
Cost (in 2017 $) $4,898,926
9 Summary of 10 Additional BCAs
Tool, Technology, or Years of Use for Cost (in Thousands, Median Net Benefit Knowledge Product Net Benefit in 2017 $) (50th Percentile) (TTKP) Calculations Research Products with Past PROTECT Applications$710 $35,505 6 Past & 4 Future Years ARMOR $1,057 $28,969 10 Years, Past Use CgSARVA $803 $5,247 One Time (Sandy) Research Products with Potential Future ADCIRC Applications$36,893 $256,411 10 Years, Future Use Engineered Swabs $1,867 $22,528 10 Years, Future Use GeoXray $273 $18,404 10 Years, Future Use TraffiCop $1,413 $10,444 10 Years, Future Use HOAX Calls $183 $4,646 10 Years, Future Use BOARD $1,018 $2,435 10 Years, Future Use E-CAT $942 $806 10 Years, Future Use TOTAL $45,161 $386,395 ROI=856% 10 10 Conclusions
• Benefit-cost analyses can be successfully conducted on OUP-funded research products • Uncertainty about future benefits • Nevertheless: Benefits for a dozen selected research products exceed OUP funding by a wide margin • A few research products are the main contributors to benefits (ADCIRC, IRIS, CT Datasets, Enhanced Millimeter Wave Scanner) • Start considering success criteria and benefits and benefit-cost analysis early – at proposal stage • Nine benefits criteria and models developed by CREATE help (e.g., reducing false alarms, increasing detection, reducing threat, informing decision-making)
11 Thank you! For more on CREATE, go to create.usc.edu
12 Supplementary Slides
13 Sensitivity Analysis for CT Datasets
Current False Positive Rate 5.0% 20.0%
Time in Secondary (Min) 5 10
Percent attributable to ATR 25% 40%
Cost of One Minute of Wait Time $0.50 $1.00
Decrease in False Positives 20% 30%
Percent attributable to CT Vendor 20% 30%
Cost of One Minute of TSA Time $0.18 $0.35
Number of Travelers/Year 719,000,000
Number of Attack Attempts Per Year 0.14
Detection probability with current system 0.9
Cost of a Successful Attack 150,000,000,000
$20,000,000 $40,000,000 $60,000,000 $80,000,000 $100,000,000 $120,000,000 $140,000,000 $160,000,000 $180,000,000 $200,000,000 $220,000,000 $240,000,000 Net Benefits for 10 Years
14 14 Uncertainty Analysis for CT Datasets
• Using wide ranges for all uncertain inputs (min, mode, max) • Using triangular distributions for each input
2500
2000
1500
y
c
n
e
u
q
e r F 1000
500
0 $- $50,000,000 $100,000,000 $150,000,000 $200,000,000 $250,000,000 $300,000,000 $350,000,000 $400,000,000 Net Benefits for Ten Years
15 15 10 Additional Research Products • ADCIRC - ADvanced CIRCulation Next generation modeling suite to predict coastal flooding CHC - Coastal Hazards Center of Excellence
• ARMOR - Assistant for Randomized Monitoring Over Routes Software that randomizes schedules, plans, or actions for security agencies CREATE – Center for Risk and Economic Analysis of Terrorism Events
• BOARD - Bus Operator Awareness and Research Development Training program for bus operators to respond to security threats NTSCOE – National Transportation Security Center of Excellence
• CgSARVA – Coast Guard Search and Rescue Visual Analytics Interactive analysis of trends, patterns, anomalies, and distribution of Search and Rescue cases VACCINE – Visual Analytics for Command, Control and Interoperability Environments
• E-CAT – Economic Consequence Analysis Tool Provides rapid estimates of economic impact of threats CREATE – Center for Risk and Economic Analysis of Terrorism Events 16 16 • Engineered Detection Swab Swabs to improve detection of trace explosives on luggage and persons ALERT - Awareness and Localization of Explosives-Related Threats
• GeoXray Tool to display geographic area and overlay relevant emergency response CCICADA – Command, Control and Interoperability Center for Data Analysis
• HOAX Calls State-of-the-art tool for identifying hoax emergency calls CCICADA – Command, Control and Interoperability Center for Data Analysis
• PROTECT – Port Resilience Operational/Tactical Enforcement to Combat Terrorism Software to randomize schedules in a port environment CREATE – Center for Risk and Economic Analysis of Terrorism Events
• TraffiCop Analysis of Twitter and social media to detect human trafficking CCICADA – Command, Control and Interoperability Center for Data Analysis 17
ADCIRC - ADvanced CIRCulation Storm surge/inundation modeling COE: CHC - Coastal Hazards Center of Excellence Median Benefit (50th Percentile) $293,804,000
• Storm surge model that combines rain, atmospheric pressure, and wind forecasts Cost (in 2017 $)
• Predicts when, where, and $36,893,000 to what extent flooding will inundate a coastal community
18 18 E-CAT – Economic Consequence Analysis Tool Rapid estimates of the economic impact of threats COE: CREATE- Center for Risk and Economic Analysis of Terrorism Events
Median Benefit (50th Percentile) $1,746,000
Cost (in 2017 $) • Provides quick $942,000 estimates of the economic impact of terrorist attacks, natural disasters, and technological accidents
19 Summary of Costs and Benefits
Tool, Technology, or Years of Use for Cost (in Thousands, Median Net Benefit Knowledge Product Net Benefit in 2017 $) (50th Percentile) (TTKP) Calculations Research Products with Past PROTECT Applications$710 $35,505 6 Past & 4 Future Years ARMOR $1,057 $28,969 10 Years, Past Use CgSARVA $803 $5,247 One Time (Sandy) Research Products with Potential Future ADCIRC $36,893 Applications $256,411 10 Years, Future Use Engineered Swabs $1,867 $22,528 10 Years, Future Use GeoXray $273 $18,404 10 Years, Future Use TraffiCop $1,413 $10,444 10 Years, Future Use HOAX Calls $183 $4,646 10 Years, Future Use BOARD $1,018 $2,435 10 Years, Future Use E-CAT $942 $806 10 Years, Future Use TOTAL $45,161 $386,395 ROI=856% 20 20 Nine Benefits Criteria and Models
• Cost savings with the same risk level – Savings in operational costs (ARMOR, E-CAT) – Savings by stretching expenses (cgSARVA) • Increased security at same cost – Reduction of threat (HOAX Calls) – Reduction of vulnerability (PROTECT) – Reduction of negative consequences (TraffiCop) • Improved signal detection – Reduction of false alarm rates (Engineered Swabs) – Improvement of detection rates (3D CT Scans) • Value of information – Reduction of uncertainty (GeoXray) – Improvement of decision quality (ADCIRC)
21 21