COMPREHENSIVE TESTING OF IMMINENT THREAT PUBLIC MESSAGES FOR MOBILE DEVICES: UPDATED FINDINGS First Responders Group August 2015 AUTHORS Michele Wood Hamilton Bean Brooke Liu Marcus Boyd COMPREHENSIVE TESTING OF IMMINENT THREAT PUBLIC MESSAGES FOR MOBILE DEVICES: FINAL REPORT Prepared for Department of Homeland Security Science and Technology Directorate, Support to the Homeland Security Enterprise and First Responders Group Comprehensive Testing of Imminent Threat Public Messages for Mobile Devices ACKNOWLEDGEMENTS Our deep appreciation extends to the Department of Homeland Security Science and Technology Directorate First Responders Group, Office for Interoperability and Compatibility, Wireless Emergency Alerts (WEA) Research, Development, Test and Evaluation (RDT&E) Program, for sponsoring this work, especially program manager Denis Gusty. Further, we offer our gratitude to the emergency responders whose dedication and commitment ensure the safety of our families, our communities and our nation. This report is a tribute to their service. For information about this publication or other START research, contact NATIONAL CONSORTIUM FOR THE STUDY OF TERRORISM AND RESPONSES TO TERRORISM 8400 Baltimore Ave., Suite 250 College Park, MD 20740 Tel (301) 405-6600 [email protected] www.start.umd.edu Comprehensive Testing of Imminent Threat Public Messages for Mobile Devices Table of Contents Executive Summary ..................................................................................................................................................i 1. Background ......................................................................................................................................................... 1 1.1 Key Findings from START’s Prior Research ............................................................................................... 1 1.2 New Research Questions............................................................................................................................... 2 Table 1. Research Questions by Research Phases ........................................................................................... 3 2. Research Design Overview ................................................................................................................................. 3 2.1 Exploratory Focus Groups ............................................................................................................................ 4 2.2 Experiments and Confirmatory Focus Groups.............................................................................................. 4 Table 2. Phase IV experiments conducted and research questions asked ....................................................... 4 3. Exploratory Focus Groups .................................................................................................................................. 4 3.1. Exploratory Focus Groups: Recruitment and Composition ......................................................................... 5 3.2. Exploratory Focus Groups: Scenario, Questions, Messages and Maps ....................................................... 5 Figure 1. Sample Map Tested in Exploratory Focus Groups .......................................................................... 5 3.3. Exploratory Focus Groups: Findings ........................................................................................................... 5 4. Experiments ...................................................................................................................................................... 11 4.1. Experiments: Method ................................................................................................................................. 11 Figure 2. Sample Message Tested in Experiments........................................................................................ 11 4.2. Experiments: Findings................................................................................................................................ 14 5. Confirmatory Focus Groups ............................................................................................................................. 17 Figure 3. Messages Tested in Confirmatory Focus Groups .......................................................................... 18 5.2. Confirmatory Focus Groups: Scenario, Questions, Messages and Maps .................................................. 18 5.3. Confirmatory Focus Groups: Findings Summary ...................................................................................... 19 6. Conclusions and Future Research ..................................................................................................................... 29 7. References ......................................................................................................................................................... 33 Appendix A: Exploratory Focus Groups’ Interview Guide, Messages and Maps................................................ 34 Figure 1. Focus Groups’ Interview Guide for Apps, Hyperlinks and Maps ................................................. 34 Figure 2. 1,380-character Message and Maps ............................................................................................... 35 Appendix B: Example Experiment Questionnaire................................................................................................ 37 Table 1. Experiments: Sample Description ................................................................................................... 54 Table 1. (Continued) Experiments: Sample Description............................................................................... 55 Table 2. Experiments: Descriptive Statistics ................................................................................................. 56 Table 3. Experiment 1: Regression of 280-Character Message Outcomes on Message Content Order ....... 58 Table 4. Experiment 2: Regression of 280-Character Message Outcomes on Message Source ................... 60 Table 5. Experiment 3: Regression of 280-Character Message Outcomes on Map Features ....................... 61 Table 6a. Experiment 4: Means and Standard Deviations for Message Outcomes by Message Length and Map Inclusion ................................................................................................................................................ 62 Table 6b. Experiment 4: Two-Way Analysis of Variance of Message Outcomes by Message Length and Map Inclusion ................................................................................................................................................ 63 Figure 1. Experiment 4: Understanding by Message Length and Map Inclusion ......................................... 64 Figure 2. Experiment 4: Belief by Message Length and Map Inclusion ....................................................... 65 Figure 3. Experiment 4: Personalization by Message Length and Map Inclusion ........................................ 66 Figure 4. Experiment 4: Deciding by Message Length and Map Inclusion .................................................. 67 Figure 5. Experiment 4: Milling by Message Length and Map Inclusion..................................................... 68 Comprehensive Testing of Imminent Threat Public Messages for Mobile Devices Table 7. Experiment 5: Regression of 280-Character Message Outcomes on Message Elements ............... 69 Table 8. Experiment 6: Regression of 280-Character Message Outcomes on Hazard Type ........................ 70 Table 9. Experiment 7: Regression of 280-Character Message Outcomes on Added Text .......................... 71 Table 10. Experiment 8: Regression of 280-Character Message Outcomes on General Links and Personalized Apps ......................................................................................................................................... 72 Appendix D: Study Design ................................................................................................................................... 74 Table 1. Experiment 1 Design: Effect of Content Order ............................................................................... 74 Table 2. Experiment 2 Design: Effect of Message Source............................................................................ 75 Table 3. Experiment 3 Design: Effect of Different Map Features ................................................................ 76 Table 4. Experiment 4 Design: Effect of Amount of Information Provided—Length (90, 140, 280, 1,380 Characters) and Maps .................................................................................................................................... 77 Table 5. Experiment 5 Design: Effect of Different Message Elements ........................................................ 78 Table 6. Experiment 6 Design: Effect of Hazard Type ................................................................................. 78 Table 7. Experiment 7 Design: Effect of Spending the Additional 190 Characters on Different Message Elements ........................................................................................................................................................ 79 Table 8. Experiment 8 Design: Effect of Including Links and Apps (general text, personalized text, maps) ......................................................................................................................................................................