Utilization of Real-Time Social Media Data in Severe Weather Events

Evaluating the Prospects of Social Media Data Use for Severe Weather Forecasting, Communication, and Post-Event Assessments

FINAL REPORT

March 2015

Carol L. Silva, Principal Investigator Joseph Ripberger, Research Scientist Hank C. Jenkins-Smith, Co-Principal Investigator Jack Friedman, Research Scientist Paul Spicer, Co-Principal Investigator Peter J. Lamb, Co-Principal Investigator

The University of Oklahoma

AWARD #: NA12OAR4590120

Acknowledgements

The authors wish to express appreciation to the following individuals whose support and research assistance made this project possible.

Matthew Henderson, OU Center for Risk and Crisis Management Deputy Director for IT Development and Design

Kerry Herron, OU Center for Risk and Crisis Management Research Scientist

Nina Carlson, OU Center for Risk and Crisis Management Deputy Director for Operations

Chloe Magee, OU Center for Risk and Crisis Management Undergraduate Research Fellow

Wesley Wehde, OU Center for Risk and Crisis Management Undergraduate Research Fellow

Hayley Scott, OU Center for Risk and Crisis Management Undergraduate Research Fellow

Harold Brooks, National Severe Storms Laboratory Senior Research Scientist

Makenzie Krocak, Iowa State University Ernest F. Hollings Undergraduate Scholar

Annelise Russell, University of Texas Graduate Research Assistant

Contents

1. Introduction and Executive Summary ...... 7 2. The Evolution of Reception, Reliance, and Trust: Public Usage of Information from Social Media About Severe Weather ...... 14 2.1 Introduction ...... 14 2.2 Data ...... 14 2.3 Findings ...... 16 Reception ...... 16 Reliance ...... 17 Trust ...... 19 2.4 Conclusions ...... 20 3. Defining the User Base: Who Uses Social Media to Collect Information About Severe Weather? ...... 22 3.1 Introduction ...... 22 3.2 Data ...... 22 3.3 Methods ...... 22 3.4 Measures ...... 23 Reception ...... 23 Reliance ...... 24 Trust ...... 25 Demographic Attributes ...... 26 3.5 Findings ...... 27 3.6 Conclusions ...... 30 4. Context for and Everyday Use of Social Media Among Forecasters in Warning Forecast Offices ...... 31 4.1 Introduction ...... 31 4.2 Data ...... 31 4.3 Findings ...... 32 Forecaster Perceptions of the Public’s Understanding of Weather and Weather Science ...... 32 Usage of/Beliefs About Social Media in WFOs ...... 34 4.4 Conclusions ...... 39 5. Skepticism Towards and Resistance to Social Media in Warning Forecast Offices ...... 41 5.1 Introduction ...... 41 5.2 Data ...... 41 5.3 Findings ...... 41 General Skepticism About Adoption of Social Media ...... 43 Skepticism About Weather Reports Submitted Through Social Media . 43 Skepticism, “Healthy Skepticism,” and Combating Skepticism ...... 47 5.4 Conclusions ...... 49

6. Searching for a Signal in the Noise: A Temporal Comparison of Social Media and Severe Weather Activity ...... 50 6.1 Introduction ...... 50 6.2 Data ...... 50 6.3 Findings ...... 53 6.4 Conclusions ...... 55 7. Close Up (Part I): Twitter Users Before, During, and After the 2013 Newcastle-Moore-South Oklahoma City Tornado ...... 57 7.1 Introduction ...... 57 7.2 Data ...... 57 7.3 Findings ...... 60 7.4 Conclusions ...... 63 8. Close Up (Part II): The Location of Twitter Users Before, During, and After the 2013 Newcastle-Moore-South Oklahoma City Tornado ...... 64 8.1 Introduction ...... 64 8.2 Data ...... 64 8.3 Findings ...... 66 8.4 Conclusions ...... 70 9. Close Up (Part III): The Evolution of Message Content Before, During, and After the 2013 Newcastle-Moore-South Oklahoma City Tornado ...... 71 9.1 Introduction ...... 71 9.2 Data ...... 71 9.3 Findings ...... 73 9.4 Conclusions ...... 78 10. Close Up (Part IV): The Evolution of Message Quality Before, During, and After the 2013 Newcastle-Moore-South Oklahoma City Tornado ...... 79 10.1 Introduction ...... 79 10.2 Data ...... 79 10.3 Findings ...... 82 10.4 Conclusions ...... 87 11. Moving Forward: Where Do We Go From Here? ...... 88 References ...... 92

List of Tables

Table 1: Regression Models Predicting Reception of, Reliance on, and Trust in Information About Severe Weather from Social Media Sources ...... 28 Table 2: Summary of Forecaster Interviews Conducted ...... 32 Table 3: Twitter User Categories, Example Tweet Descriptions, and Counts of Sampled Tweets ...... 59 Table 4: Time Categories, Time Periods, and Counts of Sampled Tweets ...... 61 Table 5: Time Categories, Time Periods, and Counts of Geolocated Tweets ...... 68 Table 6: Content Categories, Example Message Text, and Counts of Sampled Tweets ...... 72 Table 7: Time Categories, Time Periods, and Counts of Sampled Tweets ...... 75 Table 8: Time Categories, Time Periods, and Counts of Sampled Information Dissemination Tweets ...... 85

List of Figures

Figure 1: Approximate Location of Survey Respondents ...... 16 Figure 2: Warning Reception by Severe Weather Information Source and Year ...... 17 Figure 3: Importance of Severe Weather Information Sources by Year ...... 18 Figure 4: Trust in Severe Weather Information Sources by Year ...... 20 Figure 5: Importance of Social Media as a Source of Severe Weather Information by Year ...... 25 Figure 6: Trust in Social Media as a Source of Severe Weather Information by Year ...... 26 Figure 7: The Predicted Effect of Age on Reception, Reliance, and Trust ...... 29 Figure 8: Tweets Collected by Word/Phrase (April 24, 2012 – June 30, 2014) ...... 52 Figure 9: Tweets Containing the Word “Tornado” by Day (April 24, 2012 – June 30, 2014) ...... 52 Figure 10: Temporal Comparison of Twitter Activity, Tornado Watches, Tornado Warnings, and Tornado Occurrences (Apr. 25, 2012 – Nov. 11, 2012) ...... 54 Figure 11: Linear Regression Model Predictions of Twitter Activity as a Function of Severe Weather Activity (Apr. 25, 2012 – Nov. 11, 2012) ...... 55 Figure 12: Tornado Tweets and Users Per Day (May 18 – 22, 2014) ...... 58 Figure 13: The Evolution of Twitter Usership Before, During, and After the Event .. 62 Figure 14: Tornado Tweets and Geolocated Tornado Tweets Per Day (May 18 – 22, 2014) ...... 65 Figure 15: Predicted Probability of Location Information by User Type ...... 67 Figure 16: Geolocated Tornado Tweets by Time Period ...... 68 Figure 17: Word Clouds of Sampled Tweets in Top Three Information Dissemination Categories ...... 73 Figure 18: Word Clouds of Sampled Tweets in First Person Expression Categories ...... 75 Figure 19: The Evolution of Message Content Before, During, and After the Event .. 76 Figure 20: Verifiability and Accuracy of Information Dissemination Tweets ...... 82 Figure 21: Actionability and Timeliness of Information Dissemination Tweets ...... 83 Figure 22: The Evolution of Message Quality Before, During, and After the Event ... 86

1. Introduction and Executive Summary

According to a recent report by the Department of Homeland Security, “social media and collaborative technologies have become critical components of emergency preparedness, response, and recovery” (2013). These technologies are critical because they provide a centralized mechanism for two-way communication before, during, and after disasters that allows federal/state/local officials, emergency managers, the media, and affected communities to disseminate and receive information about a hazard in near real-time. Recognizing the potential utility of these technologies, the National Weather Service (NWS) recently began experimenting1 with the use of social media to educate the public and share critical information about weather, water, and climate issues as part of its effort to build a Weather-Ready Nation2. As yet, however, we know relatively little about who participates in the exchange of weather, water, and climate information that occurs on social media platforms and how that exchange of information evolves throughout the course of extreme weather and water events.

In this report, we begin to fill this void by answering three basic, yet important, research questions about social media usage before, during, and after one type of extreme weather event—tornadoes: 1. Who uses social media to get information about severe weather and how has this evolved over time? 2. What do NWS forecasters and weather scientists think about social media and how has it changed the way they approach their jobs?

1 Effective October 31, 2014, the NWS transitioned their use of Twitter (one type of social media) as an environmental information service from “experimental” to “operational” status. 2 For more information on the Weather-Ready Nation initiative, see http://www.nws.noaa.gov/com/weatherreadynation/about.html#.VQx3F2R4qvE.

INTRODUCTION AND EXECUTIVE SUMMARY PAGE | 7 3. How does social media usage evolve throughout the course of a severe weather event?

In Chapters 2 and 3, we use data from regional surveys of U.S. residents who live in tornado prone regions of the country to investigate the extent to which residents who live in these regions are using social media to collect and disseminate information about severe weather. The first survey was fielded in 2012 and the second in 2013. Among our key findings were the following: • Relative to traditional sources of severe weather information (such as television and radio), public usage of information from social media about severe weather is rather low. • Despite these relatively low rates of use, public reliance on information from social media about severe weather appears to be growing over time, whereas public usage of traditional sources may be declining over time. • Public utilization of social media sources to collect information about severe weather varies rather significantly across individuals and demographic groups. • Younger people are more likely than older people to use information about severe weather that comes from a social media source • To a lesser extent, the same is true of women and minority groups—women and members of minority groups are more likely than men and members of non-minority groups to use social media to obtain information about severe weather.

In Chapters 4 and 5, we shift away from a focus on the public and instead focus on NWS forecasters and weather scientists. More specifically, we use data from in- depth qualitative interviews of 44 forecasters and 5 weather scientists to address our second research question—what do forecasters and weather scientists think about social media and how has it changed the way they approach their jobs? Our interviews reveal a number of important insights, including:

INTRODUCTION AND EXECUTIVE SUMMARY PAGE | 8

• Forecasters and weather scientists are—for the most part—willing and eager to share information with the public via social media sources. • In general, however, forecasters and weather scientists are less willing and eager to use information (i.e., weather reports) provided by the public via social media sources than they are to share information with the public via social media sources. • This lack of enthusiasm towards weather reports on social media stems from an underlying skepticism about the quality of information that social media users are capable of and willing to provide—some forecasters and weather scientists believe that social media users are “good intentioned” but not sufficiently trained to provide useful reports; others are worried that social media users may provide intentionally misleading reports that—if used— would jeopardize the quality of information that forecasters are able to provide to the public.

In Chapter 6, we begin to investigate the extent to which this skepticism is justified by looking for a meaningful information “signal” in the social media “noise.” We do so by systematically comparing weather signals to social media signals, by collecting and analyzing millions of severe weather messages (tweets) that were published on Twitter between April 24th and June 20th, 2014. Our analyses of these data show that: • There is a severe weather signal in the social media noise, suggesting that relevant, credible, and/or valid information about severe weather is present in social media posts over the course of severe weather events. • Nevertheless, substantial noise remains, even on extreme weather days when communication about severe weather is most important.

In Chapters 7, 8, 9, and 10 we provide a more comprehensive look at the relationship between the signal and the noise by commencing a “close-up” study of

INTRODUCTION AND EXECUTIVE SUMMARY PAGE | 9 the messages published on Twitter before, during, and after a specific event – the 2013 Newcastle-Moore-South Oklahoma City tornado. In this study, we systematically identify the kinds of users who published these messages, where they were when they published them, what they said, and – if the message contained information of some sort – we evaluate the quality of this information. We then explore the extent to which these attributes of social media posts changed over the course of the severe weather event.

In Chapter 7, our study of social media users reveals that: • Individual Twitter users were responsible for the majority of messages containing the word “tornado” that were published on Twitter before, during, and after the event; organizations, by comparison, published a relatively small portion of the messages. • Before the event occurred, organizations were a bit more active than average whereas individuals were a bit less active than average; during and after the event, this relationship reversed – individuals were more active than normal and organizations were less active. • The overwhelming majority of messages published before, during, and after the event were published by users that were not affiliated, by way of self- identification, with the weather enterprise.

In Chapter 8, our analysis of where these users were when they published their messages shows that: • A very small fraction of the severe weather messages that Twitter users published before, during, and after the event included geographic information that utilized Twitter’s geolocation feature. • There was no discernable bias in the type of users who chose to include this geographic information. • The modal location of Twitter users changed over the course of the event – in the days leading up to the tornado, the majority of messages came from areas

INTRODUCTION AND EXECUTIVE SUMMARY PAGE | 10 of the country that were affected by severe weather on those days; as the storm approached and produced the tornado, the majority of messages came from users that were located in the regions of the country that were expecting/experiencing it; in the hours and days after the tornado occurred, messages began to appear throughout the region, across the country, and around the world.

In Chapter 9, our analysis of what Twitter users were sharing on social media throughout the course of the event indicates that: • The majority of the messages contained some sort of “objective” information about the event, or tornadoes in general; most messages contained general information about the event, information about how to help the victims, and/or information about the storm as it approached. • A smaller portion of the messages contained a personal or “subjective” expression of some sort; most of these messages expressed a general comment, feeling, or opinion about tornadoes or support/encouragement for the victims of the tornado; the remaining messages in this category included first person accounts of what the authors were doing/experiencing as the event unfolded. • The content of Twitter messages evolved throughout the course of the event; alert messages, for example, were published rather frequently in the hours and minutes leading up to the storm, whereas information about how to help the victims started to appear more frequently in the minutes, hours, and days that followed the event.

In Chapter 10, our assessment of the quality of information contained in messages that were published before, during, and after the 2013 Newcastle-Moore-South Oklahoma City tornado suggests that: • The information contained in the Twitter messages that were published over the course of the event was relatively high in quality; the vast majority of

INTRODUCTION AND EXECUTIVE SUMMARY PAGE | 11 messages that contained verifiable information were accurate and the majority of messages that contained actionable information were timely. • Like user type, location, and message content, information quality varied throughout the course of the event; information quality peaked in the days, hours, and minutes leading up to the event, but declined in the aftermath of the event.

In Chapter 11, we conclude the report with a brief discussion about next steps – how do we further develop and expand upon the insights gleaned from this early research on social media and severe weather? We offer a number of concrete proposals for future research, including: • Continue to field scientific (random and/or representative) surveys designed to systematically measure public reception of, reliance on, and trust in severe weather information from a variety of traditional and “new” media sources, including social media – is reliance on social media continuing to grow relative to other sources of severe weather information? Or, has the market for social media information plateaued? • Work with forecasters and weather scientists to systematically evaluate the quality of severe weather reports on social media – how often do members of the public publish verifiable weather reports on social media and, more importantly, how accurate are these reports when they are published? • Move beyond the 2013 Newcastle-Moore-South Oklahoma City tornado to study the evolution of social media usage before, during, and after other tornadoes and other types of severe weather – are our findings generalizable to other tornado (and other severe weather) events? Are the patterns evident in this analysis consistent across time, space, and weather phenomena?

This report provides a systematic point of departure for developing an empirically grounded understanding of the roles played by social media in severe weather

INTRODUCTION AND EXECUTIVE SUMMARY PAGE | 12 events. In the pages that follow we provide an assessment of the roles that social media currently play that may contribute to building a Weather-Ready Nation. It is important to note that our findings are taken from a specific period of time, whereas the world of severe weather forecasting, communication, response, and recovery is dynamic and evolving. Thus, in the interest of continuing to develop a Weather- Ready Nation that is resilient in the face of increasing vulnerability to extreme weather and water events, we must continue to study and monitor the changes in technical, natural, and social systems that can facilitate community resilience.

Social media is one such change in the technical system that – in a relatively short period of time – has significantly altered the way in which communication about extreme weather and water events occurs in the social system. In the eyes of many, this change has, to date, facilitated a net increase in community resilience. In the eyes of others, this change could, in the near future, obstruct resilience by making it all too easy for untrained members of the public to send and receive low quality or deceptive information that confuses weather decision makers and other members of the public who rely on that information to make highly consequential, protective action decisions. While the research provided in this report provides some reassurance that roles played by social media are more helpful than hurtful, more and continued research is urgently needed as the NWS continues to experiment with and begins to implement social media in forecasting and warning efforts.

INTRODUCTION AND EXECUTIVE SUMMARY PAGE | 13

2. The Evolution of Reception, Reliance, and Trust: Public Usage of Information from Social Media About Severe Weather

2.1 Introduction In recent years, much has been written about how members of the U.S. public are using social media gather information about extreme weather and water events. As yet, however, we know relatively little about what this “use” entails. For example, we do not know if members of the public using social media instead of traditional sources of information (like television and radio) or in addition to traditional sources of information. Moreover, we know relatively little about how this use has evolved over time. In this chapter, we begin to address these questions by focusing on public usage of social media information for one type of extreme weather event—severe weather. To accomplish this, we address three research questions: 1. Relative to other sources of information, how many people receive information about severe weather from social media sources? Is reception increasing over time? 2. Relative to other sources of information, how much do people rely on information about severe weather from social media sources? Is reliance increasing over time? 3. Relative to other sources of information, how much do people trust information about severe weather from social media sources? Is trust increasing over time?

2.2 Data To answer these questions, we designed and administered a survey to U.S. residents that live in tornado-prone regions of the country. The survey instrument contains approximately 140 questions that gauge perceptions about weather, tornadoes, and warnings, as well as a variety of socio-demographic characteristics, including

THE EVOLUTION OF RECEPTION, RELIANCE, AND TRUST: PUBLIC USAGE OF PAGE | 14 INFORMATION FROM SOCIAL MEDIA ABOUT SEVERE WEATHER

geographic location, income, race, ethnicity, and education. The instrument also contains a battery of questions designed specifically to measure the extent to which respondents receive, rely on, and trust information about severe weather from a variety of sources, including social media platforms such as Facebook and Twitter.

We administered the survey in 2012 and in 2013. The 2012 survey was fielded in eight weekly waves between September 12th and November 1, 2012. In each weekly wave, we collected responses from approximately 500 unique members of an online survey panel that is recruited and maintained by Survey Sampling International (SSI). The 2013 survey was fielded in eight weekly waves between May 8th and June 27, 2013. Again, each weekly wave consisted of roughly 500 unique responses from members of SSI’s online survey panel.

Because we are interested in the extent to which individuals use social media to get information about severe weather (namely tornadoes), we geographically conditioned our selection of potential respondents from SSI’s panel such that the people asked to take the 2012 or 2013 survey had to reside in a “tornado-prone” region of the U.S. Members of the online panel were considered to live in a tornado- prone region if the address they registered with SSI was located in one of the high vulnerability regions listed by Ashley (2007) in his climatological study of significant and fatal tornadoes between 1880 and 2005. In addition to this condition, we oversampled members of the panel that lived in rural settings so as to maximize geographic coverage and combat the urban bias associated with Internet access and participation in web-based surveys (Couper 2000). Finally, we used enforced Census-based demographic quotas during the completion of both surveys to ensure that our respondents are demographically representative of the target populations.3

3 Because of these conditions and quotas, our respondents are, in general, demographically and geographically representative of the target populations. However, our samples—like the majority samples that are drawn from online surveys panels—were not randomly drawn from the target populations (probabilistic). As such, we follow the advice of the American Association for Public Opinion Research (AAPOR) by avoiding estimates of sampling error and precise population values (AAPOR 2010).

THE EVOLUTION OF RECEPTION, RELIANCE, AND TRUST: PUBLIC USAGE OF PAGE | 15 INFORMATION FROM SOCIAL MEDIA ABOUT SEVERE WEATHER

In total, this process yielded data from 3,989 respondents in 2012 and 3,975 respondents in 2013 that reside at the locations depicted in Figure 1.

Figure 1: Approximate Location of Survey Respondents

(a) 2012 (b) 2013

2.3 Findings

Reception To determine how many of our respondents have received information about severe weather from social media sources, we asked them to answer two interrelated questions. First, we asked them if they remember ever receiving a tornado warning for the neighborhood where they live. Approximately 78 percent (n = 3,113; n = 3,092) of the people who responded to the 2012 and 2013 surveys (respectively) said “yes” to this question and, in turn, were asked to answer the following question:

In which of the following ways did you receive the most recent tornado WARNING? Please select all that apply. 1 – Broadcast radio 2 – Weather radio (National Weather Service Radio) 3 – Television 4 – Siren or other alarm 5 – Internet 6 – Social media such as Twitter or Facebook 7 –Word-of-mouth (including telephone or text messages, email, etc.) from family, friends, neighbors, employers, co-workers, etc. 8 – Other sources (please specify)

THE EVOLUTION OF RECEPTION, RELIANCE, AND TRUST: PUBLIC USAGE OF PAGE | 16 INFORMATION FROM SOCIAL MEDIA ABOUT SEVERE WEATHER

Figure 2: Warning Reception by Severe Weather Information Source and Year

90 86% 85% 2012 80 2013 70

60 57% 58% 50 40

29% 30 27% 27% 28% % % Respondents 25% 25% 22%

20 18% 11%

10 7% 6% 4% 0 Television Siren or Weather Broadcast Word of Internet Social Other Other Alarm Radio Radio Mouth Media Sources

As indicated by Figure 2, approximately 7 percent (n = 214) of the 2012 respondents who remembered receiving at least one tornado warning said that they received their last warning by way of social media—among other sources. This percentage jumped to 11 percent (n = 354) in 2013, which represents a modest but statistically significant increase of 4 percent (95% CI: 3.1% – 6.0%).4 These results indicate that social media is slowly penetrating the warning dissemination market, but remains a distant outlier when compared to sources like television, which – consistent with other research (e.g., Hammer and Schmidlin 2002) – was the most commonly cited source of warning reception in both samples.

Reliance To assess how reliant our respondents were on information about severe weather from social media sources, we asked them to indicate how important the information they get from social media is, relative to the information they get from other sources. We accomplished this by way of the following question:

4 Test: two-sample test for equality of proportions with continuity correction. THE EVOLUTION OF RECEPTION, RELIANCE, AND TRUST: PUBLIC USAGE OF PAGE | 17 INFORMATION FROM SOCIAL MEDIA ABOUT SEVERE WEATHER

Warnings and information about severe weather are available from multiple sources that can vary in importance based on availability, location, and preference. On a scale from zero to ten, where zero means not at all important, and ten means extremely important, please indicate the importance to you, personally, of each of the following sources of information about severe weather such as tornadoes. 1 – Broadcast radio 2 – Weather radio (National Weather Service Radio) 3 – Television 4 – Internet web pages focused on weather forecasts, such as those provided by the National Weather Service or the Weather Channel 5 – Social media such as Twitter or Facebook 6 – Word-of-mouth (including telephone calls or texts) from family, friends, neighbors, employers, co-workers, etc. 7 – Automated text or phone notifications from news or emergency services

Figure 3: Importance of Severe Weather Information Sources by Year

9 8.9 8.8 2012

8 7.9 7.8 7.9 7.6 2013

7 6.9 6.8 6.7 6.7 6.7 6.7 6 5 4.3

4 3.9 Mean 3 2 1 0 Television Weather Broadcast Internet Word of Text or Phone Social Radio Radio Mouth Notifications Media

As indicated by Figure 3, social media received a mean importance score of 3.9 among 2012 respondents, which is below the midpoint (5) on the 0 – 10 scale. In 2013, this score increased by roughly 10 percent (0.4) to 4.3, which represents a statistically significant change (95% CI: 0.25 – 0.56).5 As with reception, these

5 Test: two-sample t-test with Welch correction. THE EVOLUTION OF RECEPTION, RELIANCE, AND TRUST: PUBLIC USAGE OF PAGE | 18 INFORMATION FROM SOCIAL MEDIA ABOUT SEVERE WEATHER

figures suggest that the public is becoming more reliant on social media for information about severe weather, but that other, more traditional sources of information like television, weather, and broadcast radio are still more important. It is interesting to note, however, that public reliance on these traditional sources may be declining, whereas relatively new sources of information like social media and automatic text messaging are on the rise.

Trust To determine how much trust our respondents had in information about severe weather from social media sources, we asked them to answer the following question:

Tornado WATCHES and WARNINGS issued by the National Weather Service are provided to the public by various sources. In some cases, national weather information is supplemented with regional or local information from observations and radar. Using a scale from zero to ten, where zero means no trust and ten means complete trust, how much do you trust the accuracy of each of the following sources of information about severe weather that may affect you or your family? 1 – Radio weather broadcasts for your local area from the National Weather Service 2 – Radio weather broadcasts for your local area from regional or local stations 3 – Television weather broadcasts for your local area from national services such as major networks and the Weather Channel 4 – Television weather broadcasts for your local area from regional or local stations 5 – Weather information from social media, such as Twitter and Facebook 6 – Text messages and phone calls from friends and family

THE EVOLUTION OF RECEPTION, RELIANCE, AND TRUST: PUBLIC USAGE OF PAGE | 19 INFORMATION FROM SOCIAL MEDIA ABOUT SEVERE WEATHER

Figure 4: Trust in Severe Weather Information Sources by Year

9 8.8 8.7 8.6 8.6 8.5 8.5 8.4 8.2 2012 8 2013

7 6.6 6.4 6 5 4.4 4 4 Mean 3 2 1 0 Local Weather National Local Friends and Social Television Radio Television Radio Family Media

In 2012 social media received a mean trust score of 4.0, which is just below the midpoint (5) on the 0 – 10 trust scale. In 2013, average trust in social media sources increased by approximately 9.5 percent (0.4) to 4.4, which, again, represents a statistically significant increase (95% CI: 0.22 – 0.50).6 In line with the results on reception and reliance, Figure 4 suggests that public trust in social media is growing, but continues to lag more traditional sources of severe weather information like television and radio. Again, however, public trust in these more traditional sources appears to be declining, whereas trust in relatively new sources of information like social media and automatic text messages is rising.

2.4 Conclusions Our findings in this chapter indicate that, relative to other sources of information, public usage of information from social media about severe weather is rather low. On average, members of the public are far less likely to have received their last

6 Test: two-sample t-test with Welch correction. THE EVOLUTION OF RECEPTION, RELIANCE, AND TRUST: PUBLIC USAGE OF PAGE | 20 INFORMATION FROM SOCIAL MEDIA ABOUT SEVERE WEATHER

warning by way of social media than from traditional sources of warning information such as television, sirens, and radio. The same is true of reliance and trust – on average, public reliance on and trust in social media as a source of severe weather information is rather low when compared to reliance on and trust in other sources of information like television and radio. Despite these relatively low numbers, our findings suggest that social media usage is increasing over time. More people received their last tornado warning via social media (among other sources) in 2013 than in 2012. Likewise, public reliance on and trust in information about severe weather from social media sources like Facebook and Twitter was higher in 2013 than 2012.

When considered in tandem, these findings suggest that residents of tornado-prone areas of the U.S. rely on a mix of information sources. Within that mix, social media lags behind more traditional sources in importance and trust, but on average the gap appears to be getting narrower. But reception of, reliance on and trust in social media is unlikely to be uniform across the population, and some subgroups may be more likely to rely on social media as a warning source than others. The next chapter therefore focuses on variations across population subgroups in reliance on social media as a source of weather warnings.

THE EVOLUTION OF RECEPTION, RELIANCE, AND TRUST: PUBLIC USAGE OF PAGE | 21 INFORMATION FROM SOCIAL MEDIA ABOUT SEVERE WEATHER

3. Defining the User Base: Who Uses Social Media to Collect Information About Severe Weather?

3.1 Introduction In Chapter 2, we explored the public usage of information from social media about severe weather by exploring the evolution of reception of, reliance on, and trust in social media over time and in comparison to traditional sources of severe weather information. In this chapter, we define the social media user base by answering the following questions: 1. Who receives information about severe weather from social media sources? 2. Who relies on information about severe weather from social media sources? 3. Who trusts information about severe weather from social media sources?

3.2 Data To answer these questions, we use the survey instrument and data discussed in Chapter 2. Recall that the instrument contains approximately 140 questions that gauge perceptions about weather, tornadoes, and warnings; a variety of socio- demographic characteristics, including geographic location, income, race, ethnicity, and education; and a battery of questions designed to measure the extent to which respondents receive, rely on, and trust in information about severe weather from a variety of sources, including social media outlets such as Facebook and Twitter. We fielded the survey in 2012 and 2013 to members of an online survey panel that reside in tornado-prone regions of the U.S. This process yielded data from 3,989 respondents in 2012 and 3,975 respondents in 2013 that are geographically diffuse and representative of the U.S. public.

3.3 Methods To help us identify the types of people that are most likely to use social media to collect information about severe weather, we fit a set of three regression models DEFINING THE USER BASE: WHO USES SOCIAL MEDIA TO COLLECT PAGE | 22 INFORMATION ABOUT SEVERE WEATHER?

that assess the extent to which reception, reliance, and trust vary as a function of individual level demographic and socio-economic attributes (income, education, age, gender, race, ethnicity, and urbanicity). To ensure that our findings are consistent across samples and over time, we fit each model twice – once using the data we collected in 2012 and once using the data we collected in 2013.

3.4 Measures

Reception As discussed in Chapter 2, we use three different indicators to measure social media use – reception, reliance, and trust. Recall that reception is measured by asking respondents to answer two interrelated questions. First, we asked if they ever remember receiving a tornado warning for the neighborhood where they live. If they said yes, we asked them this follow-up question:

In which of the following ways did you receive the most recent tornado WARNING? Please select all that apply. 1 – Broadcast radio 2 – Weather radio (National Weather Service Radio) 3 – Television 4 – Siren or other alarm 5 – Internet 6 – Social media such as Twitter or Facebook 7 – Word-of-mouth (including telephone or text messages, email, etc.) from family, friends, neighbors, employers, co-workers, etc. 8 – Other sources (please specify)

In 2012, approximately 7 percent (n = 214) of the respondents who remembered receiving at least one tornado warning said that they received their last warning by way of social media, among other sources. In 2013, this percentage increased to 11 percent (n = 354) in 2013. In the analysis that follows, we treat responses to this question as binary variables where the 7 percent and 11 percent of respondents who received their last warning via social media receive a 1 and respondents who did not receive a 0.

DEFINING THE USER BASE: WHO USES SOCIAL MEDIA TO COLLECT PAGE | 23 INFORMATION ABOUT SEVERE WEATHER?

Reliance Reliance is measured by asking respondents to indicate how important the information they get from social media is, relative to the information they get from other sources. We used the following question to accomplish this:

Warnings and information about severe weather are available from multiple sources that can vary in importance based on availability, location, and preference. On a scale from zero to ten, where zero means not at all important, and ten means extremely important, please indicate the importance to you, personally, of each of the following sources of information about severe weather such as tornadoes. 1 – Broadcast radio 2 – Weather radio (National Weather Service Radio) 3 – Television 4 – Internet web pages focused on weather forecasts, such as those provided by the National Weather Service or the Weather Channel 5 – Social media such as Twitter or Facebook 6 – Word-of-mouth (including telephone calls or texts) from family, friends, neighbors, employers, co-workers, etc. 7 – Automated text or phone notifications from news or emergency services

In 2012, social media received a mean importance score of 3.9. In 2013, the mean score increased to 4.3. Both mean scores are slightly below the midpoint (5) on the importance scale. However, as indicated in Figure 5, responses to this question varied rather significantly across respondents. In 2012, for example, approximately 55 percent of the respondents selected a 4 or lower on this scale, whereas 35 percent of the respondents selected a 6 or higher. In 2013, these percentages were 50 percent and 39 percent, respectively. In the analysis that follows, we attempt to explain this variation by identifying some of the individual level attributes that are systematically associated with the importance score.

DEFINING THE USER BASE: WHO USES SOCIAL MEDIA TO COLLECT PAGE | 24 INFORMATION ABOUT SEVERE WEATHER?

Figure 5: Importance of Social Media as a Source of Severe Weather Information by Year

31% 30 2012 25% 2013 25 20 15

Percent 11%11% 11%11% 10 8% 7% 7% 7% 7% 7% 7% 6% 6% 6% 6% 6% 6% 5% 5% 5% 5 0 0 1 2 3 4 5 6 7 8 9 10 Not at Importance of Social Media Extremely All Important Important

Trust As was described in Chapter 2, trust is measured by asking respondents to answer the following question:

Tornado WATCHES and WARNINGS issued by the National Weather Service are provided to the public by various sources. In some cases, national weather information is supplemented with regional or local information from observations and radar. Using a scale from zero to ten, where zero means no trust and ten means complete trust, how much do you trust the accuracy of each of the following sources of information about severe weather that may affect you or your family? 1 – Radio weather broadcasts for your local area from the National Weather Service 2 – Radio weather broadcasts for your local area from regional or local stations 3 – Television weather broadcasts for your local area from national services such as major networks and the Weather Channel 4 – Television weather broadcasts for our local area from regional or local stations 5 – Weather information from social media, such as Twitter and Facebook 6 – Text messages and phone calls from friends and family DEFINING THE USER BASE: WHO USES SOCIAL MEDIA TO COLLECT PAGE | 25 INFORMATION ABOUT SEVERE WEATHER?

In 2012 social media received a mean trust score of 4.0. In 2013, the mean score increased to 4.4. Like importance, both scores are slightly below the midpoint (5) on the trust scale, but as indicated in Figure 6, trust in the accuracy of social media varied rather significantly across respondents. In 2012, for instance, approximately 53 percent of the respondents selected a 4 or lower on this scale, whereas 33 percent of the respondents selected a 6 or higher. In 2013, these percentages were 49 percent and 37 percent, respectively. Again, the following section endeavors to explain this variation by identifying some of the individual level attributes that are systematically associated with trust.

Figure 6: Trust in Social Media as a Source of Severe Weather Information by Year

21% 20 2012 17% 2013

15 14% 13%

10 9% 9% 9% 8% 8% 8% 8% 8% 8% 8% 8% 8% Percent 7% 7% 6% 6% 5% 5 4% 0 0 1 2 3 4 5 6 7 8 9 10 No Trust in Social Media Complete Trust Trust

Demographic Attributes We measure individual level demographic attributes in a standard manner. Income is measured by asking respondents to estimate their annual household income in 2011 and 2012 (the year before they took the survey). Education is measured by asking respondents to list the highest level of education they have completed. To

DEFINING THE USER BASE: WHO USES SOCIAL MEDIA TO COLLECT PAGE | 26 INFORMATION ABOUT SEVERE WEATHER?

simplify and populate otherwise sparse categories, we use responses to this question to create a binary “college” variable, where respondents who have completed a four-year college degree receive a 1 and respondents who have not completed college receive a 0. We measure age in years by asking respondents how old they are and gender by asking respondents if they are male or female. Race is measured by asking respondents which of the following categories best describes their race – White, Black, American Indian, Asian, Native Hawaiian or Pacific Islander, Two or More Races, or Other. Ethnicity is measured by asking respondents if they consider themselves to be Hispanic or Latino or to have Hispanic or Latino origins. Like education, we simplify race and ethnicity by creating binary “minority” variables, where respondents who identify as Black, American Indian, Asian, Native Hawaiian or Pacific Islander, Two or More Races, or Other, and/or Hispanic or Latino receive a 1 and White, non-Hispanic respondents receive a 0. Finally, we measure the urban/rural/suburban nature of the respondents’ residences by asking the following question:

Which of the following categories best describes the location of your primary residence? 1 – Urban: within the incorporated boundaries of a city or town that provides emergency services such as fire, rescue, and storm warnings for your residence 2 – Suburban: near or in a suburb or town that provides emergency services such as fire, rescue, and storm warnings for your residence 3 – Rural: not within the incorporated boundaries of a city or town; emergency services such as fire, rescue, and storm warnings for your residence usually are provided by county, state, or federal entities

3.5 Findings Table 1 lists the parameters derived from the regression models that predict reception of, reliance on, and trust in information about severe weather provided via social media. These results provide a number of findings that are consistent both across samples and over time. For example, the parameters clearly indicate that younger respondents are more likely than older respondents to receive, rely on, and trust information about severe weather from social media sources like Facebook DEFINING THE USER BASE: WHO USES SOCIAL MEDIA TO COLLECT PAGE | 27 INFORMATION ABOUT SEVERE WEATHER?

and Twitter.

Table 1: Regression Models Predicting Reception of, Reliance on, and Trust in Information About Severe Weather from Social Media Sources

Reception1 Reliance2 Trust3 2012 2013 2012 2013 2012 2013 Suburban (vs. -0.44* -0.37** -0.32* -0.23 -0.12 -0.15 Urban) (0.18) (0.14) (0.13) (0.13) (0.12) (0.12) Rural (vs. -0.27 -0.15 -0.14 -0.02 -0.01 0.05 Urban) (0.19) (0.16) (0.14) (0.14) (0.13) (0.13) Age -0.07*** -0.06*** -0.06*** -0.06*** -0.04*** -0.04*** (0.01) (0.00) (0.00) (0.00) (0.00) (0.00) Education 0.11 0.11* -0.05 0.04 -0.05 0.01 (0.06) (0.05) (0.05) (0.05) (0.04) (0.04) Male (vs. -0.59*** -0.53*** -0.59*** -0.43*** -0.37*** -0.25* Female) (0.16) (0.12) (0.11) (0.11) (0.10) (0.10) Income 0.01 0.03 -0.03 -0.02 -0.00 -0.02 (0.02) (0.02) (0.02) (0.02) (0.01) (0.01) Minority (vs. -0.00 0.11 1.18*** 0.70*** 0.85*** 0.66*** Non-Minority) (0.16) (0.13) (0.13) (0.13) (0.12) (0.11) Intercept 0.25 0.05 7.05*** 6.90*** 6.18*** 6.44*** (0.30) (0.24) (0.24) (0.23) (0.22) (0.21) BIC 1399.88 1965.08 20231.38 20192.76 19517.83 19468.23 N 3015 3000 3833 3814 3844 3833 R2 N/A N/A 0.12 0.09 0.08 0.07 Notes: 1Binomial Logistic Regression; 2Ordinary Least Squares Regression; 3Ordinary Least Squares Regression; *p < 0.5, **p < 0.01, ***p < 0.001

The direction and magnitudes of the effects of age on reception of, reliance on, and trust in severe weather information on social media are illustrated in Figure 7. As indicated by the figure, this effect is statistically significant and substantively sizable. All else equal, the predicted probability that an 18 year old received their most recent tornado warning from a social media source (among others sources) is almost 0.3; this probability falls to less than 0.1 among 40 year olds and approaches 0.01 among people who are 70 or older (|Δp| = 0.29).7 Predicted reliance and trust

7 Predictions represent point estimates that were derived by setting the specified parameters (i.e., age) to their specified values (i.e., 18, 40, and 70) and the other parameters in the model to their mean or modal DEFINING THE USER BASE: WHO USES SOCIAL MEDIA TO COLLECT PAGE | 28 INFORMATION ABOUT SEVERE WEATHER?

follow similar patterns. Predicted reliance falls from 5.5 to almost 2.7 (|Δ| = 2.8) and predicted trust goes from 5.3 to 3.0 (|Δ| = 2.3) as age shifts from 20 to 70.

Figure 7: The Predicted Effect of Age on Reception, Reliance, and Trust

(a) Reception (a) Reliance (a) Trust 7 7 0.35 6 6 0.30 5 5 0.25 4 4 0.20 3 3 0.15 Predicted Trust Predicted Predicted Reliance Predicted 2 2 0.10 Predicted Probability of Reception Probability Predicted 1 1 0.05

95% Confidence Interval 0 95% Confidence Interval 0 95% Confidence Interval 0.00 18 28 38 48 58 68 78 18 28 38 48 58 68 78 18 28 38 48 58 68 78 Age Age Age

Like age, gender appears to influence all three dimensions of social media usage. On average, male respondents were less likely than female respondents to receive, rely on, and trust information about severe weather from social media sources in both samples/years. Though statistically discernable, the substantive effect of gender is relatively small when compared to the effect of age. All else equal, the predicted probability that men received their most recent tornado warning from a social media source (among other sources) is 0.04, whereas the predicted probability for women is 0.06 (|Δp| = 0.02). Similarly, predicted reliance and trust are 3.5 and 3.8 for men, and 4.0 and 4.1 for women, respectively (|Δ|reliance = 0.5; |Δ|trust = 0.3).

Minority respondents were (on average) more reliant on and trusting of information about severe weather from social media sources than were non-minority respondents. Again, however, the substantive differences between minorities and non-minorities are small in comparison to those associated with age, but somewhat larger than the differences between men and women. All else equal, predicted values. |Δp| denotes the absolute change in predicted probability that occurs when the specified values are varied (i.e., 18 vs. 70). DEFINING THE USER BASE: WHO USES SOCIAL MEDIA TO COLLECT PAGE | 29 INFORMATION ABOUT SEVERE WEATHER?

reliance among minorities is 5.0, whereas predicted reliance among non-minorities is 4.0 (|Δ| = 1.0). Predicted trust in social media sources, by comparison, is 4.8 for minorities, and 4.1 for non-minorities (|Δ| = 0.7).

The fourth and final finding that is consistent across year and sample is somewhat surprising. On average, respondents who live in the suburbs were a bit less likely than respondents who live in urban areas to receive information about severe weather from social media sources (|Δp| = 0.03). In and of itself, this is not surprising—in general, urbanites tend to use social media more often than suburbanites (Pew 2013). Rather, the surprising finding (or lack thereof) is that the difference between urban and rural reception was not statistically significant, despite the well documented gap in general social media usage between the two groups (Pew 2013).

Finally, it is worth noting the null effects associated with education and income. Controlling for urbanicity, age, gender, and minority status, we find no evidence of a relationship between education or income and individual reception of, reliance on, or trust in information about severe weather from social media sources. These findings (or lack thereof) are generally consistent with extant research on social media usage more generally (Pew 2013).

3.6 Conclusions Our findings in this chapter indicate (1) that public usage of social media sources to collect information about severe weather varies significantly across individuals and (2) that part of this variation is systematically related to individual-level demographic differences. Most notably, younger people are more likely than older people to receive, rely on, and trust information about severe weather that comes from a social media source. To a lesser extent, the same is true of women and minorities – they too are more likely than men and non-minorities to receive, rely on, and trust information from social media.

DEFINING THE USER BASE: WHO USES SOCIAL MEDIA TO COLLECT PAGE | 30 INFORMATION ABOUT SEVERE WEATHER?

4. Context for and Everyday Use of Social Media Among Forecasters in Warning Forecast Offices

4.1 Introduction To this point, we have focused on the public as either the recipient of weather information via social media or the producer of weather information via social media. This chapter and the next are motivated by a set of important questions that examine forecasters as the recipients of weather information via social media. Specifically, we examine the role of social media in National Weather Service Weather Forecasting Offices (NWS WFOs). In this chapter, we consider: 1. How do forecasters perceive the public’s understanding of weather and weather science? 2. How do forecasters currently use social media?

In the next chapter (Chapter 5) we will focus specifically on the issue of forecaster skepticism toward social media.

4.2 Data To answer these questions, we conducted face-to-face, one-on-one, semi-structured interviews with 44 forecasters and weather scientists (representing 11 different WFOs around the country) and also observed forecasters during severe storm operations in WFOs and throughout the 2014 Storm Prediction Center (SPC)/National Severe Storms Laboratory (NSSL) Spring Hazardous Weather Testbed (HWT) experiments. Interviews, on average, lasted 1.25 hours and were recorded, transcribed, and de-identified. Three NWS WFOs were studied in detail, involving field visits to these WFOs to observe operations and to interview as broad a sample of the forecasters within that WFO as possible. These three WFOs represented 31 of the 44 interviewees (in these cases, the Meteorologist in Charge and as many forecasters as possible/available were interviewed). An additional 8 of CONTEXT FOR AND EVERYDAY USE OF SOCIAL MEDIA AMONG PAGE | 31 FORECASTERS IN WFOs

the 44 interviewees were forecasters from various NWS WFOs from around the country who were interviewed during the 2014 Spring Experiment in the NOAA Hazardous Weather Testbed. This set of 8 WFOs represented a wide range of geographical locations – from the Northeast to the Southeast to the Mountain West and the desert Southwest – which adds to the generalizability of findings associated with the interviews. Finally, 5 of the 44 interviewees were weather scientists who were either administrators or basic scientists (working on developing improved models or radar, were in charge of meteorological research settings, etc.). Table 2 provides a summary of the 44 interviews conducted.

Table 2: Summary of Forecaster Interviews Conducted

NWS WFO Forecasters Non-NWS Weather Scientists Observed WFOs Non-Observed WFOs

31 8 5

4.3 Findings In this chapter, we will divide our discussion of findings into two sections. First, we will discuss how interviewees characterized the public’s understanding of weather and weather science. Second, we will summarize our findings about how forecasters describe their uses of and beliefs about social media.

Forecaster Perceptions of the Public’s Understanding of Weather and Weather Science

Because we are trying to understand the extent to which forecasters trust social media as a source of actionable information for making decisions regarding severe weather, we sought to learn how our interviewees perceive the public’s understanding of weather and weather science. To accomplish this, we refrained from asking general questions about the “intelligence” of the public. Instead, we

CONTEXT FOR AND EVERYDAY USE OF SOCIAL MEDIA AMONG PAGE | 32 FORECASTERS IN WFOs

asked forecasters to discuss their perceptions of the average American’s knowledge about the relatively specialized scientific workings of meteorology.

In general, most forecasters felt that the public to which they provide forecasts understood relatively little about weather science. In describing how to communicate weather information to the public, one forecaster put it this way:

Essentially, dumb it down. And one of my biggest things is, communicate how you're going to communicate. Pretend that you're talking to your parents or your child. Communicate that weather message like you’re pretending to talk to them. And then that's how you should get it across, because … you can't talk about geostrophic flow and omega and all this stuff. What does that stuff mean? Essentially, dumb it down. (#34)

While there was general agreement among the forecasters interviewed that the public’s understanding of weather and weather science was limited, there were a few important caveats to this general trend. For instance, regional differences emerged. Most of the forecasters in the South Central Plains believed that the public in their region was more “weather wise,” particularly in regard to severe, convective weather, like tornadoes. By comparison, most of the forecasters from other parts of the country felt like the people in their regions were ignorant of most weather science. It was notable that one forecaster from outside of the South Central Plains expressed genuine surprise when he saw that television meteorologists in Central Oklahoma would talk about a “dry line” or use the word “convective” to describe weather threats. He said that people in his forecasting region would have no idea what words like that meant. However, despite these regional differences, most forecasters continued to perceive the public as lacking a basic understanding of the fundamental principles of atmospheric physics or the terminology used by forecasters.

Sometimes the challenges in communicating to the public—or, by extension, drawing on public knowledge about the weather—were recognized to be far more complex than what could be fixed by “dumbing it down.” For instance, many

CONTEXT FOR AND EVERYDAY USE OF SOCIAL MEDIA AMONG PAGE | 33 FORECASTERS IN WFOs

forecasters brought up the common problem of communicating forecasts about precipitation to the public. In these cases, it was acknowledged by most forecasters that the “problem” was, in fact, not simply a problem with the public’s education level, but rather it was a more complex problem that combined a) the knowledge- base of the public, b) assumptions about the meaning of various forecasting language (for instance, “probabilities of precipitation”), and c) perceptions coming from experience in different climatic regions of the country. For instance, one forecaster drew a contrast between experiences in a WFO in the southeast and experiences in a WFO in the Central Plains:

Well, it’s very interesting because in [the WFO in the southeast], afternoon days in the summer, we put 20 to 30% chance of thunderstorms because our minimum is 20%. [At the WFO in the Central Plains], the minimum is actually 10% that we can put in. But if we put 20% chance of thunderstorms in the afternoon, and about 20% of the [southeast state] would see rain that afternoon. [In the Central Plains], we put 20% in because that’s kind of the understood minimum in the weather service. We can’t do 10%, and I think we do a pretty good job with the 10% here, but you’ll see a lot of 20% [in the Central Plains] that [means] maybe one storm forms. So the 20% is maybe a little bit…the mentality in [the southeast] is 20% chance, well, it’s probably about 50/50. Whereas the mentality [in the Central Plains] is 20% means…well, I don’t know what it means, honestly. And that is a very good point. I don’t know what it means to people. Some people will say there’s a chance of rain, and any chance of rain means there’s a chance of rain. But I think the general mentality [in the Central Plains], especially through the three drought years we’ve had, is 20%, it’s not going to happen. (#26)

Usage of/Beliefs About Social Media in WFOs Having briefly discussed forecaster perceptions about the public, we turn now to their use of social media as one of many ways that forecasters communicate with the people in their regions. In so doing, we discuss the ways in which forecasters described the usages of social media at their current WFO and their thoughts on social media more generally (who uses it, what they use it for, etc.). In other words, we discuss practices and prevailing perceptions.

CONTEXT FOR AND EVERYDAY USE OF SOCIAL MEDIA AMONG PAGE | 34 FORECASTERS IN WFOs

As we begin this discussion, it is important to note that most of our interviewees were concerned with balancing what they felt they were “supposed to say” (to laud the NWS’s social media initiatives) with what they “really thought” (which was often more complicated than a straightforward embracing of social media in their work). Because we anticipated this ambivalence before beginning the study, the ethnographer conducting the interviews layered many different kinds of questions, including asking about how forecasters used social media in their own lives, and how they thought “regular people” used social media, instead of only focusing on questions about how social media was used within a WFO’s operations. This allowed interviewees to provide a more complex and nuanced expression of their perceptions of social media as a tool in WFO operations.

All but two forecasters reported that they used social media in their personal lives (outside of work contexts). Most of the younger forecasters reported having been early adopters of Facebook, signing up for their Facebook accounts when they were still in university during the period when a Facebook account was tied to university users. Most older forecasters reported that they started using social media either due to exposure to it on a regular basis at work or because a spouse or family member convinced them to join.

The use of social media at WFOs was primarily driven by the use of two platforms, Facebook and Twitter. However, how these platforms were used, who was targeted, and beliefs about the effectiveness of these two platforms differed dramatically between WFOs and even between forecasters within WFOs. All WFOs that were included in this study reported having active Facebook and Twitter accounts; however, in general, most of them reported having significantly more activity on their Facebook accounts.

When asked to explain any differences in their social media “followers,” many forecasters explained these differences as a function of rural vs. urban populations served. Facebook was used by “everybody” they explained, while Twitter was more CONTEXT FOR AND EVERYDAY USE OF SOCIAL MEDIA AMONG PAGE | 35 FORECASTERS IN WFOs

“young” and “urban” (it should be noted that, of the WFOs represented in our study, most of them served large “rural” regions rather than large “urban” regions). Most forecasters, however, did not have a strong, empirically-based sense of who their social media followers were, so many of them admitted to speculating wildly on this topic. They guessed that they had more Facebook followers because there were more “middle-aged women” sitting at home following the weather on Facebook. Others speculated that they had very little social media following at all because the population that they served was “very rural.” Others speculated that, culturally, the population that they served was not a “Twitter population,” which they went on to classify as “urban youth.” Overall, one of the most telling findings from asking forecasters about “who their social media followers were” was the sense that they not only did not know who their followers were, but, that they also had strong stereotypes about these social media followers that were not well-supported by evidence.

In general, these stereotypes came out in the exploratory questions that we asked to all interviewees to gauge their impressions of what the typical users of social media look like. Answers were very diverse; however, they tended to cluster around three themes. (1) “Today, just about everyone uses social media.” Forecasters reported that, unlike in the “early days” of social media, it seemed like “everyone” was using social media today. Forecasters who took this approach to answer this questions, when prompted to say more, tended to say things like “Well, my 12 year old daughter uses it, but so does my 70 year old mother!” Others stressed that social media, which seemed to have been a “young, urban” trend in the early days, now seems to have found its way to remote, rural users. When asked, as a follow-up question, “Who doesn’t use social media?” forecasters tended to temper their responses, and acknowledged that there were people who did not use it: the elderly, the techno-phobic, those who were socially

CONTEXT FOR AND EVERYDAY USE OF SOCIAL MEDIA AMONG PAGE | 36 FORECASTERS IN WFOs

isolated or “don’t like people,” people without internet access, and those who were too poor to afford to be regularly connected through the internet.

(2) “Mostly younger people…” Other forecasters emphasized that social media was primarily being used by “younger people.” For the most part, this view was most common among those few interviewees who self-identified as being reluctant or resistant users of social media. Regardless, the perception that social media is most heavily used by and most influenced by the usage patterns of “younger people” was a common theme among almost all of the forecasters.

(3) “It depends on whether we’re talking about Facebook or Twitter…” This was an important distinction that many interviewees made when answering this question. In general, they believed that Facebook covered a broader demographic – particularly in terms of age – than Twitter. Facebook was seen as something that many older people used, while Twitter was characterized as something that tended to be dominated by younger people. In addition, Facebook and Twitter were also contrasted in terms of “what people use them for” (one of the follow-up questions that was asked to all of the interviewees). Facebook was characterized as mostly being used for personal communication – for sharing news with friends and family, for expressing strong opinions regarding current events, for keeping abreast of news and upcoming events related to very immediate social networks (e.g., a person’s church, a club they belong to, etc.). On the other hand, Twitter was characterized in two ways (again, responding to the question: “What do people use [social media] for?”): some interviewees characterized Twitter as a platform for generally vacuous or narcissistic information – “people posting pictures of what they had for lunch” or “what some celebrity is doing right now.” On the other hand, others characterized Twitter as the best platform for gathering carefully selected news – whether this was general news (from sources like CNN), sports news (receiving news about a person’s favorite team), or news about special

CONTEXT FOR AND EVERYDAY USE OF SOCIAL MEDIA AMONG PAGE | 37 FORECASTERS IN WFOs

interests (for instance, one person described following a Twitter account that shared news about the microbrewing beer world, a hobby in which this interviewee participated). When asked about whether they subscribed to weather-related Twitter accounts or to their own WFO’s Facebook page, almost all forecasters acknowledged that they did.

Besides the ubiquitous, if not always consistent or robust, use of social media as a platform for communicating information from the WFO to the public, all of the forecasters at all WFOs said that there were contexts in which they drew on reports from the public provided through social media while reporting, tracking, or verifying severe weather. One forecaster put it this way:

We ask [for reports of the weather], but we have so many followers now that know just to go ahead and volunteer. But sometimes, you know, if we feel like we’re not getting a lot of reports we’ll fire out a post and say, hey, you know, radar’s showing a band of really heavy snow going through the [X city] area right now, you know, any reports from that area would be really appreciated. You know, we might do something like that. But typically we’ll get a lot of reports even without doing that. Especially, I mean, [X city] might not have been that good of an example because that’s one of our bigger cities, we normally get a lot of reports anyway. But maybe we might say, hey, a band of heavy snow is going through the [mountains]. And then, you know, that might prompt one or two people in a more rural area to chime in and tell us what’s going on. And, you know, I think that data is fairly reliable. Sure, sometimes you’ll get people that will exaggerate or sometimes you’ll get people that will flat-out lie because, I don’t know, they’re just, I don’t know, sociopaths or whatever. You know, you can usually tell when that’s happening, and I think the vast majority of people make a good-faith effort to give us good information. (#41)

At the same time, due to their skepticism of the veracity of social media reports (see Chapter 5: “Skepticism About Social Media”), forecasters were generally unlikely to pass on reports (or, at least, to feel confident about those reports) unless a photograph accompanied the report. A common narrative about how social media reports were integrated into daily operations during a severe weather event often took the following form:

CONTEXT FOR AND EVERYDAY USE OF SOCIAL MEDIA AMONG PAGE | 38 FORECASTERS IN WFOs

And then, say I’m on radar and say [Bill] is working the social media area, and he said, “Oh, report of baseball size hail.” I’ll put that, if it’s deemed [reasonable], I’ll put that in my warning. You know, like: “Baseball-sized hail was reported in [this region]” or something like that. [Interviewer: Even if the report came in from social media and not from one of your spotters?] Yeah. If it has a picture, then, probably yes we would, because that’s more visual validation of it. But, you know, we definitely take the spotter reports more seriously because we went out and trained those folks … and these [social media reports] are just general public – we don’t know them … So, there is still some, you know, trepidation in “do we use it or not?” (#28)

On the other hand, other forecasters reported having developed a social relationship with some of the people who posted reports on social media – interacting with them via social media, reading their posts closely to see how they made observations, even, at times, talking to them on the phone, or asking them to come to official spotter trainings because of the care that they put into their reports. In these cases, just as was the case with “trusted” trained spotters, some forecasters took reports from certain trusted social media posters very seriously. One forecaster described such a situation:

First of all, more so than the geography, it is just knowing – being familiar with the people who are tweeting or posting things. I mean, I have, I can you know, there are certain people that you know we are just kind of on a different list almost that you know, I trust them, I know they are plugged in to what is going on from a variety of sources maybe. One of the best sources I had, she moved away, but she was just like the assignment editor at one of the TV stations. She was listening to every scanner and plugged in to everything and so if she tweeted something, I trusted it. I knew that she would not tweet it unless she had, you know, internally vetted it and thought about it. So a lot of it is that, a lot of it is knowing who is out there and who to follow because if you are just looking at the [weather hashtag], it is knowing that there is a pooling … a signal from the noise is almost impossible. And if you do not know that [then] it is nearly impossible for me to do that [drawing a signal out of the noise] too. I mean it has gotten to where I do less of that - I mean [I am] just looking for those key, the key people and seeing what they are talking about. (#11)

4.4 Conclusions Our findings in this chapter indicate that forecaster willingness to embrace weather reports provided via social media depends on many things, including: (1) CONTEXT FOR AND EVERYDAY USE OF SOCIAL MEDIA AMONG PAGE | 39 FORECASTERS IN WFOs

forecasters’ general perceptions of the public’s knowledge of weather and weather science and (2) forecasters’ general usage of and beliefs about social media.

Regarding the first of these factors, we found that all of the forecasters in our study expressed some concern about the level of knowledge about weather and weather science among the general public. For many forecasters, this concern translates into uncertainties about how much to “trust” reports that come from the public. At the same time, taking into account regional differences, we see that forecasters in some regions of the U.S. believe that the people in their forecasting regions might be more or less knowledgeable than those in other regions of the country. Those who are in regions where they feel their audience is more “weather wise” were often the same forecasters who were more open to the idea of the public providing useful observations/reports on weather phenomena via social media.

Regarding the second factor – forecaster usage of and beliefs about social media – it was, perhaps, remarkable how stereotypical many of the interviewee’s perceptions of social media were. Many of the general observations and themes that emerged from interviews with forecasters – “more young people than older people use social media,” “Facebook is for older people while Twitter is for younger people,” “who knows who is on social media so why should we trust them,” “people use Twitter just to send photos of their meals or for narcissistic reasons,” etc. – are common stereotypes that do not fully acknowledge the diversity of uses for and users of social media.

We believe that the two factors that we have considered in this chapter – a belief in the poor weather knowledge base of the general public and a stereotypical understanding of who uses social media and for what purposes – contribute to the broader skepticism toward the use of social media in WFOs. In the next chapter we will systematically explore forecaster skepticism toward social media.

CONTEXT FOR AND EVERYDAY USE OF SOCIAL MEDIA AMONG PAGE | 40 FORECASTERS IN WFOs

5. Skepticism Towards and Resistance to Social Media in Warning Forecast Offices

5.1 Introduction This chapter builds on Chapter 4, shifting the focus to the degree of skepticism and resistance to the use of reports of weather phenomena via social media in everyday operations in a WFO. Specifically, in this chapter we consider: 1. How do forecasters frame their skepticism about the use of social media reports of weather phenomena? 2. What are the types of skepticism expressed by forecasters, and, are there meaningful differences among these types of skepticism? 3. How do forecasters balance “healthy skepticism” with the potential value of reports about weather phenomena provided through social media?

5.2 Data To answer these questions, we rely upon the interview data described in Chapter 4. Recall that we conducted face-to-face, one-on-one, semi-structured interviews with 44 forecasters and weather scientists that represent 11 different WFOs around the country. The majority (31) of these interviews were conducted in three WFOs that were studied in detail.

5.3 Findings All of the forecasters interviewed in this project expressed some level of skepticism about the use of social media in their job. In some cases, this skepticism was of minimal concern to the forecaster’s willingness to embrace the use of public reports of weather phenomena provided via social media. In other cases, it was clear that this skepticism was a significant obstacle to the use of social media in the work of the forecaster. Regardless, all forecasters expressed a complex and, at times,

SKEPTICISM TOWARDS AND RESISTANCE TO SOCIAL MEDIA IN WFOs PAGE | 41

ambivalent relationship toward social media – something that was captured by the kind of qualitative research methods we drew on for this portion of this research.

To illustrate the complex relationship that many forecasters expressed toward the adoption of social media as a sources of data in their work, we begin with a quote from one forecaster who said:

You know, you'll have people that post certain things and then, somebody else will say, “well, I live there and we didn't get …” – so, you even get some of that, which is entertaining. We just sort of sit back and watch it but – the other thing I'll say, that some of our forecasters – and they do have doubts because I do, too, about some of the information. It's like, well, certainly if it's people giving, reporting a tornado or something really important, you're not going to just run with that. You are going to make sure that's valid. And then, in the other cases, is it really that big of a deal if somebody said they had inch and a half hail and it was only one inch? I mean, who cares? That's part of the thing I try to get across to people – it's only hail. It's not like they said it was softballs and it wasn't. It's – and that's something I struggle with in almost every aspect of what we do, is there's – in some situations, this over – this – it's like a feeling of over-importance of – not that what we do is not important. (#42)

This quote illustrates many of the different, often conflicting, perceptions of social media expressed by forecasters in our study. In this 90 second fragment of a 75 minute interview, the forecaster (1) expressed concerns about the accuracy and consistency of reports of weather phenomena found on social media, (2) distinguished between reports associated with severe events that could endanger life/property (tornadoes or softball-sized hail) and more everyday events, and (3) expressed the opinion that forecasters might put too much emphasis on the minutiae of an overly-precise weather report when reports about a weather event might be helpful even with coarse observations.

In this chapter, we consider the nature of forecaster skepticism toward social media. We have found that skepticism is influenced by many different factors, including the ways in which a forecaster might perceive the public’s knowledge of weather/weather science and the way a forecaster thinks about social media in

SKEPTICISM TOWARDS AND RESISTANCE TO SOCIAL MEDIA IN WFOs PAGE | 42

general (topics explored in the previous chapter). Here, we consider how forecasters explain why and when they are skeptical of weather phenomena reported via social media.

General Skepticism About Adoption of Social Media In general, the older the forecaster, the more likely they were to report that they had only joined social media with great reluctance. One interviewee (#34) distinguished between those who were “36 and younger” (who were comfortable with social media) and those who were “50 and older” (who were “resistant” to social media). In general, the generation in between (37-49 years old) was characterized as a mixed group of forecasters in terms of their willingness to embrace social media. Some of these generational preconceptions were corroborated by our interviews. For instance, many of the “older” forecasters were less likely to have a Twitter account or they only used it infrequently; however, while this was a trend, there were many older users who reported having quickly embraced Twitter as a useful source of condensed information about special topics in which they were interested (e.g., following sports teams, etc.). Often, older forecasters also reported having been reluctantly “talked into” signing up for a Facebook account by spouses or extended family members who wanted to keep in contact with them. In general, older forecasters were more likely to report using social media with more reticence than younger forecasters, though, with the exception of only one interviewee, all of the forecasters in this study reported using some form of social media in their personal lives on a semi-regular basis.

Skepticism About Weather Reports Submitted Through Social Media Most forecasters, regardless of age, region/WFO, or position in the NWS, expressed some level of skepticism regarding the veracity of reports of weather events posted to social media. In general, forecasters divided potentially misleading social media reports into one of two categories – (1) “good intentioned” reports that are untimely and/or inaccurate and (2) “bad intentioned” reports that are intentionally deceptive.

SKEPTICISM TOWARDS AND RESISTANCE TO SOCIAL MEDIA IN WFOs PAGE | 43

“Good intentioned” reports of local weather on social media are meant to be helpful but are not particularly useful to forecasters in real-time, because they are poorly timed and/or of questionable accuracy. When describing the former, some of the forecasters we interviewed recalled instances of social media reports that could have been useful to verify an event and assist with evolving warnings/alerts, if they had been published with a timestamp during the incident as opposed to hours or days after the event had transpired and without a relevant timestamp. When recalling instances of this sort, a few of the forecasters talked about instances where unstamped re-tweets or re-postings of reports reoccurred days or even weeks after an event, which made it difficult to sort between current and outdated reports. For example, one forecaster likened the problem to that of TV stations running old footage of storms that could be confusing to a viewer:

The most frustrating thing is when the TV station decides to run old video from two hours earlier and we glance up and see a tornado on a screen and it’s like, oh my gosh. So there is that – I do not know if I call it a diversion or a distraction. It is a necessary part what we do, but if sometimes those reports – all the information is good, but sometimes, and we teach this in our spotter training class, that you have all the information you can to help avoid this. Sometimes that information creates distractions in the office and makes, creates more work. I mean it is necessary to do that but… sometimes if you get a random report something pops up and you go, well, there’s that – it’s pretty apparent that they are saying something happened in [one city], well the storm is now in [a neighboring city] and there is not even a cloud in [the original city]. That is over simplification. Those are easy but if it is – well there could be a tornado there, there could have been baseball sized hail and maybe the timing’s off, maybe their location’s off. (#11)

Fortunately, the forecasters we interviewed said that these poorly timed social media reports have never (to their knowledge) prompted them to incorrectly issue or verify a warning. Nevertheless, the possibility that this could happen clearly reduced some forecasters’ perceptions of the reliability of social media.

More frequently, forecasters reported concerns about the accuracy of social media reports. It was common for forecasters to emphasize that social media reports

SKEPTICISM TOWARDS AND RESISTANCE TO SOCIAL MEDIA IN WFOs PAGE | 44

“could be coming from anyone,” so there was great skepticism regarding the poster’s capacity to identify or properly measure/record weather events. While forecasters who were interviewed mentioned problems with the misapprehension of rotating wall clouds, dust devils, or gustnadoes as tornadoes, reports about local snowfall levels and the size of hail were mentioned far more frequently. For instance, reports of snowfall that did not seem to match forecasters’ expectations were often dismissed by assuming that the reporter did not properly measure the snow accumulation (e.g., they measured a drift, they did not account for pre-existing snowfall accumulations, they measured snow on the wrong surface, etc.). Similarly, while reports of hail size are infamously difficult to assess, even photos of hail were criticized for their lack of precision (at least, this occurred when hail reports contradicted the expectations of forecasters). It was common to hear forecasters joke about the possible size of hail in photographs that were submitted by the public when there was nothing to provide scale for the hail. One group of forecasters observed during operations joked, when they saw a photo of large pieces of hail in a person’s hand, that the hail was not as large as it seemed because “that’s just a child’s hand!” One of our interviewees, who, during his career, had worked in a WFO in the Northeast, noted that:

In general, it’s the kind of thing, like, if we get trees down, if we hear about trees down on Facebook and we had a warning out, we’ll go, hey, good. And if we didn’t have a warning out we go, uh-oh. You know, I mean, people take it seriously when they hear about that. You know, a lot of times if it’s contradicting – if it’s contradicting a forecast, like if we don’t have a warning out and somebody calls in and says, oh, it looked like a tornado went through here, that’s something we feel like people in the Northeast don’t know – they don’t actually know what a tornado looks like. So we do get a lot. If we hear on Facebook about someone saw a tornado, we take that with quite a bit of skepticism. Now, if trees are down – now, of course, people know what trees down look like. There – and say we don’t have a warning and we get something on Facebook that says that trees are down – there may be a little bit of skepticism there, but that will prompt us to make our calls or try to find out if anybody else is seeing that. (#41)

SKEPTICISM TOWARDS AND RESISTANCE TO SOCIAL MEDIA IN WFOs PAGE | 45

Here, the forecaster is acutely aware of the population that is being served by the WFO. The accuracy of some reports, if trees have been knocked down by a storm, is not something that would be questioned. On the other hand, the accuracy of a report of a tornado, something that the forecaster argues is a rare enough event in the Northeast to question the general public’s capacity to identify it properly, is something that the forecaster would question. This example illustrates a broader theme that was revealed throughout the course of our interviews: there are many different ways in which “accuracy” can be categorized by the public and by forecasters. In the quote that we began this chapter with, the forecaster argues that the difference between one-inch and one-and-a-half inch hail might not be particularly important as long as forecasters are getting confirmation of significant hail on the ground from people who are actually experiencing it. On the other hand, an “inaccurate” report of a tornado might lead to dramatic or unintended consequences. Accuracy, then, while being extraordinarily important for the forecasters interviewed in our research, can be seen as a more subjective category than it might appear at first glance.

While concerns about the timing or the accuracy of reports about the weather coming from social media presumed the public to be well-intentioned if not particularly skilled, by far the most common skepticism expressed by forecasters was that people could post intentionally misleading or deceptive reports via social media. For instance, at one of the WFOs where interviews were conducted, forecasters talked about a recent day (this was during spring convective weather season) when someone posted a photograph of what they claimed was a tornado currently on the ground. From the onset, the forecasters were skeptical of the report; there was nothing on the radar that would suggest the possibility of a tornado and they had not heard anything from anyone else. So, they took a closer look at the picture only to realize that is was a well-known picture of a tornado that had touched down in another state in 2007.

SKEPTICISM TOWARDS AND RESISTANCE TO SOCIAL MEDIA IN WFOs PAGE | 46

This was one of very few examples of this type of actually recorded, intentionally deceptive reporting; but many forecasters, whether they had seen one of these deceptive reports or not, believed that even a single instance of this kind of deceptive reporting represented a broader risk associated with trusting social media for reports about the weather. For instance, one forecaster started telling the interviewer about how valuable a social media report of hail could be for the forecasters, but, quickly transitioned into a general, cautionary statement about the dangers of trusting a “non-truthful” report:

We feel the social media would be a good way to get reports, too. So, if we were trying to build up the followers, in terms of, like, when we need it. Say we get a tornado [here] or like severe weather moves through [the area] … you know, they’ll put on there “baseball size hail fell at 34th and Main” … Of course, you have to kind of weed through some of that because we don’t know these people and there have been instances where people like to post things, non-truthful things, that are going on. (#28)

This kind of skepticism – the belief that the public could provide the forecaster with intentionally deceptive reports – was a narrative that frequently undermined an interviewee’s multiple stories of how helpful weather reports had been when provided via social media.

Skepticism, “Healthy Skepticism,” and Combating Skepticism In most cases, forecasters who were interviewed as part of this study stressed that, rather than being categorically opposed to the use of social media reports of weather phenomena, it was best to have a degree of “healthy skepticism” toward reports provided via social media. At the same time, when an unexpected report provided via social media could be verified, it gave forecasters a growing sense that social media could provide a useful complement to existing tools (models, radar, satellite, etc.). One forecaster described how he was won over by the role of social media reports during a winter storm in the prior year. He said:

It needs to be taken with a grain of salt. Especially when I’m just talking about, say, John Q. Public. If I’m going to watch, as an employee, watch for reports, I

SKEPTICISM TOWARDS AND RESISTANCE TO SOCIAL MEDIA IN WFOs PAGE | 47

give them a public confidence. And what that means is, in our office specifically and I obviously can only speak for our office, but I think it’s a kind of general rule in the Weather Service – public reports have – hold less water than one from a trained spotter. A storm chaser to an extent, and I know we trust them because we know them and so on and so forth. If I know someone is an NWS employee, I’m going to believe them. Public report, questionable until I can feel I can prove it. And what I’m saying by that is, I had an instance in December with a snowstorm. I could not verify an insane rate of snow and we’re getting – we had a forecast of six inches and I’m getting a report of 13 and four and a half of it fell in 30 minutes. That’s very hard to fathom. We’re talking earlier about data and believability. That data that we got was very unbelievable to begin with and it set off a firestorm with me and inside of our office. It was really kind of just me and the one other person that was there verifying it because I did not – and it actually, the sad part about this whole story right here and it’s going to be a caveat to what I just said is, it came from [XXX] a trained spotter. But the report was so unbelievable. To me, at that point, it didn’t matter who it came from. I needed proof and so I went to social media as the last resort. I called other people. Some of them said yes, some said no. I finally went to social media and said—this was Twitter social media, I didn’t go to Facebook, Twitter only. I said, send us your snow reports, especially – or, does anyone in [this location] have a snow report? Pictures preferred. Because a picture to me – I know there are a lot of false pictures going around, so I understand that, but a picture of 13 inches of snow, I’m going to believe. What I put in the report when I then turned around and finally reported it, because it was accurate, as a public report via Twitter so that everyone else knew where I got that information. I feel that in my ability, I verified it to my abilities. I believed it but I want everyone else to know where I got that information as well, so that they know, hey it was on Twitter, or they can look at it for themselves if they want to, based on how social media works. (#22)

While the above quote comes from a forecaster who has tried to show his/her colleagues the value of social media in an ad hoc manner, in other cases, certain people in a WFO have taken a more active role (either formally or informally) in combating forecaster skepticism. Central to this has been the role of Warning Coordination Meteorologists (WCMs) in many (though not all) WFOs. Because WCMs are tasked with, among other things, being active liaisons between the WFO and the users/consumers of their weather forecasts – including Emergency Managers, industry, spotters, and the general public – they are, in general, more favorable in their perception of the value of social media in the broader weather enterprise. One WCM put it this way:

SKEPTICISM TOWARDS AND RESISTANCE TO SOCIAL MEDIA IN WFOs PAGE | 48

I see my role as, I'm, like, one of the lone [social] experts in the office. I feel like my role is to try to keep those guys grounded, as far as, not being too scientific, not being – how to relate their message to the public, how to communicate with the public. And I'm essentially, the liaison with the TV meteorologists or any, kind of, media and emergency managers. So, my job is building relationships and trust. […] And when you're communicating a threat, your job doesn't stop by just issuing a tornado warning and you kick back and put your feet up, like, my day’s done - Where's the newspaper? That's not the end of it. And I like to think that I'm part of a culture change. My age group coming in is just – we're a culture change in the Weather Service. And that’s – I want to be a part of that, leading that way of this isn’t good enough, we have work to do. (#34)

5.4 Conclusions In this chapter, we have identified some of the reasons given by our interviewees for why and when they are skeptical of public reports of weather phenomena provided via social media. The broad categories that we have identified here – particularly, the distinction between “good intentioned” reports and “bad intentioned” reports – is important to highlight because forecaster assumptions about the intentionality of the reporting public should be the topic of future research and forecaster training. We will return to these themes in the final chapter of this report.

SKEPTICISM TOWARDS AND RESISTANCE TO SOCIAL MEDIA IN WFOs PAGE | 49

6. Searching for a Signal in the Noise: A Temporal Comparison of Social Media and Severe Weather Activity

6.1 Introduction As noted in previous chapters, public reception of, reliance on, and trust in information about severe weather from social media sources is growing, but remains low when compared to other, more traditional sources of information. The same is true of forecasters who, in general, believe that social media provides a useful mechanism for sending information to the public but are skeptical about the extent to which public reports of local weather on social media can or should be used to improve/validate their forecasts. As described in Chapter 5, this skepticism stems in part from experience with cases of irrelevant, unreliable, and/or invalid information (“noise”) that appear on social media. When searching for information about the local weather on Twitter, for example, one may have to sort through pictures of grumpy cats in raincoats, misleading forecasts, and outdated warnings before landing upon relevant, credible, and/or valid information. In this chapter, we ask and answer a relatively simple question – is there a “signal” amid all this noise? To answer this question, we systematically study the temporal relationship between social media activity and severe weather.8 If a signal exists, we should detect a meaningful positive relationship between what happens in the atmosphere and what happens on social media.

6.2 Data To study this relationship, we need data on the way in which social media activity and severe weather vary over time. To collect data on social media activity, we developed a program that interfaces with Twitter’s Streaming application

8 For more information on this analysis, see Ripberger, Jenkins-Smith, Silva, Carlson, and Henderson 2014. SEARCHING FOR A SIGNAL IN THE NOISE: A TEMPORAL COMPARISON PAGE | 50 OF SOCIAL MEDIA AND SEVERE WEATHER ACTIVITY

programming interface (API) to continuously collect and archive public messages (“tweets”) on Twitter that contain one or more of the following terms/phrases that are commonly used in social media forums when discussing severe weather: #weather, #wx, bad weather, hail, severe storm, severe weather, thunderstorm, or tornado.9 In addition to the text of each tweet, the program collects and archives a set of metadata provided by Twitter about the tweet itself and the user that published it. With respect to the former, the program collects and archives information such as the date and time that the tweet was published and the number of times it was “re-tweeted” (re-posted by another Twitter user). With respect to the latter, the program collects and archives details like the username of the author of each tweet, the user description and URL (link) the user provided (when available), the number of Twitter users who follow that user, and the latitude and longitude of the user when they published the tweet (when available).10

Between April 24, 2012 and June 30, 2014, we used this Streaming API program to collect and archive nearly 47 million tweets (and associated metadata) that referenced severe weather using one of the abovementioned terms/phrases. As indicated by Figure 8, tornado was the most commonly used key term/phrase in the data we collected.

9 We use Twitter data as opposed to data from other social media platforms, like Facebook, because Twitter has developed a number of application programming interfaces (APIs) that make it possible for third-party researchers to collect and analyze large volumes of high quality data. As yet, it is difficult (if not impossible) to obtain this sort of data from other social media platforms. 10 The collection of these data and the protection of the user identity are undertaken using a protocol approved by the University of Oklahoma Internal Review Board for the protection of human research participants. SEARCHING FOR A SIGNAL IN THE NOISE: A TEMPORAL COMPARISON PAGE | 51 OF SOCIAL MEDIA AND SEVERE WEATHER ACTIVITY

Figure 8: Tweets Collected by Word/Phrase (April 24, 2012 – June 30, 2014)

tornado hail #weather thunderstorm bad weather

Term/Phrase severe storm severe weather Total: 46,875,383 tweets #wx 2000000 4000000 6000000 8000000 10000000 12000000 14000000 16000000 Number of Tweets

As indicated by Figure 9, almost 16 million tweets (and re-tweets) containing the word “tornado” were published in this 26-month timeframe. The most active day was May 21, 2013, the day after an EF-5 tornado touched down in Newcastle, Moore, and Oklahoma City, OK. The second-most active day was May 20, 2013. The third and fourth most active days were November 17, 2013 and April 28, 2014, respectively. The spike in activity on November 17, 2013 corresponds with a deadly outbreak of tornadoes in the Midwest, the costliest and deadliest of which occurred in Illinois. The spike in activity on April 28, 2014 parallels a deadly outbreak of tornadoes that touched down in central and southern parts of the U.S., including Arkansas, Mississippi, and Alabama.

Figure 9: Tweets Containing the Word “Tornado” by Day (April 24, 2012 – June 30, 2014)

1200000 Total: 15,778,288 tweets Mean: 20,491 tweets per day 2013-05-21 1000000 Median: 11,192 tweets per day 800000

600000

400000 2013-11-17 2014-04-28 Number of Tweets Number 200000

0 2012-04-24 2012-05-28 2012-07-01 2012-08-04 2012-09-07 2012-10-11 2012-11-14 2012-12-18 2013-01-21 2013-02-24 2013-03-30 2013-05-03 2013-06-06 2013-07-10 2013-08-13 2013-09-16 2013-10-20 2013-11-23 2013-12-27 2014-01-30 2014-03-05 2014-04-08 2014-05-12 2014-06-15 Date (UTC)

These trends suggest that high impact severe weather events seem to prompt social SEARCHING FOR A SIGNAL IN THE NOISE: A TEMPORAL COMPARISON PAGE | 52 OF SOCIAL MEDIA AND SEVERE WEATHER ACTIVITY

media activity. Tweets containing the word tornado are more frequent when significant events occur. This is consistent with the idea that there is a signal in the noise. However, we would like to know if this signal remains during less visible incidents of severe weather. To answer this question, we compare Twitter activity to weather activity during the relatively “quiet” period on the left-hand side of Figure 9. Specifically, we analyze the temporal relationship between tweets containing the word tornado that were published between April 25, 2012 and November 11, 2012 and tornado predictions/occurrences during that time period.11

To capture the temporal dynamics associated with tornado predictions, we created a set of variables that count the number of tornado watches and warnings issued by the NWS on each day of our analysis. To measure the temporal dynamics associated with tornado occurrences, we created a variable that counts the number of tornadoes that occurred on each day of our analysis. The warning data we use comes from the Iowa Environmental Mesonet,12 the watch data we use comes from the NWS Storm Prediction Center,13 and the tornado data we use comes from the Storm Events Database,14 which is populated by the NWS and maintained by the National Climatic Data Center (NCDC).

6.3 Findings The top pane in Figure 10 plots the number of tweets containing the word tornado that were published each day between April 25, 2012 and November 11, 2012.15 The second and third panes plot the number of tornado watches and warnings issued by the NWS each day and the fourth (bottom) pane plots the number of tornadoes that occurred each day in the same time period. The green stripes on each pane highlight relative “spikes” that occurred in daily Twitter activity during that

11 For a complete account of this analysis, see Ripberger, Jenkins-Smith, Silva, Carlson, and Henderson 2014. 12 Data available here: http://mesonet.agron.iastate.edu/request/gis/watchwarn.phtml 13 Data available here: http://www.spc.noaa.gov/archive/ 14 Data available here: http://www.ncdc.noaa.gov/stormevents/ 15 The “blank” points on this line represent missing data that were not collected because our program went down. SEARCHING FOR A SIGNAL IN THE NOISE: A TEMPORAL COMPARISON PAGE | 53 OF SOCIAL MEDIA AND SEVERE WEATHER ACTIVITY

time period. A quick look at this figure reveals a pattern wherein the majority of spikes in Twitter activity (green stripe) correspond with a spike in at least one of the other variables, suggesting that Twitter activity is sensitive to severe weather forecasts and incidents, even on low intensity days.

Figure 10: Temporal Comparison of Twitter Activity, Tornado Watches, Tornado Warnings, and Tornado Occurrences (Apr. 25, 2012 – Nov. 11, 2012)

Tornado Tweets 140000 120000 100000 80000 60000 40000

Daily Count Daily 20000 0 Tornado Watches 7 6 5 4 3 2

Daily Count Daily 1 0 Tornado Warnings 50 40 30 20

Daily Count Daily 10 0 Tornadoes 20 15 10 5 Daily Count Daily 0 2012-04-25 2012-05-01 2012-05-07 2012-05-13 2012-05-19 2012-05-25 2012-05-31 2012-06-06 2012-06-12 2012-06-18 2012-06-24 2012-06-30 2012-07-06 2012-07-12 2012-07-18 2012-07-24 2012-07-30 2012-08-05 2012-08-11 2012-08-17 2012-08-23 2012-08-29 2012-09-04 2012-09-10 2012-09-16 2012-09-22 2012-09-28 2012-10-04 2012-10-10 2012-10-16 2012-10-22 2012-10-28 2012-11-03 2012-11-09 Date (UTC)

To statistically confirm this visual pattern, we estimated a set of two linear regression models that predict daily Twitter activity as a function of severe weather activity. In the first model, we simply regressed daily tweet counts on severe weather activity. In the second model, we logged daily tweet counts before regressing them on severe weather activity so as to ensure that our results are not biased by abnormally high volume days. Before estimating these models, we used principal component analysis to create a composite indicator of severe weather activity that simultaneously accounts for the number of watches, warnings, and tornadoes on each day in the analysis. This indicator provides a standardized measure of severe weather activity that has a mean of 0, a standard deviation of 1,

SEARCHING FOR A SIGNAL IN THE NOISE: A TEMPORAL COMPARISON PAGE | 54 OF SOCIAL MEDIA AND SEVERE WEATHER ACTIVITY

and ranges from -0.6 to 6. We created this measure to reduce the inherent redundancy associated with watches, warnings, and tornadoes.

Figure 11: Linear Regression Model Predictions of Twitter Activity as a Function of Severe Weather Activity (Apr. 25, 2012 – Nov. 11, 2012)

(a) Model 1 (b) Model 2

95% Confidence Interval 95% Confidence Interval 70000 11.0 50000 10.5 10.0 30000 9.5 Predicted Number of Tweets Number Predicted log(Predicted Number of Tweets) Number log(Predicted 10000 -0.6 0.6 1.8 3 4.2 5.4 -0.6 0.6 1.8 3 4.2 5.4 Severe Weather Activity Severe Weather Activity

Figure 11 graphically displays the predictions derived from this set of linear regression models. As indicated by the figure, there is a strong positive relationship between Twitter activity and severe weather activity in both models. On days in which severe weather activity is relatively low (-0.6), the model predicts that relatively few messages will be published on Twitter (roughly 11,000 tweets will be published). On highly active days (6), by comparison, the model predicts that people will be significantly more active on Twitter (roughly 57,000 tweets will be published). The fact that these findings are consistent across the two (non-logged and logged) models suggests that there is a relatively strong signal in the noise that is not biased by high volume social media days.

6.4 Conclusions Our findings in this chapter indicate that there is a “severe weather signal” in the social media noise, suggesting that relevant, credible, and/or valid information about severe weather likely exists amid the irrelevant, unreliable, and/or invalid noise that has suppressed public and forecaster trust in social media as a source of

SEARCHING FOR A SIGNAL IN THE NOISE: A TEMPORAL COMPARISON PAGE | 55 OF SOCIAL MEDIA AND SEVERE WEATHER ACTIVITY

accurate, credible, and/or timely information. Nevertheless, it is important to recognize that some noise remains, even on extreme weather days when communication about severe weather is most important. Understanding the nature and content of this noise requires a closer look at the tweets themselves – what types of users are active on social media before, during and after incidents of severe weather? Where are they when they publish their messages? What do their messages say? What do the images in their messages contain? The remaining chapters in this report address these important questions.

SEARCHING FOR A SIGNAL IN THE NOISE: A TEMPORAL COMPARISON PAGE | 56 OF SOCIAL MEDIA AND SEVERE WEATHER ACTIVITY

7. Close Up (Part I): Twitter Users Before, During, and After the 2013 Newcastle-Moore-South Oklahoma City Tornado

7.1 Introduction As described in Chapter 6, there appears to be a severe weather signal in the social media noise, suggesting that meaningful information about severe weather likely exists amid the irrelevant, unreliable, and/or invalid information that is published about severe weather before, during, and after a given incident. Nevertheless, it is important to recognize that some noise remains. Understanding the nature and content of this noise requires an in-depth examination of the social media activity that accompanies each incident. We begin this examination by systematically studying the tweets on Twitter that were published before, during, and after a single event – the violent tornado that touched down in Newcastle, Moore, and Oklahoma City, OK on May 20, 2013. In this chapter we focus on exploring the type of users that were active on Twitter throughout the course of this event.

7.2 Data To accomplish this, we use a subset of the Twitter data described in the previous chapter. As mentioned previously, we collected these data by developing a program that interfaces with Twitter’s Streaming API to continuously collect and archive tweets that contain one or more words, like tornado, that are commonly used in social media forums when discussing severe weather. In addition to the text of each tweet, the program collects and archives a set of metadata provided by Twitter about the tweet itself and the user that published it, such as user ID, username, and user description. Between April 24, 2012 and June 30, 2014, we used this program to collect and archive almost 47 million tweets (and associated metadata) that referenced severe weather of some sort.

CLOSE UP (PART I): TWITTER USERS BEFORE, DURING, AND AFTER THE PAGE | 57 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

Figure 12: Tornado Tweets and Users Per Day (May 18 – 22, 2014)

(a) Tweets Per Day (b) Users Per Day 655161 1040650 700000 1000000 500000 600000

399728 300000 Number of Users Number

Number of Tweets Number 217326 307918 188355

118764 200000 62299 30947 100000 25271 0 0 05/18/13 05/19/13 05/20/13 05/21/13 05/22/13 05/18/13 05/19/13 05/20/13 05/21/13 05/22/13 Date Date

To characterize the users who published potentially relevant messages before, during, and after the 2013 Newcastle-Moore-South Oklahoma City tornado, we limited this dataset to the 1,898,007 tweets containing the word tornado that were published on May 18–22, 2013 by 978,933 different Twitter users. This timespan includes the two days before the event occurred (May 18–19, 2013), the day of the event (May 20, 2013), and two days after the event (May 21–22, 2013). As illustrated in Figure 12, over half of the tornado-related Twitter activity during this time period was on May 21, 2013, the day after the event occurred. That said, a significant number of messages were published by a large number of users on the days leading up to the event, the day of the event, and two days after the event.

To characterize the types of users who published these messages, we drew a random sample of 5,000 tweets that were published during this timespan and then used the usernames and user descriptions associated with each account to classify users according to the multi-level scheme depicted in Table 3. The categories in the table were inductively derived using pilot data from a separate sample of tweets containing the word tornado. As shown in Table 3, we classified each user who published a tweet in the sample according to two dimensions. The first dimension differentiates between organizations, individuals, and others (and the subcategories

CLOSE UP (PART I): TWITTER USERS BEFORE, DURING, AND AFTER THE PAGE | 58 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

associated with each of those categories). The second dimension differentiates between users that mentioned the weather in their username or description and those that did not.

Table 3: Twitter User Categories, Example Tweet Descriptions, and Counts of Sampled Tweets

Count of User Category Example Description Sampled Tweets (%) Organization 466 (9.3) Official Twitter of the City of Coppell Fire Government 13 (0.3) Department […] Media The Oklahoman/NewsOK.com […] 316 (6.3) Proudly grilling Nathan's Famous All Beef Hot Commercial 73 (1.5) Dogs […] The Episcopal Diocese of Southwestern Nonprofit 37 (0.7) Virginia Other We are the Penn State Storm Chase Club […] 27 (0.5) Individual 4,065 (81.3) Unaffiliated Stay at home mom, liberal, progressive […] 3,622 (72.5) Government Private First Class in the United States Army 18 (0.4) Affiliate […] Media Affiliate Associate Editor, http://weather.com […] 265 (5.3) Commercial President, Valdes Entertainment Enterprises 95 (1.9) Affiliate […] Nonprofit Affiliate Salvation Army Officer […] 12 (0.2) Other Affiliate The latest Carrie Underwood gossip […] 53 (1.1) Other 467 (9.3) Jornal-laboratório universitário Unisanta Not English 157 (3.1) Online […] Not Enough Straight Edge 310 (6.2) Information

Weather Association Weather Producer for Weather Center Live […] 213 (4.3)

As with most attempts to quantify qualitative (text) data, some of the classification decisions were necessarily based on informed judgment. For example, some users listed multiple affiliations in their descriptions. In those instances, we did our best to classify users according to their “primary” affiliation, which occasionally required a subjective decision. To measure this subjectivity we employed a standard inter- CLOSE UP (PART I): TWITTER USERS BEFORE, DURING, AND AFTER THE PAGE | 59 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

coder reliability test, called Fleiss’ kappa (Fleiss 1971). The test required three different researchers to independently classify a sample of 500 tweets (from the larger sample of 5,000) according to both dimensions. We then used Fleiss’ kappa to assess inter-rater reliability. The kappa value associated with the first dimension was 0.54, which is surprisingly high given the multi-level complexity of the scheme. The kappa value associated with the second dimension was 0.88, which is high but not surprising given the simplicity associated with the classification task. In both cases, our inter-coder reliability results confirmed that the coding process was sufficiently reliable for this analysis.

7.3 Findings As indicated in Table 3, approximately 80 percent of the Twitter users who published tweets containing the word tornado before, during, and after the Newcastle-Moore-South Oklahoma City tornado were individuals who did not mention an organizational affiliation in their description. Of the individual users who provided an affiliation, the media was the most common, followed by commercial, government, and then nonprofit affiliations. Roughly 10 percent of the tweets published during this time period were authored by organizations. Media organizations were the most active organizations during this time period, followed by commercial, nonprofit, and then government organizations. The remaining set of users that posted messages containing the word tornado during this time period (roughly 10 percent) could not be classified along the first dimension because they did not provide enough information in their description or the information they provided was not in English.

Turning to the second dimension, we found that 4.3 percent of the Twitter users we analyzed indicated an interest, association, or expertise in the weather. Approximately 70 percent of these users were classified as individuals (mostly unaffiliated) and 30 percent were classified as organizations (mostly media). When considered in tandem, these findings suggest that the preponderance of severe

CLOSE UP (PART I): TWITTER USERS BEFORE, DURING, AND AFTER THE PAGE | 60 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

weather information published on Twitter before, during, and after this event was provided by “ordinary” (unaffiliated) members of the public, not organizations, and most of these individuals were not identified as associated with the weather enterprise.

Table 4: Time Categories, Time Periods, and Counts of Sampled Tweets

Count of Time Category Time Period (CDT) Sampled Tweets (%) 2013-05-18 00:00:00 - 2013-05-19 Days Before Event 543 (10.9) 23:59:59 2013-05-20 00:00:00 - 2013-05-20 Morning Before Event 51 (1.0) 12:13:59 2013-05-20 12:14:00 - 2013-05-20 Hours Before Event 36 (0.7) 14:55:59 2013-05-20 14:56:00 - 2013-05-20 During Event 99 (2.0) 15:35:59 2013-05-20 15:36:00 - 2013-05-20 Hours After Event 1,691 (38.8) 23:59:59 2013-05-21 00:00:00 - 2013-05-22 Days After Event 2,578 (51.6) 23:59:59

With these (baseline) figures in mind, we proceeded to explore the evolution of user types before, during, and after the event. To accomplish this, we subset the coded sample of tweets by the time periods listed in Table 4 and then looked at the percentage of tweets published by each user type during each time period. Figure 13 plots the results of this procedure. The solid (pointed) lines show these percentages by category and time period and the dashed lines indicate mean percentages by category over the course of the event.

CLOSE UP (PART I): TWITTER USERS BEFORE, DURING, AND AFTER THE PAGE | 61 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

Figure 13: The Evolution of Twitter Usership Before, During, and After the Event

100 Individuals 100 No Weather Association 80 80 60 60 40 40

Percent of Percent Tweets Organizations of Percent Tweets

20 20 Weather Association 0 0

Days Morning Hours During Night Days Days Morning Hours During Night Days Before Before Before Event After After Before Before Before Event After After Time Period Time Period

As indicated by the figure, individual Twitter users with no identifiable affiliation with a weather association were responsible for publishing the majority of messages we coded in each time period. A comparison of the solid lines to the dashed lines, however, reveals an interesting temporal pattern. In the days before the event, individual users were more active than average, whereas organizations were less active than average. In the morning and hours before the event, this pattern flipped – individual Twitter users were less active than normal, whereas organizational users (like the NWS and other governmental agencies) were more active than normal. During and after the event, this relationship returned to the “days before” state wherein individuals were publishing more messages than usual and organizations were publishing fewer.

A similar, though somewhat delayed and less dramatic pattern characterizes the relationship between users with a self-defined weather association and users with no such identifiable association. In the days before the Newcastle-Moore-South Oklahoma City tornado, users with a weather association were a bit more active than normal, whereas users with no affiliation were a bit less active than average. In the morning before the event, groups regressed to their respective means. In the hours before the event and the minutes during which the tornado was on the

CLOSE UP (PART I): TWITTER USERS BEFORE, DURING, AND AFTER THE PAGE | 62 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

ground, this split reemerged – weather enthusiasts were more active than usual and others were less active. After the event occurred, the pattern changed – users with ties to the weather community were less active than average, whereas users with no such ties were more active than average.

There are a number of factors that may explain these trends. It may be, for instance, that organizations and/or individuals with an interest in severe weather were more active than normal in the hours leading up to the event because they were busy publishing information about the dangerous storm that was developing (i.e., tornado watches and warnings). As the event unfolded, organizations and weather enthusiasts may have stepped aside (a bit) as unaffiliated individuals began posting and re-posting messages about the catastrophic event that had transpired.

7.4 Conclusions Our findings in this chapter indicate that individual users were responsible for the majority of messages containing the word “tornado” on Twitter before, during, and after the Newcastle-Moore-South Oklahoma City tornado. Most of these individual users made no mention of an organizational affiliation in their description. Organizations (including media, governmental, commercial, and nonprofit organizations) and unidentifiable users, by comparison, published a relatively small portion of the messages. Also, our findings suggest that the overwhelming majority of messages published before, during, and after the event were published by users that are not tied, by way of self-identification, to organizations that focus on the weather. Our results also point to a number of interesting temporal trends, wherein the characteristics of users change throughout the course of the event. These trends may in part relate to differences in the types of messages that these users were publishing. In Chapter 9, we revisit this possibility when discussing the content of messages published by individuals and organizations before, during, and after the storm. Before turning to that issue, however, we provide a brief look at the locations of the users when they published their messages.

CLOSE UP (PART I): TWITTER USERS BEFORE, DURING, AND AFTER THE PAGE | 63 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

8. Close Up (Part II): The Location of Twitter Users Before, During, and After the 2013 Newcastle-Moore- South Oklahoma City Tornado

8.1 Introduction In the previous chapter, we undertook an in-depth examination of the social media activity that accompanied the violent tornado that touched down in Newcastle, Moore, and Oklahoma City, OK on May 20, 2013. In this chapter, we continue this examination by looking at the location of Twitter users as they posted potentially relevant messages. Again, our goal in this analysis is twofold: 1. characterize the nature and evolution of the social media activity that surrounded the event; and 2. identify signals that may exist amidst the social media noise.

8.2 Data To accomplish these goals, we use a subset of the Twitter data as described in detail in Chapter 6. Recall that we collected these data by developing a program that interfaces with Twitter’s Streaming API to continuously collect and archive tweets that contain one or more words, like tornado, that are commonly used in social media forums when discussing severe weather. In addition to the text of each tweet, the program collects and archives a set of metadata provided by Twitter about the tweet itself and the user that published it, such as user ID, username, user description, and (when available) geographic location. Between April 24, 2012 and June 30, 2014, we used this program to collect and archive nearly 47 million tweets (and associated metadata) that referenced severe weather of some sort.

CLOSE UP (PART II): THE LOCATION OF TWITTER USERS BEFORE, DURING, AND PAGE | 64 AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

Figure 14: Tornado Tweets and Geolocated Tornado Tweets Per Day (May 18 – 22, 2014)

(a) Tweets Per Day (b) Geolocated Tweets Per Day

1040650 9926 10000 1000000 8000

6173 6000 600000

399728 3870 Number of Tweets Number 4000 307918 2814 Number of Geolocated Tweets of Geolocated Number

118764 2000 200000 30947 498 0 0 05/18/13 05/19/13 05/20/13 05/21/13 05/22/13 05/18/13 05/19/13 05/20/13 05/21/13 05/22/13 Date Date

To identify the geographic location of users who published potentially relevant messages before, during, and after the 2013 Newcastle-Moore-South Oklahoma City tornado, we limit this dataset in two ways. First, we limit the dataset to the tweets containing the word tornado that were published between May 18th and 22nd, 2013. This timespan includes the two days before the event occurred (May 18-19, 2013), the day of the event (May 20, 2013), and two days after the event (May 21-22, 2013). Second, we limit the dataset to the tweets published during this time period by users that (1) had enabled Twitter’s geolocation service when publishing their message from (2) a device that is capable of providing geographic information (i.e., an iPhone, iPad, or Android device). When these two conditions were met, our program automatically collected and archived the geolocation (latitude and longitude) of the user when they published their tweet. As suggested by Figure 14, very few (1.23 percent) of the tweets we collected met these two conditions. This is consistent with other research in the area – estimates suggest that somewhere between 1 and 2 percent of all tweets are published with geographic information attached (Burton et al. 2012).

CLOSE UP (PART II): THE LOCATION OF TWITTER USERS BEFORE, DURING, AND PAGE | 65 AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

The fact that very few tweets contain precise geographic information is unfortunate in that it limits the potential utility of information on Twitter for NWS and affiliate organizations. Weather reports, for example, are more useful to forecasters when they know exactly where a social media user is when he publishes his message. Nevertheless, it may be the case that the geographic information provided by this small sample of geolocated tweets provides a valid proxy for the geographic distribution of all the tweets published during this time period. If this is the case, then we can use this relatively small sample of tweets to draw inferences about where, in general, Twitter users were when they published messages containing the word tornado before, during, and after the Newcastle-Moore-South Oklahoma City tornado.

Before we turn to these inferences, however, we briefly evaluate the validity of using geolocated tweets as a rough proxy for all of the Twitter activity that occurred around the event. We do so by assessing the relationship between the major user types outlined in the previous chapter (individual, organization, and other) and the provision of geographic information. Specifically, we use the coded sample of 5,000 tweets to estimate a set of bivariate logistic regression models that allow us to assess the probability that different types of users included geographic information in the tweets they published. If certain types of Twitter users, such as representatives of organizations, were more likely to include geographic information in their tweets than other types of users, then the small sample of geolocated tweets that we have collected is likely biased towards that type of user. Such a bias would compromise the validity of the sample and, as a result, complicate the inferential process. This is especially true if that bias goes undetected and unmeasured.

8.3 Findings Figure 15 plots the results of the validity evaluation described above. As indicated by the figure, there were no statistically significant differences in the provision of

CLOSE UP (PART II): THE LOCATION OF TWITTER USERS BEFORE, DURING, AND PAGE | 66 AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

location information across user types – organizations were not, for example, more likely than individuals to include location information in the tweets they published before, during, and after the event. There were, however, slight differences in the point estimates across categories, suggesting that inferential caution and continued vigilance is necessary. In future research with larger sample sizes, these differences may emerge as statistically significant. Should that be the case, adjustments will have to be made to account for the measured bias.

Figure 15: Predicted Probability of Location Information by User Type

Individual

Organization

Prediction and 95% CI Other Mean Probability

0.000 0.005 0.010 0.015 0.020 0.025 0.030 0.035 Predicted Probability

For purposes of the present report, we see no need for such adjustments, and proceed as if the location information that we collected from approximately 1 percent of the tweets published during this time period is roughly representative of all the tweets that were published. With that in mind, we divided the geolocated tweets into the time increments listed in Table 5 and then plotted them on the Contiguous United States (CONUS) maps displayed in Figure 16.

CLOSE UP (PART II): THE LOCATION OF TWITTER USERS BEFORE, DURING, AND PAGE | 67 AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

Table 5: Time Categories, Time Periods, and Counts of Geolocated Tweets

Count of Time Category Time Period (CDT) Geolocated Tweets (%) 2013-05-18 00:00:00 - 2013-05-19 Days Before Event 5,953 (25.6) 23:59:59 2013-05-20 00:00:00 - 2013-05-20 Morning Before Event 451 (1.9) 12:13:59 2013-05-20 12:14:00 - 2013-05-20 Hours Before Event 327 (1.4) 14:55:59 2013-05-20 14:56:00 - 2013-05-20 During Event 471 (2.0) 15:35:59 2013-05-20 15:36:00 - 2013-05-20 Hours After Event 7,104 (30.5) 23:59:59 2013-05-21 00:00:00 - 2013-05-22 Days After Event 8,972 (38.5) 23:59:59

Figure 16: Geolocated Tornado Tweets by Time Period

As suggested by Figure 16(a), the majority of tweets published in the days leading up to the Newcastle-Moore-South Oklahoma City tornado came from states in the Midwest, including Oklahoma, Kansas, Nebraska, Missouri, Illinois, Iowa, Minnesota, CLOSE UP (PART II): THE LOCATION OF TWITTER USERS BEFORE, DURING, AND PAGE | 68 AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

and Wisconsin. That cluster of activity corresponds with the outbreak of tornadoes that occurred in that region of the country on May 18th and 19th, 2013. In addition to a number of smaller tornadoes, that outbreak produced EF-4 tornadoes near Rozel, KS on the 18th and Shawnee, OK on the 19th. On the morning before the event at hand (Figure 16(b)), the large cluster of activity in the Midwest dissipated in favor of a few significantly smaller pockets of activity in population centers and in areas of the country that were affected by the tornadoes. As the morning marched on and the May 20th storm approached (Figure 16(c)), a new cluster of activity formed around the area encompassed by the tornado watch that was issued by the SPC at 1:10 PM CDT and the multitude of tornado warnings that were issued in and around that area before the storm produced the tornado. As the tornado touched down (Figure 16(d)), that new cluster of Twitter activity collapsed into a small mass of activity in central Oklahoma that was accompanied by a number of isolated messages in other parts of the country. In the hours after the tornado occurred (Figure 16(e)), Twitter users from across the country began chiming in. There was, however, a regional flare to this national activity – the majority of messages published during this time period came from Oklahoma. In the days after the tornado occurred, this regional flare gave way to a national and international discussion about tornadoes on Twitter.

There are a number of factors that may explain this evolution of activity. The data suggest, for example, that in the days leading up to the event, Twitter users were discussing the tornadoes that occurred on the 18th and 19th. Attention to these tornadoes was significant, but not sufficient to spark the sort of national and international discussion that the Newcastle-Moore-South Oklahoma City tornado appears to have sparked. Thus, the discussion was largely regional in nature. On the morning/early afternoon of May 20th, discussion of these tornadoes appears to have died down as people began discussing the risk of the day. Again, this discussion was largely confined to the region of the U.S. that was most likely to be affected. The discussion was even more limited, both in volume and geographic scope, while the tornado was on the ground, suggesting that the discussion was largely focused on

CLOSE UP (PART II): THE LOCATION OF TWITTER USERS BEFORE, DURING, AND PAGE | 69 AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

what was happening and where. As the tornado lifted, news of the devastation slowly rippled out from central Oklahoma to rest of the state, region, country, and then to the rest of the world, suggesting (perhaps) another shift in the discourse surrounding the event.

8.4 Conclusions Our findings in this chapter indicate that relatively few Twitter users enable Twitter’s geolocation feature when publishing information about severe weather. This limits the potential utility of the information they provide. However, preliminary analysis indicates that the sample of Twitter users who chose to enable this feature is not heavily biased towards one type of user. As such, we can, with caution, use this sample to draw some inferences about the general location of Twitter users when they publish messages about severe weather before, during, and after significant events. When we did this for the Newcastle-Moore-South Oklahoma City tornado, a relatively clear signal emerged – before the tornado occurred and while it was on the ground, areas of the country that were recently affected by adverse weather (and/or likely to be affected by adverse weather in the near future) were more active than people located in areas of the country that were not affected. After the event occurred, the discussion broadened to include those who were not directly affected. This trend may in part signal a shift in the composition of social media networks and (as a result) the content of social media messages over the course of the event. In the next chapter, we explore this possibility by examining the content of Twitter messages before, during, and after the time the tornado occurred.

CLOSE UP (PART II): THE LOCATION OF TWITTER USERS BEFORE, DURING, AND PAGE | 70 AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

9. Close Up (Part III): The Evolution of Message Content Before, During, and After the 2013 Newcastle- Moore-South Oklahoma City Tornado

9.1 Introduction In Chapters 7 and 8, we began an in-depth analysis of a case study of a high-profile severe weather event by looking at the social media activity that accompanied the violent tornado that touched down in Newcastle, Moore, and Oklahoma City, OK on May 20, 2013. In Chapter 7, we focused on who, in general, was using social media to communicate about severe weather. In Chapter 8 we looked at where these users were located when they posted their messages. In this chapter, we take this analysis one step further by looking at the content users were posting on social media as the event unfolded.

9.2 Data To accomplish this, we use the sample of 5,000 tweets described in Chapter 7. As mentioned previously, we collected these data by developing a program that interfaces with Twitter’s Streaming API to continuously collect and archive tweets that contain one or more key words, such as tornado. Between April 24, 2012 and June 30, 2014, we used this program to collect and archive almost 47 million tweets that referenced severe weather. To characterize the content of potentially relevant messages that were published before, during, and after the 2013 Newcastle-Moore- South Oklahoma City tornado, we did the following: (1) limited this dataset to the 1,898,007 tweets containing the word tornado that were published on May 18-22, 2013; (2) drew a random sample of 5,000 tweets from this limited dataset; and then (3) hand classified messages according to the multi-level scheme depicted in Table 6, which was inductively derived using pilot data from a separate sample of tweets containing the word tornado.

CLOSE UP (PART III): THE EVOLUTION OF MESSAGE CONTENT BEFORE, PAGE | 71 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

Table 6: Content Categories, Example Message Text, and Counts of Sampled Tweets

Count of Content Category Example Message Text (Tweet) Sampled Tweets (%) Information Dissemination 2,959 (59.2) Tornado Warning issued for southern Alert 544 (10.9) Atoka County […] Preparation/Prevention A few tips for Tornado Season […] 49 (1.0) The 2013 Moore Oklahoma Tornado – a General Report 1,324 (26.5) synopsis […] President Obama pledges help after Policy/Politics 203 (4.1) Oklahoma tornado […] Oklahoma church opens its doors to Recovery Efforts 194 (3.9) tornado victims […] Kevin Durant has donated $1 million Relief Efforts 645 (12.9) dollars […] Personal Expression 1,452 (29.1) So sick of commercial media focus on US General Commentary 739 (14.8) tornado […] Response/Experience Tornado Warning. In the basement. 126 (2.5) Prayers and support in wake of Support/Encouragement 587 (11.7) Oklahoma tornado […] Other 587 (11.7) Satélites de la Nasa fotografían el tornado Not English 482 (9.6) […] Not Enough Information #tornado 105 (2.1)

As indicated by Table 6, we classified each message in the sample into one of three major categories – information dissemination, personal expression, and other. Then, we classified the messages within each of the first three major categories by sub- category. As was the case when coding for user type in Chapter 7, some of the content classification decisions we made required coders to make judgments about appropriate categorizations. To measure the consistency of these judgments across different coders, we asked two different researchers to independently classify a sample of 500 tweets (from the larger sample of 5,000) and then used Fleiss’ kappa

CLOSE UP (PART III): THE EVOLUTION OF MESSAGE CONTENT BEFORE, PAGE | 72 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

to assess inter-rater reliability. The kappa value was 0.92, which is very high given the complexity of the scheme.

9.3 Findings

Figure 17: Word Clouds of Sampled Tweets in Top Three Information Dissemination Categories

(a) Report about the Event (b) Relief Efforts (c) Alert

expected destruction kobebryant newcastle including photos thoughts bleacherreport shawnee storm thunder possible severestudios examiner prayforoklahoma heres scienceporn central history miles interviewpath one just efforts star coming mphjust kansas emergency hits says prayers threat ef5 cnnbrkmay hopepledges dailythunder western live twcbreaking update hugs child tollstate donating breaking weather world kdtrey5 donated confirmed ground wichita miles ripswatch videomedical millionplease cover moore nearnorth damage dogdeath knowevery durant texting will edt update dead cnn just reddonate getcountiesoklahoma may among benefitfund willhel 100 south lordkfor oklamoore hit live nba bestreliefhelp waytoday heading large storm i35 team watch wolf atheist kevin 90999find storms right severe east httpt alive okla moore victims stay warning okc today httpoklahoma can http take now schoolcity least rubble okwx retweet oklahomaokcgive effect cdtokwx west okc people survivor killed wide dollars redcross get city metro movingcounty city violent photo destroyed donationscross text new children finds okwx giving seek shelterissuedarea hail safe near news massive nowhomes disaster donation steps examiners 911buff weather nwstexas dallas office teacher blake redcrossokc donatesaid kswx breaking dangerous confirmed area prayforoklahoma affected shelton warnings devastating elementary sportscenter weatherchannel http spotted blitzer supportpledge suburb food people nwsnorman buried helping another

As indicated in Table 6, almost 60 percent of the tweets containing the word tornado that were published before, during, and after the Newcastle-Moore-South Oklahoma City tornado were written to disseminate information of some type. Most commonly, this information took the form of a “general report” about the event itself. As suggested by the words highlighted in Figure 17(a), these general reports included basic statistics about the tornado (i.e., size, location, etc.), its impact on society (i.e., damage estimates and fatality/injury reports), and a number of human-interest stories (i.e., “Woman Finds Dog in Rubble”). Organizations, as described in Chapter 7, were a bit more likely than individuals to publish information of this type. The second-most common type of information contained in this category of messages related to relief efforts and how to help victims by donating to organizations such as the Red Cross. These messages came from individuals and organizations alike. After general reports and information about relief efforts, messages containing information about pending threats (i.e., alerts, warnings, watches, etc.) appeared most frequently in the information dissemination category. The vast majority of

CLOSE UP (PART III): THE EVOLUTION OF MESSAGE CONTENT BEFORE, PAGE | 73 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

these “alert” messages were published by organizations and re-published (re- tweeted) by individuals. The remaining messages in the information dissemination category contained information about policy/politics, recovery efforts, and/or information about how to prepare for and/or mitigate the damage caused by tornadoes. Though significant, these messages appeared less frequently and came from individuals and organizations.

Roughly 30 percent of the tweets published during this time period were written from a first person perspective. Rather than disseminating “objective” information about the event, the majority of these messages contained “subjective” comments, feelings, and/or opinions about the Newcastle-Moore-South Oklahoma City tornado or an expression of support/encouragement for those affected by it. As indicated by Figure 18, many of the messages we classified as “general commentary” expressed a sense of shock or sadness brought on by the event. Individual Twitter users were responsible for the vast majority of these messages. In the messages we classified as “support/encouragement,” Twitter users (both individuals and organizations), voiced their condolences, thoughts, and prayers for those affected. The other messages in this category included first person accounts of what the authors were doing/experiencing when they published the tweet. These messages were rather diverse in nature, ranging from simple comments about watching the storm as it developed to more elaborate messages about where (specifically) the author was as the tornado passed and what (specifically) the tornado did to their house/property. Again, the majority of these “response/experience” messages came from individual users.

CLOSE UP (PART III): THE EVOLUTION OF MESSAGE CONTENT BEFORE, PAGE | 74 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

Figure 18: Word Clouds of Sampled Tweets in First Person Expression Categories

(a) General Commentary (b) Support/Encouragement (c) Response/Experience

condolences heading always horrible peoplesome destroyed never hopeplease lightning today want right wide family edmond basement take dog young homelesstweetlikeagiri right miles coming shit warning lives children time daughter love love kids moore away impacted todaygettingwatching way lol mile watching coverage loved goespray everyone hear comes awaydamn alley know see damage safe watchclose city truck like house hit lost help landedbadhope say theres dontgetlive now ive thoughtsheartsevere live delivers effected hearts rain sirens night wow widesad just watch come families prayers todayheaven get hit still man hit city stay house last now season thats justjoin hurt cant okcgoodgod just got work friend even oklahoma news sadoklahomadied can place hear one news badheart bless going warningmoore lol much will like guys okc city affected victims near news midst family homehope peopleschoolcan closeprayforoklahoma rip god coming oklahoma north think time moore cant seenstorm home chaoticstorm outbreakspraying people gained couple kids victims prayforoklahoma got still near pontifex moodyd shelterfirst good video tornados devastatingdead made back damage drivingeerie reallygoing dozens passed way two god devastation especially massive kansas die devastation please heard devastatedhundreds crazy weather storm friends sirensouth devastating footage sendingsend coverage tornadoheaven

The content of the remaining set of messages containing the word tornado that were published before, during, and after this event (slightly more than 11 percent) could not be classified because it was insufficiently specific to allow for categorization and/or it was not in English. Most of these messages came from the “other” category of users described in Chapter 7.

Table 7: Time Categories, Time Periods, and Counts of Sampled Tweets

Count of Time Category Time Period (CDT) Sampled Tweets (%) 2013-05-18 00:00:00 - 2013-05-19 Days Before Event 543 (10.9) 23:59:59 2013-05-20 00:00:00 - 2013-05-20 Morning Before Event 51 (1.0) 12:13:59 2013-05-20 12:14:00 - 2013-05-20 Hours Before Event 36 (0.7) 14:55:59 2013-05-20 14:56:00 - 2013-05-20 During Event 99 (2.0) 15:35:59 2013-05-20 15:36:00 - 2013-05-20 Hours After Event 1,691 (38.8) 23:59:59 2013-05-21 00:00:00 - 2013-05-22 Days After Event 2,578 (51.6) 23:59:59

CLOSE UP (PART III): THE EVOLUTION OF MESSAGE CONTENT BEFORE, PAGE | 75 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

With these (baseline) figures in mind, we proceeded to explore the evolution of message content throughout the course of the event. To accomplish this, we subset the coded sample of tweets by the time periods listed in Table 7 and then looked at the percentage of tweets published by content classification during each time period. Figure 19 plots the results of this procedure. The solid lines (with points) show these percentages by category and time period and the dashed lines indicate mean percentages by category over the course of the event.

Figure 19: The Evolution of Message Content Before, During, and After the Event

(a) Information Dissimination (b) Personal Expression Alert 50 25 General Commentary 40 General Report 20 30 15

Support/Encouragement 20 10 Percent of Percent Tweets of Percent Tweets 5 10

0 Relief Efforts 0 Response/Experience

Days Morning Hours During Night Days Days Morning Hours During Night Days Before Before Before Event After After Before Before Before Event After After Time Period Time Period

As shown in Figure 19(a), messages containing warnings, watches, and other information about impending weather were rather common on May 18th and 19th (shown as “Days Before” in Figure 19(a)). On the morning of the 20th, alert messages dipped in frequency, but rebounded in the hours leading up to the event and while the tornado was on the ground. As the tornado lifted, alert messages became less and less common. This trend is unsurprising given the weather and forecast patterns that unfolded as the event transpired. Alert messages were common on the 18th and 19th because Twitter users were sharing warnings and watches that corresponded with the tornado outbreaks that occurred on those days and the convective outlook that was issued for the 20th. There was no new information on the morning of the 20th, so messages of this sort died off until a tornado watch and a

CLOSE UP (PART III): THE EVOLUTION OF MESSAGE CONTENT BEFORE, PAGE | 76 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

number of warnings were issued in the hours (and minutes) leading up to the event on the 20th. Alert messages peaked during the event because Twitter users were frantically posting “warning” messages about where the tornado was located and where it was heading.

General reports and information about relief efforts, by comparison, followed a different course. General reports were most common on the morning of the 20th and after the event occurred. This trend was driven by post-event news reports. On the morning of the 20th (shown as “Morning Before” in Figure 19(a)), these post-event news reports focused on the tornadoes that had occurred on the 18th and 19th. On the evening of the 20th and in the days that followed, the post-event news reports focused almost exclusively on the Newcastle-Moore-South Oklahoma City tornado. Information about how to help tornado victims, by contrast, was relatively scarce leading up to and during the tornado on the 20th and relatively common in the hours and days that followed.

When they were not providing information about the event, Twitter users who published messages containing the word tornado during this time period were, by and large, sharing personal comments, experiences, and their condolences with other Twitter users. On May 18th and 19th, general commentary about tornadoes and the risks they pose was rather common. As the Newcastle-Moore-South Oklahoma City tornado approached, occurred, and then lifted, comments of this sort were shared less frequently.

The opposite was true for the frequency of messages offering condolences to the victims and providing personal descriptions of experiences. Tweets containing messages of support/encouragement were fairly common on May 18th and 19th. As described above, these messages were prompted by the tornadoes that had occurred in the days leading up to the May 20th event. They dipped in frequency in the hours leading up to the event, but began reappearing just after the tornado CLOSE UP (PART III): THE EVOLUTION OF MESSAGE CONTENT BEFORE, PAGE | 77 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

touched down and peaked that night after the tornado ended. Tweets containing information about what users were doing/experiencing were fairly common on the 18th and 19th as well. In these kinds of tweets users were, by and large, sharing information about how they were impacted by the tornadoes on the 18th and 19th. These reflections occurred with some frequency on the morning of the 20th, but declined substantially in the hours leading up to the event before spiking while the tornado was on the ground. Again, these patterns are unsurprising, given the weather that unfolded before, during, and after the event.

9.4 Conclusions Our findings in this chapter reveal another dimension of the signal that can be observed within the social media noise. The content of messages published on Twitter before, during, and after the Newcastle-Moore-South Oklahoma City tornado shows that people were clearly responding to the weather as it unfolded during that period of time. A relatively small portion of these messages consisted of subjective or personal statements about what was happening and how people were responding, both physically and emotionally. Though useful for research and emergency management purposes, messages of this sort are not very helpful to the growing number of people and forecasters who were using social media in real time to get information about developing severe weather. Overall however, the majority of the messages collected contained potentially helpful information of some sort. These messages called out and amplified alerts, recounted the effects of the storm, passed information on recovery efforts, and provided support and encouragement for those affected by the storm. The information contained in these messages evolved in a manner that is consistent with our expectations given the weather that occurred during the time period we studied. In the next chapter, we interrogate the quality of the information provided in these messages – was it accurate? Was it timely?

CLOSE UP (PART III): THE EVOLUTION OF MESSAGE CONTENT BEFORE, PAGE | 78 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

10. Close Up (Part IV): The Evolution of Message Quality Before, During, and After the 2013 Newcastle- Moore-South Oklahoma City Tornado

10.1 Introduction In Chapter 9, as part of our in-depth case study on the use of social media during one high-profile severe weather event, we analyzed the content of messages people were posting on social media before, during, and after the Newcastle, Moore, and Oklahoma City, OK tornado that touched down on May 20, 2013. Among other things, we found that the majority of people were posting information of some sort about the event and/or tornadoes in general. In this chapter, we take this analysis one step further by looking at the quality of this information. Was it accurate and timely or – as suggested by the skeptical forecasters we interviewed in Chapter 5 – misleading and late?

10.2 Data To analyze the quality of the information contained in the social media posts, we used the coded sample of Twitter data described in Chapter 9. As a brief reminder, we collected these data by developing a program that interfaces with Twitter’s Streaming API to continuously collect and archive tweets that contain one or more key words, like tornado. Then, we characterized the content of the messages that were published before, during, and after the 2013 Newcastle-Moore-South Oklahoma City tornado by hand classifying a random sample of 5,000 tweets, containing the word tornado, that were published during this time period into one of three content categories—information dissemination, personal expression, and other. In this chapter we are interested in the quality of severe weather information on social media, so we limited our analysis to the 2,959 tweets in the information dissemination category. CLOSE UP (PART IV): THE EVOLUTION OF MESSAGE QUALITY BEFORE, PAGE | 79 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

To evaluate the accuracy of the information contained in these messages, we followed a two-step procedure. First, we identified the subset of information dissemination messages that contained “verifiable” pieces of information that could be checked against the best available information to-date. For example, messages like “Another tornado forming in Dallas County” and “Tornado Watch for portions of the area in NE until 3:00am CDT” contain verifiable information that, when combined with data on when the tweets were published, could be checked against the record of what actually happened. Messages like “Tornado warning!” and “Tornado is coming,” by comparison, could not be checked against the record because they do not include enough information (i.e., the location of the warning/tornado).

After identifying the subset of verifiable messages, we proceeded to verify or “fact check” the information contained in them. Did a tornado really form in Dallas County when the abovementioned tweet was published? Was a tornado watch that expired at 3:00 AM actually issued for the Northeast part of the area on the day that message was posted? As mentioned above, we accomplished this verification procedure by comparing the verifiable pieces of information contained in these messages to the best available information to-date, as opposed to the best available information when the tweet was published. In most instances these were the same – the official record did not change after the event. In a select few instances, however, the record changed – the best available information at the time of publication was different than the best available information to-date. For example, in the immediate aftermath of the Newcastle-Moore-South Oklahoma City tornado, Oklahoma state officials reported that the tornado had produced almost 100 fatalities. As time went on, state officials lowered the fatality count to 24. This discrepancy made it difficult to evaluate the accuracy of tweets like “At least 91 dead as massive tornado strikes US city” that were published in the immediate aftermath of the event. At the time, they may have appeared to the author of the post to be accurate. In hindsight, we CLOSE UP (PART IV): THE EVOLUTION OF MESSAGE QUALITY BEFORE, PAGE | 80 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

know that they were not accurate. Thus, when evaluating the accuracy of messages like this, we had to make a choice – do we use what we know now, or what the social media user could have known at the time? After careful consideration, we opted for the former because it was almost impossible to develop timeline of information that objectively defined what users could have known and when. Because of this choice, messages like the example listed above (that passed on seemingly valid but inaccurate information) were marked as inaccurate.

To assess the timeliness of the information contained in these messages, we followed a similar two-step procedure. This time, however, we started by identifying the subset of information dissemination messages that contained time-sensitive “actionable” pieces of information that social media users might have acted upon if they received it in time (i.e., seek shelter). We focused on actionable information because timeliness is critical. To act on a piece of information, a person needs to receive that information before action is required. For example, a message like “Another tornado forming in Dallas County” is only useful if it is published before the tornado occurs. If it is published after the tornado occurs, it will not inspire the action it was designed to inspire. Timeliness is less critical, by comparison, when a piece of information is not intended to inspire a time-sensitive action or any action at all. For example, information like “At least 91 dead as massive tornado strikes US city” is not meant to incite an action at all, and information like “Text “REDCROSS” to 90999 for $10 donation to help tornado victims” is designed to inspire an action, but that action is not time-sensitive, so it does not matter when the receiver gets it. As a result, the timeliness of either of these types of information is less important.

After identifying the subset of information dissemination messages that contained actionable pieces of information, we proceeded to evaluate the timeliness of these “actionable” messages. We accomplished this by reading each actionable message and answering a rather simple question – was it published in time for other users to act upon the information contained in the message. If the message contained CLOSE UP (PART IV): THE EVOLUTION OF MESSAGE QUALITY BEFORE, PAGE | 81 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

information about a warning, for instance, was it published before the warning expired? If it contained information like “Large Tornado moving towards Wichita, KS,” was it published before the storm made it to Wichita?

10.3 Findings As shown below in Figure 20(a), 1,412 (47.8 percent) of the information dissemination messages containing the word tornado that were published on Twitter before, during, and after the Newcastle-Moore-South Oklahoma City tornado contained verifiable information. In other words, slightly less than half of the tweets published in this category contained enough information to check them against the official record. The majority of messages that did contain enough information to verify conveyed an alert of some sort (i.e., a watch, warning, or report that a tornado was on the ground) to other social media users. On average, organizations and self- defined weather enthusiasts were a bit more likely to publish verifiable information than individuals and non-enthusiasts.

Figure 20: Verifiability and Accuracy of Information Dissemination Tweets

(a) Was the Information Verifiable? (b) Was the Verifiable Information Accurate?

1541 1048

1500 1412 1000 800 1000 600

Number of Tweets Number of Tweets Number 364 400 500 200 0 0 No Yes No Yes

Figure 20(b) shows that 1,048 of the 1,412 verifiable messages that were published before, during, and after the event were accurate. In other words, almost three quarters (74.2 percent) of the messages that contained enough information to “fact check” were, according to the best available information to-date, accurate. The overwhelming majority of messages that were incorrect referenced the inaccurate CLOSE UP (PART IV): THE EVOLUTION OF MESSAGE QUALITY BEFORE, PAGE | 82 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

fatality count described above and/or preliminary statistics about the tornado that were refined in the aftermath of the event (i.e., damage estimates, path length, and/or wind speed). Many, but not all of these messages, were probably “seemingly accurate” at the time they were published, suggesting that Twitter users were, by and large, sharing the best available information about the event as it unfolded. As with verifiable information, however, there was a slight difference across user types – organizations and weather enthusiasts were a bit more likely to publish accurate information than individuals and users with no interest in the weather.

Figure 21: Actionability and Timeliness of Information Dissemination Tweets

(a) Was the Information Actionable? (b) Was the Actionable Information Timely? 268 1036 250 1000 200 800 150 600

106 376 Number of Tweets Number of Tweets Number 100 400 50 200 0 0 No Yes No Yes

Turning now to the issue of timeliness, Figure 21(a) shows that only 376 (26.6 percent) of the information dissemination messages in our coded sample of 5,000 tweets that were published before, during, and after the Newcastle-Moore-South Oklahoma City tornado contained time-sensitive information that was meant to provoke a specific action. In other words, the vast majority of information dissemination messages published during this time period contained information that was not actionable. Most, but not all of these non-actionable messages, contained basic reports about the tornado and information about how to help the victims. The majority of actionable messages, by comparison, contained an alert of some sort. Organizations and self-defined weather enthusiasts were quite a bit more likely to publish actionable information than individuals and non-enthusiasts.

CLOSE UP (PART IV): THE EVOLUTION OF MESSAGE QUALITY BEFORE, PAGE | 83 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

As indicated in Figure 21(b), 268 of the 376 (71.3 percent) actionable messages that were published before, during, and after the event were timely – they were published in time for social media users to act upon the information contained in the message. Most of the timely messages included information about tornado watches and warnings that had been issued by the NWS. Most of the messages that were not timely, by comparison, contained information about a tornado that was on the ground in a specific area and/or heading in a certain direction. In many instances, these messages were accurate but were posted too late to be of actionable utility. A tornado was on the ground at one point in time, but had lifted by the time the message was published, rendering the information provided in the message obsolete (for purposes of protective action). Again, however, messages that were not timely were relatively uncommon. Due (in part) to this small sample size, there were no differences across user types—organizations and self-defined weather enthusiasts were just as likely to publish timely information as individuals and non- enthusiasts.

With these (baseline) figures in mind, we proceeded to explore the evolution of message quality throughout the course of the event. To accomplish this, we subset the coded sample of tweets by the time periods listed in Table 8 and then looked at the percentage of tweets published by quality classification during each time period. Figure 22 plots the results of this procedure. The solid lines (with points) show these percentages by category and time period and the dashed lines indicate mean percentages by category over the course of the event.

CLOSE UP (PART IV): THE EVOLUTION OF MESSAGE QUALITY BEFORE, PAGE | 84 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

Table 8: Time Categories, Time Periods, and Counts of Sampled Information Dissemination Tweets

Count of Sampled Time Category Time Period (CDT) Information Dissemination Tweets (%) 2013-05-18 00:00:00 - 2013-05-19 Days Before Event 254 (8.6) 23:59:59 2013-05-20 00:00:00 - 2013-05-20 Morning Before Event 23 (0.8) 12:13:59 2013-05-20 12:14:00 - 2013-05-20 Hours Before Event 24 (0.8) 14:55:59 2013-05-20 14:56:00 - 2013-05-20 During Event 66 (2.2) 15:35:59 2013-05-20 15:36:00 - 2013-05-20 Hours After Event 972 (32.8) 23:59:59 2013-05-21 00:00:00 - 2013-05-22 Days After Event 1,620 (54.7) 23:59:59

As shown in Figure 22(a), messages containing verifiable and accurate information were relatively common before the event occurred.16 Verifiability peaked while the tornado was on the ground, while accuracy peaked in the hours before the event. As the tornado lifted, message quality declined rather substantially – accuracy bottomed out the night after the tornado occurred and verifiable information hit a low in the days that followed. The lull in accuracy immediately after the event was largely driven by the incorrect fatality reports described above – as those reports were corrected, accuracy rebounded. The dip in verifiability was largely caused by the evolution of message content that occurred during that time period – Twitter users shifted from sharing information about storms that were approaching to sharing information about the tornado that had occurred and how to help the victims of those tornadoes. Many of the event reports and messages about relief efforts did not contain enough factual information to verify.

16 The rate of “verifiable” information content shown in Figure 22(a) is the percentage of all “information dissemination” messages that could be evaluated in each time period; “accurate” information is the percentage of verifiable messages that were found to be accurate in each time period. CLOSE UP (PART IV): THE EVOLUTION OF MESSAGE QUALITY BEFORE, PAGE | 85 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

Figure 22: The Evolution of Message Quality Before, During, and After the Event

(a) Verifiable and Accurate Messages (b) Actionable and Timely Messages 100 100 Accurate

80 80 Actionable 60 60

40 Verifiable 40

Percent of Percent Tweets of Percent Tweets Timely 20 20 0 0

Days Morning Hours During Night Days Days Morning Hours During Night Days Before Before Before Event After After Before Before Before Event After After Time Period Time Period

Figure 22(b) reveals a somewhat similar trend in the evolution of actionable and timely messages.17 Excluding the morning before the event, the messages published before and during the Newcastle-Moore-South Oklahoma City tornado contained (on average) more actionable and timely information than the messages published after the event. Again, this pattern is driven by the shift in content described above. As severe weather developed and occurred, social media users focused their efforts on providing alerts to other social media users. For the most part, these alerts contained time-sensitive and actionable information that was, by and large, timely. As severe weather faded, social media users began sharing other types of information. On the 18th and 19th, a number of storms prompted warnings and produced tornadoes, which led to a spike in alert messages, and, as a result, a spike in actionable and timely information. The same thing happened in the hours and minutes leading up to the tornado on the 20th. The morning of the 20th, by comparison, was relatively quiet, so less actionable and timely information was circulated on social media.

17 The rate of “actionable” information content shown in Figure 22(b) is the percentage of all “information dissemination” messages that contained actionable information in each time period; “timely” information is the percentage of actionable messages that were published in time for the implied action (e.g., take shelter) to be taken in each time period. CLOSE UP (PART IV): THE EVOLUTION OF MESSAGE QUALITY BEFORE, PAGE | 86 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

10.4 Conclusions Our findings in this chapter show that the information contained in messages that were published on Twitter before and during the Newcastle-Moore-South Oklahoma City tornado was of relatively high quality. In general, the verifiable information that Twitter users shared was accurate, and the actionable information was timely. There were, however, noticeable lulls in the quality of this information after the tornado occurred. These lulls were likely caused by two factors: 1) the confusion and dearth of information that immediately followed the event and 2) shifts in the types of information that people were sharing on social media. These findings suggest that social media users can be relatively confident in the information they get about severe weather before an event occurs. They should be less confident, by comparison, in information that is shared in the aftermath of an event. If we had to choose, this is probably the best-case scenario. Information quality is always important, but especially important when people need information the most – in the hours and minutes leading up to an event. During these moments, accurate, actionable, and timely information is critical and, in spite of the noise, it appears from this case study analysis that social media is able to provide it.

CLOSE UP (PART IV): THE EVOLUTION OF MESSAGE QUALITY BEFORE, PAGE | 87 DURING, AND AFTER THE 2013 NEWCASTLE-MOORE-SOUTH OKLAHOMA CITY TORNADO

11. Moving Forward: Where Do We Go From Here?

As mentioned in the introduction to this report, social media is often touted as a technology that may revolutionize the way in which members of the public and the weather enterprise communicate about and respond to extreme weather and water events. As yet, however, there has been little reliable information about who uses social media to send and receive information about the weather, what this information looks like, and how it evolves over the course of an extreme weather or water event. In this report, we have begun to fill this gap by addressing the following important questions: 1. Who uses social media to get information about severe weather and how has this evolved over time? 2. What do NWS forecasters and weather scientists think about social media and how has it changed the way they approach their jobs? 3. How does social media usage evolve throughout the course of a severe weather event?

While pursuing answers to these questions, a number of interesting themes emerged. Most notably, we found that social media usage is on the rise – both forecasters and members of the public are turning to social media when severe weather is on the horizon. To varying degrees, both groups are using social media to send and receive critical information about the weather before, during, and after extreme weather events such as tornadoes. Nevertheless, both forecasters and members of the public expressed a sense of skepticism about the extent to which social media should be used to send and receive information of this sort. Members of the public that live in tornado-prone regions of the country, for example, said that the information provided by traditional sources of severe weather information – like television, radio, and even word of mouth from friends and family – is more trustworthy than the information provided by social media. NWS forecasters and weather scientists expressed a similar skepticism about the information they receive via social media. In their view, the information provided by traditional

MOVING FORWARD: WHERE DO WE GO FROM HERE? PAGE | 88

sources of information, like trained storm spotters, is more reliable then the information they get “from the untrained masses” on social media. This skepticism begs a number of obvious and important follow-up questions that should be addressed in future research, including: 1. Why is social media usage on the rise if members of the public and scientists are skeptical about the quality of information it provides? Does social media provide something that traditional sources of weather information do not? 2. Is this skepticism justified? Is the quality of the information about extreme weather that is available on social media platforms significantly lower than the quality of information that is available via traditional sources?

Many of the research methods outlined in this report provide a systematic point of departure for developing empirically grounded answers to these questions. For example, the survey research methods discussed in Chapters 2 and 3 can be used to answer questions about why some people are turning to social media to get information about the weather—what exactly are they looking for? Are they looking for the same type of information that they could get on television? Or, are they looking for something different, such as a more (or less) detailed account of what might happen to them or their family if a given forecast materializes? Answers to these questions will provide critical guidance to forecasters and other providers of weather and water information about how to maximize their use of social media before, during, and after extreme weather events.

Focusing on forecasters, the qualitative interview methods described in Chapters 4 and 5 can be used in future research to provide more detail about what forecasters are looking for when they are using social media to collect information about severe weather. What type of information are they looking for? How are they identifying and verifying that information (if at all)? Are some types of information more helpful than others? What can they (as forecasters) do to encourage the social media users who reside in their county warning areas to provide more useful information? Answers to these questions will provide guidance to members of the public and other users of social media about how to increase the utility of the information they provide to forecasters about the conditions

MOVING FORWARD: WHERE DO WE GO FROM HERE? PAGE | 89

“on the ground.” Answers to these questions will also inform ongoing efforts to develop a computer program capable of automatically identifying information on social media that may be useful to forecasters in “near real-time” as an extreme weather or water event develops and unfolds. Preliminary development of the programming for a web-accessible “dashboard”, accessible to weather forecasters, that will track and display verified real- time Twitter posts on tornadic, hail, and other severe weather events has shown very promising results.

Finally, the methods developed in Chapters 6-10 can be used to systematically address the skepticism that currently pervades discussions of weather information on social media. In Chapter 10, for example, we developed an empirical approach for evaluating the accuracy and timeliness of information that was published on Twitter before, during, and after a specific event – the 2013 Newcastle-Moore-South Oklahoma City tornado. This approach provides a baseline procedure for use in the future to assess the quality of information associated with other weather and water events. Does the quality of the weather information that is available on social media decline during less notable or unexpected events? Our approach can also be used in future research to assess the quality of the information that is provided via other (more traditional) sources of severe weather information. Such an assessment would allow for a comparison of information quality across sources: to what extent do television broadcasts, for example, provide higher quality information than social media messages during weather events? A similar approach can be utilized to evaluate the quality of severe weather reports on social media. How often do members of the public and other social media users publish verifiable local weather reports on platforms like Twitter? Are the reports that they publish accurate and timely? Answers to these questions will shed light on critical debates about the extent to which members of the public and forecasters should continue to rely on social media when extreme weather approaches. In so doing, these answers will also provide important information about the extent to which the weather enterprise should continue to invest time and resources into social media and collaborative technologies.

Social media represents an important technological innovation that, in a relatively short

MOVING FORWARD: WHERE DO WE GO FROM HERE? PAGE | 90

period of time, has altered the way in which communication about extreme weather and water events occurs. The findings presented in this report suggest that this innovation has, in general, advanced the goals of the Weather-Ready Nation initiative by enhancing community resilience. It is important to note, however, that our findings are incomplete and static, whereas the world of severe weather forecasting, communication, response, and recovery is dynamic. Thus, to make significant contributions that enable a Weather- Ready Nation, we must expand upon the findings produced by this research by continuing to systematically and empirically explore the extent to which social media and other technical, social, or natural changes interact to facilitate (or undermine) community resilience in the face of increasing vulnerability to extreme weather and water events.

MOVING FORWARD: WHERE DO WE GO FROM HERE? PAGE | 91

References

APPOR. (2010). “Research Synthesis: AAPOR Report on Online Panels” Public Opinion Quarterly 74(4):711-781.

Ashley, W. (2007). “Spatial and Temporal Analysis of Tornado Fatalities in the United States: 1880–2005.” Weather and Forecasting 22(6):1214-1228.

Burton, S., Tanner, K., Giraud-Carrier, C., West, J., and Barnes, M. (2012). "Right time, right place: health communication on Twitter: value and accuracy of location information.” Journal of Medical Internet Research 14(6):e156.

Couper, M. (2000). “Web Surveys: A Review of Issues and Approaches.” Public Opinion Quarterly 64(4):464-494.

Department of Homeland Security: Virtual Social Media Working Group and DHS First Responders Group. (2013). “Lessons Learned: Social Media and Hurricane Sandy.”

Fleiss, J. (1971). “Measuring nominal scale agreement among many raters.” Psychological Bulletin 76(5): 378-382.

Hammer, B., & Schmidlin, T. (2002). “Response to warnings during the 3 May 1999 Oklahoma City tornado: reasons and relative injury rates.” Weather and Forecasting 17(3):577-581.

Pew Research Center. (2013). “Social Media Update 2013.” Report Available at: http://www.pewinternet.org/files/2013/12/PIP_Social-Networking-2013.pdf.

Ripberger, J.T., Jenkins-Smith, H.C., Silva, C.L., Carlson, D.E., and Henderson, M. (2014). “Social Media and Severe Weather: Do Tweets Provide a Valid Indicator of Public Attention to Severe Weather Risk Communication?” Weather, Climate, and Society 6(4):520-530.

REFERENCES PAGE | 92