© Copyright 2020 Ahmer Arif Troubling Matters: Examining the Spread of Misinformation and Disinformation on Social Media During Mass Disruption Events Ahmer Arif A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy University of Washington 2020 Reading Committee: Kate Starbird (Chair) Jennifer Turns (Chair) David M. Levy Program Authorized to Offer Degree: Human Centered Design & Engineering University of Washington Abstract Troubling Matters: Examining the Spread of Misinformation and Disinformation on Social Media During Mass Disruption Events Ahmer Arif Chairs of the Supervisory Committee: Associate Professor Kate Starbird Department of Human Centered Design & Engineering Professor Jennifer A. Turns Department of Human Centered Design & Engineering Most users want Twitter feeds, Facebook pages, and other information streams to be free of misleading content. Whether this misleading content was spread unintentionally (misinformation) or on purpose (disinformation), understanding and responding to its flows has never been more important. This is especially true during periods of collective stress and uncertainty — like large-scale emergencies, disasters, and political protests — where misleading information can have potentially broad-reaching societal consequences. This dissertation aims to clarify some of the dynamics of online mis- and disinformation in such settings. It also seeks to provide insights to help researchers and designers formulate more human-centered responses to the challenges posed by misleading information — i.e. responses that can support human skill and ingenuity rather than rendering them passive. It does this by asking the following questions: Research Question 1: How do well-intentioned members of the online crowd, like journalists and ordinary people, understand their own actions when they unwittingly circulate misleading information on social media while using it for sensemaking? Research Question 2: How do state-affiliated actors opportunistically exploit these sensemaking efforts to spread disinformation? Research Question 3: How do disinformation campaigns invite their audiences to make sense of the information landscape through a lens of suspicion? To address the first two questions, I present three studies that give account of how different groups of social media users enacted the spread of misleading information as they participated in collective sensemaking, which is the process by which people build shared awareness around events that disrupt normal routines. Across the studies, I employ a combination of methods, including computational analyses of large Twitter datasets, interviews with social media users, and a qualitative analysis of alternative media websites. I address the third research question by integrating the findings from these separate studies and drawing on the literary theory of postcritique to unpack the interpretive gestures made by the disinformation campaigns examined in this research. The results of this inquiry yield contributions along several dimensions. Some of the main results illuminate how: i) social media users engage in correcting misinformation and reason about their choices; ii) disinformation campaigns blend their activities with online activism, making their efforts participatory and difficult to isolate; and iii) these campaigns manipulate critical perspectives to foster doubt and division. While making these dynamics more legible, I also draw forward some implications and articulate a design direction to broaden our repertoire of ideas for how we might address mis- and disinformation. In the larger picture, this inquiry makes empirical and theoretical contributions that can help us think about the appropriateness of how issues pertaining to misleading information are being framed, interventions are being shaped, and social and material possibilities are being constrained down the line. TABLE OF CONTENTS List of Figures ................................................................................................................................ iv List of Tables .................................................................................................................................. v Chapter 1. Introduction ................................................................................................................... 1 1.1 The ‘Problem’ of Misleading Information ...................................................................... 1 1.2 Misleading Information and Mass Disruption Events .................................................... 2 1.3 Aims and structure of this dissertation............................................................................ 3 1.4 Dispelling the God-Trick: Introducing your Narrator .................................................... 6 1.5 Integration of Prior Work................................................................................................ 8 1.6 Motivation and background on research questions ......................................................... 9 1.7 Key disciplinary perspectives this inquiry draws on .................................................... 15 Chapter 2. Background: misleading information and collective sensemaking ............................. 21 2.1 Social Media ................................................................................................................. 21 2.2 Sensemaking ................................................................................................................. 22 2.3 Misleading Information ................................................................................................ 28 Chapter 3. Methodology ............................................................................................................... 34 3.1 Epistemological aspects of this investigation ............................................................... 34 3.2 Empirical aspects of this investigation ......................................................................... 36 3.3 Limitations of this investigation ................................................................................... 41 Chapter 4. Study 1: A closer look at the self-correcting crowd .................................................... 46 i 4.1 Abstract ......................................................................................................................... 48 4.2 Introduction ................................................................................................................... 48 4.3 Background and Related Work ..................................................................................... 50 4.4 Methods......................................................................................................................... 52 4.5 Trace Analysis Findings ............................................................................................... 57 4.6 Interview Findings ......................................................................................................... 60 4.7 Discussion ..................................................................................................................... 69 4.8 Acknowledgements ....................................................................................................... 73 4.9 Reflections: Lessons from taking a close look at the self-correcting crowd ................ 74 Chapter 5. Study 2: Acting the part .............................................................................................. 80 5.1 Abstract ......................................................................................................................... 82 5.2 Introduction ................................................................................................................... 83 5.3 Literature Review.......................................................................................................... 84 5.4 Background ................................................................................................................... 87 5.5 Methods......................................................................................................................... 89 5.6 Findings......................................................................................................................... 93 5.7 Discussion ................................................................................................................... 108 5.8 Conclusion .................................................................................................................. 111 5.9 Acknowledgements ..................................................................................................... 112 5.10 References to Problematic or Misleading Content in this Chapter ............................. 113 5.11 Summary and Takeaways ........................................................................................... 114 Chapter 6. Study 3: Ecosystem or Echo-System? ....................................................................... 116 ii 6.1 Abstract ....................................................................................................................... 117 6.2 Introduction ................................................................................................................. 117 6.3 Background ................................................................................................................