OSoMe: the IUNI observatory on social media Clayton A. Davis1,2,*, Giovanni Luca Ciampaglia1,3,*, Luca Maria Aiello4, Keychul Chung2, Michael D. Conover5, Emilio Ferrara6, Alessandro Flammini1,2,3, Geoffrey C. Fox2, Xiaoming Gao7, Bruno Goncalves¸ 8, Przemyslaw A. Grabowicz9, Kibeom Hong2, Pik-Mai Hui1,2, Scott McCaulay3, Karissa McKelvey10, Mark R. Meiss11, Snehal Patil12, Chathuri Peli Kankanamalage3, Valentin Pentchev3, Judy Qiu2, Jacob Ratkiewicz11, Alex Rudnick11, Benjamin Serrette3, Prashant Shiralkar1,2, Onur Varol1,2, Lilian Weng13, Tak-Lon Wu14, Andrew J. Younge2 and Filippo Menczer1,2,3 1 Center for Complex Networks and Systems Research, Indiana University, Bloomington, United States 2 School of Informatics and Computing, Indiana University, Bloomington, United States 3 Network Science Institute, Indiana University, Bloomington, United States 4 Bell Labs, London, United Kingdom 5 LinkedIn Inc., Mountain View, CA, United States 6 Information Sciences Institute, University of Southern California, Marina Del Rey, CA, United States 7 Facebook Inc., Boston, MA, United States 8 Center for Data Science, New York University, New York, NY, United States 9 Max Planck Institute for Software Systems Saarbrücken, Germany, 10 US Open Data, Oakland, CA, United States 11 Google Inc., Mountain View, CA, United States 12 Yahoo Inc., Sunnyvale, CA, United States 13 Affirm Inc., San Francisco, CA, United States 14 Amazon Inc., Seattle, WA, United States * These authors contributed equally to this work. ABSTRACT The study of social phenomena is becoming increasingly reliant on big data from online social networks. Broad access to social media data, however, requires software Submitted 2 May 2016 development skills that not all researchers possess. Here we present the IUNI Observa- Accepted 29 August 2016 tory on Social Media, an open analytics platform designed to facilitate computational Published 3 October 2016 social science. The system leverages a historical, ongoing collection of over 70 billion Corresponding author public messages from Twitter. We illustrate a number of interactive open-source tools Giovanni Luca Ciampaglia, [email protected] to retrieve, visualize, and analyze derived data from this collection. The Observatory, now available at osome.iuni.iu.edu, is the result of a large, six-year collaborative effort Academic editor Silvio Peroni coordinated by the Indiana University Network Science Institute. Additional Information and Declarations can be found on Subjects Data Science, Network Science and Online Social Networks, Social Computing, World page 14 Wide Web and Web Science DOI 10.7717/peerj-cs.87 Keywords Social media, Observatory, Twitter, Web science, Network science, Meme diffusion, Computational social science, Big data, API, OSoMe Copyright 2016 Davis et al. INTRODUCTION Distributed under Creative Commons CC-BY 4.0 The collective processes of production, consumption, and diffusion of information on OPEN ACCESS social media are starting to reveal a significant portion of human social life, yet scientists How to cite this article Davis et al. (2016), OSoMe: the IUNI observatory on social media. PeerJ Comput. Sci. 2:e87; DOI 10.7717/peerj-cs.87 struggle to get access to data about it. Recent research has shown that social media can perform as `sensors' for collective activity at multiple scales (Lazer et al., 2009). As a consequence, data extracted from social media platforms are increasingly used side-by-side with—and sometimes even replacing—traditional methods to investigate hard-pressing questions in the social, behavioral, and economic (SBE) sciences (King, 2011; Moran et al., 2014; Einav & Levin, 2014). For example, interpersonal connections from Facebook have been used to replicate the famous experiment by Travers & Milgram (1969) on a global scale (Backstrom et al., 2012); the emotional content of social media streams has been used to estimate macroeconomic quantities in country-wide economies (Bollen, Mao & Zeng, 2011; Choi & Varian, 2012; Antenucci et al., 2014); and imagery from Instagram has been mined (De Choudhury et al., 2013; Andalibi, Ozturk & Forte, 2015) to understand the spread of depression among teenagers (Link et al., 1999). A significant amount of work about information production, consumption, and diffusion has been thus aimed at modeling these processes and empirically discriminating among models of mechanisms driving the spread of memes on social media networks such as Twitter (Guille et al., 2013). A set of research questions relate to how social network structure, user interests, competition for finite attention, and other factors affect the manner in which information is disseminated and why some ideas cause viral explosions while others are quickly forgotten. Such questions have been addressed both in an empirical and in more theoretical terms. Examples of empirical works concerned with these questions include geographic and temporal patterns in social movements (Conover et al., 2013b; Conover et al., 2013a; Varol et al., 2014), the polarization of online political discourse (Conover et al., 2011b; Conover et al., 2011a; Conover et al., 2012), the use of social media data to predict election outcomes (DiGrazia et al., 2013) and stock market movements (Bollen, Mao & Zeng, 2011), the geographic diffusion of trending topics (Ferrara et al., 2013), and the lifecycle of information in the attention economy (Ciampaglia, Flammini & Menczer, 2015). On the more theoretical side, agent-based models have been proposed to explain how limited individual attention affects what information we propagate (Weng et al., 2012), what social connections we make (Weng et al., 2013), and how the structure of social and topical networks can help predict which memes are likely to become viral (Weng, Menczer & Ahn, 2013; Weng, Menczer & Ahn, 2014; Nematzadeh et al., 2014; Weng & Menczer, 2015). Broad access by the research community to social media platforms is, however, limited by a host of factors. One obvious limitation is due to the commercial nature of these services. On these platforms, data are collected as part of normal operations, but this is seldom done keeping in mind the needs of researchers. In some cases researchers have been allowed to harvest data through programmatic interfaces, or APIs. However, the information that a single researcher can gather through an API typically offers only a limited view of the phenomena under study; access to historical data is often restricted or unavailable (Zimmer, 2015). Moreover, these samples are often collected using ad-hoc procedures, and the statistical biases introduced by these practices are only starting to be understood (Morstatter et al., 2013; Ruths & Pfeffer, 2014; Hargittai, 2015). Davis et al. (2016), PeerJ Comput. Sci., DOI 10.7717/peerj-cs.87 2/21 A second limitation is related to the ease of use of APIs, which are usually meant for software developers, not researchers. While researchers in the SBE sciences are increasingly acquiring software development skills (Terna, 1998; Raento, Oulasvirta & Eagle, 2009; Healy & Moody, 2014), and intuitive user interfaces are becoming more ubiquitous, many tasks remain challenging enough to hinder research advances. This is especially true for those tasks related to the application of fast visualization techniques. A third, important limitation is related to user privacy. Unfettered access to sensitive, private data about the choices, behaviors, and preferences of individuals is happening at an increasing rate (Tene & Polonetsky, 2012). Coupled with the possibility to manipulate the environment presented to users (Kramer, Guillory & Hancock, 2014), this has raised in more than one occasion deep ethical concerns in both the public and the scientific community (Kahn, Vayena & Mastroianni, 2014; Fiske & Hauser, 2014; Harriman & Patel, 2014; Vayena et al., 2015). These limitations point to a critical need for opening social media platforms to researchers in ways that are both respectful of user privacy requirements and aware of the needs of SBE researchers. In the absence of such systems, SBE researchers will have to increasingly rely on closed or opaque data sources, making it more difficult to reproduce and replicate findings—a practice of increasing concern given recent findings about replicability in the SBE sciences (Open Science Collaboration, 2015). Our long-term goal is to enable SBE researchers and the general public to openly access relevant social media data. As a concrete milestone of our project, here we present an Observatory on Social Media—an open infrastructure for sharing public data about information that is spread and collected through online social networks. Our initial focus has been on Twitter as a source of public microblogging posts. The infrastructure takes care of storing, indexing, and analyzing public collections and historical archives of big social data; it does so in an easy-to-use way, enabling broad access from scientists and other stakeholders, like journalists and the general public. We envision that data and analytics from social media will be integrated within a nation-wide network of social observatories. These data centers would allow access to a broad range of data about social, behavioral, and 1The OSoMe website is also available economic phenomena nationwide (King, 2011; Moran et al., 2014; Difranzo et al., 2014). at truthy.indiana.edu. The original Our team has been working toward this vision since 2010, when we started collecting website was created to host