A graph-structured dataset for Wikipedia research Nicolas Aspert, Volodymyr Miz, Benjamin Ricaud, and Pierre Vandergheynst LTS2, EPFL, Station 11, CH-1015 Lausanne, Switzerland [email protected] ABSTRACT Page 2 Page 7 Wikipedia is a rich and invaluable source of information. Its cen- Page 6 tral place on the Web makes it a particularly interesting object Page 6 of study for scientists. Researchers from different domains used Page 4 various complex datasets related to Wikipedia to study language, so- Page 3 Page 5 Page 1 cial behavior, knowledge organization, and network theory. While Page 4 being a scientific treasure, the large size of the dataset hinders pre- Page 3 processing and may be a challenging obstacle for potential new Page 7 Page 2 studies. This issue is particularly acute in scientific domains where Page 5 researchers may not be technically and data processing savvy. On Page 1 one hand, the size of Wikipedia dumps is large. It makes the parsing Day 1 Day 2 Day 3 Day 4 Day 5 Day 6 and extraction of relevant information cumbersome. On the other hand, the API is straightforward to use but restricted to a relatively Figure 1: A subset of Wikipedia web pages with viewership small number of requests. The middle ground is at the mesoscopic activity (pagecounts). Left: Wikipedia hyperlinks network, scale, when researchers need a subset of Wikipedia ranging from where nodes correspond to Wikipedia articles and edges rep- thousands to hundreds of thousands of pages but there exists no resent hyperlinks between the articles. Right: hourly page- efficient solution at this scale. view statistics of Wikipedia articles (right). In this work, we propose an efficient data structure to make requests and access subnetworks of Wikipedia pages and categories. We provide convenient tools for accessing and filtering viewership In this work, we present a convenient and intuitive graph-based statistics or "pagecounts" of Wikipedia web pages. The dataset toolset for researchers that will ease the access to this data and its organization leverages principles of graph databases that allows further analysis. rapid and intuitive access to subgraphs of Wikipedia articles and Wikimedia Foundation, the organization that hosts Wikipedia, categories. The dataset and deployment guidelines are available on makes the web activity records and the hyperlinks structure of the LTS2 website https://lts2.epfl.ch/Datasets/Wikipedia/. Wikipedia publicly available so anyone can access the records either through an API or through the database dump files. Even though KEYWORDS the data is well structured, efficient pre-processing and wrangling Dataset, Graph, Wikipedia, Temporal Network, Web Logs requires data engineering skills. First, the dumps are very large and ACM Reference Format: it takes a long time for researchers to load and filter them to get Nicolas Aspert, Volodymyr Miz, Benjamin Ricaud, and Pierre Vandergheynst. what they need to study a particular question. Second, although 2019. A graph-structured dataset for Wikipedia research. In Companion the API is well documented and easy to use, the number of queries Proceedings of the 2019 World Wide Web Conference (WWW ’19 Companion), and the response size are very limited. May 13–17, 2019, San Francisco, CA, USA. ACM, New York, NY, USA, 6 pages. Even though the API is quite convenient, it can cause repro- https://doi.org/10.1145/3308560.3316757 ducibility issues. The network of hyperlinks evolves with time so the API can only provide the latest network configuration. To 1 INTRODUCTION solve this problem, as a workaround, researchers use static pre- arXiv:1903.08597v1 [cs.IR] 20 Mar 2019 Wikipedia is one of the most visited websites in the world. Millions processed datasets. Two of the most popular datasets for Wikipedia of people use it every day searching for answers to various ques- network research are available on the SNAP archive, Wikipedia tions ranging from biographies of popular figures to definitions Network of Hyperlinks [11] and Wikipedia Network of Top Cate- of complex scientific concepts. As any other website on the Web, gories [6, 10, 16]. The initial publications referring to these datasets Wikipedia stores web logs that contain viewership statistics of every have been cited more than 1000 times, showing the high interest in page. Worldwide popularity of this free encyclopedia makes these these datasets. These archives were created from Wikipedia dumps records an invaluable resource of data for the research community. in 2011 and 2013 respectively. However, Wikipedia has evolved since then and Wikipedia research community would benefit from This paper is published under the Creative Commons Attribution 4.0 International being able to access more recent data. (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution. Multiple studies have analyzed Wikipedia from a network sci- WWW ’19 Companion, May 13–17, 2019, San Francisco, CA, USA ence perspective and have used its network structure to improve © 2019 IW3C2 (International World Wide Web Conference Committee), published Wikipedia itself or to gain insights into collective behavior of its under Creative Commons CC-BY 4.0 License. ACM ISBN 978-1-4503-6675-5/19/05. users. In [17], Zesch and Gurevych used Wikipedia category graph https://doi.org/10.1145/3308560.3316757 as a natural language processing resource. Buriol et al. [4] studied WWW ’19 Companion, May 13–17, 2019, San Francisco, CA, USA Aspert, et al. the temporal evolution of Wikipedia hyperlinks graph. Bellomi and Bonato conducted a study [3] of macro-structure of English Wikipedia network and cultural biases related to specific topics. West et al. proposed an approach enabling the identification of missing hyperlinks in Wikipedia to improve the navigation experi- ence [12]. Another direction of Wikipedia research focuses on the page- counts analysis. Moat et al. [9] used Wikipedia viewership statistics to gain insights into stock markets. Yasseri et al. [15] studied edi- torial wars in Wikipedia analyzing activity patterns in viewership dynamics of articles that describe controversial topics. Mestyán et al. [7] demonstrated that Wikipedia pagecounts can be used to predict the popularity of a movie. Collective memory phenome- non was studied in [5], where authors analyzed visitors activity to evaluate the reaction of Wikipedia users on aircraft incidents. Figure 2: Wikipedia graph structure. In blue: articles and hy- The hyperlink network structure, on one hand, and the view- perlinks referring to them. In red: category pages and hyper- ership statistics (pagecounts) of Wikipedia articles, on the other links connecting the pages or subcategories to parent cate- hand, have attracted significant attention from the research commu- gories. In green: a redirected article, i.e. Article 1 refers to nity. Recent studies open new directions where these two datasets Article 2 via the redirected page. In black: a redirection link. are combined. The emerging field of spatio-temporal data mining The blue, dashed line, is the new link created from the redi- [2] highlighted an increasing interest and a need for reproducible rection. network datasets that contain dynamically changing components. Miz et al [8] adopted an anomaly detection approach on graphs to analyze the visitors’ activity in relation to real-world events. • Get relatively large subgraphs of Wikipedia pages (1K–100K Following the recent advances of scientific research on Wikipedia, nodes) without redirects. in this work, we focus on two components: the spatial component • Use filters by the number of page views, category/sub-category, (Wikipedia hyperlinks network) and the temporal component (page- graph measures (n-hop neighborhood of a node, node degree, counts). We design a database that allows querying this hybrid data page rank, centrality measures, and others). structure conveniently (see Fig. 1). Since Wikipedia web logs are • Get viewership statistics for a subset/subgraph of Wikipedia continuously updating, we designed this database in a way that pages. will make its maintenance as easy and fast as possible. • Get a subgraph of pages with a number of visits higher than a threshold, in a predefined range of dates. The database allows its users to return subgraphs with millions 2 DATASET of links. However, requesting a large subgraph from the database may take several hours. Besides, it may require a large amount of There are multiple ways to access Wikipedia data but none of memory on the hosting server. Such queries may cause an overload them provide native support of a graph data structure. Therefore, of the database server that has to process queries from multiple if researchers want to study Wikipedia from the network science users at the same time. Therefore, instead of setting up a remote perspective, they have to create the graph themselves, which is database server, we have decided to provide the code to deploy a usually very time-consuming. To do that, they need to pre-process local or cloud-based one from Wikipedia dumps. This will allow large dumps of data or to use the limited API. researchers to explore the dataset on their own server, create new In spatio-temporal data mining [2], researchers are most inter- queries and, possibly, contribute to the project. ested in the dynamics of the networks. Hence, when it comes to Lastly, the database will be updated every month and will be con- Wikipedia analysis, one needs to merge the hyperlinks network sistent with the latest Wikipedia dumps. This gives the researchers with page-view statistics of the web pages. This is another large the ability to reproduce previous studies on Wikipedia data and to chunk of data, which requires another round of time-consuming conduct new experiments on the latest data. The dataset and the pre-processing. deployment instructions are available online [1]. After the pre-processing and the merge is completed, researchers usually realize that they do not need the full network and the entire 3 FRAMEWORK history of visitors’ activity.