Identifying, Understanding and Detecting Recurring, Harmful Behavior Patterns in Collaborative Editing – Doctoral Proposal –

Fabian Flöck Institute AIFB Karlsruhe Institute of Technology Germany fabian.fl[email protected] Supervised by Dr. Elena Simperl, University of Southampton, UK and Dr. Achim Rettinger, Karlsruhe Institute of Technology, Germany

ABSTRACT together orders of magnitude more users than their offline counter- In this doctoral proposal, we describe an approach to identify recur- parts. These environments thrive and prosper from the contribu- ring, collective behavioral mechanisms in the collaborative interac- tions of their user base, and their success hinges greatly on the ex- tions of Wikipedia editors that have the potential to undermine the tent to which interaction and collaboration among their users leads ideals of quality, neutrality and completeness of article content. We to a critical mass of useful, high-quality content. outline how we plan to parametrize these patterns in order to un- One of the most prominent examples of online collaborative plat- derstand their emergence and evolution and measure their effective forms in the past decade is undoubtedly Wikipedia. It is the di- impact on content production in Wikipedia. On top of these results rect result of the actions of a community of volunteers who create we intend to build end-user tools to increase the transparency of the and edit its articles; members of this community and the system- evolution of articles and equip editors with more elaborated quality atic behavioral patterns emerging from their interactions form, di- monitors. We also sketch out our evaluation plans and report on rectly or indirectly, intentionally or unintentionally, the decision already accomplished tasks. about which topics are included in the encyclopedia and how they are represented. In order for such “wisdom-of-the-crowds”-driven judgements to have the desired effects, and to propel the creation Categories and Subject Descriptors of high-quality articles - complete, factually correct and unbiased - H.5 [Information Interfaces and Presentation]: Group and Orga- it is essential to study and analyze the underlying community and nization Interfaces—Collaborative computing, Computer-supported its actions, and to devise methods to identify frequently occurring cooperative work, Web-based interaction; H.1 [Models and Prin- social phenomena that are likely to indicate a dysfunctional style ciples]: User/Machine Systems—Human factors of collaboration. Ownership behavior, for instance, wherein an in- dividual editor or a group of editors defend an article from being changed by third parties has been identified in the literature, as we Keywords will see in Section 1.1. Arguably, this behavior has ambivalent Wikipedia, editing behavior, user modeling, collaboration systems, effects: On the one hand, it protects articles from vandalism, on collective intelligence, web science, social dynamics the other hand, it may discourage the introduction of novel infor- mation and different points of view by outsiders. There are many 1. PROBLEM STATEMENT other behavioral patterns studied in classical social sciences which can easily emerge in online settings, causing (or facilitating) such In sociology, economy and psychology, there have been decades harmful impairments. Note that by these behavioral patterns we do of research on the emergence of harmful social interaction patterns not refer to purposeful malicious behavior like vandalism or other in certain populations, leading to a vast body of scientific work actions by individual users, but collaborative interaction patterns of aiming to understand, predict, and prevent the occurrence of such conduct in the (well-intended) editing process. phenomena. These include mass hysteria and panics, stock mar- 1 Understanding and systematically detecting social mechanisms ket bubbles and disease spread, to name only a very few. By within Wikipedia is a challenging topic; while empirical studies at- contrast, little is known about how such damaging social dynam- test the existence of a variety of recurring interaction patterns, the ics form in collaborative online environments that potentially bring effects of such interaction often remain unclear, and imprecision and hidden assumptions may lead to unwanted conclusions. This 1A plethora of research work exists on these and many related top- ics. In the scope of this PhD proposal we cannot introduce them in is partially due to the fact that assessing the quality of Wikipedia their entirety. See [19] (panics), [21] (stock market) and [6] (dis- content is highly contextual; there is not, and will never be, a con- ease spread) as examples. sistent alternative source of information, accepted by all relevant stakeholders, and containing all facts, opinions and viewpoints that Copyright is held by the International World Wide Web Conference should be objectively included in a Wikipedia article. If auto- Committee (IW3C2). IW3C2 reserves the right to provide a hyperlink matic techniques are available, they are naturally restricted to spe- to the author’s site if the Material is used in electronic media. cific sources, which the content of articles is compared against, WWW 2013 Companion, May 13–17, 2013, Rio de Janeiro, Brazil. ACM 978-1-4503-2038-2/13/05.

361 thus identifying only a share of potential inconsistencies or miss- that it is hard for new and occasional editors to change content. ing information. Such methods typically identify major quality As [22] point out, the ratio of reverted edits to the total number flaws in articles, but are less reliable for cases in which such is- of edits has increased with occasional editors experiencing greater sues are slightly less obvious, such as a biased point of view. Aside resistance compared to high-frequency ones.7 Relevant to this sce- from purely content-driven methods, means to detect the emer- nario [13] show that if an editor tries to remove words from an gence of harmful editor behavior patterns promise a complemen- article, the probability of her edit to be reverted increases signifi- tary approach; once such behavior occurs, the system generates cantly with the number of article revisions the removed words had warnings, and the community at large can react and assess the cur- ‘survived’ before (normalized for the number of removed words). rent status of an article apart from simple fact checking, sentiment In short, words which have been in the article for a longer period of analysis and biased wording. time are harder to remove, as more trust is placed in older content. Supplying readers and editors with instruments to make the “so- Likewise, viewpoints added in the early days of Wikipedia could cial evolution” of an article transparent and explicit can be an effec- have a much higher probability of being eventually represented in tive tool to induce awareness of possible incorrectness and bias and the article, supported by the observations of Kittur et al. [14, 15]. counter uncritical perception. As Wikipedia matures, so need the The cause of these phenomena seems to lie in the perceived con- methods to secure its quality. Giving Wikipedians (and any sensus between the editors, which serves as “social proof” [5] of users) more elaborated tools to see complex problematic patterns the correctness of the information. If a majority has (implicitly) in editor behavior is one step in this direction. Our work aims to assessed a content to be right, it is difficult to change it. An “in- provide such methods, which are accurate regarding the way infor- formation cascade” [3] can occur where all participants place their mation is processed and interpreted, and representative with respect decisions on the preceding editors’ choices instead of their own to the communities of readers and editors they claim to apply to. knowledge. All in all, this valid social heuristic may also hamper The has reached similar conclusions and the replacement of outdated content or the revision of biased con- incorporated them into their strategy to advance the Wikpedia plat- tent towards more balanced opinion expressions, at least for new- form. The research planned in this thesis is carried out in the con- comers or anonymous users. text of the RENDER project2, in collaboration with the German Another observation is that certain editors tend to become over- chapter of the Wikimedia foundation. Results and tools will be protective about article content against changes from other edi- deployed on Wikipedia’s sites and promoted by Wikimedia to the tors. Although explicitly discouraged by Wikipedia,8 strong feel- worldwide community.3 ings of ownership towards an article and protective behavior are not uncommon and “[...] run the risk of deterring new community 1.1 Related work member participation.” [23]. Halfaker et al. [13] infer that “[...] The work presented in the following exemplifies that Wikipedia Wikipedia’s review system suffers from a crucial bias: editors ap- articles are increasingly showing traits that hint at the emergence pear to inappropriately defend their own contributions.” Articles of exactly the harmful behavior patterns we aim to detect. A great guarded in such a way naturally run the risk of being biased as majority of the work mentioned here offers descriptive views on new contributions tend to get accepted only if conforming to the single phenomena the authors observed; as a whole, these studies owner’s taste. Members or subgroups which do not agree with the draw a consistent picture of the effects recurring behavior patterns majority decision might at first fight for their viewpoint(s) but then can have on articles, and on the potential of these effects to raise eventually just stop editing altogether. Halfaker et al.[13] find in- quality and neutrality issues in Wikipedia content.4 dicators for the drop-out of highly reverted editors in the first 16 Some findings [12] suggest that a certain share of legitimate weeks of membership in Wikipedia. Another effect is the emer- viewpoints5 might not be represented in Wikipedia because active gence of user ‘camps’ with divergent positions engaging in revert editors cover a narrow section of the offline-population’s socio- wars on highly controversial articles, which has been investigated 9 demographic scope, a problem the community has long identified in [17]. The case of Conservapedia reveals that an opposing camp and labeled the “systemic bias” of Wikipedia.6 might leave the discussion arena altogether, perhaps resulting in a For those active in the system, the widely observed phenomenon highly biased article. of a small minority of users providing most of the edits and content The phenomena indicated by the research work presented above [25, 14] to an online collaboration system holds true for Wikipedia make it seem likely that social barriers exist for new viewpoints to as well. Priedhorsky et al. [20] show that a vast majority of the be accepted in Wikipedia, even when these may objectively contain content that is actually provided by a minuscule fraction of all edi- useful information. As content matures, and the share of “hard” tors. This effect is reinforced by the fact that high-frequency editors factual information to be added to articles is diminishing, the value continue to increase their proportional number of edits [22]. and accuracy of new revisions (for example, the rewrite of an article Suh et al. [22] suggest that the declining growth of the English to include a different point of view on the topic) are far more chal- Wikipedia indicates a state of matureness where many articles are lenging to evaluate and judge. This means that more complex social close to complete on a factual level. Correlating with this devel- control mechanisms and procedures come into play, for instance, in opment, for many articles a consolidated article text has emerged form of a stronger emphasis on reputation and social proof, which since Wikipedia’s inception, which now is relatively fixed insofar may deter some editors from contributing effectively. At the same time, tools and methods to detect the emergence of those possibly 2http://www.render-project.eu negative habits in an article become essential given the scale and 3See, e.g., the WIKIGINI tool presented in section 2.5, which is complexity of the Wikipedia environment. already included in the diversity toolkit by Wikimedia. 4Note that all of the research presented here was done using data 7Indicators such as page protection, deletion, block, and other re- and observations based on the English version of Wikipedia, a prac- strictive policies exhibit a similar trend. tice that we will adopt for the work outlined in the following. 8http://en.wikipedia.org/wiki/Wikipedia: 5“Legitimate” meaning all content apart from vandalism. Ownership_of_articles 6http://en.wikipedia.org/w/index.php?title= 9The ultra-conservative “sister” of Wikipedia, founded by ex- Wikipedia:Systemic_bias&oldid=516438857 Wikipedians (http://www.conservapedia.com).

362 2. APPROACH, STATE-OF-THE-ART AND editing logs provided by Wikipedia. This data is needed for the CONTRIBUTIONS detection of nearly all relevant dynamics identified in Task 1. The first step was completed in [9], where we devised a sufficiently ac- The planned doctoral thesis aims to (i) give a comprehensive ac- curate method to detect reverts between editors. It uses a definition count of the core behavioral mechanisms that fundamentally drive of reverts rooted in the official Wikipedia definition and daily inter- social interaction phenomena leading to biased, incorrect or in- actions of editors we observed. The evaluation showed a significant complete content; (ii) systematically identify and categorize these increase in accuracy predicting antagonistic relationships between mechanisms; (iii) evaluate what their actual (negative) effects are edits compared to previous methods. likely to be on the resulting content under given circumstances; and Furthermore, we improved the accuracy of the detection of au- finally (iv) generate user-friendly support tools and interfaces that thorship of words in Wikipedia [8] compared to existing methods. inform editors about these phenomena, generating transparency re- This is a task still in progress until completely satisfactory accuracy garding the article evolution and giving an opportunity to take coun- performance is reached. teractive measures. The research approach can be divided into five tasks as follows: 2.2.1 State-of-the-art Task 1 Systematic collection of patterns For revert detection, the available research uniformly considers reverts simply as “going back to a previous revision”, leading to Task 2 Data mining and preparation straightforward and fast methods, but neglecting the full Wikipedia definition of reverts and the correct interpretation of antagonistic Task 3 Collecting a ground truth edit behavior [13, 14, 17, 20, 22, 18]. This caveat was removed by Task 4 Pattern and effect analysis our work on reverts. Few approaches exist for identifying authorship of words. One Task 5 Building end-user tools is HistoryFlow by IBM, which operates on a sentence level [24], as well as some minor tools by the Wikipedia community, which 2.1 Task 1: Systematic collection of patterns show no competitive performance in terms of speed or accuracy. A As a first step, it is important to identify those fundamental be- more elaborated approach is used by Wikitrust [1, 2]. It operates on havioral mechanisms that have the potential to do notable harm in a word level and can as well track reintroduced words and assign the collaboration process and to the resulting content, in order to the original author. An evaluation showed that a first version of make clear what patterns to look for in the existing Wikipedia edit our algorithm outperformed Wikitrust by 10% in terms of accuracy log data. This was already done to an extend by the author on the [8]. As of writing this paper, the accuracy of our method lies at one hand through related work showing the damaging effects of circa 90%, thereby surpassing the state of the art by over 50%. imitation behavior in collaborative online systems [7], and on the other hand through a first literature review [10] and initial system- 2.3 Task 3: Collecting a ground truth atization of recurring suspected harmful behaviors, similar to the In order to decide if certain behavior patterns lead to bias or oth- overview given in Section 1.1. This review will be expanded to erwise low quality, it is necessary to determine if an article exhibits give an account of all relevant empirical research work. these traits in the first place. Using only the mark-up (in the form of tags and ratings) provided by the Wikipedia community will fall 2.1.1 State-of-the-art short of taking into account the assessment of the shown content There is plenty of research work that identifies individual phe- by the majority of read-only users that don’t cross the threshold to nomena in Wikipedia, as introduced in Section 1.1. Few of these editorship or users in spe who don’t yet have access to Wikipedia. publications, however, go beyond a descriptive account of the hap- I.e., relying only on the ratings of the content by the individuals penings; explanations of the underlying behavior causing these de- that actually created the content will not be sufficient for an ob- velopments or means to measure them are largely missing. jective analysis. More importantly, most articles in Wikipedia lack As an exception, e.g., Suh et al. [22] explain the slowing growth any metadata indicating their quality, especially crucial in cases of and increasing conflict by comparing Wikipedia to a kind of infor- missing tags for low quality which can lead to misplaced trust in the mation ecosystem which is overcrowded and cannot “support” its content. Therefore, we will use crowdsourcing techniques to per- inhabitants anymore (in line with the observations of [4, 11, 17]). form content analyses through a diverse set of non-wikipedia-users. Based on their empirical findings, some works claim that tasks re- This will include (i) testing the ability of workers on crowdsourcing quiring low coordination between participants actually profit from platforms to spot different types of impairments of article content many contributors [26], whereas more complex tasks with high co- and finding the right setting parameters to optimize the reliability ordination overhead, such as building the structure of an article, are of the results and (ii) extracting the actual ratings from the crowd optimally accomplished within a comparatively smaller group of to be used as a gold standard in the remaining tasks of this doctoral users and would be probably unfeasible otherwise [15, 16, 26]. thesis. Apart from a few attempts, though, the research on Wikipedia editing dynamics so far fails to provide a comprehensive catalogue 2.3.1 State-of-the-art of the underlying, recurring mechanisms leading to the phenomena To evaluate if an article is biased or what quality level it has, described in Section 1.1, and does not evaluate them in terms of available research work almost uniformly relies on the ratings given their actual effects on the content quality of Wikipedia. to articles by the Wikipedia community itself, either in the form of specific tags assigned to them by individual editors and wiki 2.2 Task 2: Data mining and preparation projects or, much less frequently, some (semi-)external ratings, e.g. As a basis to systematically (and even automatically) spot these by the Wikipedia 1.0 project or external experts.10 We are not behavioral patterns in the Wikipedia edit data, meaningful relations between the editors as well as reliable information regarding the au- 10In the latter case, this usually includes not more than about 20 thorship of words in the articles have to be extracted from the raw manually evaluated articles.

363 aware of any work that tries to collect ratings for Wikipedia arti- 2.5.1 State-of-the-art cles from external, independent sources and avoiding self-selection A variety of tools has been developed for Wikipedia to provide problems, especially not from crowdsourcing platforms, making an overview on the editing dynamics for different purposes. One use of the immense diversity of workers they offer. of the earliest visualization tools was HistoryFlow, mentioned in Section 2.1. By tracking how long certain words remain unrevised 2.4 Task 4: Pattern and effect analysis in an article, the Wikitrust Project by Adler et al. [1] uses visual Once collected and identified in Task 1, recurring editing mecha- markups for “trusted" and “untrusted" passages in any Wikipedia nisms suspicious of harming article content will be made detectable article text in different color shades. Community extensions, like by learning which typical structures they exhibit in regard to the WikiPraise12 expand its original functionality. It does however not social network between users and the network relating users and give insights as to the diversity of the article content. Further, the words in an article, as well as distributions of edits and authorship definition of “trust" implies quality, when, as a matter of fact, con- concentration, revert rates of newcomers and many other variables, solidated article content can as well hint at outdated or biased con- all of which will be derived from the data mined in Task 2. From the tent, as we discussed in section 1.1. Wikidashboard13 provides an analysis of such statistical and social network data, measures will aggregated view of edits in an article performed over time for the be obtained that can indicate the presence and intensity of a given most active users. There is, furthermore, a wide variety of tools mechanism, e.g., ownership behavior. The methodology used will created by the Wikipedia community.14 These solutions, however, incorporate traditional statistical methods like regression analysis, have neither the purpose nor the ability to uncover neutrality issues clustering and factor analysis as well as social network methods, or to detect harmful behavioral patterns and generate warnings or including exponential random graph models and block modelling. at least an easly understandable score for a certain quality or neu- The next step will be to assess the effect of one or a combina- trality thread. tion of these patterns in an article on the resulting quality and bias, using the ground truth generated in Task 3. We will look at the 3. EVALUATION correlation between the two and conduct a careful analysis on the Every step in the outlined research methodology will be evalu- possible causalities. ated to ensure the accuracy and usefulness of the results of each task. This is even more important as the Tasks build up on each 2.4.1 State-of-the-Art other. The statistical methods described above have been traditionally Task 1 can not be evaluated in a strict sense, but an extended applied in social sciences, and social network analysis methods are literature review will be compiled and submitted as a journal arti- gaining more and more momentum in the realm of online collabo- cle, covering as much empirical evidence as possible for potential ration, proving their usefulness for gaining deeper insights into this damaging behavior patterns in Wikipedia. kind of data. We will adopt these to our specific use case. Task 2 has been evaluated for the revert detection through a user study with 35 Wikipedia editors which yielded significant statical 2.5 Task 5: Building end-user tools improvements in accuracy [9]. For authorship detection we used The goal of our work is ultimately to provide users with intuitive 250 words from 45 randomly selected articles and could report a tools and visualizations that make possibly negative developments statistically significant 10% gain in accuracy [8]. Further improve- in an article’s quasi-hidden social context easy to spot and under- ments for authorship detection are planned to be tested with even stand. This holds true for the most recent, ongoing process of con- larger samples. tent generation, but also for the historic view on the evolution of an Task 3: The user studies using crowdsourcing platforms will article. The latter can yield insights into the quality and neutrality include settings for cross-evaluating the results between different status of the article that are hard or impossible to assess only by sample groups, controlling for group effects and self-selection, as reading the article content and comparing it to external sources. well as survey variations ruling out suggestive questions. Also, a To do so, we will provide tools that are centered around the de- pre-study will determine if the workers on crowdsourcing platforms velopment over time of those critical indicators (identified in Task are able to solve these kind of rating tasks and how parameters 4) that have been singled out as influential for the unfolding of an have to be set in terms of payment, length and types of articles and article, and highlight important or unexpected events in the evo- qualification of workers. lution of these measures, such as spikes or sudden drops (in the Task 4 will be evaluated by testing the prediction capabilities simplest case). Furthermore, this information has to be put in re- of the learned patterns for biased and other problematic content lation to other article-related features such as edits and views re- through probabilistic modeling. ceived, tags and templates added, or activities on the associated Task 5 is planned to be tested by doing user studies with Wikipedia talk page. Such tools could be used to augment the outcomes of editors, readers and non-readers to gauge the usability and useful- content-analysis methods, for example algorithms examining fact ness of the tools to heighten transparency, spotting critical devel- coverage in an article against external sources such as news or so- opments and help finding the cause for content insufficiencies. Part cial media. of the tools will be included and evaluated in the frame of the offi- cial diversity toolkit by Wikimedia Deutschland, developed in the We have already developed the WIKIGINI tool, which, based 15 on the results of Task 2, determines the concentration of author- research project RENDER. 11 ship in an article, and its oscillation over time. First results of the 12http://de.wikipedia.org/wiki/Benutzer: ongoing analysis we are running with the help of this rather sim- NetAction/WikiTrust/WikiPraise ple indicator are promising in respect to its ability to facilitate the 13http://wikidashboard.appspot.com/ understanding of the evolution of articles, and the identification of 14Some examples: Wikirage (http://www.wikirage.com) critical developments that deserve special attention. shows trending topics and most editing users recently. WikiXRay (http://meta.wikimedia.org/wiki/WikiXRay) gen- 11http://toolserver.org/~RENDER/toolkit/ erates a range of quantitative descriptive statistics. WIKIGINI/ 15http://toolserver.org/~RENDER/toolkit/

364 4. CONCLUSIONS [11] A. Forte and A. Bruckman. Scaling consensus: Increasing In this doctoral proposal, we made apparent that the maturing of decentralization in wikipedia governance. In HICSS, page Wikipedia in terms of the content and the collaboration habits of its 157. IEEE Computer Society, 2008. editor base make more sophisticated methods and tools necessary [12] R. Glott, P. Schmidt, and R. Ghosh. Wikipedia survey — to keep track of possible harmful social interaction patterns. It was overview of results. Technical report, UNU- MERIT, United outlined how we plan to systematize these patterns from empirical Nations University, Maastricht, Netherlands, March 2010. findings, how we are going to mine and enrich editing log data [13] A. Halfaker, A. Kittur, R. Kraut, and J. Riedl. A jury of your and how we are going to use this data to parametrize the found peers: quality, experience and ownership in wikipedia. In Int. mechanisms to eventually build tools that can visualize them and Sym. . ACM, 2009. generate warnings for editors and readers. In this way we hope to [14] A. Kittur, E. H. Chi, B. A. Pendleton, B. Suh, and at least create transparency and awareness for the reader regarding T. Mytkowicz. Power of the few vs. wisdom of the crowd: these underlying obstacles for quality and at best provide a lever Wikipedia and the rise of the bourgeoisie. 25th Annual ACM for editors to remove them. For future work, we will focus mainly Conference on Human Factors in Computing Systems on the Tasks 3, 4 and 5, as they are still largely unsolved. (CHI’07), 2007. [15] A. Kittur and R. E. Kraut. Harnessing the wisdom of crowds in wikipedia: quality through coordination. In CSCW ’08: Acknowledgements Proceedings of the ACM 2008 conference on Computer The research for this thesis has received funding from the European supported cooperative work, pages 37–46, New York, NY, Union’s Seventh Framework Programme (FP7/2007-2013) under USA, 2008. ACM. grant agreement no. 257790, project RENDER.2 [16] A. Kittur, B. A. Pendleton, and R. E. Kraut. Herding the cats: the influence of groups in coordinating peer production. In Int. Sym. Wikis. ACM, 2009. 5. REFERENCES [17] A. Kittur, B. Suh, B. A Pendleton, and E. H Chi. He says, she says: conflict and coordination in wikipedia. In Proceedings [1] T. Adler and L. Alfaro. A content-driven reputation system of the SIGCHI conference on Human factors in computing for the Wikipedia. In WWW ’07: Proceedings of the 16th systems, CHI ’07, pages 453–462, New York, NY, USA, international conference on World Wide Web, pages 2007. ACM. 261–270, 2007. [18] S. K. Lam, S. Uduwage, Z. Dong, S. Sen, D. R. Musicant, [2] T. Adler, K. Chatterjee, L. Alfaro, M. Faella, I. Pye, and Loren Terveen, and John Riedl. Wp:clubhouse? an V. Raman. Assigning trust to Wikipedia content. In exploration of wikipedia’s gender imbalance. In WikiSym International Symposium on Wikis, 2008. 2011, Mountain View, CA, 10/2011 2011. ACM, ACM. [3] S. Bikhchandani, D. Hirshleifer, and I. Welch. A theory of [19] A.R. Mawson. Understanding mass panic and other fads, fashion, custom, and cultural change as informational collective responses to threat and disaster. Psychiatry: cascades. Journal of political Economy, pages 992–1026, Interpersonal and biological processes, 68(2):95–113, 2005. 1992. [20] R. Priedhorsky, J. Chen, S. K. Lam, K. Panciera, L. Terveen, [4] B. Butler, E. Joyce, and J. Pike. Don’t look now, but we’ve and J. Riedl. Creating, destroying, and restoring value in created a bureaucracy: the nature and roles of policies and wikipedia. In GROUP ’07: Proceedings of the 2007 rules in wikipedia. In Proceedings of the CHI ’08, pages international ACM conference on Supporting group work, 1101–1110, New York, NY, USA, 2008. ACM. pages 259–268, New York, NY, USA, 2007. ACM. [5] R. Cialdini. Influence science and practice. Gardners Books [21] R.J. Shiller, S. Fischer, and B.M. Friedman. Stock prices and Ltd, [S.l.], 2007. social dynamics. Brookings Papers on Economic Activity, [6] S. Eubank, H. Guclu, V.S.A. Kumar, M.V. Marathe, 1984(2):457–510, 1984. A. Srinivasan, Z. Toroczkai, and N. Wang. Modelling disease [22] B. Suh, G. Convertino, E. H. Chi, and P. Pirolli. The outbreaks in realistic urban social networks. Nature, singularity is not near: slowing growth of wikipedia. In 429(6988):180–184, 2004. Proceedings of the 5th International Symposium on Wikis [7] F. Flöck, J. Putzke, S. Steinfels, K. Fischbach, and and Open Collaboration, WikiSym ’09, pages 8:1–8:10, D. Schoder. Imitation and quality of tags in social New York, NY, USA, 2009. ACM. bookmarking systems – collective intelligence leading to [23] J. Thom-Santelli, D. Cosley, and G. Gay. What’s mine is folksonomies. In T. Bastiaens et al., editor, On Collective mine: territoriality in collaborative authoring. In CHI’09, Intelligence, volume 76 of Advances in Intelligent and Soft pages 1481–1484. ACM, 2009. Computing, pages 75–91. Springer Berlin / Heidelberg, 2011. [24] F. B. Viégas, M. Wattenberg, and K. Dave. Studying [8] F. Flöck and A. Rodchenko. Whose article is it anyway? – cooperation and conflict between authors with history flow detecting authorship distribution in wikipedia articles over visualizations. In Proceedings of the SIGCHI conference on time with wikigini. In Online proceedings of the Wikipedia Human factors in computing systems, CHI ’04, pages Academy 2012. Wikimedia, 2012. 575–582, New York, NY, USA, 2004. ACM. [9] F. Flöck, D. Vrandeciˇ c,´ and E. Simperl. Revisiting reverts: [25] S. Whittaker, L. Terveen, H. Hill, and L. Cherny. The accurate revert detection in wikipedia. In Proceedings of the dynamics of mass interaction. In Proceedings of the 1998 23rd ACM conference on Hypertext and social media, pages ACM conference on Computer supported cooperative work 3–12. ACM, 2012. (CSCW98), 1998. [10] F. Flöck, D. Vrandeciˇ c,´ and E. Simperl. Towards a [26] D. Wilkinson, D.M. Wilkinson, and B.A. Huberman. diversity-minded Wikipedia. In Proceedings of the ACM 3rd Assessing the value of coooperation in wikipedia. Arxiv International Conference on Web Science 2011, 06 2011. preprint cs/0702140, 2007.

365