Download Preprint

SEQUENTIAL COLLABORATION 1 1 Sequential Collaboration: Comparing the Accuracy of Dependent, 2 Incremental Judgments to Wisdom of Crowds 1 2 3 Maren Mayer & Daniel W. Heck 1 4 University of Mannheim 2 5 Philipps University of Marburg SEQUENTIAL COLLABORATION 2 6 Author Note 7 8 Maren Mayer, Department of Psychology, School of Social Science, University of 9 Mannheim, Germany. https://orcid.org/0000-0002-6830-7768 10 Daniel W. Heck, Department of Psychology, Philipps University of Marburg, 11 Germany. https://orcid.org/0000-0002-6302-9252 12 Data and R scripts for the analyses are available at the Open Science 13 Framework(https://osf.io/96nsk/). nd 14 The present work was presented at the 62 Conference of Experimental 15 Psychologists (Virtual TeaP, 2021). The present manuscript has not yet been peer 16 reviewed. A preprint was uploaded to PsyArXiv and ResearchGate for timely 17 dissemination (version from June 2, 2021). 18 This work was supported by the Heidelberg Academy of Sciences and 19 Humanities (WIN project Shared Data Sources) and the Research Training Group 20 “Statistical Modeling in Psychology” funded by the German Research Foundation 21 (DFG grant GRK 2277). 22 The authors made the following contributions. Maren Mayer: Conceptualization, 23 Investigation, Methodology, Writing - Original Draft, Writing - Review & Editing; 24 Daniel W. Heck: Conceptualization, Methodology, Writing - Review & Editing. 25 Correspondence concerning this article should be addressed to Maren Mayer, B6, 26 30-32, 68169 Mannheim. E-mail: [email protected] SEQUENTIAL COLLABORATION 3 27 Abstract 28 In recent years, online collaborative projects in which users generate extensive 29 knowledge bases such as Wikipedia or OpenStreetMap have become increasingly 30 popular while yielding highly accurate information. Collaboration in such projects is 31 organized sequentially with one contributor creating an entry and the following 32 contributors deciding whether to adjust or maintain the presented information. We refer 33 to this process as sequential collaboration as individual judgments are dependent on the 34 latest judgment. As sequential collaboration has not yet been examined systematically, 35 we investigated whether dependent incremental judgment as obtained in sequential 36 collaboration become increasingly more accurate and whether the final judgments are 37 more accurate than estimates obtained with equally large groups in wisdom of crowds. 38 For this purpose, we conducted three preregistered studies with groups of four to six 39 contributors using general knowledge questions as well as geographic maps on which 40 cities had to be positioned. As expected, individual judgments in sequential 41 collaboration became more accurate and the final group estimates were slightly more 42 accurate than those based on aggregated judgments in wisdom of crowds. These results 43 show that sequential collaboration can profit from dependent incremental judgments, 44 thereby extending the literature on dependent judgments and shedding light on 45 collaboration in large-scale online collaborative projects. 46 Keywords: judgment and decision making, teamwork, mass collaboration, group 47 decision making SEQUENTIAL COLLABORATION 4 48 Sequential Collaboration: Comparing the Accuracy of Dependent, 49 Incremental Judgments to Wisdom of Crowds 50 Collaborative online projects that provide user-generated content have become a 51 popular source for information gathering and acquiring over the last twenty years. The 52 most prominent example is Wikipedia, an online encyclopedia that allows users to 53 contribute semantic information to various topics in the form of structured articles 54 (Wikipedia Contributors, 2021). Another less well known example of online 55 collaboration is OpenStreetMap, a collaborative project that aims at generating a 56 comprehensive, open, and free-to-use map of the world (OpenStreetMap Contributors, 57 2021). OpenStreetMap does not only comprise geographical numeric information about 58 the locations of objects such as coordinates but also semantic information such as 59 names of streets, areas, buildings and other useful information (e.g., addresses or 60 websites of shops and restaurants). Giles (2005) showed that Wikipedia is very accurate 61 in general. Moreover, certain topics such as information on cancer or certain drugs are 62 similarly accurate as official health information or text books (Kräenbring et al., 2014; 63 Leithner et al., 2010). Comparing the accuracy of OpenStreetMap with commercial 64 map providers or governmental sources also revealed a comparable accuracy (Girres & 65 Touya, 2010; Zheng & Zheng, 2014; Zielstra & Zipf, 2010). 66 The high accuracy of Wikipedia and other online collaborative projects has often 67 been attributed to wisdom of crowds (Arazy et al., 2006; Baeza-Yates & Saez-Trumper, 68 2015; Chen et al., 2010; Kittur et al., 2007; Kittur & Kraut, 2008; Niederer & Dijck, 69 2010). However, wisdom of crowds refers to a technique of aggregating independent 70 individual judgments (Galton, 1907; Larrick & Soll, 2006; Surowiecki, 2004). The high 71 accuracy of judgments in wisdom of crowds is due to the central limit theorem which 72 ensures that errors in independent, individual judgments cancel out (Hogarth, 1978). 73 Wisdom of crowds has been shown to yield highly accurate estimates for various tasks 74 and contexts (Hueffer et al., 2013; Keck & Tang, 2020; Larrick & Soll, 2006; Steyvers et 75 al., 2009; Wagner & Vinaimont, 2010). Aggregating independent individual judgments SEQUENTIAL COLLABORATION 5 76 is especially successful when judgments bracket the true answer (Larrick & Soll, 2006; 77 Simmons et al., 2011) and are negatively correlated and unbiased (Davis-Stober et al., 78 2014; Keck & Tang, 2020). 79 In contrast, judgments in online collaborative projects are not collected 80 independently and then aggregated afterwards, but rather elicited in a dependent and 81 sequential manner. Instead of providing independent individual judgments, contributors 82 encounter already existing entries and decide whether to change the presented 83 information which reflects the latest version of an entry or whether to leave the 84 presented information as it is. We refer to this way of collaborating as sequential 85 collaboration. 86 In the following, we will first describe the process of sequential collaboration, 87 distinguish it from other forms of collaboration, and embed it into already existing 88 research on dependent judgments which has shown both positive and detrimental effects 89 of dependency. Furthermore, we compare sequential collaboration and wisdom of 90 crowds to highlight why eliciting incremental, dependent judgments in sequential 91 collaboration can be beneficial for judgment accuracy compared to aggregating 92 independent judgments. In three studies, two of them preregistered, we used general 93 knowledge questions and maps on which cities should be positioned to test whether 94 sequential collaboration yields improved judgments within small groups of four to six 95 contributors. Moreover, we tested whether the final judgments at the end of a sequential 96 chain are more accurate than estimates obtained by aggregating independent individual 97 judgments in wisdom of crowds. In line with our hypotheses, we found that judgment 98 accuracy increased over the course of sequential chain and that sequential collaboration 99 yielded more accurate results than wisdom of crowds in two of the three studies. 100 Sequential Collaboration 101 As outlined above, collaboration in collaborative online projects is organized 102 sequentially by making incremental changes to the latest available information. 103 Sequential collaboration starts with one contributor creating an initial independent SEQUENTIAL COLLABORATION 6 104 entry. The following contributors who encounter this entry can then decide whether to 105 adjust or maintain the presented information. Whenever the entry is changed, the 106 information is updated such that only the latest version of the entry is presented to the 107 following contributors. For example, a first contributor might answer the question “How 108 tall is the Eiffel Tower?” with 420 meters. A second contributor encountering this 109 judgment could simply maintain it while a third contributor might adjust the height to 110 290 meters. After several contributors have adjusted and maintained the judgment, the 111 correct height of 300 meters may be entered. In the domain of geographical maps, the 112 first contributor could create an initial entry by outlining the layout of a buildingnot 113 yet mapped in OpenStreetMap. While a second contributor could improve the outline 114 of the building, a third might not change any information, and a fourth could add 115 semantic tags to describe that the building belongs to a university. Throughout several 116 sequential steps of adjusting and maintaining the entry, the building might finally be 117 represented by an adequate outline and be tagged as a university building with 118 additional information such as the university’s website and address. The sequence of 119 decisions whether to maintain or adjust entries made by a previous contributor forms a 120 sequential chain. Figure1 displays how group estimates are generated in sequential 121 collaboration and in wisdom of crowds. In the former, the final estimate is the last 122 judgment in a sequential chain generated by adjusting and maintaining previous 123

Load more