Studying the ``Wisdom of Crowds'' at Scale
Total Page:16
File Type:pdf, Size:1020Kb
Studying the “Wisdom of Crowds” at Scale Camelia Simoiu,1 Chiraag Sumanth,1 Alok Mysore,2 Sharad Goel1 1 Stanford University, 2University of California San Diego Abstract a county fair. He famously observed that the median of the guesses—1,207 pounds—was, remarkably, within 1% of the In a variety of problem domains, it has been observed that the true weight (Galton 1907). aggregate opinions of groups are often more accurate than those of the constituent individuals, a phenomenon that has Over the past century, there have been dozens of studies been dubbed the “wisdom of the crowd”. However, due to that document this “wisdom of crowds” effect (Surowiecki the varying contexts, sample sizes, methodologies, and scope 2005). Simple aggregation—as in the case of Galton’s ox of previous studies, it has been difficult to gauge the extent competition—has been successfully applied to aid predic- to which conclusions generalize. To investigate this ques- tion, inference, and decision making in a diverse range tion, we carried out a large online experiment to systemati- of contexts. For example, crowd judgments have been cally evaluate crowd performance on 1,000 questions across 50 topical domains. We further tested the effect of different used to successfully answer general knowledge ques- types of social influence on crowd performance. For exam- tions (Surowiecki 2005), identify phishing websites and ple, in one condition, participants could see the cumulative web spam (Moore and Clayton 2008; Liu et al. 2012), crowd answer before providing their own. In total, we col- forecast current political and economic events (Budescu lected more than 500,000 responses from nearly 2,000 par- and Chen 2014; Griffiths and Tenenbaum 2006; Hill and ticipants. We have three main results. First, averaged across Ready-Campbell 2011), predict sports outcomes (Herzog all questions, we find that the crowd indeed performs bet- and Hertwig 2011; Goel et al. 2010), and predict climate- ter than the average individual in the crowd—but we also related, social, and technological events (Hueffer et al. 2013; find substantial heterogeneity in performance across ques- Kaplan, Skogstad, and Girshick 1950). However, given the tions. Second, we find that crowd performance is generally diversity of experimental designs, subject pools, and analytic more consistent than that of individuals; as a result, the crowd does considerably better than individuals when performance methods employed, it has been difficult to know whether is computed on a full set of questions within a domain. Fi- these documented examples are a representative collection nally, we find that social influence can, in some instances, of a much larger space of tasks that exhibit a wisdom-of- lead to herding, decreasing crowd performance. Our findings crowds phenomenon, or conversely, whether they are highly illustrate some of the subtleties of the wisdom-of-crowds phe- specific instances of an interesting, though ultimately lim- nomenon, and provide insights for the design of social recom- ited occurrence. mendation platforms. Moreover, it is unclear whether these findings generalize to many real-world settings where individuals make deci- Introduction sions under the influence of others’ judgments. This question Are crowds mad or wise? In his 1841 book, “Memoirs of ex- is especially relevant today, as peer influence is oftentimes traordinary popular delusions and the madness of crowds,” explicitly built into online platforms. One might choose a Charles Mackay documents a series of remarkable tales of restaurant, watch a movie, read a news story, or purchase human folly, ranging from the hysteria of the South Sea a book because of the aggregated opinions of the “crowd.” Bubble that ruined many British investors in the 1720s, Recommender systems may display top-rated products first to Holland’s seventeenth-century “tulipomania”, when in- by default, whose quality has been estimated as the most dividuals went into debt collecting tulip bulbs until a sud- popular or highly voted. In recent years, researchers have den depreciation in the bulbs’ value rendered them worth- debated whether social influence undermines or enhances less (Mackay 1841). Decades later, in yet another classic the wisdom of crowds. On the one hand, some have con- example, the statistician Francis Galton watched as eight jectured that if participants receive information about the hundred people competed to guess the weight of an ox at answers of others, that can help ground responses, leading to greater accuracy (Faria et al. 2010; King et al. 2012; Copyright c 2019, Association for the Advancement of Artificial Madirolas and de Polavieja 2015). But, on the other hand, Intelligence (www.aaai.org). All rights reserved. there is also worry that such social influence could result in herding, which in turn could decrease collective perfor- tics (Lorenz et al. 2011), rank ordering problems (e.g., rank- mance (Lorenz et al. 2011; Muchnik, Aral, and Taylor 2013; ing U.S. presidents in chronological order) (Lee, Steyvers, Salganik, Dodds, and Watts 2006). and Miller 2014; Miller and Steyvers 2011), recollecting To systematically explore the wisdom-of-crowds information from memory (Steyvers et al. 2009), and spa- phenomenon—including the effects of social influence—we tial reasoning tasks (Surowiecki 2005). But not all studies carried out a large-scale, online experiment. In one of have been able to replicate this success. For example, Bur- the most comprehensive studies of the wisdom-of-crowds nap et al. consider crowd evaluation of engineering design effect to date, we collected a total of more than 500,000 attributes and find that clusters of consistently wrong eval- responses to 1,000 questions across 50 topical areas. For uators exist along with the cluster of experts. The authors each question, we computed the “crowd” answer by either conclude that both averaging evaluations and a crowd con- taking the median response of participants (in the case sensus model may not be adequate for engineering design of open-ended, numerical questions) or the most popular tasks (Burnap et al. 2015). choice (in the case of categorical questions). This lack of consensus is also evident among the set Averaged across our full set of questions, we found that of studies that consider prediction domains. In the con- the crowd answer was approximately in the 65th percentile text of predicting outcomes for competitive sporting tour- of individual responses, ranked by accuracy. Our results thus naments, collective forecasts were found to consistently per- lend support to the idea that the wisdom-of-crowds effect form above chance and to be as accurate as predictions based indeed holds on a corpus chosen to reflect a wide variety of on official rankings (Herzog and Hertwig 2011). In another topical areas. Further, we found that crowd performance was study involving a competitive bidding task, Lee et al. con- typically more consistent than the performance of individu- sidered eleven different methods to aggregate answers, and als. That is, whereas the crowd performed at least modestly found that aggregation improves performance (Lee, Zhang, better than average on all of the questions, even the best in- and Shi 2011). In contrast, in the betting context consid- dividuals occasionally performed poorly. As a result, when ered by Simmons et al., the authors found no evidence we looked at performance at the level of topical domains, of a wisdom-of-crowds phenomenon. The authors attribute rather than individual questions, the crowd performed con- the failure to the fact that “most bettors have high intu- siderably better than individual respondents, with average itive confidence and are therefore quite reluctant to aban- performance in approximately the 85th percentile. don it”. Similarly, crowd predictions made by thousands Finally, we examined the effect of social influence, ran- of people competing in a fantasy football league were domly assigning participants to one of three different social found to predict favorites in over 90% of the games, even conditions: (1) “concensus”, in which participants saw the though favorites and underdogs were equally likely to win cumulative crowd response before providing their own an- against the spread (Simmons et al. 2010). These studies swer; (2) “most recent”, in which participants saw the three suggest that crowd wisdom may not prevail in contexts in most recent answers; and (3) “most confident”, in which par- which emotional, intuitive responses conflict with more ra- ticipants saw three answers from the most confident individ- tional, deliberative responses (Tversky and Kahneman 2000; uals, based on self-reported assessments. For the latter two Simmons et al. 2010). conditions—“most recent” and “most confident”—we found Several studies focus on the question of how to best ex- that crowd performance was qualitatively similar to the non- tract collective wisdom. Numerous studies have shown that social, control condition. However, for the “consensus” con- simple aggregation techniques (e.g., using the mean or me- dition, the crowd performed worse than when respondents dian for open-ended questions, or the majority vote for did not receive any social signals. Notably, this consensus categorical questions) often perform just as well as more condition mirrors the design of many online rating sites, in complex methods, including confidence-weighted aggrega- which users can see the aggregate rating of others before tion, Bayesian methods, and the Thurstonian latent variable providing their own rating. While such a design has value model (Miller and Steyvers 2011; Griffiths and Tenenbaum (e.g., it facilitates use by those who simply want to see the 2006; Prelec, Seung, and McCoy 2017; Budescu and Chen information, rather than providing a review themselves), our 2014; Hemmer, Steyvers, and Miller 2010). Simple aggrega- results suggest that it can also hurt the quality of results. tion, however, has often been found to perform reasonably well, if not on par with more complex models (Steyvers et Related Work al.