editorial The wisdom of crowds, for a fee Paid is coming to biology. This powerful approach will support research, though it could also promote unjust conditions for some workers.

he participation of large groups of Turk (AMT) allows anyone—a researcher, AMT workers, canvassed over a two-week amateurs in scholarly projects is for example—to post a task for payment. period, reported that they derive their Tnothing new. The Oxford English Online workers then decide whether to take principal income from online microwork. Dictionary was famously a massive collation on this task given the time involved and the Most workers are in the , of examples of word use amassed by a level of compensation. but several studies report an increasing dispersed group of annotators combing In this month’s issue of Nature Methods, contribution from people in India. Not through published texts. The Audubon researchers describe Quanti.us, a tool that surprisingly, average payment rates on Christmas Bird Count, in which tens of enables such transactions for crowdsourced AMT are estimated to be well below the thousands of volunteers contribute to an annotation of biological images (p587). US federal (which, at annual bird census, has been running The authors conclude that for object $7.25 an hour, is itself well below the for over a century. After World War II, discrimination, tracking, and segmentation, minimum wage in several US states). subatomic particles were hunted by the performance of a collective of paid Indeed, at 6 cents per image, the image- highly trained but non-scientist scanners, non-expert workers can almost always analysis tasks described in this issue are typically women, who annotated data equal that of experts and exceed that of likely to be examples of this. from bubble chambers. automated methods. They also show that The forces that compel people to take Present-day crowdsourcing depends a model trained on crowd- underpaid jobs are many; microwork heavily on digital information and accesses annotated images can perform just as well platforms cannot be held responsible for the ‘crowd’ via the . In biology, as one trained on expert-annotated ones. them. What is more, this type of flexible crowdsourcing has been used to solve Also in this issue, scientists with experience work is likely to be beneficial for some protein structures, to reconstruct neuronal in volunteer-driven crowdsourcing discuss as a source of supplementary income or connections in electron micrographs, to the strengths and weaknesses of the paid and a gap measure. Paid online microwork pick particles for cryo-electron microscopy, volunteer models (p579). could be considered as a part of the now to curate a bacterial interactome, and Quanti.us may be used in a paid or a prevalent ‘gig economy’, where workers to detect sleep spindles in human free mode. In the latter case, a project could take on short-term projects as independent electroencephalograms, to give just be distributed to a community of students, contractors rather than assuming a stable a few examples. colleagues or other collaborators. In the arrangement as an employee. Certainly, In most such projects, the crowd consists paid mode, worker engagement depends on traditional institutions like universities and of volunteers who have an interest in the both the complexity of the task (for instance, businesses are no strangers to the practice work at hand. Indeed, in its earliest sense, how many annotations are required per of lowering labor costs by hiring temporary the term “amateur” derives from amator, or image) and the pay. Not surprisingly, a more faculty or . In other words, paid “one who loves.” Even so, it can be difficult complex job needs to offer better pay for microwork is part of more general economic to keep the sustained attention of a volunteer large numbers of workers to engage. The and societal trends. group. For time-sensitive or very tedious authors report that they can find conditions Nevertheless, researchers planning projects, the myriad attractions of life and that result in much faster image annotation crowdsourcing projects should keep in the internet may offer volunteer citizen compared to what can be achieved in the mind that paid microwork could perpetuate scientists more appealing ways to spend typical lab or extrapolated from a volunteer inequality and even be exploitative for some their spare time. platform. workers. An online community of AMT Two other approaches can keep the Fast and accurate crowdsourced image workers has released sensible guidelines crowd engaged. In the first, the scientific annotation could be a boon for many that are worth considering. Should the use task is incorporated into an existing research projects. It will free scientists of paid crowdsourcing in biology become online game that already has a large and from tedium, provide annotations to train widespread, the scientific community would enthusiastic following. For instance, powerful algorithms, and do well to come to a wider agreement about researchers at the Human Protein Atlas, make possible analyses that might otherwise how to promote its principled use. which aims to map the location of all be too onerous to even contemplate. But Crowdsourcing offers a powerful and proteins in cells and tissues, have embedded paid online microwork has a less positive exciting new way to transform biological imaging data into the role-playing game side as well. data into knowledge and thus will benefit EVE Online, opening a portal to its roughly Paid crowdsourcing takes place in an society in the long run. We should remain half-million subscribed gamers. It will be unregulated labor market where workers vigilant that this does not come at the of great interest for the research community have neither formal collective bargaining expense of the economically vulnerable in to evaluate the accuracy of data obtained power nor basic government protection the shorter term. ❐ in this way. (although AMT workers do organize online Alternatively, contributors to in active communities where they review crowdsourced projects can be paid for their requesters and tasks). A 2016 Pew Research Published online: 31 July 2018 efforts. A platform like Mechanical Center study found that a full quarter of https://doi.org/10.1038/s41592-018-0101-4

Nature Methods | VOL 15 | AUGUST 2018 | 555 | www.nature.com/naturemethods 555