Citizen science frontiers: Efficiency, engagement, and serendipitous discovery with human–machine systems

Laura Trouillea,b,1, Chris J. Lintottc, and Lucy F. Fortsond

aDepartment of Citizen Science, The Adler Planetarium, Chicago, IL 60605; bCenter for Interdisciplinary Exploration and Research in Astrophysics, Northwestern University, Evanston, IL 60208; cDepartment of Physics, The University of Oxford, Oxford, OX1 3RH, United Kingdom; and dDepartment of Physics and Astronomy, The University of Minnesota-Twin Cities, Minneapolis, MN 55455

Edited by Youngmoo E. Kim, Drexel University, Philadelphia, PA, and accepted by Editorial Board Member Eva Tardos December 7, 2018 (received for review May 20, 2018)

Citizen science has proved to be a unique and effective tool in large, diverse datasets. In this Perspectives piece, we describe in helping science and society cope with the ever-growing data our recent efforts and future considerations for designing a rates and volumes that characterize the modern research land- human–machine system optimized for “happy chance discov- scape. It also serves a critical role in engaging the public with ery” (a definition of serendipity that provided guidance for the research in a direct, authentic fashion and by doing so promotes Cybernetic Serendipity exhibit) that best takes advantage of the a better understanding of the processes of science. To take full efficiencies of the machine while acknowledging the complexity advantage of the onslaught of data being experienced across of human motivation and engagement. the disciplines, it is essential that citizen science platforms lever- What is now called citizen science—the involvement of the age the complementary strengths of humans and machines. This general public in research—has a long history. An early example Perspectives piece explores the issues encountered in designing is Edmund Halley’s study of timings during the 1715 total human–machine systems optimized for both efficiency and vol- solar eclipse, which included observations from a distributed, unteer engagement, while striving to safeguard and encourage self-organized group of observers (1). Works by refs. 2 and opportunities for serendipitous discovery. We discuss case stud- 3, among others, have linked modern-day efforts to their ies from Zooniverse, a large online citizen science platform, and 19th century antecedents, for example, highlighting the role show that combining human and machine classifications can effi- played by amateur networks of meteorological observers in ciently produce results superior to those of either one alone establishing that field of study (i.e., by 1900, more than 3,400 and how smart task allocation can lead to further efficiencies observers were contributing data to a network organized by in the system. While these examples make clear the promise of George Symons, producing data on a scale that could not be human–machine integration within an online citizen science sys- matched by the professional efforts of the time). In recent tem, we then explore in detail how system design choices can decades, citizen science has gained renewed prominence, inadvertently lower volunteer engagement, create exclusionary boosted in part by technological advances and digital tools practices, and reduce opportunity for serendipitous discovery. like mobile applications, cloud computing, and wireless and Throughout we investigate the tensions that arise when design- sensor technology which have enabled new modes of public ing a human–machine system serving the dual goals of carry- engagement in research (4) and facilitated research projects ing out research in the most efficient manner possible while that investigate questions from data at scales beyond the empowering a broad community to authentically engage in this professional research community’s resource capacity (5). research. Professional citizen science organizations have been created in Europe, Australia, and the United States. In the United States, citizen science | machine learning | human computing interaction | the Crowdsourcing and Citizen Science Act of 2015 was physical sciences | biological sciences introduced to encourage the use of citizen science within the federal government and, that same year, the first Citizen he 1968 Cybernetic Serendipity exhibition (www. Science Association (CSA) conference was held (although Tstudiointernational.com/index.php/cybernetic-serendipity- some consider the 2012 European Space Agency side event 50th-anniversary) was an early imagining and exploration of on citizen science the first CSA gathering). CitizenScience.gov computer-aided creative activity, play, and interplay. The exhibit, (https://www.CitizenScience.gov) currently lists over 400 active curated by Jasia Reichardt at the Institute of Contemporary Arts citizen science projects. Participation in citizen science today in London, examined the role of cybernetics in contemporary art ranges from hands-on data collection, tagging, analysis, and and included robots; algorithmically generated movies, poetry, research projects [e.g., iNaturalist.org (https://www.iNaturalist. and music; painting machines; and kinetic interactives. Several org) (research grade observations: https://www.gbif.org/dataset/ of the works featured chance as an important ingredient in the 50c9509d-22c7-4a22-a47d-8c48425ef4a7), eBird.org (https:// creative process, reflected in the priority given in the exhibition’s www.eBird.org) (6), and CitSci.org (https://www.CitSci.org) (7)] emphasis on machine-enabled serendipity. It is useful to reflect, 50 y later, whether machines have indeed enabled serendip- itous discovery, albeit in the realm of science rather than This paper results from the Arthur M. Sackler Colloquium of the National Academy of Sciences, “Creativity and Collaboration: Revisiting Cybernetic Serendipity,” held March the arts. 13–14, 2018, at the National Academy of Sciences in Washington, DC. The complete pro- We consider this concept in the context of online citizen sci- gram and video recordings of most presentations are available on the NAS website at ence projects. These projects, which massively share the task www.nasonline.org/Cybernetic Serendipity.y of data analysis among a crowd of volunteers, in many ways Author contributions: L.T., C.J.L., and L.F.F. performed research; and L.T., C.J.L., and L.F.F. exemplify the promise of those early ideas, providing a com- wrote the paper.y pelling modern example of the transformative power of the The authors declare no conflict of interest.y integration of human and machine effort. Online citizen sci- This article is a PNAS Direct Submission. Y.E.K. is a guest editor invited by the Editorial ence not only is a powerful tool for efficiently processing our Board.y growing data rates and volumes (the “known knowns”), but Published under the PNAS license.y also can function as a means of enabling serendipitous discov- 1 To whom correspondence should be addressed. Email: [email protected] ery of the “known unknowns” and the “unknown unknowns” Published online February 4, 2019.

1902–1909 | PNAS | February 5, 2019 | vol. 116 | no. 6 www.pnas.org/cgi/doi/10.1073/pnas.1807190116 Downloaded by guest on September 27, 2021 Downloaded by guest on September 27, 2021 eiin(6,SihoinTasrp etr(https://siarchives. si.edu/collections/siris Center Transcript the Smithsonian (16), in (https://fromthepage.com), Veridian efforts Page the transcription From crowdsourced including humanities, online of number ing rcuelnsae ic h anhi 07o h Galaxy the infras- of research 2007 al. et the Trouille in of launch part the world, core Since time the a a landscape. across At prominence become tructure arts. gaining cancer has the is to zoology, Zooniverse science history to and citizen science, astronomy when exist- climate from its to disciplines, research of researchers scale of many hundreds the with across (iv) partners Zooniverse and (DIY) Platform) audience. below; a Builder ing do-it-yourself described Project for the free, as as used capabilities (iii) known be (also tasks; Builder” can “Project development open- appli- which scalable of shared (API), users and variety (i) interface from flexible, programing its reliable, input (ii) cation of and disciplines; result expertise, the across a experience, online as among software, unique regis- source platforms is million It online science 1.7 world. for with citizen the platform projects around 120 largest participants over the tered to is host paper, science, this citizen of remainder that the understanding (30). and process collective 29), a is (23, progress pro- literacy scientific knowledge scientific scientific (21–24), domain the confidence in (25–28), in increases minorities and (20); and cess women increased the of 19); (18, representation influ- making to decision environmental communities local of ence empowerment and the 18 long- therein); ref. (e.g., in references interests research increases and civic, including environmental, research, term scientific and/or in participation computing distributed for efforts, resources storage. computing” a computing har- enabled ness “volunteer/distributed which also (https://setiathome.berkeley.edu), of SETI@Home have like track advances Technological parallel platforms. (https://SciStarter. and SciStarter.org and org CitizenScience.gov See Cosmoquest and (15) Eyewire examples include Other ( synthesized science laboratories. citizen then tuber- medical online are Stanford’s like of in structures diseases tested for molecular and solutions new the to find These ways to of new culosis. molecules design generation to RNA next players Eterna fold challenges [the pattern the (14)] environment platform example, for FoldIT gaming For talent online detection. human data, (13) anomaly of inherently and quantities the large recognition of involving diverse advantage problems from taking teams solve research to enables domains analysis, data distributed (e.g., study of fields these peer-reviewed on impact of 12). major into ref. a dozens and with results date to efforts, articles their launched in annually recently upload participate these universities the and 300 annotate Over databases. them, them, their national in from characterize under- bacteria samples the other genomes, for and isolate soil curricula environments, Education collect local Sea-Phages standardized Genomic to efforts example, the students provide graduate and science for (11) (10), citizen settings; Initiative Partnership of World classroom explosion Small below. in (9), an detail out been more carried also in has described There efforts, data-processing line (https://www.epa.gov/urbanwaterspartners/ diverse-partners-brownfields-healthfields-la-watershed LA Project the community [e.g., researchers with Watershed with sensors collaboration in low-cost working environmen- using members cocreated projects of number monitoring Taste growing of tal Genetics a Science to of (8)] Museum Laboratory Denver data the hands-on [e.g., in analysis participating and data in-person contributing to https://cosmoquest.org ,wihw ou nin on focus we which (https://www.zooniverse.org), Zooniverse public of impacts positive the outlined have studies Numerous of method proven a become has which science, citizen Online o opeesv itnso iie cec projects science citizen of listings comprehensive for ) sic ,aogohr.Teeaeas grow- a also are There others. among ), 14645 ,adTasrb eta (17). Bentham Transcribe and ), ]t on- to )] h anhi uy21 ftefe rjc ule Platform Builder Project free the of of result 2015 a July is ( in which acceleration launch an the growth, rapid experienced relevant the and high- URL project project specific on citations. Zooniverse the citi- projects each reference myriad online we With from lighted, raised into platform. experiences pro- Zooniverse questions learning the paper on the and machine based This learned of science community. zen integration lessons scientific data the of quality wider around compilation producing the a a science by vides established citizen use have online projects for of 38), These and record (37, 44–46). 43), track humanities (42, 32, science (33–36), (31, climate ecology 41), astronomy in (40, physics over e.g., (39), zooniverse.org/publications list), to biomedicine (see full led the disciplines contribu- have for many significant projects across enabling Zooniverse tions publications, 32), peer-reviewed (31, 150 project Zoo icvr ihihstepwro ak(0.Afwvolunteers few A (50). Their Talk away. of far power very not the are rates. highlights they formation young), discovery (which are low star metallicity they low that are their high indicates despite because galaxies with surprise and a galaxies green were formation They metallicity compact the low These of mass, research. galaxies. (49) understanding of of Peas” “Green theoretical evolution fields Zoo’s our Galaxy frontier of pushed transformed discovery the have example, long-term For years and the interest discoveries, over attract novel to of means enabler a (48). collab- an engagement question- as and their as indirectly, driven in and and goal directly, function to both project social answering the purely in benefiting the interactions orative, from The building. range collaboration community Talk serendipitous social vol- between facilitating enabling exchange and researchers, ideas and and wide sharing unteers a for information space to of a central range provides been (47). Talk have investigations. task citizen-led forums classification these discussion noticed main “Talk” they Zooniverse prescribed unusual something The the an on came performing based flag they questions typically while to beyond research themselves, project on far with work given up adventured and/or a interest not of of object than framework their often original on science the more based citizen the have inquiry modern notice on scientific platforms 19th to participants of the volunteer ability in lines observations, scientists human new amateur innate defined as many our enabled century Just so and unexpected. and and having data unusual encouraged the of on advantage has eyes taking platform (31, discovery, (https://GalaxyZoo.org) Zooniverse serendipitous Zoo the Galaxy project 32), first its From Serendipity—Finding Unexpected Cybernetic the for Design in Lessons Early 50 over and 2017, 3–5 in 44 launching 2016, in 2018. from 26 in launching went to year Zooniverse professional per projects of time. years Project development to The months web required end. project online front typical science a displays citizen Builder development 1 its Project before Fig. transformative; the is project. Builder the from research, from discus- shots the results a about screen and content interface; team, for classification research pages the a “About” and page; forum; landing sion projects a Builder project’s with Project the create All come to website. out and fills of interface researcher series classification a a is boxes end text front and Builder forms Project The tools. drawing mark- and and classi- ing tasks, including Builder comparison interaction questions, within Project of multiple-choice cost, The types fication, common no toolkit. most at browser-based the project web supports science a citizen using online hours, an deploy and https://www.zooniverse.org/lab h ubro rjcsspotdb onvrehsrecently has Zooniverse by supported projects of number The h eedptu icvre aeb onvrevolunteers Zooniverse by made discoveries serendipitous The PNAS | eray5 2019 5, February hc nbe noet build to anyone enables which ) | o.116 vol. | o 6 no. | 1903

ECOLOGY COMPUTER SCIENCES COLLOQUIUM PAPER Fig. 1. Screenshots of Zooniverse’s free Project Builder Platform (https://www.zooniverse.org/lab). As the user inputs content into the Project Builder interface (Left), he or she can immediately view changes in the associated public-facing website (Right). Here we have displayed the “Workflow” section of the Project Builder (in which the user sets the classification task the volunteer will carry out) and the associated “Classify” page within the project’s actual website that it creates. Through the tabs located along the left-hand side of the Project Builder interface, project leads upload all of the necessary content and data for their crowdsourced research project, from inputting information about their research goals and why they need volunteers’ help in the About tab to uploading their subjects that need classifications in the “Subject Sets” tab to exporting the raw classifications provided by the volunteers through the “Data Exports” tab. The Project Builder is democratizing access to online citizen science as a tool for research and has enabled the accelerated expansion of the Zooniverse. Since its launch in July 2015, the Zooniverse has gone from launching 3–5 projects each year to launching over 50 in 2018.

had noticed the strange, compact, green blobs while classifying algorithms to automatically process the remaining data with and had started a Talk discussion forum thread humorously confidence. titled “Give peas a chance.” The researchers worked alongside the volunteers (who referred to themselves as the “Peas Corps”) Combining Human and Machine Classifiers. The current Supernova to refine the collection of objects, ultimately identifying 250 Hunters project (https://www.zooniverse.org/projects/dwright04/ Green Peas in the million-galaxy dataset. Even if the researchers supernova-hunters) provides an illustrative example of the addi- had managed to examine 10,000 of the images, they would tional power that comes from combining human and machine have only come across a few ‘Green Peas’ and would not have classifications. Each week, thousands of new Pan-STARRS tele- recognized them as a unique class of galaxies (49). Numerous scope images are flagged by machine-learning routines as con- other examples of serendipitous discovery pepper Zooniverse’s taining potential supernovae candidates. Subjects which the history—from Hanny’s Voorwerp, the ghost remnant of a model deems unlikely to be a supernova are automatically supermassive black hole outflow, offset from its central Galaxy rejected and the remaining subjects (∼5,000 each week) are Zoo galaxy and of which only a few dozen other examples uploaded for our volunteers to classify. Ref. 54 found that have been observed (51), to the discovery of a group of 19th the human classifications and machine-learning results in Pan- century female scientific illustrators and writers [a volunteer STARRS were complementary; the human classifications pro- noticed the name “Mabel Rhodes” while annotating 19th vide a pure but incomplete sample while the machine classifica- century scientific journal pages as part of the Science Gossip tions provide a complete but impure sample. By combining the project (https://www.ScienceGossip.org) and spurred a cohesive aggregated volunteer classifications with the machine-learning search and collection effort and a new research direction for results, they are able to create the most pure and complete the project (https://talk.sciencegossip.org/#/boards/BSC0000004/ sample of new supernovae candidates. discussions/DSC00004s8)]. Transfer Learning, Cascade Filtering, and Early Retirement. The Experiments in Machine Integration Camera CATalogue project (https://www.zooniverse.org/ As we enter an era of growing data rates and volumes [e.g., projects/panthera-research/camera-catalogue) further took ad- the 30 TB of data each night that will be produced by astron- vantage of the different strengths in human–machine integration omy’s Large Synoptic Survey Telescope (52) or the thousands to more efficiently classify new data. Through this project, of terabytes produced by ecology projects annually], Zooniverse the Panthera conservation organization harnesses the power has been moving toward a more complex system design, one of the crowd to tag different species in camera trap images. that takes better advantage of the complementary strengths of The researchers first used a transfer learning technique (55) humans and machines, integrating these efforts to optimize for to train a model specific to South Africa based on the much both efficiency and volunteer engagement, while striving to safe- larger Snapshot Serengeti (https://www.SnapshotSerengeti.org) guard and encourage opportunities for serendipitous discovery. dataset. The images were then passed through the logic of Below we provide a few examples of early efforts to integrate “cascade filtering” wherein the task is broken into a sequence humans and machines. of simple “yes/no” questions (e.g., ‘blank/not blank’) which In its simplest form, a number of projects have used vol- volunteers could easily do through the Zooniverse mobile app. unteer classifications to generate training sets for automated Furthermore, instead of requiring a default of five volunteer methods to efficiently classify all remaining data. For exam- classifiers to classify each image, the project used new Zooni- ple, an early project on Zooniverse, Galaxy Zoo: Supernova verse system infrastructure to automatically retire an image if (53), using data from the Palomar Transient Factory, retired only one to two volunteers agreed with the model prediction. from the system after the volunteer classifications provided a This combination of human–machine classifiers reduced hu- large enough training set for the researchers’ machine-learning man effort by 43% while maintaining overall accuracy and

1904 | www.pnas.org/cgi/doi/10.1073/pnas.1807190116 Trouille et al. Downloaded by guest on September 27, 2021 Downloaded by guest on September 27, 2021 hs outeswobcm epyeggdi n attached and in truly engaged for is that deeply it is become where though, who effort noting, volunteers human worth those focus nuance to important An us needed. allowing over the victory, take of) of can a (most machines remainder because is and project team the a retiring development process volunteers, web the training reliably Zooniverse enough the to For large machines data. a the provided for had set humans from retire once to able system was the that project) project Hunters Zooniverse Supernova a first of (the example (24, an cited research we integration, learning to contributing may as or may may who participation 63–66). who volunteers 27, their and for dif- science view so with and not relationship complex especially complex be is a experi- to This which have likely one systems predict. is in just to tasks behavior is ficult scientific volunteers’ potential in This that them clear volunteer’s maximized. involve is each be it but if to ment, is critical contribution is for efficiency, expe- user of not for psychology optimizing rewards. rience, how intermittent the highlights for and experiment invested simple design is This which effort game phenomenon where to machines, a as related slot is classifications, engagement suggested of volunteer we number found decreased 62 by blanks ref. measured However, all system. removing Zooniverse the of that their “blanks” efficiency known the all into remove increase to to (those best upload seem subjects would before it “blank” Naively project. animals) of no fraction projects certain containing trap a human– camera auto- remove the a (https://SnapshotWisconsin.org) matically ecology convey in Wisconsin example, to Snapshot efficiency like hope For for we designing system. when following, machine citi- exist the online that In an tensions system. within science integration zen human–machine of Engagement. promise and Efficiency Tensions Design 61. high and examples 60 other with refs. For volunteer see (45). type, one this images least of those gravitational at reviewed no if skill eliminated measured containing be as could allocation. classified lenses) Warps wrongly improve nega- Space (images false to the tives that within information demonstrated clas- studies performance (https://SpaceWarps.org) this assignment project used volunteer task used studies of example, and For these estimates classifications ability of derive of or Each to number 57–59). data total sification refs. vol- the (e.g., to reducing needed assignment task greatly judicious demonstrating unteers, through projects, applica- exist Zooniverse practical efficiencies from and that data work using theoretical often of tion, body growing a from Assignment. Task took it Intelligent d 400+ alone. the dataset with the over compared complete time produced to in volunteers have decrease human would 8 it of project, factor live a system the this on had Had deployed dataset. machine been galaxy-image the 200,000 online entire classifications, through the 30, over human retired day retired the and By alongside 12 application. day first learning by its trained on fully images was 70,000 on machine time The real 8. near day in off-the-shelf training an started the simulations, retire algorithm 56 to ref. machine-learning 2011) the to in 2010 contrast, the (from In y of dataset. on 1 full dataset over classified for 2 project Zoo volunteers 2 Zoo Galaxy Zooniverse Galaxy the images. with galaxy experiments ref. example, of 200,000 system For series human–machine two. a the ran the of 56 in integration real-time sophistication near of through level next the Integration. smaller Real-Time our Near for learning transfer projects. of trap camera utility the demonstrated rulee al. et Trouille saohreape nordsuso bv ferymachine- early of above discussion our in example, another As onvrehsbgnt explore to begun has Zooniverse h bv xmlsotiethe outline examples above The diinlpors a stemmed has progress Additional ahn nerto ih“eeigUp.” ( “Leveling with Integration Machine images is Future the contradiction shown for Strategies the if needed, work not more are do apparent. classifications but which directly that research, for tasks only to complete most to contribute con- the want authentic volunteers of an If one make tribution. volunteers—to undermine of also volun- motivations of but use common enthusiasm, wasteful and a time represent only teer not image. each would artificially for this required enhanced by However, classifications classify of been number to the have opportunity increasing could greater providing inclusivity through Supernova social the the clear in in project, example, not efficiency For Hunters is preferable. scientific is it inclusivity, reducing ideal, of designing of name not sci- Although alternative, is more exclusive. the exclusivity is whether more via efficiency project is for a it projects where if For instances efficient nuanced. be entifically is may efficiency, there scientific example, and inclusivity social aims, (social to research real community with ways broad inclusivity). meaningful and a valued possible empowering in manner engage and efficient carry- most efficiency) of the goals in (scientific research opposing for authentic occasionally out potential the ing the serving examine the when to however, tension opportunity of the importantly, middle the provides of More the example distribution in participate. broader a not could more when (i.e., a population hours) by release work during simply data week alleviated that the the be note of could We timing trends volunteers. judicious these dedicated possible highly is of it cohort made are small classifi- classifications a the volunteer by the of addition of majority currently terms In the contributions; is in olds. cation project Hunters 75-y Zooniverse Supernova to unequal with bias, olds most female 18-y demographics from 45% the distribution to are age overall (65+). flat projects group a age Zooniverse oldest the contrast, bias, in In gender males and among age popular Supernova most strong the being a example, has For community frequently. volunteer more and Hunters inter- project scarcity to a volunteers with perceived motivating act classification that beyond drive effects cautions to have 67 used competition ref. is scarcity However, artificial activity. where mode a in artificial creating of engagement. point encourage the of to format to scarcity) similar (even a release adopt data dedi- projects good incremental other a of that seem level recommend might it to high project, idea extremely this in the engagement other Given volunteer the beating cated data. on the of relies data to opportunity one new this volunteers of but discover amount week, small each to the released in person present a likely first candidates is novae the There possibility legitimate data. being the clas- week’s to of enough due that competition h of retire 48 sense to Within heightened h. submitted over 24 are averages first sifications project the the in weekly classifications and the 20,000 announcement out weekly sends the pre- and cedes activity morning classification Volunteer Tuesday after. the shortly a uploads announcement researcher on the subjects week Each new choices. design of impact hunters (https://www.zooniverse.org/projects/dwright04/supernova- ect Inclusion. and Efficiency difficult, processed being be now are can but their in system machines. coming supernovae), by the still are for from data if retirement searching especially its (e.g., of project experience particular a to adi nertn uasadmcie notesse and system the into machines and humans integrating in ward https://www.GravitySpy.org e.6 xlrshwterltosi ewe hs two these between relationship the how explores 67 Ref. operating imagine might one plentiful, are data when Even rvdsadfeetln hog hc ocnie the consider to which through lens different a provides ) PNAS | h urn uenv utr proj- Hunters Supernova current The eray5 2019 5, February 4,6–2 ugssoewyfor- way one suggests 68–72) (41, ) h rvt p project Spy Gravity The | o.116 vol. | 2super- ∼12 o 6 no. | 1905

ECOLOGY COMPUTER SCIENCES COLLOQUIUM PAPER optimizing for both efficiency and engagement. Through Grav- A recent study by ref. 75 finds that volunteers who experienced ity Spy, the public helps to categorize images of time-frequency the machine-learning scaffolded training performed significantly representations of gravitational wave detector glitches from the better on the task (an average accuracy of 90% vs. 54%), con- Laser Interferometer Gravitational Wave Observatory (LIGO) tributed more work (an average of 228 classifications vs. 121 into preidentified morphological classes and to discover new classifications), and were retained in the project for a longer classes that appear as the detectors evolve. This is one of our period (an average of 2.5 sessions vs. 2 sessions). The project also most popular projects, with over 12,000 registered participants exemplifies how curious citizen scientists are well situated for having provided over 3 million classifications, and has led to the serendipitous discovery of unusual objects. Gravity Spy volun- accurate classification of tens of thousands of LIGO glitches and teers have identified a number of new glitch categories, including the identification of multiple prominent and previously unknown the discovery of the “Paired Doves” class which has proved sig- glitch classes (73, 74). This popularity has been a surprise given nificant in LIGO detector characterization endeavors, as this that the images are not particularly “pretty” and the goal is to glitch resembles signals from compact binary inspirals and is categorize noise (not to directly discover a new gravitational therefore detrimental to the search for such astrophysical signals wave signal). The project’s success is a good reminder to not in LIGO data (73, 74). make assumptions on what will or will not prove engaging to the public. Machine Integration with Intelligent Task Assignment. Another In parallel with the human effort in Gravity Spy, a deep- example of the potential for thoughtful integration of machines learning model with convolutional neural network (CNN) layers within online citizen science systems comes from the Etch-a- is used to categorize images after being trained on human- cell project (https://www.zooniverse.org/projects/h-spiers/etch-a- classified examples of the morphological classes. The system also cell). Here volunteers provide detailed tracings of the boundaries maintains a model of each volunteer’s ability to classify glitches of cellular components such as the nucleus, as seen in extremely of each class and updates the model after each classification high-resolution microscopy. Modern instruments can slice a cell (i.e., increasing its estimate of the volunteer’s ability when he to produce a stack of thousands of such images, and the Etch- or she agrees with an assessment and decreasing it if he or she a-cell project aims to assemble 3D representations from these disagrees). When the volunteer model shows that a volunteer’s segmentations. At present, this is achieved by having volunteers ability is above a certain threshold, the volunteer advances to the work through each image in the stack separately and combin- next workflow level, in which he or she is presented with new ing the results. One can easily imagine using machine learning to classes of glitches and/or glitches with lower machine-learning determine where user intervention is really required; this could confidence scores. Volunteers progress through five levels within be via simple change detection or something as complex as a the Gravity Spy system, choosing from 22 different glitch classes recurrent neural network (RNN). The net effect would be to save in Level 5. In addition, images which neither the volunteers volunteer effort while increasing the variety of the experience. nor the machine confidently classify as a known glitch class are This kind of intervention, which increases both the effective- moved from one workflow level to the next through the “None of ness of volunteer contributions and the variety of the task, is, we the Above” category. The volunteers, in concert with machine- suggest, more likely to produce successful projects. learning efforts, are now working to create new glitch categories Such strategies are likely to be easiest to find for highly from these None of the Above images. Fig. 2 shows the Gravity ordered datasets. For others, such as those which draw their Spy human–machine system architecture and the flow of images data from modern astronomical surveys, this may be more prob- through that system. lematic. It is also clear that any attempt to direct effort in a Gravity Spy thus uses machine learning alongside a leveling-up classification task more efficiently reduces the possibilities for infrastructure to guide the presentation of tasks to newcomers serendipitous discovery, a key motivation for human classifica- to help them more quickly learn how to do the image classi- tion in many projects. In cases where the discovery is in the fication task while still contributing to the work of the project. background, or otherwise tangential to the main task (as in the

Fig. 2. Gravity Spy system architecture and data flow through the interconnected, interdisciplinary components of the project. Each day, LIGO detects ∼2,000 perturbations with a signal-to-noise ratio greater than 7.5 and sends them to the Gravity Spy system. To maximize the gravitational wave detection rate, glitches must be identified and removed from this dataset. The Gravity Spy project couples human classification with machine-learning models in a symbiotic relationship: Volunteers provide large, labeled sets of known glitches to train machine-learning algorithms and identify new glitch categories, while machine-learning algorithms “learn” from the volunteer classifications, rapidly classify new glitch signals, and guide the training provided to the volunteers. In parallel, LIGO engineers work to identify and isolate the physical cause(s) of the identified glitches and, if possible, eliminate them. If they cannot be physically eliminated, the glitches are flagged and removed as part of the LIGO data processing pipeline. See ref. 41 for details.

1906 | www.pnas.org/cgi/doi/10.1073/pnas.1807190116 Trouille et al. Downloaded by guest on September 27, 2021 Downloaded by guest on September 27, 2021 lthcass e e.7 o eal nGaiySycutrn fot odt n e.7 o akrudo h -N apn ehdused. method mapping t-SNE the on background new for identifying 78 in two ref. volunteers al. along and support et clustering date Trouille to the to analyses shows efforts clustering The view clustering these data. This of Spy category. Spy results Gravity Above Gravity the on the with use details of that use for None tools for 71 the building in ref. 71 is fall See ref. team that classes. Spy by subjects glitch Gravity developed as The networks well dimensions. neural as possible classes, many deep glitch of using known algorithm different machine-learning the unsupervised indicate symbols an is DIRECT space. feature 3. Fig. can classification machine and effi- human called that shown of been have full combination We have a like. cient once what be might for would serendipity) signposts cybernetic (what as system serve above human–machine described efforts The Summary volunteers. by investigated the categories be own then simplify be their can form might which still will it objects system, can unusual hybrid truly a Zoo—they that such expected Galaxy In volunteers. of to presented case task the that strongly in are see cat- egories they morphological to if be several Even blurring can easy dataset. example, objects contaminated—for whole is of the number it classify small to anal- then a leveraged an just groups, of such classification separate by volunteer produced uniquely clusters are volunteers If allow ysis clusters. clus- to similar built these a being explore to tools to subject with been 3), categories. (Fig. also separate analysis have 200 tering Spy roughly Gravity in into data (77) The Near-infrared Zoo Galaxy Assembly clas- by previously Cosmic sified (CANDELS) Survey the Legacy from Extragalactic algo- can Deep clustering data classification) unsupervised sorting the completely in rithm, a included with with morphology approached (albeit as be classification well cluster- galaxy for as that but color demonstrated classification, 76 for Ref. not ing. learning machine using Clustering. in with Integration Machine evident been classifica- not date. for has to images. out which efficiency carried serendipity, at projects between in for looking tension designing clear spent and a time tion thus of is roughly amount presumably There Voorwerp), the are made Hanny’s to being and discovery proportional a Peas such Green of Zoo’s odds the Galaxy the of case -tcatcniho medn tSE apn fteGaiySytann e nteDe IciiaieEbdigfrCuTrn (DIRECT) ClusTering for Embedding DIscRiminative Deep the in set training Spy Gravity the of mapping (t-SNE) embedding neighbor t-stochastic oeta ouinexists solution potential A oaHnespoet entdta nrmna i hscase, this (in incremental that noted Super- we real the project, of with Hunters example ways the nova meaningful Through efficient and inclusivity). valued (social most broad research in the a engage empowering in to and research community efficiency) authentic (scientific possible out manner carrying of goals when experience user consider which to interventions. better designing manip- projects is clas- a in It than need experience. rather implications ulated science not with ethical do engagement most has “authentic” which the offer this images (selecting but boring images sification), inserting interesting or of artificially beautiful could frequency One assignment. the task alter intelligent of feature a ily project. depend the in therefore spent which time and extended on attitudes to scientific changing relating when goals or important with engagement especially combined spent is data—are time the tension by aims—classifying This measured site. as the engagement loss on user resulting the reduced yet variety efficiency, increase of to way project obvious Serengeti an Snapshot seems the and in identify images to animal-free classifier the remove automated an increas- Using rendered monotonous. tasks in ingly participate to as volunteers of emerges willingness which projects, such make considered. in are far interventions inherent so complex out tension carried the experiments clear simple come. the to even years many However, for human– competitive that the remain believe available, will us systems makes become machine interest data of objects training many of further scarcity as the con- to while improve expected and to be own, might tinue its learning deep on modern one of either performance to superior results produce eas xlrdtetnin htaiei evn h dual the serving in arise that tensions the explored also We necessar- not is intervention this in seen variety of loss The the reduce might assignment task complex that noted we First, PNAS | eray5 2019 5, February | o.116 vol. | o 6 no. | 1907

ECOLOGY COMPUTER SCIENCES COLLOQUIUM PAPER weekly) data release with the potential for discovery encourages classification subjects are highly linked. The example given is high levels of engagement. Yet the perceived scarcity and compe- in layered microscopy, but this solution holds potential for a tition have also resulted in a skewed volunteer base (in this case, range of projects, e.g., planet hunting where the task is change the majority of the classifications are made by a small cohort of detection. The most extreme machine/human hybrid classifier highly dedicated volunteers, the vast majority of whom are male introduced in the previous section is the use of machine clus- and 65+ y in age). tering to dramatically reduce the need for human classification; We then discussed strategies for addressing these design ten- the success of such a scheme is likely, however, to depend on the sions and thoughtfully attempting to optimize for both efficiency purity of clusters that can be achieved without significant manual and engagement, while leaving space for serendipitous discovery. intervention. The leveling-up model explored through the Gravity Spy project Despite the increasing sophistication and complexity of many provides one such opportunity. In this project, machine learn- deployed citizen science systems, therefore, it is clear that there ing guides the presentation of tasks to newcomers to quickly is much more to do in project design. Keeping open the pos- train them in the image classification task while still contribut- sibility of volunteer serendipitous discovery with the largest of ing work to the project. Volunteers are promoted from one all upcoming datasets will require the development of new and level to the next as they pass certain thresholds of classifica- flexible tools for interacting with sorting algorithms. However, tion counts and quality. In parallel, images that do not conform the performance of experiments carried out to date gives us to an existing class are passed through the levels and eventu- confidence that we can succeed, and we should expect citizen sci- ally retire as None of the Above. The volunteers, in concert entists to be experiencing cybernetic serendipity for many years with machine-learning efforts, then identify new classes through to come. visual inspection, clustering analyses, and a combination of ACKNOWLEDGMENTS. We gratefully acknowledge the tremendous efforts the two. of the Zooniverse web development team, the research teams leading each There also exist possibilities for more creative uses of machine Zooniverse project, and the worldwide community of Zooniverse volunteers learning, which become apparent once one’s design goal switches who make this all possible. This publication uses data generated via the from trying to replace humans with machines to one of build- Zooniverse.org platform, development of which is funded by generous sup- port, including a Global Impact Award from Google, and by a grant from the ing complex “social machines” that involve both sorts of worker. Alfred P. Sloan Foundation. We also acknowledge funding in part for sev- Examples of such opportunities detailed above include the abil- eral of the human–machine studies from the National Science Foundation ity to use machine effort to direct attention in datasets where Awards IIS-1619177, IIS-1619071, and IIS-1547880.

1. Pasachoff JM (1999) Halley and his maps of the Total Eclipses of 1715 and 1724. J Canada’s parks and protected areas: A climate futurescapes approach. J Environ Plann Astron Hist Heritage 2:39–54. Manag 60:1016–1035. 2. Dawson G, Lintott C, Shuttleworth S (2015) Constructing scientific communities: 21. Raddick MJ, et al. (2010) Galaxy Zoo: Exploring the motivations of citizen science Citizen science in the nineteenth and twenty-first centuries. J Vic Cult 20:246–254. volunteers. arXiv:0909.2925. Preprint, posted March 3, 2016. 3. Shuttleworth S (2018) Old weather: Citizen scientists in the 19th and 21st centuries. 22. Raddick MJ, et al. (2013) Galaxy Zoo: Motivations of citizen scientists. arXiv:1303.6886. Sci Mus Group J 3:150304. Preprint, posted March 27, 2013. 4. Bonney R, Phillips TB, Ballard HL, Enck JW (2015) Can citizen science enhance public 23. Masters K, Cox J, Simmons B, Lintott CJ (2016) Science learning via participation understanding of science? Publ Understanding Sci 25:2–16. in online citizen science. J Sci Commun. Available at https://doi.org/10.22323/2. 5. Miller-Rushing A, Primack R, Bonney R (2012) The history of public participation in 15030207. Accessed January 1, 2017. ecological research. Front Ecol Environ 10:285–290. 24. Greenhill A, et al. (2016) Playing with science. Aslib J Inf Manage 68:306–325. 6. Sullivan BL, et al. (2009) eBird: A citizen-based bird observation network in the 25. Crall AW, et al. (2012) The impacts of an invasive species citizen science training pro- biological sciences. Biol Conserv 142:2282–2292. gram on participant attitudes, behavior, and science literacy. Publ Understand Sci 7. Wang Y, Kaplan N, Newman G, Scarpino R (2015) CitSci.org: A new model for 22:745–764. managing, documenting, and sharing citizen science data. PLoS Biol 13:e1002280. 26. Cronje R, Rohlinger S, Crall A, Newman G (2011) Does participation in citizen science 8. Boxer EE, Garneau NL (2015) Rare haplotypes of the gene TAS2r38 confer bitter taste improve scientific literacy? A study to compare assessment methods. Appl Environ sensitivity in humans. SpringerPlus 4:505. Educ Commun 10:135–145. 9. Caruso SM, Sandoz J, Kelsey J (2009) Non-STEM undergraduates become enthusiastic 27. Jones G, Childers G, Andre T, Corin E, Hite R (2018) Citizen scientists and non-citizen phage-hunters. CBE Life Sci Educ 8:278–282. scientist hobbyists: Motivation, benefits, and influences. Int J Sci Educ 8:287–306. 10. Caruso JP, Israel N, Rowland K, Lovelace MJ, Saunders MJ (2016) Citizen science: The 28. Trumbull DJ, Bonney R, Bascom D, Cabral A (2000) Thinking scientifically during small world initiative improved lecture grades and California critical thinking skills participation in a citizen-science project. Sci Educ 84:265–275. test scores of nonscience major students at Florida Atlantic University. J Microbiol 29. Brossard D, Lewenstein B, Bonney R (2005) Scientific knowledge and attitude change: Biol Educ 17:156–162. The impact of a citizen science project. Int J Sci Educ 27:1099–1121. 11. Shaffer CD, et al. (2014) A course-based research experience: How benefits 30. Ruiz-Mallen´ I, et al. (2016) Citizen science. Sci Commun 38:523–534. change with increased investment in instructional time. CBE Life Sci Educ 13:111– 31. Lintott CJ, et al. (2008) Galaxy Zoo: Morphologies derived from visual inspection of 130. galaxies from the Sloan digital sky survey. Mon Not R Astron Soc 389:1179–1189. 12. Davis E, et al. (2017) Antibiotic discovery throughout the small world initiative: A 32. Fortson L, et al. (2012) Galaxy Zoo: Morphological classification and citizen sci- molecular strategy to identify biosynthetic gene clusters involved in antagonistic ence. Advances in Machine Learning and Data Mining for Astronomy, eds Way MJ, activity. MicrobiologyOpen 6:e00435. Scargle JD, Ali KM, Srivastava AN (CRC Press, Taylor & Francis Group, Boca Raton, FL), 13. Lee J, et al. (2014) RNA design rules from a massive open laboratory. Proc Natl Acad pp 213–236. Sci USA 111:2122–2127. 33. Swanson A, Kosmala M, Lintott C, Packer C (2016) A generalized approach for pro- 14. Cooper S, et al. (2010) Predicting protein structures with a multiplayer online game. ducing, quantifying, and validating citizen science data from wildlife images. Conserv Nature 466:756–760. Biol 30:520–531. 15. Kim JS, et al. (2014) Space–time wiring specificity supports direction selectivity in the 34. Matsunaga A, Mast A, Fortes JAB (2016) Workforce-efficient consensus in crowd- retina. Nature 509:331–336. sourced transcription of biocollections information. Future Gener Comput Syst 16. Daniels C, Holtze TL, Howard R, Kuehn R (2014) Community as resource: Crowd- 56:526–536. sourcing transcription of an historic newspaper. J Electron Resour Librarian 35. Arteta A, Lempitsky V, Zisserman A (2016) Counting in the wild. Computer Vision– 26:36–48. ECCV 2016, eds Leibe B, Matas J, Sebe N, Welling M (Springer International 17. Causer T, Grint K, Sichani A-M, Terras M (2018) ‘Making such bargain’: Transcribe Ben- Publishing, Amsterdam), pp 483–498. tham and the quality and cost-effectiveness of crowdsourced transcription. Digital 36. Anderson TM, et al. (2016) The spatial distribution of african savannah herbivores: Scholarship Humanit 33:467–487. Species associations and habitat occupancy in a landscape context. Philos Trans R Soc 18. Dicksinson J, Bonney R (2012) Citizen Science: Public Participation in Environmen- B Biol Sci 371:20150314. tal Research (Comstock Pub Assoc, Ithaca, NY). Available at https://www.amazon. 37. Williams AC, et al. (2014) A computational pipeline for crowdsourced transcriptions com/Citizen-Science-Participation-Environmental-Research-ebook/dp/B007NK5I16? of ancient greek papyrus fragments. 2014 IEEE International Conference on Big SubscriptionId=0JYN1NVW651KCA56C102&tag=techkie-20&linkCode=xm2&camp= Data (Big Data), eds Samad T, Arnold G, Goldgof D, Hossain E, Lanzerotti M (IEEE, 2025&creative=165953&creativeASIN=B007NK5I16. Accessed July 1, 2018. Washington, DC). 19. Newman G, et al. (2012) The future of citizen science: Emerging technologies and 38. Grayson R (2016) A life in the trenches? The use of operation war diary and crowd- shifting paradigms. Front Ecol Environ 10:298–304. sourcing methods to provide an understanding of the british army’s day-to-day life 20. Groulx M, Lemieux CJ, Lewis JL, Brown S (2016) Understanding consumer behaviour on the western front. Br J Mil Hist. Available at https://bjmh.org.uk/index.php/bjmh/ and adaptation planning responses to climate-driven environmental change in article/view/96. Accessed February 1, 2018.

1908 | www.pnas.org/cgi/doi/10.1073/pnas.1807190116 Trouille et al. Downloaded by guest on September 27, 2021 Downloaded by guest on September 27, 2021 2 ennC,e l 21)Ccoecne:Cnctznsinit mrv rpclcyclone tropical improve scientists citizen Can characteriza- center: Cyclone detector (2015) ligo al. et advanced CC, Hennon integrating Spy: 42. Gravity (2017) al. et M, Zevin 41. publication. ATLAS for scale project large science citizen for A public HiggsHunters: general (2017) the A Hass Crowdsourcing (2015) 40. al. et FJ, Reis dos Candido 39. 4 lfo ono 21)PA tla lse uvy I nrmd rjc cluster project Andromeda II. survey. cluster stellar PHAT (2015) L Johnson Clifton science 44. citizen of validation Quantitative Forests: Floating (2018) al. et IS, Rosenthal 43. 0 uskvk ,L -,FiFiL(05 eto ohwrd:Hmnmciecollabora- Human-machine worlds: both of Best (2015) L Fei-Fei L-J, Li O, Russakovsky 60. tasks. hierarchical crowdsourcing for Planning (2015) E collec- Horvitz for E, metric Kamar information-theoretic 59. An bit: the by Pay (2013) of TP combination galaxy Bayesian Waterhouse Dynamic in 58. (2013) intelligence A Smith machine I, and Psorakis S, human Roberts Integrating E, Simpson (2018) 57. al. in et features MR, are Beck transferable 56. How (2014) H Lipson machine and Y, human Bengio combined J, using Clune search J, transient Yosinski A 55. (2017) al. et DE, supernovae. zoo Wright Galaxy (2011) 54. al. et AM, Smith 53. (2008) al. et echo? Z, light quasar Ivezic a voorwerp’, 52. ‘Hanny’s zoo: Galaxy (2009) al. et CJ, Lintot scientific volunteer-led 51. of study compact A of chance: a class scientists a citizen of Giving Discovery (2016) peas: M Straub green zoo 50. Galaxy (2009) al. et C, Cardamone 49. hunters Planet (2014) CB Jackson K, Crowston KD, Hassman C, Østerlund G, Mugar 48. of discovery the Crowdsourcing of outside I. Luczak-R araneiforms of Terrains–Discovery 47. - Four: Planet WARPS (2018) al. SPACE et ME, (2016) Schwamb 46. al. et PJ, Marshall 45. rulee al. et Trouille nest records? intensity (IOP science. C citizen Tull and R, learning, machine Mount 1.0. tion, p eds 898, Physics, Vol Francisco), Nuclear San and Series, Conference Energy Physics: High of Journal in Computing on ence cancer. in studies pathology molecular catalog. posted Preprint, arXiv:1801.08522. classifications. 2018. 25, consensus January from generated data dPctM(EE otn,p 2121–2131. pp Boston), (IEEE, M Pecht ed Recognition, Pattern for and Foundation annotation. object (International for P tion Yolum 1191–1199. pp G, SC), Weiss Richland, Systems, Multiagent E, Multiagent and and Elkind Agents Agents R, Autonomous Autonomous Bordini on Conference eds International Systems, 2015 the of ings Work Cooperative judgment. human tive 1–35. pp Berlin), (Springer, D Wolpert classifiers. imperfect multiple tasks. 3320–3328. classification pp morphology 2, Vol C, MA), Cortes Cambridge, Press, M, (MIT Welling KQ Z, Weinberger Ghahramani ND, eds Lawrence NIPS’14, Systems, Processing Information ral networks? neural deep classifications. Preprint, e-prints:0805.2366. arXiv Products. 2008. Data 15, May Anticipated posted and Design erence Soc Astron R 2018. 1, June discovery. W galaxies. Lutters star-forming S, extremely Fussell eds ’14, 109–119. pp Computing–CSCW York) New Social Press, & (ACM Work Cooperative ported 315–324. explorers. pp seafloor CA), and Alto, Palo Press, (AAAI A Goel ed Media, Social and science. citizen online in dynamics deposits. layered polar south the lenses. gravitational srpy J Astrophys shM ta.(04 h o’ lestl ou?Cnetadcommunity and Content us? to talk aliens won’t Why (2014) al. et M, osch ¨ iie c ho Pract Theor Sci Citizen 399:129–140. o o srnSoc Astron R Not Mon ulAe eerlSoc Meteorol Amer Bull d adsE comc AM e ok,p 623–638. pp York), New (ACM, H McCormick E, Tardos eds , 802:127. MNRAS o h STClaoain ST rmSineDiest Ref- to Drivers Science From LSST: Collaboration. LSST the For rceig fte21 ofrneo optrSupported Computer on Conference 2013 the of Proceedings rceig fte1t C ofrneo optrSup- Computer on Conference ACM 17th the of Proceedings rceig fte2t nentoa ofrneo Neu- on Conference International 27th the of Proceedings rceig fteIE ofrneo optrVision Computer on Conference IEEE the of Proceedings 455:1171–1190. d u ,KryM, Karny T, Guy eds Imperfection, and Making Decision vial tht:/o.r/053/sp4.Accessed http://doi.org/10.5334/cstp.40. at Available . o o srnSoc Astron R Not Mon MNRAS Icarus ihhItrainlAA ofrneo Weblogs on Conference AAAI International Eighth 472:1315–1323. EBioMedicine 308:148–187. 399:1191–1205. 96:591–607. lsia unGravity Quan Classical o o srnSoc Astron R Not Mon 2:681–689. 476:5516–5534. 34:064003. 412:1309–1319. o Not Mon Proceed- Confer- 8 a e atnLP itnG 20)Vsaiigdt sn t-SNE. using classifica- data Visualizing (2008) morphological GE Hinton LJP, visual Maaten der van Quantitative 78. Zoo: galaxy Galaxy of (2016) taxonomy automatic BD An Simmons (2017) 77. N Davey Yi, Sun JE, Geach A, Hocking 76. coor- support to Folksonomies (2018) M Harandi log C, Østerlund electronic K, Crowston LIGO C, Jackson Advanced doves. 75. Paired class: glitch LIGO New Advanced (2016) 9. A October Lundgren on 74. PCAL-Y misbehaving from Glitches (2015) J Smith 73. 2 e K rwtnK aad ,ØtrudC ilrG(08 peln odifferent to Appealing (2018) G Miller C, Østerlund M, Harandi K, Crowston TK, of Lee clustering for 72. embedding discriminative Deep DIRECT: (2018) al. et S, Bahaadini work: 71. crowd advanced Coordinating (2018) C Østerlund E, Mitchell K, learning Crowston human 70. and machine Blending (2017) TK Lee C, Østerlund the and K, machines Crowston Humans, 69. Spy: Gravity (2017) Team Spy Gravity science. The citizen K; online Crowston across 68. behavior volunteer of Patterns (2018) self- al. Gender, et science: H, Spiers doing 67. and Using (2014) CE George-Jackson agency. MM, human Williams in 66. mechanism Self-efficacy (1982) A Bandura 65. in participation multimodal Understanding play: Connecting (2012) for D case Fields The Y, learning: Kafai and interaction Tangible 64. (2011) MU Bers RJ, Crouser MS, Horn 63. intentionally image This (2015) G Miller A, Swanson C, Lintott V, Maidel A, Bowyer 62. intelligence machine and human Combining (2012) E Horvitz S, Hacker E, Kamar 61. in o 800glxe rmCANDELS. 9:2579–2605. from galaxies 000 48 4447. for tions learning. machine 1108–1129. unsupervised using morphology 647–678. pp 27, Vol York), New (Springer, K Schmidt ed folksonomies. of coordination and dination 2018. 15, June https://alog.ligo-wa.caltech.edu/aLOG/index.php?callRep=27138. Accessed at available 27138 https://alog.ligo-la.caltech.edu/aLOG/index.php? at 2018. 15, June available Accessed callRep=21463. 21463 log electronic 2018. 1, June Accessed oiain namsaet eri iie cetss eut fafil experiment. field a https://jcom.sissa.it/archive/17/01/JCOM of at Results Available scientists: Commun. citizen recruit Sci to J message a in motivations 748–752. pp York), data. Sciences, LIGO System on Conference International (Hawaii T 1681–1690. pp Bui Honolulu), ed Sciences, System on science. citizen Extending pp Honolulu), Sciences, System on Conference International (Hawaii 65–73. T Bui ed ences, 163–166. pp York), processes. New (CSCW, S Poltrock C, Lee eds Computing, Social and science. citizen of future 93–94. pp Geneva), Committee, Steering Conferences 2018 Web Conference Web Wide The of Proceedings Companion ’18 WWW stem. in Eng students Sci undergraduate of identity science and efficacy, 265–272. pp York), New Press, 147. (ACM al. et L, Morency eds ’12, ICMI - interaction worlds. virtual approach. hybrid a Alto, Palo (AAAI, P Ipeirotis E, 209–210. Gerber pp eds CA), Computation, Human on participation. Conference science 2015 citizen increase images Mundane blank: 1, left Vol SC), (International Richland, M Systems, Luck Multiagent C, and 467–474. Sierra pp Agents eds Autonomous Systems, for Multiagent Foundation and Agents Autonomous crowdsourcing. large-scale in 20:99–126. rceig fte5t aaiItrainlCneec nSse Sci- System on Conference International Hawaii 50th the of Proceedings dPctM(EE New (IEEE, M Pecht ed Processing, Image on Conference International IEEE rceig fte1t C nentoa ofrneo Multimodal on conference international ACM 14th the of Proceedings esUiutu Comput Ubiquitous Pers PNAS C ofrneo optrSpotdCoeaieWork Cooperative Supported Computer on Conference ACM rceig fte5s aaiItrainlConference International Hawaii 51st the of Proceedings | rceig fte1t nentoa ofrneon Conference International 11th the of Proceedings eray5 2019 5, February optrSpotdCoeaieWork Cooperative Supported Computer 16:379–389. o o srnSoc Astron R Not Mon | o o srnSoc Astron R Not Mon o.116 vol. ItrainlWorld (International mPsychol Am oe Minorities Women J rceig fthe of Proceedings | ahLanRes Learn Mach J 1701 o 6 no. 2018 464:4420– | 37:122– 1909 A02. 473: ,

ECOLOGY COMPUTER SCIENCES COLLOQUIUM PAPER