Cerberus: The Mars Crowdsourcing Experiment Innovations J. S. S. van ‘t Woud J. A. C. Sandberg B. J. Wielinga BlackShore – creative Informatics Institute, FNWI Informatics Institute, FNWI ESA business incubation programme University of Amsterdam University of Amsterdam Noordwijk, Netherlands Amsterdam, Netherlands Amsterdam, Netherlands E-mail: [email protected] [email protected] Keywords: Mars, Crowdsourcing, Serious Gaming, Citizen Scientist Summary This article discusses the use of crowdsourcing in a serious game. A computer game, called Cerberus, which allows play- ers to tag surface features on Mars, has been developed. Developing the game has allowed us to investigate the effects of different help levels in supporting the transfer of knowledge, and also how changing the game features can affect the quality of the gaming experience. The performance of the players is measured in terms of precision and motivation. Pre- cision reflects the quality of the work done and motivation is represented by the amount of work done by the players. Games with an explicit help function combined with a “rich gaming experience” resulted in significantly more motivation among the players than games with an implicit help function combined with a “poor gaming experience”. There was no significant difference in the precision achieved under different game conditions, but it was high enough to generate Martian maps exposing aeolian processes, surface layering, river meanders and other concepts. The players were able to assimilate deeper concepts about Martian geology, and the data from the games were of such high quality that they could be used to support scientific research. Introduction Gaming for science Players are encouraged to play the game by Because crowdsourcing demands a cer- earning points and the chance to earn “sci- Crowdsourcing science uses many individ- tain effort from its contributors, the play- entific glory” (Viñas, 2008; The Economist, uals (the crowd) to process scientific data ers must be motivated to participate. Two 2008). Another example is Galaxy Zoo, an and is mainly used with datasets where hu- methods are utilised: 1. Small financial initiative in which galaxies and their behav- man perception exceeds the capabilities of rewards are offered for processing data iour are classified by users (Darg et al., computers. Humans are still often better units. 2. The computer game environment 2009). Astronomers can only cover a small and faster than automatic devices at rec- is made sufficiently entertaining that play- portion of the amount of data that needs ognising shapes and objects (Hoffmann, ers will process data for free. to be analysed, so the data is prioritised, 2009). This project investigates whether and new discoveries may remain hidden in the crowd can recognise and apply high The concept of crowdsourcing has proven the lower priority data. This is where Galaxy level semantic concepts to features in itself in serving science. The quality of the Zoo comes in. The players analyse the pho- photos of the Martian surface and thus data analysis performed by crowds within tographs collectively, so that each time a support scientific research. In this research certain fields of research is superior to the photograph is analysed the reliability of the project, crowdsourcing was conducted us- results obtained by individuals and even classifications in the galaxy database in- ing a serious game. Different types of help sometimes to those of the experts involved creases. The results prove to be just as ac- function were investigated to establish (Hoffmann, 2009). curate as if the analysis had been carried the level required to provide players with out by expert astronomers. By the end of enough knowledge for them to identify sur- An example of a serious crowdsourcing the year 2009 over 220 000 people had par- face features on Mars. Another research game used for science is Foldit, a game ticipated in this project and they had con- goal was to investigate which game fea- in which players create new protein chains. tributed to the discovery of a new type of tures are needed to motivate players. The goal is to contribute to cancer research. object (Charles, 2009). 28 CAPjournal,CAPjournal, No.No. 12,12, AprilMay 2012 CAPjournal, No. 12, May 2012 Exploring Mars Knowledge is added. These levels are derived from Since November 2006 NASA’s Mars Panofsky’s (1962) and Shatford’s (1986) Reconnaissance Orbiter (MRO) has used t models, but each differentiates between in- Type the High Resolution Imaging Science 1. Technique dividual objects and the scene as a whole. 2. Global Distribution Experiment (HiRISE) to acquire data about Syntax/Percep In order to describe these six conceptual the surface of Mars. The MRO transmits 3. Local Structure levels, from general to specific, an increas- 4. Global Composition colour imaging data back to Earth, cover- 5. Generic Objects ing level of knowledge about the subject is ing objects with sizes down to 25 centi- required (Hollink, 2004). The annotations oncept6. Generic Scene C metres. This high level of detail generates 7. Specific Objects for BaM are confined to the perceptual lev- a vast amount of data that needs to be an- 8. Specific Scene els (levels 1 to 4, Figure 1) with the occa- mantics/Visual9. Abstract Objects e alysed (McEwen, 2010). The first research S sional cautious foray into the first concep- 10. Abstract Scene phase, or Primary Science Phase (PSP), tual level (NASA & Microsoft, 2009), as, for ran until December 2008 and has photo- Figure 1. Ten-level model (Jaimes, 2000). example, a crater would be categorised as graphed approximately 0.55% of the plan- a generic object within Jaimes’s model. et’s surface, consisting of 8 terapixel of data (McEwen et al., 2009). The scientific Martian map. As a reward the user earns However, when we look at the HiRISE re- research covers 18 themes, such as differ- experience points and rewards in the form search themes, annotations that are con- ent types of erosion, with each theme pro- of medals, badges and “inside” NASA in- fined to the perceptual levels are no longer cessing specific surface features to learn formation (NASA & Microsoft, 2009). sufficient. The themes, for example, de- more about Mars. scribe specific geological processes and Knowledge levels therefore demand a deeper level of knowl- The non-academic world can participate Jaimes (2000) presented a model which edge to be able to recognise and describe in HiRISE via an Education and Public describes images in terms of ten different them (McEwen, et al., 2009). In terms of Outreach (EPO) initiative (McEwen et al., levels of perception and semantics. The Jaimes’s model (2000) the HiRISE research 2009). The Clickworkers project is an ex- model is shown schematically in Figure 1, themes would position themselves within ample. This was a “citizen science” initia- with the first four perceptual levels at the the specific and abstract conceptual levels tive whereby internet users had to classify top of the pyramid. The topmost level de- (levels 7–10 in Figure 1). The specific levels and annotate geological objects such as scribes the image in terms of its most basic could describe processes like wind ero- dunes and craters to generate a searchable properties, such as the JPEG 2000 file for- sion, while the abstract levels could con- database (McEwen et al., 2009). NASA’s mat of the HiRISE images (HiRISE, 2009). cern hidden craters buried beneath the researchers assumed that the average The next three perceptual levels go deeper planet surface, which cannot be seen di- person had enough commonsense knowl- into the superficial features of the image rectly with the eye (HiRISE, 2009). edge to accurately screen photographs for and describe colour, shape and texture, craters. Starting in November 2000, over distinguishing between characteristics of The current generation of Mars games cre- 80 000 people measured two million cra- the image as a whole, its distinguishable ated by NASA and Microsoft (2009) only ters and classified the age of 300 000 cra- elements and the composition of these el- extends to the generic levels (levels 5 and 6 ters in a year (Szpir, 2002). ements (Hollink, 2004; Jaimes, 2000). The in Figure 1), which do not go deep enough general image characteristics described to make annotations within the HiRISE Become a Martian by these perceptual levels show a strong (2009) research themes. But can users In November 2009, the website, Become a resemblance to the way in which the types reach a deeper semantic level without pos- Martian (BaM), was launched on Microsoft’s of feature that have to be picked out and sessing expert knowledge? (Darg, 2009; Developers Conference as a cooperation annotated in Galaxy Zoo and Be a Martian Bulletin of the Atomic Scientists, 2001) between Microsoft and NASA. Two games are categorised. Whether crater counting are offered through this interface and or classifying galaxies the user has to rec- which allow users to simultaneously learn ognise circles or varieties of them. The user The problem about Mars and contribute to planetary also has to align patterns, and both these exploration. Microsoft and NASA empha- tasks can be carried out without special- The investigations into whether crowd- sise that BaM does not only inform people ist knowledge (Charles, 2009; NASA & sourcing using a game can be used to about Mars and NASA’s activities, but that Microsoft, 2009). make annotations within the specific levels it enables large groups of people to ana- (levels 7 and 8 in Figure 1) or even within the lyse data in a field where computers still lag The six “lower” semantic (or conceptual) still higher abstract levels ( levels 9 and 10 behind (Brown, 2009; Microsoft, 2009). An levels are divided into generic, specific and in Figure 1), and using the HiRISE research introductory movie is shown in the Martian abstract levels, each of which is further themes, have covered three aspects.