What Makes Citizen Science Projects Successful, and What Can We Learn from Them for Future Projects?

Technion – Israel Institute of Technology

What makes Citizen Science projects successful, and what can we learn from them for future projects?

Literature review of Citizen Science projects

Yaela Naomi Golumbic Academic advisor: Boris Koichu Technion Citizen Science Project (TCSP) 2015 Table of Content Executive summary ...... 2 Definitions ...... 9 Classifications of Citizen Science projects ...... 12 Principals for choosing and assessing cases ...... 15 Case studies ...... 17 CoCoRaHS- Community Collaborative Rain, Hail and Snow Network ...... 17 eBird ...... 22 Foldit ...... 27 Galaxy Zoo ...... 31 OPAL- The Open Air Laboratories ...... 34 PatientsLikeMe ...... 40 What makes projects successful? ...... 45 Appendix ...... 48 References ...... 53

Executive summary

This report provides a systematic review and meta-analysis for Citizen Science projects, submitted to the Technion Citizen Science Project (TCSP). The aim of this report is to better understand what makes Citizen Science projects successful, and to highlight what can be learned and implemented in future practice, from existing successful projects.

For this report, we defined principals for choosing projects for case study assessment, according to the definitions of the TCSP, which include scientific excellence, innovation and citizen scientists' participation. In the attempt to choose well known and acknowledged Citizen Science projects, indicating successful management and project planning, we reviewed six established Citizen Science reviews (Bonney et al. 2009, European commission report 2013, Silvertown 2009, Dickinson 2010, Wiggins and Crowston 2011, Franzonia and Sauermann 2014), listing Citizen Science projects. A total of 139 projects were collectively mentioned in these reviews (see Appendix A). After filtering the list of projects by being cited in at least three of the six reviews, having over five unique scientific publications and matching similar topics, six final project were chosen for further assessment: CoCoRaHS, eBird, Fold It, Galaxy Zoo, OPAL and Patients-like-me.

The chosen projects were reviewed and analyzed in order to shape the understanding of successful Citizen Science projects that have the capability to serve as models for future project planning. A summary of the projects main features, and the main element the can be learned from each project is hereby presented.

CoCoRaHS Summary

CoCoRaHS (Community Collaborative Rain, Hail and Snow Network) is a community-based network of volunteers working together to measure and map precipitation (rain, hail and snow) across North America. CoCoRaHS originated at the Colorado Climate Center at Colorado State University (CSU) in the spring of 1998 and is now the largest provider of daily precipitation observations in the United States and most of North America.

CoCoRaHS primary goal is to provide accurate high-quality precipitation data for natural resource, education and research applications.

Data collection is done by participants throughout North America who record daily precipitation observation using a standard rain gauge. Volunteers can also submit reports of intense rain or hail (estimate of the total rainfall over a limited time and flooding indications), or quick reports that includes starting time, approximate hailstone size and texture and any evidence of damage. Data is submitted using an online form or by phone.

Accessibility- The network has an active website, blog, Facebook page and twitter feed with reports about project progress, developments and recent findings. All meteorological information is available online and is presented as a daily precipitation map or as summarized in tables, bar charts, and graphs. Among the many end users of CoCoRaHS data, are the National Weather Service, meteorologists, hydrologists, emergency managers and more.

What can TCSP learn from CoCoRaHS?

CoCoRaHS defines itself as a community-based network. As such, all information collected by volunteers is clearly presented in the website and is disseminated for the use of many organizations and private users free of charge.

Since the task of measuring precipitation includes several different activities, CoCoRaHS organizes face-to-face work-shops and training session that help participants learn how to install and use equipment and how to accurately read measurements. Training information is also available on the CoCoRaHS home page with specialized slides, videos and manuals. eBird summary eBird is an online tool for collecting and distributing basic information on bird abundance and distribution in diverse spatial and temporal measures. It was launched in 2002 by the Cornell Lab of Ornithology and National Audubon Society, and provides a simple and intuitive web-based interface for submitting and viewing bird observations. eBird is to date, one of the largest biodiversity data resources, with over one-million bird observations reported every month from tens of thousands of participants, and with more than 100,000 people entering data every year. eBird’s goal is to is to maximize the use and accessibility of the many bird observations made each year by recreational and professional bird watchers.

Data collection is done by participants from a broad range of professional backgrounds, including both amateur and expert bird observers. All data is submitted using an online checklist program, participants simply enter when, where, and how they went birding, then fills out a checklist of all the birds seen and heard during that time. Data are validated with an automated data quality filters, developed by regional bird experts, before they enter the database, local experts review unusual records that are flagged by the filters. Data are then classified and grouped according to species, location and time of observation.

Accessibility- The project has an active website, Facebook page and twitter feed with recent report about project progress, developments and interesting findings and photos. All birding information is available on the project website and application and can be accessed in English, Spanish, French, Portuguese and Chinese. Data can be presented as tables, bar charts, line graphs or presented on a map and are also available for download fur further use and analysis.

What can TCSP learn from eBird? eBird started out from the assumption, that there is an unconsolidated amateur birdwatcher community, and focused their work on providing services that would appeal to this community and consolidate it. The tool built by eBird enables both the formulation of a birding community and contributes to in-depth scientific research and knowledge construction. Taking this approach of what the public wants and needs, is the main source

3 for eBirds' success, according to eBird initiators, and resulted in extensive growth of eBird project, in terms of the number of participants and the amount of data submitted.

An additional feature that is important for a project of this magnitude is the assistance of volunteers who serve as reviewers of the submitted data and support the data quality verification. The project initiators and directors cannot review all the amounts of data that flows through the system every day and hence the assistance of these volunteers is essential.

Foldit Summary

Foldit is a multiplayer online game in which players compete and collaborate to find well folded protein structures. Puzzles that are presented to players are computational unsolvable, highlighting the players' contribution to science.

The game was initially launched in May 2008 by a group of scientists from the University of Washington, and by September of that year it had engaged 50,000 users. One year after launch there were about 200,000 active Foldit players and activity remains steady with about 2,000 active players who play more than once a week.

Foldits' primary goal is to produce accurate protein structure models through gameplay.

Data collection is done by recording gameplay data from participants. Data collected includes biochemical structures, scores, algorithms, tool and algorithm usage, progress, and time played. Structures built and modified by players are reviewed by scientists who analyze the structures' likelihood to fold in space and track recipe (folding strategies) usage and development employed by Foldit players.

Accessibility- Foldit has a website with project news, information about puzzles, a blog which describes some of the outcomes and results of the folding work and a Wiki page with general information for beginner and advanced players. Scientific discoveries and publications are also made available on the project website.

The game is downloaded by the players to their private computer and is open for all users around the world, including players with no scientific training and no previous exposure to molecular biology.

What can TCSP learn from Foldit?

Foldit has an initial goal of finding accurate protein structures for unknown problems. However, Foldit does not limit the game to the few scientific puzzles at stake, and opens the option for participants to create new puzzle in the form of contests, define and describe the type of puzzle and choose activity dates. Although these contest are not used as part of the scientific research as they are not logged, scored or looked at by the scientists, it is an important feature of the game with much activity and participation.

Foldit also has a number of community building features including chat rooms, opportunities to share folding strategies and a wiki page written partly by experience Foldit players. They give an important role to the personal connection with players maintaining a blog where projects progression and outcome are presented and offering scheduled chats with scientist

4 and developers where participants can communicate directly with the people working on the project and get a view of what is happening behind the scenes of Foldit.

Galaxy Zoo Summary

Galaxy Zoo is an online platform for classifying galaxies from the Sloan Digital Sky Survey (SDSS). Participants visually inspect and classify pictures of galaxies, on the project site, assisting astronomers in studying our universe.

The site was first launched at June 2007 and within 24 hours of launch received almost 70,000 classifications an hour. During the sites first year, more than 50 million classifications were received, contributed by more than 150,000 people. Since, Galaxy Zoo has launched a number of classification versions (the fourth launched at 2014), each asking volunteers to classify galaxies in slightly different ways.

Galaxy Zoo goal is to classify galaxies by having volunteers look telescopic pictures and sort them by shape. Galaxy Zoo hopes to assist astronomers in studying the way galaxies were formed and to better understand our universe.

Data collection is done by participants visualizing an image presented on the website and answering a series of simple questions as to the figure shape.

Accessibility- submitting information is very easy in Galaxy Zoo. Galaxy zoo also has a platform for discussion in which participants can have conversations and ask questions. In addition there is a blog introducing new developments, research results and news.

Results and final products of the project are published on the website and blog. Information of finished and published classifications is available for download of raw data, however data in-analysis process is not presented.

What can TCSP learn from Galaxy Zoo?

40% of participants in a motivation survey indicated their top motivation for participating in Galaxy Zoo is contribution to scientific research. Other dominant responses included love for astronomy and seeing new discoveries. These indicate that Galaxy Zoo users are highly scientific oriented and care about scientific findings.

The Galaxy Zoo platform is very simple, submitting information is very easy and many classifications can be added in just a few minutes.

OPAL Summary

OPAL (open air laboratories) is a Citizen Science initiative active across the UK, which motivates the public to get closer to their local environment while collecting scientific data. Lunched at 2007 and led by the Imperial College London, OPAL includes leading museums, universities and environmental organizations across the UK (OPAL, 2015).

With over half a million people who have actively participated in the OPAL programs, OPAL promotes active participation and involvement with nature, and encourages participants to

5 take the next step and record their observations, develop ecological knowledge and apply it. This is done with OPAL national surveys that combine observations of wildlife with data on air, soil and water condition.

OPALs' primary goal is to carry out high quality environmental research with maximum public engagement and to promote environmental local knowledge in the community.

Data collection is done by conducting national surveys across a broad range of environmental disciplines (soil, water, climate change, air and biodiversity) with self- explanatory field guides designed to be and suitable for a wide age range and background. Participants answer the questions in the survey, identify species and note characters, in their chosen researched location. Data is submitted using an online form or via free post.

Accessibility- The OPAL website has very detailed information, and include pictures, short videos and tips for things to look out for while monitoring. The website also has a scientist blog which provides News, updates and opinions from community scientists and a newsletter is distributed quarterly via email. Scientific results and conclusions are presented online after the termination of a survey, and real time data is available graphically using results-maps for each of the surveys examined.

What can TCSP learn from OPAL?

OPAL has invested a lot of time and effort in designing resources and tasks that are suitable for everyone. The information available and the accessible and clear format of the work guides make it a fun and educational experience that people want to be a part of. Clear instructions are provided for every step of the surveys, including introduction of the topic, explanatory pictures and diagrams and clear multi-choice questions.

The regional community-based science teams, helps promote community building and development of deprived areas working directly with local people motivating them to get involved in OPAL activities. These regional communities also have regional meetings, workshops and open days, enhancing public participation.

PatientsLikeMe Summary

PatientsLikeMe is a health data-sharing platform, a tool for patients, researchers, and caregivers that helps users make treatment decisions, manage symptoms, and improve health outcomes. PatientsLikeMe has a vision to transform the way patients manage their own conditions, change the way industry conducts research and improve patient care.

PatientsLikeMe was founded in 2004 by three MIT engineers, it was opened publicly at 2006 and a year and a half later, the community contained 1570 verified patients and currently (June 2015) has 325,000 members across over 2,400 medical conditions.

PatientsLikeMe primary goal is to provide a platform for patients who want to share their health information to create collective knowledge about disease, health, and treatments.

Data collection is done by patient members who share their personal information through the website platform. Information includes demographic information, longitudinal treatment, symptoms, outcome data, and treatment evaluations.

Accessibility- The project has a very detailed website featuring the shared information about members conditions, treatments and symptoms. Public profiles show patient username and profile photo, charts featuring shared information about patient condition such as outcome scores, lab values, treatments, treatment evaluations, symptoms and weight. Data is also presented as averages by conditions, treatment and symptoms. Personal data is presented back to members as individual-level graphical health profiles and aggregated into reports accessible on the site.

What can TCSP learn from PatientsLikeMe?

The platform was built thinking of the patients and their uses. The site offers a variety of tools to help patients record the conditions and treatments, and to promote accessibility of information. The resources on the site are designed to help members answer the question: “Given my status, what is the best outcome I can hope to achieve, and how do I get there?

Members can discuss data sets with other members either within a group forum or individually through private messages.

PatientsLikeMe has an Open Research Exchange platform for creating health outcome measurements (www.openresearchexchange.com). The platform puts patients at the center of the clinical research process. It helps medical researchers pilot, deploy, share, and validate new ways to measure diseases within PatientsLikeMe’s community.

Table 1. Summary of number of participants and Initial recruitment methods in each project.

Project name Number of participants Initial recruitment methods CoCoRaHS >10,000 submitting data Schools. Local TV, radio, newspaper. Face daily to face recruitment. eBird >100,000 submitting data Birding community. annually >1 M visiting annually Foldit >200,000 players annually Rosetta@home community >2,000 players weekly Galaxy Zoo >180,000 for Galaxy Zoo1 BBC website >50 M classifications OPAL >500,000 volunteers in 6 Schools. Local community work. years Patients 325,000 members Paid ads, internet search, PR, press, LikeMe provider referrals, word of mouth.

After thoroughly reviewing all six projects briefly describes above, common characteristics were identified in order to better understand what makes these project successful. The most

7 common features found, other than a strong scientific goal, were related to the platform and participant interface built for the project. These features include data submission interface, social platforms and extra material available onsite.

All projects have a simple and clear platform for collecting data, whether the data is physically submitted by participants or derived from online platforms. In addition, five of the six projects promote social interactions between participant by providing a social platform such as forums and chat rooms. These social platforms serve as community building tools, enable participant to share their findings, collaborate, have discussion and ask each other questions. In many cases these platforms have been shown to enhance motivation and enjoyment. All projects provide much information regarding the topic of the study in there web platform, including learning material, educational plans and study results. Of the information provided in the web sites, all six projects present recent result written in clear and accessible fashion in the form of blogs, newsletters or reports.

From our analysis it seems that an extra emphasis is put, in all successful projects reviewed, on the user experience and in creating motivation for participant to continue to participate and contribute to the research. This is done by creating user friendly platforms for submitting data or for game play, providing information about the study progress and results and by creating social platforms for additional involvement and interest.

Definitions

Citizen Science

Citizen Science is the involvement of volunteers in science. Beyond this very broad description, there is no one absolute definition of Citizen Science today and it refers to a diverse range of projects with different scientific, educational and engagement goals.

Alan Irwin, in the preface to his book "Citizen Science": A Study of People, Expertise, and Sustainable Development" (1995), defines Citizen Science as conveying the relationship between science and citizens as it "evokes a science which assists the needs and concerns of citizens… and implies a form of science developed and enacted by citizens themselves".

As opposed to Irwins` very social definition, Bonney et al. (2009) defines Citizen Science simply as "a research technique that enlists the public in gathering scientific information". This is done, as they describe, through projects in which volunteers collect data for use in organized scientific research. Projects cover a wide range of topics that may engage anywhere between a handful of participants in a local project to many thousands of citizens across the whole world (Bonney et al. 2013).

Similarly, the UK- Environmental Observation Framework (2011) defined Citizen Science as a “Non-salaried involvement in collecting environmental observations / measurements which contribute to expanding our knowledge of the natural environment”. This definition, similar to Bonney et al. (2009) covers volunteer engagement in Citizen Science as contributing data to the scientific process in existing studies done by professional researches.

Roy et al. (2013) in a final report on behalf of the UK- Environmental Observation Framework, recommends expanding this definition of Citizen Science beyond the contributory model to include collaborative and co-created projects, which engage volunteers beyond the collection of data and observations. They define Citizen Science as an "overall term" for the various approaches of utilizing volunteers in science in a range of activities and participation levels. This definition includes people as data collectors, as participants forming the projects, assessing the data and using the information themselves (Roy et al. 2013).

Wiggins and Crowston (2011) interpret Citizen Science as "a form of research collaboration involving members of the public in scientific research projects to address real-world problems". They differentiate active engagement in the scientific process, such as volunteer monitoring, from other forms of public participation in scientific research where volunteers may take less active roles, for example providing computing resources or participating as a research subject.

Public participation in scientific research (PPSR)

An umbrella concept that refers to a range of project that engage participants in the scientific process to varying degrees. The different PPSR models yield different types of learning outcomes, and suggest that project developers be deliberate in their project designs, carefully matching design to desired outcomes (Bonney et al. 2013).

PPSR essentially includes all projects that include citizens in the scientific process, similar to Roy et al. (2013) definition of Citizen Science.

Amateur science

I did not find a term of amateur science in the literature. "Amateur" in the context of Citizen Science refers to individuals who are not experts on the topic they are involved in as citizen scientists.

Citizen observatory

Frameworks that enable citizens to use their mobile devices to collect environmental data for use in monitoring and decision making. These projects are being developed by several EU-funded Seventh Framework Programs (FP7), exploring the potential of Citizen Science for informing policy and participatory democracy (EU environmental Citizen Science report, 2013).

Citizen sensing

A type of urban sensor network, where citizens use technology or act as sensors themselves sharing information, observations and views using mobile devices and Web services (Boulos et al. 2011).

Goodchild (2007) defines three types of sensors:

1. Static sensors designed to capture specific measurements of a local environment. 2. Sensors carried by humans, vehicles, or animals. 3. Humans as sensors, each equipped with five senses and the intelligence to compile and interpret their meaning.

Mobile devices equipped with cameras, GPS, and microphones are commonly used to gather, analyze and share local knowledge acting as sensor nodes. These devices, along with human participation, help form interactive human-in-the-loop sensing (Boulos et al. 2011) that have the ability to contextualize, discriminate and filter data, learn, use background knowledge, commonsense and perception, when looking at full sets of data (Sheth 2009).

Crowd science

Wide base volunteer participation project with the goals to obtain a deeper understanding of the natural world and to find solutions to scientific problems. Participation is not restricted to any individuals by location or qualification and is often based on self-selection in response to a general call for participation. Openly disclose a substantial part of the intermediate inputs used in the knowledge production such as data sets or problem solving approaches (Franzoni and Sauermann 2014).

Crowd sourcing

A production model that makes an open call for contributions from a large, undefined network of people (Wiggins and Crowston, 2011), relying on the power of the masses to achieve their goals for organizational designs and information.

Project sponsors are often private organizations that seek to gain a competitive advantage by maintaining unique access to research results and new technologies, therefore, data and even final project results are often not disclosed (Franzoni and Sauermann 2014).

Crowd science

Citizen science by Bonney et al. PPSR/ citizen science by Roy et.al

Crowdsourcing

Crowd sensing

Figure 1- Interactions between the different definitions explained above

Classifications of Citizen Science projects

Since Citizen Science is generally defined as the involvement of citizens in scientific research, many typologies were suggested over the past decade, characterizing the involvement and participation of citizens in these projects (Cooper 2007, Wilderman 2007, Bonney et al. 2009). These typologies focus mainly on the structure of participation and the involvement of the public in different steps of the scientific research. However, typologies of public participation in scientific research, are strongly linked to general public participation typologies in civilian and political domains and this topic will therefore be discussed hereunder.

We will begin this chapter, discussing Public participation according to Sherry Arnstein in her well known article "A Ladder of Citizen Participation" (1969), and demonstrate its relevance to Citizen Science. We will then proceed to discuss public participation in scientific research typologies regarding participation level (Bonney 2009, Hakley 2013) and project goals and physical environment (Wiggins and Crowston 2011).

Public participation

Sherry Arnstein in her well known article "A Ladder of Citizen Participation" (1969) phrases citizen participation as citizen power. The power of the public to determine "how information is shared, goals and policies are set, tax resources are allocated, programs are operated, and benefits like contracts and patronage are parceled out".

Arnstein sets a typology of 8 levels of public participation (and nonparticipation), dividing them to three categories:

1. Nonparticipation- These levels are seen by some as participation, but are in fact methods to educate and persuade the public.  Manipulation- Illusory form of participation, where people are placed on rubberstamp positions in order to "educate" them or engineer their support.  Therapy- Citizens are assumed to have mental illness and are therefore engaged in group therapy rather than hearing their voice and causing change. 2. Tokenism- The public can hear and be heard, but there is no assurance that their views will be considered.  Informing- A one-way flow of information, from officials to citizens, with no opportunities for citizens to influence or oppose.  Consultation- Consulting people using surveys, meetings, and public hearings, but with no assurance that citizen concerns and ideas will be taken into account.  Placation- Citizens have opportunities to advise by being placed on committees and boards, but power-holders continue to have the right to decide. 3. Citizen power- The public has increasing degrees of influence on decision-making.  Partnership- A shared mechanism for policy planning and decision-making, between citizens and power-holders.

 Delegated Power- Citizens achieve dominant decision-making authorities over a particular plan or program by having a majority of seats or votes.  Citizen Control- A degree of power which guarantees that participants have full power on program or institution policy and management, and are able to negotiate the circumstances for making changes.

Although this typology is directed towards national-civilian programs such as urban planning, it could be adjusted to fit other environments such as schools, universities and city halls (Arnstein 1969). Citizen Science research is characterizes with experts- professional scientist, and laymen- the general public, similar to "power-holders" and "have-nots", described by Arnstein. Therefore, when discussing participation of the public in scientific research, Arnsteins' typology could apply and may help clarify the process of scientific research planning and the complex public-scientist relationship (Haklay 2013).

Level of participation in Citizen Science

An additional way to approach public participation, is by focusing on the level of participation, and not the power participants have. This approach is characterizes by the involvement pattern of public participation, according to the answers for following questions: "Who is it that is actually defining the problem? That is, who is setting the agenda for the research? Who is it that is actually designing the study? Who is that is collecting the samples? Who is it that is analyzing the samples? Who interprets the data?" (Wilderman 2007). The answers to these questions can vary, from being all done by scientists to being all done by citizens.

Consistent with this description, Bonney et al. (2009) define three levels of public participation (that have been widely accepted) and divide all Citizen Science projects into one of the following levels:

1. Contributory- projects that engage the public in simple contribution of information to established research designed by scientists. 2. Collaborative- Projects that involve the public in data collection and have an additive value of engaging the public in data analysis and interpretation. 3. Co-created- projects that engage the public in all aspects of the research process including project designing, analyzing and disseminating conclusions.

According to this classification, each project has a specific participation level, determined by who the research initiators and executers are. Different participation levels yield different outcomes, therefore the participation level chosen, should be most suitable for achieving the project goals (Shirk 2012).

However, according to Hakley (2013), one project can be classified, in many cases, to more than one category, depending on the participants' desire and not the project definition. For example- in some projects, most participant may be at the bottom level, while few very active participant, may move to the next level and contribute in additional fashions. Some participants may even communicate directly with the scientist and suggest analysis approaches and new research directions.

Haklay (2013) describes a four level typology:

1. Crowdsourcing- Participation is limited to providing resources, such as computing and carrying sensors, and the cognitive engagement is minimal. 2. Distributed intelligence- usage of participants' cognitive ability as the primary resource. 3. Community science- Definition of the problem is set by the participants, who are engaged in data collection and the research progress is accompanied by scientist and exports (common in environmental justice cases). 4. Extreme Citizen Science- A fully collaborative activity of professional and non- professional scientists in problem definition, data collection, analysis and publication or utilization of results.

When attempting to compare the two typologies, it may seem they are quite similar, however, it is important to look at the details of each level suggested, in order to highlight the differences. For example: A collaborative program described by Bonney et al. (2009) may seem similar to Haklays' (2013) description of community science: they both engage the public in data analysis and interpretation, a process that is accompanied by scientist however, while Bonneys' description defines scientists as the project initiators and leaders Haklays' description clearly puts the public in-charge while scientists' role is advisory and supportive.

Project goals and physical environment

A different approach for classifying Citizen Science, was suggested by Wiggins and Crowston (2011) according to project goals and applications. Wiggins and Crowston (2011) examined 80 common characteristics of Citizen Science projects. They then grouped similar projects that share conditions for successful research operation and identified five types of projects based on their goals and execution methods:

5. Action- Project that encourage participants' involvement in local concerns, using scientific research as a tool to support these activities. Usually planned by citizens, and engage professional researchers as consultants. 6. Conservation- Projects that support natural planning and management of resources. They tend to be local and engage citizens mainly for data collection activities. 7. Investigation- Projects focused on scientific research and findings collected from physical environments. They can be regional or international and engage thousands of participants. 8. Virtual- Projects focused on scientific research in a virtual online participation method. They use advanced technology tools creating online tasks and games and work very effectively with large numbers of volunteers, which is critical for success. 9. Education- Projects with primary goals of education and outreach. The emphasis in these projects in not generating scientifically valid results though is it generally organized by professional research partners.

Principals for choosing and assessing cases

The Technion Citizen Science Project (TCSP) science committee at the 30/06/2014 meeting, defined Criteria for projects do be selected for pilot study within TCSP activity. According to the decision, projects will "contain scientific excellence, innovation and participation by citizen scientists". They further described 16 principles that the projects are expected to cover. In line with these guidelines, the principles for choosing projects for the following case study assessment, and the indicators for analyzing them, were defined and are described below. Six Citizen Science projects were chosen according to these principals, and were reviewed and analyzed in order to shape an integrative understanding of successful Citizen Science projects that have the capability to serve as models for future project planning.

Principles for choosing projects

The initial criteria for projects to be included in this case study assessment, was defined as active involvement of citizens in the scientific research process. In addition, representation of projects from different scientific fields is necessary in order to cover the variety of potential fields to be involved with TCSP (e.g. astronomy, biology, computer science, environment, medicine, education). Projects should be well known and acknowledged Citizen Science projects, indicating successful management and project planning. They should also be based on scientific excellency, providing academic level publications in recognized scientific journals.

Projects selection process

Projects were selected by gathering all 139 projects listed in six established Citizen Science reviews cited in this report (Bonney et al. 2009, European commission report 2013, Silvertown 2009, Dickinson 2010, Wiggins and Crowston 2011, Franzonia and Sauermann 2014) (see Appendix A). A total of 15 projects who were listed in three or more reviews, were defined as candidate projects. For each candidate project, an extensive search was conducted for academic level publications, using pubmed, google scholar and the project website. In order to count as a relevant publication for this search, the article was required to address the relevant scientific field (e.g. articles examining educational outputs were excluded in scientific fields), and derived with the use of public participation in at least one of the research steps. Projects with under 5 publications were excluded from the candidates (total of 5 projects) and similar scientific fields were matched (e.g. three projects that covered bird distribution, projects ecological monitoring), only one project was chosen from each match. This resulted in 5 projects from different scientific fields. Since there was no medical research included in the final project list for case study assessment, and the medical field is relevant and important for TCSP to review, the project "patients-like-me" was chosen from the initial project list (the139 project list) in addition to the final 5 projects. It should be noted that patients- like-me has over 30 scientific publications.

Projects chosen:

 eBird  Fold It

 Galaxy Zoo  Community Collaborative Rain, Hail & Snow Network  OPAL – Open Air Laboratories  Patients-like-me

Methods for project evaluation

Each of the projects selected here for an in-depth assessment, was evaluated as follows. At the first stage, the project website was thoroughly examined and information about project applications, goals and participation was identified. Then I registered as participants in the project, learned about the public interface and the ways participants can contribute and are awarded. This gave me a good grasp of the project progression and the options offered to participants. Next, relevant academic level publications were found for each project and the relevant parts were summarized.

Indicators for evaluating projects

1. The course of the study (Primary goal, research questions, research field, recruitment methods, data collection and analysis methods) 2. Level of participation 3. Time and effort investment (by the participants) 4. Channels and scope of communication (including transparency of results, dissemination of conclusions) 5. Contribution to citizens 6. Ethical, legal and privacy issues

Case studies CoCoRaHS- Community Collaborative Rain, Hail and Snow Network

Project description

CoCoRaHS is a community-based network of volunteers working together to measure and map precipitation (rain, hail and snow) across North America. CoCoRaHS originated at the Colorado Climate Center at Colorado State University (CSU) in the spring of 1998 and is now the largest provider of daily precipitation observations in the United States and most of North America.

The data obtained by CoCoRaHS is used extensively and disseminated automatically to the National Weather Service (NWS) offices throughout the United States.

The course of the study (Primary goal, research questions, research field, recruitment methods, data collection and analysis methods)

CoCoRaHSs' aim is to provide high quality data of precipitation for natural resource, education and research applications (CoCoRaHS, 2015). Their four goals are:

1) Provide accurate high-quality precipitation data for all users. 2) increase the density of precipitation data available throughout the country 3) Encourage citizens to participate in meteorological science, while enjoying the process and increasing their awareness about weather. 4) Provide enrichment activities in water and weather resources for teachers, educators and the community.

Recruitment methods

CoCoRaHS is a community project. As such, everyone can help and be involved, including children and adults of all ages and backgrounds. Schools are also encouraged to participate in the measurements and special lesson plans were developed in order to engage school teachers and students. While bringing CoCoRaHS to Central Great Plains in 2003, Stories about CoCoRaHS and their need for volunteers, were distributed by TV, radio, newspapers and magazines in addition to efforts of the National Weather Service (Cifelli et al 2006). Face to face recruiting was also done at local fairs, conservation district offices and public meetings.

Data collection and analysis methods

Data collection is done by volunteers who have registered to CoCoRaHS and have a listed station on their name. Every day, and each time a rain, hail or snow storm crosses the area, participants record measurements of precipitation and submit the information collected through the online form, application or by telephone. The data are then displayed and organized for end users to analyze and apply and set as precipitation maps that are automatically updated throughout the day.

Data quality verification

CoCoRaHS are trying to get the most consistent data possible using volunteer observers. To do so, they encourage participants to attend a training session, or at least read all of the online training information, available online, before entering data.

Data quality is verified by automated checks that identify and reject incorrect station numbers, dates, time, or unrealistic precipitation values. Participants are asked to confirm their data entry, and one or more of CoCoRaHS volunteers check precipitation maps each day to visually identify potential errors (Cifelli et al., 2005). Daily data is mapped only if it is collected within two hours of 7am.

Level of participation

Participation in CoCoRaHS is based on a contributory model where the public collects data and contributes information to a large database managed by scientists. Data is then provided on the project web site in the form of tables and maps.

Process for submitting data

In order to submit data to the database, registration to CoCoRaHS is necessary. Printable forms will help are available, assisting data collection and submission is possible by an online form and application or by phone. The data submitted appears on the website immediately after submission in both report and map form.

Participation is limited to members who have purchased a standard high capacity 4” diameter rain gauge. This is because the majority of automated rain gauges, when summed over several months or years, report less precipitation than actually fell by a significant amount -- sometimes 25% or more. Moreover, none of the automated gauges work well in areas that receive snow. This is not acceptable for CoCoRaHS project because they are interested in observing and understanding natural precipitation variability, as accurately as possible (CoCoRaHS, 2015).

Time and effort investment (by the participants)

The main investment of participants is to measure and report data daily. Ideally, participation lasts at least one season, but the longer one contributes, the more valuable the data becomes.

In order to participate, members must have a standard rain gauge which is emptied daily. Such a rain gauge can be purchased from WeatherYourWay.com for approx. $30.00. Hail pads can be picked up from CoCoRaHS Headquarters in Fort Collins or from your county's local coordinator (state dependent as not all states are measuring hail). Making hail pads at home is also possible with relatively easy materials.

Participants learn how to use equipment, install and read measurements. They are strongly encouraged to attend a 2-hour training course which covers the placement of rain gauges and hail pads for maximizing gauge catch efficiency and practice at measuring precipitation from a variety of storm types. Training slides our also found on the CoCoRaHS home page.

Some additional opportunities for increased participation include:

 Becoming a local county coordinator  Preparing training materials  Organizing training sessions  Planning educational programs  Recruiting volunteers  Data quality control  Organizing and making hail pads

Channels and scope of communication

CoCoRaHS websites has much information about the project and meteorological information. CoCoRaHS has a bi-monthly newsletter called "the catch" written by the Colorado State Climatologist, a blog and a "message of the day", all describing news, features and progress in the project. In addition, many states produce their own "CoCoRaHS Newsletters" on a monthly or quarterly basis.

CoCoRaHS has a You-tube channel with training animations, tutorials and webinars in addition to Educational Series and hour long talks by experts in different fields of meteorology. There are a number of Unofficial CoCoRaHS Discussion Group Sites, including a Facebook page, where participants share their ideas, thoughts and interesting links and articles.

All precipitation data is available on-line in a number of forms. Daily averages are presented using an interactive map, and can be seen as a list of personal measurements (see Fig. 2). Summarized charts and tables are also available in the projects website for each station or as total observations. A variety of data reports are produced that allow quick and easy analysis of frequency, magnitudes, and real extent of rain, snow, and hail. Data can be downloaded in excel files, for personal use.

CoCoRaHS is used by a wide variety of organizations and individuals. The National Weather Service, other meteorologists, hydrologists, emergency managers, city utilities (water supply, water conservation, storm water), insurance adjusters, USDA, engineers, mosquito control, ranchers and farmers, outdoor & recreation interests, teachers, students, and neighbors in the community are just some examples of those who visit our Web site and use our data.

Figure 2- CoCoRaHS inteactive map presenting daily precipitation data. 29/03/2015 http://data.cocorahs.org/cocorahs/maps

Contribution to citizens

Participants have the chance to do important activities for science and learn some new things along the way. They also have the chance to participate in local activities and make some new friends. In some areas, activities are organized for network participants including training sessions, field trips, special speakers, picnics, pot-luck dinners, and photography contests.

CoCoRaHS creates Climate Resources for Master Gardeners Guide which introduces elements of climate important to gardeners. An overview of climate patterns and differences is shown and information about regional climate is given.

In addition, CoCoRaHS staff have worked with science teachers to develop a series of lesson plans and activities for a variety of grade levels. Educational resources on meteorology can be found on the projects website.

Ethical, legal and privacy issues

CoCoRaHS privacy policy, states that a master list of all participants is kept by CoCoRaHS headquarters. However, this list is not shared or made public. Regional coordinators may

20 be provided with access to the participant contact list on an as-needed basis for the purpose of management and operations of the CoCoRaHS network, but not for private or commercial use.

During registration volunteers are asked to provide personal information (name, email, phone number), the location of their station (address and Coordinates), and state they have a rain gauge or intend the purchase one.

21 eBird

Project description eBird is an online tool for collecting and distributing basic information on bird abundance and distribution in a variety of spatial and temporal measures. It was launched in 2002 by the Cornell Lab of Ornithology and National Audubon Society, and provides a simple and intuitive web-interface for submitting and viewing bird observations (a record of bird species in space and time). eBird is to date, one of the largest biodiversity data resources in existence, with over one- million bird observations reported every month from tens of thousands of participants, more than 100,000 people entering data in any given year and more than one million unique visitors annually who access the on line visualization and analysis tools (Lagoze, 2014). Most of the bird observations are located in the United States, but there is a growing community of eBirders around the world.

The course of the study (Primary goal, research questions, research field, recruitment methods, data collection and analysis methods) eBird’s goal is to "harness the power of everyday birders in an effort to better understand bird distribution and abundance across large spatio-temporal scales and to identify the factors that influence bird distribution patterns" (Sullivan et al., 2009, p. 2). This is done by an interactive online tool that gathers, organizes, and disseminates observations of birds. With time, these data sets will become the basis for a better understanding of bird distribution across the world.

Recruitment methods

The main focus of eBird is on providing services that appeal to the birding community and the project participants, in the process of finding and identifying birds. This model is different than traditional Citizen Science, which is often based on a specific scientific question and engages participants to help find the answer. The shift to this model, as demonstrated by Sullivan et al. (2009) has resulted in extensive growth of eBird project, both in terms of the number of participants and the amount of data submitted.

Data collection and analysis methods

Data collection is done by participants from a broad range of professional backgrounds, including both amateur and expert bird observers. All data is submitted using an online checklist program as described with detail below. Data are classified and grouped according to species, location and time observed and are accessible for further research and analysis1.

Data quality verification

1 See- http://ebird.org/ebird/eBirdReports?cmd=Start

All data submitted to eBird are verified using a two-stage verification system: 1. Automated data quality filters process the data during the data entry. 2. Local experts who examine all records flagged by the automated filters.

The filters, which are built and maintained by the regional editor experts, check all user- entered data, comparing it to the average daily count for each species and location, before adding it to the database. The flagged observations are then sent for confirmation to the user, and if confirmed, sent for processing to the appropriate regional editor (Sullivan et al. 2009). There are over 500 expert reviewers who volunteer to review all flagged observations and devote much of their personal time to this cause. They participate because they share the belief in the purpose and goals of eBird (eBird, 2015).

Level of participation

Participation in eBird is based on a contributory model where the public is engaged in data collection and contribution of information. Scientific data are provided on the project web site and are accessible and open for all scientists and amateurs for additional analysis.

Process for submitting data

Submitting data is done via an online checklist program. A birder simply enters when, where, and how they went birding, then fills out a checklist of all the birds seen and heard during the outing (see Fig. 3). At the first stage, participants are asked to identify the location where the observations take place, this can be done with an interactive map, using latitude and longitude data or selecting a region or county, if the birding was done in a very large area. At the next stage, participants are asked to provide observation date and time, and indicate observation type (what they were doing during observation): traveling (walking a trail, driving a refuge loop, field birding), stationary (observing from a fixed location), historical (cannot estimate start time, duration, and distance), incidental (birding was not the primary purpose- noting a bird while driving or gardening…) or other. Next, the bird observation itself is submitted using the provided list of bird species to be checked. Bird checklist can be sorted taxonomically, alphabetically or by the most likely species to be seen in a specific location. The most likely species are calculated by the frequency of complete checklists reporting the species for a specific location during a three week window surrounding the observation date. Species are grouped by "Frequent", "Infrequent", and "Not Reported" (eBird, 2015).

Figure 3- Three stage process for data submission. Images provided by eBird (www.ebird.org) and created [25/02/2015].

Time and effort investment (by the participants) eBird is designed to serve as a tool for several effort-based sampling types (traveling, stationary, historical or incidental). Some participants may report birds only occasionally, while investing time and effort in a designated location, while others may submit daily checklists of all species they have seen that day from several locations - home, work, or while

24 walking the dog. Everyone with an interest in birds can participate and spent as much time and effort he or she chooses to spend.

Channels and scope of communication

The project has an active website, Facebook page and twitter feed with recent report about project progress, developments and interesting findings and photos. The project website also has an on-going "news and features" blog with updates and news about project events, interesting findings and the "birder of the month" competition.

All birding information is available on the project website and application in a clear and accessible fashion2. eBird users and all others interested, can browse the website and view bird observations, distribution and checklist submissions. Data are presented in several different configuration, enabling each individual to choose the best display for his or her needs. For example: bird distribution can be found by entering a specific location of interest, by exploring bird hotspots (public birding locations) around the world on an interactive map or by searching specific species on the world map (see Fig 4). In every one of these cases, additional details can be found by zooming in or by clicking on the point of interest. Detailed lists of species and submitted checklist are publicly available in every location, in addition to bar charts and line graphs that can be created interactively by all interested (eBird, 2015).

Figure 4- Interactive map presenting bird hotspots, with the number of species and checklist submitted in a specific location of interest. Image provided by eBird (www.ebird.org) and created [25/02/2015].

In addition to checklist information submitted by eBirders, lists of yearly arrivals and departures of bird species for a country, province, county, or hotspot, are available, as well as lists of all-time first and last records and species high-count records. Detailed reports can be produced and printed, summarizing all collected data for a specific location over a designated time or period and raw data can be freely downloaded for noncommercial use in order to

2 See- http://ebird.org/ebird/eBirdReports?cmd=Start

25 personally analyze birding data. All the features described above are available in English, Spanish, French, Portuguese and Chinese (eBird, 2015).

The community using the data, is divided between academic, governmental, private organizations, NGOs and individuals and who are interested in topics such as migration, conservation, modeling, mapping, climate, population, conservation and habitat (Lagoze, 2014).

Contribution to citizens

As described above, the main focus of eBird is on providing useful services to the birding community, which will assist them in the process of finding and identifying birds. This tool enables easy submission and consolidation of all user-specific bird observations and lists, while providing exposure and recognition for each individuals' work. These user-specific bird lists create incentive for birders to enter both current and historic data and the motivation to continue and increase the effort invested (Sullivan et al. 2009).

In addition, eBird provides a community based platform enabling birders to compare their birding accomplishments to those of fellow birders and to improve and advance their bird observation skills by learning from others experiences. These features, take advantage of the crowd sourced structure of eBird platform and serve as an example of how a Citizen Science project can both motivate participation and contribute to scientific knowledge construction (Lagoze, 2014).

Ethical, legal and privacy issues eBird does not have an ethical statement, but does have a privacy policy published in its web site. The privacy policy states that by registering, participants allow eBird to associate their bird observations with a unique identifier (login name or your name). Participants' email address and contact information is stored for internal use (for personal contact if necessary), but will not be sold or distributed outside the Cornell Lab of Ornithology and other eBird partners (eBird, 2015).

All details of bird observations, submitted by participants, and their associated location, are available to all users registered with eBird, scientists, and others who are interested in examining the data. Downloaders of raw data must be registered users (anyone can register) and must agree not to use the data for commercial use and to attribute the Cornell Lab of Ornithology in the results obtained by use of the data.

Foldit

Project description

The game was initially launched in May 2008 by a group of scientists from the University of Washington, and by September of that year it had engaged 50,000 users. One year after launch there were about 200,000 active Foldit players (Franzoni and Sauermann 2014), and activity remains steady with about 2,000 active players who play more than once a week.

The course of the study (Primary goal, research questions, research field, recruitment methods, data collection and analysis methods)

Foldits' primary goal is to produce accurate protein structure models through gameplay (Cooper 2010). They aim to: A. have human folders work on proteins that do not have a known structure and B. take folding strategies that players have come up with while playing the game, and automate them to make effective protein-prediction software. In addition, Foldit has a goal in promoting protein design in assistance to protein engineering strategies with the goal of bringing these structure to reality (Foldit 2015).

Research field

Foldit is a result of a collaboration between the Computer Science and Engineering Department and the Biochemistry Department at the University of Washington. With the knowledge and skills coming from each of the two departments this successful project was built which studies and makes scientific Biochemistry advances through computing, while continually improving Computer Science technology.

Recruitment methods

Foldit is open for all users around the world. It was planned to fit the use of players with no scientific training and no previous exposure to molecular biology. To do so, many technical terms were replaced with more common terms and new concepts are introduced through a series of introductory levels puzzles (Copper et al., 2010).

Some of Foldit earliest participants came from the Rosetta@home community, a project that was initiated by the same research group in the University of Washington. This influenced and assisted the initial recruitment work of participant, as the Rosetta@home community was already interested in protein folding issues (Copper et al., 2013).

Data collection and analysis methods

Level of participation

Foldit is considered a contributory project according to Bonney et al. (2009) definition of public participation, this is due to the fact that participants contribute information, in the form of protein folding options, but do not participate in analyzing and verifying the folded structures' ability to fold and exist in reality.

Process of game playing

This on-online game enables players to shape protein structure, with the move of the mouse, by moving, rotating and shaking protein structures.

Incorrect folded protein conformations are presented as folding puzzles and players can inspect the structure from different angles, interactively reshape them by rotating or flipping chain branches in the direction they believe will lead to a better structure and highest score in the game (see Fig 5).

There are three basic biological principles that determine the score each protein structure receives: 1. Protein package (The smaller the protein, the better) 2. Hiding hydrophobic and exposing hydrophilic sidechains. 3. Keeping different sidechains apart and clearing the clashes between them (Foldit, 2015).

Figure 5- Folded up protein puzzle. Image provided by Foldit (http://fold.it/portal/info/about)

Foldit allows player to improve the game by teaching the computer and other participants their strategies, and build upon each other's work by forming groups, in which they can share structures to get ideas. Each player in the game has his own “cookbook” of recipes, which are folding strategies players come up with and develop. Recipes can be created by applying new algorithms or by modifying and recombining successful recipes developed by other players. Using these recipes, players can utilize a variety of interactive automated strategies, they can share their recipes with the rest of the Foldit community, continue to develop their own or others' recipes or decide to keep their recipes to themselves (Khatib et al. 2011).

Time and effort investment (by the participants)

Beginner players have a series of introduction puzzles to solve before starting to solve "real" science puzzles. These levels teach the game’s tools and visualizations, and certain strategies that can be used. After completing these puzzles, players can proceed to science puzzles of contests and play as much or as little as they choose.

Channels and scope of communication

Foldit has a website with project news, information about puzzles, participants and groups and a Wiki page with much information for beginner and advanced players. The internet site has a blog which describes some of the outcomes and results of the folding work done by participants, it provides an insight to future challenges and developments, and gives a better understanding of what scientific work is done with foldit to date and where they hope to go in the future (Foldit, 2015). Foldit also have scheduled scientist and developer chats where participants can communicate directly with the people working on the project and get a view of what is happening behind the scenes of Foldit (Cooper et al., 2013).

Dozens of videos about Foldit are available on Foldits' youtube channel including tutorials, puzzle specific videos, messages from the Foldit team and general Foldit related videos including Foldit TED talk, interviews and news reports.

Scientific discoveries and publications are made available on the project website.

Contribution to citizens

Playing Foldit is intended to be a fun and enjoyable experience. Scoring of structures is immediate, and players know how they're doing in the game, and can continue modifying their structure until satisfaction. Players learn biochemistry concepts while playing, and can choose to broaden this knowledge with videos and lessons available on the game website, connecting players to the science behind the game

Foldit promotes social interactions by providing dedicated technological infrastructure such as discussion forums, contests and chat channels. These networks allow participants to team

29 up and collaborate, to exchange strategies and compete in groups, promoting social relationships and enjoyment (Franzoni and Sauermann 2014)

Foldit also has an option for creating private groups, for educators to use in their classrooms. The moderator of the group (e.g. teacher) can view all of the members of his group by logging into the website and view user's achievements. Teachers can also create contests that are specific for their class and can only be accessed by the members of the group.

Ethical, legal and privacy issues

Foldit has their "terms of service and consent" published in their website, and users are required to accept these terms before playing.

The terms of service states that Foldit records gameplay data for analysis and research. The data is associated with the players account, or collected anonymously if the player did not set an account or plays offline.

Data collected includes biochemical structures, scores, algorithms, tool and algorithm usage, progress, and time played. General chat channels may be logged to monitor for abuse, but are not used for research purposes. Private group channels and private messages will not be logged unless, in extreme cases, needed to prevent abuse of the system. Since chatrooms and website are publicly available, Foldit terms of service states that a third party may independently log these channels.

Scientific discoveries will be made publicly available and the University of Washington will handle the ownership of discoveries. Individual players who contributed to the discovery will be considered co-inventors for any discovery produced through play. Data logs of player activity will assist in determination of attribution. In addition, all data used for research may be made public, shared with collaborators, shown in the game, on the website, or be used in external publications, presentations, promotional materials, and other public venues (Foldit 2015).

Graber and Graber (2013) investigated the ethical aspects of on-line research games such as Foldit, and termed the concept "gremes" - game-research hybrids. According to them, gremes participants are themselves objects of study and thus research of this form should be subjected to IRB (institutional review board) review. Furthermore, Graber and Graber (2013) claim that participation in gremes can cause proximate harm to participants, reducing the time spent on other activities such as exercise, family time and school work and can contribute to internet addiction, a recognized psychological disorder. Graber and Graber (2013) therefore conclude that there should be IRB review process that assures participants are informed of the risks and potential outcomes of greme participation.

Galaxy Zoo

Project description

Galaxy Zoo is an online platform for classifying galaxies from the Sloan Digital Sky Survey (SDSS). The site was first launched at June 2007 and within 24 hours of launch received almost 70,000 classifications an hour. During the sites first year, more than 50 million classifications were received, contributed by more than 150,000 people. Since, Galaxy Zoo has launched a number of classification versions (the fourth launched at 2014), each asking volunteers to classify galaxies in slightly different ways (Galaxy Zoo, 2015).

The course of the study (Primary goal, research questions, research field, recruitment methods, data collection and analysis methods)

The goal of Galaxy Zoo is to classify galaxies by having volunteers look telescopic pictures and sort them by shape. Galaxy Zoo hopes to assist astronomers in studying the way galaxies were formed and to better understand our universe.

Recruitment methods

During project initiation, a news story on a BBC Web site was published which got thousands of entrances to the site.

Data collection and analysis methods

Galaxy classification is done by inviting volunteers to visually inspect and classify pictures of galaxies, on the project site.

Data quality verification

Obtaining morphologies of the galaxies by direct visual inspection, avoids introducing biases that are associated with proxies for morphology such as color, concentration or structural parameters. Each galaxy was viewed and classified multiple times, with a mean of 38 classifications per galaxy, increasing the validity of the classification. Galaxy Zoo results were found to be consistent with the results obtained by professional astronomers, and demonstrate the data accuracy and robustness (Lintott et al, 2008).

Level of participation

Participation in Galaxy Zoo is based on a contributory model since participants are engaged in contribution of information, in the form of galaxy classifications. Participants sometimes contribute additional information and initiate discussions in the site blog, however generally they do not participate in further analysis of data.

Process for submitting data

Visitors to the site were initially asked to read a brief tutorial giving examples of each class of galaxy, and then to correctly identify a set of ‘standard’ galaxies, classified by team members. Those who correctly classify 11 of the 15 were allowed to proceed to the main study (Lintott et al, 2008). Today (June 2015) in the fourth version of the site, no login is required and no tutorial seem to be mandatory.

Submitting information is simply done by visualizing the image presented and answering a series of questions as to the figures shapes. A decision tree which shows all the possible paths Galaxy Zoo users can take when classifying a galaxy, can bee see in Figure 6.

Figure 6- Visualization of the decision tree for Galaxy Zoo 2 (GZ2), by C. Krawczyk. Image provided by Galaxy Zoo (http://data.galaxyzoo.org/gz_trees/gz_trees.html)

Time and effort investment (by the participants)

Time investment is not defined and participants can classify anywhere between one galaxy to several hundred or thousand. Each classification takes 10-30 seconds. Participants have an option to discuss the pictures featured to them, write comments or ask questions.

Channels and scope of communication

Galaxy zoo has a platform for discussion called "talk" in which participants can have conversations and ask question. In addition there is a blog introducing new developments, research results and news.

Contribution to citizens

In a survey taken place by Raddick et al (2010), twelve motivations for participating in galaxy Zoo were identified. These motivations include contribution to scientific research, helpful platform for learning about astronomy, discovering galaxies that few people have seen before, beautiful pictures and others. In a follow up research (Raddick et al., 2013), researchers found that "contribution" is the most important motivation for Galaxy Zoo volunteers participating in the survey, with close to 40% responses.

"Astronomy" (I am interested in astronomy) and "discovery" were the next frequent replies with about 13% and 10% responses respectfully.

Galaxy zoo also has a platform for educators including many lessons plans and learning resources for student of different ages. Users can open private groups for classifying images and compare classifications done by members of the group.

Ethical, legal and privacy issues

Foldit has their "user agreement and privacy policy" published in their website, and users are required to accept these terms when registering to the site. The agreement states that uses grant the license to: use, reproduce, modify, adapt, publish, translate, create derivative works from, distribute, and exercise all copyright and publicity rights with respect the contribution made in the site.

Galaxy zoo recognizes the personal contribution of each of their volunteers, these efforts are individually acknowledged on their website3.

The site collect some personal information such as e-mail address, and usage information such as log in timing, Pages viewed, Classifications made and posts on Talk pages. Collected data is not shared with third parties unless it is anonymized.

3 See http://authors.galaxyzoo.org

OPAL- The Open Air Laboratories

Project description

OPAL is a Citizen Science initiative active across the UK, which motivates the public to get closer to their local environment while collecting scientific data. Lunched at 2007 and led by the Imperial College London, OPAL includes leading museums, universities and environmental organizations across the UK (OPAL, 2015).

OPAL promotes active participation and involvement with nature, and encourages participants to take the next step and record their observations, develop ecological knowledge and apply it. This is done with OPAL national surveys that combine observations of wildlife with data on air, soil and water condition.

With over half a million people who have actively participated in the OPAL programs, studying more than 25,000 sites across England, (OPAL Community Environment Report, 2012) OPAL has lately grown out of the boarders of England and is now active in Scotland, Wales and Northern Ireland.

The course of the study (Primary goal, research questions, research field, recruitment methods, data collection and analysis methods)

The main aim of OPAL is to carry out high quality research with maximum public engagement (Davies et al. 2011). To do so, OPAL has set five main objectives directing its activity (OPAL 2015, Davies et al. 2011):

1. Make people more aware of the open spaces and conservation sites around them. 2. Construct knowledge and confidence in environmental issues through new learning approaches. 3. Inspire a new generation of environmentalists. 4. Understand more about the natural environment in sparse locations, by ensuring participation options for everybody, especially disadvantaged communities. 5. Strengthen partnerships between the community, voluntary and statutory sectors.

Recruitment methods

OPAL aims to encourage the public, throughout the UK, to get closer to nature in their local environment. To do so, OPAL is open to all members of the general community, including people from different backgrounds, ages, disadvantaged communities etc. According to the OPAL Community Environment Report (2012), 50% of the surveys produced are distributed directly to schools.

The OPAL regional network of universities extends the outreach work and connection to local communities, and provides the national infrastructure for direct community engagement. Each regional university employs an OPAL Community Scientist who works directly with local

34 people, motivating them to get involved in OPAL activities (OPAL Community Environment Report, 2012).

Data collection and analysis methods

OPAL has five national research centers (soil center, water center, climate change center, air center and biodiversity center), each responsible for producing an ecological survey from a series of national surveys developed by OPAL. Each survey explores the relationship between a group of organisms (biotic) and habitat quality (abiotic) and promotes current policies that address pollution and environmental protection. These surveys are the primary mechanism through which communities explore and learn about their local environment.

Data quality verification

Before preceding analysis of the public data by professional scientists, all submissions are screened and flagged if location information is not thought to be accurate, or is outside the UK. Duplicate records are identified as records where all of fields including the location information are exactly the same.

In addition, the survey data is compared with existing sources and databases to evaluate the state of agreement and the quality of the data submitted. For example: The soil classification done in the OPAL soil survey was compared to the World Reference Base for Soil Resources system and to the European Soil Database (ESDB) (Bone et al., 2012).

Public participation

Level of participation

Participation in OPAL is based on a contributory model where the public is involved in data collection alone. Participants are encouraged to use the knowledge and data they have obtained for their own use and benefit.

The program essentially provides the opportunity for the local community to participate in scientific research, collecting environmental data for both personal use and for research purposes. Participants explore and understand some of the uncertainties in nature, demonstrating the reasons scientists carry out research of this sort and how these datasets can be used (Davies et al. 2011).

Process for surveying

Special packs are produced by OPAL for each survey, including field guides, identification charts, and all materials and equipment needed for the survey. Clear instructions are provided for every step of the survey, including introduction of the topic, explanatory pictures and diagrams and clear multi-choice questions (See Fig. 7). Packs can also be downloaded from the internet site (excluding the materials and equipment that can generally be found at home or a nearby store). Participants walk through the instruction given in the survey field guide, answering the questions, identifying species and noting characters, in their chosen researched location.

Figure 7- Field guide of the Tree health survey. Provided by OPAL (www.opalexplorenature.org/treesurvey) and created [23/03/2015].

Submitting data

Entering data is done with an on-line form that includes location and date of the survey, level of expertise, which is determined by asking a relevant species identification question, and answers on the multiple choice questions in the survey. In addition, the form has a serious of questions about the user experience and personal gain and attitude of participants following participation (e.g. Did you develop new skills? Will you change your behavior towards the environment?) and an option to add personal notes or comments (OPAL, 2015).

According to the OPAL Community Environment Report (2012) who presented data of submitted surveys, out of 230,000 OPAL surveys that were produced and distributed, and an additional 100,000 surveys downloaded from the OPAL website, only 25,000 surveys have been submitted to the OPAL database. They estimate that at least five times as many surveys have been completed, but not submitted. Davies et al. (2011) gives three main reasons for this gap: a) participants enjoyed the activity but did not want to enter data on the computer. b) Participants did not have access to a computer. c) Participants were not confidence in their data quality. Due to these reasons OPAL has given the option to submit data via free post mail.

Time and effort investment (by the participants)

Participation in OPAL is not limited in time, participants can choose to survey as many times as they want and survey time can vary between participants. Since one of the main goals of OPAL is to get people investigating the nature around them, even starting a survey without completion or without submitting the data, may be still be considered as a project success. This is demonstrated is the website where resources from closed surveys are still available for download and learn from.

Channels and scope of communication

The OPAL web site has very detailed information, written in a clear accessible fashion directed to participants from all ages backgrounds and abilities. The explanations include pictures, short videos and tips for things to look out for and things that could be confused with what is being monitored (e.g. while monitoring Oak mildew, a plant disease that infects young oak leaves and shoots, one should not be confused with other powdery mildews on other plants that are different species of this disease) (OPAL, 2015). The OPAL website also has a scientist blog which provides News, updates and opinions from community scientists across the UK. They give the opportunity to go behind the scenes and learn more about the work done at OPAL. Interviews with the leading scientist are available on line, explaining how to become a scientist, what OPAL scientists do etc. A quarterly email newsletter is distributed to registered participants with news and update about the research done in OPAL.

Communal activities are conducted regionally by the Community Science team and includes regional meetings, workshops and open days. In addition these teams give training, materials and support to participants and oversee collaborations and involvement with local government, government agencies, local communities and voluntary sector organisations (Davies 2011).

Scientific results are available graphically using on-line maps results, for each of the surveys examined (See example in fig. 8). Results of finished surveys are also presented online, in a clear summarized version highlighting the main findings. However, the main use of the data on a personal level is directed to the use of individual information collected personally by the participant in his proximity.

Figure 8- OPAL results map for water quality survey (Davies et al. 2015).

Contribution to citizens

OPAL focuses on the contribution for participants as one of the main goals of the project, and calls itself "a community-focused research and education partnership". As such, OPAL has invested much, in informative and educational resources than can all be easily accessed online. The OPAL surveys are designed to be self-explanatory and are suitable for a wide age range. Special activities for children including games and activities, easy explanations and quizzes are also available. Primary school education packs, lesson plans and curriculum guides are all downloadable, in addition to ideas for designing personal science projects, using OPAL data and methods.

Ethical, legal and privacy issues

OPAL has a privacy policy published on her website, in addition to their acceptable use policy, and the website terms and conditions. These policies state that by using the site, one gives his consent to the listed conditions.

In addition, all information placed by a participant on the site is owned by the OPAL website who can use, reproduce, translate and distribute the material (including text, photographs or video).

PatientsLikeMe

Project description

PatientsLikeMe was founded in 2004 by three MIT engineers, it was opened publicly at 2006 and a year and a half later, the community contained 1570 verified patients (frost et al. 2008), and currently (June 2015) has 325,000 members across over 2,400 medical conditions (PatientsLikeMe, 2015).

These patient-reported data can provide a new source of evidence about secondary uses and potentially identify targets for treatments to be studied systematically in traditional efficacy trials.

The course of the study (Primary goal, research questions, research field, recruitment methods, data collection and analysis methods)

PatientsLikeMe goal is to provide a platform for patients who want to share their health information to create collective knowledge about disease, health, and treatments. Their four core values include: putting patients first, promoting transparency (“no surprises”), fostering openness and creating “wow".

Recruitment methods

Members of PatientsLikeMe find out about the site through a variety of channels: search, paid advertisements, public relations, press mentions, academic collaborations, word of mouth from patients, and provider referrals (Wicks et al, 2010).

Data collection and analysis methods

Members enter their personal information through a website platform, including demographic information, longitudinal treatment, symptoms, outcome data, and treatment evaluations.

PatientsLikeMe believe that sharing healthcare experiences and outcomes is good. Because when patients share real-world data, collaboration on a global scale becomes possible. New treatments become possible. Most importantly, change becomes possible.

Data quality

PatientsLikeMe admits that their data may be biased due to unequal representation of the population. For example, their users tend to be slightly more likely to be female, are a few years younger, a few percentage points more likely to be white, and a few percentage points more likely to have a higher level of education than the general population. In addition, the platform in only available in English and 70% of users are located in the United States.

Participation

Level of participation

The basic involvement of participant can be seen as contributory, where each patient enters data regarding his personal condition. However, besides the use of the information as a medical database, the information is first of all used on a personal level where each member can analyze the data of others for his own benefit. Although these "mini" studies are not publish in scientific papers, they are still engaging participants in higher levels of research, beyond the classical contributory model.

Process for submitting data

Types of information members can enter through the website (PatientsLikeMe, 2015):

 Biographical information, e.g. photograph, biography, gender, age, location (city, state and country), general notes;  Condition/disease information, e.g. diagnosis date, first symptom, family history;  Treatment information, e.g. treatment start dates, stop dates, dosages, side effects, treatment evaluations;  Symptom information, e.g. severity, duration;  Primary and secondary outcome scores over time, e.g. ALSFRS- R, MSRS, PDRS, FVC, PFRS, Mood Map, Quality of Life, weight, InstantMe;  Sensor information, e.g. personal activity trackers;  Laboratory results and biomarkers, e.g. CD-4 count, viral load, creatinine, voice features, images;  Genetic information, e.g. information on individual genes and/or entire genetic scans;  Individual and aggregated survey responses;  Information shared via free text fields, e.g. the forum, treatment evaluations, surveys, annotations, journals, feeds, adverse event reports; and  Connections to other people on the Site, e.g. invited care team member, mentors, feeds, subscriptions.

The website offers a variety of tools to help patients record the treatments they are taking, supported by a drug database to promote accurate data entry. On an individual basis, each patients can see an historical profile of his treatment over time, using visual displays (Wicks et al., 2010).

Time and effort investment (by the participants)

Members of PatientsLikeMe can choose to include as much information as they want and use the information presented by other as much as they need. Patients are encouraged to add as much information as possible, as this information aids other patients.

Channels and scope of communication

The project has a very detailed website featuring the shared information about members' conditions, treatments and symptoms, and includes a blog with articles about the project, its uses and the team. In addition it has a Facebook page and twitter feed with recent advances in the project, and personal stories, it has a podcast series and a youtube channel with many videos including patients stories (the patient voice), and interviews with doctors and researches.

Participants can decide if their profile it publicly available or open only to other members of PatientsLikeMe. Public profiles show patient username and profile photo, charts featuring shared information about patient condition such as outcome scores, lab values, treatments, treatment evaluations, symptoms and weight. In addition Public profiles show demographic information including age, gender and location and biographical information added to the profile (See figure 9). Data is also presented as averages by conditions, treatment and symptoms.

Figure 9- Example of charts featuring shared information about patient condition in a public profile. Provided by patientslikeme (https://www.patientslikeme.com/patients) and created [15/06/2015].

PatientsLikeMe enables the existence of local communities, and has for example an ALS community, HIV community and epilepsy community. Members can discuss the profiles and reports as well as general health concerns through the Forum, private messages, and comments they post on one another’s profile (Frost et al. 2008).

Contribution to citizens

The platform was built thinking of the patients and their uses. The site offers a variety of tools to help patients record the treatments they are taking, supported by a drug database to promote accurate data entry. By sharing information on the site, patients can put their disease experiences in context and find answers to the questions they have. Some of the options for patients include:

 Get detailed information about every medication, supplement, or device used to treat patients with similar conditions and learn about what works.  Connect easily with other patients that have the same conditions, experiencing the same symptoms or using similar treatments.  Learn from other patients' experience.

The data is presented back to members as individual-level graphical health profiles and aggregated into reports accessible on the site (see fig. 10). Members can discuss these data sets either within a group forum or individually through private messages. The resources on the site are designed to help members answer the question: “Given my status, what is the best outcome I can hope to achieve, and how do I get there? (Wicks et al. 2010).

From a survey testing the potential benefits of PatientsLikeMe to memberes, the greatest benefit Users perceived, was in learning about a symptom they had experienced. Patients also found the site helpful for understanding the side effects of their treatments and nearly half of patients agreed that the site had helped them find another patient who had helped them understand what it was like to take a specific treatment for their condition. In general 72% of users rated the site “moderately” or “very helpful.” (Wicks et al. 2010).

Figure 10- Charts comprising the personal profile (Frost et al. 2008)

Ethical, legal and privacy issues

Most healthcare websites have a Privacy Policy. Naturally, we do too. But at PatientsLikeMe has a Privacy Policy, but states that they are more excited about their Openness Philosophy, which states that the more shared information, the more options for collaboration, treatments and change.

When a Member enters Personal Information, including name and email address, as part of registering to use PatientsLikeMe, that Personal Information is treated as Restricted Data. Additional restricted data includes: Password, Mailing address, Date of birth and private messages.

What makes projects successful? After reviewing six successful Citizen Science projects, active around the world, common characteristics were identified in order to better understand what makes these project successful. Fifteen features were identified, that repeatedly emerged in many of the projects reviewed, divided to four main categories: goals, community, platform and dissemination. All projects obtained at least seven of the fifteen features identified, and most of the features were presents in at least three of the six projects (See table 2).

Table 2. Presence of fifteen features in each of the six reviewed projects.

Galaxy Patients total projects CoCoRaHS eBird Foldit OPAL Zoo LikeMe for each feature Goals scientific goal X X X X X X 6 educational goal X - - - X - 2 social goal - X - - X X 3 Community existing community X X - - - X 3 does not requires previous X - X X X X 5 knowledge provides social platform - X X X X X 5 network of supporting X X - - X - 3 volunteers connection with scientists - - X X X - 3 platform simple platform X X game X X X 5 application X X - - X X 4 provides learning platform- X - X - X - 3 online provides learning platform- X - - - X - 2 personal (workshops) educational materials X - X X X - 4 dissemination dissemination of raw data X X - - X X 4 facilitated dissemination of X X X X X X 6 results total features in each 12 9 7 7 14 9 project

All projects reviewed, had a very clear scientific goal which is the essence of a Citizen Science project. However four of the six projects had additional educational or social goals. Although all six projects have clear contributions to the citizens participating, the four project who have set educational or social goals have made a special effort to facilitate the project in this way. Some of these efforts include building specific services and tools that appeal to the volunteering community (in eBird), designing work guides and tasks that are accessible to all ages, group and communities (in OPAL) and organizing face-to-face workshops and working teams (in CoCoRaHS).

Another important feature identified in all projects reviewed, is the simple and clear platform for collecting data. Whether data is submitted from physical data collection or derived from online platforms, the interface is always very user-friendly and intuitive. Although some project require registration and some do not, once ready to submit data, the process is easy and clear.

Three of the six projects reviewed, directed themselves towards an existing community, making it somewhat easier to recruit participants. Still, five of the six projects do not require previous knowledge in order to participate, eBird being the exception. Three of the projects provide online learning platforms of the topics investigated, in the form of videos, tutorials, manuals and work guides, and two projects have additional scheduled workshops. Although eBird requires some previous birding knowledge in order to accurately identify bird species, and do not have learning materials on their internet site, other online resources are available online, including the Cornell lab of ornithology "all about bird" guide4. Three of the project have options to chat with scientists, communicate directly with the people working behind the scenes of the project, ask professional questions and get a better insight as to the uses of the data and its analysis.

Four of the five projects have additional educational materials online, providing teachers and educators opportunities to implement the use of the platform in their classroom. These include lessons plans and learning resources for student of different grades and ages. Foldit, for example, has an option of creating private groups and contest giving the teacher the opportunity to view the activity and achievements of all members of the group.

An important feature identified in five of the six projects is the promotion of social interactions between participant facilitated by forums and chat rooms. These social platforms enable participant to share their findings, have discussion and ask questions. They also serve as community building tools enabling collaborations, exchange of strategies (e.g in Foldit), provide learning opportunities from others experiences and promote social relationships and enjoyment.

Organizing huge databases and constant flow of information can sometime be too much for a local group to manage. For this reason, three of the projects reviewed have a local network of volunteers participating in additional tasks of management, data verification and training. CoCoRaHS for example, has volunteers involved as local county coordinators, in preparing training and educational materials, organizing training sessions, recruiting volunteers etc.

Dissemination of result in a blogs, newsletters or reports is available in all six projects reviewed, and is always written in a clear and accessible fashion. Some projects have a "message of the day" or "volunteer of the month" features, with additional general or personal information presented. Four of the six projects also have an option to view full data sets in the form of interactive maps, charts and tables. Volunteers can hence analyze the data, identify trends and interesting phenomenon and create their own "mini research".

4 See www.allaboutbirds.org

Galaxy Zoo and FoldIt were exceptional in many of the features identified and had the fewest common features among the projects reviewed (e.g. don't have social or educational goals, a smartphone application and do not disseminate raw data). This may be due to the different characteristics of the projects which to not collect physical data from the public, but rather derive the data from the active online participation of volunteers in a game-like platform. Galaxy Zoo and FoldIt are defined by Wiggins and Crowston (2011) as virtual projects rather than investigative projects, which mean they use advanced technological tools to create online tasks for data collection. Virtual projects work very effectively with large numbers of volunteers and these large numbers are critical for project success. Hence, these two projects may use slightly different implementation methods in order to be successful.

Conclusions

From the analysis presented here, a picture is formed of the features that make Citizen Science projects successful, in addition to the importance of scientific excellency. The major features identified in all successful projects reviewed, and that we believe are fundamental for project success, are the user experience and the creation of motivation for participant. These features include user-friendly interfaces, simple platforms for collecting data, online videos, tutorials, and work guides, social platforms and clear dissemination of results. We believe that the presence of these features motivates participants to continue to participate and contribute to the on-going research.

Appendix

Appendix A- 139 projects listed in six established Citizen Science reviews.

Wiggins Franzonia European Bonney Slivertown Dickinson and and Sum Projects commission (2009) (2009) (2010) Crowston Sauermann citation (2013) (2011) (2014) Achuar-Amazon Watch initiative X 1 Air Quality Egg X 1 ALLARM Acid Rain Monitoring Project X X 2 Amateurs as Experts X 1 Ancient Lives X 1 Argus X 1 Avian Knowledge Network X 1 Barnegat Bay Partnership X 1 Bat Detective X X 2 Bay Area Ant Survey X 1 Big Butterfly Count X 1 Big Garden BirdWatch X 1 Birdhouse Network (now NestWatch) X X X X 4 Britain’s Common Birds Census X 1 British Trust for Ornithology X X 2 Buckeye Lady Beetle Blitz X 1 Butterfly Conservation Europe – Butterfly X 1 Monitoring Chicago Wilderness Project 0 Christmas Bird Count X X X 3 Citizen Science Canada X 1 Citizen-led activities at Lake Kirkkojärvi X 1 CitizenWeather Observer Program X 1 ClimatePrediction.net X 1 Community Collaborative Rain, Hail & X X X 3 Snow Network Community Health Effects of Industrial X X 2 Hog Operations Connect2Decode X 1 Corfe Mullen Bio-Blitz X 1 CreekFreaks X 1 Crowd Computing X 1 Cyclone Center X 1 Did You Feel It? X 1 Discover life X 1 eBird X X X X X 5 Eco21.PL X 1 Erie Rising X 1 EteRNA X 1

48 evolution mega lab X X 2 Eye on Earth X 1 Firefly Watch X X 2 Fold It X X X 3 Fossil Finders X 1 French Garden Butterfly Monitoring X 1 FrogWatch USA X 1 FuturICT X 1 Galaxy Zoo X X X X 4 Garden Moths Count X 1 Globe at Night X 1 Gravestone Project X 1 Great Backyard Bird count X 1 Great LakeWormWatch X 1 GWAP X 1 Hunter Valley rehabilitation project X 1 Ice Hunters X 1 Invasive crabs (US) project X 1 Invasive Plant Atlas of New England X X X 3 Invasive species survey 0 Invasive Tracers X 1 ITDG Zimbabwe project to restore local X 1 food security (p9) Lost Ladybug Project X X X X 4 Milkyway Project X 1 Missouri Stream Team Program X 1 Monarch Larva Monitoring Project X X X X 4 Monitoring H¨aufige Brutv ¨ oge X 1 Moon Zoo X 1 MykoWeb X 1 NASA’s Earth Observatory X 1 National Bat Monitoring Programme X 1 National BioBlitz Network X 1 National Biodiversity Network 0 Nature North X 1 North American Amphibian Monitoring X 1 Program North American Bird Phenology Program X 1 North American Breeding Bird Survey X 1 Northeast Phenology Monitoring X 1 Notes from Nature X 1 Ocean Optical Monitoring X 1 Old Weather X X 2 OMNISCIENTIS – Odour Monitoring and X 1 Information System OPAL – Open Air Laboratories X X X 3 Open Dinosaur Project X X 2 Open Street Map project X 1 pacewraps X 1

Pan-European Common Bird Monitoring X 1 Scheme Patientslikeme X 1 Peppered moth X 1 Perfect Pitch Test X 1 Phylo X 1 Picture Post X 1 PigeonWatch X X 2 Planet Hunters X 1 plant watch X 1 Polymath X 1 Project Budburst, including floracaching X 1 game Project FeederWatch X 1 Project Implicit X 1 Protea Atlas Project 0 Radio Jove Project X 1 Reclam the Bay X X X 3 REEF X 1 Road Watch X 1 Royal Docks Noise Mapping X 1 Salal Harvest Sustainability Study X X 2 Science, Democracy and Community X 1 Decisions on Fracking Seafloor Explorer X X 2 Secchi App X 1 SETI@Home X 1 Setilive X 1 Sherman’s Creek Conservation X X X 3 Association Snapshot Serengeti X 1 SnowTweets X 1 Solar Stormwatch X 1 Space NEEMO X 1 Spotting the Weedy Invasives X X X 3 Stardust@home X X 2 Sun Lab X 1 SusClime X 1 Swedish Species Gateway 0 Tag a Tiny X 1 The Chicken Coop Stakeout X 1 The Great Sunflower Project X X X X 4 The Shale Network X 1 The Smell Experience Project X 1 Twitter Earthquake Detection Program X 1 UK Butterfly Monitoring Scheme X 1 UK Ladybird Survey X 1 USA-National Phenology Network X 1 USGS Frog Quizzes X 1 UVA Bay Game X 1

Vigie Nature X 1 WESENSEIT X 1 West Visayas State University teacher X 1 training Whale Song X 1 What on Earth X 1 What’s Invasive X X 2 What’s the score X 1 Who’s Whoo-ing X 1 Wormwatch X 1 Yardmap X X X 3

Appendix B- funding for the reviewed projects

CoCoRaHS-

At late 2002, CoCoRaHS was selected for funding by the National Science Foundation Informal Science Education program. For the first time, funds became available to hire a project webmaster and volunteer coordinator. In 2003, staff members were hired and an advisory board consisting of members from three adjacent Central Great Plains states was convened. In 2006, CoCoRaHS was the recipient of one of NOAA’s (National Oceanic and Atmospheric Administration) Environmental Literacy grants encouraging outreach and partnership with NOAA offices across the country (Reges et al., 2008).

As grant funding becomes more difficult to obtain for such a large program national in scope, the burden of support is likely to shift from national to local. Local partners and collaborators may become an integral part of project support and funding as well as volunteer recruiting, training and data application (Reges et al., 2008).

Has a long list of sponsors in their site: http://www.cocorahs.org/Content.aspx?page=sponsors eBird-

The majority of the funding to develop eBird was provided by a National Science Foundation award (NSF ESI-0087760). While this initial development of the eBird cyberinfrastructure was a significant investment, the cost per observation is quite low. For example, in 2008 eBird gathered almost 10 million observations of birds. When considering the annual budget for eBird, the cost per observation is three cents. This cost continues to drop as the number of eBird participants increases (Sullivan 2009).

Foldit-

Supported by: UW Center for Game Science, UW Department of Computer Science and Engineering, UW Baker Lab, DARPA, NSF, HHMI, Microsoft, and Adobe.

OPAL-

OPAL was awarded £13 m by the UK Big Lottery Fund (BLF, 2010a) to deliver the programme. Half of the funding goes towards the research programme and half for support services (Davies 2011).

PatientsLikeMe-

PatientsLikeMe website, has a list of four investors groups: CommerceNet, Omidyar Network, Collaborative Seed and Growth Partners, LLC and Invus, LP.

In addition, there is a long list of partners listed in their website: www.patientslikeme.com/about/partners#research

References

Arnstein, S. R. (1969). A ladder of citizen participation. Journal of the America Institute of Planners, 35(4), 224.

Bone, Archer, Barraclough, Eggleton, Flight, Head, et al. (2012). Public participation in soil surveys: Lessons from a pilot study in England. Environmental Science & Technology, 46(7), 3687-3696.

Bonney, R., Ballard, H., Jordan, R., McCallie, E., Phillips, T., Shirk, J., et al. (2009). Public participation in scientific research: Defining the field and assessing its potential for informal science education. A CAISE Inquiry Group Report.

Bonney, R., Cooper, B. C., Dickinson, J., Kelling, S., Phillips, T., Rosenberg, V. K., et al. (2009). Citizen Science: A developing tool for expanding science knowledge and scientific literacy. BioScience, 59(11), 984.

Bonney, R., Shirk, J., Phillips, T. B., & Science, C. (2013). Citizen Science. In Gunstone Richard (Ed.), Encyclopedia of science education (pp. 154) SpringerReference.

Boulos, M. N. K., Resch, B., Crowley, D. N., Breslin, J. G., Sohn, G., Burtner, R., et al. (2011). Crowdsourcing, citizen sensing and sensor web technologies for public and environmental health surveillance and crisis management: Trends, OGC standards and application examples. International Journal of Health Geographics, 10(1), 67.

Cifelli, R., Doesken, N., Keneddy, P., Carey, L., D, Rutledge, S., A, Gimmestad, C., et al. (2005). The community collaborative rain, hail, and snow network. Informal education for scientists and citizens. American meteorological society, 1069- 1077.

Cifelli, R. Doesken, N., and Rutledge, S., (2006) Bringing CoCoRaHS to the Central Great Plains: An Informal Science Education Project for Rural Communities.

Cooper, C. B., Dickinson, J., Phillips, T., & Bonney, R. (2007). Citizen Science as a tool for conservation in residential ecosystems. Ecology and Society, 12(2), 440.

Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., et al. (2010). Predicting protein structures with a multiplayer online game. Nature, 466(7307), 756-60.

Cooper, S., Khatib, F., & Baker. (2013). Increasing public involvement in structural biology. Structure, 21(9), 1484.

CoCoRaHS. 2015. Community Collaborative Rain, Hail and Snow Network [web application]. CoCoRaHS, Colorado Climate Center. Available: www.cocorahs.org. (Accessed: 29/03/2015).

Davies, L., Bell, J. N. B., Bone, J., Head, M., Hill, L., Howard, C., et al. (2011). Open air laboratories (OPAL): A community-driven research programme. Environmental Pollution, 159(8?9), 2210.

Davis, L., Gosling, L., Bachariou, C., Eastwood, J., Fardera, R., Manomaiudom, N., et al. (2013). OPAL community environment report. exploring nature together. London.

Dickinson, J.L., Zuckerberg, B. and Bonter, D.N. 2010. Citizen Science as an Ecological Research Tool: Challenges and Benefits. Annual Review of Ecology, Evolution, and Systematics. 41, 1 (Dec. 2010), 149–172. eBird. 2015. eBird: An online database of bird distribution and abundance [web application]. eBird, Cornell Lab of Ornithology, Ithaca, New York. Available: www.ebird.org. (Accessed: 25/02/2015).

Foldit. 2015. Foldit: solve puzzles for science [web application]. Center for Game Science, University of Washington. Available: www.fold.it/portal (Accessed: 12/03/2015).

Franzoni, C., & Sauermann, H. (2013). Crowd science: The organization of scientific research in open collaborative projects. Research Policy, 43, 20.

Frost, J. H., & Massagli, M. P. (2008). Social uses of personal health information within PatientsLikeMe, an online patient community: What can happen when patients have access to one another’s data. Journal of Medical Internet Research, 10(3)

Galaxy Zoo. 2015. Few have witnessed what you're about to see. [Zooniverse web application]. Available: www.galaxyzoo.org (Accessed: 30/06/2015).

Goodchild, M. (2007). Citizens as sensors: The world of volunteered geography. GeoJournal, 69(4), 221. doi:10.1007/s10708-007-9111-y

Graber, M. A., & Graber, A. (2012). Internet-based crowdsourcing and research ethics: The case for IRB review. Journal of Medical Ethics, 39, 115-118.

Haklay, M. (2013). Citizen Science and volunteered geographic information – overview and typology of participation. In D. Sui, S. Elwood & M. Goodchild (Eds.), Sui, D.Z., elwood, S. and M.F. goodchild (eds.), 2013. crowdsourcing geographic knowledge: Volunteered geographic information (VGI) in theory and practice (pp. 122). Berlin: Springer Netherlands.

Irwin, A. (1995). Citizen Science: A study of people, expertise, and sustainable development. Routledge.

Lagoze. (2014). eBird: Curating Citizen Science data for use by diverse communities. International Journal of Digital Curation, 9(1).

Lintott, C.G., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., . . Vandenberg, J. (2008). Galaxy zoo: Morphologies derived from visual inspection of galaxies from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society, 389(3), 1179- 1189.

OPAL. 2015. OPAL- Open Air Laboratories [web application]. OPAL Centre for Environmental Policy Imperial College London. Available: www.opalexplorenature.org. (Accessed: 25/03/2015).

PatientsLikeMe. 2015. PatientsLikeMe, making healthcare better for everyone through sharing, support and research [web application]. PatientsLikeMe Inc. Cambridge, MA. Available: www.patientslikeme.com (Accessed: 15/06/2015).

Raddick, M. J., Bracey, G., Gay, P. L., Lintott, C. J., Cardamone, C., Murray, P., . . . Vandenberg, J. (2013). Galaxy Zoo: Motivations of Citizen Scientists . Astronomy Education Review, 12(1), 27.

Raddick, M. J., Bracey, G., Gay, P. L., Lintott, C. J., Murray, P., Schawinski, K., . . . Vandenberg, J. (2010). Galaxy zoo: Exploring the motivations of Citizen Science volunteers. Astronomy Education Review, 9(1), 18.

Roy, H. E., Pocock, M. J. O., Preston, C. D., Roy, D. B., & Savage, J. (2012). Understanding Citizen Science and environmental monitoring. Final Report on Behalf of UK-EOF. NERC Centre for Ecology & Hydrology and Natural History Museum,

Science Communication Unit, U. of the W. of and England, B. 2013. Science for Environment Policy IN-DEPTH REPORT : Environmental Citizen Science. Repord produced for the European Commision DG Environment. 9 (2013).

Sheth, A. (2009). Citizen sensing, social signals, and enriching human experience. IEEE Internet Computing, 13(4), 92.

Silvertown, J. 2009. A new dawn for Citizen Science. Trends in ecology & evolution. 24, 9 (Sep. 2009), 467–71.

Sullivan, B. L., Wood, C. L., Iliff, M. J., Bonney, R. E., Fink, D., & Kelling, S. (2009). eBird: A citizen-based bird observation network in the biological sciences. Biological Conservation, 142(10), 2292.

Wicks, P., Massagli, M., Frost, J., Brownstein, C., Okun, S., Vaughan, T., Heywood, J. (2010). Sharing health data for better outcomes on PatientsLikeMe. Journal of Medical Internet Research, 12(2)

Wiggins, A. and Crowston, K. 2011. From Conservation to Crowdsourcing: A Typology of Citizen Science. 2011 44th Hawaii International Conference on System Sciences. (Jan. 2011), 1–10.

Wilderman, C. C. (2007). Models of community science: Design lessons from the field. In R. B. C. McEver, & J. Dickinson, S. Kelling, K. Rosenberg, and J. L. Shirk (Eds.), Citizen Science toolkit conference. Ithaca, NY: Cornell Laboratory of Ornithology.