technology feature

Finding the crowd 1070 Testing, gaming 1072 Folding as play 1073

Neuroscience waves to the crowd Vivien Marx Researchers scale up how to crowdsource the mapping of neural circuits. These projects entice crowds by tapping into their spirit of play.

Nonscientists can help with sophisticated ahead for now. “Unusual means may get us images of brain slices and to turn the two- neurobiology challenges. Researchers will- to some really cool goals,” he says. dimensional (2D) image information into ing to follow in the tracks of labs pioneer- Being ‘bold’ and ‘ambitious’ is now fash- connectivity maps of how the run, ing this approach need to know that work- ionable in , as illustrated by branch and connect throughout a section of ing with crowds means more than sharing large-scale efforts to map the human brain the brain. EyeWire is the father of crowd- neuroscience and computing knowledge. and accelerate technology development sourcing in this field, a gaming project initi- Initiating and managing peopled projects in neuroscience—such as the Brain ated by Seung and the first to “crowdsource takes stamina, management the ,” he says. expertise, unconventional The online project invites recruiting methods and slight- volunteers to compete against ly more-than-healthy amounts one another to map the neu- of pizza and caffeinated soft rons in a region of the mouse drinks. . The goal is to better In 2009, when neurosci- understand how visual infor- ence crowdsourcing ventures mation is processed in the began, skepticism reigned brain and to help train com- about the prospect of map- puters to get better at this kind ping neurons of entire cir- of tracing. cuits; crowdsourcing this chal- Around 82,000 partici-

© 2013 Nature America, Inc. All rights reserved. America, Inc. © 2013 Nature lenge to interested volunteers pants, who are almost all seemed to arouse even more citizen scientists as opposed skepticism. “That is a good to formally trained scientists, sign. If you’re doing something have played the game so far. npg new, there should be measured As EyeWire’s creative direc-

skepticism—otherwise it is Neurobiology MPI for tor Amy Robinson says, their not new enough”, says Moritz Neuroscientist Moritz Helmstaedter (middle) and Brainflight programmers want surveys show that ‘EyeWirers’ Helmstaedter, a neuroscien- crowdsourced neuroscience to be fun. range from high school stu- tist at the Max Planck Institute dents to retirees and hail (MPI) of Neurobiology. He from basically all walks of life. and , a computational Research through Advancing Innovative Some players are researchers, but they are neuroscientist at Massachusetts Institute Neurotechnologies Initiative launched by mainly just people interested in science, says of Technology (MIT), and a few other neu- US President Barack Obama. Anatomical Seung. The top players, he says, are a mix roscientists are crowdsourcing in connec- mapping of the brain’s connections is one of of men and women. There is even a slight tomics, a field in which researchers seek to the initiative’s top priorities as described in skew toward more female players among generate and analyze detailed anatomical the first report recently issued to the direc- EyeWirers. “It’s pretty hard to predict who maps of entire nervous systems. tor of the US National Institutes of Health. will like this game.” As projects have drawn crowds and gen- Separately, scientists have found that The project takes two long-standing neu- erated publications, researchers now face crowdsourcing is a promising way to analyze roscience traditions of manual labor into the challenges in scaling up the ways crowds increasingly large data sets generated in labs realm of computer games. One of these is follow neurons through a stack of electron that perform neuroanatomical mapping. skeletonization, in which researchers trace microscopy (EM) images. The ability to per- the likely paths of neurons through stacks form this tracing is “a key bottleneck right EyeWire of EM images, he says. The other is one in now in ,” says Helmstaedter. Crowds in neuroscience are already helping which scientists trace and render the neu- Crowdsourcing is the only way to move researchers to analyze high-resolution EM rons and their paths in three dimensions,

nature methods | VOL.10 NO.11 | NOVEMBER 2013 | 1069 technology feature

precisely outlining every ’s contour Seung is happy about the success of the and curve. project thus far in garnering so many par- Performed with “human intelligence,” ticipants for the game and the competitions. both these tasks done manually are pains- He acknowledges that his project’s players takingly slow, says are far fewer than the millions of players Seung. His idea has reached with games unrelated to science. been to accelerate Laughing, he points out that the game the process with Angry Birds has 2 billion downloads across many volunteers all platforms. “If you compare EyeWire with competing with a commercial game, it looks kind of pathet- one another and ic,” he says. “It needs to be much more fun, by harnessing arti- and that’s a challenge.”

K. Krug/PopTech ficial intelligence. It is hard to recruit game developers to Sebastian Seung’s In the lab, his work on neuroscience. Helmstaedter—who EyeWire is the father team uses software is also trying to use gaming to grow the scale of crowdsourcing in called Omni to of neural circuit tracing in his lab—takes connectomics. render neurons in unusual approaches to recruiting potential three dimensions. programmers. EyeWire uses a slightly simplified version of Omni that lets players see one 3D and Finding the crowd one 2D view. In 2009, Helmstaedter and some students There is also a trained convolutional neural positioned themselves at the bottom of network at the heart of EyeWire, the code for the staircase leading to the cafeteria of the which is part of the paper Seung coauthored University of Heidelberg. A large projec- with Helmstaedter and others, in which the tion screen showed what one student was team presented a connectomic reconstruc- doing on a computer: tracing neurons in tion of a layer of the mouse retina1. EyeWire EM images. taps into the crowd to follow the path of neu- “It got so much attention,” Helmstaedter rons and perform volume reconstruction says of the recruiting session for his neuron- of neurons and the circuits to which they tracing project. People stopped, asked ques- belong. “It’s like coloring in a neuron with the tions, picked up flyers. Excitement mount- help of the computer,” Seung says. ed, and the crowd of his volunteers grew its Separately, he and his team also run algo- ranks to the list of 224 names that accom- rithm development competitions. “We put a pany his latest Nature paper, completed with

© 2013 Nature America, Inc. All rights reserved. America, Inc. © 2013 Nature data set online, and we let people try to sub- Seung, Winfried Denk from the Max Planck mit their algorithms to reconstruct from it, Institute for Medical Research and research- and we score them and we see who the win- ers in the UK, Germany and the USA1. ner is,” Seung says. The plan for the future is The recruited crowd was made up of npg to feed this algorithmic knowledge gained paid undergraduate and graduate students. back into EyeWire. Applying a software tool called KNOSSOS2 EyeWire, MIT/Seung lab MIT/Seung EyeWire, Around 82,000 participants have played EyeWire thus far.

1070 | VOL.10 NO.11 | NOVEMBER 2013 | nature methods technology feature

members. After ing company, in which he learned to fos- some training ses- ter an environment with a group “driven sions, the crowds set by the goal, not by hierarchies,” he says. out to establish con- Helmstaedter then switched back to science nectivity between and completed an MD-PhD; he has conse- neurons and anno- quently brought his management experi- tate the microscopy ence to neuroscience crowdsourcing. data. With the skel- For the volunteers to begin their neu- etons, Helmstaedter ral reconstruction tasks, scientists pre- was then able to selected points in the EM images where apply the machine the volunteers were to start tracing. Then,

EyeWire, MIT/Seung lab MIT/Seung EyeWire, learning algorithm Helmstaedter says, the volunteers followed used in EyeWire to a neurite for as long as they could, and the Volunteers compete to map and reconstruct neurons in EyeWire. “blow up the skel- results were returned to the research team etons into full vol- for evaluation. umes,” says Seung. Scientists could developed at the MPI, they helped to recon- Helmstaedter believes that anyone can cross-check the struct the circuit of 950 neurons in a layer of run a crowdsourcing project, but the project delivered results the mouse retina called the inner plexiform leaders will need to fully embrace it. After because multiple that had been imaged through serial block- finishing his master’s degree in physics, students were face EM. “KNOSSOS is made to ‘fly along’ Helmstaedter spent a few months working assigned the same neurites” to reconstruct skeletons of neurons, for a consulting company, which offered neuron, he says. Helmstaedter says. “EyeWire ‘paints’ neu- “great experience in how to manage very MPI Neurobiology More than 20,000 rons,” generating volume reconstructions. large teams, how to motivate people,” he says. “A lot of pizza was annotator hours He distributed hard disks of EM data sets Frequent meetings and benchmarking involved,” says Moritz led to 2.6 meters of Helmstaedter. and the KNOSSOS software to all the team were part of the project for the consult- skeleton. © 2013 Nature America, Inc. All rights reserved. America, Inc. © 2013 Nature npg

nature methods | VOL.10 NO.11 | NOVEMBER 2013 | 1071 technology feature

Testing, gaming Dominic Bräunlein is testing several games—all in the prototype stage—that are intended to help annotate neurosci- ence data. His start-up, named scalable minds, is programming the games in a team Helmstaedter assembled. He encouraged them to start a company, doing so also as a strategy to recruit software engineers to neuroscience. Bräunlein founded the firm with four oth-

Thinkstock ers in late 2011; it is part of a project called Brainflight. The other company found- It is not easy to recruit game developers to work ers—Tom Bocklisch, Tom Herold, Norman on neuroscience. Rzepka, and Thomas Werkmeister—were Bräunlein’s classmates and friends at the Human brains are better than machines computer science–focused Hasso Plattner at seeing the less distinct boundaries Institute, where they recently completed between neurons. “We’re very good at software engineering degrees. figuring out those difficult occasions; The group is tight-lipped about the actu- however, we’re very bad at being always al plots of the neuroscience games under and constantly attentive,” he says. As he development. But Bräunlein shares that they followed the crowd’s progress, he saw that are not shoot-up or kill-the-alien games the attention span and exhaustion limit and that they appeal to different age groups. for neural tracing was around 4 hours, “One idea is more quiet and more for older including breaks. persons, more like a puzzle,” Bräunlein says. Attention-related errors occur when, for “The other is more like a typical 3D game example, volunteers miss a neurite branch. that appeals to guys in my age group or teen- “It’s rather related to your personal caffeine age boys,” he says. “Or girls,” he adds. intake on that day or level of distraction you With trepidation, Bräunlein recently have related to the data,” he says. showed a middle-aged woman a newly Because the errors are attention related designed computer game. She didn’t like it. and not data related, a second person trac- And so the software engineer had to go back ing the same neurite is less likely to make to the drawing board, and his team had to

© 2013 Nature America, Inc. All rights reserved. America, Inc. © 2013 Nature the same mistake. This pattern allowed rethink the game—as he has done plenty of the scientists to average out the mistakes times before with other volunteer testers. by comparing multiple students’ tracings. Building software is tough, but turning A software tool called the redundant- a scientific problem into game play is per- npg skeleton consensus procedure (RESCOP), haps harder, says Helmstaedter. The goal also developed in the lab, was used to rec- is to imagine an app that people play while oncile the students’ tracings. waiting at a bus stop and in that time “help To encourage the crowds, every us massively solve our big reconstruction 2–3 months Helmstaedter met with all problems.” the members to discuss progress. “A lot of Bräunlein says that Brainflight wants to pizza was involved,” he says. They validated take a more playful approach than EyeWire results in the group setting. “We wouldn’t to crowdsource tracing. “People need to tell them ‘you were wrong’,” he says. The play it without feeling like they are doing group of peers found errors together and good work for scientists,” he says. “My per- addressed them together to avoid embar- sonal goal is to have it feel like you would rassing individual members. want to play it irrespective of the science- “What Moritz did is a highly significant related task.” innovation, which is to demonstrate you Seung looks forward to the Brainflight can use a lot of people’s effort,” says Seung. games and acknowledges the friendly com- And now Helmstaedter is scaling his crowd petition. But he also teases the project devel- and tools further by embedding the sci- opers about the fact that their games have ence and the tracing activity into a gam- not yet been released. “There is plenty of ing approach that is intended to be unlike room for scalable minds or for many other EyeWire. But it takes testing, testing and people to come in,” he says. No group has yet more testing to turn science into a game. scored a chart-breaking hit.

1072 | VOL.10 NO.11 | NOVEMBER 2013 | nature methods technology feature

As a contractor for Helmstaedter’s lab, kind of local forecasting, looking at both surprised him to see “just how much the the start-up is also helping build a scalable the direction and speed with which the general public really wants scientists to suc- version of the annotation tool KNOSSOS, analysis is likely to unfold by an individual ceed and would like to help us, either for the called Oxalis, so that the software can be using the software. advancement of science or just the joy and used by crowds working through a Web thrill of participating in it.” browser. With this step, the crowdsourcing Folding as play Many technical challenges have to be project takes a tool that was formerly used Neuroscientists interested in crowdsourcing tackled to crowdsource projects, often by scientists and makes it possible for non- can look to the success of similar projects in ways that are specific to a given scien- scientists to use it. And in this next phase in other research disciplines. Galaxy Zoo tific field. But scientists generally need to the crowds will no longer need to have the lets participants help classify galaxies in become more comfortable with crowd- EM image data on a hard disk. telescope images from the Sloan Digital Sky sourced solutions, says Pande. In the long The plan is to make Oxalis freely available Survey. Folding@home began at Stanford term that approach will make it easier for for researchers, and it might be open source. University and has allowed researchers to other researchers to use these methods, too. The group is still making those decisions. tap into over 260,000 computers of volun- Internally, he says, reengineering the One software challenge facing the teers to explore how proteins fold. The proj- scientific problem and the software at the team is that the data are big, terabyte- ect also includes a crowdsourced protein- heart of Folding@home in order to scale it sized images. As a consequence the soft- folding game, , in which participants has been a challenge. But he and many of his ware cannot place too much data into the solve puzzles and fold proteins for points colleagues like to “think as big as we can, so browser at any one time. Bräunlein and and thereby help scientists to predict protein scaling up is always the most exciting part his colleagues engineered the software to folding or design proteins3. of all of this.” anticipate users’ movements through the Folding@home’s initiator is Stanford The software development for this proj- images as they annotate. “We load small University chemist Vijay Pande, who is ect has all been performed at Stanford, but cubes of data,” he says. That way no one happy to see crowdsourcing ventures in “we’ve moved to employing more and more stares at a blank screen because the data neuroscience. “I think there will be more professional software engineering prac- are still loading. The software performs a and more successes like these,” he says. It tices,” he says. “Company connections have © 2013 Nature America, Inc. All rights reserved. America, Inc. © 2013 Nature npg

nature methods | VOL.10 NO.11 | NOVEMBER 2013 | 1073 technology feature

been very helpful to get Folding@home into ity. “The burden,” he says, is on the scien- “That device will allow us to speed that people’s hands.” tists using these methods “to ensure these up 60-fold, roughly, which will be more His project has many nonprofit, gov- are not concerns.” like three terabytes an hour,” he says. But, ernment and commercial supporters, Crowdsourcing in neuroscience might he adds, and laughs as he speaks, as these and he sees a number of ways for crowd- become more common as data sets grow data set dimensions become common, he sourced science projects to grow. It might in size and as the neuroscience commu- wonders whether “there are enough peo- begin in a lab and involve companies nity launches large-scale projects. In a lab, ple out there” to crowdsource the analy- later. “Bringing in companies is a bit more current image data sets can be hundreds sis of such large data sets. Researchers in unorthodox, but they can help scale up, of terabytes large—but they are bound to neuroscience and the crowds they recruit too,” says Pande. get larger. Harvard University neurobiolo- will have their work cut out for them. Similarly to the neuroscience proj- gist Jeff Lichtman likes the idea of crowd- ects, Folding@home also originally faced sourcing image analysis. 1. Helmstaedter, M. et al. Nature 500, 168–174 (2013). skepticism because crowdsourcing is an He and Winfried Denk are collaborat- 2. Helmstaedter, M., Briggman, K.L. & Denk, W. unusual approach in the often conserva- ing with microscope manufacture Zeiss as Nat. Neurosci. 14, 1081–1088 (2011). tive world of science. And there is good the company develops a prototype scan- 3. Cooper, S. et al. Nature 466, 756–760 (2010). reason for the conservatism, he says, ning electron microscope that uses 61 because “the integrity of the results is electron beams on a sample, as opposed paramount.” That standard means more to the typical single electron beam. Vivien Marx is technology editor for complexities for crowdsourced projects, Currently, the Lichtman lab generates Nature and Nature Methods. such as the need to address reproducibil- around a terabyte of image data per day. ([email protected]). © 2013 Nature America, Inc. All rights reserved. America, Inc. © 2013 Nature npg

1074 | VOL.10 NO.11 | NOVEMBER 2013 | nature methods