WHO WE ARE WHAT WE CAN DO WHERE WE’RE HEADED

BEYOND LIMITS

neuroengineering Mind in ble for translating electrical signals into digital motor commands so that the Motion exoskeleton can first stabilize the kick- The idea that paralyzed people er’s body weight and then induce the ro- botic legs to begin the back-and-forth co- might one day control their limbs ordinated movements of a walk over the manicured grass. Then, on approaching just by thinking is no longer the ball, the kicker will visualize placing a foot in contact with it. Three hundred a Hollywood-style fantasy milliseconds later brain signals will in- struct the exoskeleton’s robotic foot to By Miguel A. L. Nicolelis hook under the leather sphere, Brazilian style, and boot it aloft. This scientific demonstration of a radi- cally new technology, undertaken with collaborators in Europe and , will convey to a global audience of billions that brain control of machines has moved from lab demos and futuristic speculation to a new era in which tools capable of bringing mobility to patients incapacitated by inju- in 2014 billions of viewers worldwide may ry or disease may become a reality. We are remember the opening game of the on our way, perhaps by the next decade, to World Cup in Brazil for more than just technology that links the brain with me- the goals scored by the Brazilian national chanical, electronic or virtual machines. team and the red cards given to its adver- This development will restore mobility, sary. On that day my laboratory at Duke not only to accident and war victims but University, which specializes in develop- also to patients with ALS (also known as ing technologies that allow electrical sig- Lou Gehrig’s disease), Parkinson’s and oth- nals from the brain to control robotic er disorders that disrupt motor behaviors limbs, plans to mark a milestone in over- that impede arm reaching, hand grasping, coming paralysis. locomotion and speech production. Neu- If we succeed in meeting still formi- roprosthetic devices—or brain-machine dable challenges, the first ceremonial interfaces—will also allow scientists to do kick of the World Cup game may be made much more than help the disabled. They by a paralyzed teenager, who, flanked by will make it possible to explore the world the two contending soccer teams, will in revolutionary ways by providing saunter onto the pitch clad in a robotic healthy human beings with the ability to body suit. This suit—or exoskeleton, as augment their sensory and motor skills. we call it—will envelop the teenager’s In this futuristic scenario, voluntary legs. His or her first steps onto the field electrical brain waves, the biological al- will be controlled by motor signals origi- phabet that underlies human thinking, nating in the kicker’s brain and transmit- will maneuver large and small re- ted wirelessly to a computer unit the size motely, control airships from afar, and of a laptop in a backpack carried by our perhaps even allow the sharing of patient. This computer will be responsi- thoughts and sensations of one individu-

58 Scientific American, September 2012 Illustration by Kemp Remillard September 2012, ScientificAmerican.com 59 WHO WE ARE WHAT WE CAN DO WHERE WE’RE HEADED

BEYOND al with another over what will become a feel what it was like to touch an object LIMITS collective brain-based network. with virtual fingers controlled directly by their brain. Thought Machines The Walk Again consortium—assisted the lightweight body suit intended for by its international team of neuroscien- the kicker, who has not yet been selected, tists, roboticists, computer scientists, is still under development. A prototype, neurosurgeons and rehabilitation profes- Miguel A. L. Nicolelis has pioneered though, is now under construction at the sionals—has begun to take advantage of the field of . He lab of my great friend and collaborator these animal research findings to create a is Duke School of Medicine Professor Gordon Cheng of the Technical Universi- completely new way to train and rehabili- of and co-director of the Duke University Center for ty of Munich—one of the founding mem- tate severely paralyzed patients in how to Neuroengineering. bers of the Walk Again Project, a nonprof- use brain-machine interface technologies it, international collaboration among the to regain full-body mobility. Indeed, the Duke University Center for Neuroengi- first baby steps for our future ceremonial neering, the Technical University of Mu- kicker will happen inside an advanced nich, the Swiss Federal Institute of Tech- virtual-reality chamber known as a Cave nology in Lausanne, and the Edmond Automatic Virtual Environment, a room and Lily Safra International Institute of with screens projected on every wall, in- of Natal in Brazil. A few cluding the floor and ceiling. After don- new members, including major research ning 3-D goggles and a headpiece that institutes and universities all over the will noninvasively detect brain waves world, will join this international team in (through techniques known as electroen- the next few months. cephalography—EEG—and magnetoen- The project builds on nearly two de- cephalography), our candidate kicker— cades of pioneering work on brain-ma- by necessity a lightweight teenager for chine interfaces at Duke—research that this first iteration of the technology—will itself grew out of studies dating back to become immersed in a virtual environ- the 1960s, when scientists first attempted ment that stretches out in all directions. to tap into animal to see if a neural There the youngster will learn to control signal could be fed into a computer and the movements of a software body avatar thereby prompt a command to initiate through thought alone. Little by little, the motion in a mechanical device. Back in motions induced in the avatar will in- 1990 and throughout the first decade of crease in complexity and will ultimately this century, my Duke colleagues and I pi- end with fine-motor movements such as oneered a method through which the walking on a changing terrain or un- brains of both rats and monkeys could be screwing a virtual jelly jar top. implanted with hundreds of hair-thin and flexible sensors, known as micro­ Plugging into wires. Over the past two decades we have the mechanical movements of an exo- shown that, once implanted, the flexible skeleton cannot be manipulated as readi- In Brief electrical prongs can detect minute elec- ly as those of a software avatar, so the trical signals, or action potentials, gener- technology and the training will be more Brain waves can now control the ated by hundreds of individual neurons complicated. It will be necessary to im- functioning of computer cursors, distributed throughout the animals’ fron- plant electrodes directly in the brain to robotic arms and, soon, an entire suit: manipulate the robotic limbs. We will an exoskeleton that will allow tal and parietal cortices—the regions that a paraplegic to walk and maybe define a vast brain circuit responsible for need not only to place the electrodes un- even move gracefully. the generation of voluntary movements. der the skull in the brain but also to in- Sending signals from the brain’s This interface has for a full decade crease the number of neurons to be outer rindlike cortex to initiate used brain-derived signals to generate “read” simultaneously throughout the movement in the exoskeleton movements of robotic arms, hands and cortex. Many of the sensors will be im- represents the state of the art legs in animal experiments. A critical planted in the motor cortex, the region of for a number of bioelectrical breakthrough occurred last year when two the frontal lobe more readily associated technologies perfected monkeys in our lab learned to exert neural with the generation of the motor pro- in recent years. The 2014 World Cup in Brazil will control over the movements of a comput- gram that is normally downloaded to the serve as a proving ground for a er-generated avatar arm that touched ob- spinal cord, from which neurons directly brain-controlled exoskeleton if, as jects in a virtual world but also provided control and coordinate the work of our expected, a handicapped teenager an “artificial tactile” feedback signal di- muscles. (Some neuroscientists believe delivers the ceremonial opening kick. rectly to each monkey’s brain. The soft- that this interaction between mind and ware allowed us to train the animals to muscle may be achieved through a nonin-

60 Scientific American, September 2012 chronology vasive method of recording brain activity, like EEG, but that goal has yet to be prac- tically achieved.) Gary Lehew in my group at Duke has The Long Road to devised a new type of sensor: a recording cube that, when implanted, can pick up Brain-Controlled Prosthetics signals throughout a three-dimensional volume of cortex. Unlike earlier brain sen- Replacement limbs have existed for millennia—a rational response to the need sors, which consist of flat arrays of micro- to address war wounds or other types of trauma and birth defects. Today the electrodes whose tips record neuronal technology is so sophisticated that an artificial limb can be controlled by electrical electrical signals, Lehew’s cube extends signals channeled directly from the brain. sensing microwires up, down and sideways

throughout­ the length of a central shaft. 1500–1000 B.C. The current version of our recording FIRST HISTORICAL REFERENCE cubes contains up to 1,000 active record- A Hindu holy book written during this period ing microwires. Because at least four to mentioned Vishpala, who had a leg amputation after six single neurons can be recorded from a wound sustained during battle. She had the limb replaced with an iron version that let her walk and each microwire, every cube can poten- return to her troops. tially capture the electrical activity of be- tween 4,000 to 6,000 neurons. Assuming that we could implant several of those Fourth Century B.C. cubes in the frontal and parietal corti- Ancient artifact ces—areas responsible for high-level con- One of the oldest artificial limbs discovered—a copy trol of movement and decision making— of which is shown here—was dug up in southern Italy in 1858. Fabricated in about 300 B.C., it was made of we could obtain a simultaneous sample copper and wood and designed, it appears, for a below- of tens of thousands of neurons. Accord- knee amputee. ing to our theoretical software modeling, this design would suffice for controlling the flexibility of movement required to 14th Century operate an exoskeleton with two legs and GUNS AND AMPUTATIONS The arrival of gunpowder at the European battlefront to restore autonomous locomotion in our greatly amplified the number of injuries sustained by patients. soldiers. In response, in the 16th century Ambroise To handle the avalanche of data from Paré, the royal surgeon for several French kings, these sensors, we are also moving ahead developed techniques to attach both upper and lower on making a new generation of custom- limbs to patients and reintroduced the use of ligatures designed neurochips. Implanted in a pa- to tie off blood vessels. tient’s skull along with the microelec- trodes, they will extract the raw motor 1861–1865 Civil War commands needed to manipulate a whole- The War between the States resulted in many amputations. body exoskeleton. One person affected was Brigadier General Of course, the signals detected from Stephen Joseph McGroarty, who lost the brain will then need to be broadcast to an arm. An influx of government funding and the avail­ the prosthetic limbs. Recently Tim Han- ability of anesthetics that allowed for longer operations improved prosthetic technology during this era. son, a newly graduated Ph.D. student at Duke, built a 128-channel wireless record- ing system equipped with sensors and 1963 PRIMITIVE BRAIN INTERFACE

) officer War Civil chips that can be encased in the cranium José Manuel Rodriguez Delgado implanted a radio- and that is capable of broadcasting record- controlled electrode in the caudate nucleus deep or b i s (

; C ed brain waves to a remote receiver. The in a bull’s brain and stopped the animal dead in its tracks by pressing a button on a remote transmitter; first version of these neurochips is cur- his device was a predecessor to contemporary brain- rently being tested successfully in mon- machine interfaces. keys. Indeed, we have recently witnessed ) artificial leg and Paré the first monkey to operate a brain-ma- rary ( rary chine interface around the clock using 1969 PIONEERING EXPERIMENTS wireless transmission of brain signals. We Eberhard Fetz of the University of Washington performed a study in which monkeys were trained filed in July with the Brazilian govern- to activate electrical signals in their brain to control ment for permission to use this technolo- the firing of a single , duly recorded by gy in humans. a metal microelectrode.

Science & Society Picture & Society Li b Science For our future soccer ball kicker, the

September 2012, ScientificAmerican.com 61 WHO WE ARE WHAT WE CAN DO WHERE WE’RE HEADED

BEYOND chronology data from the recording systems will be LIMITS relayed wirelessly to a small computer 1980s processing unit contained in a backpack. LISTENING TO BRAIN WAVES Apostolos Georgopoulos of Johns Hopkins University Multiple digital processors will run vari- discovered an electrical firing pattern in the motor ous software algorithms that translate neurons of rhesus macaques that occurred when they motor signals into digital commands rotated their arm in a particular direction. that are able to control moving parts, or actuators, distributed across the joints of the robotic suit, hardware elements that Early 1990s PLUGGING IN John Chapin, now at S.U.N.Y. Downstate University, adjust the positioning of the exoskele- and Miguel A. L. Nicolelis introduced a technique ton’s artificial limbs. that allowed for simultaneous recording of dozens of widely dispersed neurons using permanently force of brainpower implanted electrodes, thus paving the way for the commands will permit the exoskele- research on brain-machine interfaces. ton wearer to take one step and then an- other, slow down or speed up, bend over 1997 BETTER MOVES or climb a set of stairs. Some low-level ad- The microprocessor-controlled C-Leg knee justments to the positioning of the pros- prosthesis, which in its current version allows thetic hardware will be handled directly the wearer to turn on customized settings that can be by the exoskeleton’s electromechanical used for activities such as bicycling, was introduced. circuits without any neural input. The space suit–like garment will remain flexi- 1999–2000 GOOD FEEDBACK ble but still furnish structural support to The Chapin and Nicolelis laboratories published its wearer, a surrogate for the human spi- the first description of a brain-machine interface nal cord. By taking full advantage of this operated by activity from rat brains, whereby interplay between brain-derived control the animals sensed the movement through a visual feedback signal. The following year the signals and the electronic reflexes sup- Nicolelis lab published the first study in which a plied by the actuators, we hope that our monkey controlled the movements of a robotic brain-machine interface will literally car- arm using only brain activity. ry the World Cup kicker along by force of willpower. The kicker will not only move but also 2008–2011 BLADE RUNNER feel the ground underneath. The exoskel- After failing to qualify for the 2008 Summer Olympics Games, Oscar Pistorius eton will replicate a sense of touch and swept the 2008 Summer Paralympic Games and balance by incorporating microscopic then got to the 400-meter semifinals at the 2011 sensors that both detect the amount of International Association of Athletics Federations force from a particular movement and World Championships in Daegu, South Korea. convey the information from the suit back to the brain. The kicker should be able to feel that a toe has come in contact 2011 MONKEY THINK, AVATAR DO with the ball. Nicolelis’s team at the Duke University Center for Neuroengineering demonstrated that a Our decade-long experience with brain-

monkey was able to use thoughts to manipulate machine interfaces suggests that as soon the movements of a software avatar. as the kicker starts interacting with this ) BrainGate ( exoskeleton, the brain will start incorpo- ); Pistorius rating this robotic body as a true exten- AP Photo ( Photo AP 2012 FROM MY BRAIN sion of his or her own body image. From TO MY ARM training, the accumulated experience ob- b raingate2.org y o f y m e d ic h ini John Donoghue of Brown University and his s w colleagues showed with their BrainGate tained from this continuous feeling of courte neural interface system that contact with the ground and the position an d re a subject with a could manipulate of the robotic legs should enable move-

a robotic arm to pick up a drink. artificial leg );

ment with fluid steps over a soccer pitch ( ); avatar monkey (

or down any sidewalk. All phases of this i s project require continuous and rigorous t h care 2014 OPENING KICK testing in animal experiments before we l e nico l . a. The Nicolelis lab intends to provide an h ea l ock begin in humans. In addition, all proce- exoskeleton for a handicapped teenager otto b otto l m igue to make the first kick of the opening event dures must pass muster with regulatory y o f y y o f y s of the World Cup in Brazil. agencies in Brazil, the U.S. and Europe to s courte ensure proper scientific and ethical re- courte

62 Scientific American, September 2012 view. Even with all the uncertainties in- should take. Idoya continued to produce volved and the short time required for the brain patterns required to make CB1 the completion of its first public demon- walk even though her own body was no stration, the simple idea of reaching for longer engaged in this motor task. This such a major milestone has galvanized transcontinental brain-machine inter- Brazilian society’s interest in science in face demonstration revealed that it is ways rarely seen before. possible for a human or a simian to read- ily transcend space, force and time by Remote Control liberating brain-derived commands from the opening kickoff of the World Cup—or the physical limits of the biological body a similar event, say, the 2016 Olympic that houses the brain and broadcasting and Paralympic Games in Rio de Janeiro, them to a man-made device located far if we miss the first deadline for any rea- from the original thought that generated son—will be more than just a one-time the action. stunt. A hint of what may be possible These experiments imply that brain- with this technology can be gleaned from machine interfaces could make it possi- a two-part experiment already complet- ble to manipulate robots sent into envi- ed with monkeys. As a prelude, back in ronments that a human will never be able 2007, our research team at Duke trained to penetrate directly: our thoughts might rhesus monkeys to walk upright on a operate a microsurgical tool inside the treadmill as the electrical activity of body, say, or direct the activities of a hu- more than 200 cortical neurons was re- manoid worker trying to repair a leak at a corded simultaneously. Meanwhile Gor- nuclear plant. don Cheng, then at ATR Intelligent Ro- The interface could also control tools botics and Communication Laboratories that exert much stronger or lighter forces in Kyoto, built an extremely fast Internet than our bodies can, thereby breaking free protocol that allowed us to send this of ordinary constraints on the amount of stream of neuronal data directly to Kyo- force an individual can exert. Linking a to, where it fed the electronic controllers monkey’s brain to a humanoid robot has of CB1, a humanoid robot. In the first already done away with constraints im- half of this across-the-globe experiment, posed by the clock: Idoya’s mental trip Cheng and my group at Duke showed around the globe took 20 milliseconds— that the same software algorithms devel- less time than was required to move her oped previously for translating thoughts own limb. into control of robotic arms could also Along with inspiring visions of the far convert patterns of neural activity in- future, the work we have done with mon- volved in bipedal locomotion to make keys gives us confidence that our plan two mechanical legs walk. may be achievable. At the time of this The second part of the experiment writing, we are waiting to see whether yielded a much bigger surprise. As one of the International Football Association our monkeys, Idoya, walked on the tread- (FIFA), which is in charge of organizing mill in Durham, N.C., our brain-machine the ceremony, will grant our proposal to interface broadcast a constant stream of have a paraplegic young adult participate more to explore her brain’s electrical activity through in the opening ceremony of the inaugural Controlling Robots with the Mind. Cheng’s Internet connection to Kyoto. game of the 2014 World Cup. The Brazil- Miguel A. L. Nicolelis and John K. There CB1 detected these motor com- ian government—which is still awaiting Chapin in Scientific American,Vol. 287, No. 4, pages 46–53; October 2002. mands and began to walk as well, almost FIFA’s endorsement—has tentatively sup- Cortical Control of a Prosthetic immediately. CB1 first needed some sup- ported our application. Arm for Self Feeding. Meel Velliste port at the waist, but in later experi- Bureaucratic difficulties and scientific et al. in Nature, Vol. 453, pages 1098– ments it began to move autonomously in uncertainties abound before our vision 1101; June 19, 2008. response to the brain-derived commands is realized. Yet I cannot stop imagining Beyond Boundaries: The New Neu- roscience of Connecting Brains with generated by the monkey on the other what it will be like during the brief but Machines—and How It Will Change side of the globe. historic stroll onto a tropical green soccer Our Lives. Miguel Nicolelis. St. Mar- What is more, even when the tread- pitch for three billion people to witness a tin’s Griffin, 2012. mill at Duke stopped and Idoya ceased paralyzed Brazilian youth stand up, walk Scientific American walking, she could still control CB1’s leg again by his or her own volition, and ulti- Online Inspect an exoskeleton movements in Kyoto by merely observ- mately kick a ball to score an unforgetta- prototype at Scientific­American. ing the robot’s legs moving on a live vid- ble goal for science, in the very land that com/sep2012/exoskeleton eo feed and imagining each step CB1 mastered the beautiful game.

September 2012, ScientificAmerican.com 63