TECHNOLOGY QUARTERLY January 6th 2018 -COMPUTER INTERFACES

Thought experiments

20180106_TQbrain.indd 1 06/12/2017 10:24 TECHNOLOGY QUARTERLY Brain-computer interfaces

Brains and machines Thought experiments

Brain-computer interfaces sound like the stuff of science fiction. Andrew Palmer sorts the reality from the hype N THE gleaming facilities of the Wyss Centre A channel of communication of this sort requires a for Bio and Neuroengineering in Geneva, a lab brain-computer interface (BCI). Such things are al- technician takes a well plate out of an incuba- ready in use. Since 2004, 13 paralysed people have ALSO IN THIS TQ tor. Each well contains a tiny piece of brain tis- been implanted with a system called BrainGate, first Non-invasive devices sue derived from human stem cells and sitting developed at Brown University (a handful of others Headache on top of an array of electrodes. A screen dis- have been given a similar device). An array of small Iplayswhatthe electrodesare pickingup: the character- electrodes, called a Utah array, is implanted into the Implants istic peak-and-trough wave forms offiring . motor cortex, a strip of the brain that governs move- Inside intelligence To see these signals emanating from disembodied ment. These electrodes detect the neurons that fire tissue is weird. The firingofa is the basic build- when someone intends to move his hands and arms. Data processing ing block of intelligence. Aggregated and combined, These signals are sent through wires that poke out of Translation required such “action potentials” retrieve every memory, guide the person’sskull to a decoder, where theyare translat- every movement and marshal every thought. As you ed into a variety of outputs, from moving a cursor to Brain scan read this sentence, neurons are firing all over your controlling a limb. Eberhard Fetz brain: to make sense of the shapes of the letters on the The system has allowed a woman paralysed by a Looking for page; to turn those shapes into phonemes and those stroke to use a robotic arm to take her first sip of coffee serendipity phonemes into words; and to confer meaning on without help from a caregiver. It has also been used by Grey matter, red tape those words. a paralysed person to type ata rate ofeightwords a mi- This symphony of signals is bewilderingly com- nute. It has even reanimated useless human limbs. In a plex. There are as many as 85bn neurons in an adult study led by Bob Kirsch of Case Western Reserve Uni- human brain, and a typical neuron has10,000 connec- versity, published in the Lancet this year, BrainGate tions to othersuch cells. The job ofmappingthese con- was deployed artificially to stimulate muscles in the nections is still in its early stages. But as the brain gives arms of William Kochevar, who was paralysed in a cy- up its secrets, remarkable possibilities have opened cling accident. As a result, he was able to feed himself up: of decoding neural activity and using that code to for the first time in eight years. control external devices. Interactions between and machines have 1 The Economist January 6th 2018 1 TECHNOLOGY QUARTERLY Brain-computer interfaces

2 changed lives in other ways, too. The opening ceremony of the cells in the visual cortex of macaque mon- football World Cup in Brazil in 2014 featured a paraplegic man keys encoded 50 different aspects of a per- who used a mind-controlled robotic exoskeleton to kick a ball. A son’s face, from skin colour to eye spacing. recent study by Ujwal Chaudhary of the University of Tübingen That enabled them to predict the appear- and four co-authors relied on a technique called functional near- ance of faces that monkeys were shown infrared spectroscopy (fNIRS), which beams infrared light into the from the brain signalstheydetected, with a brain, to put yes/no questions to four locked-in patients who had spooky degree of accuracy. But conducting been completely immobilised by Lou Gehrig’s disease; the pa- scientific research on human brains is tients’ mental responses showed up as identifiable patterns of harder, for regulatory reasons and because blood oxygenation. they are larger and more complex. Neural activity can be stimulated as well as recorded. Cochlear Ultimately Even when BCI breakthroughs are implants convert sound into electrical signals and send them into it may be made on humans in the lab, they are diffi- the brain. Deep-brain stimulation uses electrical pulses, delivered cult to translate into clinical practice. Wired via implanted electrodes, to help control Parkinson’s disease. The possible to magazine first reported breathlessly on the technique has also been used to treat other movement disorders then new BrainGate system back in 2005. and mental-health conditions. NeuroPace, a Silicon Valley firm, meld An early attempt to commercialise the monitors brain activity for signs of imminent epileptic seizures technology, by a company called Cyberki- and delivers electrical stimulation to stop them. together netics, foundered. It took NeuroPace 20 It is easy to see how brain-computer interfaces could be ap- years to develop its technologies and nego- plied to other sensory inputs and outputs. Researchers at the Uni- human and tiate regulatory approval, and it expects versity ofCalifornia, Berkeley, have deconstructed electrical activ- artificial that only 500 people will have its elec- ity in the temporal lobe when someone is listening to con- trodes implanted this year. versation; these patterns can be used to predict what word intelligence Current BCI technologies often require someone has heard. The brain also produces similar signals when experts to operate them. “It is not much use someone imagines hearing spoken words, which may open the if you have to have someone with a mas- door to a speech-processing device for people with conditions ters in standing next to such as aphasia (the inability to understand or produce speech). the patient,” says Leigh Hochberg, a neu- Researchers at the same university have used changes in blood rologist and professor at Brown University, who is one of the key oxygenation in the brain to reconstruct, fuzzily, film clips that peo- figures behind BrainGate. Whenever wires pass through the skull ple were watching. Nowimagine a device thatcould work the oth- and scalp, there is an infection risk. Implants also tend to move er way, stimulating the visual cortex of blind people in order to slightly within the brain, which can harm the cells it is recording project images into their mind’s eye. from; and the brain’s immune response to foreign bodies can If the possibilities of BCIs are enormous, however, so are the create scarring around electrodes, making them less effective. problems. The most advanced science is being conducted in ani- Moreover, existing implants record only a tiny selection of the mals. Tinysilicon probescalled Neuropixelshave been developed brain’s signals. The Utah arrays used by the BrainGate consor- by researchers at the Howard Hughes Institute, the Allen Institute tium, for example, might pick up the firing of just a couple of hun- and University College London to monitor cellular-level activity dred neurons out of that 85bn total. In a paper published in 2011, in multiple brain regions in mice and rats. Scientists at the Univer- Ian Stevenson and Konrad Kording of Northwestern University sity ofCalifornia, San Diego, have built a BCI that can predict from showed thatthe numberofsimultaneouslyrecorded neurons had priorneural activity what songa zebra finch will sing. Researchers doubled every seven years since the 1950s (see chart). This falls far at the California Institute of Technology have worked out how shortofMoore’slaw, which hasseen computingpowerdouble ev-1

2 The Economist January 6th 2018 TECHNOLOGY QUARTERLY Brain-computer interfaces

2 ery two years. Indeed, the Wyss Centre in Geneva exists because it is so hard to get out of the lab and into clinical practice. John Donoghue, who heads the centre, is another of the pioneers of the BrainGate system. He says it is designed to help promising ideas cross several “valleys of death”. One is financial: the combi- nation oflengthy paybackperiods and deep technology scares off most investors. Another is the need for multidisciplinary exper- tise to get better interfaces built and management skills to keep complex projects on track. Yet another is the state of itself. “At its core, this is based on understanding how the brain works, and we just don’t,” says Dr Donoghue.

Me, myselfand AI This odd mixture of extraordinary achievement and halting pro- gress now has a new ingredient: Silicon Valley. In October 2016 Bryan Johnson, an entrepreneur who had made a fortune by sell- ing his payments company, Braintree, announced an investment of $100m in Kernel, a firm he has founded to “read and write neu- ral code”. Mr Johnson reckons that the rise ofartificial intelligence (AI) will demand a concomitant upgrade in human capabilities. “I find it hard to imagine a world by 2050 where we have not inter- vened to improve ourselves,” he says, picturing an ability to ac- quire new skills at will orto communicate telepathically with oth- ers. Last February Kernel snapped up Kendall Research Systems, a spinoff from the Massachusetts Institute of Technology (MIT) that works on neural interfaces. Kernel is not alone in seeing BCIs as a way forhumans to co-ex- ist with AI rather than be subjugated to it. In 2016 Elon Musk, the boss of SpaceX and Tesla, founded a new company called Neura- link, which is also working to create new forms of implants. He Non-invasive devices has gathered together an impressive group of co-founders and set a goal of developing a BCI for clinical use in people with disabil- ities by 2021. Devices for people without such disabilities are Headache about eight to ten years away, by Mr Musk’s reckoning. is not saying what exactly it is doing, but Mr Musk’s thinking is outlined in a lengthy post on Wait But Why, a website. In it, he describes the need for humans to communicate far more quickly with each other, and with computers, if they are not to be AI left in the dust by . The post raises some extraordinary possibili- Can the brain be deciphered without opening up the skull? ties: beingable to access and absorb knowledge instantly from the cloud or to pump images from one person’s retina straight into the ATRICKKAIFOSH’S left hand lies flat on the table in front visual cortex of another; creating entirely new sensory abilities, of him. Occasionally his fingers twitch or his palm rises from infrared eyesight to high-frequency hearing; and ultimately, up slightly from the surface. There is nothing obvious to melding together human and artificial intelligence. connect these movements with what is happeningon the In April it was Facebook’s turn to boggle minds as it revealed tablet in front of him, where a game of asteroids is being plans to create a “silent speech” interface that would allow people played. Yet he is controllingthe spaceship on the screen as to type at100 words a minute straight from their brain. A group of Pit spins, thrusts and fires. more than 60 researchers, some inside Facebook and some out- What enables him to do so is a sweatband studded with small side, are working on the project. A separate startup, Openwater, is gold bars that sits halfway up his left forearm. Each bar contains a also working on a non-invasive neural-imaging system; its foun- handful ofelectrodes designed to pickup the signals of motor un- der, Mary Lou Jepsen, says that her technology will eventually al- its (the combination of a motor neuron, a cell that projects from

low minds to be read. the spinal cord, and the muscle fibres it controls). These data are

¡

£

Many BCI experts react to the arrival of ¢alley visionaries processed bymachine-learningalgorithmsand translated into the by rolling their eyes. Neuroscience is a work in progress, they say. actions in the game. Dr Kaifosh, a co-founder of CTRL-Labs, the An effective BCI requires the involvement of many disciplines: startup behind the device, has learned to exercise impressive con- materials science, neuroscience, machine learning, engineering, trol over these signals with hardly any obvious movement. design and others. There are no shortcuts to clinical trials and reg- Some say that the claims of Dr Kaifosh and Thomas Reardon, ulatory approval. his co-founder, that CTRL-Labs has created a brain-machine inter-

In all this, the sceptics are right. Many of the ambitions being face are nonsense. The sweatband is nowhere near the brain, and

£ ¥ aired look fantastical. Still, this is a critical moment for BCI ¤ast the signals it is picking up are generated not just by the firing of a amounts of money are pouring into the field. Researchers are try- motor neuron but by the electrical activity of muscles. “If this is a ing multiple approaches. Mr Musk in particular has a track record BCI, then the movement of my fingers when I type on a keyboard of combining grandiose aspirations (colonising Mars) and practi- is also a brain output,” sniffs one researcher. Krishna Shenoy, who cal success (recovering and relaunching rockets via SpaceX). directs the neural prosthetics systems lab at Stanford University To be clear, “The Matrix” is not imminent. But BCIs may be and acts as an adviserto the firm, thinks it is on the right side of the about to take a big leap forward. For that to happen, the most im- divide. “Measuring the movement of the hand is motion capture. portantthingisto find a betterwayofconnectingwith the brain. 7 They are picking up neural activity amplified by the muscles.” 1 The Economist January 6th 2018 3 TECHNOLOGY QUARTERLY Brain-computer interfaces

2 Whateverthe semantics, itisinstructive can be identified. Studies have shown strong correlations be- to hear the logic behind the firm’s decision tween people’s mental states as shown by an EEG and theirability to record the activity of the peripheral ner- to spot weapons in X-rays ofluggage. vous system, rather than looking directly Yet the uses of EEGs remain limited. In a real-world environ- inside the head. The startup wantsto create ment like a cockpit, a car or an airport, muscle activity and ambi- a consumer product (its potential uses in- ent electricity are likely to confound any neural signals. As for clude being an interface for interactions in Neurable’s game, it relies not solely on brain activity but also de- virtual reality and augmented reality). It is ploys eye-tracking technology to see where a player is looking. Dr not reasonable to expect consumers to un- Alcaide says the system can workwith brain signals alone, but it is dergo brain surgery, say the founders, and It is not hard for a user to disentangle the two. current non-invasive options for reading Othernon-invasive options also have flaws. Magnetoencepha- the brain provide noisy, hard-to-read sig- reasonable lography measures magnetic fields generated by electrical activity nals. “For machine-learning folk, there is in the brain, but it requires a special room to shield the machinery no question which data set—cortical neu- to expect from Earth’s magnetic field. Functional magnetic resonance imag- ronsormotorneurons—youwould prefer,” ing(fMRI) can spotchangesin blood oxygenation, a proxyfor neu- says Dr Reardon. consumers ral activity, and can zero in on a small area of the brain. But it in- This trade-off between the degree of in- to undergo volves a large, expensive machine, and there is a lag between vasiveness and the fidelity of brain signals neural activity and blood flow. is a big problem in the search for improved brain If any area is likely to yield a big breakthrough in non-invasive BCIs. But plenty ofpeople are trying to find recording of the brain, it is a variation on fNIRS, the infrared tech- a better way to read neural code from out- surgery nique used in the experiment to allow locked-in patients to com- side the skull. municate. In essence, light sent through the skull is either ab- The simplest way to read electrical ac- sorbed orreflected backto detectors, providinga picture ofwhat is tivity from outside is to conduct an electro- going on in the brain. This technique does not require bulky encephalogram (EEG). And it is not all that equipment, and unlike EEG it does not measure electrical activity, simple. Conventionally, it has involved wearing a cap containing so it is not confused by muscle activity. Both Facebook and Open- lots of electrodes that are pressed against the surface of the scalp. water are focusing their efforts on this area. To improve the signal quality, a conductive gel is often applied. The obstacles to a breakthrough are formidable, however. Cur- That requires a hairwash afterwards. Sometimes the skin of the rent infrared techniques measure an epiphenomenon, blood oxy- scalp isroughened up to geta betterconnection. Asa consumer ex- genation (the degree ofwhich affects the absorption of light), rath- perience it beats going to the dentist, but not by much. er than the actual firing of neurons. The light usually penetrates Once on, each electrode picks up currents generated by the fir- only a few millimetres into the cortex. And because light scatters ing of thousands of neurons, but only in the area covered by that in tissue (think of how your whole fingertip glows red when you electrode. Neurons that fire deep in the brain are not detected ei- press a pen-torch against it), the precise source of reflected signals ther. The signal is distorted by the layers of skin, bone and mem- is hard to identify. brane that separate the brain from the electrode. And muscle ac- Facebook is not saying much about what it is doing. Its efforts tivity (of the sort that CTRL-Labs looks for) from eye and neck are being led by Mark Chevillet, who joined the social-media movements or clenched jaws can overwhelm the neural data. giant’s Building 8 consumer-hardware team from Johns Hopkins Even so, some EEG signals are strong enough to be picked up University. To cope with the problem oflight scattering as it passes pretty reliably. An “event-related potential”, for example, is an through the brain, the team hopes to be able to pick up on both electrical signal that the brain reliably gives off in response to an ballistic photons, which pass through tissue in a straight line, and external stimulus of some sort. One such, called an error-related what it terms “quasi-ballistic photons”, which deviate slightly but potential (Errp), occurs when a user spots a mistake. Researchers can still be traced to a specific source. The clock is ticking. Dr Che- at MIT have connected a human observer wearing an EEG cap to villet has about a year of a two-year programme left to demon- an industrial robot called Baxter as it carried out a sorting task. If strate that the firm’s goal ofbrain-controlled typing at100 words a Baxter made a mistake, an Errp signal in the observer’s brain alert- minute is achievable using current invasive cell-recording tech- ed the robot to its error; helpfully, if Baxter still did not react, the niques, and to produce a road map for replicating that level ofper- human brain generated an even stronger Errp signal. formance non-invasively. Openwater is much less tight-lipped. Ms Jepsen says that her Ifthe cap fits San Francisco-based startup uses holography to reconstruct how Neurable, a consumer startup, has developed an EEG headset lightscattersin the body, so itcan neutralise thiseffect. Openwater, with just seven dry electrodes which uses a signal called the P300 she suggests, has already created technology that has a billion to enable users to play a virtual-reality (VR) escape game. This sig- times the resolution of an fMRI machine, can penetrate the cortex nal is a marker of surprise or recognition. Think of the word to a depth of10cm, and can sample data in milliseconds. “brain” and then watch a series of letters flash up randomly on a Openwater has yet to demonstrate its technology, so these screen; when the letter “b” comes up, you will almost certainly be claims are impossible to verify. Most BCI experts are sceptical. But giving off a P300 signal. In Neurable’s game, all you have to do is Ms Jepsen has an impressive background in consumer electronics concentrate on an object (a ball, say) for it to come towards you or and display technologies, and breakthroughs by their nature up- be hurled at an object. Ramses Alcaide, Neurable’s boss, sees the end conventional wisdom. Developer kits are due out in 2018. potential for entertainment companies like Disney (owner of the In the meantime, other efforts to decipher the language of the Star Wars and Marvel franchises) to license the software in theme brain are under way. Some involve heading downstream into the parks and arcade games. peripheral nervous system. One example of that approach is Thorsten Zander of the Technische Universität in Berlin thinks CTRL-Labs; another is provided by Qi Wang, at Columbia Univer- that “passive” EEG signals (those that are not evoked by an exter- sity, who researches the role ofthe locus coeruleus, a nucleus deep nal stimulus) can be put to good use too. Research has shown that in the brain stem that plays a role in modulating anxiety and brainwave activity changes depending on how alert, drowsy or stress. Dr Wang is looking at ways of stimulating the vagus nerve, focused a person is. Ifan EEG can reliablypickthisup, perhaps sur- which runs from the brain into the abdomen, through the skin to geons, pilots or truckdrivers who are becoming dangerously tired see ifhe can affect the locus coeruleus. 1 4 The Economist January 6th 2018 TECHNOLOGY QUARTERLY Brain-computer interfaces

2 Others are looking at invasive approaches that do not involve ofdebate. In people who suffer from movement disorders such as drilling through the skull. One idea, from a firm called SmartStent, Parkinson’s disease, spaghetti-like leads and big electrodes are using technology partly developed with the University of Mel- used to carry out deep-brain stimulation over a fairly large area of bourne, isto use a stent-like device called a “stentrode” thatisstud- tissue. Such treatment is generally regarded as effective. Andrew ded with electrodes. It is inserted via a small incision in the neck Jackson ofthe University ofNewcastle thinks that recording activ- and then guided up through blood vessels to overlie the brain. itybyensemblesofneurons, ofthe sortthatgetspicked up byelec- Once the device is in the right location, it expands from about the trocorticography arrays, can be used to decode relatively simple size ofa matchstickto the size ofthe vessel and tissue growsinto its movement signals, like an intention to grasp something or to ex- scaffolding, keeping it in place. Human trials of the stentrode are tend the elbow. due to start next year. But to generate fine-grained control signals, such as the move- Another approach is to put electrodes under the scalp but not ment of individual fingers, more precision is needed. “These are under the skull. Maxime Baud, a neurologist attached to the Wyss very small signals, and there are many neurons packed closely to- Centre, wants to do just that in order to monitor the long-term sei- gether, all firingtogether,” saysAndrewSchwartzofthe University zure patterns of epileptics. He hopes that once these patterns are of Pittsburgh. Aggregating them inevitably means sacrificing de- revealed, they can be used to provide accurate forecasts ofwhen a tail. After all, individual cells can have very specific functions, seizure is likely to occur. from navigation to facial recognition. The 2014 Nobel prize for ¦et others think they need to go directly to the source of action medicine was awarded for workon place and grid cells, which fire potentials. And that means heading inside the brain itself. 7 when animals reach a specific location; the idea of the “Jennifer Aniston neuron” stems from research showing that single neu- rons can fire in response to pictures ofa specific celebrity. Companies like Neuralinkand Kernel are betting that the most Implants ambitious visions of BCIs, in which thoughts, images and move- ments are seamlessly encoded and decoded, will require high-res- olution implants. So, too, is America’s Defense Advanced Re- Inside intelligence search Projects Agency (DARPA), an arm of the Pentagon, which this year distributed $65m among six organisations to create a high-resolution implantable interface. BrainGate and others con- tinue to workon systems oftheir own. But the challenges that these researchers face are truly daunt- ing. The ideal implantwould be safe, small, wirelessand long-last- The hunt for smaller, safer and smarter brain implants ing. It would be capable of transmitting huge amounts of data at high speed. It would interact with many more neurons than cur- ALK to neuroscientists about brain-computer interfaces rent technology allows (the DARPA programme sets its grant recip- (BCIs) for long enough, and the stadium analogy is al- ients a target of1m neurons, along with a deadline of 2021for a pi- mostbound to come up. Thiscomparesthe neural activi- lot trial to get under way in humans). It would also have to ty ofthe brain to the noise made by a crowd at a football navigate an environment that Claude Clément ofthe Wyss Centre game. From outside the ground, you might hear back- likens to a jungle by the sea: humid, hot and salty. “The brain is not ground noise and be able to tell from the roars whether a the right place to do technology,” he says. As the chief technology Tteam has scored. In a blimp above the stadium you can tell who officer, he should know. has scored and perhaps which players were involved. Only inside it can you askthe fan in row 72 how things unfolded in detail. Da neuron, ron, ron Similarly, with the brain it is only by getting closer to the action That is not stopping people from trying. The efforts now being that you can really understand what is going on. To get high-reso- made to create betterimplants can be divided into two broad cate- lution signals, for now there is no alternative to opening up gories. The first reimagines the current technology of small wire the skull. One option is to place electrodes onto the sur- electrodes. The second heads offin new, non-electrical directions. face of the brain in what is known as electrocorticogra- Start with ways to make electrodes smaller and better. Ken phy. Another is to push them right into the tissue ofthe Shepard is a professor of electrical and biomedical engineering at brain, for example by using a grid of microelectrodes Columbia University; his lab is a recipient of DARPA funds, and is like BrainGate’s Utah array. aiming to build a device that could eventually help Just how close you have to come to indi- blind people with an intact visual cortex to see by vidual neurons to operate BCIs is a matter stimulating precisely the right neu-

The Economist January 6th 2018 5 TECHNOLOGY QUARTERLY Brain-computer interfaces

2 rons in order to produce images inside their brains. He thinks he them as a string of zeros; and second, by concentrating on the can do so by using state-of-the-art CMOS (complementary metal- wave forms of specific action potentials rather than recording oxide semiconductor) electronics. each point alongtheircurves. Indeed, he sees data compression as Dr Shepard is aware that any kind of penetrating electrode can being the company’s big selling-point, and expects others that cause cell damage, so he wants to build “the mother of all surface want to create specific BCI applications or prostheses simply to recording devices” which will sit on top of the cortex and under plug into its feed. “We see ourselves as the neural data backbone, the membranes that surround the brain. He has already created a like a Qualcomm or Intel,” he says. prototype ofa first-generation CMOS chip, which measures about 1cm by 1cm and contains 65,000 electrodes; a slightly larger, sec- Meshy business ond-generation version will house 1m sensors. But like everyone Some researchers are trying to get away from the idea of wire im- else trying to make implants work, Dr Shepard is not just cram- plants altogether. At Brown University, for example, Arto Nur- ming sensors onto the chip. He also has to add the same number mikko is leading a multidisciplinary team to create “neurograins”, ofamplifiers, a converterto turn the analogue signals ofaction po- each the size of a grain of sugar, that could be sprinkled on top of tentials into the digital 0s and 1s of machine learning, and a wire- the cortex or implanted within it. Each grain would have to have less link to send (or receive) data to a relay station that will sit on built-in amplifiers, analogue-to-digital converters and the ability the scalp. That, in turn, will send (or receive) the data wirelessly to to send data to a relay station which could powerthe grains induc- external processors fordecoding. tively and pass the information to an external processor. Dr Nur- The device also has to be powered, another huge part of the mikko is testing elements ofthe system in rodents; he hopes even- implantables puzzle. No one in the field puts faith in batteries as a tually to put tens ofthousands ofgrains inside the head. source of power. They are too bulky, and the risk of battery fluid Meanwhile, in a lab at Harvard University, Guosong Hong is leaking into the brain is too high. Like many of his peers, Dr Shep- demonstrating another innovative interface. He dips a syringe ard uses inductive coupling, whereby currents passing through a into a beaker of water and injects into it a small, billowing and coiled wire create a magnetic field that can induce a current in a glinting mesh. It is strangely beautiful to watch. Dr Hong is a post- second coil (the way that an electric toothbrush gets recharged). doctoral fellow in the lab of Charles Lieber, a professor of chemis- That job is done by coils on the chip and on the relay station. try; theyare both workingto create a neural interface thatblursthe Over on America’s west coast, a startup called Paradromics is distinction between biology and electronics. Their solution is a also using inductive coupling to power its porous net made of a flexible polymer implantable. But Matt Angle, its boss, does called SU-8, studded with sensors and con- not think that souped-up surface record- ductive metal. ings will deliver sufficiently high resolu- The mesh isdesigned to solve a number tion. Instead, he is working on creating tiny ofproblems. One hasto do with the brain’s bundles ofglass and metal microwires that immune response to foreign bodies. By can be pushed into brain tissue, a bit like a replicating the flexibility and softness of Utah arraybutwith manymore sensors. To neural tissue, and allowing neurons and stop the wires clumping together, thereby other types of cells to grow within it, it reducing the number of neurons they en- should avoid the scarring that stiffer, solid gage with, the firm uses a sacrificial po- probes can sometimes cause. It also takes lymerto splay them apart; the polymer dis- up much less space: less than 1% of the vol- solves but the wires remain separated. ume of a Utah array. Animal trials have They are then bonded onto a high-speed gone well; the next stage will be to insert CMOS circuit. A version of the device, with the mesh into the brains of epilepsy pa- 65,000 electrodes, will be released next tients who have not responded to other year foruse in animal research. forms oftreatment and are waiting to have That still leaves lots to do before Para- bits oftissue removed. dromics can meet its DARPA-funded goal A mile away, at MIT, members ofPolina of creating a 1m wire device that can be Anikeeva’s lab are also trying to build de- used in people. Chief among them is cop- vices that match the physical properties of ing with the amount of data coming out of neural tissue. Dr Anikeeva is a materials the head. Dr Angle reckons that the initial scientist who first dived into neuroscience device produces 24 gigabits of data every at the lab of Karl Deisseroth at Stanford second (streaming an ultra-high-definition University, who pioneered the use ofopto- movie on Netflix uses up to 7GB an hour). genetics, a way of genetically engineering In animals, these data can be transmitted cells so that they turn on and off in re- through a cable to a bulky aluminium sponse to light. Her reaction upon seeing a head-mounted processor. That is a hard (mouse) brain up close for the first time look to pull off in humans; besides, such was amazement at how squishy it was. “It quantities of data would generate far too is problematic to have something with the much heat to be handled inside the skull or elastic properties of a knife inside some- transmitted wirelessly out ofit. thingwith the elastic properties ofa choco- So Paradromics, along with everyone late pudding,” she says. else trying to create a high-bandwidth sig- One way she is dealing with that is to nal into and out of the brain, has to find a borrow from the world of telecoms by cre- way to compress the data rate without ating a multichannel fibre with a width of compromising the speed and quality of in- 100 microns (one micron is a millionth ofa formation sent. Dr Angle reckons he can do metre), about the same as a human hair. this in two ways: first, by ignoring the mo- That is denser than some ofthe devices be- ments of silence in between action poten- ing worked on elsewhere, but the main tials, rather than laboriously encoding thing that distinguishes it is that it can do1 6 The Economist January 6th 2018 TECHNOLOGY QUARTERLY Brain-computer interfaces

2 multiple things. “Electronics with just cur- Data processing rent and voltage is not going to do the trick,” she says, pointing out that the brain communicates not just electrically but Translation required chemically, too. Dr Anikeeva’s sensor has one channel for recording using electrodes, but it is also able to take advantage of . A second channel is designed to deliver Introducing channelrhodopsin, an algal protein that Once data have been extracted from the brain, how can can be smuggled into neurons to make they be used to best effect? tiny, flexible them sensitive to light, and a third to shine a light so that these modified neurons can OR those who reckon that brain-computer interfaces will materials be activated. nevercatch on, there is a simple answer: they already have. It is too early to know if optogenetics Well over 300,000 people worldwide have had cochlear into the can be used safely in humans: channelrho- implants fitted in their ears. Strictly speaking, this hearing dopsin has to be incorporated into cells us- device does not interact directly with neural tissue, but the brain ing a virus, and there are question-marks effect is not dissimilar. A processor captures sound, which about how much light can safely be shone Fis converted into electrical signals and sent to an electrode in the creates a into the brain. But human clinical trials are inner ear, stimulating the cochlear nerve so that sound is heard in “wet under way to make retinal ganglion cells the brain. Michael Merzenich, a neuroscientist who helped devel- light-sensitive in people whose photo- op them, explains that the implants provide only a crude repre- noodle” receptor cells are damaged; another of the sentation of speech, “like playing Chopin with your fist”. But giv- recipients of DARPA funds, Fondation Voir en a little time, the brain works out the signals. problem et Entendre in Paris, aims to use the tech- That offers a clue to another part of the BCI equation: what to nique to transfer images from special gog- do once you have gained access to the brain. As cochlear implants gles directly into the visual cortex of com- show, one option is to let the world’s most powerful learning pletely blind people. In principle, other machine do its stuff. In a famous mid-20th-century experiment, senses could also be restored: optogenetic two Austrian researchers showed that the brain could quickly stimulation of cells in the inner ear of mice has been used to con- adaptto a pairofglassesthatturned the image theyprojected onto trol hearing. the retina upside down. More recently, researchers at Colorado Dr Anikeeva is also toying with another way ofstimulating the State University have come up with a device that converts sounds brain. She thinksthata weakmagneticfield could be used to pene- into electrical impulses. When pressed against the tongue, it pro- trate deep into neural tissue and heat up magnetic nanoparticles duces different kinds of tingle which the brain learns to associate that have been injected into the brain. If heat-sensitive capsaicin with specific sounds. receptors were triggered in modified neurons nearby, the in- The brain, then, is remarkably good at working things out. creased temperature would cause the neurons to fire. Then again, so are computers. One problem with a hearing aid, for Another candidate for recording and activating neurons, be- example, is that it amplifies every sound that is coming in; when yond voltage, light and magnets, is ultrasound. Jose Carmena and youwantto focuson one person in a noisyenvironment, such as a Michel Maharbiz at the University of California, Berkeley, are the party, that is not much help. Nima Mesgarani ofColumbia Univer- main proponentsofthisapproach, which again involvesthe inser- sity is working on a way to separate out the specific person you tion of tiny particles (which they call “neural dust”) into tissue. want to listen to. The idea is that an algorithm will distinguish be- Passing ultrasound through the body affects a crystal in these tween different voices talking at the same time, creating a spectro- motes which vibrates like a tuning fork; that produces voltage to gram, or visual representation of sound frequencies, of each per- power a transistor. Electrical activity in adjacent tissue, whether son’s speech. It then looks at neural activity in the brain as the muscles or neurons, can change the nature of the ultrasonic echo wearer of the hearing aid concentrates on a specific interlocutor. given offby the particle, so this activity can be recorded. This activity can also be reconstructed into a spectrogram, and the Many of these new efforts raise even more questions. If the ones that match up will get amplified (see diagram). ambition is to create a “whole-brain interface” that covers multi- Algorithms have done better than brain plasticity at enabling ple regions of the brain, there must be a physical limit to how paralysed people to send a cursor to a target using thought alone. much additional material, be it wires, grains or motes, can be in- In research published earlierthisyear, forexample, Dr Shenoyand1 troduced into a human brain. If such particles can be made suffi- ciently small to mitigate that problem, another uncertainty arises: would they float around in the brain, and with what effects? And how can large numbers of implants be put into different parts of the brain in a single procedure, particularly if the use of tiny, flex- ible materials creates a “wet noodle” problem whereby implants are too floppy to make their way into tissue? (Rumour has it that Neuralink may be pursuing the idea of an automated “sewing machine” designed to get around this issue.) All thisunderlineshowhard itwill be to engineera newneural interface that works both safely and well. But the range of efforts to create such a device also prompts optimism. “We are approach- ing an inflection-point that will enable at-scale recording and stimulation,” says Andreas Schaefer, a neuroscientist at the Crick Institute in London. Even so, being able to get the data out of the brain, or into it, is only the first step. The next thing is processing them. 7 The Economist January 6th 2018 7 TECHNOLOGY QUARTERLY Brain-computer interfaces

2 his collaborators at Stanford University recorded a big improve- ment in brain-controlled typing. This stemmed not from new sig- nals or whizzier interfaces but from better maths. One contribution came from Dr Shenoy’s use of data generat- ed during the testing phase of his algorithm. In the training phase a user is repeatedly told to move a cursor to a particular target; machine-learning programs identify patterns in neural activity that correlate with this movement. In the testing phase the user is shown a grid of letters and told to move the cursor wherever he wants; that tests the algorithm’s ability to predict the user’s wish- es. The user’s intention to hit a specific target also shows up in the data; by refittingthe algorithm to include that information too, the cursor can be made to move to its target more quickly. But although algorithms are getting better, there is still a lot of room for improvement, not least because data remain thin on the ground. Despite claims that smart algorithms can make up for bad signals, they can do only so much. “Machine learning does nearly magical things, but it cannot do magic,” says Dr Shenoy. Consider the use of functional near-infrared spectroscopy to identify sim- ple yes/no answers given by locked-in patients to true-or-false statements; they were right 70% ofthe time, a huge advance on not hand has 27. Visual-cortex researchers often work with static im- being able to communicate at all, but nowhere near enough to ages, whereas humans in real life have to cope with continuously have confidence in their responses to an end-of-life discussion, moving images. Workon the sensory feedbackthat humans expe- say. More and cleaner data are required to build better algorithms. rience when they grip an object has barely begun. It does not help that knowledge of how the brain works is still And although computational neuroscientists can piggyback so incomplete. Even with better interfaces, the organ’s extraordi- on broader advances in the field of machine learning, from facial nary complexities will not be quickly unravelled. The movement recognition to autonomous cars, the noisiness of neural data pre- of a cursor has two degrees of freedom, for example; a human sents a particular challenge. A neuron in the motor cortex may fire1 Brain scan Eberhard Fetz

The man who helped launch brain-computer interfaces in 1969 has not finished yet BCIs have deep roots. In the 18th century tional control ofa BCI was possible, and forging entirely new ones. Luigi Galvani discovered the role of elec- that the brain was capable oflearning As an example, he points to research tricity in nerve activity when he found how to operate one without any help. in which the recording ofan action poten- that applying voltage could cause a dead Some 48 years later, Dr Fetz is still at the tial in the brain prompts not only the frog’s legs to twitch. In the 1920s Hans University ofWashington, still fizzing with normal firing ofa motor neuron in the Berger used electroencephalography to energy and still enthralled by the brain’s spinal cord but also a parallel stimulus record human brain waves. In the 1960s plasticity. He is particularly interested in from a BCI delivered to the same site. The José Delgado theatrically used a brain the possibility ofartificially strengthening idea is to take advantage ofa relationship implant to stop a charging bull in its connections between cells, and perhaps made famous in an aphorism by a Cana- tracks. One ofthe field’s father figures is dian psychologist, Donald Hebb: “Neu- still hard at workin the lab. rons that fire together, wire together.” This Eberhard Fetz was a post-doctoral reinforced stimulus strengthens the con- researcher at the University ofWashing- nection between the original action ton in Seattle when he decided to test potential and the motor neuron, which whether a monkey could control the could help recovery from spinal-cord needle ofa meter using only its mind. A injuries. Such stimulation might also paper based on that research, published encourage stronger bonds in the brain in 1969, showed that it could. Dr Fetz itself—between the speech-processing tracked down the movement ofthe need- area ofa stroke victim’s brain, say, and the le to the firing rate ofa single neuron in region that controls movements ofthe the monkey’s brain. The animal learned lips and mouth. to control the activity ofthat single cell Asked to explain the slow rate of within two minutes, and was also able to progress since his breakthrough paper, Dr switch to control a different neuron. Fetz points to the technical difficulties of Dr Fetz disclaims any great insights in recording from single cells and the high setting up the experiment. “I was just hurdle to doing this sort ofworkin peo- curious, and did not make the association ple. But he does not thinkthere is a need with potential uses ofrobotic arms or the to find out much more about the brain in like,” he says. But the effect ofhis paper order to make further advances: “I have was profound. It showed both that voli- had a lot ofprogress by just jumping in.”

8 The Economist January 6th 2018 TECHNOLOGY QUARTERLY Brain-computer interfaces

2 at a rate of 100 action potentials a second when someone thinks to shake rather than slur his words. about moving his right arm on one occasion, but at a rate of115 on Giving humans more options of this sort will be essential if another. To make matters worse, neurons’ jobs overlap. So if a some of the bolder visions for brain-computer interfaces are to be neuron has an average firing rate of 100 to the right and 70 to the realised. Hannah Maslen from the UniversityofOxford is another left, what does a rate of85 signify? ethicist who works on a BCI project, in this case a neural speech At least the activities of the motor cortex have a visible output being developed by a consortium of European re- in the form of movement, showing up correlations with neural searchers. One of her jobs is to think through the distinctions be- data from which predictions can be made. But othercognitive pro- tween innerspeech and public speech: people need a dependable cesses lack obvious outputs. Take the area that Facebook is inter- mechanism for separating out what they want to say from what ested in: silent, orimagined, speech. Itisnotcertain thatthe brain’s they think. representation of imagined speech is similar enough to actual That is only one of many ethical questions that the sci-fi ver- (spoken or heard) speech to be used as a reference point. Progress sions of brain-computer interfaces bring up. What protection will ishampered byanotherfactor: “We have a century’sworth ofdata BCIs offer against neural hacking? Who owns neural data, includ- on how movement is generated by neural activity,” says Brain- ing information that is gathered for research purposes now but Gate’s Dr Hochberg dryly. “We know less about animal speech.” may be decipherable in detail at some point in the future? Where Higher-level functions, such as decision-making, present an does accountability lie if a user does something wrong? And if even greaterchallenge. BCI algorithmsrequire a model thatexplic- brain implants are performed not for therapeutic purposes but to itly defines the relationship between neural activity and the pa- augment people’s abilities, will that make the world an even more rameter in question. “The problem begins with defining the pa- unequal place? rameter itself,” says Dr Schwartz ofPittsburgh University. “Exactly what is cognition? How do you write an equation for it?” From potential to action Such difficulties suggest two things. One is that a set of algo- For some, these sorts ofquestions cannot be asked too early: more rithms for whole-brain activity is a very long way off. Another is than any other new technology, BCIs may redefine what it means that the best route forward for signal processing in a brain-com- to be human. For others, they are premature. “The societal-justice puter interface is likely to be some combination ofmachine learn- problem of who gets access to enhanced memory or vision is a ing and brain plasticity. The trick will be to develop a system in question for the next decades, not years,” says Thomas Cochrane, which the two co-operate, not just for the sake of efficiency but a neurologist and director of neuroethics at the Centre for Bioeth- also for reasons ofethics. 7 ics at Harvard Medical School. In truth, both arguments are right. It is hard to find anyone who argues that visions of whole-brain implants and AI-human sym- biosis are impossible to realise; but harderstill to find anyone who Looking for serendipity thinks something so revolutionary will happen in the near future. This report has looked at some of the technological difficulties as- sociated with taking BCIs out of the lab and into the mainstream. Grey matter, red tape But these are not the only obstacles in the way of “brain mouses” and telekinesis. The development path to the eventual, otherworldly destina- tion envisaged by organisations like Neuralink and Kernel is ex- tremely long and uncertain. The money and patience of rich indi- How obstacles to workable brain-computer interfaces may viduals like Elon Musk and Bryan Johnson can help, but in reality be overcome each leg ofthe journey needs a commercial pathway. Companies such as CTRL-Labs and Neurable may well open EUROTECHNOLOG§has long been a favourite of sci- the door to consumer applications fairly quickly. But for invasive ence-fiction writers. In “Neuromancer”, a wildly inven- technologies, commercialisation will ini- tive bookbyWilliam Gibson written in 1984, people can tially depend on therapeutic applications. use neural implants to jack into the sensory experi- That means overcoming a host of hurdles, ences of others. The idea of a neural lace, a mesh that from managing clinical trials to changing grows into the brain, was conceived by Iain M. Banks in doctors’ attitudes. Frank Fischer, the boss Nhis “Culture” series of novels. “The Terminal Man” by Michael of NeuroPace, has successfully negotiated Crichton, published in 1972, imagines the effects ofa regulatory approval for his company’s epi- on someone who is convinced that machines are takingover from lepsy treatment, but it has been a long, humans. (Spoiler: not good.) hard road. “If we tried to raise money to- Where the sci-fi genre led, philosophers are now starting to fol- day knowing the results ahead of time, it low. In Howard Chizeck’s lab at the University of Washington, re- would have been impossible to get fund- searchersare workingon an implanted device to administer deep- ed,” he says. Brain- brain stimulation (DBS) in orderto treat a common movement dis- Start with regulation. Neural interfaces order called essential tremor. Conventionally, DBS stimulation is are not drugs but medical devices, which computer always on, wasting energy and depriving the patient of a sense of means that clinical trials can be completed control. The lab’s ethicist, Tim Brown, a doctoral student of philos- with just a handful ofpatients forproof-of- interfaces ophy, says that some DBS patients suffer a sense of alienation and principle trials, and just a couple of hun- may redefine complain offeeling like a robot. dred for the trials that come after that. Even To change that, the team at the University of Washington is us- so, ensuringa supply ofpatients for experi- what it ing neuronal activity associated with intentional movements as a ments with invasive interfaces presents trigger for turning the device on. But the researchers also want to practical difficulties. There is only one means to be enable patients to use a conscious thought process to override good supply of these human guinea pigs: these settings. Thatismore useful than itmightsound: stimulation epilepsy patients who have proved unre- human currents for essential tremor can cause side-effects like distorted sponsive to drugs and need surgery. These speech, so someone about to give a presentation, say, might wish patients have already had craniotomies 1 The Economist January 6th 2018 9 TECHNOLOGY QUARTERLY Brain-computer interfaces

gery. The precedents ofonce-rare, now-routine procedures such as laser eye and cosmetic surgery suggest that invasiveness alone need not stop brain implants from catchingon. More than 150,000 people have had electrodes implanted for deep-brain stimulation to help them control Parkinson’s disease. But it is also a matter of functionality: plenty ofamputees, for example, prefersimple met- al hooks to prosthetic arms because they are more reliable.

Waiting forNeuromancer These are all good reasons to be cautious about the prospects for BCIs. But there are also reasons to thinkthat the field is poised for a great leap forward. Ed Boyden, a neuroscientist at MIT who made hisname asone ofthe people behind optogenetics, points outthat innovations are often serendipitous—from Alexander Fleming’s chance discovery of penicillin to the role of yogurt-makers in the development of CRISPR, a gene-editing technique. The trick, he says, is to engineer the chances that serendipity will occur, which means pursuing lots ofpaths at once. That is exactly what is now being done with BCIs. Scientific ef- forts to understand and map the brain are shedding ever more light on how its activity can be harnessed by a BCI and providing ever more data for algorithms to learn from. Firms like CTRL-Labs and Neurable are already listening to some of the more accessible neural signals, be it from the peripheral nervous system or from outside the skull. NeuroPace’sclosed-loop epilepsysystem creates a regulatory precedent that others can follow. Above all, researchers are working hard on a wide range of new implants for sending and receiving signals to and from the brain. That is where outfits like Kernel and Neuralink are focused in the short term. Mr Musk’s four-year schedule for creating a BCI for clinical use is too ambitious forfull clinical trials to be conclud- 2 and electrodes implanted so that doctors can monitor them and ed, butitismuch more realisticforpilottrials. This isalso the rough pinpoint the focal points oftheir seizures; while these patients are timeframe to which DARPA is working with its implantables pro- in hospital waiting for seizures to happen, researchers swoop in gramme. With these and other efforts running concurrently, ser- with requests oftheirown. But the supply ofvolunteers is limited. endipity has become more likely. Where exactly the electrodes are placed depends on clinical Once a really good, portable, patient-friendlyBCI is available, it needs, not researchers’ wishes. And because patients are often de- is not hard to think of medical conditions that affect a large num- liberately sleep-deprived in order to hasten seizures, their capaci- berofpeople and could potentiallyjustifysurgery. More than 50m ty to carry out anything but simple cognitive tasks is limited. people worldwide suffer from epilepsy, and 40% of those do not When it comes to safety, new technologies entail lengthier ap- respond to medication. Depression affects more than 300m peo- proval processes. Harvard’s Dr Lieber says that his neural mesh re- ple worldwide; many of them might benefit from a BCI that mon- quires a new sterilisation protocol to be agreed with America’s itored the brain for biomarkers of Food and Drug Administration. Researchers have to deal with the such mental disorders and deliv- question of how well devices will last in the brain over very long ered appropriate stimulation. The OFFER TO READERS periods. The Wyss Centre has an accelerated-ageing facility that quality of life of many older peo- Reprints of Technology exposes electrodes to hydrogen peroxide, in a process that mimics ple suffering from dysphagia (dif- Quarterly are available from the Rights and Syndication the brain’s immune response to foreign objects; seven days’ expo- ficulty in swallowing) could be Department. A minimum order sure in the lab is equivalent to seven years in the brain. improved by a device that helped of five copies is required. The regulators are not the only people who have to be won them swallow whenever they over. Health insurers (or other gatekeepers in single-payer sys- wanted to. “A closed-loop system CORPORATE OFFER tems) need to be persuaded that the devices offervalue formoney. for recording from a brain and re- Customisation options on The Wyss Centre, which aims to bow out of projects before de- sponding in a medically useful corporate orders of 100 or more vices are certified for manufacturing, plans with this in mind. One way is not a small market,” says are available. Please contact us ofthe applicationsitisworkingon isfortinnitus, a persistentinter- Dr Hochberg. to discuss your requirements. nal noise in the ears of sufferers which is often caused by overac- That may still bring to mind For more information on how to order special reports, reprints or

tivity in the auditory cortex. The idea is to provide an implant the aphorism of Peter Thiel, a Sil-

© any queries you may have please which gives users feedback on their cortical activity so that they ic ¨alley grandee, about hav- contact: can learn to suppress any excess. Looking ahead to negotiations ing been promised flying cars and with insurers, the Wyss is trying to demonstrate the effectiveness getting 140 characters. There is a The Rights and Syndication of its implant by including a control group of people whose tinni- large gap between dreamy talk of Department tus is being treated with cognitive behavioural therapy. symbiosis with AI, or infrared The Economist That still leaves two other groups to persuade. Doctors need to eyesight, and takingyearsto build 20 Cabot Square be convinced that the risks ofopeningup the skull are justified. Mr a better brain implant formedical London E14 4QW Fischer says that educating physicians proved harder than expect- purposes. But if a device to deliv- Tel +44 (0)20 7576 8148 ed. “The community does not find it natural to think er a real-time, high-resolution, Fax +44 (0)20 7576 8492 about device therapy,” he says. long-lasting picture of neural ac- e-mail: [email protected] Most important, patients will have to want the devices. This is tivity can be engineered, that gap www.economist.com/rights partly a question of whether they are prepared to have brain sur- will shrinkspectacularly. 7 10 The Economist January 6th 2018