Trends & Controversies

Brain-Computer Interfacing for Intelligent Systems

Anton Nijholt, University of Twente Desney Tan, Microsoft Research

dvances in cognitive neurosci- interfaces. Additionally, knowing the José del R. Millán describes real-time, ence and brain-imaging tech- user’s state as well as the tasks they’re robust control of brain-actuated robots Anologies give us the unprec- performing might provide key informa- and neuroprostheses. He focuses on edented ability to interface directly with tion that would help us design context- how to optimally blend a human user’s brain activity. These technologies let us sensitive systems that adapt themselves mental capabilities with a robot’s in- monitor the physical processes in the for optimal user support. This could telligence to operate complex devices brain that correspond with certain forms prove useful to healthy users who might through a low-bit-rate BCI based on of thought. Driven by society’s growing be situationally disabled—that is, they electroencephalography. recognition of the needs of people with might lack full access to traditional, Brendan Allison and Bernhard Graimann physical disabilities, researchers have physically based communication modali- present specific situations in which BCI begun using these technologies to build ties. It also opens a whole new domain research aimed at the physically disabled brain-computer interfaces (BCIs)—com- of niche applications, carefully designed can apply to healthy users. munication systems that don’t depend to exploit this novel modality’s specific Finally, Florin Popescu, Benjamin on the brain’s normal output pathways affordances, perhaps in conjunction Blankertz, and Klaus-R. Müller ground of peripheral nerves and muscles. In with more traditional input devices. We the opportunities in the hardware, BCIs, users explicitly manipulate their believe that games might be an area of computational, and social challenges brain activity instead of motor move- early adoption—first, because games we face as we work to create BCIs ments to produce signals that control have traditionally pushed us to consider that work effectively in real-world computers or communication devices. completely new usage paradigms, and environments. This research has extremely high impact, second, because gamers tend to be especially for disabled individuals who fairly tolerant of new technologies. Edu- can’t otherwise physically communicate. cation could be another such domain. Although removing the need for mo- The four short articles in this issue’s Anton Nijholt is full professor of tor movements in computer interfaces is Trends & Controversies provide a quick computer science at the University of challenging and rewarding, we believe overview of the past, present, and fu- Twente and chair of its Human Media the full potential of brain imaging as ture of BCIs. They are written primarily Interaction subdepartment. Contact an input mechanism lies in the rich in- by European researchers working with him at [email protected]. formation it provides about the user’s noninvasive techniques, which repre- state. Having access to this state is im- sent a focused subset of the broader re- Desney Tan is a researcher at Microsoft portant to researchers because it might search and viewpoints in the field. Research, where he manages the Com- let us derive more direct measures of Gert Pfurtscheller and Clemens Brun- putational User Experiences group. He traditionally elusive phenomena such ner begin with a state-of-the-art survey. also holds an affiliate faculty appoint- as task engagement, cognitive work- They discuss brain signals that can be ment in the Computer Science and En- load, surprise, satisfaction, or frustra- measured with various devices, ways to gineering Department at the University tion. These measures could open new control these signals, and how to train of Washington. Contact him at desney@ avenues for evaluating systems and users to do this. microsoft.com.

The State-of-the-Art in BCIs or attentional deficit hyperactivity disorders (ADHD); or Gert Pfurtscheller and Clemens Brunner, controlling computer games.1,2 Graz University of Technology Every mental activity—for example, decision making, intending to move, and mental arithmetic—is accom- A brain-computer interface (BCI) is a novel communica- panied by excitation and inhibition of distributed neural tion system that translates human thoughts or intentions structures or networks. With adequate sensors, we can re- into a control signal. In this way, a BCI provides a new, cord changes in electrical potentials, magnetic fields, and nonmuscular communication channel that system develop- (with a delay of some seconds) metabolic supply when the ers can use in a variety of applications, such as assisting activated neuron population exceeds some critical mass. people with severe motor disabilities; supporting biofeed- Consequently, we can base a BCI on electrical potentials, back training in people suffering from epilepsy, stroke, magnetic fields, or metabolic/hemodynamic recordings.

72 1541-1672/08/$25.00 © 2008 IEEE IEEE INTELLIGENT SYSTEMS Published by the IEEE Computer Society Figure 1 presents a schematic of the principal BCI components. The compo- Feature Preprocessing Classification nents involve signal acquisition, prepro- extraction cessing, feature extraction, classification, and an application interface together with the application. When we talk about a BCI, Signal Application acquisition interface we must consider several component op- tions. Signal recordings can be either inva- Brain signal Control signal sive or noninvasive. Signal features require Brain-computer interface (BCI) analysis and classification methods. Con- Thought trol functions require selecting a suitable mental strategy as well as operational and feedback mechanisms. Closed-loop system Application Suitable brain signals Invasive BCI methods place electrodes di- rectly on or inside the cortex. One method records electrical potentials for subsequent analysis of the electrocorticogram (ECoG). Feedback (visual, auditory, haptic) Another method places a multiunit elec- trode array in the cortex to record the neu- Figure 1. The brain-computer interface: (a) schematic of principal BCI components ral firing of a small population of neurons. and (b) three applications: playing table tennis (top), using a spelling system Both signal types have a superior signal- (middle), and restoring grasp functions (bottom). to-noise ratio, need little user training, and are suitable for replacing or restoring lost resolution, which can be as fine as 2 to 3 corded (and possibly preprocessed in motor functions in patients with damaged millimeters. Researchers have studied BCIs suitable ways), the next step is to extract parts of the neuronal system. using MEG data, but they haven’t been able prominent features that describe impor- Noninvasive BCIs, on the other hand, to demonstrate significant advantages over tant discriminative signal properties. This can use a variety of brain signals as input, EEG-based systems. processing stage aims simply to reduce such as electroencephalograms (EEG), Unlike EEG and MEG systems, which data and adequately transform it such that magnetoencephalograms (MEG), blood- detect the electromagnetical activity of cor- the subsequent classification process is oxygen-level-dependent (BOLD) signals, tical neurons, near-infrared spectroscopy optimal. Example features used in EEG and (de)oxyhemoglobin concentrations. (NIRS) measures the metabolic activity of processing are the power in a specific fre- The EEG, which is basically the sum of specific cortical regions. NIRS uses light quency band (band power), autoregressive many postsynaptic potentials in the cortex, in the near-IR spectrum (typically between parameters, and synchronization measures. is the most widely used brain signal for op- wavelengths of 630 to 1,350 nm) to deter- erating a BCI system. We can extract two mine the oxygenation of the tissue, and Choosing types of changes from the ongoing EEG researchers have recently applied it to BCI the mental strategy signals: one is time- and phase-locked research. The potential advantages of real- Operant conditioning is a learning pro- (evoked) to an externally or internally izing a BCI with this technique include its cess with the goal of self-regulating brain paced event, while the other is also time- insensitivity to typical EEG artifacts such potentials (such as slow cortical potential locked but not phase-locked (induced). To as the electrooculogram (EOG), electro- shifts) or brain waves (such as sensorimo- the former class belong the event-related myogram (EMG), and electrode failures. tor rhythms) with the help of suitable feed- potentials (ERPs), including the P300, However, the technique also requires sev- back. This process doesn’t require continu- steady-state visual evoked potentials eral seconds to pass before it can measure ous feedback, but it does require a reward (Ssveps), and slow cortical negative shifts; the metabolic response, which is a long time for achieving the desired brain potential to the latter class belong the event-related compared to EEG and MEG. The spatial (wave). Researchers have used operant desynchronizations (ERDs) and event-re- resolution also lies in the centimeter range.3 conditioning to realize a communication lated synchronizations (ERSs). Like NIRS, functional magnetic reso- system for completely paralyzed (“locked- The MEG can measure brain activity nance imaging (fMRI) measures the in”) patients. by detecting weak magnetic fields caused metabolic changes in the brain. Based Another frequently used mental strategy by current flows in the cortex. These small on traditional MRI principles, the fMRI is motor imagery. Research results from magnetic fields in the picotesla to femto­ neuroimaging technique can also be used this strategy provide strong evidence that tesla range are measured with multichannel to control a BCI. To measure the hemody- motor imagery activates cortical areas Squid (superconducting quantum inter- namic response, fMRI studies usually use similar to those activated by executing the ference device) gradiometers in a shielded the BOLD signal. The stimulus response same movement. Consequently, we place environment. This technique combines time is in the range of some seconds.4 the EEG electrodes over the primary sen- excellent time resolution with good spatial After the brain signals have been re- sorimotor areas. When a user learns such a

May/June 2008 www.computer.org/intelligent 73 motor imagery task in a number of training this way, the computer learns to recognize shown the possibility of doing so. sessions, characteristic ERD/ERS patterns the users’ mental-task-related brain pat- How can brainwaves directly control ex- are associated with different types of motor terns. This learning process is highly sub- ternal devices? The current focus is mainly imagery and detectable in single trials in an ject-specific, so each user must undergo the on invasive approaches that provide de- online system. training individually. The learning phase tailed, single-neuron activity recorded from Other mental tasks besides motor imag- produces a classifier that we can use to clas- microelectrodes implanted in the brain.1 ery are suitable to modulate the brain sig- sify the brain patterns online and provide The motivation for invasive approaches is nals—for example, mental arithmetic and suitable feedback to the users. Visual feed- broad evidence that ensembles of neurons imaging the rotation of geometric objects. back has an especially high impact on the in the brain’s motor system—motor, premo- Focused attention or gaze control on visual dynamics of brain oscillations that can fa- tor, and posterior parietal cortex—encode stimuli, such as flickering lights or flashed cilitate or deteriorate the learning process. the parameters related to hand and arm letters, is especially suitable to realize The training phase is relatively short with movements in a distributed, redundant way. spelling devices with a P300-based BCI or P300 or SSVEPs, but can last weeks or For humans, however, noninvasive ap- to control neuroprostheses with a Ssvep- even months with mental tasks.2 proaches avoid health risks and associated based BCI. ethical concerns. Most noninvasive brain- computer interfaces (BCIs) use electroen- Self-based and References cephalogram (EEG) signals—electrical cue-based BCI systems 1. J.R. Wolpaw et al., “Brain-Computer In- brain activity recorded from electrodes on The mode of operation determines the type terfaces for Communication and Control,” the scalp. The EEG’s main source is the Clinical Neurophysiology, vol. 113, no. 6, of data processing, either in a predefined 2002, pp. 767–791. synchronous activity of thousands of corti- time window of some seconds following a 2. G. Pfurtscheller, C. Neuper, and N. Birbau- cal neurons. Thus, EEG signals suffer from cue stimulus (synchronous BCI) or continu- mer, “Human Brain-Computer Interface,” a reduced spatial resolution and increased ously sample-by-sample (asynchronous Motor Cortex in Voluntary Movements, noise when measurements are taken on the BCI). The cue might contain information A. Riehle and E. Vaadia, eds., CRC Press, scalp. Consequently, current EEG-based 2005, pp. 367–401. for users (for example, it might let them 3. S. Coyle et al., “On the Suitability of Near- brain-actuated devices are limited by low know whether they should imagine moving Infrared (NIR) Systems for Next-Genera- channel capacity and are considered too the left or right hand during training), or it tion Brain-Computer Interfaces,” Physi- slow for controlling rapid and complex se- might be neutral. In the latter case, the us- ological Measurement, vol. 25, no. 4, 2004, quences of robot movements. ers are free to choose one of the predefined pp. 815–822. Recently, however, my coworkers and 4. N. Weiskopf et al., “Principles of a Brain- mental tasks after the cue. I at the Idiap Research Institute and the Computer Interface (BCI) Based on Real- A synchronous BCI system is not avail- Time Functional Magnetic Resonance École Polytechnique Fédérale de Lausanne able for control outside the cue-based Imaging (fMRI),” IEEE Trans. Biomedical have shown for the first time that online processing window. In the asynchronous Eng., vol. 51, no. 6, 2004, pp. 966–970. EEG signal analysis, if used in combina- mode, no cue is necessary, so the system tion with advanced robotics and machine is continuously available to the users. learning techniques, is sufficient for hu- They can decide freely when they wish to mans to continuously control a mobile ro- Gert Pfurtscheller was professor of med- 2 3 generate a control signal. Such a system ical informatics and is head of the Brain- bot and a wheelchair. is more complex and demanding, and the Computer Interface Lab at the Graz Univer- great challenge is to maximize the inten- sity of Technology’s Institute for Knowledge Spontaneous EEG tional control (true positives) while mini- Discovery. Contact him at pfurtscheller@ and asynchronous operation tugraz.at. mizing the nonintentional control (false We can classify noninvasive EEG-based positives) at the output. We used such an Clemens Brunner is a postdoctoral re- BCIs as evoked or spontaneous. An evoked asynchronous BCI successfully to operate searcher at the Brain-Computer Interface BCI exploits a strong characteristic of the a spelling device and to navigate in a vir- Lab at the Graz University of Technology’s EEG, the evoked potential, which reflects tual environment. Institute for Knowledge Discovery. Contact the immediate automatic responses of the him at [email protected]. brain to some external stimuli. Examples of Organizing training evoked potentials include P300 and Ssvep and feedback (steady-state visual evoked potentials). In To employ a BCI successfully, users must Brain-Controlled Robots principle, evoked potentials are easy to de- first go through several training sessions to José del R. Millán, Idiap Research tect with scalp electrodes. However, evok- obtain control over their brain potentials Institute and École Polytechnique ing them requires external stimulation, so (waves) and maximize the classification ac- Fédérale de Lausanne they apply to only a limited task range. curacy of different brain states. In general, In my view, a more natural and suit- the training starts with one or two pre- The idea of moving robotic or prosthetic able alternative for interaction begins with defined mental tasks repeated periodically devices not by manual control but by mere analyzing components associated with in a cue-based mode. In predefined time “thinking”—that is, by human brain activ- spontaneous, intentional mental activity. windows after the cue, we record the brain ity—has fascinated researchers for the past This is particularly the case for controlling signals and use them for offline analyses. In 30 years. But only now have experiments robotics devices. As in driving a car, the

74 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS subject’s attention must focus on driving the performance was only marginally bet- and not on external stimuli. ter than the mental performance. Spontaneous BCIs are based on the anal- More recently, we extended this work to ysis of EEG phenomena associated with the mental control of both a simulated and various aspects of brain function related to a real wheelchair (see figure 2).3 We per- mental tasks that the subject carries out at formed this work in the framework of the will. For example, the subject might imag- European project MAIA (Augmentation ine limb movements, such as the right or through Determination of Intended Action, left hand, or cognitive operations, such as www.maia-project.org) and in cooperation arithmetic or language. with Katholieke Universiteit Leuven. In But volunteer mental control isn’t enough this case, we incorporated shared-control for steering a wheelchair or a prosthesis. principles to blend the two intelligences.7 These tasks require subjects to also make Although our first brain-actuated robot had self-paced decisions. In such asynchronous a form of cooperative control, shared con- protocols, the subject can deliver a mental trol is a more principled, flexible framework command at any moment without waiting for and gives users a finer degree of control. external cues.2,4 This contrasts with synchro- nous interaction, where the EEG is time- Challenges and locked to externally paced cues. Only asyn- future research directions chronous controls can send the appropriate For brain-actuated robots, in contrast to mental command at the right time to make augmented communication through BCI, the wheelchair turn and cross the desired Figure 2. A brain-actuated wheelchair. fast decision making is critical. In this doorway while it’s moving continuously. The subject guides the wheelchair sense, real-time control of brain-actuated through a maze, using a BCI that devices, especially robots and neuropros- The statistical recognizes the subject’s intent from theses, is the most challenging BCI applica- machine learning way analysis of noninvasive EEG signals. tion. While researchers have demonstrated (photo courtesy of the MAIA project) Training is a critical BCI development is- brain-actuated robots in the laboratory, the sue—that is, how do users learn to operate technology isn’t yet ready for use in real- the BCI? Like other groups,5,6 we follow a wrong turns or bring back the wheelchair world situations. We still need to improve mutual-learning approach to facilitate and to the desired doorway. the BCI’s robustness to make it a more accelerate the user’s training period. The practical and reliable technology. user and the BCI are coupled together and A blending of intelligences A first line of research is online adapta- adapt to each other. In other words, we use How is it possible to control a robot that tion of the interface to the user to keep the machine learning approaches to discover must make accurate turns at precise mo- BCI constantly tuned to its owner.8 This the individual EEG patterns characterizing ments using signals that arrive at a rate of would account for the new capabilities— the mental tasks users execute while learn- about one bit per second? and corresponding new brain signals—that ing to modulate their brainwaves in a way The key aspect of our brain-actuated subjects gain with experience. In addition, that will improve system recognition of robots is combining the subject’s mental brain signals change naturally over time. In their intentions. capabilities with the robot’s intelligence. particular, they can change from one session We use statistical machine learning tech- That is, the subject delivers a few high-level that supplies the data to train the classifier niques at two levels: selecting the features mental commands (for example, “Turn to the next session that applies the classifier. and training the classifier embedded in the right at the next occasion”), and the robot Online learning can help adapt the classifier BCI. In particular, the statistical classifier executes these commands autonomously throughout its use and keep it tuned to drifts achieves error rates below 5 percent for using the readings of its onboard sensors. In in the signals it receives in each session. three mental tasks, but correct recognition other words, the EEG conveys the subject’s The second line is the analysis of neural is 70 percent. In the remaining cases, the intent, and the robot performs it to generate correlates of high-level cognitive and affec- classifier doesn’t respond because it consid- smooth, safe trajectories. tive states such as errors, alarms, attention, ers the EEG samples to be uncertain. This approach makes it possible to con- frustration, and confusion. The EEG has Incorporating rejection criteria to avoid tinuously control a mobile robot—emu- information about these states embedded in making risky decisions is an important lating a motorized wheelchair—along it, together with the mental commands in- BCI concern. From a practical viewpoint, nontrivial trajectories requiring fast and tentionally generated by the user. The abil- a low classification error is a critical BCI frequent switches between mental tasks.2 ity to detect and adapt to these states would performance criterion. Otherwise, users In a few days, two human subjects learned enable the BCI to interact with the user in a can become frustrated and stop using it. to mentally drive a robot between rooms in much more meaningful way. One of these Furthermore, not executing probable wrong a house-like environment and visit three or high-level states is the awareness of errone- commands increases the BCI’s theoretical four rooms in a prescribed order. Further- ous responses. The neural correlate for this bit rate and improves the robot’s trajec- more, when the subjects later controlled the awareness arises in the millisecond range, tories. The subject won’t need to correct robot manually along the same trajectories, so user commands are executed only if no

May/June 2008 www.computer.org/intelligent 75 error is detected in this short time frame. José del R. Millán is an adjunct professor conventional handheld controls and control Recent results have shown satisfactory sin- at the Swiss Federal Institute of Technology special features through a BCI. gle-trial error recognition that significantly in Lausanne (EPFL) and a senior researcher New BCI subjects sometimes perform improves BCI performance.9 In addition, at the Idiap Research Institute. Contact him effectively within about 10 minutes despite at [email protected]. this new type of error potential—which is background distraction and electrical noise, generated in response to errors made by the but researchers haven’t yet studied the ef- BCI rather than by the user—can provide fects of intensive usage as might occur in performance feedback that, in combination Why Use a BCI gamers.1–3 Nor have they fully studied the with online adaptation, improves the BCI If You’re Healthy? precision and timing of translating user in- while it’s being used. Brendan Allison and Bernhard Graimann, tent into control signals through BCIs. University of Bremen Typical research BCIs allow communi- cation only via electrodes and so exhibit Most brain-computer interface (BCI) very low bandwidth. Hybrid interfaces Acknowledgments research focuses on restoring commu- could combine BCIs with other interfaces The Swiss National Science Foundation sup- 1,2 ported this work through the National Centre of nication for severely disabled users. to provide an additional independent signal Competence in Research on Interactive Multi- However, BCIs could also treat disabili- or modify other commands,1 which might modal Information Management and also by ties such as stroke, autism, epilepsy, or allow moving while crouching, dodging, the European Information Society Technologies emotional disorders, and they might even firing, communicating, spellcasting, and/or Programme, Future and Emerging Technolo- 3,4 gies Project FP6-003758. The article reflects become useful to healthy users. At pres- mentally levitating an object. only the author’s views, and funding agencies ent, BCIs have several serious drawbacks The BCI “distraction quotient” is un- aren’t liable for any use that might be made of relative to conventional interfaces such as known in these scenarios. How can BCIs the information it contains. keyboards or mice. They’re much slower, best be integrated with other interfaces? less accurate, and operational only at very Which BCIs work best with other interfaces, low bandwidths. They require cables and environments, and games? How do these is- References unfamiliar, expensive hardware, including sues vary across users with different person- 1. J.M. Carmena et al., “Learning to Control an electrode cap. The cap requires hair alities, backgrounds, motivations, abilities, a Brain-Machine Interface for Reaching gel and several minutes of preparation and experience, training, and other characteris- and Grasping by Primates,” PLoS Biology, vol. 1, no. 2, 2003, pp. 193–208. cleanup. Some BCIs require training, are tics? These questions will become increas- 2. J.d.R. Millán et al., “Noninvasive Brain- difficult to use, and fail with some sub- ingly important as pressure to build a practi- Actuated Control of a Mobile Robot by Hu- jects or in noisy environments. BCIs often cal BCI mounts from commercial sources. man EEG,” IEEE Trans. Biomedical Eng., seem intimidating, exotic, Orwellian, or vol. 51, no. 6, 2004, pp. 1026–1033. even nerdy. They rarely show up in main- Induced disability 3. F. Galán et al., “An Asynchronous and stream markets, and this won’t change Healthy users might communicate via BCIs Non-Invasive Brain-Actuated Wheelchair,” Proc. 13th Int’l Symp. Robotics Research, soon. when conventional interfaces are inad- 2007, pp. 45–54. Hence, the prevailing view is that BCIs, equate, unavailable, or too demanding. Sur- 4. J.d.R. Millán, “Adaptive Brain Interfaces,” at best, enable people to send the same geons, mechanics, soldiers, cell phone users, Comm. ACM, vol. 46, no. 3, 2003, pp. information available much more quickly drivers, and pilots can experience induced 74–80. and easily via other interfaces. This per- disability when hand or voice communi- 5. B. Blankertz et al., “The Berlin Brain- spective is wrong. Here, we’ll discuss why cation is infeasible. BCIs might help them Computer Interface: Machine Learning Based Detection of User Specific Brain healthy people might eventually use BCIs request tools, navigate maps or schematics, States,” J. Universal Computer Science, in specific situations. We’ll consider BCIs access data, or perform otherwise difficult, vol. 12, no. 6, 2006, pp. 581–607. with scalp-mounted electrodes because distracting, dangerous, or impossible tasks. 6. G. Pfurtscheller and C. Neuper, “Motor other neuroimaging approaches are typi- Hybrid interfaces could also help when Imagery and Direct Brain-Computer Com- cally impractical.1,3,5 conventional interfaces provide insufficient munication,” Proc. IEEE, vol. 89, no. 7, 2001, pp. 1123–1134. bandwidth. Expert gamers often use many 7. D. Vanhooydonck et al., “Shared Control BCIs for healthy users keys at once. Console games require us- for Intelligent Wheelchairs: An Implicit A few BCI R&D projects envisioned ing several fingers on both hands. A major Estimation of the User Intention,” Proc. 1st healthy subjects as end users. Modern BCI benchmark will be the first BCI that reli- Int’l Workshop Advances in Service Ro- simulations or games usually allow one ably provides supplemental information botics, Fraunhofer IRB Verlag, 2003, pp. 176–182. or two degrees of freedom or 1D to 2D without impairing mainstream interface 8. J.d.R. Millán et al., “Adaptation in Brain- graded control. Turning, moving, or lean- performance. Computer Interfaces,” Towards Brain- ing are often possible, sometimes in a vir- Computer Interfacing, G. Dornhege et al., tual environment. For example, research- Ease of use in hardware eds., MIT Press, 2007, pp. 303–325. ers have demonstrated BCIs intended to The keyboard and mouse seem like natu- 9. P.W. Ferrez and J.d.R. Millán, “Error-Re- let healthy users navigate maps while their ral, intuitive, convenient interfaces—when lated EEG Potentials Generated during 6,7 Simulated Brain-Computer Interaction,” hands are busy. Game companies such expert users just happen have them handy. IEEE Trans. Biomedical Eng., vol. 55, no. as NeuroSky and Emotiv advertise games Users who wear electroencephalography 3, 2008, pp. 923–929. that allow people to move a character with (EEG) sensors might find BCIs easier to

76 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS use. EEG sensor technology is becoming regional attention. Software might magnify, Healthy target markets more practical.1 New electrodes require link, remember, or jump to interesting areas Most healthy BCI users today are research little or no gel, scalp contact, or prepara- of the screen or auditory space. EEG-based scientists, friends, research subjects, and vis- tion and cleanup time. As electronics and assessment of global attention, frustra- itors at expositions. A few people order com- signal processing improve, smaller, better, tion, alertness, comprehension, exhaustion, mercial BCIs, forming a crucial fifth cat- cheaper sensors and amplifiers could oper- or engagement could enable software that egory in which no BCI expert prepared the ate with devices or clothing on or near the adapts much more easily to the user. The software or hardware for individual users. head. Bluetooth, the ubiquitous wireless challenge of developing new opportunities Gamers are likely early adopters. They Internet, and related technologies facilitate for integrating BCI-based signals into con- often wear headgear, enjoy novelty and wireless BCIs. BCIs might eventually be- ventional and emerging operating systems technical challenges, have money and time come more convenient and accessible than might be as fun as Douglas Engelbart’s available for peripherals and training, and cell phones, watches, remote controls, or daunting task of integrating the mouse into are competitive and increasingly numerous. car dashboard interfaces. a world then dominated by keyboards. Specific military or government person- Laziness is the wayward child of inven- nel follow technology validated elsewhere. tion. Laziness can induce disability, and it Improved training or performance Highly specialized users such as surgeons, can be very motivating. Although televi- Some BCIs train subjects to produce spe- welders, or mechanics are also likely sec- sions have viable interfaces, people typi- cific activity over sensorimotor areas, so ond-generation adopters. Electrooculograms, cally prefer more portable alternatives that BCI training might improve movement electromyograms, electrocardiograms, and provide no advantage except remote control. training or performance. Subjects’ athletic other signals might supplement EEG control BCIs could also help people who retype and motor background and skills might in many BCI and related applications. words or sentences (rather than cut and influence BCI parameters. These avenues More mainstream applications, such as paste via mice) by letting them instead se- might be useful for motor rehabilitation or error correction hybridized with word pro- lect, drag, or click via the BCI, thus avoid- finding the right BCI for each user.3,4 cessors, are more distant. These approaches ing temporarily disengaging from the key- require new software development, much board. BCIs could allow sending messages Confidentiality better EEG sensors, and encouraging vali- without the hassle of a keyboard, micro- BCIs might be the most private communica- dation. BCIs might instead seem unreliable, phone, or cellphone numberpad. Humanity tion channel possible. With other interfaces, useless, unfashionable, dangerous, intru- might finally escape the various inconve- eavesdropping simply requires observing sive, or oppressive, spurred by inaccurate niences of finding handheld interfaces or the necessary movements. This important reporting. Websites such as bci-info.org, pressing buttons. security problem also shows up in competi- proper dissemination of results, and positive tive gaming environments. For example, appearances at conferences, expositions, in- Ease of use in software many console gamers have chosen an offen- terviews, or other events can educate people The activities that control most BCIs and sive football play, then noticed an adjacent and reduce miscommunication. conventional interfaces differ fundamen- opponent select a corresponding defensive BCIs won’t soon replace conventional in- tally from desired outputs. Noticing flashes play after overt peeking. terfaces, but they might be useful to healthy or moving fingers across a keyboard isn’t users in specific situations. Integrating them like natural communication. However, Speed with other interfaces raises many questions some BCIs allow walking or turning by Relevant EEGs are typically apparent one best addressed with parametric research in­ imagining foot or hand movements,2,7 and second before a movement begins and volving different users, interfaces, mental these might offer new frontiers of usability might precede the decision to move.1 Future activities, goals, output devices, and train- for all users. As with other interfaces, re- BCIs might be faster than natural pathways. ing parameters. search should address which mental activi- Further research should provide earlier ties seem most natural, easy, and pleasant movement prediction with greater preci- References for different users in different situations. sion and accuracy, integrate predicted with 1. B.Z. Allison, E.W. Wolpaw, and J.R. Wol­ actual movements smoothly, and evaluate paw, “Brain-Computer Interface Systems: Progress and Prospects,” Expert Rev. of Otherwise unavailable information training and side effects. Medical Devices, vol. 4, no. 4, 2007, pp. Available interfaces have heavily influenced 463–474. all software. Operating systems would look Novelty 2. G. Pfurtscheller et al., “15 years of BCI Re- very different if eye trackers and voice Some people might use a BCI simply be- search at Graz University of Technology: commands were the dominant interfaces. cause it seems novel, futuristic, or excit- Current Projects,” IEEE Trans. Neural Sys- tems and Rehabilitation Eng., vol. 14, no. Just as keyboards and mice are inherently ing. This consideration, unlike most others, 2, 2006, pp. 205–210. suited to typing and dragging, BCIs are in- loses steam over time. BCIs will become 3. N. Birbaumer and L.G. Cohen, “Brain Com- herently better suited to certain tasks. The more flexible, usable, or better hybridized as puter Interfaces: Communication and Res- error-related negativity and P300 that often research continues. However, as BCIs im- toration of Movement in Paralysis,” J. Phys- develop after a subject recognizes a mis- prove, public perception will follow a pattern iology, vol. 579, pt. 3, 2007, pp. 621–636. take could allow real-time error recogni- reminiscent of microwaves and cell phones. 4. B. Graimann, B.Z. Allison, and A. Gräser, “New Applications for Non-invasive Brain- 1 tion. The P300, steady-state visual evoked BCIs will first be exotic, then novel, wide- Computer Interfaces and the Need for En- potential (Ssvep), and other signals reflect spread, unexceptional, and finally boring. gaging Training Environments,” Proc. Int’l

May/June 2008 www.computer.org/intelligent 77 Conf. Advances in Computer Entertain- EEG seems for now the only practical BCI illiteracy ment Technology, ACM, 2007, pp. 25–28. brain-machine interaction choice (cost and A long-standing problem of BCI designs 5. J.R. Wolpaw et al., “BCI Meeting 2005: ITR limitations hamper other noninvasive that detect EEG patterns related to a volun- Workshop on Signals and Recording Meth- methods). As such, we ask here not how tarily produced brain state is that such para- ods,” IEEE Trans. Neural Systems and Re- habilitation Eng., vol. 14, no. 2, 2006, pp. further signal-processing and machine- digms work with varying success among 138–141. learning improvements might increase the different subjects or patients. We distinguish 6. L.J. Trejo, R. Rosipal, and B. Matthews, ITR.1,2 BCI researchers already know that mental-task-based BCI, such as “movement “Brain-Computer Interfaces for 1D and 2D many complex technical problems remain: imagination” BCI, from paradigms based Cursor Control: Designs Using Volitional such problems have been the field’s main on involuntary stimulus-related potentials Control of the EEG Spectrum or Steady- State Visual Evoked Potentials,” IEEE concern up to now. Nor will we will discuss such as P300. These stimulus-related poten- Trans. Neural Systems and Rehabilitation EEG-BCI applications. Instead, we concen- tials are limited to very specific applica- Eng., vol. 14, no. 2, 2006, pp. 225–229. trate on outlining the challenges that remain tions, such as typing for locked-in patients, 7. R. Scherer, G.R. Müller-Putz, and G. in adapting EEG-BCI from the laboratory and they require constant focus on stimuli Pfurtscheller, “Self-Initiation of EEG- to real-world use by healthy subjects. extraneous to the task at hand. Based Brain-Computer Communication Using the Heart Rate Response,” J. Neural In a recent study, with 10 untrained us- Eng., vol. 4, 2007, pp. L23–L29. Dry electrodes ers,2 our research group took a close look The most elementary EEG-BCI challenge at how fast the users achieved their best for healthy users isn’t—at first glance—a performance (by skill acquisition) during Brendan Allison is a researcher at the Uni- computational one. Standard EEG practice a small number of BCI sessions and how versity of Bremen’s Institute of Automation. Contact him at [email protected]. involves the tedious application of con- much this performance varied among sub- ductive gel on EEG electrodes to provide jects. We confirmed the results in a follow- Bernhard Graimann is a researcher at accurate measurements of the microvolt- up study with 13 novice subjects.4 Although the University of Bremen’s Institute of Au- level scalp potentials that constitute EEG machine learning techniques allow use of tomation. Contact him at graimann@iat. uni-bremen.de. signals. Without “dry-cap” technology, minimal calibration data recording (< 20 the proper set-up of BCI sessions in, say, minutes) before the BCI system is ready to a home environment, is too tedious and use, the subjects’ peak-performance pla- messy to be practical. Some dry electrode teaus, even after multiple sessions, varied Computational Challenges designs that use a combination of EEG greatly. Using this and other unreported for Noninvasive Brain and electromyogram (EMG) have been an- data by many research groups, we estimate Computer Interfaces nounced for home entertainment use. The that Florin Popescu, Fraunhofer Institut für EMG originates from body and face mus- Rechnerarchitektur und Softwaretechnik cles; in BCI studies, it’s considered an arti- • about 20 percent of subjects don’t show (First) Berlin fact. Although EMG is stronger and easier strong enough motor-related mu-rhythm Benjamin Blankertz and Klaus-R. Müller, to read than EEG, it doesn’t truly constitute variations for effective asynchronous Berlin Institute of Technology a mental interface. Our research group has motor-imagery BCI, developed an EEG-BCI dry-cap design and • another 30 percent exhibit slow perfor- Electroencephalography (EEG) is unique tested its performance (and the absence of mance (< 20 bits per minute), and among functional brain-imaging methods muscle artifacts) in a controlled study.3 • up to 50 percent exhibit moderate to high in that it promises a means of providing a For ease-of-use and cost reasons, all performance (20–35 bits/min.). cost-efficient, safe, portable, and easy-to- foreseeable systems will use fewer elec- use brain-computer interface (BCI) for trodes than found on standard EEG caps It’s still a matter of debate as to why BCI both healthy users and the disabled. An today. The computational challenges we’ve systems exhibit “illiteracy” in a significant already-extensive corpus of experimental addressed include optimal placement of the minority of subjects and what can be done work has demonstrated that, to a degree, reduced number of electrodes and robust- about it in terms of signal processing and EEG-based BCI can detect a person’s men- ness of BCI algorithms to the smaller set machine learning algorithms. From inter- tal state in single trials of mental imagi- of recording sites. With only six unipolar nal investigations (as well as the results of nation using sophisticated mathematical electrodes, we can achieve about 70 percent BCI Competition II, data set Ib5), BCI illit- tools; but this work has also outlined clear of full-gel-cap BCI performance at sites eracy in a subject appears to depend not so challenges. The first challenge is the rather above the motor cortex, while being able to much on the algorithm used but on a prop- limited information transfer rate (ITR) discount any potential influence of muscle erty inherent in the subject. achievable through EEG, which is—in the and eye movement artifacts. EEG is sensitive to sources in cortical most optimistic of cases—about an order of Most other remaining dry-cap challenges folds, so it might not be able to read motor- magnitude lower than invasive BCI meth- are of an engineering design nature, exclud- imagery activity in some subjects because ods currently provide. That said, the po- ing perhaps the computational reduction of the particular cortical region involved is tential benefits of brain implant-based BCI artifacts produced not by unrelated electro- tangential to the scalp. An observation con- haven’t yet proved worth the associated cost physiological activity but by measured low- sistent with this explanation is that in certain and risk in the most disabled patients, let frequency voltage variations caused by the subjects some classes—that is, types of alone healthy users. head’s physical movement. imagined movements—are detectable and

78 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS others not. Calibration sessions should there- offers a clear trade-off between high idle- References fore select subject-specific classes along with class accuracy (that is, the thresholds are 1. B. Blankertz et al., “The Noninvasive frequency bands necessary for feature gen- high) and fast speed of response or high Berlin Brain-Computer Interface: Fast Ac- quisition of Effective Performance in Un- eration to minimize the illiteracy problem. ITR (the thresholds are low). Remaining trained Subjects,” NeuroImage, vol. 37, no. challenges are to find a classifier that can 2, 2007, pp. 539–550. Idle class induce a rest state without a relax cue and 2. G. Dornhege et al., eds., Toward Brain-Com- Most commonly, BCI controllers involve to optimize the relationship between clas- puter Interfacing, MIT Press, 2007, p. 83. two classes, which can move a monitor- sifier output and BCI command. Because 3. F. Popescu et al., “Single Trial Classifica- displayed cursor toward, say, left and right. of physiological variations in background tion of Motor Imagination Using Six Dry EEG Electrodes,” PLoS ONE, vol. 2, 2007, Although these controllers can perform EEG activity, where fatigue is a main fac- p. e637. asynchronously—that is, at their own in- tor, we believe an adaptive classifier and 4. B. Blankertz et al. “The Berlin Brain- dependent pace—useful cursor control is controller are necessary for maximal per- Computer Interface: Accurate Performance difficult. The user must either continuously formance. Our group has undertaken some from First-Session in BCI-Naive Subjects,” imagine one of the two classes or lose con- efforts toward optimizing a true idle-state to be published in IEEE Trans. Biomedical Eng., 2008. trol of the cursor. BCI paradigm by balancing idle-class ac- 9 5. B. Blankertz et al., “The BCI Competition Besides self-pacing, BCI would greatly curacy and ITR. 2003: Progress and Perspectives in Detec- benefit from integrating an “idle” or “rest” tion and Discrimination of EEG Single Tri- class with the BCI’s active classes—that Future challenges als,” IEEE Trans. Biomedical Eng., vol. 51, is, those corresponding to mentally imag- and implementations no. 6, 2004, pp. 1044–1051. ining a particular task and implying the While these three computational challenges 6. J.d.R. Millán and J. Mouriño, “Asynchro- nous BCI and Local Neural Classifiers: An desire to transmit the activation of a cor- are, we believe, the most urgent, other im- Overview of the Adaptive Brain Interface responding command. This would keep provements might also be beneficial. Al- Project,” IEEE Trans. Neural System Re- the cursor from responding when no ac- though 20 minutes of calibration for a novel habilitation Eng., vol. 11, no. 2, 2003, pp. tive class (from a set of two or more) is subject isn’t excessive, usability would ben- 159–161. activated. efit from knowing the minimal number of 7. J.F. Borisoff et al., “Brain-Computer Inter- face Design for Asynchronous Control Ap- The idle state might take one of two calibration trials needed to achieve moder- plications: Improvements to the LF-ASD forms: a relax state, where the subject stays ate performance and rule out BCI illiteracy, Asynchronous Brain Switch,” IEEE Trans. still and tries to “think of nothing,” or a such that a classifier can then adapt to the Biomedical Eng., vol. 51, no. 6, 2004, pp. state where the subject can do almost any user during normal use. For applications 985–992. mental task other than those belonging to such as gaming, or voluntary self-paced 8. G.R. Muller-Putz et al., “Brain-Computer the active classes. In the case of deliberate interaction with an unstructured environ- Interfaces for Control of Neuroprostheses: From Synchronous to Asynchronous Mode relaxation, usability is obviously limited, ment, this adaptation should work even in of Operation,” Biomedizinische Technik although signal processing is easier, given cases where class labels aren’t available (Berl), vol. 51, 2006, pp. 57–63. that relaxation tends to increase EEG power (unsupervised adaptation).10 9. S. Fazli et al., “Asynchronous, Adaptive in the alpha band. For example, research- We envisage an EEG BCI scenario in BCI Using Movement Imagination Train- ers have shown that alpha band modulation which users purchase an affordable com- ing and Rest-State Inference,” Proc. Arti- ficial Intelligence and Applications (AIA played a strong role in detecting relaxation puter peripheral that is simply placed on the 08), ACTA Press, 2008, pp. 85–90. when subjects closed their eyes during an head and requires no gel. New users will 10. M. Krauledat et al., “Reducing Calibra- idle state.6 undergo a one-time calibration procedure tion Time for Brain-Computer Interfaces: Relying on alpha power modulation is that takes maximally 10 minutes, ideally A Clustering Approach,” Proc. Advances complicated by the involuntary variation even less. They then proceed to use the in Neural Information Processing Systems (NIPS 06), vol. 19, MIT Press, 2007, pp. of background alpha in physiological as BCI system in a game environment to, for 753–760. opposed to experimental conditions—for example, control a robot or wheelchair. The example, due to fatigue. Furthermore, re- system’s performance slowly adapts to the Florin Popescu is a research scientist at laxing itself induces drowsiness. user’s brain patterns, reacting only when Fraunhofer First’s Intelligent Data Analysis A neurofeedback-style, low-frequency he or she intends to control it. At each re- Group. Contact him at florin.popescu@first. modulation approach has shown promise peated use, the system recalls parameters fraunhofer.de. as an idle-state paradigm, but it requires from previous sessions, so recalibration is Benjamin Blankertz is a researcher in the intensive subject training, exhibits lim- rarely, if ever, necessary. Machine Learning Laboratory at the Berlin ited ITR, and has only one active class.7 We strongly believe such a system, ca- Institute of Technology and in the Intelligent The Graz group has begun work toward pable of an average performance of about Data Analysis group at Fraunhofer First. idle-state control with a relax cue,8 but so 15 to 20 bits/min, is achievable within the Contact him at [email protected]. far there is little hard data on idle-state next few years. Challenges such as BCI il- Klaus-R. Müller is a professor and chair duration and accuracy. This is important, literacy are likely to be only partially met. of the Machine Learning Department at because two-class classifier output (a noisy Still, if this percentage decreases further, the Berlin Institute of Technology and head signal) is usually integrated until it hits a it shouldn’t prevent noninvasive BCI sys- of the Intelligent Data Analysis group at Fraunhofer First. Contact him at krm@ threshold (for example, left or right cur- tems from reaching a large user population, cs.tu-berlin.de. sor movement). The level of this threshold healthy or disabled.

May/June 2008 www.computer.org/intelligent 79