Eduardo Reck Miranda,* Ken Sharman,† On Harnessing the Kerry Kilborn,‡ and Alexander Duncan§ *Computer Music Research—Neuroscience of Electroencephalogram Music Group, School of Computing, Communications, and Electronics, for the Musical Braincap University of Plymouth, Drake Circus, Plymouth, Devon PL4 8AA, United Kingdom [email protected] † Instituto Tecnolo´ gico de Informa´tica Universidad Polite´cnica de Valencia, Camino de Vera s/n, 46071 Valencia, Spain [email protected] ‡ Department of Psychology University of Glasgow, 58 Hillhead Street, Glasgow G12 8QB, United Kingdom [email protected] § The Sun Centre Prades, 48160 St. Martin de Boubaux, France [email protected] Before you can be fitted with your Braincap, face for a system whereby a monkey controlled a you have to be completely bald. A faint cursor on a computer screen (Turner 2002). At first, drumming sound accelerated until it became the monkey used a joystick to move the cursor. the lowest of audible Cs, then raced up the After a while, the joystick was disconnected, and musical scale until it disappeared beyond the the monkey, who had not realized this, continued range of the human hearing. He presumed moving the cursor by means of tiny electrical sig- that his neuromuscular control was being nals emanating from an electrode implanted on the tested. (Clarke 1997) monkey’s motor cortex (the main brain area for motor control). The braincap, as described in 3001: The Final Od- yssey, the concluding edition of Arthur C. Clarke’s We are interested in developing thought- science fiction classic, is the ultimate human- controlled musical devices, and to this end we are computer interface: it connects the brain to a sys- currently working on the design of a musical brain- tem that is able to read thoughts and upload new cap. We are developing technology to interface the information. The wearer can in minutes acquire new brain with music systems and compositional tech- skills that would otherwise take years to master. niques suitable for thought control. Currently, however, a system that uploads infor- This article focuses on extracting and harnessing mation into the brain cannot exist outside the tiny electrical brain signals from electroencephalo- realm of science fiction, although machines that grams (EEGs) that can be captured with electrodes can read signals from the brain are becoming pres- on the scalp. We present three experiments whose ent-day reality. Furthermore, we should soon be results provide the basis for building systems to able to control all sorts of devices by our thoughts automatically detect information in the electroen- alone. In 1998, a paper presented at the 9th Euro- cephalogram associated with musical mental activ- pean Congress of Clinical Neurophysiology already ities. Then, we describe how these results are reported impressive advances in research on an currently being embedded in the design of the mu- electroencephalogram-based system to control a sical braincap. Before we present the experiments, prosthetic hand (Guger and Pfurtscheller 1998). we briefly introduce the growing field of Brain- More recently, scientists at Brown University re- Computer Interfaces (BCI), followed by an introduc- ported the development of a brain-computer inter- tion to the EEG and the signal processing techniques we employed to harness it. Computer Music Journal, 27:2, pp. 80–102, Summer 2003 Before we continue, it is necessary to clarify the ᭧ 2003 Massachusetts Institute of Technology. meaning of the expression ‘‘thought control.’’ In 80 Computer Music Journal Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892603322022682 by guest on 02 October 2021 this context, thought control should not evoke the User-Oriented Systems idea of a person imagining a specific piece of music In user-oriented BCI systems, the computer adapts that is then magically generated exactly as imag- to the user. Metaphorically speaking, these systems ined. This is beyond the capabilities of current sci- attempt to ‘‘read’’ the mind of the user to control a ence and technology. By thought control, we mean device. For example, Anderson and Sijercic (1996) simply using brain signals associated with specific reported on the development of a BCI controller mental activities to interact with musical devices. that learns how to associate specific EEG patterns The sophistication of this interaction will depend from a subject to commands for navigating a upon the nature of the mental activities that one is wheelchair. The prosthetic hand and the monkey able to identify in the EEG, the efficiency of the experiment mentioned earlier also fit into this EEG signal-processing techniques employed, and category. above all, the design of the system in question. We are by no means proposing the naı¨ve scenario of a Computer-Oriented Systems ‘‘high-tech Mozart’’ who would simply imagine music that the technology then creates. With computer-oriented BCI systems, the user adapts to the computer. These systems rely on the capacity of the users to learn to control specific as- pects of their EEG, affording them the ability to ex- Brain-Computer Interfaces and Music ert some control over events in their environments. Examples have been shown where subjects learn Generally speaking, a brain-computer interface how to steer their EEG to select letters for writing (BCI) is a system that allows one to interact with a words on the computer screen (Birbaumer et al. certain device by means of signals emanating di- 1999). rectly from the brain. There are basically two ways of tapping brain signals: invasively and non- Mutually-Oriented Systems invasively. Whereas invasive methods require the Finally, mutually-oriented BCI systems combine placement of sensors connected to the brain inside the functionalities of both categories, where the the skull, non-invasive methods use sensors that user and computer adapt to each other. The com- can read brain signals from the outside the skull. bined use of mental task pattern classification and Invasive technology is becoming increasingly so- biofeedback-assisted online learning allows the phisticated, but brain prosthetics is not a viable op- computer and the user to adapt. Prototype systems tion for this research. The most viable non-invasive to move a cursor on the computer screen have been option for tapping the brain for BCI currently is the developed in this fashion (Peters, Pfurtscheller, and EEG. It is a well-known phenomenon that brain ac- Flyvberg 1997; Penny et al. 1999). Co-evolving sys- tivity produces a range of electrical signals in the tems of humans and computers belong in this cate- cerebral cortex, generating electrical fields that can gory. be captured using electrodes placed on the scalp. These signals are generally referred to as the EEG. BCI Music Controllers There is a growing number of people developing EEG-based BCI, as could be witnessed at the first To date, most efforts of BCI research have been international meeting Brain-Computer Interface aimed at developing ways to help severely impaired Technology: Theory and Practice, held at the New people communicate via computer systems and/or York State Department of Health, in June 1999. control mechanical tools, such as a wheelchair or a (Unfortunately, no proceedings were published.) prosthetic organ. However, very little has been It is possible to identify three categories of BCI done to address the use of BCI technology for musi- systems: user-oriented, computer-oriented, and cal applications; such applications could undoubt- mutually-oriented. edly improve the life quality of physically impaired Miranda, Sharman, Kilborn, and Duncan 81 Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892603322022682 by guest on 02 October 2021 people in many forms, ranging from entertainment was really detecting shift of attention in the EEG. to therapy. In the case of Mr. Rosenboom’s work, however, Those who have attempted to employ EEG as this was not necessarily a problem. On the con- part of a music controller have done so by associat- trary, his objective was to allow the performing ing certain EEG characteristics, such as the power subject’s EEG to influence the evolving musical of the EEG alpha waveband to specific musical ac- forms, regardless of whether they were always tions. These are essentially computer-oriented sys- aware of the events. All the same, his work is un- tems, as they require the user to learn to control doubtedly a landmark in the field of BCI for musi- their EEG in a certain way. This is very difficult to cal applications, as it indicates that the notion of achieve without appropriate training. An effective thought-controlled musical systems is indeed pos- method for learning to achieve specific mental sible. The core idea is to control or influence gener- states is based upon the notion of biofeedback. Bio- ative musical processes using EEG information feedback is a therapeutic technique whereby pa- about the musical experience of a performer during tients are trained to improve their condition by the unfolding of the piece. The sophistication of altering body functions that are involuntary, such such a system is largely dependent upon its ability as blood pressure, body temperature, and EEG (Rob- to harness the EEG signal and to devise suitable bins 2000). generative music strategies. We believe that we can The idea of thought-controlled music can be push Mr. Rosenboom’s initial ideas much further traced back to the 1960s, when Alvin Lucier com- by taking the progress made in the last decade in posed Music for Solo Performer, a piece for percus- the fields of artificial intelligence, EEG analysis sion instruments played by the vibrations produced technology, and digital signal processing. from the performer’s EEG (Lucier 1976). Lucier Several commercial systems can play music from placed electrodes on his scalp, amplified the sig- EEG data.