Facing page: Peter Driessen PEng in his University of Victoria laboratory testing the response Designing New Musical Instruments characteristics of the radio drum, an electronic musical instrument that detects the performer’s The Artist/Engineer Collaboration movements to control and trigger sound (photo: UVic). Left: Andrew Schloss performs on the radio drum in his home studio using /MSP, a language designed W Andrew Schloss PhD hat was the first musical instrument? We may never for real-time music programming (photo: Irene Mitri). Peter Driessen PEng PhD know the answer, but we do know that music is University of Victoria W much older than recorded history — earliest archeological musical instrument specimens go back 30,000 years or more. Primitive early musical instruments were more discoveries than inventions, but for thousands of years they have evolved to what we now know as modern instruments.Have musical instruments stopped sound via gestures, and the range of ex- pression is determined by the complex- evolving? Highly unlikely — if anything, the pace has increased. ity available in each of the above three components. Fully decoupling the gesture from During the course of human history and the organ both have some kind of and improvisations. One such new in- the sound (and putting a computer in until this century, all music was played mechanism between the player’s fingers strument is the radio drum,whose devel- between) creates enormous possibili- acoustically,and thus it was always phys- and the sound. opment is being refined through ties. The instrument can now do any- ically evident how the sound was pro- In the 20th century, the invention of collaborative research among faculty thing as a result of the gesture, which duced. The player controlled the sound electronic musical instruments has seen and graduate students of Music and En- can trigger and control any sound (not directly by playing (eg, plucking, blow- a “decoupling” between the performer gineering at the University of Victoria. only drum sounds).The gesture can also ing,striking) the instrument,and the per- and the resulting sound. This has led to control image and video displays, or any former’s gestures were directly vast possibilities for entirely new types The Radio Drum Explained other device that can be controlled via translated by the instrument into sound. of “intelligent” musical instruments The radio drum was first developed at an electronic input signal. This direct physical relationship started where the contemporary performing Bell Laboratories in the mid 1980s as a One could say that the mechanism to become indirect in recent centuries artist uses technology to create innova- three-dimensional mouse, which failed (or “action”) in the conventional piano, with keyboard instruments; the piano tive new compositions, performances as a computer peripheral and instead became a musical instrument. A kind of receiving antennas in the drum. The strings and soundboard have now be- “virtual instrument,”it is a descendant of drum generates six separate analog sig- come electronic circuits, chips and com- the , the amazing musical in- nals that represent the x, y, z position of puters. Sound generation is now strument invented in 1917 that gener- each stick versus time t. A key attribute accomplished via computer chip and ates sound in response to hand of the radio drum is the multidimen- nothing vibrates except the loudspeaker movements in free space within an elec- sional nature of the gestural signal.This at the end of the chain. tromagnetic field. is in contrast with unidimensional con- This allows the creation of totally new The radio drum works on a similar trollers such as keyboards, which have sounds — sounds that would be impos- principle: the drum itself does not pro- velocity sensitivity only. sible to replicate physically.Examples are duce any sound. As a gesture sensor, like playing 88 notes at once on a piano, di- the theremin, it detects the performer’s Components of New Musical viding the octave into microsteps to cre- movements to trigger and control sound; Instruments ate new tonalities, or creating new the performer does not have to physi- In order to consider musical perform- timbres (character of sounds) with spec- cally touch any surface to create a sound. ance with new musical instruments tra unknown in acoustic music, etc. The radio drum consists of two parts: such as the radio drum, three compo- These open possibilities are very ex- a rectangular surface (drum) with em- nents are required: hilarating artistically. However, an bedded antennae and two transmitters • a gesture sensor that detects the per- equally important issue arises: how do embedded in conventional sticks that former’s movements and responds to you play these new electronic/computer use different radio frequencies.The drum them by creating control signals; instruments? In other words, even if we surface is covered with a layer of foam to • a sound engine that generates can make these wonderful new sounds, provide a quiet, elastic playing surface sounds according to a sound synthe- how will we control and perform them? and to avoid striking the circuit board. sis model with parameters; and The radio tracking technique de- • control algorithms that map the con- The Performance Question pends on the electrical capacitance be- trol signals into the parameters of the Early electronic music did not worry so tween the radio transmission antenna sound . much about performance; early works in the end of each stick and the array of The performer therefore controls the from the mid 20th century were often

14 INNOVATION DECEMBER 2002 INNOVATION DECEMBER 2002 15 new musical instruments is mension can be discrete, as in a chromatic division of the this: how effective (respon- octave (like a piano), or it can be continuous, in which case sive, subtle, musical) is the it can play microtones or glissandi.The important improve- instrument in a live perform- ment here would be investigating how to map parameters ance? within the “stripes” that will have subtle control over differ- Most of the control soft- ent physical models instead of samples,as is currently done. ware for new musical instru- • Quantizing x and y into gridsquares and triggering sounds ments is written in Max/ MSP, only when the stick hits the surface at z = 0.This “virtual drum a language designed for real- kit”maps different kinds of percussion samples to each grid- time music programming square, but within each square there is information that is that is widely used in the “thrown away.”Here, the obvious improvement would be to community. allow subtle timbre variation within each gridsquare, It allows very quick prototyp- mapped to the respective physical model. ing, has a very intuitive • Mapping a gesture dimension to melodic patterns,an exam- graphical interface,and is op- ple of “macroscopic”or “process”level of control.These inter- timized for both asynchro- actions can be seen as closer to the conductor paradigm than nous event-driven programs the musical instrument paradigm, but this is really a contin- and synchronous DSP pro- uum.This level of control may or may not map to the synthe- grams for sound synthesis sis technique,but the innovations here involve a better ability and analysis.The figure at left to interpret the gestures of the performer in a more musical shows an example program way, and applying it to a high-level control of sound. in Max/MSP that implements • Mapping the y axis to, for example, the speed of pulses that algorithms to control the trigger sonic events, while modifying parameters of these sound via gestures. events with gestures in the x and z axes. In one example,the algo- rithms control an acoustic Onstage Results piano via gestures on the Schloss has been testing the hardware and software in very de- Max/MSP graphical user interface showing top-level concert patch (NYU concert.pat) that the artist manding concert situations. He has performed on the radio sees during the performance (top), along with subpatches that are opened by double-clicking on radio drum. (A special “com- Andrew Schloss performing on the radio drum at the Banff Centre the parent patch. The NMundo subpatch includes mappings from regions of the drum to different puter-controlled” acoustic drum with world-class Cuban pianists Chucho Valdés and Hi- (photo: Banff Centre). sounds; the settimbres sub-subpatch assigns different timbres to the synthesizer to prepare it for this grand piano, the Yamaha lario Durán in Cuba,Canada and the US,including a live national particular mapping of the drum. Graphical user interfaces are custom made for specific artistic Disklavier, has solenoids in broadcast on CBC Radio Two. These performances involve jazz wide variety of musical contexts, from contemporary classical to projects; no two are the same. See www.cycling74.com for more information. every key and is played by improvisation, which is a fairly rigorous test of robustness of Cuban jazz to performing on Broadway in New York.His research is sending it note-on, note-off musical ideas, mappings, algorithms, response and latency. in the areas of interactive performance, improvisation and intelli- and velocity commands.) He has also recently collaborated with Randy Jones in the gent musical instruments.His website is www.radiodrum.com. The control algorithms, or creation of UNI, a multimedia work that incorporates the cre- recorded in a studio and not performed live.Even so,there was mappings,are designed to “play”the piano in ways that may be ation of animation in real time, directly driven by the perform- Peter Driessen PEng PhD is Associate Professor in the Department always an interest in live performance and the ability to control impossible for a human performer while also making musical ers.UNI received its premiere in Havana,Cuba and has also been of Electrical and Computer of Engineering at the University of Vic- the sounds in real time. sense — and, most importantly, retaining the perception of performed in Canada and the United States in several festivals. toria. His research interests include radio propagation, wireless Computers have finally become powerful enough to create “cause and effect” in the eyes of the audience. The mappings Most recently,he has collaborated with San Francisco com- communications,audio/video signal processing and interdiscipli- complex music in real time. Sound synthesis techniques have may trigger entire sequences of notes, but always in response poser David Jaffe, using the radio drum to control physical nary work.His website is www.driessen.ca. taken advantage of this computing power and have evolved to to a clearly observable gestural input. models of automobile and jet engines in the context of a new include physical modeling of an acoustic instrument, in which Creating mappings from the gesture sensor to the sound composition by Jaffe entitled “Racing Against Time,” which The authors acknowledge financial support for this project from the physics of the instrument are implemented in software. synthesis engine is an unsimulatable process that makes the was premiered earlier this year at the Brandon New Music Fes- the Natural Sciences and Engineering Research Council and the Gesture sensing techniques have also improved significantly system musical, satisfying and performable: it is necessary to tival in Manitoba. Canada Council for the Arts. using new devices such as pressure-sensitive fiber optics. physically play the instrument when “tuning” the algorithms. However, the control algorithms and techniques have not There is no other way to calculate, theorize or program this Conclusion evolved nearly as much.In terms of performance,the problem mapping. It has to be done experientially; the performer has to It took decades before the saxophone was accepted as a viable OYEN WIGGS GREEN & MUTALA remains — what kind of control algorithms or mappings make be in the loop at all times. musical instrument,and centuries for the piano to develop into musical sense? This experimental approach is an effective way to create its current shape and form. Engineers and artists are in the Intellectual Property Lawyers • Patent & Trademark Agents Electronic keyboard instruments — even those with physical something usable and musical, responding in subtle ways to process of creating new musical instruments of the future that modeling sound synthesis — still respond only to note number small changes in input and thus fully exploiting the musician’s meet the requirements of a virtuoso performer. No one knows Gerald O. S. Oyen Blake R. Wiggs Bruce M. Green and velocity gesture inputs.Thus,because of the missing control performance skills that were honed over many years. Catherine D. Mutala David J. McGruder Thomas W. Bailey which instruments will prevail and which will disappear. Gavin N. Manning George F. Kondor Hilton W. C. Sue component,the opportunity to create new musical instruments The radio drum has been used successfully in very de- Craig A. Ash Richard A. Johnson Todd A. Rattray is largely unexplored. Mapping Examples manding artistic contexts, but there is still work to be done in Some examples of mappings already completed with the generalizing many of its capabilities and evolving towards an 480 - 601 West Cordova Street Mappings for Musical Expression original radio drum, and possible extensions to be pursued, instrument that is truly limitless.Interdisciplinary collaboration Vancouver, British Columbia, V6B 1G1 The subtle and nuanced motions we make while playing a mu- are as follows: to create new forms of expression for performing artists con- Tel: (604) 669-3432 Fax: (604) 681-4081 sical instrument relate closely to what and how we intend to • Mapping of x to pitch, y to timbre and z to volume or veloc- tinues, and the future possibilities are exciting. E-mail: [email protected] www.patentable.com play and what sound we intend to make.Thus,good musical in- ity.The y axis can be a continuous change of timbre or it can struments are capable of responding to subtle changes in ges- be mapped in horizontal “stripes” that map to discrete tim- W Andrew Schloss PhD is Associate Professor in the School of Music Patents Trademarks Copyright Litigation tures by making subtle changes in sound.An important test for bres; for example, different instruments. Similarly, the x di- at the University of Victoria. He has performed internationally in a Canadian, U.S. and International

16 INNOVATION DECEMBER 2002 INNOVATION DECEMBER 2002 17