Dreams of Computer Music: Then and Now Author(s): F. Richard Moore Source: Computer Music Journal, Vol. 20, No. 1 (Spring, 1996), pp. 25-41 Published by: The MIT Press Stable URL: http://www.jstor.org/stable/3681267 Accessed: 28/01/2010 22:31

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at http://www.jstor.org/action/showPublisher?publisherCode=mitpress.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed page of such transmission.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected].

The MIT Press is collaborating with JSTOR to digitize, preserve and extend access to Computer Music Journal.

http://www.jstor.org F.Richard Moore of Department of Music, and Center for Research Dreams Computer in Computing and the Arts University of California, San Diego Music-Then and Now La Jolla, California 92093-0037, USA [email protected]

Computer music was first created nearly 40 years observations, first published in 1514, that led him ago in universities and research laboratoriesby to suggest that the earth revolves around the sun people dreaming of magical instruments for music. rather than vice versa, an observation that directly These instruments were like genies in that they contradicted the teachings of the Roman Catholic could grant wishes-musical wishes-to anyone church. In 1633, just a few years after Bacon's who possessed the required "bottle" of knowledge death, the church condemned Galileo Galilei to about computers and music. These dreams also had life in prison on a charge of "vehement suspicion a correspondingtechnology in the real world, but of heresy" for teaching Copernicanism. Francis more than anyone would have dared to wish at the Bacon'sphilosophical works were held in high es- beginning, the technology of computer music was teem by many important scientists, including Rob- to prove itself unique; computers are possibly the ert Boyle, Robert Hooke, Isaac Newton, and only thing in existence that have actually gotten Thomas Hobbes, so much so that in the 18th cen- cheaper over the years while simultaneously becom- tury, Voltaire and Denis Diderot considered Francis ing more powerful. Now nearly everyone who Bacon to be the father of modern science. wishes it has the means to create computer mu- In 1626, about two years before his death, Francis sic-but do they have the dreams? Bacon wrote The New Atlantis. This is, ostensibly, the first work of what we today call science fiction. Bacon was both a practical man and a dreamer.In Precursors,Science Fiction, and Music The New Atlantis, he described a society stumbled upon by shipwrecked sailors, a kind of island uto- Dream:Power Over Sound and Music pia somewhere in the Atlantic Ocean. As the sail- Reality:Purely Imaginary ors were shown around the wonders of this place, Bacon describes many aspects of nature being stud- As strange as it may sound, the idea that observa- ied. One of these places of study was described tion should be the basis of knowledge was not al- with the following words. The and ways commonplace. great philosopher We have also our where we statesman Francis Bacon much to the sound-houses, prac- gave impetus tice and demonstrate all and their of modern science that sounds, gen- development by suggesting eration. We have harmonies which do observation of nature should be the basis for knowl- you not, of quarter-sounds,and lesser slides of sounds. edge. At the time of Francis Bacon, the church, the Diverse instruments of music, likewise to you state, and the nobility all had another theory; they unknown, some sweeter than any you have; to- believed that authoritative statements by high- gether with bells and rings that are dainty and ranking officials should be the basis for knowledge sweet. We represent small sounds as great and truth, rather than observations that could be and deep; likewise great sounds extenuate and made by just about anybody.The real revolution of sharp;we make diverse tremblings and war- science was the replacement of authority with ob- blings of sounds, which in their original are servation as the basis for truth. In fact, the very entire. We represent and imitate all articulate term revolution comes from Nicolaus Copernicus's sounds and letters, and the voices and notes of beasts and birds. We have certain helps which Computer Music Journal,20:1, pp. 25-41 Spring 1996 set to the ear do further the hearing greatly.We ? 1996 Massachusetts Institute of Technology. have also diverse strange and artificial echoes,

Moore 25 I

reflecting the voice many times, and as it were I wish now to consider the development of the tossing it: and some that give back the voice field of computer music in terms of the dreams of louder than it came; some shriller, and some several of those who were active in its establish- deeper;yea, some rendering the voice differing ment. While Francis Bacon was dreaming in 1624 in the letters or articulate sound from that of doing what many now take for granted, I con- they receive. We have also means to convey sider the real history of a separate field called com- sounds in trunks and pipes, in strange lines puter music to have started in the mid-1950s with and distances. the pioneering work of LejarenHiller and Max Ma- thews. Obviously such a treatment as this must be In this small paragraphfrom The New Atlantis, somewhat speculative, for most of what remains is Francis Bacon described much of the modern field the work of such people rather than records of their of computer music almost perfectly. In 1624, Bacon dreams. But there is ample evidence that dream was already dreaming of fundamental power over they did, and I suggest that it is these dreams- sound and music. What is remarkableis not so every bit as much as the work-that form the basis much the dream itself, for Bacon was simply trying for what is now an established and increasingly to list everything that he could imagine doing with important field of inquiry. sound, but that it would take hundreds of years to achieve most of it. Alexander Graham Bell invented the "trunks and pipes" and "strangelines" to con- Musique Concrete and Elektronische Musik vey sounds over distances-otherwise known as the telephone-only in the 19th century. Hearing There were many other precursorsto computer mu- aids, which Bacon described as "certain helps sic. Thaddeus Cahill's 200-ton Dynamophone (also which set to the ear do further the hearing greatly" called the Telharmonium), invented around 1906, are still a of intensive research, though toured the United States in several railroadboxcars, much has been achieved recently through the use astonishing and delighting audiences with its amaz- of neural implants and microelectronics. Modern re- ing ability to create musical sounds through the cording studios would be lost without their "di- use of huge electromagnetic inductors. Leon There- verse strange and artificial echoes," and computer min invented his famous mood-altering instrument music techniques like the phase vocoder allow us about 1924, around the time of the Ondes Mar- to "representsmall sounds as great and deep; like- tenot used by Olivier Messian and the Trautonium wise great sounds extenuate and sharp,"among used by Paul Hindemith. Sometime in the 1930s other things. The important thing is that Bacon the great conductor Leopold Stokowski wrote, "I imagined-he dreamed-of being able to "practice can see coming ahead a time when the musician and demonstrate all sounds, and their generation." who is creator can create directly in tone not on As with Leonardoda Vinci's visions of flight paper."In the late 1940s, with the advance of mag- through the air, and Jules Verne'svisions of travel netic wire and tape, it became possible to manipu- through space, without dreams, reality could not late sounds on recordingsin a manner similar to possibly follow. the manipulation of images on photographicpaper. Dreams in this sense are a pure combination of Such manipulations gave rise to the school of mu- desire and imagination without necessity of the sique concrete by the Groupe de Recherches Musi- means to achieve something in reality. Dreaming is cales at the national radio station of France in the imagining what is desirable without having to early 1950s, with the work of Pierre Shaeffer and worry about how to achieve it. The purpose of Pierre Henry and others. Musique concrete was to dreaming is therefore to discover what we wish to be based on the "concrete" sounds of nature rather do. Only when we awaken do dreams become than the "abstract"sounds of traditional musical attached to the typical clauses and conditions start- instruments. ing with "if only...." Also in the early 1950s, at the Cologne Radio Sta-

26 Computer Music Journal Figure 1. John Cage and graph originally appeared LejarenHiller at the con- in Source: Music of the sole of the ILLIACcom- Avant-Garde2(2), 1968, puter they used in their and is reprinted with per- collaborative piece mission of the Source Ar- HPSCHD, realized at the chives, Music Library,Uni- University of Illinois in Ur- versity of North Texas, bana in 1968. (This photo- Denton, Texas, USA.) tion in Germany, Herbert Eimert and Robert Beyer began to generate sound waves from their elemen- tal building blocks-sine waves-according to the famous mathematical theory of Jean-Baptiste Jo- seph Fourier. Because the sine waves were created electronically, the result, as practiced by composer and many others, was col- lectively known as elektronische Musik. In the United States, Vladimir Ussachevsky and Otto Leuning produced the first concert of what they called "tape music" at the Museum of Modern Art in New York City on 28 October 1952.

ComputerMusic: The First Decade, 1955-1965

Dream:Mind of Science/Heart of Art Reality:Existence Proofs set the first note to number 40 (middle C). Given that first choice, the rules might disallow any skip Hiller and the ILLIAC greater than an octave, so if the computer chooses a second note at random less than 28 or greater True computer music did not begin, however, until than 52, the choice would be rejected and the com- a young research chemist and composer named puter must try another, and another, until a choice Lejaren A. Hiller began to investigate ways to apply is found that doesn't break any of the "rules." Such a process known as stochastic modeling to under- a technique is called a "Monte Carlo method." The standing the process of musical composition. Sto- hard part of this procedure is coming up with rules chastic methods had been successfully used for mo- that are both possible to obey and that give results lecular modeling in chemistry for some time, and that sound musical. This is much more difficult computers were a perfect vehicle to exploit such than it may appear at first glance; so difficult, in techniques. Computers of the mid-1950s were not fact, that Hiller tried several alternatives to the yet capable of sensing and producing arbitrary whole procedure, most notably those based on the sounds. They were, however, capable of the high- mathematics of Andrei Andreevich Markov. Using speed calculations that made otherwise laborious Markov's techniques, one examines any set of ex- stochastic modeling a practical reality. Because he isting music, perhaps that of a composer generally worked at the University of Illinois, Lejaren Hiller regarded to be reasonably good, such as Mozart. had access to one of the earliest examples of what The basic idea of Markov modeling is as follows. we would now call a supercomputer: the ILLIAC. Looking through one or more pieces by Mozart, He is shown in Figure 1 standing at the console of one can compute the probability that the note the ILLIAC with his collaborator John Cage. middle C will be followed by D, or E, or E-sharp, or Lejaren Hiller's basic idea was simple: have the F-sharp, etc. Once these probabilities are deter- computer generate a list of random numbers repre- mined by analyzing existing music, the computer senting anything we like about music, such as can be programmed to produce music that has ex- pitches, rhythms, dynamics, etc., then program a actly the same statistics, yet is different from the set of "rules" by which one choice would be al- analyzed music. If the statistics are based on Mo- lowed to follow another. For example, if we wish to zart, the computer would be more likely to follow compose a melody, we might number the pitches middle C with D than, say, F-sharp. Such tech- on a piano keyboard from 1 to 88, then (arbitrarily) niques can be extended to pairs of notes, or triplets

Moore 27 of notes, and so on, yielding ever more restrictive are embodiments of objective research results. probabilities, while still allowing some freedom of They are laboratorynotebooks. Sometimes the choice. Eventually one can imagine doing such results have surprisedus because a composi- things as feeding in Beethoven'snine symphonies tional routine seems less effective than ex- and asking the computer to compose the Tenth pected, sometimes more so. If I were to delete Symphony that Beethoven might have written if he everything that disturbs me aesthetically, I had had the time. would falsify the research record. So, for the LejarenHiller's dream was not only to compose present, my objective in composing music by music with a computer, but to understand both means of computer programmingis not the im- how a composer thinks and even what makes one mediate realization of an aesthetic unity, but piece of music good and another poor. Because such the providing and evaluation of techniques work had never been attempted before, the dreams whereby this goal can eventually be realized. clearly (and typically) overshot any possibility of For this reason, in the long run I have no per- realization. This problem turned out to be much sonal interest in using a computer to generate harder than anyone realized, before anyone had at- known styles either as an end in itself or in or- tempted to solve it, as problems often do. Neverthe- der to provide an illusion of having achieved a less, the dream was no less than grandiose-deep valid musical form by a tricky new way of stat- insight into the intelligence of the human mind, in ing well-known musical truths. this case, the musical intelligence. The computer allowed an experiment that most would say failed (if we listen to the resulting music), but I contend that the experiment succeeded in producing one of Mathews: MUSIC I (1957) the most brilliant dreams ever had about music. The dream exceeded even those of Francis Bacon, Even in 1876, Alexander Graham Bell knew that whose brilliant imagination stopped short of includ- sound pressure waves could be converted into anal- ing scientific investigation of the process of human ogous electrical signals, and vice-versa, through musical thought. Sound example 1 (CD index 2) is the use of microphones and loudspeakers-that's the second movement of LejarenHiller and Leon- why we call them analog signals. The advantageof ard Isaacson's 1957 piece Illiac Suite for String electrical signals is that they travel much faster Quartet. through wires than sound does through air, Granted, LejarenHiller's music may sound pretty allowing electrical signals to travel much further in bad, but does that make it good science? Were a small amount of time; hence the name tele (far Hiller and his associates actually using the scien- off) plus phone (sound). By continually reconnect- tific method suggested by Bacon and Galileo and ing a network of wires efficiently, the telephone sys- others, or were they simply playing with the new tem has probably done more to shrink the size of technology of computers? The following words, the world than the airplane. When electrical signals written by LejarenHiller in the record liner notes travel through wires, however, they pick up the about the Illiac Suite, address this exact question. electrical equivalent of dust and dirt, called noise. If the wire is long enough, the signal becomes so Should a person listen to these two pieces as contaminated that it is no longer possible to under- he might "ordinarymusic?" Yes, I think, but stand the original sound. Practically the entire with this important qualification: They are story of telecommunications is about ways to pre- much more didactic than expressive compared vent, or at least to slow, this inevitable noise degra- to most music. These pieces are truly experi- dation of signals as they are transmitted from one mental because they are concerned with reveal- place to another. ing process as well as being final product. They Digital technology was developed in so many

28 Computer Music Journal ways and places that there is no particular person On this recording,we illustrate another ad- identified with its origin. By the 1950s it was al- vancement in the realm of tools available to ready known that digital signals could be transmit- the music-maker: the computer and the ted from one place to another far more reliably digital-to-sound transducer.This new "instru- than analog electrical signals such as those in Bell's ment" combination is not merely a gadget or a original telephone. In a digital signal, information complicated bit of machinery capable of pro- about, say, a sound, is encoded as a series of binary ducing new sounds. It opens the door to the digits, or "bits,"each of which can have one of only exploration and discovery of many new and two possible values ("on/off,""yes/no," "1/0," etc.). unique sounds. However, its musical use- When a bit is transmitted electrically from one fulness and validity go far beyond this. With place to another, it picks up noise just like any sig- the development of this equipment carried out nal. The receiver, however, only needs to be able to at the Bell Telephone Laboratories,the com- distinguish whether a "yes" or a "no" was origi- poser will have the benefits of a notational sys- nally transmitted to have essentially perfect knowl- tem so precise that future generations will edge of the original signal. As long as it is not too know exactly how the composer intended his powerful compared to the "on/off" part of the sig- music to sound. He will have at his command nal, the noise can be eliminated completely, mak- an "instrument" which is itself directly in- ing digital transmission and recording far superior volved in the creative process. In the words of to analog for many purposes. This fact was not lost three of the composers whose works are heard on the telephone company, and in the mid-1950s a on this recording, man's music has always been young telecommunications engineer and amateur acoustically limited by the instruments on violinist named Max Mathews was hard at work at which he plays. These are mechanisms that AT&T Bell Telephone Laboratoriesinvestigating its have physical restrictions. We have made potential. At that time the telephone company was sound and music directly from numbers, sur- interested only in speech. From a technical stand- mounting conventional limitations of instru- point, however, music is just like speech with a few ments. Thus, the musical universe is now cir- more frequencies-both are structured sound. cumscribed only by man's perceptions and Using a special device called a digital-to-analog creativity. converter, a computer can control the movements of loudspeakers with a precision never before avail- Especially given this vision of the computer as a able. Max Mathews and others reasoned that be- completely general musical instrument, perhaps cause computers could be used to investigate the most obvious question one can ask about his improvements in telephone speech analysis, pro- original computer music sound-synthesis program cessing, and synthesis, they could also be used to is, How did it sound? The answer is simple: in improve the analysis, processing, and synthesis of musical terms, it sounded rather terrible. For one musical sounds. Mathews therefore wrote the first thing, the audio quality of 1950s digital audio tech- music-synthesizing programsin about 1957, turn- nology was poor by today's standards (as today's ing the computer into a new kind of musical instru- technology will seem soon enough). Also, the musi- ment that was capable, in principle at least, of pro- cal quality was extremely stilted and unnatural due ducing any sound that could come from a to the simplistic means for specifying the musical loudspeaker.Along with LejarenHiller's efforts to sound, which left out the performeraltogether. Fi- use computers to investigate musical intelligence nally, expectations placed on the output of early and composition, the field of computer music was computers were often unrealistic. Who would have now born. The liner notes of Music from Mathe- expected that the first musical sounds produced by matics, the first published recording of computer a multimillion-dollar example of our most ad- music, explained it as follows. vanced technology would sound more like a child's

Moore 29 first violin lesson than the pinnacle of musical evo- ComputerMusic: The Second Decade, 1965-1975 lution? Despite its potential to act as a completely general musical instrument, the computer's first Dream:Truly Musical Sounds music lesson sounded no better than yours or Reality:Research mine-and (not knowing how yours sounded), per- haps a lot worse. For the remainder of computer music's first de- Risset and the Trumpet cade, most work occurred in a few laboratories and universities where expensive and hard-to-use com- It is difficult to make music on a musical instru- puters were available for this purpose. Programs ment that does not sound like a musical instru- were typically written in assembly language using ment. Therefore, one of the first problems to be punched cards or paper tape. To run a single com- tackled seriously in the field of computer music puter programoften took a full day or more, and was that of synthesizing sounds with truly musical the results were usually disappointing, assuming characteristics. Two basic approachesto this prob- the computer itself worked properly (which was lem are either to imitate the sounds of traditional often not the case). Printed output was usually on musical instruments or to synthesize completely large fan-folded sheets of computer paper.If the new sounds that are musically satisfying. Both of computer also produced a sound, it was in the form these problems are difficult, but for different rea- of a large computer tape containing millions of sons. Imitations are always tricky, because even the numbers representing the sound in digitized form. slightest discrepancy will alert the listener to try to If the sound-synthesis programsucceeded in pro- find others. Elsewhere I have called this the "oleo- ducing the tape (which one often determined by margarineeffect" by analogy with another famous watching the tape move while the programran), it example of how substitutes rarely work perfectly. then had to be carried to a separate machine spe- The other approach,that of synthesizing new cially designed to feed these numbers at high speed sounds with musical properties, is also difficult through a digital-to-analog converter,to convert the because listeners have such ingrained habits of numbers into voltages. This signal could then be thought regardingwhat sounds are "musical."De- processed and fed into a loudspeaker (in the mid- spite years of the "liberation of noise" and other 1960s, computer music researchersfrom Princeton campaigns by modern musicians, most people sim- University had to drive to -a 150-km ply have different aesthetic expectations when they round trip-about once a week with one or two hear traditional musical instruments, such as vio- such tapes). One always had to remember to turn lins and guitars, as opposed to sounds never before on the reel-to-reel tape recorderbefore starting the encountered. A kind of "categoricalperception" oc- conversion into sound, or the whole effort might be curs in which some sounds are placed into the "mu- lost if the computer tape broke. The results typi- sical" category while others (such as speech and cally fell into these categories: (1) silence (the most noise) are not. Some interesting sounds have been common result), (2) a long, loud noise (also a com- discovered, however, most notably an entire cate- mon result), (3) a patterned noise sounding nothing gory of "acoustical illusions," named for their abil- like expected (usually the result of a programming ity to "fool the ear" in ways analogous to the man- error),or (4) a thin, weak, pale, sickly musical ner in which optical illusions "fool the eye." sound (which happened in perhaps one of every five By the mid-1960s, computer sound synthesis had attempts). The sounds may have been sickly and advanced to the point where musical researchers pale, and the work may have been laborious and could begin to address the problem of musical tone painstakingly slow, but these results proved that color, or timbre. Based on the assumption that it computer music existed. Now all it needed was would be necessary to understand what makes the improvement! sounds of traditional instruments "musical" to be-

30 Computer Music Journal Figure2. Jean-Claude Ris- set demonstrating a trum- pet tone synthesized by computer at Bell Tele- phone Laboratoriesin 1965. (Photographcour- tesy of Bell Telephone Laboratories.) ally any musical sound, but musicians are almost always after a particular set of musical characteris- tics, usually a matter of perception. One of the things early computer music researchers quickly learned was that our knowledge of the relationship between the physical and psychological characteris- tics of sound-psychoacoustics-is about as lim- ited as our knowledge of every other aspect of the I human mind. Looking at waveform plots, for ex- 91 -t ample, we soon learned that fairly large-looking dif- ferences between two sound waves don't necessarily correspond to large-sounding differences, and the opposite is also true (apparentlyhuman beings can both see and hear because we get truly different information about the environment from each of these senses). Jean-ClaudeRisset's investigation into trumpet sound led to a key insight into the sounds of virtu- ally all brass musical instruments. The physical op- eration of brass instruments causes the amplitude, gin with before new ones could be invented, Jean- or strength, of the sound to build up, reach a more- Claude Risset, a physicist and composer working or-less steady state, then die away at the end, all in with Max Mathews at Bell Labs, investigated the more-or-less exponential fashion (this is a common sound of the trumpet. What he learned confirmed characteristic of many physical systems). Risset what had been suspected for some time-that determined that the bandwidth of the sound-es- there is not a single recipe for the entirety of any sentially the number of harmonics-grows in musical sound. Rather, the recipe-the relative pro- direct proportion to the amplitude of the sound. portions of sine waves that are added together to While that simple fact is not all there is to brass- form a complete sound-must change in charac- instrument sounds, it is so important that virtually teristic ways from moment to moment within a any sound with this characteristic will sound brass- sound, even from the beginning to the end of a like, or at least more like brass than any other type single note. In technical terms, the spectrum of the of instrument. sound waveform must vary over time in the correct Despite its limitations, Jean-ClaudeRisset's way. The equivalent problem for cooking might be "brass tone hypothesis" was a real breakthrough, to try to invent a food that tastes and feels like a and an encouragement that at least some im- green apple when you bite into it, then gradually be- portant characteristics of musical timbre could be comes more like roast beef as you chew it, then understood. Risset celebrated his discovery by fea- tastes and feels like port wine as you swallow it. turing his synthetic trumpet in computer-generated That is essentially what all musical sounds do, ex- portions of the incidental music he wrote for Little cept that taste and feel are replaced by timbre, the Boy, a play that tells the true story of the pilot who time-varying mixture of frequency components flew the plane that droppedthe atomic bomb on that fuse to form a single musical sound. Figure 2 Hiroshima. The flight crew had named the bomb shows Risset at Bell Labs in 1965 presenting his "Little Boy."Risset used his computer-synthesized work on trumpet tones. trumpet to accompany sequences in the play in Max Mathews's synthesis programs allowed which the pilot dreamed about happier times in a people to specify the physical properties of virtu- jazz club he had frequented. The pilot-who later

Moore 31 I

went insane-was tormented by nightmares in of which is that it requires intimate familiarity which he "became" the bomb as it fell endlessly with mathematical . Another problem with toward its target. Here Risset found an eloquent physical modeling is that it is difficult even for the application for a computer-synthesized acoustic computer, because an enormous amount of compu- illusion. Based on recently understood principles of tation must usually be done to obtain even modest psychoacoustics, he created a long tone whose results. This fact relegates most physical modeling pitch gradually descends forever without getting to the exotic realm of supercomputers. But the ad- any lower (like the barberpole stripes that appear vantages of physical modeling are also very real, to go down forever without getting anywhere) to ac- especially if one is interested in synthesizing the company the pilot's dream of endless falling. Sound sound of traditional musical instruments. Pierre example 2 (CD index 3) is an excerpt from Jean- Ruiz's early work in this field was so demanding Claude Risset's 1968 Suite from Little Boy. that, despite obtaining the best synthetic violin tones made at the time, he left the field of com- puter music altogether, as have many others when Ruiz and Physical Models they discovered the true cost of some dreams. One might have expected someone with Ruiz's mathe- Jean-ClaudeRisset's success was based on under- matical abilities to investigate other issues in sci- standing the physical characteristics of brass instru- ence or technology, but such was not the case. He ments and human perception. Max Mathews's went into experimental theater. programsallowed users to specify the physical char- acteristics of virtually any musical sound. Lejaren Hiller borrowed stochastic methods used in model- Music V and GROOVE ing molecular physics to model the music composi- tion process. A young graduate student of Hiller's, When I first arrivedat Bell Labs in 1967, after Pierre Ruiz, working at the University of Illinois studying music composition and performance at and later at Bell Labs with Risset, took computer Carnegie-Mellon University and the University of music synthesis a big step further. Physicists have Illinois, one of my first jobs was to write the com- known for some time how to write differential puter programthat processed computer tapes con- equations that model the behavior of physical sys- taining sounds. I also became deeply involved in tems, including those of musical instruments. The Mathews's last sound synthesis project, called Mu- trouble is that such equations quickly grow more sic V because it was the fifth version of that pro- complicated than any person can solve in a general gram he had designed over about ten years. While mathematical way. The computer, however, can be Music V embodied many ideas and technical ad- programmedto simulate the moment-to-moment vances of interminable interest to programmers, results of such equations even when they cannot be perhaps its most important feature was that it was solved generally.If the equations correctly model the first such programwritten in a portable pro- the behavior of the musical instrument, then the gramming language-in this case FORTRANIV- computer should be able to compute the resulting rather than machine-specific assembly language. sound. Ruiz applied this approach,called physical This allowed Music V to be carriedfrom one com- modeling, to the violin. puter to another and still stand a fairly good chance While this may be said of all computer music of working. As a public service to the new field of sound-synthesis techniques, physical modeling is computer music, Bell Labs gave the Music V pro- truly a technique in which mathematical equations gram away for free, primarily to universities that become musical instruments. The computer, in ef- owned suitable computer systems such as IBM fect, "plays" the formulae in ways that no human 7094s, Honeywell/GE 645s, and Control Data ever could, thereby obtaining sounds. There are 6600s. Keep in mind that a typical computer sys- many problems with this technique, not the least tem was the size of a small house, and cost many

32 Computer Music Journal Figure3. The IBM 650 com- ers. New York:Ticknor Figure 4. The high- GE 635/645, on which puter (first delivered in and Fields, p. 210. Reused performance CDC 6600 Music V was developed. 1954). (From S. Augarten, by permission.) computer of 1963. Other (FromAugarten 1984, 1984. Bit by Bit-An Illus- examples of this genera- p. 219. Reused by per- trated History of Comput- tion of machines were the mission.) IBM 7090 or 7094 and the

Figure 3 millions of dollars. The Music V program was dis- tributed in several boxes, each having a capacity of roughly 2,000 punched cards (don't worry-they were numbered in case someone dropped them!). Music V was a flexible and powerful tool for both making computer-synthesized music and for study- ing the relationship between the physics of sound and its perception. It was also clear that the modus operandi of Music V, which consisted of typed de- scriptions of one or more virtual musical "instru- ments" in a special language designed for that pur- pose, followed by a typed list of notes to be played on these instruments, mimicked well the concepts embodied in traditional written music. The pro- gram treated its input as the specification for a sound, which it then proceeded to synthesize. Figure4 What this paradigm leaves out is anything in par- ticular having to do with a performer's contribution exists before the movie-making process even be- to music. When a performer plays a piece of music, gins. A performance is filled with nuances and in- a great deal happens, most of which is poorly indi- flections associated with such things as the general cated-if at all-in a written score. There are usu- historical period and style of a piece, the perform- ally noteworthy differences between reading the er's personal "interpretations,"and the physical script for a play and seeing it performed on stage or characteristics of the listening space. In the case of screen. In both theater and music, these differences music, the physical characteristics of an instru- are called performance. The long list of names at ment also contribute a great deal to the resulting the end of a movie testifies to all the people who sound. Without these refinements, a piece of music have added something to the book, which usually usually tends to sound flat and lifeless, as when a

Moore 33 primitive computer speech synthesizer reads a love Everything must be planned in such a situation- poem. every gesture, every scene-no spontaneous "im- There are two general approachesto solving the provisation" is possible. If computer music was performance problem in computer music. One is to ever to sound as lively and expressive as other mu- write more complicated scores that are designed to sic, we needed a way to account for what perform- specify the precise final sound of the finished mu- ers contribute to music. Since we didn'treally un- sic-a much more complex task than writing a tra- derstand performance well enough to specify it ditional musical score. The second approachis to with written instructions (and we still don't),we allow computer music to be played in the tradi- had to resort to the capturing approach-and to cap- tional manner with keyboardsand other control- ture what performers do, a computer music system lers, thereby altogether avoiding the necessity of un- must operate in real time. derstandingperformance. In other words, we can For the GROOVEsystem, Mathews and I used a either try to understand performance or simply relatively small computer (one that occupied only "capture"it. While some composers have been up the space of an apartment rather than a house) to to the first task, most are not, partly because many control another roomful of voltage-controlled ana- composers are not themselves performers,but log synthesis equipment that had become the basis mostly because we do not necessarily understand for such popular (at that time) electronic music de- what happens when performersperform. Because vices as the Moog synthesizer (of Switched-On it involves additional understanding, the first ap- Bach fame). It is shown in Figure 5. The voltages proach is a matter of further research, even today. that controlled the pitches and amplitudes of the The second approachavoids the necessity for under- synthesis equipment were obtained through a bank standing by simply capturing aspects of perfor- of 14 digital-to-analog converters attached to a Hon- mance and using them directly. As we will see eywell DDP-224 computer. These voltages were up- later, this "capturing"approach has also been ap- dated by the computer at a rate of between 100 and plied successfully in the area of sound synthesis. 200 times per sec instead of the 10,000 to 50,000 As a pianist and percussionist as well as a com- times per sec needed for direct digital sound synthe- poser, I was equally interested in the computer mu- sis. Though the sound potential of the analog syn- sic problems of sound synthesis and performance. thesis equipment was far more limited than direct The only trouble was that by the late 1960s there digital synthesis, by using digital-to-analogconvert- was no performance capability in computer music, ers to control only such things as pitch and ampli- which suggested to Max Mathews and me that we tude rather than the actual sound waveform itself, should work on one. The result was a system we we gained a factor of about 100 in available compu- called GROOVE,for Generated Real-time Opera- tation time, and 5 or 10 msec computers-even tions On Voltage-controlledEquipment. 1969 computers as big as apartments-can do a fair At that time, there were no digital computers ca- amount of computing! We then built and con- pable of synthesizing sound in real time. Music V nected piano-like keyboards,pedals, joysticks, was an extremely powerful tool, but it worked inde- knobs, and switches to the computer, all of which pendently of real time. This means that it synthe- could affect the sound being synthesized in real sized the specified sounds as quickly as possible, time according to the user's program.Like a player but not as the sound was heard. Non-real-time piano, the GROOVEsystem recordedthe actions sound synthesis is similar to creating an animated we made pushing keys and twisting knobs, rather film; all the individual drawings must be com- than the sounds produced. For the first time, we pleted before the film can be viewed and the mo- could truly play the computer like a musical in- tion implicit in those frames can be experienced. strument-even though it still didn'treally sound While virtually anything can appearon those like one. frames, the synthesis process generally takes much The result was that for the first time computers longer than viewing the final result in real time. could be involved in improvisation and every other

34 Computer Music Journal Figure 5. The analog side thews and the author. The log synthesizer modules quency tuning knobs) and of the early-1970s left-most equipment rack above it. The other racks filters, and patch bays for GROOVE system, devel- contains an Ampex two- include a variety of analog configuring the intercon- oped at Bell Telephone channel tape recorderand voltage-controlled oscilla- nection of these elements. Laboratories by Max Ma- a collection of Moog ana- tors (with the large fre-

kind of spontaneous musical decision-making. three examples from the GROOVE system. The These are things that trained musicians are ex- piece at CD index 4 is an excerpt from Emmanuel tremely good at, and, like walking, are far easier to Ghent's work Phosphones, and indices 5 and 7 are do than to explain. Much research on real-time sys- two short GROOVE pieces by Laurie Spiegel. tems for computer music has been done for the pur- The example at index 6 on the CD was produced pose of circumventing deeper questions regarding on the real-time interactive synthesizer developed how performers perform. Fortunately, the very exis- by Hal Alles and co-workers at Bell Telephone Labo- tence of real-time computer music systems has ratories; it is an excerpt from one of Laurie Spiegel's made it feasible to study performance in ways improvisations on the machine. never before possible. Robert Willey, a bright young graduate student of mine at University of Califor- nia, San Diego, recently earned a doctorate in mu- Frequency Modulation sic by studying the effect of instrument response de- lay on performers. This is just one of the many In the late 1960s, another young graduate student questions that can now be asked and answered of music named John Chowning was using Music using real-time computer music systems. V at Stanford University to investigate the musical The CD that accompanies this issue includes effects of extreme vibrato. Due to a programming

Moore 35 Figure 6. A flow chart of a half-circles; the trape- Music V-style instrument zoidal figures are time- for frequency modulation varying envelope gener- (FM).The two oscillators ators. have icons that resemble

accident (as he himself describes it) he specified a sound with a vibrato rate much larger than a nor- mal vibrato, which is usually about 5 or 6 vibra- carrier tions per sec. In fact, the vibrato he specified was as high or even higher than the fundamental fre- frequency quency of the sound itself-something no singer or } modulator traditional instrument could possibly produce. ij ratio Much to Chowning's surprise, what came out was something that didn't sound like it had a vibrato at all. Instead, the result was a completely different tone quality. The simple process of vibrato, when carried to such an extreme, can be understood by carrier a from borrowing theory radio transmission, called frequency frequency modulation (FM).Even though FM radio transmission had been well understood since the 1930s, no one ever thought of listening directly to the modulated waveform itself before, and that's not how John Chowning thought of it either. What he did was accidentally transfer the FM theory from radio to musical psychoacoustics, and voila! a fascinating new music synthesis technique was born. one had any idea that such a discovery in the com- Of course, FM is not musically interesting be- puter music field could be so profitable. Stanford cause its theory comes from radio. It is interesting University patented the discovery,which was later because it is very simple to compute (one need licensed to the Yamaha Corporation,giving Ya- only modulate the frequency of one sine wave with maha a distinct early advantagein the computer another-a simple and efficient process). Further- music instrument marketplace. For several years, more, two things that are easy to control in FM are this patent produced royalties in excess of any ever the modulating frequency, which corresponds to issued to StanfordUniversity. The discovery of FM the vibrato rate, and something called the modula- synthesis-more than any other single develop- tion index, which is simply related to the vibrato ment-was responsible for the proliferation of com- depth (the extent to which the frequency varies puter music from universities and laboratories into above and below the main, or carrierfrequency). It studios, onto stages, into concerts, onto recordings, turns out that the ratio of the modulating and car- and ultimately, into homes all over the world. rier frequencies determines whether the resulting Sound example 7 (CD index 8) is John Chowning's musical timbre has a harmonic or inharmonic spec- 1972 piece Turenas. trum, which is an extremely important musical characteristic in perceptual terms. Furthermore, the modulation index is directly related to another ComputerMusic: The Third Decade, 1975-1985 perceptually important quality of sound-its band- width, or "brightness."Figure 6 shows the block Dream:Proliferation diagram of an FM instrument. Reality:Development Before John Chowning's wonderful discovery,no one had any idea that such a simple and efficient The Digital Revolution synthesis process could provide such straightfor- ward control over many perceptually important In 1960 the first integrated circuits were fabricated, characteristics of musical sound. Furthermore,no marking the real beginning of the digital revolution

36 Computer Music Journal Figure 7. A 1961-vintage conical shapes around the integrated circuit (a dual center of the figure are the flip-flop) constructed by transistors. The entire de- the Fairchild Camera and vice is 0.06-inch in diame- Instrument Corp. The ter. (FromAugarten 1984, white lines are aluminum p. 224. Reused by per- interconnects; the four mission.)

(see Figures 7 and 8). About 10 years later, in 1971, Marcian Hoff of Intel Corporation invented the mi- croprocessor, placing an entire computer central processor unit (CPU) on a single chip of silicon about the size of a fingernail. Such facts gave rise to a well-known maxim in the semiconductor industry known as Moore's Law (after Gordon Moore of the Intel Corporation). Moore's Law states that the complexity of inte- grated circuits is proportional to 2Y-1960,where Y is the current year. In other words, every year (starting in 1960) the complexity of integrated circuits dou- bles, meaning that roughly twice as many gates, bits of memory, or whatever, can be put on a single chip as compared with the year before. A corollary to Moore's Law states that cost is approximately proportional to the square root of complexity, mean- ing that the cost of doing any particular thing with integrated circuits will be cut in half about once ev- ery two years. Moore's Law and its corollary have continued to be valid (more or less) ever since they were intro- duced. They are important because they codify one of the most important aspects of computer technol- ogy, namely, that it is getting cheaper as time goes by. Not only that, it is getting cheaper at a remark- able rate. Computer systems aren't getting cheaper quite so fast, of course, because not everything in a computer is an integrated circuit. The cost of an for many things, but by the time a computer is two equivalent computer is only falling at the rate of generations old it becomes nearly useless, like last about one half every three years, still an incredible month's newspapers. price reduction compared with virtually everything else in the known universe. There is a dark side to Moore's Law, however. It Machines, Machines ... says that for the same price, complexity will in- crease by a factor of four every two years or so. In The same principles hold true for any devices based other words, computers keep getting more and on integrated circuits, like digital music synthe- more capable, so much so that after a computer is sizers. With the beginning of the real-time digital just a few years old, it can no longer compete with music synthesis era, research in computer music newer models in either performance or cost. At was booming. In 1975 I had temporarily left Bell some point it actually becomes cheaper to replace Labs to learn how to make digital synthesizers at an old computer than to upgrade or even repair it. Stanford University, where I studied computer engi- Moore's Law is therefore an important basis for the neering and built the FRMbox, one of the world's dreaded phenomenon known as technological obso- first all-digital music synthesizers shown in lescence. Computer designs progress by "genera- Figure 9. tions" that are only about three or four years long. Computers were still the size of cars and small A "last-generation" computer may still be usable trucks, and, much as with the GROOVE system,

Moore 37 Figure 8. Three historic top), 8008 (1972; right-top) lk-bit memory chip (1970; garten 1984, p. 266. microprocessor chips from and 8080 (1974; left- right-bottom). (FromAu- Reused by permission.) Intel-a 4004 (1970; left- bottom), and an Intel 1103

one entire computer had to be dedicated to control designing a digital synthesizer, working first with my synthesizer. Its design was documented in my , and later with and dissertation, and represented a kind of digitization many others. Hal Alles built a research digital mu- of the analog synthesis portion of the GROOVE sys- sic synthesizer at Bell Labs around one of the new tem. Peter Samson of the Systems Concepts com- LSI-11 microcomputer systems, shrinking the size pany also built a large digital synthesizer that be- of the control computer from pickup truck to TV came the basis for much music created at Stanford set. In 1979 I left Bell Labs to found the Computer University's Center for Computer Research in Mu- Audio Research Laboratory (CARL) project at the sic and Acoustics (CCRMA) which John Chowning University of California, San Diego, an effort dedi- and others founded around that time. Pierre Boulez cated to doing everything possible to sound and mu- created the Institute de Recherche et Coordination, sic with computers, and vice versa. Acoustique/Musique (IRCAM) in Paris, which All of these efforts to build computer music in- opened in 1978. At IRCAM, a brilliant young nu- struments, among several others, were research clear physicist named Giuseppe Di Giugno started projects that consumed fairly large efforts and bud-

38 Computer Music Journal Figure 9. The 1975 oped around the same Figure 10. A Digital Equip- (FromAugarten 1984, FRMbox digital synthe- time as the Systems Con- ment Corp. DEC VAX-11/ p. 258. Reused by per- sizer developed by the au- cepts Digital Synthesizer 780 of the late 1970s. mission.) thor at Stanford Univer- (or "Samson box") used sity. This was one of the at CCRMA, and the "4" first all-digital sound syn- series of machines built thesis machines, devel- at IRCAM in Paris.

dreamed of the day when real-time computer mu- sic synthesis would cost no more than a single grand piano, making it possible for a dedicated indi- vidual musician to afford one. Then computer mu- sic might begin to be on the same social basis as the rest of music, and we were sure that it would proliferate widely after that. Our dreams of proliferation were far too modest. Within a few years, the Yamaha Corporation had de- veloped inexpensive chips that could do much of what the large, expensive research prototypes could do. The first digital music synthesizer marketed in the United States was the Synclavier, designed by Sidney Alonso and others at New England Digital Company, a small computer company. Such ma- chines, which appearedin the late 1970s, carried price tags similar to average automobiles, and were fairly limited in their capabilities. But nothing could really compare on the price- gets, especially if the cost of the computer was in- performance curve with the first good, cheap syn- cluded. The Digital Equipment Corporation thesizer, the Yamaha DX-7, introduced in 1983. It VAX-11/780 (see figure 10) computer system that cost about US$ 2,000. Based on John Chowning's was the mainstay of the CARL project for many FM synthesis algorithm, within a few years the years cost about half a million dollars (US$), DX-7 had sold more than 500,000 units to the inter- though it did serve many users simultaneously (al- national market. It was not only the world's most beit not too well, by today's standards). While it did popular digital music synthesizer, but possibly the not seem unreasonable that a major research effort world's most popular musical instrument. in music might cost as much as a single grand op- In addition to the DX-7, another key develop- era production, tensions rose as with the costs of ment in computer music was the agreement among these efforts conflicted with the needs of individual several important manufacturers to a voluntary musicians who were used to working on more mod- standardfor connecting digital music synthesizers est budgets. In those days we wanted real-time digi- to each other (regardlessof brand!)and to the new, tal computer music synthesis, and thought in small personal computers. The resulting Musical In- terms of how many grand pianos it would cost. We strument Digital Interface, or MIDI, allows people

Moore 39 to combine the features of different digital synthe- display have already combined with interchange to sizers and computers in various ways-a critical create what is now called the World-WideWeb means of combating technological obsolescence. (WWW),but that is just the beginning. In effect, Whatever their shortcomings, MIDI and the DX-7 the interconnected computers of the world are be- created a proliferation that was completely unprece- ginning to wake up, just as their users are begin- dented in the field of music. Many other instru- ning to realize how powerful an instrument of ments and companies followed. mind (as Marvin Minsky called them) computers One device that deserves special mention in this really are. As we prepareto enter the 21st century, context was the Fairlight Computer Music Instru- computers are empowering individuals in ways rem- ment (CMI).Just as it is possible to solve the perfor- iniscent of the scientific revolution-no longer is mance problem without understanding it, it is pos- the world's most powerful technology available sible to solve the sound synthesis problem in a only to those in privileged positions. similar manner. The Fairlight was the first of a new Computer music has also jumped out of the labo- genre of digital music instruments that works by ratory and into the fire. Traditionalmusical instru- recording sounds in its digital memory (a process ments continue to get more expensive (like every- called "sampling")in such a way that they can be thing else) while computer music instruments just played back at various pitches and amplitudes un- keep getting better and cheaper. Most new pianos der control of a keyboard.The so-called sampling sold are now digital, and this trend shows no signs synthesizer could put a recording of virtually any of abating. Computer music is no longer an eso- sound under any key, though there are some limits teric corner of music but has in fact become part to how much pitch modification can be done with- of the mainstream, at least in terms of the market- out degradingthe sound. place. The new face of computer music is as shallow as it is broad, however. Rather than promoting re- search into basic musical questions, most com- ComputerMusic: The FourthDecade, 1985 to puter music now merely substitutes newer, cheaper the Present (and Beyond) technology for strings and pipes. In that sense the field of music has effectively "absorbed"computer Dream:Interfacing Mind with Reality music with little or no effect on what music is pro- Reality:Virtual duced or our understanding of it. Perhaps this is as it should be. Technology is In about 1985 the complexion of computer music changing much more rapidly than people. began to change; what was research oriented was After all is said and done, the dreams of today are becoming market oriented. Following the tremen- not much different from those of our ancestors, ex- dously successful proliferation of computer music cept, of course, that we have many more opportuni- instruments from the laboratories and into homes ties to realize those dreams than they did. and studios, shifts began to occur that are still pro- Perhaps too, it is difficult to realize what a dream foundly affecting the music field in general. Com- is about while one is in the midst of having it. puters have continued to shrink and get cheaper at Since my own interest in computer music over the rates predicted by Moore's Law to the point where years has been largely oriented toward creating new they are now commonplace. In fact, computers, capabilities, perhaps the best way to access current telephones, fax machines, television, radio, cinema, "dreams"is to examine what capabilities still do and print media all appearto be converging into a not exist but need to. single information technology that is inherently Clearly, the technology of computer music will centered around real time, telecommunicating, and continue to improve and get cheaper. General- audio-visual computer-based information exchange. purpose computers are becoming so fast that soon Explosions in information storage, processing, and we will have to slow them down-rather than

40 Computer Music Journal I

speed them up with specialized-but-limited hard- pilot. Another result is that airplanes sometimes ware-in order to achieve real-time performance. crash due to programmingerrors. This means that soon we will be able to retain the Imagine now a computer-basedmusic machine generality and flexibility of the Music V model and that senses the musical desires of an individual lis- have real-time interaction, too. Computers are- tener. The listener might simply turn a knob one or computing is-becoming an essentially free re- way when the computer plays something the lis- source. This inevitably will put more and more of tener likes, the other way when the computer does our experience of the world into the "computer- something less likable. Or, better yet, the computer mediated" category,meaning that our distinction could sense the listener's responses directly using, between reality and what we now call virtual real- say, body temperature, pulse rate, galvanic skin re- ity will gradually disappear.Rather than being a sponse, pupil size, blood pressure, etc. Imagine "dial-a-dream"situation, the real world will actu- what music would sound like if it continually ally be under the power and control of the mind. adapted itself to your neurophysiological responses, As an example of this, consider the "fly-by-wire" for as long as you wish. Such music might be more systems in modern aircraft.Instead of directly con- addictive than any known drug, or it might cure trolling the airplane, a pilot operates controls that any of several known medical disorders. are sensed by the computer, which in turn flies the Of course, such a music machine cannot be built airplane. In effect, the pilot's input only expresses today but that is precisely my point. Without such his wishes-the computer then responds to these dreams we have no idea how to direct our increas- wishes in a manner that is screened for such things ingly marvelous technology. Without dreams we as flight safety and passenger comfort. One result can only repeat what has already happened. is that far more complex airplanes can be built Dreams are the only true agents of change. than ever could be flown by an unassisted human

Moore 41