Nyssim Lefford Machine Listening Group An Interview Massachusetts Institute of Technology (MIT) Media Laboratory with Barry Vercoe Cambridge, Massachusetts 02139, USA [email protected]

Barry L. Vercoe (see Figure 1) is one of the foremost listeners and performers, and machine listening. developers and disseminators of His publications span many fields of research, in- technology. He is best known as the inventor of the cluding music theory (Vercoe 1968), signal pro- Music 360 (Vercoe 1973), Music 11 (Vercoe 1978), cessing (Vercoe 1982), music perception (Vercoe (Vercoe 1985), and RTCsound (Vercoe and 1997), and audio coding (Vercoe, Gardner, and Ellis 1990) languages for digital sound synthesis, Scheirer 1998). Among other awards, Mr. Vercoe which have been used by thousands of composers received a Guggenheim Fellowship in 1982–1983 around the world. He is also a respected composer for his innovative work exploring synthetic per- and a broad thinker who was one of the founding formers (Vercoe 1984; Vercoe and Puckette 1985) faculty members of MIT’s Media Laboratory. and other forms of automatic accompaniment sys- Mr. Vercoe was born in New Zealand in 1937. tems. In 1992, he received the Computer World/ He received bachelors’ degrees in music and math- Smithsonian Award in Arts and Media. The study ematics from the , followed of interaction between human and computer per- by the MusD degree from the University of Michi- formers remains his closest research interest. gan, where he studied under Ross Lee Finney. After This interview was conducted in February 1999 brief appointments at Princeton, Oberlin, and Yale, as part of the Media Lab’s “Digital Rewind” cel- he settled at MIT in 1971, where he was granted ebration of the 25th anniversary of the Experimen- tenure in 1974 and became a full . In 1973 tal Music Studio. Mr. Vercoe founded the MIT Experimental Music Studio (EMS), the first facility in the world to dedi- Lefford: Describe the first MIT experimental stu- cate digital computers exclusively to research and dio. Where was it located? composition in computer music. Vercoe: We began that work when I first arrived in The EMS was one of the innovating studios of its 1971. The first studio we had was in the basement time, and it oversaw the development or improve- of Building 26, where we had a computer given to ment of technologies such as real-time digital syn- MIT by Mathews—the Honeywell DDP-24. thesis, live keyboard input, graphical score editing, Max initially developed his Groove system on this graphical patching languages, synchronization be- machine, and was kind enough to give it to MIT tween natural and synthetic sound in composition, when I joined the faculty. It was a well-worn piece and advanced music languages. In 1976, the EMS of hardware, but I was ably assisted in its mainte- hosted an international conference on computer nance and software development by MIT graduate music; in 1981, Mr. Vercoe encouraged the MIT Steve Haflich. Press to take over publication of Computer Music That computer was put into the basement room Journal beginning with Volume 4. In 1985, the at the far end of Building 26, just at the time they EMS was integrated into the new MIT Media Labo- started constructing Buildings 36 and 38, the new ratory to carry on its work in a new, cross-disci- EE [Electrical Engineering] buildings. There was plinary context of multimedia research. lots of construction going on next door to Building At the Media Lab, Barry Vercoe has directed re- 26; they had to dig down deep. The roads here search groups on music and cognition, synthetic aren’t too high above the water level and the river, so there was lots of flooding. I think when they Computer Music Journal, 23:4, pp. 9–17, Winter 1999 drove in the piles for the new buildings, Building 26 © 1999 Massachusetts Institute of Technology. actually cracked a little. Anyway, we had basement

Lefford 9

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892699559968 by guest on 29 September 2021 Figure 1. Barry Vercoe.

This coincided with Professor [Amar] Bose mov- ing out of some space on the third floor of Building 26. He had stopped doing research at MIT at that point; he had been given what I think remains the only post of a full professor at MIT who teaches just one class and doesn’t do any research on campus. He moved out of his lab, and we promptly moved in because it had a nice acoustic treatment and was a much better environment for the new PDP-11. For the next twelve years, this was the area that the composers lived in. Lefford: What influenced your decision to build an all-digital experimental music studio? Did you draw from the experiences you had at other institutions? Vercoe: As a composer, I had no experiences in whatsoever in New Zealand where I did my undergraduate degrees. When I ar- rived at Ann Arbor, Michigan in 1962, there was a flooding. Fortunately, the DDP-24 was high enough visiting composer by the name of Mario off the ground that we never had problems from the Davidovsky who had helped Ross Finney do some six or eight inches of water that we would some- work at the Columbia University studios. Ross times find in the studio. That was our first place! asked Mario to come and help set up the studio at At the time, the composers taking my composi- Ann Arbor. tion class were primarily MIT students, plus a So for my three years at the University of Michi- couple of outsiders. They were actually generating gan, there was an analog studio under develop- sound using my Music 360 program, which ran on ment. I had a little bit of experience with the the IBM 360 or 370 system in the adjacent build- cut-and-splice technique, but I wasn’t really enam- ing. Mostly, they lived in the keypunch room in ored of that—although Mario was clearly a master the EE computer facility, but used our DDP-24 for at that classical studio method of constructing digital-to-audio conversion. pieces, and I admired his music. In fact, years In 1973, after spending two years developing a later, on several occasions, I conducted his music, real-time digital design—which I still which I still like very much. think was the best thing I ever did at MIT—I re- In parallel with this, I was supporting myself as ceived an offer from Jerry Weisner [president of a grad student at the work- MIT at the time] to have Digital Equipment (DEC) ing as a statistician. My degree in mathematics got build this machine for us. This would have been me a job in human genetics and muscular dystro- the world’s first real-time digital synthesizer, but I phy research. I learned to program out of sheer never followed up on it because along came an- need. Fortran wasn’t around; well, maybe it was, other offer, from Jerry and Ed Fredkin, who was but I learned to program using the Michigan Algo- the head of the Computer Science Lab at the time. rithmic Decoder (MAD) language. So, this MAD Jerry and Ed asked DEC to give us an entire language got me first into computing. PDP-11/50, the top of the line at the time, so that By this time I was finishing my doctoral work at we could use it to build the whole synthesizer in Ann Arbor, and had accepted a teaching job at software. We would never have to worry about Oberlin Conservatory; but I was already familiar hardware obsolescence, and this had a certain ap- with the possibilities of digital sound synthesis, peal to me. So I accepted that solution in the sum- and spent the next summer at Princeton with mer of 1973. Godfried Winham. There was no DAC there, so we

10 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892699559968 by guest on 29 September 2021 would travel to to do digital-to-audio Vercoe: Yes, in the sense that I’ve always practiced conversions from a big 2,400-foot reel. By the end that. Whenever I’ve run summer workshops for of the summer, I had written parts and pieces of composers, I’ve forced them to come to grips with things that ended up getting incorporated into real the computer language. It’s not very difficult to pieces I wrote later that year as the composer-in- patch oscillators and filters together using some residence for the city of Seattle. I wrote a big piece kind of symbolic representation. Some of them find for the Music Educators National Convention, it difficult to get into, but if they have some degree which was in Seattle that year, for a very large or- of control over their instrument, they are able to chestra: wind ensemble, brass band, string en- intuit more naturally, to be more imaginative and semble, two choirs (one singing Latin, the other a to be more creative. text by the Japanese Toyohiko Kagawa), percus- Moreover, they have faith in the system, and be- sion ensemble, soloist, and computer sounds [Di- lieve it will be able to do what they want it to do. I gressions, recorded on Crest Records, 1968]. Those think when you buy an innovative but shaky computer sounds had come from my work at black box—particularly computing devices that Princeton that summer. sometimes crash and need rebooting—if you don’t After my year as composer-in-residence in Se- know the technology inside at some level, you’re attle, 1967–1968, I decided to go to Princeton and just in fear and dread that this thing is suddenly work seriously on digital sound synthesis. I did going to gobble up your favorite tune. this for the next two years, the result of which was Some kind of familiarity is essential. Even dur- Music 360. Next, I spent a year at Yale where I ing the 1970s, although we had developed quite an taught sound synthesis, as well as Schenkerian orchestra-instrument library that one could tap analysis and theory. By the time I was invited to into, whenever I taught a course to composers, I MIT, I was totally committed to digital synthesis, never told them about the instrument library. I as opposed to constantly trying to keep analog de- just told them about the language and how to put vices in tune. There was no doubt in my mind that things together—oscillators and filters—and said, it was a digital studio that I was going to develop. “Okay, go to it.” Only later would they discover I think my appointment here was because MIT’s the library and say, “Well, this could have saved and David Epstein had visited Bell us a lot of time.” But also, it would have saved Labs under Jerry Weisner’s urging to see what they them learning things, and I think it was the could find out about the electronic music scene. knowledge that gave them power. They apparently heard one sound that Max Lefford: Did the EMS have a unique compositional Mathews made on his Groove system under com- aesthetic? If so, what characterized the works cre- puter control, looked at each other, and said, ated there? “That’s the future.” Vercoe: I think the relatively few pieces that I com- I was appointed because I was so totally com- posed at the studio tended to combine computer- mitted, and, by that time, somewhat expert in processed sound with live instruments. I come from digital sound synthesis. This thereby affected the a tradition of live performance, and in particular, as nature of the studio that took off at MIT. Whereas a conductor, I’m still in love with choir music. I’ve some people might have thought in 1971 that always been in love with the live aspects of music, starting an electronic music studio at MIT would and I see them as extensions of natural body motion. mean buying a Moog synthesizer, putting it in the I don’t think computers will ever replace that. basement, and handing keys out to undergradu- There’s a contribution that humans do make, ei- ates, my idea was very different. So, we started out ther through vocal cords or tactile control or what- with a computer studio from the word “go.” ever, that is an essential human communication. Lefford: Do you think it’s important to build tools Despite some claims to the contrary, I’ve never that can be used directly by the composers without found it a really successful venture to replace this the aid of an engineer? with cybernetic activity.

Lefford 11

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892699559968 by guest on 29 September 2021 My work, at that time, was writing pieces like I think those two models, either Mario’s or the one I did for [MIT professor] Marcus Thomp- mine, inspired the pieces by composers like Peter son, which was Synapse for viola and computer Child and Martin Brody. Their pieces gave the (recording CRI 393, 1981). It was rather like a viola same challenge to performers. They had a canned, concerto with the computer playing the orchestra precomputed part to synchronize with, so that the part. But you can tell from the name Synapse that entire load of synchronization fell on the shoul- I was very interested in the real “touchy-feely” ders of the performers. The idea of seeing these as contact between the live acoustic instrument and two forces of equal importance but of very differ- the computer. That is, whatever that could mean ent natures made for a more interesting piece. in an environment where the computer part had to This was at the back of the composers’ minds. be precomputed, preprocessed, prerecorded, and The other half of the pieces we produced were canned on a fixed tape, invariant under any perfor- for computer alone. Since we had very high-qual- mance conditions. ity four-channel floating-point D-A converters At that time, human-computer interaction, or (due to the work of Richard Steiger), we could get the relation of the two forces, was strictly on the wonderful sounds that had their own appeal to shoulders of the live performer. This obviously de- composers. For instance, when I went to Paris in served some attention, so when I went over to the year of 1983–1984, I invited Jim Dashow to Paris [on the Guggenheim Fellowship] to work come and take over my classes. Jim was a with Pierre Boulez and his group, I developed an Brandeis-trained composer, living and working in automatic synthetic following system. Notice that Italy. He had been a real expert user of Music 360, I describe that as a following system, since I regard and while at MIT wrote a wonderful piece for the human performer as the leader and the com- computer alone called In Winter Shine. This work puter as responding with accompaniment. I have is a big four-channel piece with lots of interaction never really successfully modeled, nor even had among all four channels. very much faith in, the notion of the computer This piece is an example of just what a carefully leading the way. It doesn’t seem very appealing to constructed computer piece can do. It controls the me. It might occasionally be an interesting thing timbres in special ways, where the harmonic over- to experience, but I still have primary interest in tones of one sound are carefully related to the har- humans as the entities I’m communicating with monic overtones of an adjacent sound. Jim built and hearing from. complex tone structures that related to adjacent Lefford: And the other composers that came to work tone structures in the way that Schubert often re- here? Did this characterize their work as well? lated one key area to another key area using pivot- Vercoe: It was about half-and-half. Half of the note modulations. I found this work of Jim’s quite people were inspired by the models I had given fascinating, the sort of thing that only a carefully them, which models were, in fact, based on Mario crafted computer-synthesis system could deliver. Davidovsky’s models. He was writing synchro- We were also challenged to deliver that sound in nisms, which as their name implies, seek some concert form. So, every summer we put on big sort of synchronization between the tape part and concerts that came from this repertoire of EMS the live instruments. In point of fact, the synchro- pieces; the concerts were very popular and suc- nization is [often] very sparse and not tightly orga- cessful. They always presented a mix between nized at all. live-instrument participation and computer alone. What I set out to do in the 1970s with my pieces I found from a concert-programming standpoint was to engage the live performer and the computer that you could not have more than two computer- in a tightly constructed composition where the alone pieces played back to back in concerts before performer had to listen carefully to what was go- the audience became a little restless—they needed ing on in order to achieve the proper ensemble. something to look at. So, we would typically alter- The pieces demanded that. nate them with pieces involving live players too.

12 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892699559968 by guest on 29 September 2021 I think the composers felt the same way. They ers. The others would be from industry, looking might do one piece for computer alone; the next into this new world of digital audio and wishing to time they would do a piece including live instru- get a head start in it. ments that they were probably born and bred to The money from the industry attendees, who love. They would never just change over com- usually paid full freight of US $1,000 or so, helped pletely if they had any sense. support the composers, who were mostly without Lefford: Did the EMS collaborate with other insti- industry support. The composers would not only tutions and studios? attend that workshop but were then able to stay Vercoe: Not directly. I visited places, of course: I on for another three or four weeks and write a had visited Dartmouth; I spent two and a half years piece. In that way you might say those early com- at Princeton; and I taught at Yale for a bit. So, there posers were composers-in-residence, because they was some interaction even before I came to MIT. were being assisted by industry to stay for the ad- After I came here, people from other institutions ditional time, to get a piece out the door and into a would come and visit and perhaps take the sum- concert, plus a review in the Boston Globe. mer workshops that I was conducting every year. I didn’t make the final decision as to who got to I don’t think one would say there was any real stay on until the second week of the workshop. collaboration until three of the leading studios at There was nothing guaranteed, but toward the end the time, MIT, San Diego, and Stanford, each got of the second week it became clear to me who had funding from the System Development Founda- the potential to really do pieces for our public con- tion. In our case, we already had our own com- cert. It was typically the trained composers who puter. Stanford had been using the PDP-10 that had the musical maturity to get a piece out the belonged to their AI Lab, which was moving onto door. But sometimes the people from industry were the main campus, so they were in real need. San musically surprising, and some composers turned Diego had just purchased a VAX computer, but out to be technologically inept and simply hopeless had no way to maintain or support it. in the medium. I think that during several years of The support from the System Development Foun- summer concerts, only one of the selected people dation was timely in that regard. We’d already had ever failed to complete a piece in time. the PDP-11 running for some time, so we didn’t By the time we moved to the Media Lab, I had need a new VAX, but the funding did enable us to been giving these summer workshops for six to support some grad students working on serious re- eight years, and I was quite worn out from it. After search projects. John Stautner and I came back from Paris and my Guggenheim Fel- did a lot of their early work in this field, and they lowship at IRCAM, I really was looking to do turned out some first-rate papers on various topics. something different. Lefford: Were most of the composers-in-residence At that time, we had Judy Whipple, who was a familiar with computer-assisted composition when wonderful assistant and a real source of ideas for they arrived, or were they exploring the medium our continuation. Judy wrote wonderful proposals for the first time? to the National Endowment for the Arts based on Vercoe: Well, the composer-in-residence notion the workshops we had been giving. As a result of didn’t really crystallize until we moved to the Me- the music from the workshops, we managed to get dia Lab in 1984–1985. Prior to that, the composers money from both the Massachusetts Arts Council came during the summer as part of my composers’ and the National Endowment for the Arts to bring workshop and paid a summer workshop fee to in real composers-in-residence for three or four come and work here, typically for six weeks or so. months at a time. Some would come for one The first two of these workshops included an in- month, get to know things, then go away, write a depth crash course on how to use computer tech- piece, and come back and spend three months put- niques. There would be about 35 people in the ting it together. Others would be in residence workshop, 12 or maybe 15 being serious compos- throughout.

Lefford 13

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892699559968 by guest on 29 September 2021 What I enjoyed about those interactions was Lefford: What technological revolutions have been that having composers around for that long en- important in shaping the EMS and the field of com- abled the Media Lab students to really rub shoul- puter music? ders with them, to become assistants in their Vercoe: In the early days at Princeton, we initially endeavors, to interact with them, and to learn a lot worked on the IBM 7094 and IBM 360 computers, from the whole creative process. This doesn’t hap- which were by today’s standards very slow; to syn- pen these days. We don’t have artists coming thesize a good 3-min high-fidelity piece took some through and being in residence for periods like 3–6 hr of computer time. We must have had some that. In part this is because the Endowment for the belief in where it was all going; the difference these Arts has ceased to be so active in areas like this. days is that you can now do pieces of that complex- But also, it is because in the early days of the Me- ity at real time or better. dia Lab, we were looking for any points of applica- Once the level of synthesis gets to be comput- tion that would set us apart from other activities on able in real time, you have a very different situa- campus or even around the country. I think we’ve tion. If it’s 99 percent up to real time, then it has become a little too set in our ways of late. We’ve to be recorded and played back later. But once it settled down into doing much more fundamental re- hits 100 percent, you’re suddenly in a situation search as opposed to artistic exploration. Maybe where you can play the entire piece and interact that’s what the world believes is the only way to with it while it’s actually being created. That in- have technology transfer from a lab into industry. duces a whole new way of thinking about things, But I think the most progress is made when the and it engenders new, interactive opcodes in the technology is being pushed by the creative artist. computer music language. I do miss the creative challenges that went along The pieces that can happen these days, on ei- with the early days when we were exploring all ther big expensive workstations which most com- kinds of things, where the visiting composers had posers don’t have access to, or relatively good support from the endowments. We would inexpensive PCs with good, fast DSP cards, can typically obtain for them a US$ 8,000–10,000 sti- happen in real time. This invites all manner of in- pend for a 3–4 month residency, plus additional teraction and ways of thinking about audio music money to produce the piece. We had good rehears- processing and control processing, and enables a als and smashing performances. very different aesthetic. I think, if anything, the When I conducted the premiere of Jonathan technological breakthrough has just been the Harvey’s From Silence, I had some wonderful play- natural one of computers getting faster and chang- ers—Lucy Stoltzman, Marcus Thompson—superb ing our low-expectancy ideas into real-time, right- performers, with adequate rehearsal time, which now things. That has been the breakthrough. most contemporary music performances around Of course, there have always been new techniques town don’t get. We were able to put on a really de- for synthesis, from phase vocoders to physical mod- finitive performance of those pieces. But, that eling. The practical signal-processing techniques are phase has passed. I have less energy now, and we driven not by music/audio needs alone, but from don’t have the financial support to keep doing that. DSP research. We then become the beneficiaries, Besides, running over and putting on concerts at just as composers in every day and age have always Kresge [the main MIT performance hall] on the been the beneficiaries of the technology at the time. other side of campus was always such an effort. People have always learned to develop musical in- We did operate in the Media Lab’s Cube (where struments with responsive control. People were able the assisting computers could be right in the next to build violins instead of viols, make instruments room) for a few years, but now we’ve lost that as a that could stand out from an ensemble and be solo convenient performance space. [The Villers Experi- instruments, and put valves on trumpets. mental Performance Space at the Media Lab was When you have a technology that enables com- converted into offices in 1998.] posers and performers to do new and different

14 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892699559968 by guest on 29 September 2021 things, that will always excite the imagination. Leacock had pioneered filmmaking with the Super- That happened with instruments in the Western 8 camera. Each of us was struggling to apply tech- culture around 1600, when they became a real in- nology to artistic expression. But fortunately, we dependent force, able to do more than just double were perceived by MIT’s president Jerry Weisner to the voices. What we have in this day and age, be- be people that, again, were worth betting on. sides voices and instruments, is a third medium— What would happen if we put all these people in the electronic medium. It is still in its relative one place? The notion of a Media Lab came from youth, but it will benefit from parallel industrial the desire to bring these disparate efforts from the developments which composers and artists, who different corners of campus to one place so they are always the scavengers of contemporary tech- could gain strength in numbers, and to foster some nologies, will also put to creative use. interactions between them. Lefford: What sort of research did you pursue dur- The initiative was first called Arts and Media ing your Guggenheim Fellowship? Technology, or Computers in the Arts. The word Vercoe: The Guggenheim Fellowship was really for “media” was there, but the idea of calling it a composition, but when I got to Paris, I developed a “media lab” was Nicholas’s. In many ways, MIT is system using computers to follow live performers. very territorial. You can’t suddenly call yourself a On going back and reading my Guggenheim applica- computer science department. Similarly combin- tion, it was quite clear that this was on my mind at ing “computers” and something like “art” will im- the time, but I had only sketchy ideas of how to do it. mediately raise eyebrows in some other corner. I’m quite sure I didn’t have the technological cre- But when Nicholas proposed the word “media,” dentials to convince a Guggenheim scientific com- which carried such bad connotations around MIT, mittee that this project would be a good thing to everyone around campus said “Sure, you’re wel- back. Nor did I yet have enough proof that this was come to it.” I think they simply lacked the fore- a sure-fire way to the future. I think I was a good sight to realize that media was going to be big bet because of the strength of our summer work- business in the near future. shops for composers, and because we had caused a We got our own space and our own curriculum lot of innovative computer music to be produced. because no one wanted the association. We were I’ve always been torn, devoting half my life to expected to fail, of course. We had lots of meetings one thing and half to other things. My misspent on building plans, and how the groups were going youth was to spend hours making music (my dad to interact. Then on moving day everyone moved was a jazz performer) and the other hours explor- the old equipment from their labs in here, and ing math and science, the two never seeming to spent time making it all work the same way again. have a future together until they suddenly found The six originals were Marvin [Minsky], Seymour confluence in this new field of computer music. I [Papert], Nicholas [Negroponte], Ricky Leacock, was fortuitously equipped in both fields, able to Muriel Cooper, and myself. We were six almost- guide the synthesis of their combination. independent entities just operating under the same Lefford: What were the early days of the Media Lab roof, but the difference was that we were now able like? What was the EMS’s involvement? to share students. And while the senior faculty Vercoe: In my early years at MIT, there were sev- rarely got to collaborate, the students were the eral activities on campus that were using comput- ones who combined the individual territories and ers in one way or another. One was the EMS. In put together something greater than the parts. parallel, Nicholas [Negroponte] was developing the Lefford: How have your research interests changed Architecture Machine, which began as a sophisti- since the Media Lab opened? cated architectural drafting system, then got in- Vercoe: I don’t think they have. I have always creasingly into graphics and human interaction. been interested in finding out enough about how Muriel Cooper and Ron McNeil were getting com- music works, as to make technology a part of it. puters to control mural painting, and Ricky The musically responsible thing is to avoid having

Lefford 15

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892699559968 by guest on 29 September 2021 the technology tail wagging the artistic dog, but have very loud or very soft representations of the rather to have the technology following and even same signal-to-noise [SNR] ratio. This is basically enabling the musician. a floating-point representation. What has changed and will continue to change is He became interested in this problem of creating how I implement that research interest. Getting a floating-point D-A converter, which he helped us computers to follow live performers is one thing, build. He put his top designer, Bob Talimberus, on but there are still some real problems, such as how this, and we built the first floating-point digital do you separate one sound from another? If com- audio converters. They were wonderful gold-plated puters are hearing music the way humans do, then things, beautifully engineered, with a very wide they must be able to do signal separation and in- range and clean four-channel sound, and they strument recognition. served us well for many years. These are things that I had to gloss over in my We thought that when the audio world ulti- early instantiation. Now, I’m happy to see the stu- mately went digital—which it had to eventually—it dents deeply involved in what we call “machine would be with a floating-point representation that listening.” For me, machine listening is a subset of covered all of the capacities of the ear in the most the larger goal of synthetic listeners and perform- efficient way. We had shown that to be about 13 ers in a total musical sense. bits of SNR as the fractional part, and 3 bits (a base- Lefford: You continue to have a closer working 2 factor of 8) of floating-point dynamic range—in relationship with industry than some of the other effect, a 21-bit dynamic range, which is about 125 experimental music groups. How does this impact dB. But we were wrong. The industry eventually your research? went for 16-bit linear (96 dB), which never made Vercoe: The original Experimental Music Studio any sense to me, because the ear is not linear. This was not supported by industry at all, except for the has all sorts of noise-floor problems in a 16-bit original gift from Digital. The first research support DAC. At that time, it was hard to guarantee the was from the System Development Foundation, stability of the least-significant bit, because that re- which had been a spin-off of Rand Corporation. I quires the most-significant bit resistor to maintain suppose you might call that an industrial relation, its accuracy to 1 part in 65 thousand, which was be- but I don’t think that what we were doing in music yond consumer technology at the time. research and music production was of serious inter- One of the MIT who was an early est to industry except in very small ways. skeptic of our building an all-digital studio was In the early days, we needed a reliable, high- Francis Lee, who was teaching a digital course in quality, digital-to-audio conversion system. The EE. Soon after we had built our floating-point D-A best DAC converters were those manufactured by converters, Francis used them in his own research companies like Analogic Corp., north of Boston, on digital audio reproduction. A bit later, he was and Analog Devices, south of Boston. We went to doing audio work in his new company, which he Analogic president Bernie Gordon and said, “You called Lexicon. So, despite his initial skepticism, may make the best DACs in the industry, but they he must have felt we were on the right track and don’t have enough dynamic range for real music.” saw there was opportunity here for industry. To which Bernie responded, “How come?” Although the fractional part of our D-A convert- We explained that the ear has a sensitivity to ers was from Analogic, the floating-point scaling distortion of about 70 dB but also an adaptability was from Analog Devices (using their multiplying to local loudness levels that moves really fast. We DAC). We showed Analog Devices that this could outlined for him [graduate student] Rick Steiger’s be used potentially in a consumer context. notion of building a floating-point D-A system, I think what artists often do as a function in soci- which would convert a sound from digital to audio ety is show engineers new ways of doing things— and also scale it like the ear scales incoming sig- creative things. Engineers like to feel they’re nals—a sort of automatic gain control. You could creative too, but they must realize that artists are

16 Computer Music Journal

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892699559968 by guest on 29 September 2021 creative in a different way. Perhaps only artists ones who will make sure that technology is of the know how to push the limits of devices and thereby form that society really needs. engender a rethinking of how they can be used. Lefford: What is the next step in synthetic per- former and machine listening research? Acknowledgment Vercoe: I think progress in synthetic listeners and performers will depend on how much energy there I am grateful to Eric Scheirer for his help with is, and how many people become committed to the this article. work of insuring that technology remains a tool in the hands of the human artists. Human arts and crafts are, after all, what’s important—peoples’ References ways of communicating. It’s so easy to turn on a bit of technology these Vercoe, B. L. 1968. “Reviews of Books on Computers and days, just let it go “widdly-widdly,” and some will Music.” Perspectives of New Music 10(1):323–330. think that’s music or art. To some degree that’s Vercoe, B. L. 1973. The MUSIC 360 Language for Digital okay, but I like to believe there will always be people Sound Synthesis. Program documentation. Cambridge, who have a higher sense of what art and expression Massachusetts: MIT Experimental Music Studio. Vercoe, B. L. 1978. The MUSIC-11 Language for Digital can be, and that technology must be a servant of art Sound Synthesis. Program documentation. Cambridge, and craft. To make sure it’s a servant, it needs to be Massachusetts: MIT Experimental Music Studio. well schooled and well trained and practiced in serv- Vercoe, B. L. 1982. “New Dimensions in Computer Mu- ing our needs in the musical arts—to serve the needs sic.” Trends and Perspectives in Signal Processing of composers and performers in their goal of commu- 2(2):15–23. nicating with audiences. Vercoe, B. L. 1984. “The Synthetic Performer in the If computers can find a way to be active or to Context of Live Performance.” Proceedings of the have responsible roles in collaborative production, 1984 International Computer Music Conference. San then they can be musically mature fountains, just Francisco: International Computer Music Associa- like us. But if computers remain rack-mounted tion, pp. 199–200. buttons and switches at arm’s distance, then that Vercoe, B. L. 1985 (rev. 1996). Csound: A Manual for the Audio-Processing System. Program documenta- will be their only role. It’s a question of where tion. Cambridge, Massachusetts: MIT Media Lab. people think technology can go, what they aspire Vercoe, B. L. 1997. “Computational Auditory Pathways technology to be. It also depends whether the com- to Music Understanding.” In I. Deliège and J. Sloboda, posers, the creative artists, are willing to give up eds. Perception and Cognition of Music. London: Psy- some composing time to make sure technology chology Press, pp. 307–326. does get there. It won’t happen without the artists’ Vercoe, B. L., and D. P. W. Ellis. 1990. “Real-Time participation, since the engineers don’t know what Csound: Software Synthesis with Sensing and Con- we need. But if we put some energy into redirect- trol.” Proceedings of the 1990 International Com- ing the technology, then this can be the third me- puter Music Conference. San Francisco: International dium we have been waiting for. Computer Music Association, pp. 209–211. [The EMS] was a lot of fun. There’s a lot of mu- Vercoe, B. L., W. G. Gardner, and E. D. Scheirer. 1998. “Structured Audio: The Creation, Transmission, and sic and reliable software out there in the world Rendering of Parametric Sound Representations.” now that wouldn’t have happened without it. Proceedings of the IEEE 85(5):922–940. While it was very time consuming and I gave up a Vercoe, B. L., and M. S. Puckette 1985. “Synthetic Re- lot of composing time to make it happen, it was a hearsal: Training the Synthetic Performer.” Proceed- sure way to move the technology ahead in artistic ings of the 1985 International Computer Music ways. I think a lot of people have benefited from Conference. San Francisco: International Computer this, and I’m hopeful that the goals will continue Music Association, pp. 275–278. to be a driving force for some people. They are the

Lefford 17

Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/014892699559968 by guest on 29 September 2021