Experienced-Based Music Composition

EXPERIENCE-BASED MUSIC COMPOSITION

CARL WESCOTT and ROBERT LEVINSON Department of Computerand Information Sciences University of California, Santa Cruz Santa Cruz CA 95064 USA

1 Overview 2 Related Research

"If a complex structure is completely "[Musical fragments] that please me unredundant - if no aspect of its struc- I retain in memory, and am accustomed, ture can be inferred from any other - then as I have been told, to hum them to my- it is its ownsimplest description. Wecan self. If I continue in this way, it soon exhibit it but we cannot describe it by a occurs to me how I mayturn this or that simpler mechanism." - Herbert Simon, morsel to account, so as to make a good The Architecture of Complexity, p. 478. dish of it, that is to say, agreeably to the rules of counterpoint, to the peculiar- ities of the various instruments, etc. All Humancreativity produces new forms, or this fires my soul, and, provided I amnot patterns, from previous experience, in such di- disturbed, my subject enlarges itself, be- verse activities as composingmusic, writing prose, comes methodized and defined, and the and playing chess. We believe that humans have whole, though it be long, stands almost domain-independent pattern-based neurobiologi- complete and finished in my mind, so cal mechanisms that provide the foundations for that I can survey it, like a fine picture creativity, and propose that systems which com- or a beautiful statue, at a glance. Nor bine and alter simple patterns to model complex do I hear in nay imagination the parts systems can effectively simulate creative intelli- successively, but I hear them, as it were, gence. all at once." Wolfgang AmadeusMozart, To study these mech- in Vernon, Creativity, p. 55. anisms and experience-based creativity, Smurph, System for Musicomposition Using Repeated Pat- The original computer music composi- tern Hierarchies, has been developed. Smurph is tions were stochastic music composed using ran- intelligent software that learns to composemusic dom numbers or processes [Hiller & Isaacson, based on discovery of repeated musical patterns 1959] [Xenakis, 1971]. Often, these aleatoric de- and its creative combinations and variations of vices were interactive compositional tools to assist those patterns. The goal is for Smurphto learn to the humancomposer, such a.s Koenig’s mentor sys- compose original works of music entirely from its tems Project One and Project Two [Laske, 1981]. experience of ’listening’ to music and its analyses Another approach has been to use formal gram- thereof, with virtually no music-theoretic knowl- mars to model and generate music [Laske, 1973] edge or heuristics. [Bernstein, 1976] [Smoliar, 1976] [Winograd, 1968] This paper summarizes related research, [Holtzman, 1981] [Lerdahl & Jackendoff, 1983] describes the design of our experience-based chess [Roads, 1978]. and music programs, expounds on the patterns, More sophisticated music composition hierarchical structures, and linguistic properties programs have been written that make use of rules of music, and concludes by outlining ongoing re- based on the music-theoretic heuristics of tradi- search. tional counterpoint species [Schottstaedt, 1984]

119 [Ebcioglu, 1986]. These systems attempt to min- pattern ranging from 0 to 1, with 0 indicating a imize penalties for broken rules using backtrack- sure loss and 1 indicating a win. Unlike most in- ing to compose an original piece of music or har- telligent chess programs, Morph does not search monize an existing one phrase-by-phrase. More the tree of possible movesfor future consequences recently, there have been several neural net im- of its actions. Instead, Morphevaluates the pat- plementations of compositional algorithms [Todd, terns that it finds in its possible legal movesand 1991] [Mozer, 1991] [Lewis, 1991] [Kohonen, Laine, chooses the movewith the best possible combina- Tiits, & Torkkola, 1991] and algorithms to learn tion of patterns. After the game is over, Morph and produce well-formed melodies [Underwood, adds and deletes patterns and adjusts its pattern- 1992] [Gilbert, 1993]. Someresearchers are study- weights using temporal difference learning [Sut- ing the related field of real-time musical accompa- ton, 1988] and the simple feedback of whether it niment [Dannenberg, 1984] [Vercoe, 1984] [Baird, won, drew, or lost the game. Blevins, & Zahler, 1989]. To study the application of similar princi- Our computational music research builds ples to another domain besides chess, music com- on the work of David Cope, whose Experiments in position was chosen, since: Musical Intelligence (EMI) software has created ¯ pieces in the style of Bach, Mozart, Joplin, Gersh- Music has relatively simple, discrete patterns win, and Bartok, amongothers. In many respects, that can be sequentially analyzed and unam- Cope combines some of the above techniques: EMI biguously represented. uses an Augmented Transition Network (ATN) * Due to the hierarchical structure of music, grammaras well as a rule-base to describe the syn- each note can be interpreted on manydiffer- tax of music. EMIbuilds a dictionary of signature ent levels, as part of a bar, motive, phrase, phrases by listening to a composer’s works, and section, or theme. then composesmusic using these stylistic patterns and its rule-base [Cope, 1991a]. ¯ Music composition, like winning chess, is of- Smurph does not use transformational ten cited as an exampleof intelligent and cre- grammarsnor musical heuristics. Instead, it uti- ative behavior. lizes only its probabilistic analyses of input music ¯ There are digitally stored musical databases to determine the most suitable patterns to follow that can easily be accessed on-line. its current output. It stores repeated patterns in a hierarchical database, and combines and alters ¯ Psychological aspects of human music com- them to form new musical phrases. position have been widely studied and pub- lished. 3 Morph & Smurph Like Morph, Smurph stores musical patterns that it recognizes in a hierarchical database "While Babbage dreamt of creating using a more-general-than operator. Rather a chess or tic-tac-toe automaton, [Lady than storing weights with these patterns, though, Ada Lovelace] suggested that his Engine, Smurph keeps track of the probabilities of oc- with pitches and harmonies coded into its currence of these patterns, modeling music as spinning cylinders, ’might compose elab- the result of a multiple-order Markovinformation orate and scientific pieces of music of any source. Thus, when composing, Smurph combines degree of complexity or extent.’ " - Dou- these smaller patterns to produce music that has glas Hofstadter, Godel, Escher, Bach, p. some of the style or signatures of the input music 25. pieces that it has listened to. Smurph’s patterns currently use only the Morph, an experience-based, adaptive, melody of a musical piece, but Smurphwill eventu- pattern-oriented program that learns to play chess ally analyze all the voices in the input data stream. by watching or playing games, has already been The input music is all transposed to the key of C, built [Levinson, 1991]. Morph was developed and rhythms are represented only in a temporally based on our APS (Adaptive-Predictive Search) relative manner. Thus, two eighth notes followed paradigm for machine learning. Starting with lit- by a quarter note are perceived to be identical to tle a priori knowledge, Morphstores the patterns two quarter notes and a half note, assuming of that it finds on the chess board in a hierarchical course that the pitches mapone-to-one after trans- database, along with its estimated weight for each position. Smurph’s database routines are written

120 in C++ and interface with Macintosh MIDILISP unary operators like Generalize, Invert, and Re- code to parse and output MIDIevents to and from ?)erse. the computer. 5 The Hierarchical 4 Patterns Structure of Music "One of the most interesting aspects "As a piece of music unfolds, its of the world is that it can be considered rhythmic structure is perceived not as to be made up of patterns. A pattern is a series of discrete independent units essentially an arrangement. It is charac- strung together in a mechanical, additive terized by the order of the elements of way like beads, but as an organic pro- which it is made, rather than by the in- cess in which smaller rhythmic motives, trinsic nature of these elements." - Nor- while possessing a shape and structure bert Wiener, in Gonzalez & Thomason, of their own, also function as integral Syntactic Pattern Recognition: An Intro- parts of a larger rhythmic organization." duction, p. 1. - Grosvenor Cooper & Leonard Meyer, The Rhythmic Structure of Music, p. 2. The easiest way to form complex patterns from simple ones is to group the smaller patterns According to Heinrich Schenker, the orig- together. Experimental evidence shows that the inator of modern harmonic analysis, music can be Gestalt principles of temporal proximity and sim- understood on three principal levels: foreground, ilarity are important determinants of grouping in middleground, and background. Schenker advo- music. Grouping objects together improves per- cates a universal deep structure for music based on ception and recognition visually as well as aurally: his formalisms not unlike Chomsky’s. This struc- "Regular, symmetrical, simple shapes will be more ture has two components, a fundamental melodic readily perceived, appear more stable, and be bet- line and a bass line, which may be discovered ter remembered than those which are not. Thus, by reeursively removing extraneous musical notes for instance, conjunct pitch sequences (the law of from a piece to reveal its harmonic and contrapun- proximity), continuing timbres (the law of similar- tal skeleton. ity), cyclic formal structures (the law of return)- The structure of music is the bridge be- all help to facilitate perception, learning, and un- tween the composer and the listener. Music’s var- derstanding." [Deutsch, 1982] ious aspects - pitch, intensity, duration, timbre, Musical groups or phrases can also be texture, and harmony - are interwoven at many altered by the simple procedures of retrograde, different levels in an architectonic fashion. [Nar- inversion, and transposition. "Tonal melodies mour, 1990] Experimental evidence shows that hu- are often generated, either consciously or uncon- mans retain pitch information in their memories sciously, from one or more motives (gestures less at different levels of abstraction. [Deutseh, 1982] than a phrase long). These typically fall so nat- Like pitch, the rhythms in a piece of music can be urally into the logic of the line as not to be dis- analyzed on multiple hierarchical levels. [Cooper tinguishable. Motives are varied by transposition & Meyer, 1960] Each note is analyzed in terms of (different levels of the scale), by inversion (inter- its contextual tonal hierarchies, [Krumhansl, 1990] vals in the opposite direction), and by extension offering the musical stability which leads to emo- or truncation." [Cope, 1991a] Traditional fugues tion and meaning in music. [Meyer, 1956] consist almost entirely of multivoice variations on Smurph’s hierarchical storing and analy- their melodic themes. Marvin Minsky believes sis of patterns is consistent with the multilevel that the most significant aspect of these simi- hierarchical structures in many forms of Western lar phrases, motives, and rhythms is the higher tonal music. The classical sonata, for example, is level differences between otherwise identical ob- a hierarchy of subunits, and the "subunits in one jects. [Minsky & Laske, 1992] branch are generally found in identical or varied Currently Smurph stores only patterns form in other branches. For example, the themes that actually occur in the music that it listens from the exposition are also heard in the devel- to, but mechanisms are being added so that vari- opment, recapulation, and coda. but in different ants can be produced by using binary pattern orders, keys, and tonalities. Similarly, the mo- transforms such as Most CommonSubpattern, and tives from one theme also appear throughout the

121 movement,but transposed, inverted, stated in ret- able at all stages of the compositional process, it rograde, diminished or augmented, or otherwise reflects music’s potential ambiguities through its altered. Although the formal structure of a musi- stochastic generator. Also, rather than having to cal composition varies widely from one composer rewrite its grammar to create music in another or style to another, the hierarchical nature of the style, Smurphsimply digests, analyzes, and emu- form remains nearly universal." [Bateman, 1980] lates new input music.

6 Music as a Language 7 Domain-Independent "Just as letters are combined into Creativity words, words into sentences, sentences into paragraphs, and so on, so in mu- One of the primary high-level goals of this sic individual tones becomegrouped into research is to explore to what extent creativity can motives, motives into phrases, phrases be achieved in individual domains using principles into periods, etc. This is a famil- that are essentially domain-independent. In the iar concept in the analysis of harmonic case of Smurph the goal then is for Smurph to and melodic structure." - Grosvenor exhibit the musical characteristics of those that Cooper and Leonard Meyer, The Rhyth- have formal music training, without explicitly be- mic Structure of Music, p. 2. ing given music theory as in knowledge-based composition systems. Smurph’s expertise must come This multilevel hierarchical structure al- from its experience in "listening to" well-formed lows music to be viewed as a language. Besides music. the more formal approaches to describing music The first primitive version of Smurphwas in terms of languages’ semantics, parts of speech, based strictly on its hidden Markovmodel. As was transformations, and musical grammars, [Bern- expected, if Smurph used a high order of proba- stein, 1976] it is widely acknowledgedthat the hi- bilistic synthesis, redundant repetition resulted. If erarchical structure of music lends itself well to the order of synthesis was too low, the resulting another analogy. Music groups itself into phrases, output note sequences were mostly random and motives, and section, just as language is composed not typical of the input pieces’ style. of words, sentences, and paragraphs. Also, like Smurph’s first compositions contained co- language, music creates tension and relief through herent phrases, but lacked some of the basic el- its use of dissonance and cadences. ements which characterize Western tonal music. As a language, it possesses the same uni- Since each note in its compositions was determined versal deep structure that Chomskydiscusses in by probability, the overall structure which perme- linguistics. [Chomsky, 1965] Music can be dis- ates most music was missing. Repetition, self- cussed in terms of the abstract expressions of reference, and theme and variation occured only formal language theory. Lerdahl and Jackend- infrequently. The concept of modulating between off concentrate on the hierarchical aspects of mu- keys was completely alien to Smurph. Musical ten- sic. They employ grouping analysis of musical sion and release were rare, and particularly glaring sections, phrases, and motives as well as metrical was the absence of definitive endings. analysis to break downhierarchies like two 3-note Rather than encoding music-theoretic groups that are part of a larger 6-note group. They heuristics into the system, Smurphwas directed to also use time-span reduction and prolongational improve its playing through other means. Smurph reduction to isolate pitch hierarchies. [Lerdahl & now stores the patterns of its compositions in a Jackendoff, 1983] separate database, and occasionally consults this There have been attempts to compose database to develop recurring musical phrases and music based on formal grammars, but one prob- create theme and variation. Smurph produces lem with this approach is that it can never be endings to its pieces by using the endings of the "adequate to mirror the total ambiguity which is pieces it has listened to. Starting from the last so important in music, of the boundary between note, usually the tonic or a fifth above it, Snmrph new figures and variations of an old one." [Lidov works backwards to write a bar or two of the fi- & Gabura, 1973] Smurph captures the essence of nal phrase which sometimes combines the tension the structure of music without using formal gram- and release of the traditional V7-I cadence with mars to output music. Since its patterns are avail- the patterns of the melody.

122 In giving Smurphthe capability of gener- are successful in other related domains. This ap- ating music that follows someof the rules of music proach may lead to plausible move generators for theory, we are actually giving it the potential to chess, organic synthesis, and automatic theorem produce creative music based on generalizations proving, each being an area in which humans have of these qualities. If Smurphcan learn the laws of shown superiority to computers. Finally, map- traditional music theory then it can break them. pings of pattern structures across multiple prob- Thus Smurph by judiciously extending the bound- lem domains will be explored, such as in music aries of Western music may eventually be truly composition based on chess patterns. creative. [Boden, 1990] Our presentation furnishes specific re- sults, including the creative output of the system, further details on its inner workings, and a critical 8 Prediction analysis of its strengths and weaknesses.

"Prediction is very difficult, espe- cially of the future." - Niels Bohr 10 Acknowledgements The authors are indepted to David Cope Like speech recognition systems and adap- and Kevin Karplus for valuable discussions and tive coding techniques, Smurphuses the context of suggestions. input it has seen so far to predict upcoming pat- References terns. Humans,whether they realize it or not, also predict upcoming sections of music as they listen [Apostel, Sabbe, and Vandamme,1986] L. Apostel, to it. In fact, the emotional aspect of music is H. Sabbe, and F. Vandamme,eds. Reason, Emo- mainly based on expectancy and surprise [Meyer, tion, and Music: Towardsa commonstructure for the arts, sciences, and philosophies, based on a 1956] [Narmour, 1990]. Furthermore, experimen- conceptual frameworkfor the description of music. tal evidence shows that even children can predict Ghent, Belgium, 1986. whena piece of music is finished [Serafine, 1988]. [Baird, Blevins, & Zahler, 1989] B. Baird, D. Blevins, and N. Zahler. The artificially intelligent computer performer on the MacintoshII and a pattern match- 9 Conclusions and Ongoing ing algorithm for real-time interactive performance. Research In Proceedings of the 1989 International Computer Music Conference. The MIT Press, Cambridge, 1992. Smurph creates original motives in the style of the input compositions. Whereas some [Balaban, Ebcioglu, and Laske, 1992] Mira Balaban, previous experiments in statistical analysis of KemalEbcioglu, and Otto Laske, eds. Understand- melodies [Olson, 1967] have only calculated zero ing Music with AI: Perspectives on Music Cogni- tion. The MITPress, Cambridge, 1992. through third-order analyses, Smurph produces [Bateman, 1980] Wayne Bateman. Introduction to higher-order analyses. Part of Smurph’s challenge Computer Music. John Wiley & Sons, New York, is using the right order of synthesis and combining 1980. multiple orders of probabilistic information, and [Bean, 1961] Calvert Bean, Jr. Information Theory herein ties part of its creativity and our challenge Applied to the Analysis o] a Particular FormalPro- as researchers. WhenSmurph sounds good enough cess in Tonal Music. Dissertation. University of in subjective listening test, it will be evaluated as Illinois, Urbana,1961. in a double-blind Turing test. Also, Smurph will [Bernstein, 1976] Leonard Bernstein. The Unan- itself be given a test to determine which of two swered Question: Six Talks at Harvard. Harvard input pieces is Mozart’s (or another composer’s), University Press, Cambridge,1976. after listening to selected pieces by that composer. [Boden, 1990] Margaret Boden. The Creative Mind: By creating a computer program which Myths and Mechanisms.Weidenfeld and Nicholson, learns a creative art through experience in another London, 1990. domain besides chess, the hypothesis that the ba- [Chomsky,1965] NoamChomsky. Aspects of the The- sis for creativity is domain-independent is being ory of Syntax. The MITPress, Cambridge, 1965. tested. Webelieve that problem solving and the [Chowning and Roads, 1985] John Chowning and arts mayutilize similar mechanismsto recognize, Curtis Roads. John Chowningon Composition. In alter, and combine patterns. Based on this the- Composers and the Computer. William Kaufmann, ory, we hope to build intelligent systems which Los Altos, 1985.

123 [Cooper & Meyer, 1960] Grosvenor Cooper and [Lerdahl & Jackendoff, 1982] Fred Lerdahl and Leonard B. Meyer. The Rhythmic Structure of Mu- Ray Jazkendoff. A Grammatical Parallel Between sic. The University of Chicago Press, Chicago, Music and Language. In Music, Mind, and Brain: 1960. The Neuropsychology of Music, edited by Manfred [Cope, 1988] David Cope. Music: The Universal Lan- Clynes, Plenum Press, New York, 1982. guage. In Proceedings of the American Association [Lerdahl & Jackendoff, 1983] Fred Lerdahl and Ray o] Artificial Intelligence Workshop on Music and Jackendoff. A Generative Theory of Tonal Music. AL AAAI, Menlo Park, 1988. The MIT Press, Cambridge, 1983. [Cope, 1991a] David Cope. Computers and Musical [Levinson, 1991] Robert Levinson. Experience-based Style. A-R Editions, Madison, 1991. Volume 6 of creativity. In Proceedings of Symposium on AI, the Computer Music and Digital Audio Series. Reasoning, and Creativittj, Kluwer Academic Press, [Cope, 1991b] David Cope. On Algorithmic Repre- 1991. To appear in the Kluwer book A1 and Cre- sentation of Musical Style. In Musical Intelligence, ativity. Also appears as UCSCTecli Report 91-37. Balaban, Ebcioglu, and Laske, eds. AAAIPress, [Lewis, 1991] J. P. Lewis. Creation by Refinement Menlo Park, 1991. and the Problem of Algorithmic Music Composi- [Dannenberg, 1984] R. B. Dannenberg. An on-line tion. In Music and Conn~ctionism, Todd & Loy, algorithm for real-time accompaniment. In Pro- eds., The MIT Press, Cambridge, 1991. ceedings o] the 1984 International Computer Music [Lidov & Gabura, 1973] David Lidov and A. James Conference. Gabura. A Melody-Writing Algorithm Using a For- [Deutsch, 1982] Diana Deutsch. Grouping Mecha- mal Language Model. In Computer Studies in the nisms in Music. In The Psychology of Music, Humanities and Verbal Behavior, 4 (1973): 138-148. Deutsch, editor. Academic Press, New York, 1982. [Mandelbrot, 1983] Benoit Mandelbrot. The Fractal [Ebcioglu, 1986] Kemal Ebcioglu. An Expert System Geometry of Nature. W.H. Freeman and Company, for the Harmonization of Chorales in the Style of J. New York, 1983. S. Bach. Dissertation, 1986. [Meyer, 1956] Leonard B. Meyer. Emotion and Mean- [Gilbert, 1993] Stephen Gilbert. Training an HMM ing in Music. The University of Chicago Press, to Generate Appropriate Melodies. Unpublished, Chicago, 1956. January 11, 1993. [Meyer, 1967] Leonard B. Meyer. Music, the Arts, [Gould & Levinson, 1992] Jeff Gould and Robert and ldeas: Patterns and Predictions in Twentieth- Levinson. Experience-based Adaptive Search. To Century Culture. The University of Chicago Press, appear in Machine Learning: A Multi-Strategy Ap- Chicago, 1967. proach, Morgan Kauffman, 1992, vol. 4. [Minsky & Laske, 1992] Marvin Minsky and Otto [Hiller & Isaacson, 1959] Lejaren Hiller and L. Isaac- Laske. A Conversation with Marvin Minsky. In son. Experimental Music. McGraw-Hill, 1959. A1 Magazine, Fall 1992, pp. 31-45. [I-Iofstadter, 1979] Douglas R. Hofstadter. Godei, Es- [Mozer, 1991] Michael Mozer. Connectionist Music chef, Bach: An Eternal Golden Braid. Random Composition Based on Melodic, Stylistic, and Psy- House, New York, 1979. cliophysical Constraints. In Music and Connection- [I’Ioltzman, 1981] S. Holtzman. Using Generative ism, Todd & Loy, eds., The MIT Press, Cambridge, Grammars for Music Composition. In Computer 1991. Music Journal, 5, 1 (Spring 1981): 51-64. [Narmour, 1990] Eugene Narmour. The Analysis [Kohonen, Lalne, Tiits, & Torkkola, 1991] Teuvo Ko- and Cognition of Basic Music Structures: The honen, Pauli Laine, Kalev Tiits, and Kari Torkkola. Implication-Realization Model. The University of A Nonheuristic Automatic Composing Method. In Chicago Press, Chicago, 1990. Music and Connectionism, Todd & Loy, eds., The MIT Press, Cambridge, 1991. [Olson, 1967] Harry F. Olson. Music, Physics, and Engineering. Dover Publications, New York, 1967. [Krumhansl, 1990] Carol Krumhansl. Cognitive Foundations o] Musical Pitch. Oxford University [Pickover, 1990] Clifford A. Pickover. Computers, Press, Oxford, 1990. Pattern, Chaos, and Beauty. St. Martin’s Press. New York, 1990. [Laske, 1973] Otto Laske. In Search of a Generative Grammar for Music. In Perspectives o] New Music [Roads, 1978] Curtis Roads. Composing Grammars. 12, 1 (Fall-Winter 1973): 351-378. Computer Music Association, San Francisco, 1978. [Laske, 1981] Otto Laske. Music and Mind: An Arti- [ScliiUinger, 1948] Joseph Schilfinger. The Mathemat- ficial Intelligence Perspective. San Francisco, Com- ical Basis of the Arts. Philosophical Library, New puter Music Association, 1981. York, 1948.

124 [Schottstaedt, 1984] Bill Schottstaedt. Automatic Species Counterpoint. Report No. STAN-M-19, Center for Computer Research in Music and Acous- tics, 1984. [Serafine, 1988] Mary Serafine. Music as Cognition: The Development of Thought in Sound. Columbia University Press, NewYork, 1988. [Smoliar, 1976] S. Smoliar. Music programs: An Ap- proach to Music Theory Through Computational Linguistics. Journal of Music Theory. Yale Univer- sity Press, New Haven, 1976. [Sutton, 1988] Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3(1):9-44, August 1988. [Todd, 1991] Peter M. Todd. A Connectionist Ap- proach to Algorithmic Composition. In Music and Connectionism, Todd & Loy, eds., The MIT Press, Cambridge, 1991. [Underwood, 1992] Rebecca C. Underwood. Chopin- NET: A sequential neural network for learning to reproduce melodies. Unpublished, June 5, 1992. [Vercoe, 1984] B. Vercoe. The Synthetic performer in the context of live performance. In Proceedings of the 1984 International Computer Music Conference. [Winograd, 1968] Terry Winograd. Linguistics and Computer Analysis of Tonal Harmony. In Journal of Music Theory 12 (Spring), 2-49. Yale University Press, New Haven, 1968. [Xenakis, 1971] Iannis Xenakis. Formalized Music. Indiana University Press, Bloomington, 1971.

125