SoundPaint { Painting Music Jurgen¨ Reuter Karlsruhe Germany http://www.ipd.uka.de/~reuter/ Abstract the composition was strongly bound to a very We present a paradigm for synthesizing electronic particular technical setup of electronic devices. music by graphical composing. The problem of map- Consequently, the composer easily became the ping colors to sounds is studied in detail from a only person capable of performing the compo- mathematical as well as a pragmatic point of view. sition, thereby often eliminating the traditional We show how to map colors to sounds in a user- distinction of production stages. At least, new definable, topology preserving manner. We demon- notational concepts were developed to alleviate strate the usefulness of our approach on our proto- the problem of notating electronic music. type implementation of a graphical composing tool. The introduction of MIDI in the early 80s was Keywords in some sense a step back to electro-acoustic, electronic music, sound collages, graphical compos- keyed instruments music, since MIDI is based ing, color-to-sound mapping on a chromatic scale and a simple note on/off paradigm. Basically, MIDI supports any instru- 1 Introduction ment that can produce a stream of note on/off Before the advent of electronic music, the west- events on a chromatic scale, like keyed instru- ern music production process was clearly di- ments, wind instruments, and others. Also, vided into three stages: Instrument craftsmen it supports many expressive features of non- designed musical instruments, thereby playing keyed instruments like vibrato, portamento or a key role in sound engineering. Composers breath control. Still, in practice, mostly key- provided music in notational form. Performers boards with their limited expressive capabilities realized the music by applying the notational are used for note entry. form on instruments. The diatonic or chromatic The idea of our work is to break these limi- scale served as commonly agreed interface be- tations in expressivity and tonality. With our tween all participants. The separation of the approach, the composer creates sound collages production process into smaller stages clearly by visually arranging graphical components to has the advantage of reducing the overall com- an image, closely following basic principles of plexity of music creation. Having a standard set graphical notation. While the graphical shapes of instruments also enhances efficiency of com- in the image determine the musical content of posing, since experience from previous compo- the sound collage, the sound itself is controlled sitions can be reused. by color. Since in our approach the mapping The introduction of electro-acoustic instru- from colors to actual sounds is user-definable ments widened the spectrum of available in- for each image, the sound engineering process is struments and sounds, but in principle did not independent from the musical content of the col- change the production process. With the intro- lage. Thus, we resurrect the traditional separa- duction of electronic music in the middle of the tion of sound engineering and composing. The 20th century however, the process changed fun- performance itself is done mechanically by com- damentally. Emphasis shifted from note-level putation, though. Still, the expressive power of composing and harmonics towards sound engi- graphics is straightly translated into musical ex- neering and creating sound collages. As a re- pression. sult, composers started becoming sound engi- The remainder of this paper is organized as neers, taking over the instrument crafts men's follows: Section 2 gives a short sketch of image- job. Often, a composition could not be no- to-audio transformation. To understand the tated with traditional notation, or, even worse, role of colors in a musical environment, Section 3 presents a short survey on the traditional use sive and thus became seldom. Mozart wrote a of color in music history. Next, we present and manuscript of his horn concert K495 with col- discuss in detail our approach of mapping col- ored note heads, serving as a joke to irritate the ors to sounds (Section 4). Then, we extend our hornist Leutgeb { a good friend of him(Wiese, mapping to aspects beyond pure sound creation 2002). In liturgical music, red color as con- (Section 5). A prototype implementation of our trasted to black color remained playing an ex- approach is presented in Section 6. We already traordinary role by marking sections performed gained first experience with our prototype, as by the priest as contrasted to those performed described in Section 7. Our generic approach is by the community or just as a means of read- open to multiple extensions and enhancements, ability (black notes on red staff lines). Slightly as discussed in Section 8. In Section 9, we com- more deliberate application of color in music pare our approach with recent work in related printings emerged in the 20th century with tech- fields and finally summarize the results of our nological advances in printing techniques: The work (Section 10). advent of electronic music stimulated the devel- opment of graphical notation (cp. e.g. Stock- 2 Graphical Notation Framework hausen's Studie II (Stockhausen, 1956) for the In order to respect the experience of tradition- first electronic music to be published(Simeone, Wehinger ally trained musicians, our approach tries to 2001)), and uses colors in an au- Ligeti stick to traditional notation as far as possi- ral score(Wehinger, 1970) for 's Artic- ble. This means, when interpreting an image ulation to differentiate between several classes as sound collage, the horizontal axis represents of sounds. For educational purposes, some au- time, running from the left edge of the image thors use colored note heads in introductory to the right, while the vertical axis denotes the courses into musical notation(Neuh¨auser et al., pitch (frequency) of sounds, with the highest 1974). There is even a method for training ab- pitch located at the top of the image. The verti- solute hearing based on colored notes(Taneda cal pitch ordinate is exponential with respect to and Taneda, 1993). Only very recently, the use the frequency, such that equidistant pitches re- of computer graphics in conjunction with elec- sult in equidistant musical intervals. Each pixel tronic music has led to efforts in formally map- row represents a (generally changing) sound of ping colors to sounds (for a more detailed dis- a particular frequency. Both axes can be scaled cussion, see the Related Work Section 9). by the user with a positive linear factor. The While Wehinger's aural score is one of the color of each pixel is used to select a sound. The very few notational examples of mapping col- problem of how to map colors to sounds is dis- ors to sounds, music researchers started much cussed later on. earlier to study relationships between musical and aural content. Especially with the upcom- 3 Color in Musical Notation History ing psychological research in the late 19th cen- tury, the synesthetic relationship between hear- The use of color in musical notation has a long ing and viewing was studied more extensively. tradition. We give a short historical survey in Wellek gives a comprehensive overview over order to show the manifold applications of color this field of research(Wellek, 1954), including and provide a sense for the effect of using colors. systems of mapping colors to keys and pitches. Color was perhaps first applied as a purely no- Painters started trying to embed musical struc- tational feature by Guido von Arezzo, who tures into their work (e.g. Klee's Fugue in invented colored staff lines in the 11th century, Red). Similarly, composers tried to paint im- using yellow and red colors for the do and fa ages, as in Mussorgsky's Pictures at an Exhi- lines, respectively. During the Ars Nova period bition. In Jazz music, synesthesis is represented (14th century), note heads were printed with by coinciding emotional mood from acoustic black and red color to indicate changes between and visual stimuli, known as the blue notes in binary and ternary meters(Apel, 1962). While blues music. in medieval manuscripts color had been widely applied in complex, colored ornaments, with the 4 Mapping Colors to Sounds new printing techniques rising up in the early 16th century (most notably Petrucci's Odhe- We now discuss how colors are mapped to caton in 1501), extensive use of colors in printed sounds in our approach. music was hardly feasible or just too expen- For the remainder of this discussion, we define a sound to be a 2π-periodic, continuous func- Unfortunately, there is no mapping between tion s : R 7! R; t ! s(t). This definition meets the function space of 2π-periodic functions and the real-world characteristic of oscillators as the R3 that fulfills all of the three constraints. most usual natural generators of sounds and the Pragmatically, we drop surjectivity in order fact that our ear is trained to recognize periodic to find a mapping that fulfills the other con- signals. Non-periodic natural sources of sounds straints. Indeed, dropping the surjectivity con- such as bells are out of scope of this discus- straint does not hurt too much, if we assume sion. We assume normalization of the periodic that the mapping is user-definable individually function to 2π periodicity in order to abstract for each painting and that a single painting does from a particular frequency. According to this not need to address all possible sounds: rather definition, the set