Proceedings of SMC Sound and Music Computing

Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

Visions of sound: The Centro di Sonologia Computazionale, from Computer Music to Sound and Music Computing

Sergio Canazza Giovanni De Poli Alvise Vidolin CSC-DEI, Univ. Padova CSC-DEI, Univ. Padova CSC-DEI, Univ. Padova [email protected] [email protected] [email protected]

ABSTRACT and hardware devices (filters, computers) conducted by Pa- duan researchers have since then produced state-of-the art Centro di Sonologia Computazionale (CSC) scientific re- results from both the technological and musical standpoints, search was the premise for subsequent activities of musi- and have generated collaborations with several renowned cal informatics, and is still one of the main activities of the contemporary composers. CSC engineering skills have been Centre. Today CSC activities rely on a composite group used to build electronic and digital instruments, augmented of people, which include the Center board of directors and reality systems, immersive video-games, and measuring personnel, guest researchers and musicians, and particu- instruments. It has also led to advances in such widely larly on master students attending the course “Sound and differing fields as sound design, musical cultural heritage Music Computing” at Dept. of Information Engineering preservation and promotion, and cognitive/physical reha- (DEI), which is historically tightly linked to the CSC. The bilitation. dissemination of scientific results as well as the relation- The CSC is carrying out a project for the preservation ship between art and science is hard and surely not trivial. and restoration of electrophone equipments and audio doc- With this aim, this paper describes an exhibition that illus- uments. An important (from both scientific and dissemi- trated the history of CSC, from the scientific, technolog- nation points of view) moment in this project was the re- ical and artistic points of view. This exhibition is one of alization of an exhibition by the University of Padova, in the first examples of “a museum” of Computer Music and collaboration with the SaMPL Lab of the Conservatory “C. SMC researches. Pollini”: Visions of sound. Electronic music at the Uni- versity of Padova, open from April 3 to July 18, 2012 at 1. INTRODUCTION the exhibition halls of the Botanical Garden. The exhibition showed the history of the computer music produced in Since the invention of musical instruments, art and tech- Padova and was assisted by various events, including a se- nology have stimulated and benefit one another. The crafts- ries of educational seminars held by CSC researchers and manship required to make a violin is a classic example, some concerts. The dissemination of the Computer Music but the invention of music-writing techniques was also an history is hard and complicated, because of its multi-faced achievement, often based on complex mathematics, which nature. It is necessary to emphasize the communication enabled musicians in the late Middle Ages to create intri- of all its different aspects, in particular it is important that cate combinations of sounds. general public understand also the genesis of the computer Over the centuries, Padova institutes, musicians and schol- music works. This paper presents our experience. ars have helped to revolutionize the science and art of sound. The exhibition illustrated the history of CSC, from scien- During the late 20th century, the Centro di Sonologia Com- tific, technological and artistic points of view. From the putazionale 1 (CSC, Center of Computational Sonology) first experiments by Teresa Rampazzi and by the group of Padova University (Italy) and the Electronic Music class Nuove Proposte Sonore (NPS) in the sixties, the close col- at the Conservatory “Cesare Pollini” in Padova gave birth laboration among the Conservatory, the CSC and the Com- to a unique scientific, technological, and artistic experi- puting Centre of the University, to the present, it was possi- ence, which stemmed from individual collaborations and ble to expose historic equipments, as the original magnetic multidisciplinary exchanges. tape recorder used by Teresa Rampazzi, a Synthi AKS, Between the 1970s and 1990s, CSC emerged as one of an ARP 2500 (now the last example in Italy), and the 4i the world leading centres for research into “Computer mu- System, devices which allowed the realization of the elec- sic”. The design and development of software programs tronic music of the last decades, art music as well as con- 1 CSC was founded by Giovanni Battista Debiasi (1928-2012): this sumer music. It was also possible to listen to some major paper is humbly and affectionately dedicated to his memory, a leading researcher and an outstanding teacher whose brightness and kindness we works realized at the CSC, e.g., Prometeo by Luigi Nono, will always remember. Perseo e Andromeda by Salvatore Sciarrino, and Medea by Adriano Guarnieri: for this latter musical work the original Copyright: c 2013 Sergio Canazza et al. This is an open-access article distributed multi-channel installation was recreated, for the first time under the terms of the Creative Commons Attribution 3.0 Unported License, which after its premiere` in 2002 at the Teatro La Fenice in Venice. permits unrestricted use, distribution, and reproduction in any medium, provided The exhibition was enriched by numerous interactive in- the original author and source are credited. stallations, specially designed and realized by researchers

639 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

development of computer music in the world (the musical works realized in CSC are listed at http://csc.dei. unipd.it/musical_productions.html). At the same time, with its own set of electronic equipment (filters, digital signal processors, computers) specially designed and programmed by researchers at the Department of Informa- tion Engineering, it is a striking witness to the technological era and its evolution in recent decades (Sec. 4). CSC, today directed by Giovanni De Poli, was founded by Giovanni Battista Debiasi (fig. 1). In 1957 Giovanni Battista Debiasi, at the University of Padova, proposed an original work about an electronic organ based on photodi- odes. This was the first step of a multidisciplinary future for electric/electronic engineering and music in Padova. In the early seventies Debiasi carried out research on speech analysis and synthesis, in collaboration with Gian Antonio Figure 1. Giovanni Battista Debiasi, Vicenza (Italy), 4th Mian and Carlo Offelli [2, 3]. In the eighties and nineties, June 1928 – Padova (Italy), 24th June 2012. in advance to the international scientific community, De- biasi studied issues related to the preservation and restoration of cultural musical heritage. He trained hundreds of students: his research fields are now everywhere, in Italy and in the world, and this gives the sign of the importance that he played in the birth and development of Sound and Music Computing (see, at least, [1, 4, 5]).

Fig. 2 shows the system panels for recording, sound synthesis and processing in 1979: this system was also used in the Summer Schools organized in CSC and that were considered as world references in the field of computer music. Among the various hardware systems of CSC, particularly important from the history and the musicology points of Figure 2. System panels for recording, sound synthesis view, was the project – granted by the Laboratory for Com- and processing in 1979. puter Music at the La Biennale (LIMB) in Venice, in collaboration with IRCAM in Paris – that led to the realization of the 4i System (fig. 3 and Sec. 7). at CSC – which today is with the Department of Informa- CSC has been mainly a centre of promotion and cultural tion Engineering (DEI) – to introduce visitors to the world diffusion of music informatics since its foundation. Thanks of sound, applications of technology, the result of the novel to close collaboration among experts of various disciplines, research in the Sound and Music Computing field, in par- it has been possible to create an interdisciplinary group, ticular immersive reality systems, preservation and restora- which has become an international reference in the field, tion of musical cultural heritage, information systems for and has come to be part of contemporary music history. enhanced learning and for rehabilitation of disabled peo- Activities of CSC can be grouped into four main areas: ple. scientific research, music research, production and perfor- This exhibit aims at showing the deep connections be- mance of music works, teaching and dissemination. tween academic research and the multifaceted world of The rapid evolution known by computers and microelec- sound, and their influences on both art music and popular tronic devices in the second half of the last century has music, especially from the Seventies. led to the development of several sound synthesis methods This exhibition was important (i) as a moment of cultural (Sec. 5) and to reduction the processing times, allowing reflection, because it has led to the comparison of the dif- to recover the performer-instrument relationship and then ferent research areas that have occurred since the sixties to reintroducing the causality between gesture and sound typ- the present, and (ii) as a stimulus to overcome the prob- ical of the musician with his/her instrument. This evolu- lems related to the preservation and restoration of cultural tion permitted to integrate the electronic medium with tra- heritage music the CSC. ditional instruments, mixing freely the sound of mechan- In Sec. 2 the history of the Centre is concisely summa- ical devices with sound processing generated during the rized. Then all the sections of the exhibition are detailed, as performance: arising the live electronics performer, which an example of dissemination to a general public, reported allowed to recover the absence of the performer typical of here for consideration by the SMC community. electroacoustic music (Sec. 7), when the public was con- fused in front of stages with only loudspeakers. The com- 2. CENTRO DI SONOLOGIA COMPUTAZIONALE The CSC was born in 1979 [1], but it was already active since the late sixties as a point of reference for the birth and

640 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

science of harmony) by Giuseppe Tartini, 1754. In this treatise, published in 1754, the violinist and composer Giuseppe Tartini accounted for his research on the phenomenon of the third sound. He included el- ements of physics, arithmetic, and geometry, organized into a complex theory which sparkled a lively discussion; • original ancient violins and bows.

4. WELL-CALCULATED MUSIC: THE 20TH CENTURY Figure 3. 4i System developed by Giuseppe Di Giugno. When the first instruments able to generate “new” sounds appeared in the 1950s’, composers and musicians welcomed enthusiastically this revolution. Electronic music was born, puter allows to control individual processes (synthesis and i.e. music realized with either analogue electronic (1950s’ sound processing) to a more abstract level than that reached and 1960s’) or digital (since the 1970s’) devices. In the by the electrophone equipments of the sixties (generally most important international research centre, technology based on voltage control). The use of systems with mul- was used to create new sounds, or to explore and process tiple speakers, thanks to which the sounds came from dif- sounds recorded and produced with this equipment. The ferent directions (front, back, side, top, bottom) made ob- new music had no performers, and the loudspeaker – the solete the traditional concert halls and the placement of main mean to deliver sound to listeners – became the new chairs lined up in the theater. Even at this stage the CSC “star” of concert halls. Musical structures became more played a pioneering role (see Sections 7, 8, and 9), becom- free, while the need for accurate control of durations and ing a leader in the opera Prometeo by Luigi Nono (Venice for adequate notation posed new problems. La Biennale, 1984; Teatro alla Scala, 1985) and the work At the CSC, Teresa Rampazzi 2 was an electronic-music Perseo e Andromeda by Salvatore Sciarrino (Staatstheater pioneer. She and Ennio Chiggio set up the NPS Group Stuttgart, 1991; Teatro alla Scala, 1992). Nuove Proposte Sonore (New Sound Proposals) in Padova Now the CSC is carrying out researches in all the areas of in 1965. Chiggio was part of Gruppo Enne, a group which SMC field (Sec. 6). applied kinetics to visual art. The NPS Group conducted research into the timbre and density of “sound events”, 3. WELL-CALCULATED MUSIC: PREMISES creating “sound objects” (or “sounding objects” according to Rampazzi’s own terminology) and more or less com- The first section of the exhibition introduces the roots of plex tracks which explored acoustic phenomena. In 1972, music in Padova. One of the major breakthroughs in the Rampazzi donated her equipment to the Conservatory of 14th century Italy was the development of written music Padova, which was one of the very first italian Conservato- and musical symbols. Music had traditionally been handed ries where electronic music classes were started – follow- down orally, but musicians and composers had come to ing Firenze. realize that complex musical constructs had to be writ- In this section of the exhibition the following equipments ten down and that symbols were needed to set the time- were exhibited: values between different sounds. The composer and the- orist Marchetto da Padova pioneered these developments • the original ARP 2500 (see fig. 4). It is an early and his arithmetic- and geometry-based studies paved the 1970s analogue synthesizer: it was one of the most way for musical notation, the forerunner of modern mu- versatile and powerful professional synthesisers of sic scores. Mathematical studies are also at the core of its time. The synthesizer came with a wide range Giuseppe Tartini’s theories (1692-1770). He was “first vi- of compatible modules which could be connected to olin and head of concerts” at St. Antonio’s Basilica in generate and manipulate sound; Padova. He discovered a “terzo suono” (literally a “third sound”), which he heard when two different notes were • EMS Synthi AKS portable analogue synthesizer man- played together on a violin. His work was devoted to link- ufactured by Electronic Music Studios in London ing the physics to a musical and metaphysical theory. Tar- (fig. 5). Its built-in pin matrix, sequencer and key- tini is known for his art of bowing, as he used a specially board pack the power of an electronic-music labora- design bow to create virtuoso effects. tory into a portable briefcase; In this section of the exhibition the following items were • TEAC A 3340 S (see fig. 4). Four-track tape recorder exhibited: introduced in the mid-1970s. It played tracks through • original manuscripts (unique source worldwide) of four loudspeakers and paved the way for modern the first half of 15th century; “surround sound”; • the original Trattato di musica secondo la vera scienza 2 The title of this section of the exhibition was an usual question by Teresa Rampazzi to her students: “but do you have it [your music] well dell’armonia (Treatise on music according to the true calculated?”

641 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

velopment of a formal musical notation language for computer [7]. The CSC is among the pioneers of the most innovative and interesting methods of synthesis, based on sound source (e.g., a musical instrument) modeling, instead of signal modeling [8]. This synthesis uses algorithms that produce the sound as a side effect of a process of simulation of physical phenomena, i.e., reproducing what occurs in nature. The bow-string interaction in the physical reality, studied by Tartini in his treatise (Sec. 3), in this way becomes a mathematical model.

Figure 4. The original ARP 2500, the stop-watch and the The results of the research conducted by computer music TEAC reel to reel tape recorder (4-tracks) used by Teresa brings a terrific deepening of knowledge within the acous- Rampazzi. tic and psychoacoustic. It is with these studies that the foundations are laid for the development of auditory communication in multimedia and multimodal environments (virtual and augmented reality). In this section multimedia installations were exhibited, in which the visitors could interact with different sound synthesis techniques; a digital juke-box with some of the most important musical works realized in CSC, restored on purpose by the authors; a printout MARCR J578 A, the publication by Enore Zaf- firi Musica per un anno (Music for a year), DUCHAMP Center; a folder NPS (Nuove Proposte Sonore), with various enclosed documents; a copy of the magazine Oggetti Sonori (Sound Objects), or Oggetto Sonoro (Sound Ob- ject); two original video works by Ennio Chiggio: small Figure 5. The Synthi AKS used by Teresa Rampazzi. television in plexiglass display cabinet, with video board, and Dischi a rotazione apparente (Discs with apparent ro- • Junghans stop-watch – used by Teresa Rampazzi for tation) – Marcel Rotour (1967, Photographic tape, plex- the realization of her well-calculated music (see fig. iglass and wooden frame, 50x50x20 cm). These differ- 4); ent items helps general public to contextualize the musical works, showing the relationship among music and others • Teletype, electromechanical device used to transmit arts. text messages and employed by early computers at CSC for data input/output purposes; 6. SOUND AND SOCIETY • Digital-to-analogue and analogue-to-digital convert- At the end of the Nineties, the international computer mu- ers at 12 and 16 bits, originally connected to the IBM sic research domain evolved into Sound and Music Com- System/7, with programmable clock filter, low-pass puting (SMC), which also includes non-musical areas re- filters at 4.5 kHz, 7 kHz, and 14 kHz. lated to research on sound. The results are manifold. These items, following the musical instruments history (Sec. Researchers in CSC developed multimodal interactive sys- 3), allow the general public to understand the genesis of the tems for teaching with special interfaces, specifically de- Computer Music. signed to enhance the learning of students with disabilities. The research on the preservation and restoration of audio documents (see [9] for a review) are combined with tech- 5. NUMBER AND SOUND nology’s innovations in information retrieval to meet the In the Seventies, composers discovered the potential of in- needs of today society where everything has to be stored, formation technology and adopted computers and electron- browsable, and available “anybody, anytime and everywhere”. ics devices: the born of Computer Music. in the interna- This implies the definition of new strategies for data stor- tional field sound synthesis had an extraordinary impact age and study of new techniques of content search (e.g., on music writing, allowing composers to better understand by humming) in data mining, as well as listening strategies the way in which sounds are formed and their aural effect, appropriate to each situation (the living room, the concert transforming sometimes even the orchestral writing [6]. hall, the walkman/iPod headphones). Innovative 3D au- Sec. 3 showed how Padova in the 14th century has been dio techniques [10] allow to virtually recreate an environ- a research laboratory in musical writing (in particular with ment in which various sound sources are located at dif- Marchetto da Padova). In line with this, it is interesting to ferent moving points in space, with important applications note that in the Seventies the CSC contributed to the de- in virtual reality systems, from immersive video games

642 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

CAM in Paris and LIMB of the Venice La Biennale jointly developed the 4i System, a digital signal processors based system for live electronics. This system was used in some of the most important musical works of the second half of the Twentieth century, including Prometeo, la tragedia dell’ascolto (1984-85) by Luigi Nono, based on the movement of sound in space. The fig. 6 shows the arrangement of the choir and orchestra of the Prometeo in a model (displayed in the exhibition) of the original building designed by Renzo Piano for the representation at the Venice La Bi- ennale in 1984. In this section the following items were exhibited: • 4i System (see fig. 3). It is realized by means of a 128-kbyte memory PDP11 computer with a 4i digi- Figure 6. The model – as shown in the exhibition – of tal sound processor (designed by Giuseppe Di Giugno), the original building designed by Renzo Piano in 1984 for a 16-bit digital-to-analogue converter and a control Prometeo by Nono. interface for performance parameters; • an interactive system (developed on purpose) in which the user can control the live electronics software of the Prometeo and contemporaneously observe the original gesture of the live electronics performer; • original scores with notes handwritten by Luigi Nono; • heliography of the Prologue of Prometeo (in the 1984 version), with several original corrections and anno- tations, probably added during the early rehearsals in Venice; • a multimedia installation for the interactive listening of the Perseo e Andromeda (1990) by Salvatore Figure 7. An interactive installation dedicated to explain Sciarrino, in which the synthesized sounds replace the results of the 3D audio research domain. The visitor the traditional orchestra. The visitors can listen the could appreciate the change of the sound spatialization de- entire work, some parts and/or the single sound ob- pending on of his head movement. ject, observing the related score. to flight simulators (see an example in fig. 7). Micro- 8. MEDEA BY ADRIANO GUARNIERI (2002) phones arrays systems with variable geometry are specifically designed for both monitoring of urban environments In the exhibition two large and innovative musical works for homeland security and as musicians tracking system for were showed: the musical theatre opera Medea (2002) by live electronics [11]. Adriano Guarnieri and the interactive multimedia installation Casetta delle immagini by Carlo De Pirro. Medea is a video-opera in three part loosely based on Eu- ripide’s tragedy, for video sequences, soloists, chorus, or- 7. MUSIC AND SPACE chestra and live electronics, in which the sound direction The initial absence of typical performers in the electronic becomes almost visual and the spatial sound seems to al- music repertoire is overcome with the development of com- ternate close-ups and overviews. It was showed in this ex- puters able to generate electronic sounds in live contest and hibition (see fig. 8) by means of the original stage sound- to process the sound signal (voices or musical instruments) design, using the eight-channel audio recording made dur- in real time. Live electronics was born [12], which now is ing the first performance at the PalaFenice in Venice. used in a large music repertoire all over the world and with The mythical story of Medea, represented by three female it also grows new professional figures with a double train- voices, merges with the play of the dynamics of sound in ing: musical and scientific. space. The sound produced by the singers and by the or- The CSC also developed new interfaces to play these in- chestra is detected by 68 microphones, processed by live struments, necessary to control the musical timbre and the electronics software and finally diffused by dozens of speak- virtual space and polarizing the interest of many composers. ers distributed among the public. The sound movement in The traditional keyboard organ is not suited to control mul- the room, besides, is controlled in various ways (e.g.,musicians’ tiple parameters simultaneously, synthesis algorithms, and gestures) and reinterpreted in real time by live electronics sound spatialization. In the Eighties, CSC in Padua, IR- software. This work is one of the greatest artistic studies

643 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

In this section the original Casetta delle immagini was exhibited, restored on purpose by the authors.

10. CONCLUSIONS CSC scientific research was the premise for the other activities of musical informatics, and it is the main focus of the Centre. Today the CSC still supports production of musical works, thanks to significant investments in research that begun in 1979 when the Centre was officially founded. In the early days the research was mainly focused on sound synthesis. Nowadays, the Centre is work- ing, in synergy with the SaMPL Lab of the Conservatory of Padova, on preservation and restoration of audio doc- Figure 8. The stage of Medea – as represented in the exhi- uments, new sound synthesis techniques, analytical tools, bition – using the original eight-channel audio recording as techniques of sound spatialization, complex dynamic sys- well as three video output. Guarnieri himself, visiting the tems and analysis and morphing of expressive content in exhibition, recognized this installation as “this is my real music performances. Today CSC activities rely on a com- Medea”. posite group of people, which include the Center board of directors and personnel, guest researchers and musicians, and particularly on master students attending the courses in expressive gesture and sound interaction, a domain born “Sound and Music Computing” at Dep. of Information En- in the late 90s, which bring interesting results even in the gineering of the University of Padova. analysis of musical performance. The CSC is carrying out a project for the preservation and This section of the exhibition showed also some of the restoration of electrophone equipments and audio docu- better models developed at CSC related to most recent re- ments. The principal output of this project is the realiza- search studying the possible connections between two uni- tion of an interactive “museum” of Computer Music and of verses that may seem antithetical, the emotions and the ma- researches in SMC field. The first attempt was the exhibi- chines, deepening the procedures that enable the comput- tion Visions of sound. Electronic music at the University of ers to communicate and simulate expressive components, Padua. In the authors’ opinion, it is time to start a debate emotions, intentions and affects [13]. on how the scientific SMC community wants to preserve its history and what kind of access tools we are able to 9. CASETTA DELLE IMMAGINI develop, in order to communicate the (scientific and ap- BY CARLO DE PIRRO (2002) plicative) potential of its researches to the general public and (no less important) to potential investors [15]. This interactive multimedia installation was designed by the composer Carlo De Pirro 3 at the CSC for Piazza Pinoc- Acknowledgments chio, the Italian space at the Expo 2002 in Neuchatelˆ (Switzer- land), Section Artificial Intelligence and Robotics. The In the realization of the exhibition Visions of sound the ef- work uses the results of research in the fields of analy- forts of the CSC researchers, of the entire University of sis, modeling and communication of expressive content Padova, of the Conservatory “C. Pollini”, of Padova and of and emotional non-verbal interaction, by means of multi- the Veneto Region were terrific. The authors deeply thank sensorial interfaces in mixed reality environments. The in particular the scientific and the organization boards of Casetta delle immagini (Little house of the appearances) is the exhibition, the Luigi Nono’s Archive (in particular the a sort of magic room for children, where every gesture be- president Nuria Schoenberg-Nono), the Rectorate and the comes sound, images and colors. The visitors’ movements Museums Center of the University of Padova, the direc- were captured by cameras and analyzed by specially devel- tor of the Conservatory Maria Nevilla Massaro and Nicola oped software able to process a virtual gesture model and Bernardini (professor of Electronic Music), and Ivo Rossi thus generate projection images and rhythmic sequences of (vice-Mayor of Padova). music. A similar idea is now implemented in Stanza Logo-motoria 11. REFERENCES [14], a systems for educational purposes used in many Ital- ian schools, that exploits a multimodal interactive environ- [1] G. B. Debiasi, G. De Poli, G. Tisato, and A. Vidolin, ment aimed at learning through the movement and can be “Centro di Sonologia Computazionale C. S. C. Uni- used in situations of learning difficulties or for children versity of Padova,” in Proc. of International Computer with multi-disabilities. Music Conference, 1984, pp. 287–294. 3 Adria, 1956 – Padova, 2008. Carlo, professor of Music Composi- [2] G. Debiasi, G. De Poli, G. A. Mian, G. Mildonian, and tion at Rovigo Conservatoire, collaborated with the CSC for more than fifteen years: his musical compositions were (and are) a great stimulus C. Offelli, “Italian speech sinthesis from unrestricted for the researches carried out in CSC, thanks to his innovative and artistic text for an automatic answerback system,” in Proc. of approach. 8th Inf. Congress of Acoustics, London, 1974, p. 296.

644 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

[3] G. A. Mian, F. Morgantini, and C. Offelli, “An applica- related transfer function features,” IEEE Trans. Audio, tion of the linear prediction technique to efficient cod- Speech, and Language Process., vol. 21, no. 3, pp. ing of speech segments,” in Proc. of 1976 IEEE Int. 508–519, March 2013. Conf. Acoustic, Speech, Signal Processing, Philadel- phia, April 12-14, 1976, p. 722. [11] D. Salvati and S. Canazza, “Adaptive time delay estimation using filter length constraints for source lo- [4] G. Francini, G. Debiasi, and R. Spinabelli, “Study of calization in reverberant acoustic environments,” IEEE a system of minimal speech reproducing units for ital- Signal Processing Letters, vol. 20, no. 5, pp. 507–510, ian,” JASA, vol. 43, pp. 1282–1286, 1968. 2013. [5] G. B. Debiasi and M. Rubazzer, “Architecture for a [12] A. Vidolin, “Musical interpretation and signal process- digital sound synthesys processor,” in Proc. of ICMC, ing,” in Musical Signal Processing, C. Roads, S. T. 1982, pp. 225–231. Pope, A. Piccialli, and G. De Poli, Eds. Lisse: Swets and Zeitlinger, 1997, pp. 439–459. [6] G. De Poli, “A tutorial on digital sound synthesis techniques,” Computer Music Journal, vol. 7, no. 4, pp. [13] S. Canazza, G. De Poli, A. Roda,` and A. Vidolin, 8–26, 1991. “Expressiveness in music performance: Analysis, [7] G. B. Debiasi and G. De Poli, “MUSICA, A Language models, mapping, encoding,” in Structuring Music for the Transcription of Musical texts for Computers,” through Markup Language: Designs and Architec- Interface, vol. 11, no. 1, pp. 1–27, 1982. tures, J. Steyn, Ed. IGI Global, 2012, pp. 156–186. [8] G. Borin, G. De Poli, and A. Sarti, “Algorithms and [14] S. Zanolla, A. Roda,` F. Romano, F. Scattolin, structures for synthesis using physical models,” Com- S. Canazza, and G. L. Foresti, “When sound teaches,” puter Music Journal, vol. 16, no. 4, pp. 30–42, 1992. in Proc. of Sound and Music Computing Conference, 2011, pp. 64–69. [9] S. Canazza, “The digital curation of ethnic music audio archives: from preservation to restoration,” Inter- [15] N. Bernardini and G. De Poli, “The future of sound national Journal of Digital Libraries, vol. 12, no. 2-3, and music computing,” Journal of New Music Research pp. 121–135, 2012. (special issue), vol. 36, no. 3, pp. 139–239, 2007. [10] S. Spagnol, M. Geronazzo, and F. Avanzini, “On the relation between pinna reflection patterns and head-

645 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

SMOOTHNESS UNDER PARAMETER CHANGES: DERIVATIVES AND TOTAL VARIATION

Risto Holopainen

ABSTRACT far from smooth responses to changes in physical control variables (e.g. overblowing in wind instruments). Apart from the sounds they make, synthesis models are The smoothness of transitions has been proposed as a cri- distinguished by how the sound is controlled by synthesis terion for evaluating sound morphings [4]. As the mor- parameters. Smoothness under parameter changes is often phing parameter is varied between its extremes, one would a desirable aspect of a synthesis model. The concept of expect the perceived sound to pass through all intermediate smoothness can be made more accurate by regarding the stages as well. However, because of categorical perception synthesis model as a function that maps points in parameter some transitions may not be experienced as gradual. It may space to points in a perceptual feature space. We introduce be impossible to create a convincing morph between, say, new conceptual tools for analyzing the smoothness related a banjo tone and a sustained trombone tone. to the derivative and total variation of a function and apply Quantitative descriptions of the smoothness of a synthesis them to FM synthesis and an ordinary differential equation. parameter should use a measure of the amount of change in The proposed methods can be used to find well behaved the sound, which can be regarded as a distance in a percep- regions in parameter space. tual space. Similarity ratings of pairs of tones have been used in research on timbre perception, where multidimen- 1. INTRODUCTION sional scaling is then used to find a small number of dimensions that account for the perceived distances between Some synthesis parameters are like switches that can as- stimuli [5]. In several studies, two to four timbral dimen- sume only a discrete set of values, other parameters are like sions have been found and related to various acoustic cor- knobs that can be seamlessly adjusted within some range. relates, often including the attack time, spectral centroid, Only the latter kind of parameter will be discussed here. spectral flux and spectral irregularity [6]. The importance Usually, a small change in some parameter would be ex- of spectrotemporal patterns was stressed in a more recent pected to yield a small change in the sound. As far as this study [7] where five perceptual dimensions were found. is the case, the synthesis model may be said to have well Most timbre studies have focused on pitched, harmonic behaved parameters. sounds, in effect neglecting a large part of the possible A set of criteria for the evaluation of synthesis models range of sounds that can be synthesized. At the other ex- were suggested by Jaffe [1]. Three of the criteria seem rel- treme, the problem of similarity between pieces of music evant in this context: 1) How intuitive are the parameters? has been addressed in music information retrieval [8]. The 2) How perceptible are parameter changes? 3) How well difficulty in comparing two pieces of music is that they behaved are the parameters? The vague notion of smooth- may differ in so many ways, including tempo, instrumen- ness under parameter changes (which is not the name of tation, melodic features and so on. Most synthesis models one of Jaffe’s criteria) can be made more precise by the of interest to musicians are also able to vary along several approach taken in this paper. dimensions of sound, e.g., pitch, loudness, modulation rate From a user’s perspective, the mapping from controllers and many timbral aspects. A thorough study of the per- to synthesis parameters is important [2]. In synthesis mod- ceived changes of sound would include listening tests for els with reasonably well behaved parameters, there are good each synthesis model under investigation. A more tractable prospects of designing mappings that turn the synthesis solution is to use signal descriptors as a proxy for such model and its user interface into a versatile instrument. tests. However, a synthesis model does not necessarily have to There are numerous signal descriptors to choose from [9], have well behaved parameters to be musically useful. De- but the descriptors should respond to parameter changes in spite the counter-intuitive parameter dependencies in com- a given synthesis model. For example, in a study of the plicated nonlinear feedback systems, some musicians are timbre perception of a physical model of the clarinet, the using them [3]. Likewise, acoustic instruments may have attack time, spectral centroid and the ratio of odd to even harmonics were found to be the salient parameters [10]. Copyright: c 2013 Risto Holopainen et al. This is Since a synthesis model may be well behaved with re- an open-access article distributed under the terms of the spect to certain perceptual dimensions but not to others, Creative Commons Attribution 3.0 Unported License, which permits unre- the smoothness may be assessed individually for each of a stricted use, distribution, and reproduction in any medium, provided the original set of complementary signal descriptors. author and source are credited. A synthesis model will be thought of as a function that

646 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden maps a set of parameter values to a one-sided sequence of space, and another distance metric is needed for points in 0 real numbers, representing the audio samples. It will be the space of sample sequences. Let dp(c, c ) be a metric 0 assumed that all synthesis parameters are set at the begin- in parameter space, and let ds(X(c),X(c )) be a metric in ning of a note event and remain fixed during the note. Dy- the sequence space. The derivative can then be defined as namically varying parameters can be modelled by an LFO the limit or envelope generator, but for simplicity we will consider d (X(c),X(c + δ)) only synthesis parameters that remain constant over time. lim s (1) The effects of parameter changes may be studied either kδk→0 dp(c, c + δ) locally near a specific point in parameter space, or glob- where δ ∈ Rp is some small displacement in parameter ally as a parameter varies throughout some range. The lo- space. The limit, if it exists, is the derivative evaluated at cal perspective leads to a notion of the derivative of a syn- the point c. thesis model, which is developed in section 2. Parameter In general, synthesis parameters do not make up a uni- changes over a range of values are better described by the form space. Different parameters play different roles; they total variation, which is introduced in section 3. Then, sec- affect the sound subtly or dramatically and may interact so tions 4 and 5 are devoted to case studies of the smoothness that the effect of one parameter depends on the settings of of FM synthesis and the Rossler¨ attractor. Some applica- other parameters. This makes it hard to suggest a general tions and limitations of the methods are discussed in the distance metric that would be suitable for any synthesis conclusion. model. Our solution will be to consider the effects of varying a single synthesis parameter cj at a time, so the distance 0 0 2. SMOOTHNESS BY DERIVATIVE dp(c, c ) in (1) reduces to cj − cj . Furthermore, consider a scalar valued signal descriptor φ(i)(c) ≡ φ(i)(X(c)) In order to formalize the notion of smoothness, we will for- which itself is a signal that depends on the sample se- mulate a synthesis model explicitly as a function and de- quence and the parameter value. Thus, we arrive at a kind scribe what it means for that function to be smooth. First, of partial derivative evaluated with respect to the parameter we define a suitable version of the derivative. Then, in Sec- (i) cj using a signal descriptor φ , tions 2.2 and 2.3, the practicalities of an implementation are discussed. ∂φ(i) ◦ G(c) d (φ(i)(c), φ(i)(c + he )) = lim s j (2) 2.1 Definition of the derivative ∂cj h→0 h

p Consider a synthesis model as a function G : R → RN where ej is the jth unit vector in the parameter space. that maps parameters c ∈ Rp to a one-sided sequence of Clearly the magnitude of this derivative depends on the samples xn, n = 0, 1, 2,..., where the sample sequence specifics of the signal descriptors used and which synthesis will be notated X(c) to indicate its dependence on the pa- parameters are considered. In a finite dimensional space, rameters. Then the question of smoothness under param- all partial derivatives should exist and be continuous for eter changes is related to the degree of change in the se- the derivative to exist. Such a strict concept of derivative quence X(c) as the point c in parameter space varies. In does not make sense in the present context where any num- practice, the distance in the output of the synthesis model ber of different signal descriptors can be employed, so only will be measured through a signal descriptor rather than the partial derivatives (2) will be considered. from the raw output signal. If a distance were to be cal- Before discussing the implementation, let us recall some culated from the signals themselves, two periodic signals intuitive conceptions of the derivative. As William Thurston with identical amplitude and frequency but different phase has pointed out [11], mathematicians understand the deriva- might end up being widely separated according to the met- tive in multiple ways, including the following. ric, despite sounding indistinguishable to the human ear. • The derivative is the slope of a line tangent to the Signal descriptors that are clearly affected by the synthesis graph, if it has a tangent. parameters and that can be interpreted in perceptual terms are preferable. d n n−1 • In terms of symbolic operations, dx x = nx . In order to treat the synthesis model as a function, it will be assumed to be deterministic in the sense that the same • The derivative is the best linear approximation to the point in parameter space always yields identical sample se- function near a point. quences. The idea of relating how much a function f(x) • It is the limit of what you get by looking at a function changes as the independent variable x changes by a small under a microscope of higher and higher power. amount leads to the concept of derivative. Functions that have derivatives of all orders are called smooth. A more Synthesis models are typically very complicated if con- refined concept is to say that a function is k times considered as mathematical functions; hence the analytic ap- tinuously differentiable; the larger k is, the smoother the proach to differentiation is out of the question and one has function. to rely upon numerical approximations. The various intu- Now, we would like to apply some suitably defined deriva- itions of what the derivative is may guide a practical nu- tive to synthesis models considered as functions. To this merical implementation in different directions, as will be end, a distance metric is needed for points in the parameter further discussed in Section 2.3.

647 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

Numerical estimation of the derivative is highly sensitive Although the synthesis model is assumed to be deter- to measurement noise. Here one source of measurement ministic, all signal descriptors will introduce measurement noise are the signal descriptors. Whereas one would like noise. If a number of windowed segments of the signal are to magnify a curve in order to find its derivative at a point, analyzed, then the spectrum of these segments will fluctu- doing so will also reveal more fine details caused by the ate unless some integer number of periods fit exactly into noise, which may lead to false estimates. When properly the window. The fluctuation can be reduced by using the estimated, the derivative will exaggerate irregularities and time-averaged version of the distance metric (4). make them easier to detect. Several methods for the estimation of derivatives exist [12]. Theoretically, it may be possible to arrive at ana- 2.2 Pointwise or time-average distance? lytical expressions for the derivative of a synthesis model considered as a function, at least in some trivial cases. In The distance metric d in sequence space has so far been s practice, numerical estimates have to be used. A simple left unspecified. We propose two alternatives, each suit- approach would be to evaluate (2) directly at two points able in different situations. The signal descriptors that will c and c0. Another approach is to fit a polynomial to the be used are based on short-time Fourier transforms of the curve φ(c), and then do a symbolic differentiation of the signal X(c) at regular intervals, using a hop size equal to polynomial. the FFT window length, L. Hence, the signal descriptor The method of estimation of derivatives that will be used is a sequence which we write concisely as φ (c), where m here is similar to one described in ref. [12, p. 231] but m = bn/Lc is a time index. slightly simpler. The derivative at a point c is approx- Using a pointwise distance metric, one may follow the 0 imated by a sequence of symmetric differences with de- two signals over time and take the sum over their distances creasing distance h. A linear regression of this sequence |φ (c) − φ (c0)| at each moment. Since these are infi- m m gives the derivative as the intercept. Suppose a sequence nite sequences, the sum may not converge. Therefore, an of slopes exponentially decaying weighting function is applied in the distance metric φ(c0 + hi) − φ(c0 − hi) yi(c0; hi) = (6) 2hi " ∞ #1/2 are given. Then the limit as h → 0 can be found as the 0 X m 0 2 ds(X(c),X(c )) = γ (φm(c) − φm(c )) y-intercept of the fitted line m=0 (3) y = d + bh + η , (7) where γ ∈ (0, 1) controls the decay rate. Convergence is i i i then guaranteed if the signal descriptors φm are bounded. which gives the estimated derivative d. This method also The second approach involves first taking an average over provides a hint about the badness of fit, for which the root the sequence φm(c), m = 0, 1,...,M and then compar- mean square error (RMSE) of the residuals η can be used. ing averages of two sequences. Thus, the distance becomes

0 0 3. TOTAL VARIATION ds(X(c),X(c )) = |hφ(c)i − hφ(c )i| (4) Whereas the derivative is concerned with local behaviour where we take time averages of a function, an even more useful perspective on the smooth-

M−1 ness of a synthesis model may be to look at its properties 1 X over intervals of a parameter. One possible way to do so is hφ(c)i = lim φm(c) (5) M→∞ M m=0 to measure the length of the curve that a signal descriptor traces out as the parameter traverses some interval. If this before computing the distance. For time-varying signals, curve is highly wrinkled, the curve becomes rather long, the drawback of the second approach is that two different whereas a straight line connecting the endpoints means that temporal sequences φ may average to the same value. m the parameter changes are smooth. The total variation of As an illustration, consider two signals of equal average a function may be used for such a measure; intuitively, it amplitude, the first having constant amplitude and the sec- measures the length travelled back and forth on the y-axis ond with a periodic amplitude modulation. Suppose we of a function y = f(x), x ∈ [a, b]. compare the RMS amplitudes of the two signals using the Let f(x) be a real function defined on an interval x ≤ second approach (4). When averaged over sufficiently long 0 x ≤ x , and suppose x < x < ··· < x is a partition of time, both signals will appear to have the same average k 0 1 k the interval. Then the total variation of f(x), x ≤ x ≤ x amplitude. In contrast, the pointwise distance measure (3) 0 k is defined as will detect their difference. k X 2.3 Estimation of the derivative Vxk (f) = sup |f(x ) − f(x )| x0 j j−1 (8) A numerical computation of the derivative may return a j=1 number even if the limit (1) or (2) does not exist. There- taking the supremum over all partitions of the function. If fore, a measure of the reliability of the estimate, or “degree f is differentiable, the total variation is bounded and can of differentiability”, should be added. be expressed as

648 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

x Z k Vxk (f) = |f 0(x)| dx. x0 (9) 0.15 x0 Also, recall that one way for a function to fail to be differentiable is that its total variation diverges to inﬁnity. 0.10 The mesh of the partition, which is the greatest distance fc : fm = 1

|xj − xj−1|, needs to be fine enough when estimating the Centroid total variation numerically. A global description of the function’s smoothness is obtained from considerations of 0.05 the limit of the total variation as the mesh gets finer. Sup- fc : fm = 1 2 pose the partition of [x0, xk] is uniform with each point separated from its nearest neighbours by |xj − xj−1| = ∆. Then, the question is whether a limit exists as ∆ → 0. 0 3 6 9 12 For the present purposes it will suffice to consider ap- Modulation index proximations of the total variation using a small but fixed mesh. Certain functions may appear to have different amounts of total variation when observed at different scales. A slow increase in total variation as the mesh is successively made finer indicates that the estimation process goes as intended. 0.025 An alternative to measuring the total variation would be to measure the arc length, which can be thought of as the length of a string fitted to the curve if it is continuous. 0.0

Fractal curves on the plane have the property that their arc Derivative length grows as the measurement scale gets smaller. When measuring the total variation of a signal descriptor over a range of synthesis parameter values, there are still −0.025 two possible approaches to how the distance is measured. As discussed above in section 2.2, either a pointwise dis- 0 3 6 9 12 tance may be taken, or the distance may be taken over time averages of the signal descriptors. The latter approach will be used here because it is better suited for the case of static parameters. Applications of the derivative and total varia- Figure 1. FM synthesis. Top: centroid as a function of tion to two synthesis models will be demonstrated next. modulation index for fc = fm = 440 Hz (solid line) and fc = 311.1, fm = 440 Hz (dashed line). The outer lines indicate one standard deviation of the centroid. Bottom: 4. FM SYNTHESIS the derivative of the centroid at fc = fm = 440 Hz. With only three synthesis parameters, basic FM synthesis is convenient for investigations of the smoothness of its parameter space. The formula that will be used is 1024 point Hamming window. As can be seen, the C:M ratio 1 gives a rather bumpy curve with a general rising trend of the centroid, but with several local peaks. The bottom

xn = sin(2πfcn/fs + I sin(2πfmn/fs)) (10) part shows the derivative, estimated with the method described in the end of Section 2.3. Evidently, the derivative with modulation index I, carrier frequency fc, modula- is discontinuous at each of the peaks. The RMSE of the tor frequency fm and sample rate fs = 48 kHz. Since linear regression used in the estimation of the derivative is the spectrum of the signal (10) is governed by a sum of typically very small, but has sharp peaks around the dis- Bessel functions [13], it may actually be possible to esti- continuities. It turned out to be necessary to re-initialize mate some related signal descriptors directly from the for- the oscillator’s initial phase at the beginning of each run at mula, although we will not attempt to do so. The oscil- a new parameter value, otherwise there would be oscilla- lations of the Bessel functions give FM synthesis its char- tions in the centroid as a function of modulation index that acteristic timbral ﬂavour of partials that fade in and out as would prevent the derivative from converging. the modulation index I increases, with the overall bright- The total variation of the centroid over the range 0 < ness increasing with the modulation index. Brightness is I ≤√12.5 is about 0.127 for the inharmonic ratio fc/fm = related to the spectral centroid, which will be used to study 1/ 2, and increases to about 0.188 for fc/fm = 1. We the effects of parameter changes. may now ask how the total variation changes as a function In the top of Figure 1, the centroid is shown as a function of the C:M ratio. This is shown in Figure 2. Narrow peaks of I at two different carrier to modulator (C:M) ratios. The arise at the simple C:M ratios 1 : 2, 1 and 3 : 2. Inso- centroid, given in units of normalized frequency, is mea- far as FM synthesis is reputed for its timbral variability as sured as the time average over 25 FFT windows using a the modulation index varies, this phenomenon is more pro-

649 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

FM / spectral centroid 0.8 I ∈ (0, 12.5) 0.20 0.7 0.6 0.15 Total Variation Total Spectral entropy Spectral 0.5 increasing modulation index (I = 0.25 − 20) increasing modulation index 0.4 0.10 0.5 1.0 1.5 C:M ratio 0 1 2 3 C:M ratio Figure 2. Total variation of the centroid of FM signals for I ∈ [0, 12.5] as a function of the C:M ratio. Figure 3. Spectral entropy of FM as a function of C:M ratio (horizontal) and modulation index (vertical). nounced at the simple C:M ratios that result in harmonic spectra. Since the density of the spectrum depends on the modulation index as well as on the C:M ratio, signal descriptors related to spectral density may provide additional insights. The spectral entropy will be used for this purpose. Spec- tral entropy is measured from the amplitude spectrum, normalized so that all bins ak sum to 1. Then, the normalized entropy is

1 X H = − a log a (11) norm k k k where a perfectly ﬂat spectrum yields the maximum spectral entropy H = 1, and a sinusoid results in the smallest Figure 4. Poincare´ section of the Rossler¨ system showing possible entropy of a signal that is not completely silent. bifurcations for c ∈ [1, 8] and a = b = 0.3. In Figure 3, the spectral entropy is shown as a function of the C:M ratio as well as the modulation index. De- In particular, there are many nonlinear oscillators capa- spite an even geometric progression of the modulation in- ble of both chaotic and periodic behaviour. Rossler’s¨ sys- dex I ∈ [0.25, 20], the curves are slightly irregularly dis- tem [14], tributed. Two dips in spectral entropy can be seen at the simple ratios C : M = 1, 2. These dips can be understood to result from the fact that, at harmonic C:M ratios, several x˙ = −y − z partials overlap (negative frequencies match positive frequencies), whereas for inharmonic ratios, there are more y˙ = x + ay (12) distinct partials in the spectrum. z˙ = b + z(x − c) The total variation of spectral entropy over the range of C:M ratios shown in Figure 3 is about 1 for I = 0.25, and is known to have a chaotic attractor at a = b = 0.2, c = it increases monotonically to a maximum value of 2.5 at 5.7. For lower values of c there are periodic solutions. I = 1.25. For higher modulation indices, the total varia- A Poincare´ section across the ray x = −y, x ≥ 0 at tion decreases. These results can be interpreted as indicat- a = b = 0.3 and a range of values of c reveals a period ing that, if the modulation index is set at a ﬁxed value and doubling route to chaos, after which there is a period two the C:M ratio is varied, then the sounds will change less for window (see Figure 4). In the following, (12) is solved low modulation indices, and the maximum change occurs with the fourth order Runge-Kutta method. The system is for I = 1.25. allowed time to approach an attractor by iterating at least 25000 time steps of size 0.025 before any measurements are taken. 5. THE ROSSLER¨ SYSTEM The system rotates in the xy-plane, with occasional spikes Ordinary differential equations with bounded and oscil- in the z variable. Therefore, the x and y variables are suit- lating solutions are good candidates for sound synthesis. able for use as audio signals, after they have been suitably

650 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

●● ●●●●●●● ●● ●● ●●●● ●●●● ●● δ ● the total variation over a short interval of length centred ● ●● ●●● ●●● ●●● ● ●●● ●●● ● ●●● ● ●●● ● ●●● ● ●● ●●● ● ●●● ●● ● ● ● ● ●●● ● ● ● ●●● ● ●●● ● ● ● ● ●● ● ● ●●● ● ● ●●● ● ● ● ● ● x ● ● ● about a point : ● ●● ● ●● ● ● ● ● ● ● ● ●●● ●● ● ●● ● ● ●

8 ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●●● ●● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ●● ● ● ● ●● ● ● ●● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●●●● ● ● ●● ● ●● ● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● x+δ/2 ● ● LV (f; x, δ) = V (f) (13) ● ● ● x−δ/2 ● ● ● ●

●● ●● ●●

5 ● ●● ●● ●●●● A mathematical definition of the LV would probably in- ● ● ● ● ●●● ●● ● ● ● ● ●●● ●●●● ● ● ●● ●●● ● ● ●●● ● ● ● ● ●● ●● ● δ → 0 ● ● ● volve taking the limit , but for practical purposes a ● ● ● ● ●●● ● ●●● ● ● ●● ● ● ●●●● ●● ● ● ● ●●● ● ● RMS amplitude ● ● ●●●●●●● ●●● ● ● ● ● ●●● ●●● ●● ●● ● ●●● ●●●● ●● ●● ●● ● ● ● ● ● ●● ●●●●● ●● ●● ● ●●●●●●●●● ●●●●● ● ●●●● small but finite interval must be used. Now the smooth- ●●● ●●●● ●●●● ●●●● ●●●●●●●●●●● ●●●●●●●●● ●●●●●●●● ●●●●●●● ●●●●●● ●●●●●● ●●●●● ●●●●● ●●●●● ●●●● ●●●● ●●●● ●●●● ●●●● ●●●● 2 ●● ness of a curve may be described in the neighbourhood of ● ●●● ●●● (x + y)/2 ●●● ●●● ●●● ●●● ●●● ●●● ●●● ●●● ●●●●●●●● z ●●●● ●●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●● ●●●●●●●●●● x ●●●● any point , which is computed by partitioning the inter- ● ●●●●●●●●●●● 0 ●●●●●● ●●●●●●● val into a suitably large number of points and proceeding 1 3 5 7 c as described above in Section 3. In the following example, δ = 0.02 has been subdivided into 16 steps to find the local Figure 5. RMS amplitude of the Rossler¨ system; the aver- variation. age of x and y is greater than z for low values of c. scaled in amplitude. The first thing to check with an ordinary differential equation intended for use as an audio 0.15 oscillator is its amplitude range and stability. As can be seen from Figure 4, the amplitude grows approximately 0.1 linearly with c over the displayed range. By measuring the λ RMS amplitude of each coordinate, one gets a more detailed overview of the amplitude’s dependence on the parameter c (see Figure 5). Because the amplitudes of x and 0.05 y are typically not very different, their average has been

plotted together with the amplitude of the z coordinate. 0 Bifurcation plots already reveal a few things about the smoothness under parameter changes. Each bifurcation is 1 3 5 7 a point where the system’s behaviour changes in a discon- c tinuous way, whereas the behaviour between bifurcations can be expected to vary more smoothly. 6 Before going further, let us recall that dynamic systems L.V. of may depend critically on the initial condition. Indeed, chaos RMS amplitude is deﬁned in terms of the exponential divergence of two orbits starting from inﬁnitesimally separated initial condi- 4 tions, which is measured with the largest Lyapunov exponent [15]. Even more dramatically, different initial con-

ditions may lead to different kinds of behaviour. In con- 2 servative systems, orbits may be periodic, quasiperiodic or Local Variation chaotic depending on the initial condition. Dissipative systems, such as Rossler’s,¨ have a basin of attraction of points that end up on the attractor, but should an orbit be started 0 from outside the basin of attraction, it may wander off to 1 3 5 7 inﬁnity. It is important to distinguish the properties of the orbit itself (chaotic versus regular) from the bifurcation scenarios as a parameter is varied. When looking at bifurcation di- Figure 6. Greatest Lyapunov exponent (top) and local agrams, there are intervals of smooth change and intervals variation of the RMS amplitude (bottom) for the Rossler¨ that are very irregular. It is tempting to guess that the ir- system as a function of the parameter c. regular parts correspond to chaotic orbits, and the smooth parts to periodic orbits. This is only a half-truth; in fact, The local variation of the average RMS amplitude of the there are periodic windows interspersed with all the chaos. x and y coordinates of the Rossler¨ system are shown in As already seen, the RMS amplitude changes smoothly Figure 6 below a plot of the largest Lyapunov exponent in some regions and irregularly in others. A quick compar- over the same parameter range. When λ = 0, the dynamics ison with the largest Lyapunov exponent λ indicates that is regular (either periodic or quasi-periodic), whereas λ > the irregular parts correspond to chaotic regions (see Fig- 0 indicates chaos. It is worth noting that regions of regular ure 6). Although it is easy to pick out “irregular regions” dynamics correspond to low values of the local variation, by visual inspection, a localized version of total variation i.e., the amplitude changes smoothly. At chaotic regions, can also achieve this. The local variation (LV) is deﬁned as the local variation obtains higher values, although there is

651 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

acoustic instruments using mechanical transducers to ex- 0.5 cite them. Mechanical transducers may be needed also for the automated control of acoustic instruments by MIDI or ● other means, but the response characteristics of the trans- ● ducer and the instrument considered together may not be d|z| 0.25 known in advance and need to be mapped out. Analog, voltage controlled synthesizers can be similarly studied by ● applying some control voltage to one of its inputs. Then,

0 studying the signal’s response to changes in control volt- 1 2 3 4 age can further elucidate input to output relations and the smoothness of the parameter. Although smoothness prop- c erties can be roughly assessed by visual inspection, the derivative, and the total and local variations provide quan- Figure 7. Derivative of the peak amplitude of the z coor- titative measures of smoothness. dinate as a function of c. Points of bifurcations are marked Comparisons of smoothness properties across different with circles. synthesis models are, however, not so straightforward. One might intuitively want to argue that the Rossler¨ system is no simple correlation between λ and LV. The higher values less smooth, on the whole, than FM synthesis, but the set of LV in chaotic regions can be partly explained by the of synthesis parameters have entirely different meanings existence of periodic windows which may be very thin, yet in the two models, so a direct comparison will be prob- are known to be dense in the chaotic regions. lematic. The same signal descriptors and distance metrics In the interval 1 ≤ c ≤ 4, there is a sequence of pe- must of course be used for both synthesis models, and one riod doubling bifurcations. Most changes in amplitude are must decide what parameter ranges to compare. too subtle to notice directly (compare Figure 5), but taking Noise is used in many kinds of synthesis. If the noise is the derivative, as shown in Figure 7, reveals points where prominent in the output signal, it will increase the variance the slope changes. In fact, the bifurcation points would be of the signal descriptors and make the estimation of deriva- even easier to detect by plotting the second derivative of tives and total variation more complicated. If the noise the peak amplitude. is mild enough not to alter the behaviour of the synthe- In this study of the Rossler¨ system, the effects of tran- sis model altogether, one can take ensemble averages over sients and dynamic parameter changes have been mini- many runs of the system. Stochastic synthesis such as Xe- mized. On the contrary, in a performance situation when nakis’ Gendyn algorithm [16] may however be beyond the using the Rossler¨ system as an audio oscillator, its param- scope of the present methods. eters would typically change over time. Then one may no- Ordinary differential equations and nonlinear feedback tice effects of hysteresis near bifucrations and in the chaotic systems may exhibit hysteresis. In synthesis models with regions. Approaching the same parameter value from dif- hysteresis, there is no longer a unique correspondence be- ferent directions may then result in different behaviour. tween the point in parameter space and the resulting output signal. This fact invalidates the assumption that the synthesis model can be thought of as a function that maps 6. CONCLUSION points in parameter space to sequences in the sample se- By conceiving of a synthesis model as a function from quence space. Sometimes a transition from one type of points in parameter space to one-sided real sequences of behaviour to another may depend not only on the direction audio samples, we have introduced a concept of derivative of the changing parameter, but also the speed of its change. and total variation that can be used to describe the smooth- We began by making the assumption that signal descrip- ness properties of the synthesis model. The derivative re- tors could be used instead of conducting listening tests. lates to local properties near specific points in parameter This is obviously an exaggeration. Firstly, one needs to space, whereas the total variation characterizes the amount know what perceptual characteristics of sound are captured of change over intervals of a parameter. Interesting find- by various signal descriptors. Second, we have been look- ings were that the total variation of the centroid with re- ing at rather small variations in these descriptors and mag- spect to the modulation index in FM synthesis is greater nified them with the derivative or considered their total for simple harmonic C:M ratios than for other ratios. In variation. It is very easy to gain a false impression that other words, FM becomes smoother for inharmonic C:M minor variations or roughnesses in the curves would be au- ratios than for simple ratios. In the study of the Rossler¨ dible. Listening tests would be necessary in order to assess system, we found that regular dynamics corresponds to how the smoothness and irregularity of parameter changes smooth variation in the RMS amplitude. Chaotic regions are really perceived. are generally less smooth in parameter space, but there The assumption that maximally smooth parameters are is some variation and relatively smooth parameter regions always preferable is not necessarily true. Monotonicity may exist where the system is chaotic as well. and smoothness may be good, because then the parame- The methods of characterizing the smoothness of synthe- ter can be remapped in a way that is more practical for the sis models can be applied to analog synthesis and even to user. Nevertheless, the rugged appearance of the parame-

652 Proceedings of the Sound and Music Computing Conference 2013, SMC 2013, Stockholm, Sweden

ter space of a chaotic system should not detract musicians [14] O. Rossler,¨ “An equation for continuous chaos,” from using them. Physics Letters, vol. 57A, no. 5, pp. 397–398, July 1976. 7. REFERENCES [15] T. Tel´ and M. Gruiz, Chaotic Dynamics. An Introduc- [1] D. Jaffe, “Ten criteria for evaluating synthesis tech- tion Based on Classical Mechanics. Cambridge Uni- niques,” Computer Music Journal, vol. 19, no. 1, pp. versity Press, 2006. 76–87, Spring 1995. [16] I. Xenakis, Formalized Music. Thought and Mathemat- [2] A. Hunt, M. Wanderley, and M. Paradis, “The impor- ics in Music. Stuyvesant: Pendragon Press, 1992. tance of parameter mapping in electronic instrument design,” in Proceedings of the 2002 Conference on New Instruments for Musical Expression (NIME-02), Dublin, Ireland, 2002.

[3] D. Sanﬁlippo and A. Valle, “Towards a typology of feedback systems,” in Proc. of the ICMC 2012, Ljubl- jana, Slovenia, September 2012, pp. 30–37.

[4] M. Caetano and N. Osaka, “A formal evaluation frame- work for sound morphing,” in Proc. of the ICMC 2012, Ljubljana, Slovenia, 2012, pp. 104–107.

[5] J. Grey, “Multidimensional perceptual scaling of musical timbres,” J. Acoust. Soc. Am, vol. 61, no. 5, pp. 1270–1277, May 1977.

[6] A. Caclin, S. McAdams, B. Smith, and S. Winsberg, “Acoustic correlates of timbre space dimensions: A conﬁrmatory study using synthetic tones,” J. Acoust. Soc. Am, vol. 118, no. 1, pp. 471–482, 2005.

[7] T. Elliott, L. Hamilton, and F. Theunissen, “Acoustic structure of the ﬁve perceptual dimensions of timbre in orchestral instrument tones,” J. Acoust. Soc. Am, vol. 133, no. 1, pp. 389–404, January 2013.

[8] J.-J. Aucouturier and F. Pachet, “Music similarity measures: What’s the use?” in Proceedings of the Interna- tional Symposium on Music Information Retrieval (IS- MIR), Paris, France, October 2002.

[9] G. Peeters, B. Giordano, P. Susini, N. Misdariis, and S. McAdams, “The timbre toolbox: Extracting audio descriptors from musical signals,” J. Acoust. Soc. Am, vol. 130, no. 5, pp. 2902–2916, November 2011.

[10] M. Barthet, P. Guillemain, R. Kronland-Martinet, and S. Ystad, “From clarinet control to timbre perception,” Acta Acustica united with Acustica, vol. 96, pp. 678– 689, 2010.

[11] W. Thurston, “On proof and progress in mathematics,” Bulletin of the American Mathematical Society, vol. 30, no. 2, pp. 161–177, April 1994.

[12] W. Press, S. Teukolsky, W. Vetterling, and B. Flannery, Numerical Recipes. The Art of Scientiﬁc Computing, 3rd ed. Cambridge University Press, 2007.

[13] J. Chowning, “The synthesis of complex audio spectra by means of frequency modulation,” Journal of the Au- dio Engineering Society, vol. 21, no. 7, pp. 526–534, September 1973.

653