Artists’ Article Motion Tracking of a Fish as a Novel Way to Control Electronic Music Performance S h a l T i E l Eloul, Gil Zi S S u , Y E h i E l h . a M o a n d N o r i J a C o b Y The authors have mapped the three-dimensional motion of a fish T of Aristotle, who, in the 4th century BCE, tried to relate a C onto various electronic music performance gestures, including loops, fish’s fins to its straight motion, somewhat anticipating Isaac r melodies, arpeggio and DJ-like interventions. They combine an element S Newton’s identification of the fins as the physical basis of of visualization, using an LED screen installed on the back of an ABT a aquarium, to create a link between the fish’s motion and the sonified the generation of thrust for locomotion in water in the 17th music. This visual addition provides extra information about the fish’s century [2]. role in the music, enabling the perception of versatile and developing This unique locomotion can be both patterned and ran- auditory structures during the performance that extend beyond the dom, and therefore our translation of the fish’s movement sonification of the momentary motion of objects. into musical syntax observed the following rules: On the one hand, the fish’s repetitive paths could be captured as rhythm, Popular electronic music performances are situations in while on the other hand, occasional random movements which highly structured music defines the mood and social could trigger surprising musical responses and effects. An interactions among people within the same space. This style electronic club music genre that has analogous properties of music typically combines accessible musical phrases with is minimal techno [5], characterized by slowly developing variations that enhance the experience for the human ear [1]. rhythms that exploit the use of repetition while making gen- Many clubs and electronic venues also create multisensory erous use of synthesized sounds and prerecorded realistic environments in an attempt to enhance club-goers’ experi- sounds. This genre is an inspiration for our work on translat- ences. Introducing an interactive musical aquarium is one ing the motion of a fish into various electronic music styles such enhancement, creating subtle similarities between the as a new way to express the interaction between visual and movement of fish in an aquarium and the electronic music auditory senses in club and electronic music scenes. style. Music computing artists have been increasingly interested From auditory to visual domains, fish exhibit a wide vari- in identifying new methods of expressive engagement with ety of swimming patterns and behaviors in response to their interactive music systems [6–8], i.e. systems that express ges- environments. Their movements are visually attractive to the tures to generate and control musical parameters and sig- human eye and have also proven to be valuable in cognitive nals for virtual instruments [9–13]. Another emerging area studies [2] and psychological therapy [3]. The hydrodynamic of study is the sonification of various natural phenomena shape and body movements of fish (which are related to their by analyzing data or tracking objects. This includes water size and type) merge into the aquatic locomotion that has sonification [14], music based on climate [15], music maps captivated human onlookers for thousands of years [2,4]. of the routes of stars [16] and sounds based on molecules The earliest recorded reference to fish motion is perhaps that and biomolecules [17, 18]. The aim of our work is to create an expressive interactive performance with a stimulating Shaltiel Eloul (artist-student), St Cross College, Oxford, St Giles, OX1-3LZ U.K. phenomenon, in this case the sonification of fish motion, by Email: <[email protected]>. using motion tracking. Gil Zissu (artist), University of the Arts, London, WC1V 7EY U.K. Tracking visual motion (e.g. dancing, natural phenomena Email: <[email protected]>. or animals’ motion [19]) in real time and sonification of mo- Yehiel H. Amo (artist), Beer-Sheva, Israel. Email: <[email protected]>. tion are now relatively manageable tasks owing to the wide Nori Jacoby (advisor and researcher), Department of Music, Bar Ilan University, Ramat Gan, Israel, 52900, and Edmond and Lily Safra Center for the Brain Sciences, availability of high-speed cameras and tracking devices such Hebrew University of Jerusalem, Jerusalem. Email: <[email protected]>. as the Kinect [20, 21]. Yet the translation of these tasks into See <www.mitpressjournals.org/toc/leon/49/3> for supplemental files associated intriguing onstage musical performances remains challeng- with this issue. ing, both technologically and artistically [22]. This difficulty ©2016 ISAST doi:10.1162/LEON_a_01072 LEONARDO, Vol. 49, No. 3, pp. 203–210, 2016 203 Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LEON_a_01072 by guest on 23 September 2021 Fig. 1. Sketch of equipment used during performances and exhibitions. All the equipment is portable, and setup requires approximately one hour. (© Shaltiel Eloul) is likely a result of the signifi cant gap between the demon- study of sonifi cation and the implementation of sound from stration of an idea in a research framework (“in the lab”) and visual events. However, their auditory results did not seem the creation of a compelling live performance. When per- to attempt to stimulate the listener’s attunement to various forming, the artist must consider many parameters, such as music phrases or to develop music structures over time. the quality of the music, visual interpretation, the audience’s Moreover, their works were focused mostly on solving tech- ability to understand the artistic ideas and, above all, the nological issues in sonifi cation, such as motion tracking with artist’s ability to maintain the audience’s interest throughout multiple objects, rather than directly applying it to the cre- the performance. Th erefore, the artist’s challenge is to turn ation of music or to the combination of their technology with a new technological art idea into an enjoyable and pleasant time-based performance. experience for the audience in a live show, exhibition or club In 2011, Nikolaidis and Weinberg [26] created a general performance. Here, we present the idea of creating and play- model for dynamic sonifi cation of visual movements into ing music generated by the motion of a fi sh, which we have “low-level” elements such as pitch and panning as well as developed into an onstage performance. “high-level” elements such as melodic attraction. Still, music that relied solely on this model would be limited to tracked PErForMaNCE baCkGrouNd objects over a specifi c time. To avoid such specifi c momen- Th e fi rst major work to address the challenge of translating tary movements, we used a background screen that can in- fi sh movement into music was produced at the San Francisco teract with the movement of the tracked object, rendering Tape Music Center by Ramon Sender in 1962 [23]. In his the melody and musical ideas more complex and diverse. Tropical Fish Opera, four instrumentalists used a tank to For example, “painting” the movement of the fi sh in real create music scores for an improvisational performance. time can provide a visual record of the fi sh’s routes or pre- Th e players improvised the music by drawing staff lines vious positions, as shown below in the section “Audio and on the fi sh’s tank and trying to play the positions of the Visual Translation Patches.” Th en, aft er this information has fi sh as it swam up and down in the tank. been captured on the screen, we can use it to implement Sonifi cation using optical tracking was proposed by large-scale musical ideas (such as create rhythm, arpeggios Pendse et al. in 2008 [24] and more recently by Baldan et or loops) without losing the relation between the music and al. in 2012 [25]; these works describe the idea of multiple the fi sh’s motion. Above all, we must also take into account fi sh sonifi cations created according to the animals’ size and aspects of visual and music perception, aesthetic value and location in an aquarium. Th e studies demonstrate a restricted logistical organization, so as to provide a performance that method of sonifi cation, which is important for the general appeals to both musicians and nonmusicians. 204 Eloul et al., Motion Tracking of a Fish as a Novel Way to Control Electronic Music Performance Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/LEON_a_01072 by guest on 23 September 2021 Inst allaTioN Sketches of the aquarium equipment and photo- graphs from live installations are shown in Figs 1 and 2, respectively. We positioned the fish in an aquarium (Fig. 1a) onstage, as depicted in the upper photo in Fig. 2. To detect the three- dimensional motion of the fish, we placed two FireWire cameras at the top (Fig. 1b) and the right side of the aquarium (Fig. 1c). Three parameters —position, velocity and acceleration—are trans- lated to control musical gestures and triggers. To compute the fish’s motion and all music ele- ments, we used two laptop computers (Fig. 1d), a synthesizer (Fig. 1e) and one portable audio card (Fig. 1f). We further combined a visual interpretation with an LED screen (Fig. 1g) positioned on the back of the aquarium. This visual element is con- trolled by the fish’s motion and by music triggers using a third laptop. We found that this added visual interpretation is crucial to the audience’s understanding of the interaction between the fish and the music, as we explain below.