Proceedings of the International Conference on New Interfaces for Musical Expression

Designing Sound for Recreation and Well-Being

Anders-Petter Andersson Birgitta Cappelen Fredrik Olofsson Kristianstad University Oslo School of Architecture & Design Musikproduktion 29188 Kristianstad 0175 Oslo Boxhagener Str. 16 QG Sweden Norway 10245 Berlin [email protected] [email protected] Germany [email protected]

ABSTRACT with and ambient sounding furniture. We try to understand how we In this paper we explore how we compose sound for an interactive can compose dynamically changing music for many relations like tangible and mobile interface, where the goal is to improve health these. Music that motivates users to act and shift roles between those and well-being for families with children with disabilities. We relations. Something that according to our observations of families describe the composition process of how we decompose a linear interacting contribute to health and well-being. beat-based and vocal sound material and recompose it with real-time We know that musical activities are good for health and well- audio synthesis and composition rules into interactive Scenes. Scenes being, based on the last 15 years of research [3, 12, 8], and on that make it possible for the users to select, explore and recreate practice in peoples everyday use of music as “soundtracks of our different sound worlds. In order to recreate, the users interact with the lives” for regulating emotions and social relations (6) and in Music tangible interface in different ways, as instrument, play with it as a Therapy. Music has a documented ability to evoke emotions, help friend, improvise and create music and relax with it as ambient mastering tasks and situations such as playing instruments, and sounding furniture. We discuss composition techniques for mixing managing the own body [12]. It further strengthens social relations sound, tangible-physical and lighting elements in the Scenes. Based through singing in choirs, dancing and listening. Finally music on observations we explore how a diverse audience in the family and creates individual, cultural and social meaning and a sense of at school can recreate and improvise their own sound experience and coherence in life [2, 12]. We like to call this perspective a relational play together in open and non-therapeutic everyday situations. We and resource-oriented perspective emphasising what a person can do conclude by discussing the possible impact of our findings for the rather than his or her weakness or illness [11, 12]. The concept of NIME-community; how the techniques of decomposing, relation is used by musicologist Christopher Small and is linked to recomposing and recreating sound, based on a relational perspective, his term “musicking” [13]. He describes music, not as an aesthetic could contribute to the design of new instruments for musical object or work of Art, but as potential relations between persons, expression. their experiences of music, and activities of all sorts of music making. Many traditional and electronic music technologies used for well- being have limitations and even show negative effects on well-being Keywords and health. This is confirmed by Music Therapist and Music and decompose, recompose, recreate, mobile, tangible, well-being, Health researchers [9, 5] working with assistive technologies like the SuperCollider, Inclusive Design, Assistive Technologies, relational ultrasound sensor SoundBeam (http://www.soundbeam.co.uk) and 1. INTRODUCTION switch-based Paletto (http://www.kikre.com). In our view, the reason for negative effects is a belief that users need to establish a causal It is normal in many composition traditions to borrow musical understanding of stimuli-response prior to that they can start creating material from others. In classical, modern and popular music the music, express relations and experience meaning. Even if the composer tends to borrow musical elements and structures. In intention may be good, the focus on a tool-oriented cause-and-effect interactive computer based composition the focus normally is more can make a physically or cognitively weak user experience fatigue on the development of new instruments; either as tools for the and disempowerment before being able to master actions like hitting composition, but more often, as parts of the composition. A typical a particular sensor and create music [9, 5]. “NIME-interface” often is a unique interpretation of how action In previous papers we have showed an alternative strategy to connects to sound, and "...understanding this interpretation can be as building interfaces based on tool-oriented cause-and-effect [1, 5]. aesthetically rewarding as listening to the sound itself." [10, p. 57] Instead we have expanded the role of the traditional music instrument Our experience, though, is that something else happens if you bring [4]. As alternative we have designed computer based interactive the instruments out into the everyday world of amateurs. An interface music interfaces building on the relational and resource-oriented such as Reactable (www.reactable.com) is more than a tool and perspectives described above [13, 11, 12]. Interfaces that learn, composition. It has a physical presence. When Reactable is exhibited remember, and adapt their responses after diverse users’ abilities. at a museum where children play, it creates many relations to its Interfaces that are rich tangible multimedia that change the sound users. Yes, it becomes a tool for creating music, but also toys to play responses musically over time and with the user interaction. The result is that persons interacting are motivated to create, strengthen Permission to make digital or hard copies of all or part of this work for personal social relations, play and improvise, experience sense of coherence or classroom use is granted without fee provided that copies are not made or and well-being. distributed for profit or commercial advantage and that copies bear this notice Earlier studies of computer based music and assistive technologies and the full citation on the first page. To copy otherwise, to republish, to post on for persons with disabilities that follow a relational perspective are servers or to redistribute to lists, requires prior specific permission and/or a fee. rare to find. There are even fewer studies for this group investigating NIME’14, June 30 – July 03, 2014, Goldsmiths, University of London, UK. practical-creative work with composition techniques and expressive Copyright remains with the author(s). qualities of such computer based technologies. It makes our

preliminary findings [4, 5, 1] unique.

529 Proceedings of the International Conference on New Interfaces for Musical Expression

The research question we like to explore in this paper is how one combined dynamically by the users over time. Our earlier findings (1, can compose sound for an interactive mobile and tangible interface, 4, 5, 14, 15) show that musical variation and change over time is motivating and strengthening well-being for families with children important for creating expectations and motivate persons on different with severe disabilities. levels of mastery to continue to be active and create relations. Again Our practical example is the tangible interface Reflect, and our this is a complement to the tool-oriented focus on cause-and-effect suggestion for composition techniques is a process of decomposing, discussed above. Further, when recomposing rules for change over recomposing and recreating. The composer decomposes linear music time, the composer take into consideration both the musical consisting of strong musical elements such as riffs, jingles and possibilities to create expectations and the users’ everyday stories in grooves or beats and recomposes and programs rules for them into relation to the particular piece. For instance expecting a guitar riff in what we call Scenes. The music is recreated by users interacting with an up-coming chorus motivates the users to dance and mimic playing the interface in real-time, creating relations to each other, the sounds the guitar. The composer therefore needs to consider movement, and the interface. visuals and lighting response that goes with the music, and get help to design them into Scenes. A single sound, light or movement chunk 2. METHOD and its composition rules are designed so that they are possible to change dynamically when the person interacting, changes his or her 2.1 The Multidisciplinary Rhyme Project relationship with the music. Consequently, it is not enough to attach a The context for this paper is the 5-year research project Rhyme static sounding sample to an interactive object. A user goes from (http://www.rhyme.no) in the area of Internet-of-things, mobile viewing the music as a tool, getting motivated by short accentuated computing and social media for health. The goal is to improve and repeated sounds, separating the sound events stressing the break health and well-being for families with children with severe and the causal relation between action and sound, to dialogue. A disabilities in open, non-therapeutic everyday situations. We dialogue with the interactive tangible interface, considering it as a co- have so far explored multimodal, tangible and mobile computer player that answers back with over time varying sound and light. based designs to motivate persons on different levels of musical We define recreation as the users’ activities to interpret and co- mastery, activity levels and with different abilities [14, 15, 5]. create content with the system and other humans. It is possible if the The design and research is multidisciplinary and made in a composer is doing an analysis of what activities the amateur users group consisting of an industrial designer and interaction find meaningful, in combination with creating the chunks so that they designer, a musicologist and a composer and programmer. The are coherent with the character of the Scenes and the music. development is done in 4 generations of prototypes in In summary, the composer is looking for relations between the collaboration with users and experts in Music and Health and original music, that he or she decomposes into parts, and over time Music Therapy. We have the advantage of continuous contact potentially motivating relations in the process of recomposing for with the same 8 individuals and their families over as long as 4 new situations, based on the understanding of the users years time. That makes it possible to follow-up on design interpretations in the process of recreation choices and the users’ experiences that happened from one week to the other and as long as three years back. 3. RESULTS We and our colleagues in Music and Health research have been writing about our findings in earlier texts for the NIME 3.1 Reflect community and elsewhere [1, 4, 14, 15]. The case focus on one family and discusses composition based on a relational perspective of music and well-being. 2.2 Decomposing, Recomposing, Recreating When it comes to the practical work of adopting a relational perspective [13] to composition, our suggestion is to divide the work into three processes; decomposing, recomposing and recreating. We define decomposing as the process of selecting and analysing music, based on the music preferences [7] of individual children and their families. The selection is based on an understanding of the relations between musical elements, structures, genres, and songs, on how the persons use the music and how it creates relations in their cultural context. In order to build on already established relations, we analyse music that is well known to the persons with disability and shared with their families. It is therefore important that the selection is based on the users’ individual preferences. A consequence is that the selection can be wider or targeted to one genre, e.g. avant-garde, contemporary music. If the selection is wider, it can for instance consist of a mix of popular up-tempo grooves or rock-based songs, combined with folk music, or soft lullabies and ambient music. In order to understand and suggest potential use of the music, a person who is decomposing also analyse non-musical elements, movements, tangible and visual expressions that are relevant when appreciating the music, e.g. disco dance movements, rock guitar playing, stroking Figure 1. Man holding Reflect’s belly and pointing its trunk and hugging, etc. with RFID-reader against a RFID-tag playing beats We define recomposing as the act to, create potential relations Reflect is a mobile and wireless interactive tangible installation. It based on significant parameters and structures in the music and/or offers people possibilities to select and play with music and with vocal material, such as timbre, harmonic progressions, melodies, others, and thereby reflect on their actions. It consists of a lumber-like beats, bridges, riffs and choruses. Further to recompose them into soft thing that you can play with on the floor, hold in your arms, or sound chunks and program composition rules, so that they can be over the shoulder while dancing. Reflect's embedded sensors, such as

530 Proceedings of the International Conference on New Interfaces for Musical Expression

touchable glowing stars, its speakers and lighting makes it possible character and differentiated it from the others; as the frog sound, lion for the user to create music and light. sound, monkey sound, etc. Reflect uses a RFID-reader at the end of its trunk (see figure no. 1) For the persons that wanted to play with the dolls and develop to collect and dynamically change music samples from any other relations to them as friends or co-players, we added three voice thing with RFID-tags in plastic key cards attached to them (see figure variations to each doll to toggle between to motivate dialogue. no. 2), and touch and bend sensor to dynamically mix, manipulate We also created composition rules to vary the objects sounds musically and play on and with the samples. depending on the active Scene. For instance did all objects play a Technically the software in Reflect is written in the object oriented four-note motive based on Mam-ma-Mi-a whenever a beat was programming language SuperCollider (http://supercollider.github.io) activated in Abba’s song. The rubber duck sounded Quack-quack- and is running on an iPod Touch. The hardware is a mixture of Quack-quack. The bongo drum went Boing-boing-Boing-boing, etc. custom-built circuits for sensors and light, and standard mobile phone In the song about the animals in Africa we created coherent technology like portable speaker and battery pack. It makes the variations through relation between the animal doll and the matching platform self-sufficient and wireless. verse. Consequently, we created a composition rule that let the animal a person interacted with select the matching beat, pitch range and timbre from a verse about the same animal. When the user selected another animal the sound changed on the first beat of the next bar. The result was variation that followed the users choice and still was coherent with the character in the story and the particular animal the user had chosen. In addition we used the Reflect lumber-like object’s tangible qualities to motivate hugging and patting by adding star-like glowing touch sensors to its body. The stars made squeaky high-pitched synthetic sounds when pushed as contrast to the acoustic voices. A bending sensor in one arm worked as guitar whammy, making noises and changes in pitch to the playing sample. These things motivated persons that wanted to dance and move to pick up Reflect and hold it Figure 2. Maracas and Monkey with white RFID-tags as if it was a guitar or a dance partner. During user tests we added a Apart from reading the plastic RFID-tags that can be attached to strap so that persons could strap Reflect on to mimic a real guitar. things like toys, instruments and playback sound samples and trigger lighting, the 2 digital bend sensors and 5 analogue touch sensors can 3.4 Recreate Motivating Relations filter the sound and add effects and synthesized sounds. Wendy, a teenage girl with Downs syndrome and her family entered the music room at the school. Wendy was curious and exited of what 3.2 Decompose Children Songs, Disco, she was about to meet. She smiled when she saw Reflect. The bean Maracas, Rubber Ducks and Bongos shaped object in soft velvet textile in huggable size was lying on a mat on the floor. Spread around it was 20 hand sized toys and We interviewed 8 children with families and asked what songs they instruments, such as rubber ducks, fluffy balls, small congas and would like to work with. Here we concentrate on one child and her family consisting of child, mother, father and grandmother. maracas. Wendy expected it to answer to her actions and her voice in Their favourites were Abba’s Mamma Mia; and Gimme, Gimmie, music and lighting. Her expectations were based on earlier Gimme; Alexander Rybak’s Fairy Tale; children songs like the experiences, one and two years ago. As a participant in the 5-year research project she had already tried similar interactive music Norwegian The Animals in Africa/Dyrene i Afrika by Torbjørn technologies. One of the researchers showed how Reflect worked. Egner; and Captain Sabertooth/Kaptein Sabeltann by Terje Formoe; Wendy threw herself on to the mat and started to hug Reflect. As she Philly soul disco groove beats; and Claude Debussy’s Prelude to the hugged it she squeezed the body and activated the touch sensors Afternoon of a Faun. looking like stars. The stars immediately lit up and made high- Apart from the music we looked for sounds for all sorts of playful objects that a family with children could relate to, such as toys, toy pitched whistle sounds as if Reflect where taking to her. She took a cars, dolls, rubber ducks, small music instruments like maracas and grip around the neck and asked it back by talking into the trunk/nose bongo drums and soft cute or fury objects that could go well with at the end as if it was a microphone: “Say taco”, Wendy said. Nothing happened until she bent the neck and it made swoosh each song. sounds. Wendy’s father gave her one of the fuzzy balls in pink that One example was the popular song text to the Animals in Africa, had a RFID tag attached. He asked her to hold the tag against the where chattering monkeys, soprano frogs and baritone lions all had nose containing a RFID reader. Wendy tried it out and Reflect started their own choruses. These are animals that children are used to mimic to play a looping beat in the song Mamma Mia by Abba. Both she as they sing the song and therefore have a relation to. We found matching dolls and sounds of monkeys, frogs and lions to go with and her mother immediately started to dance and moved their bodies them (see figure 2). The beats in the song were cut in 2-4 bars, 8-15 to the beat and sang along with the repeating text Mamma Mia. seconds long chunks, synchronised to the pulse, so it was possible to Wendy picked up the rubber duck and held it against the nose all by her self. Reflect played back a duck’s voice on the same melody and loop them. We also edited and selected a part of each song to become four-syllable rhythm as Mam-ma-Mi-a: “Quack-quack-Quack- a jingle. quack”. The mother and Wendy echoed the phrase with the original 3.3 Recompose Sounds Coherent with Scenes text, singing in consonant: “Mam-ma Mi-a” over and over again Slowly the scenes took shape with tangible dolls and objects and while dancing. Wendy turned the beat off and signalled to the mother sounds to go with them. In order to make the scenes coherent with that she wanted to rest and lie down on the mat and some cushions, the characters of each song and the objects in each scene; the sounds using Reflect as a blanket. Wendy invited the mother to lie down also had to vary with the role and relation that the user wanted to take beside her and started to cuddle with Reflect that lit up and made tiny in a certain situation. high-pitched sounds every time she hugged it and touched the stars. For the person trying to master the dolls as instruments, it was The mother was lying down and together with her daughter created important that the dolls firstly had a sound; and secondly, a an intimate and safe environment where the sounds from Reflect was characteristic sound that matched or added something to the physical part of the ambience where they made small talk, caressing the

531 Proceedings of the International Conference on New Interfaces for Musical Expression

Reflect while relaxing together. After a while Wendy started to talk findings show that music’s ability to change over time and create into Reflect’s nose again, talking to it as if Reflect was a person. On content that is coherent is motivating for everybody involved, leading her back she continued to explore the cushion’s form, materiality and to active persons, well-being and health. rubber band with her hands and feet. The following week and the second time Wendy interacted with 6. CONCLUSION the Reflect, she brought her mother and grandmother. The We have explored how to compose sound for the interactive tangible grandmother sat down in a sofa facing her daughter and and mobile interface Reflect, promoting health and well-being for granddaughter that danced and played with the toys. Wendy put the families with children with disabilities. Reflect cushion into the grandmothers lap, gave her two toys and We have investigated the composition process from decomposing showed her how to hold the tags against the trunk. When the a linear beat-based and vocal sound material. How a user can recreate grandmother didn’t manage to change the sound, Wendy showed her it with real-time audio synthesis into Scenes. Scenes that made it one more time until she got impatient and took Reflect off the possible for persons with diverse abilities to develop personal grandmother’s lap and strapped it on to her own body, with a relations to the music, each other and the tangible interface as an bouncing rubber strap around her neck as if holding a guitar. The instrument, improvise and play with it, hugging it as a friend and to mother still dancing made the association and asked: “Are you relax and use as ambient sounding furniture. playing the guitar?” Wendy swung her hand back and forth as if Based on observations we have explored how a family can re- playing on invisible strings while dancing and sharing together with create and play with Reflect. the mother and the grandmother, sitting in the sofa and moving her We have discusses how NIME’s readers could go further by feet to the musical pulse. working in all three phases of decomposing, recomposing and recreating as part of their compositional and reflective practices. 4. ANALYSIS NIME could make a difference in peoples lives and have impact on Reflect created different relations to its users, by changing its answer health and well-being through improvisation in sound and music. depending on different musical Scenes and user activities. Reflect took the role of a teddy bear and a friend. For instance as Wendy 7. ACKNOWLEDGMENTS talked to Reflect, asking it to “Say taco”, as if it was a person. Our thanks to the Research Council of Norway and the VERDIKT Further, Reflect took the role of a fellow improvising musician as programme that has made it possible to make the RHYME project. when the duck responded with a variation to the motive “Mam-ma Mi-a” with “Quack-quack-Quack-quack”, strengthening the role of 8. REFERENCES Wendy and her mother as co-players. Or, the role of a tool to be [1] A. Andersson, B. Cappelen. Designing Empowering Vocal and pushed when Wendy assumed a teacher’s role, teaching her Tangible Interaction. In Proc. NIME 2013. 2013. grandmother how to play. Or, the role of ambient landscape and [2] A. Antonovsky. The Salutogenic Model as a Theory to Guide furniture for Wendy and the mother to lay down and feel safe in. A Health Promotion. Health Promotion Intern., 11/2, 1996, 11-19. furniture to play out an intimate relation, normally associated with a [3] G. Bjursell, L. Westerhäll. Kulturen och Hälsan (in Swedish home-environment, and not a public office. Culture and Health), Santerus, Gothenburg, 2008. The observations also show how Wendy from the very first minute [4] B. Cappelen, A. Andersson. Expanding the role of the found it meaningful to start interacting, even if she then didn’t knew instrument In Proc. NIME 2013. 511-514, 2011. how to make sounds and operate Reflect. Even if she at that point [5] B. Cappelen, A. Andersson. Musicking Tangibles for didn’t understood the causality between her action and the sound Empowerment. ICCHP 2012, Part I, LNCS 7382. Springer- coming from Reflect, she was motivated by the sound and light while Verlag, Berlin, Heidelberg. 2012, 254–261. she was hugging Reflect and continued to interact. [6] T. DeNora. Music in everyday life. Cambridge Univ. Pr., 2000. [7] S. Lyngroth, T. Kvamme, B. Skogen. Når musikk skaper 5. DISCUSSION kontakt (in Norwegian, When Music Creates Contact – From Wendy’s interaction and how she switched between roles and Developing Individualised Music), 1, Report, 2006. relations in the different Scenes, and with her family, we believe that [8] R. MacDonald et al. Music, Health, and Wellbeing. Oxford computer based interactive tangibles can offer new possibilities University Press, 2012. leading to well-being. Well-being as the possibility to master, create, [9] W. Magee, K. Burland. An Exploratory Study of the Use of strengthen relations and sense-of-coherence based on a relational Electronic Music Technologies in Clinical Music Therapy. perspective on musicking [13]. A future development could be to Nordic Journal of Music Therapy, 17/2, 2008, 124—141. offer Reflect on a 24/7 basis, and program even more dynamic rules [10] T. Murray-Browne, D. Mainstone, N. Bryan-Kinns, and M.D. changing the music to be motivating for longer time. Another Plumbley. The Medium is the Message: Composing possibility would be for users to add music on their own, in order to Instruments and Performing Mappings. In Proc. NIME 2011. extend users’ freedom and the possibilities to make decisions [13, 6]. 2011, 56—59. To create computer based interactive tangibles that are meaningful [11] R. Rolvsjord. Resource-Oriented Music Therapy in Mental for amateurs, we believe designers and composers, need to work Health Care. Barcelona, 2010. relational and with all three processes from decomposing, [12] E. Ruud. Music Therapy: A Perspective from the Humanities. recomposing to recreating. Barcelona, 2010. They need to take responsibility for doing the selecting and [13] C. Small. 1998. Musicking : The Meanings of Performing and decomposing of genres, songs and sounds, with an understanding of Listening. Univ. Press of New England, 1998. the persons who should play and recreate with the music. In the [14] K. Stensæth. “Musical co-creation”? Exploring Health- recomposing of the material and the writing of composition rules Promoting Potentials on the Use of Musical and Interactive they should not leave the interaction design to a person without Tangibles for Families with Children with Disabilities. Intern. J. musical and aesthetical understanding and craftsmanship. of Qualitative Studies on Health and Well-Being, 8/0, 2013. Finally, we need to upgrade the status of working with users of all [15] K. Stensæth, E. Ruud. Interaktiv Helseteknologi - Nye kinds. We call it recreation because it is creative; the material is Muligheter for Musikkterapien? (in Norwegian, Interactive composed to exceed the initial ideas. The user is part of creating the Health Technology - New possibilities for Music Therapy?), music. At the same time the composer is needed more than ever. Our Musikkterapi, 2, 2012, 6—19.

532