DEGREE PROJECT IN THE FIELD OF TECHNOLOGY MEDIA TECHNOLOGY AND THE MAIN FIELD OF STUDY COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2017 Mapping physical movement parameters to auditory parameters by using human body movement OLOF HENRIKS KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF COMPUTER SCIENCE AND COMMUNICATION Mapping physical movement parameters to auditory parameters by using human body movement Mappning av fysiska r¨orelseparametrar till ljudparametrar genom anv¨andningav m¨ansklig kroppsr¨orelse By: Olof Henriks, [email protected] Submitted for the completion of the KTH programme: Master of Science in Engineering in Media Technology, Master's programme in Computer Science. Supervisor: Ludvig Elblaus, KTH, School of Computer Science and Communication, Department of Media Technology and Interaction Design. Examiner: Roberto Bresin, KTH, School of Computer Science and Communication, Department of Media Technology and Interaction Design. Work commissioned by: EU Dance Project. Date of submission: 2017-01-15 Abstract This study focuses on evaluating a system containing five different mappings of physi- cal movement parameters to auditory parameters. Physical parameter variables such as size, location, among others, were obtained by using a motion tracking system, where the two hands of the user would work as rigid bodies. Translating these vari- ables to auditory parameter variables gave the ability to control different parameters of MIDI files. The aim of the study was to determine how well a total of five partic- ipants, all with prior musical knowledge and experience, could adapt to the system concerning both user generated data as well as overall user experience. The study showed that the participants developed a positive personal engagement with the sys- tem and this way of audio and music alteration. Exploring the initial mappings of the system established ideas for future development of the system in potential forth- coming work. Sammanfattning Denna studie fokuserar p˚autv¨arderingav ett system som inneh˚allerfem olika mapp- ningar av fysiska r¨orelseparametrartill ljudparametrar. Variablerna f¨orde fysiska r¨orelseparametrarnainh¨amtades via ett motion tracking-system, d¨aren anv¨andares b¨aggeh¨anderverkade som stela kroppar. Overs¨attningav¨ dessa variabler till variabler f¨orljudparametrar m¨ojliggjorde styrning av olika parametrar inom MIDI-filer. M˚alet med studien var att besluta hur v¨alfem anv¨andare,alla med tidigare musikalisk kun- skap och erfarenhet, kunde anpassa sig till systemet g¨allandeb˚adeanv¨andargenererad data samt ¨overgripande anv¨andarupplevelse. Studien visade att anv¨andarnautveck- lade ett positivt personligt engagemang med systemet och detta s¨attatt p˚averka ljud och musik. Unders¨okning av systemets initiala mappningar etablerade id´eerf¨or framtida utveckling av systemet inom potentiellt kommande arbete. i Acknowledgements EU Dance Project, my principal. Thank you for allowing me to work with you. Giampiero Salvi, my supervisor at KTH for the first parts of my thesis. Ludvig Elblaus, my supervisor at KTH for the later parts of my thesis. Roberto Bresin, my examiner at KTH. Jimmie Paloranta, my partner in crime and endless source of assistance. ii Glossary BPM Beats Per Minute MIDI Musical Instrument Digital Interface OSC Open Sound Control Velocity (MIDI) Representation of how forcefully, or loud, a note is played iii Contents 1 Introduction 1 1.1 Purpose . 1 1.2 Question . 1 1.3 Project limitations . 2 2 Background 3 2.1 Historical background . 3 2.2 Mapping physical parameters to auditory parameters . 5 2.3 Similar studies . 9 2.4 Description of development tools . 10 3 Method 11 3.1 Equipment and experimental setup . 11 3.2 System development . 12 3.3 User tests . 13 3.4 Interview procedure . 17 4 Results 18 4.1 Mapping 1: Size-Pitch . 18 4.2 Mapping 2: Location-Spatialisation . 20 4.3 Mapping 3: Orientation-Tempo . 22 4.4 Mapping 4: Motion-Loudness . 24 4.5 Mapping 5: Distance-Duration . 26 4.6 Overall user experience . 27 5 Discussion 28 5.1 System evaluation . 28 5.2 Possible improvements . 28 5.3 Future studies . 29 6 Conclusion 31 References 32 Appendix 35 iv 1 Introduction There has been done a great amount of research and development in using contem- porary digital technology to aid human musical expressions, and it is still going on to this day as made evident by NIME conferences (NIME, 2016). One way is the mapping of physical parameters to auditory parameters within the context of music making. Physical parameters in this context are features derived from the movement of the human body, or separate body parts or limbs, as well as location based features. All of these are connected to the way humans interact or express themselves, whether it be in a group or alone. These kinds of body movements can be mapped within a system to a sound representation that corresponds to the action taken. Auditory parameters include the ways music, or audio in general, can be altered. Some examples would be pitch alteration, amplitude change (loudness), the duration of single or several segments of audio, or the pace of a musical piece (tempo). 1.1 Purpose The subject of the thesis is about mapping five physical parameters to five auditory parameters, based on common parameters derived from a meta study by Dubus & Bresin (2013). This was done by translating human body movement to physical parameters and mapping these to auditory parameters, that in turn was used to alter MIDI files. The purpose is to determine the performance of each mapping utilising user tests and interviews. 1.2 Question Given five mappings based on the meta study by Dubus & Bresin (2013), how well can a user adapt to any of these mappings in terms of both measured numerical data as well as user experience? Sub-questions: 1. Can the user generated data show a measurable increase in adaptation as the experiment progresses? 2. How do the users perceive their own skill for each of the mappings? 1 The outcome of the experiments will be an evaluation of the physical user movement parameters, the auditory parameters, and their respective mappings. The hypothesis will be tested by determining the functionality of a system used by a person that has prior music experience but does not hold any former knowledge of the system. 1.3 Project limitations Correctly determining all possible parameter variables and mappings will not be cov- ered by the thesis, but will instead give an initial evaluation of implementing physical to auditory parameter mappings. The thesis will focus on the mapping of five physical parameters to five auditory parameters, as it would not be feasible to expand into the vast amount of possible parameter mappings described by Dubus & Bresin (2013). These parameters as well as the mappings are listed below. Physical parameters Auditory parameters Distance Duration Location Loudness Motion Pitch Orientation Spatialisation Size Tempo Mappings, labelled M1-M5. • M1: Size-Pitch • M2: Location-Spatialisation • M3: Orientation-Tempo • M4: Motion-Loudness • M5: Distance-Duration 2 2 Background 2.1 Historical background Humans have always had a way to express themselves by the use of music and physical body movement. The different ways of expression ranges from social gatherings and artistic creations to war dances and religious rituals, as well as several other fields of application. 2.1.1 History of embodied musical practice One of the simplest human musical expressions are using the voice to sing, scream, whisper, and similar. According to a study by Arensburg et al (1989), a hyoid bone that dates back around 60,000 years have been discovered in Israel. This bone suggests that Neanderthals would have the same physiological voice capabilities as modern day humans, as the bone has stayed relatively unchanged since. Musical instruments can be traced back tens of thousands of years, as a flute crafted from a bear bone was found in 1996 that possible is around 43,000 years old. Al- though this has been disputed, as there are possible predator bite marks that could have created the holes of the flute and thereby rendering the crafting of the flute unintentional (Bower, 1998). It is also discussed by Conard et al (2009) that flutes discovered in Germany dates back approximately 40,000 years. The usage of human body movement in combination with music is ever present in society. It ranges from everything from rave clubs to ballroom dancing, where move- ment of the body is used to further the appreciation of musical pieces. There is however other usages for dance and similar bodily expressions. In native American culture, war dances was used to increase the morale before an upcoming confronta- tion (Wilson et al, 1988), as these exhibitions would acknowledge the warriors place within the tribe and define a sense of purpose. In the 1970s, a system named Selspot was developed to aid in the study of human motion (Welch & Foxlin, 2002). Human body movement was tracked with the use of several position sensing detectors in the form of analog optical sensors, where the user would also wear a number of light-emitting diodes. In the year 1984, Michel Waisvisz gave a concert with a computer music controller called "The Hands" (Waisvisz, 1999). This controller used sensors attached to the hands of the user, which could manipulate music by translating different physical 3 parameters (distance between hands, bowing gestures, etc.) to input data for MIDI data sequences (Krefeld & Waisvisz, 1990). 2.1.2 Sonification In Hermann & Hunt (2011), sonification is defined as the concept of delivering infor- mation using non-speech audio. The oldest known usage of sonification can be traced back to 3,500 B.C.E. in the cradle of civilization, Mesopotamia, where auditors (from the latin work auditus, meaning \I hear") could compare different reports of in-, and outbound goods, as well as those currently in stock, to determine any deficiencies (Worrall, 2009).