Human Bodily Micromotion in Music Perception and Interaction Application to FRIPRO/IKTPLUSS Alexander Refsum Jensenius, University of Oslo 1 Relevance relative to the call for proposals This project seeks to investigate the close couplings between musical sound and human bodily micromotion. Micromotion is here used to describe the smallest displacements of human body parts, either voluntary or involuntary, typically at a speed smaller than 10 mm/s. The last decades have seen an increased focus on the role of the human body in both the performance and the perception of music (Gritten and King, 2006), and today it is difficult to imagine talking about the experience of music without also focusing on how music is experienced through the body (Godoy and Leman, 2010). Up to now, however, the micro-level of these experiences has received little attention. This project will investigate such music-related micromotion, with an aim of contributing to: • knowledge about how musical sound influences human motion at the micro-level. This will be based on literature studies, theoretical modelling, and a longitudinal observational study as well as three large-scale experiments of sound-motion relationships. • a large, annotated and metadata-rich database of the micromotion recordings mentioned above. The database will be central to the current project, and will also be made available for future research in the field. • conceptual models and software tools for using micromotion to control musical sound in computer-based systems. Such musical microinteraction can be used for music perfor- mance or production, or for \active listening". The project is: • at the research frontier of basic issues in cognitive musicology, embodied music cognition, music technology, and human-computer interaction. • truly interdisciplinary, bridging over between so-called \hard" and \soft" research ap- proaches, as well as between science and the arts. • innovative, with research outputs in the form of scientific publications, an open database, and open source software tools. I am already established as a leading, young researcher within the international music technology and music cognition communities, and this project will give me the possibility to set up my own research group and carry out a bold and novel project. Results from the project will feed directly into current trends in musicology and in music psychology/cognition, where more knowledge about the human body is of great importance for the further understanding of how humans produce, perform and perceive music. The conceptual models and software tools developed in the project may also have a wider impact. For example, my previous research on (more large-scale) music-related body motion has resulted in the development of software-tools for detecting cerebral palsy (CP) in young infants (Adde et al., 2010). If succesful, this project's ambition of using micromotion to control interactive systems may have a transformational impact on how we use tomorrow's technologies. 1 2 Aspects relating to the research project 2.1 Background and status of knowledge Try to sit still for a couple of minutes. What did you observe? Did you notice the small motion happening in various parts of your body? Did you notice the rhythmic patterns of your breathing, pulse and postural adjustments? Most such microactions are produced unin- tentionally and are so small that they are not easily observed by others. But the \invisibility" of such microactions does not mean that they do not affect us and those around us. I believe that nuance and expressivity in music to a great extent are conveyed and experienced through micromotion. The underlying premise for this project is built on the idea that bodily motion is integral to both the performance and the perception of music. While this may seem obvious, it is only in the last decades that music researchers have started to study music from an embodied perspective (Leman, 2008), and there are still many unanswered research questions. My own contributions to the field include a series of observational studies of people's spontaneous motion to music, including that of \air piano" performance, \sound tracing" and \free dance" to music (Jensenius, 2008). Based on these studies I have been able to present a more thorough classification scheme of music-related motion, including the four main types: sound-producing, sound-facilitating, sound-accompanying, and communicative (Jensenius et al., 2010). Alongside this theoretical development, I have also contributed with a set of video-based analysis methods and tools for studying music-related body motion systematically (Jensenius, 2013). Towards micromotion Having studied music-related motion for ten years, it has become clear to me that most research projects in the field are generally limited to actions that are large and intentional. Music performance is rife with examples of intentional, micro-level interaction, such as the minute actions found in the mouth of wind performers, or in the fingering of string players. But such micromotion has mainly been studied from a sound-producing and acoustic perspective rather than a perceptual one. Even studies of performer's unintentional motion, such as the \ancillary" motion of clarinetists (Wanderley et al., 2005), have primarily focused on medium to large-scale motion, rather than the micro-level. This is puzzling, given that it is well documented that the \invisibility" of micromotion is at the core of how we perceive others (Ekman and Friesen, 1969). So what if we dispense with all motion larger than, say, 10 mm/s? What are we dealing with then? Exploring standstill In 2012 I carried out a pilot study on music-related micromotion to- gether with dancer-choreographer Kari Anne Vadstensvik Bjerkestrand, in which we explored the act of standing still in silence for ten minutes at a time, what is sometimes referred to as a \human pendulum" (Collins and De Luca, 1994). The experience of standing consciously still for such a long time is overwhelming. Not only does one's own micromotion become apparent, such as swaying, shifting of weight, breathing and heart beats, but it is also easy to see and relate to other people's spontaneous micromotion (Jensenius and Bjerkestrand, 2012). Based on the findings in the pilot study, I continued the exploration of micromotion in the scientific-artistic research project Sverm,1 where two musicians, two dancer-choreographers and one scenographer were recruited. Here we employed the same observation strategy as tested earlier, standing still together on the floor for 10 minutes at a time (Figure1). Using a state-of-the-art motion capture system to quantify our motion, we found an average quantity of motion (QoM) of around 6.5 mm/s for a head marker of a person standing still (Jensenius et al., 2014). We also found that the QoM values remained remarkably linear over time for 1http://www.fourms.uio.no/projects/sverm/ 2 each person for each recording, and that they were also very consistent for each person across recording sessions. More careful analysis of the data revealed clear person-specific patterns in the data sets, such as seen in Figure2. Here we could identify (at least) three levels of periodic mo- tion: (a) quasi-random motion happening on a millisecond scale, probably caused by the ankles working to keep the body in balance (Loram and Lakie, 2002), (b) periodic mo- tion at intervals of approximately five to ten seconds corresponding to our respiratory pat- terns, (c) \spikes" every two to three min- utes that were presumably explained by pos- tural adjustments. All in all, the patterns that emerged from the recordings were not as Figure 1: Picture from a standstill session in random as we expected, but rather were indi- the Sverm project. Reflective motion capture vidually systematic to such an extent that we markers can be seen on the heads of each person. could identify a person through the plots. 5 mm -5 0 2 4 6 8 10 time (min) Figure 2: Normalised vertical position of the head marker of a person standing still. Towards music-related micromotion It is well known that musical sound can induce both motion and emotion in humans (Leman, 2008). One example is that of rhythmic entrainment, meaning that two or more independent rhythmic processes may synchronise with each other (Pikovsky et al., 2003). Such entrainment has been shown to exist at multiple levels in music performance (Clayton, 2005), most likely based on the experience of both the individual pulses and the metrical structure in the rhythm (Large, 2000). Similarly, studies of human walking have shown a preference for a frequency of two steps per second (2 Hz), independent of gender, age, height, weight, or body mass index (MacDougall, 2005). These findings coincide well with studies on the perception of musical rhythm, indicating a clear preference for musical tempi of two beats per second (120{125 bpm) (Moelants, 2002), and that the preferred tempo of dance music is a little faster (125{130 bpm) (Moelants, 2008). These studies, however, are based on voluntary and large-scale motion. Would we be able to find evidence of similar types of periodicities or entrainment in micromotion recordings of people standing still? To test this, I have conducted two informal experiments, each camouflaged under the head- ing \Norwegian Championship of Standstill," so as to recruit a large amount of people. For each of the experiments around 100 people stood still for six minutes, three minutes in silence and three minutes with music. Since these were only informal experiments we have not pub- lished the results, but the preliminary analysis does support, to some extent, the hypothesis that our involuntary micromotion will synchronise with musical features. However, controlled, full-scale experiments are needed to properly validate these findings, and to find out which musical features influence different types of micromotion.