JYVÄSKYLÄ STUDIES IN HUMANITIES 176 Marc Thompson The Application of Motion Capture to Embodied Music Cognition Research JYVÄSKYLÄ STUDIES IN HUMANITIES 176 Marc Thompson The Application of Motion Capture to Embodied Music Cognition Research Esitetään Jyväskylän yliopiston humanistisen tiedekunnan suostumuksella julkisesti tarkastettavaksi yliopiston vanhassa juhlasalissa maaliskuun 21. päivänä 2012 kello 12. Academic dissertation to be publicly discussed, by permission of the Faculty of Humanities of the University of Jyväskylä, in building Seminarium, auditorium S212, on March 21, 2012 at 12 o'clock noon. UNIVERSITY OF JYVÄSKYLÄ JYVÄSKYLÄ 2012 The Application of Motion Capture to Embodied Music Cognition Research JYVÄSKYLÄ STUDIES IN HUMANITIES 176 Marc Thompson The Application of Motion Capture to Embodied Music Cognition Research UNIVERSITY OF JYVÄSKYLÄ JYVÄSKYLÄ 2012 Editors Petri Toiviainen Department of Music, University of Jyväskylä Pekka Olsbo Publishing Unit, University Library of Jyväskylä Jyväskylä Studies in Humanities Editorial Board Editor in Chief Heikki Hanka, Department of Art and Culture Studies, University of Jyväskylä Petri Karonen, Department of History and Ethnology, University of Jyväskylä Paula Kalaja, Department of Languages, University of Jyväskylä Petri Toiviainen, Department of Music, University of Jyväskylä Tarja Nikula, Centre for Applied Language Studies, University of Jyväskylä Raimo Salokangas, Department of Communication, University of Jyväskylä Cover picture by Veera Aivio URN:ISBN:978-951-39-4690-6 ISBN 978-951-39-4690-6 (PDF) ISBN 978-951-39-4689-0 (nid.) ISSN 1459-4331 Copyright © 2012 , by University of Jyväskylä Jyväskylä University Printing House, Jyväskylä 2012 ABSTRACT Thompson, Marc R. The application of motion capture to embodied music cognition research Jyväskylä: University of Jyväskylä, 2012, 86 p. (Jyväskylä Studies in Humanities ISSN 1459-4331; 176) ISBN 978-951-39-4689-0 (nid.) ISBN 978-951-39-4690-6 (PDF) Music and expressive body movements are inseparable. Musicians use movements to convey emotional expressivity, conductors use gestures to convey musical structure, and co-performers rely on corporeality to maintain group cohesion. Additionally, the spontaneously-produced movements that occur when listening to music appear to naturally synchronize with the musical pulse. Despite this tendency common among most people, movement patterns appear to be highly individualistic and perhaps related to expressive cues inherent in the music. A growing awareness of the significant role that expressive body movement plays in music-related activities has given rise to the application of embodied cognition to music. Embodied cognition is the view that knowledge is attained through sensorimotor representations of the world and that the human body acts as the main mediator between our thoughts and the environment. Music-related activities such as performance and dance are excellent examples embodied cognition, as they require online tracking of perceptual stimuli that is in turn being augmented by our own actions. Furthermore, the embodied approach to music assumes that we can interact with music at different levels of engagement ranging from synchronization with low-level features such as the beat to empathizing with the music’s higher- level features such as expressive intentions. In both cases, musical meaning is developed through goal-directed corporeal actions, which both react and contribute to musical experiences. The study of such phenomena has greatly benefited from advances in optical motion capture systems, which can track movements with a high degree of precision. For analysing the movements, semantically significant features are then extracted from the raw motion capture data. These variables can describe the kinematics, kinetic or other behavioural features of musicians and dancers, and be subjected to an infinite variety of analysis methods. The aim of this thesis is to demonstrate, through experimental research, some applications of motion capture within the area of embodied music cognition. Across six independent studies, four music-related activities were analysed: piano performance, conducting gestures, choreographed choral performance and music-induced movements. Keywords: embodied music cognition, motion capture, gestures, music-related movements Author’s address: Marc R. Thompson Department of Music PL 35 (M) 40014, University of Jyväskylä Finland [email protected] Supervisor: Dr. Geoff Luck Finnish CoE in Interdisciplinary Music Research University of Jyväskylä Reviewers: Professor Rolf Inge Godøy Department of Musicology University of Oslo Dr. Peter Keller MARCS Auditory Laboratories University of Western Sydney Opponent: W/Professor Jane W. Davidson School of Music The University of Western Australia AKNOWLEDGEMENTS The real voyage of discovery consists not in seeking new landscapes but in having new eyes. —Marcel Proust (1871-1922) I move on to another day, to a whole new town with a whole new way. —The World at Large by Modest Mouse At a crossroads such as this, it is opportune to reflect on and show gratitude to the talented people who have generously devoted their time, knowledge and tutelage to me throughout these last four years. First, I wish to thank Dr. Geoff Luck, who has supervised both my Masters and Doctoral studies. I am grateful for his mentorship, partnership and friend- ship throughout every stage of this project, and for assisting me in converting my unpinned ideas into clear and achievable goals. To Prof. Petri Toiviainen, I am grateful for his continued support and guidance since September 2005. I am appreciative to Petri for sharing his vast knowledge on a daily basis on subjects as varied as linear algebra, empirical mode decomposition, Wii-controller hacking and Finnish mushrooms, to name a few. His intelligence, kindness and modesty embody the qualities that all great academics ought to have. At McGill University, I thank Marcelo M. Wanderley, whose guidance on my bachelor’s thesis some years ago, which focused on musical gestures and the role they play in music performance, influenced me to study music cogni- tion in the first place. I am very glad to have had the opportunity to learn from him again during my six-month sabbatical at IDMIL in 2010. From an administrative point of view, working at the Music Department was made easy with the great help of Markku Pöyhönen. Markku was a con- stant informant of upcoming events, tasks and responsibilities. I believe he is the key reason the Department is as successful as it is. To Tommi Himberg, I am thankful for his invitation to join several excit- ing projects of which I am proud to have been a part of, and for the numerous conference proceedings and articles we have written together. To my fellow doctoral students, Rafael Flores Ferrer, Vinoo Alluri, Jonna Vuoskoski, Birgitta Burger, Anemone Van Zijl, Pasi Saari and Marko Punkanen, thank you all for the enlightening conversations, laughter, endless lunch hours and the vast array of sweet and salty snacks we have shared over the years. I also wish to thank Suvi Saarikallio, Oliver Lartillot, Prof. Tuomas Eerola, Prof. Jaakko Erkkilä, Esa Ala-Ruona, Jörg Fachner, Jukka Louhivuori, Pekka Toivanen, Hannes Juutilainen, Mikko Myllykoski, and Mikko Leimu who have all in their unique ways played a role in helping me achieve success at this de- partment. This summary greatly benefitted from having been reviewed by two emi- nent scholars. I thank Peter Keller and Rolf Inge Godøy for their insightful comments and support of this thesis and its constituent articles. In addition, I thank Alex Reed for help in proofreading this summary. Though away from my homeland, I never feel without family. I thank Virpi, Mikko and Lauri who have given me so much since my first months in Jyväskylä. I also thank my in-laws, Tarja, Antero, Janne, Veera and Samppa for their love, kindness and acceptance. In Canada, I thank my parents, Christopher and Nicole, and my siblings, Philippe, Charles and Amanda and their families, for supporting me in which- ever strange and unlikely avenues my journey has taken me. Most of all I am thankful to my wife Noora and our beautiful daughter Elsa, to whom I dedicate this work. Their love, support and encouragement— past, present and future—continue to be my guiding light. Jyväskylä, November 10th, 2011 Marc R. Thompson LIST OF ORIGINAL PUBLICATIONS I. Thompson, M. R., & Luck, G. (2008). Effect of pianists' expressive in- tention on amount and type of body movement. In Miyazaki, K., Hi- raga, Y., Adachi, M., Nakajima, Y. & Tsuzaki, M. (Eds.), Proceedings of the 10th International Conference on Music Perception and Cognition, (ICMPC10) (pp. 540-544). Sapporo, Japan: Hokkaido University. II. Luck, G., Toiviainen, P. & Thompson, M.R. (2010. Perception of ex- pression in conductors’ gestures: A continuous response study. Music Perception 28(1), 47-57. III. Toiviainen P., Luck, G. & Thompson, M.R. (2010). Embodied meter: Hierarchical eigenmodes in music-induced movement. Music Percep- tion 28(1), 59-70. IV. Luck, G., Saarikallio, S., Burger, B., Thompson, M.R. & Toiviainen, P.(2010). Effects of the Big Five and musical genre on music-induced movement. Journal of Research in Personality 44(6), 714-720. V. Himberg, T. & Thompson, M. (2011). Learning and synchronizing dance movements in South African songs—cross-cultural motion- capture study. Dance Research vol. 29(2), 305-328. VI. Thompson, M. R., & Luck, G. (2011). Exploring relationships