Soma: live performance where congruent musical, visual, and proprioceptive stimuli fuse to form a combined aesthetic narrative Ilias Bergstrom UCL A thesis submitted for the degree of Doctor of Philosophy 2010 1 I, Ilias Bergstrom, confirm that the work presented in this thesis is my own. Where information has been derived from other sources, I confirm that this has been indicated in the thesis. 2 Abstract Artists and scientists have long had an interest in the relationship between music and visual art. Today, many occupy themselves with correlated animation and music, called ‗visual music‘. Established tools and paradigms for performing live visual music however, have several limitations: Virtually no user interface exists, with an expressivity comparable to live musical performance. Mappings between music and visuals are typically reduced to the music‘s beat and amplitude being statically associated to the visuals, disallowing close audiovisual congruence, tension and release, and suspended expectation in narratives. Collaborative performance, common in other live art, is mostly absent due to technical limitations. Preparing or improvising performances is complicated, often requiring software development. This thesis addresses these, through a transdisciplinary integration of findings from several research areas, detailing the resulting ideas, and their implementation in a novel system: Musical instruments are used as the primary control data source, accurately encoding all musical gestures of each performer. The advanced embodied knowledge musicians have of their instruments, allows increased expressivity, the full control data bandwidth allows high mapping complexity, while musicians‘ collaborative performance familiarity may translate to visual music performance. The conduct of Mutable Mapping, gradually creating, destroying and altering mappings, may allow for a narrative in mapping during performance. The art form of Soma, in which correlated auditory, visual and proprioceptive stimulus form a combined narrative, builds on knowledge that performers and audiences are more engaged in performance requiring advanced motor knowledge, and when congruent percepts across modalities coincide. Preparing and improvising is simplified, through re-adapting the Processing programming language for artists to behave as a plug-in API, thus encapsulating complexity in modules, which may be dynamically layered during performance. 3 Design research methodology is employed during development and evaluation, while introducing the additional viewpoint of ethnography during evaluation, engaging musicians, audience and visuals performers. 4 Acknowledgments Firstly, I wish to thank my supervisors, Beau Lotto and Anthony Steed, for taking on my project, and for helping me through with the work presented in this thesis. I am further grateful to have been surrounded by very interesting people at the Lottolab. Without Lefkothea Andreou, Daniel Hulme, David Malkin, David Corney, Erwan LeMartelot, Martina Wicklein, Rich Clarke, Madi Boyd, Emma Byrne, Sam Walker, and Katherine Bash, the time spent working in the lab would have been far less stimulating. Many thanks are due to Nadia Berthouze, Nick Bryan-Kinns, Alan Blackwell, Marcelo Wanderley, Katherine Stevens, Nigel Jenkins, Olivier Ruellet, Mauritius Seeger, Mat Hourteillan, Alison Hardcastle, Anne-Marie Preston, Illi Adatto, Dor Wand, the members of Accretion and of the Gathering, for their invaluable advice and assistance towards completing the work presented here. I owe my gratitude to my family, without the support of which I would never have had the chance of writing these lines. And last but not least, Giulia! 5 Table of Contents Soma: live performance where congruent musical, visual, and proprioceptive stimuli fuse to form a combined aesthetic narrative .........................................................................................1 Ilias Bergstrom ............................................................................................................................1 Abstract ..............................................................................................................................3 Acknowledgments ..............................................................................................................5 Table of Contents ...............................................................................................................6 Table of Figures ............................................................................................................... 11 1. Introduction ..................................................................................................................... 15 1.1 Limitations with current practice ............................................................................ 17 1.2 Proposed system ..................................................................................................... 21 1.2.1 Increasing expressivity ..................................................................................... 22 1.2.2 Simplifying preparation for performance ........................................................ 23 1.2.3 Increasing achievable mapping complexity and variability ............................. 24 1.2.4 Allowing for collaborative performance .......................................................... 25 1.2.5 Soma: a new artistic practice .......................................................................... 25 1.2.6 A new performer role ...................................................................................... 27 1.3 Methodology ........................................................................................................... 29 1.4 Summary of contributions, and how each is evaluated .......................................... 33 1.5 Thesis structure ....................................................................................................... 36 2 Background ...................................................................................................................... 39 2.1 Relating visual arts to music .................................................................................... 39 2.1.1 Visual Music, Intermedia, Abstract film, and Synesthetic Art ......................... 39 2.1.2 Colour Organs and Lumia ................................................................................ 43 2.1.3 Audiovisual Composition ................................................................................. 49 2.2 Musical Instruments ................................................................................................ 54 2.2.1 Electronic musical instruments and the interface using which these are played 55 6 2.2.2 Analysis of Digital Musical Instruments ........................................................... 59 2.2.3 Mapping ........................................................................................................... 61 2.2.4 Embodied / Enactive knowledge, and music virtuosity................................... 63 2.2.5 Collaborative performance, mutual engagement and group flow.................. 64 2.2.6 The mixing engineer ........................................................................................ 65 2.2.7 Performance using Conducting Gesture DMIs ................................................ 66 2.3 Musical Narrative .................................................................................................... 69 2.3.1 Absolute music ................................................................................................ 69 2.3.2 Narrative in visual music and audiovisual art .................................................. 71 2.4 Programming as art: Creative Code ........................................................................ 73 2.4.1 New Media art ................................................................................................. 74 2.4.2 Demo Scene ..................................................................................................... 75 2.4.3 Live Coding ....................................................................................................... 77 2.5 Programming for non-computer scientists ............................................................. 78 2.5.1 Visual programming languages ....................................................................... 79 2.5.2 Procedural languages, and the Processing language ...................................... 81 2.6 Review of related research and evaluation methodology ...................................... 85 2.6.1 Evaluation of Music Technologies ................................................................... 86 2.6.2 Human Computer Interaction ......................................................................... 89 2.6.3 Review of evaluation methodology employed in recent PhD theses analogous to present work ............................................................................................................. 105 2.7 Recent related work .............................................................................................. 108 3 Mutable Mapping .......................................................................................................... 112 3.1 Related work to mutable mapping ........................................................................ 114 3.2 The mutable mapping performer .........................................................................