Copyright by Yago Parra Moutinho Stucky de Quay 2016 The Dissertation Committee for Yago Parra Moutinho Stucky de Quay Certifies that this is the approved version of the following dissertation: The Development and Experience of Gesture and Brainwave Interaction in Audiovisual Performances Committee: Sharon Strover, Supervisor Bruce Pennycook, Co-Supervisor Shanti Kumar Russell Pinkston Yacov Sharir Rui Penha António Coelho The Development and Experience of Gesture and Brainwave Interaction in Audiovisual Performances by Yago Parra Moutinho Stucky de Quay, B.Music.; M.Multimedia Dissertation Presented to the Faculty of the Graduate School of The University of Texas at Austin in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy The University of Texas at Austin December 2016 Dedication To my parents, Isabel and José Acknowledgements I want to thank Bruce Pennycook, Yacov Sharir, and Sharon Strover for their significant intellectual and artistic contributions to this work. I also want to thank A. R. Rahman for inviting me to join his fantastic project with Intel, and Lakshman Krishnamurthy for his inspiring leadership in that project. Finally, I am grateful to have had Reema Bounajem supporting me emotionally throughout the writing process. v The Development and Experience of Gesture and Brainwave Interaction in Audiovisual Performances Yago Parra Moutinho Stucky de Quay, PhD The University of Texas at Austin, 2016 Supervisor: Sharon Strover Co-Supervisor: Bruce Pennycook Software and hardware developments in the 21st century have greatly expanded the realm of possibilities for interactive media. These developments have fueled an increased interest in using unconventional methods to translate a performer’s intentions to music and visuals. And at the heart of every computer mediated performance is the question of what human action causes what audiovisual effect. This research explores this question and others by examining two performances that I created; performances featuring audiovisual media that respond to a performer’s gestures and brainwaves. The aim is to improve our understanding of what factors influence interaction design within and across the performing arts. I argue that technology, collaborators, and transparency have a decisive impact on a performer’s actions and interactive media. Furthermore, my findings suggest that synchronicity, elegance and visual feedback are important characteristics that can enhance the bond between the performer and audiovisual content. Lastly, I explain how universities and companies can improve research and development by together creating a live show. vi Table of Contents CHAPTER 1: INTRODUCTION .............................................................................................. 1 Background .................................................................................................................. 1 Methods ........................................................................................................................ 2 Data collection .............................................................................................................. 6 Dissertation outline ...................................................................................................... 7 CHAPTER 2: LITERATURE REVIEW ................................................................................... 10 Embodied music cognition ......................................................................................... 11 Audiovisual binding ................................................................................................... 14 Mediatization on stage ............................................................................................... 18 Interaction transparency ............................................................................................. 20 Interim summary ........................................................................................................ 22 Brain-computer interfaces .......................................................................................... 23 Extracting relevant neural signals .............................................................................. 28 Interim summary ........................................................................................................ 31 Examples of interactive performances ....................................................................... 32 Brainwaves .............................................................................................................. 33 Motion tracking ....................................................................................................... 37 Biosignals ................................................................................................................ 39 CHAPTER 3: CURIE .......................................................................................................... 41 Context ....................................................................................................................... 44 Origin ...................................................................................................................... 44 Media technology .................................................................................................... 47 Sensors .................................................................................................................... 52 Development .............................................................................................................. 56 Musician wristbands ............................................................................................... 57 Gesture algorithm .................................................................................................... 60 Optical motion tracking .......................................................................................... 65 Radio motion tracking ............................................................................................. 67 Finger tracking glove .............................................................................................. 71 Audience wristbands ............................................................................................... 72 Software .................................................................................................................. 76 Mappings................................................................................................................. 84 Mapping development ............................................................................................ 89 Visuals..................................................................................................................... 98 Choreography ........................................................................................................ 101 Music..................................................................................................................... 104 Analysis .................................................................................................................... 106 Technology ........................................................................................................... 108 Transparency ......................................................................................................... 114 Collaboration ......................................................................................................... 122 CHAPTER 4: AD MORTUOS .............................................................................................. 133 Context ..................................................................................................................... 136 vii Origin .................................................................................................................... 136 Media technology .................................................................................................. 138 Development ............................................................................................................ 139 Eeg headset ........................................................................................................... 140 Tablet computer .................................................................................................... 143 Led lights .............................................................................................................. 145 Motion tracking ..................................................................................................... 146 Position tracking ................................................................................................... 147 Practice software ................................................................................................... 148 Mappings............................................................................................................... 150 Mapping development .......................................................................................... 150 Visuals................................................................................................................... 153 Choreography ........................................................................................................ 155 Music..................................................................................................................... 157