Institut für Medien PhD Studium The Interface-Score Electronic Musical Interface Design as Embodiment of Performance and Composition Enrique Tomás DOCTORAL THESIS to achieve the academic grade DOCTOR OF PHILOSOPHY Supervisors: Univ. Prof. Dr. Ing. Martin Kaltenbrunner and Univ.Prof. Dr.phil. Gerhard Eckel Approved: Linz 2018 Abstract This thesis investigates the artistic and philosophical consequences of understanding musical instruments as artifacts embodying a score. In other words, it explores the extent to which musical instruments can be perceived as musical scores. This research examines the cultural and technological nature of digital musical instruments and it contributes to a material account of musical notation. It introduces a novel paradigm for tangible interface design called the tangible score. It also describes a practice-based research journey performing tangible musical interfaces. This is a thesis in art. Opposed to a thesis on art, such as art history, my research did not have any pre- scribed methodology. In fact, my artistic production during this PhD may be considered both object and method. As an artist working with technology it was mandatory contextualizing my personal artistic meth- ods. In consequence, this thesis gives answers to the question of how artists can engage artistic research with the field of human-computer interaction (HCI). I defend that the way artists can help HCI is adopting a criti- cal attitude with our research medium -namely tangible and musical interaction design in this PhD- avoiding the instrumentalization of our artistic processes and adopting the format of artistic research. For drawing up this theory, this thesis analyzes various examples of critical interfaces. In order to capture the interest of other researchers within HCI, I present practical methods to formalize the structure of an artistic research project. The notion of the inherent-score drives the beginning of my research on musical notation. I define it as the material and virtual elements of the instrument shaping and inspiring a player’s performance. This notion gives us language to describe how the instrument mediates embodiment at the exact moment of per- formance. This is coherent with many improvisers’ intuition affirming that the instrument is the score of what they play. I propose that it is possible to design new interfaces for musical expression emphasizing their in- herent score. For exemplifying this option, I have built and performed a series of digital musical instruments called Tangible Scores. They are the center of my artistic practice during this doctorate. This thesis contributes to a vision of digital musical instruments as symbolic machines. In my opinion, the principal characteristic of digital music instruments is their symbolic apparatus. The apparatus always features a program defining all possible musical realizations. For dealing with the complexities of the appa- ratus, designers have to deal with representations. They are helpful to inform performers about the status of the machine while they also serve for constraining the possible interactions with the program. For this reason, the representational dimension of musical interfaces has been addressed in this thesis. I present a vi- sion about musical interfaces as a collection of related texts and symbols with a given arbitrary meaning. As I show in this thesis, they radically mediate our embodiment with the instrument. For this reason, I contribute towards a non-linguistic and non-representational approach to performing with musical interfaces. Materiality is another important field of study in this thesis. I contribute to a materialistic account of mu- sical interfaces defending that matter must not be only understood as the mere support for an interface’s functionality. My thesis asserts that matter is an active agent able to inspire, provoke and shape perfor- mances. Therefore, my research aims at developing a new materialist sensibility at tangible and musical interaction design. Finally, this thesis addresses the cultural and political dimensions of musical interface design. I defend that the cultural always reconfigures and redefines the technical in musical interface design. Politically, mu- sical interfaces are never transparent technologies. A full range of cultural and political values are scripted into our musical interfaces. They are good examples of cultural interfaces. In conclusion, this is a thesis dedicated to the study of the issues of musical interface design from an artistic standpoint. The field of study is highly interdisciplinary and cultural studies, technology and design -among many other disciplines- merged at my practice. For this reason, the background of this thesis is art, music, engineering and philosophy. iii Acknowledgements I want to thank many people.. First of all, my supervisors Martin Kaltenbrunner and Gerhard Eckel, for their dedication in guiding this thesis. Many aspects of this text have been inspired and shaped by their insights, comments and experience. Nothing comes out of nowhere. This thesis would not be possible without the support of the Kunstuniversität Linz and many of its aca- demic staff, Christa Sommerer, Laurent Mignonneau, Gertrude Hörlesberger, Fabricio Lamoncha, Michaela Ortner, André Zogholy, Veronika Schwediauer, Reinhard Kannonier, Rainer Zendron... Four years of this the- sis have been funded by scholarships of the Kunstuniversität Linz. During six months I received a scholarship from the Akademie der bildenden Künste Wien. I also have to thank my colleagues and friends César Escudero Andaluz, Alex de las Heras, Martín Nadal, Reinhard Gupfinger, Jens Vetter... and five generations of students at the Interface Culture department. Thanks to Kevan Croton this text has been proofread. I have received feedback and comments from Thor Magnusson, Nicolas Collins (the American), Norbert Schnell, Damian Schwartz, Volkmar Klien, Michael Maierhof, Frederic Bevilacqua, Pavlos Antoniadis, Rupert Huber, Ana Morales, Atau Tanaka, Laura Benítez Valero, Hiroshi Ishii, Caroline Hummels, Jonathan Impett, Sally-Jane Norman, Günther Gessert, Fadi Dorninger, Andi Kurz, Thomas Gorbach, the intra-sonic duo, Alice Eldridge, Chris Kiefer, Peter Kirn, Alberto Bernal, Joao Pais, Eckart Waage, Pablo Martin Jones, Clemens Pich- ler and many other anonymous people at workshops, exhibitions and concerts who helped me to understand better what I was doing. Thank you very much to the academic staff of the Anton Bruckner Privat And gracias a mi familia, Ursula und Elsa, Enrique y Isabel, Helmut und Gudrun, and all the others in Madrid and in Hallein. You supported me so much! Thank you very much! Enrique Tomás Linz, April 2018 v Contents 1 Introduction 1 1.1 Prelude.............................................. 1 1.2 Research Field . 2 1.3 Establishing the research questions . 4 1.4 Research Methods: Art and Design Methods . 5 1.4.1 Practice-Based Research Methods in the Arts . 5 1.4.2 User-centered Design Methods . 8 1.4.3 Collaborative Design Methods . 9 1.4.4 Critical Design Methods . 9 1.5 Research Contributions of this PhD . 10 1.6 Outcomes . 11 1.7 Structure of this Thesis . 13 2 Methodologies: How Can THE Arts Help Human Computer Interaction? 15 2.1 Introduction: Tangible Interaction and the Arts . 15 2.2 Interfaces: Definitions . 16 2.3 Critical Interfaces: Interface Aesthetics . 17 2.3.1 Interfaces as Aesthetic Artifacts . 17 2.3.2 Critical Interfaces . 18 2.3.3 Traditional visions about the Arts in HCI. 20 2.4 Artistic Research and Human Computer Interaction . 20 2.4.1 The Limits of Artistic Research . 21 2.5 Examples of Critical Interfaces . 22 2.5.1 Interfight: Functional and Media Realism . 23 2.5.2 Hatching Scarf: Illusionistic Realism . 23 2.6 Discussion: How to Formalize Artistic Research Projects at HCI Labs . 24 2.6.1 Problematization of the examples . 26 2.6.2 Methodologies of the examples . 26 2.6.3 Artistic Inquiry of the examples . 27 2.7 Conclusion: How the Arts can help Human Computer Interaction . 28 2.8 Contributions of this chapter . 28 3 Musical Interfaces 29 3.1 The Compound Nature of Digital Musical Instruments . 29 3.2 Musical Interfaces as Technology . 30 3.2.1 Definitions . 30 3.2.2 The Apparatus of Digital Musical Instruments . 35 3.2.3 A Post-Hermeneutic Approach to Apparatuses. 38 3.2.4 Taxonomies, Critical Organology and Musical Organics . 39 3.2.5 Mediated Relationships Among Performer, Instrument and Sound Work . 40 3.2.6 Physical Embodiment . 42 3.3 Musical Interfaces as Cultural Interfaces . 43 3.3.1 Politics of Musical Interfaces . 45 3.3.2 Politics of Musical Interface Design . 46 3.3.3 Disenchantment: Post-Digitalism and Interface Objectification . 48 3.3.4 The Politics of the Novelty: the "World’s First" Case . 51 3.3.5 Marketing and Musical Controllers . 52 3.3.6 Euphoric Controllerism . 52 3.3.7 Politics: Conclusions. 54 3.4 Contributions of this Chapter . 55 vii viii Contents 4 Notation 57 4.1 The Instrument - Score Duality . 57 4.2 Notation Today . 57 4.2.1 Notation and Live Electronic Music . 57 4.3 What can be notated? . 58 4.4 Dance and Notation. 59 4.4.1 Choreographic Objects. 59 4.4.2 Mallarmé, Dance and Notation . 64 4.5 The Circuit as Score . 64 4.5.1 The Inherent Score: Origins of this Practice . 64 4.5.2 Composed and Self-Composed Instruments . 69 4.5.3 The Inherent Score of Acoustic Instruments . 70 4.5.4 A Material Account of Scores: Performative Materiality . 70 4.6 Formalization as Notation . 72 4.6.1 Preliminary Questions . 72 4.6.2 Notational Systems . 72 4.6.3 Inherent Scores Notational Scheme . 74 4.6.4 Non-Notational Systems and Musical Graphs . 74 4.6.5 Instrumentality of Graphic and Inherent Scores . 75 4.6.6 The Form of an Inherent Score: A Hybrid Art Form. 76 4.6.7 Inherent Scores: Typology of Symbols . 78 4.7 Preliminary Conclusions on Inherent Scores . 78 4.8 A Framework for Composing Inherent Scores .