Direct Manipulation for Musical Interfaces Based on Visual Software: Design and Evaluation Jeronimo Barbosa Department of Music Technology Input Devices and Music Interaction Laboratory McGill University Montreal, Canada July 2019 A thesis submitted to McGill University in partial fulfillment of the requirements for the degree of Doctor of Philosophy. © 2019 Jeronimo Barbosa 2019/07/09 i Abstract With the popularization of mobile computing and software-distribution platforms like the Apple Store and Google Play, a plethora of remarkable software-based musical interfaces with dynamic visual feedback are being designed today. Recent examples such as the Loopy, the Borderlands, the Ocarina illustrate that powerful visual interfaces are capable of captivating amateur and expert musicians alike by making music manipulation more direct and straightforward. However, while successful examples are available, effectively designing such direct interfaces remains challenging, typically proceeding “as more art than science”. For example, unlike acoustic instruments, sound production is physically decoupled from input controls, yielding infinite possi- bilities for mapping. Moreover, increasing ease-of-use for novices can lead to toy applications that disinterest experts. Finally, analyzing successful crafts remain unusual, and little music interface research is being carried to better understand these interfaces. This thesis attempts to understand these musical interfaces better, aiming at providing empir- ical evidence to inform their design and evaluation. Borrowing from human-computer interaction research, I hypothesize that one effective strategy concerns adopting an interaction model known as direct manipulation–characterized by continuous visual representation of the object of interest, physical actions to manipulate the visual representations, rapid visual feedback, and incremental and easily reversible actions. To investigate this hypothesis, I report exploratory case studies where direct manipulation is designed and evaluated in three specific contexts of music tools: a) live looping tools;b)tools for creating interactive artistic installations; and c) music authoring tools. As a result, I introduce three novel software music tools: Voice Reaping Machine; ZenStates; and StateSynth. Finally, I present a set of interface design strategies and evaluation approaches for effectively designing direct manipulation in musical interfaces based on visuals. ii Sommaire Grâce à la popularisation des plateformes d’informatique mobile et de distribution de logiciels, telles que l’Apple Store et Google Play, de nombreuses interfaces musicales basées sur des logiciels avec retour visuel sont en cours de conception. Des exemples récents tels que Loopy, Borderlands et Ocarina montrent que de puissantes interfaces logicielles visuelles sont capables de captiver autant les musiciens amateurs que les experts, tout en fournissant une manipulation musicale plus directe. Bien que certains exemples soient disponibles, il demeure difficile de conceptualiser de telles in- terfaces directes, puisque l’accent est porté davantage sur “l’art plutôt que la science”. Par exemple, contrairement aux instruments acoustiques, la production du son est physiquement découplée des commandes d’entrée, offrant ainsi des possibilités infinies de mappage. De plus, la facilité d’accès et d’utilisation grandissante pour les novices peut conduire à des applications ressemblant plutôt à des jouets, ce qui désintéresse les experts. Enfin, l’analyse des interfaces musicales existantes reste inhabituelle et peu de recherches sont menées afin de mieux comprendre ces interfaces. Cette thèse a pour objectif de mieux comprendre ces interfaces musicales dans le but de fournir des bases empiriques qui informeront les principes entourant leur conception et leur évaluation. Tout en me basant sur les recherches conduites dans le domaine de l’interaction personne-machine, je propose l’hypothèse qu’une stratégie efficace consiste à adopter un modèle d’interaction appelé manipulation directe (MD) caractérisé par une représentation visuelle continue de l’objet en ques- tion, des actions physiques pour manipuler les représentations visuelles, un retour visuel rapide, et des actions incrémentales facilement réversibles. Afin d’étudier cette hypothèse, je présente des études de cas exploratoires au sein desquelles la MD est conçue et évaluée selon trois contextes spécifiques reliés à des outils musicaux: a) outils de bouclage en temps réel;b)outils de création d’installations artistiques interactives;etc)outils de création musicale. En conséquence, je présente trois nouveaux outils logiciels de musique: le Voice Reaping Machine;leZenStates;etleStateSynth. Enfin, je présente un ensemble de stratégies de conception d’interface et d’approches d’évaluation permettant de conceptualiser efficacement la MD pour les interfaces musicales logicielles. iii Acknowledgements This thesis would not exist without the invaluable support of many people. I am immensely grateful to: • My advisor, Dr. Marcelo M. Wanderley, and my co-advisor, Dr. Stéphane Huot; • All music technology faculty and my comprehensive exams committee–Dr. Catherine Guas- tavino, Dr. Gary Scavone, Dr. Isabelle Cossette, and Dr. Eleanor Stubley–, for their insightful comments and early feedback; • Darryl Cameron and Yves Méthot, for their technical support with the prototypes and setting up experiments; • All collaborators–especially Dr. Sofian Audry and Dr. Chris Salter–for the opportunity and the exchange; • Current and past colleagues from IDMIL, Music Tech Area, and CIRMMT; • My friends in Canada, for the daily support and encouragement, and in Brazil, for their wisdom and endless inspiration. May many carnivals be ahead of us; • My family, Mamadi, Papadi, and Bocodi, for their love and unconditional support; • Kamilla de Souza, for the unspoken. What else to say? I dedicate this thesis to the memory of my old grandpa, Macário Costa. “O sorriso na chegada da fazenda de meus filhos e amigos, que adoro e quero bem”. iv Contribution of Authors This document is organized as a manuscript-based thesis, gathering five different papers writ- ten during my PhD studies. Overall, it presents two key contributions: 1) Three novel software music tools: the Voice Reaping Machine; the ZenStates; and the StateSynth; and 2) A set of interface design strategies and evaluation approaches for effectively designing direct manipula- tion in software-based visually-oriented musical interfaces. Following, I detail contributions per manuscript, describing my individual contributions. Chapter 2 Jeronimo Barbosa, Joseph Malloch, Stéphane Huot, and Marcelo M. Wanderley. “What does ’Evaluation’ mean for the NIME community?” In Proceedings of the 2015 International Con- ference on New Interfaces for Musical Expression, Baton Rouge, USA. This chapter presents an exploratory survey on the usage of evaluation studies within musical interface design. Here, I have carried both research and writing. Collaborators have contributed by discussing strategies for data analysis, discussing results, and reviewing early versions of the text. Chapter 3 Jeronimo Barbosa, Marcelo M. Wanderley, and Stéphane Huot. “Exploring Playfulness in NIME Design: The Case of Live Looping Tools.” In Proceedings of the 2017 International Confer- ence on New Interfaces for Musical Expression, Copenhagen, Denmark. This chapter describes a case study on using design rationale as process for creating playful- oriented NIMEs–in particular, applied to the specific context of Live looping. A second contribu- tion outcomes from this process: a visual direct manipulation-based live looping tool called the Voice Reaping Machine. Here, I have carried all the design research, technical development, and v writing. Collaborators have contributed by reviewing early versions of the design space and the guidelines, as well as providing feedback during the entire manuscript writing. Chapter 4 Jeronimo Barbosa, Sofian Audry, Christopher L. Salter, Marcelo M. Wanderley, and Stéphane Huot. “Designing Behavioral Show-Control for Artistic Responsive Environments: The Case of the ‘Qualified-Self’ Project”. This chapter contributes with a design study investigating the composition practice of artistic responsive environments, driving the development of two visual behavioral show-control systems (one of them is ZenStates, introduced on Chapter 5). A second contribution concerns lessons learned from an exploratory real-world application of the ZenStates in a professional artistic piece. My individual contributions are four-folded: (1) defining and carrying out the design study; (2) the technical development of the ZenStates prototype; and (3) carrying out the preliminary real- world application; and (4) most of the manuscript writing. Collaborators have contributed with: discussing and taking part of the design study; carrying out the technical development of the ZenTrees prototype; providing feedback on early versions of the ZenStates; and writing parts of the manuscript–especially the sections ‘Introducing ZenTrees’ and final paragraphs of ‘Related Works’. Chapter 5 Jeronimo Barbosa, Marcelo M. Wanderley, and Stéphane Huot. "ZenStates: Easy-to-Understand Yet Expressive Specifications for Creative Interactive Environments." In 2018 IEEE Sympo- sium on Visual Languages and Human-Centric Computing (VL/HCC), Lisbon, Portugal, 167–175. © 2018 IEEE. https://doi.org/10.1109/VLHCC.2018.8506491 Building upon the last chapter, this chapter introduces a visual