Exploring User-Driven Techniques for the Design of New Musical Interfaces through the Responsive Environment for Distributed Performance Dalia El-Shimy Department of Electrical and Computer Engineering McGill University Montreal, Canada November 2014 A thesis submitted to McGill University in partial fulfillment of the requirements of the degree of Doctor of Philosophy. c 2014 Dalia El-Shimy i Abstract The problem of enriching distributed communication has been tackled extensively by conventional investigations of videoconferencing technology. However, standard tools for computer-mediated communication have yet to offer a level of social engagement that compares to the gold standard of face-to-face interaction. While a number of systems attempt to bridge this gap by providing reasonable support for non-verbal cues, such systems typically attempt to improve collaboration of a practical nature. As such, we were interested in exploring the extent to which computer-mediated technology could support human activities necessitating a greater level of creativ- ity, playfulness and spontaneity. An additional motivation was understanding how distributed collaboration could improve on its co-present counterpart, by leveraging its underlying technology towards further assisting target users in effectively accom- plishing the activity at hand. As musical performance can be considered creative activity necessitating multi- ple levels of communication and interaction among its participants, we decided to investigate the topics above within the context of distributed musical performance. Given that Computer-Supported Cooperative Work research has illustrated that suc- cessful collaboration over a network is contingent not only on resolving technological complexities, but also on the development of appropriate interaction paradigms, we opted to undertake our research from a user- rather than technology-driven per- spective. The project was initially propelled by a traditional user-centered design methodology. As such, we began by developing a thorough understanding of our target end user, the musician, by means of contextual observations, interviews, and persona profiles. The information we acquired subsequently inspired a number of sim- ple prototype designs that, in turn, were used in formal user experiments to validate the basic premise of augmenting distributed performance. Eventually, we increased the level of user involvement through a long-term deployment and collaboration with a three-piece ensemble, and a participatory design cycle with a composer. The final result of our design and development efforts is a responsive environment that augments distributed performance with dynamic, real-time, hands-free control ii over several aspects of the musicians' sound, enabling them to seamlessly change vol- ume, affect reverb, adjust the mix, and perceive spatialized audio rendering, without detaching themselves from their higher-level activity. Furthermore, a derivative of this system that provides such features within the context of musical composition was also developed. Our user-driven design of a novel interactive musical system was not without its share of difficulties. The non-utilitarian nature of the users' tasks poses special chal- lenges, requiring attention to benchmarks, evaluation techniques and alternatives to formal quantitative testing that are suitable to the exacting nature of musical performance. Such challenges are by no means unique to the context of musical performance, but inherent to many creative and artistic domains. As such, this dis- sertation contributes two novel artefacts|a responsive environment for distributed performance, and a responsive environment for composition|along with a set of recommendations based on our experiences working with a unique and creative end user whose needs cannot easily be defined. In turn, our solutions may be of help to developers looking to acquire a deeper understanding of the user experience that the traditional notion of usability alone does not afford. iii R´esum´e Le probl`emede l'enrichissement de la communication distribu´eea ´et´elargement abord´epar les enqu^etesclassiques de la technologie de vid´eoconf´erence. Cependant, les outils standards pour la communication m´ediatis´eepar ordinateur n’offrent pas encore un niveau d'engagement social qui compare a l'´etalon-orde l'interaction face- `a-face. Bien qu'un certain nombre de syst`emestentent de combler cet ´ecarten fournissant un soutien raisonnable pour les indices non-verbaux, ces syst`emestentent g´en´eralement d'am´eliorerla collaboration de nature pratique. Pour cette raison, nous nous sommes int´eress´es`aexplorer la mesure dans laquelle la technologie m´ediatis´ee par ordinateur puisse soutenir les activit´eshumaines qui n´ecessitent un plus grand niveau de cr´eativit´e,ludisme et de spontan´eit´e.Une motivation suppl´ementaire ´etait notre d´esirde comprendre la mani`eredont laquelle la collaboration distribu´eepuisse surpasser son homologue du cas co-pr´esent, en profitant de sa technologie sous-jacente pour aider les utilisateurs `aaccomplir leurs activit´esplus effectivement. Comme la performance musicale peut ^etreconsid´er´eeune activit´ecr´eative qui n´ecessiteplusieurs niveaux de communication et d'interaction entre ses participants, nous avons d´ecid´ed'enqu^eterles sujets ci-dessus dans le cadre de la performance mu- sicale distribu´ee.Puisque la recherche sur le travail coop´eratifassist´epar ordinateur a d´emonstr´eque le succ`esde la collaboration sur un r´eseaudepend non seulement sur la r´esolutiondes complexit´esde la technologie, mais aussi sur le d´eveloppement des paradigmes d'interactions appropri´es,nous avons opt´ed'entreprendre notre recherche selon une perspective ax´eesur l'utilisateur, plut^otque sur la technologie. Le project ´etaitinitialement propuls´epar la m´ethodologie de conception centr´eesur l'utilisateur traditionelle. Nous avons ensuite hauss´ele niveau de participation des utilisateurs `a travers un d´eploiment et collaboration `along terme avec un ensemble de trois pi`eces, et un cycle de conception participative avec un compositeur. Le r´esultatfinal de nos efforts de conception et de d´eveloppement est un syst`eme qui augmente la performance musicale distribu´eeavec des contr^olesdynamique, aux mains libres, en temps r´eelsur plusieurs aspects du son musical. Notre syst`emeper- met aux musiciens de changer leur volumes, d'ajuster leurs niveaux de r´everb´eration, iv de r´eglerleur mixage, et de percevoir des effets audio spatialis´es,tout d'une fa¸con transparente qui ne n´ecessitepas qu'ils se d´etachent de leur activit´ede niveau sup´erieure.En outre, un d´eriv´ede ce syst`emequi fournit ces m^emescaract´eristiques dans le contexte de la composition musicale a ´egalement ´et´ed´evelopp´e. Notre conception ax´eesur l'utilisateur d'un syst`ememusical interactif n'´etaitpas sans part sa part de d´efis.Le caract`erenon-utilitaire des t^aches des utilitsateurs pose des difficult´esparticuli`eres,exigeant une attention sp´ecialeaux indices de r´ef´erence et techniques d'´evaluation, et des alternatives aux tests quantitatifs formels qui sont adapt´es`ala nature de la performance musicale. Ces d´efisne sont pas uniques au context musical, mais sont aussi inh´erent `aplusieurs domaines cr´eatifset artis- tiques. En cons´equence,cette th`esecontribue deux objects fabriqu´esnouveaux|un environment r´eactiffor la performance distribu´eeet un environment r´eactiffor la composition musicale|accompani´espar une s´eriede recommendations bas´eessur nos ´experiences en travaillant avec des utilisateurs cr´eatifset uniques, les besoins de qui ne peuvent pas ^etrefacilement d´efinis. A` leur tour, nos solutions pourrait ^etre utiles aux d´eveloppeurs qui cherchent `aacqu´erirune compr´ehensionplus profonde de l'exp´eriencede l'utilisateur que la notion traditionelle d'utilisabilit´ene peut pas offrir. v Dedication To Hala and Reda vi Acknowledgments Big up to all my homies who (somehow) managed to put up with me through this insane process, even though they probably wanted to kill me at times. You know how you are. I'd also like to thank my supervisor Jeremy R. Cooperstock, and the members of my committee, Frank Ferrie and Marcelo M. Wanderley, for their invaluable guidance and advice; Thomas Hermann for inspiring me to get the ball rolling; Steve Cowan for being a wonderful collaborator; the members of People on Pause and the Visitation for stepping up to the plate when I needed to run one too many of those last-minute sessions; and the current and past members of the Shared Reality Lab, who offered immense support while tolerating my non-stop ranting for the past six years. vii Contents 1 Introduction 1 2 Background and Previous Works 6 2.1 Computer-Supported Cooperative Work . .6 2.1.1 From Videoconferencing to Telepresence . 12 2.2 Network Musical Performance . 15 2.2.1 Embracing the Network as a Performance Environment . 17 2.2.2 Classification of systems for Network Musical Performance . 19 2.2.3 Visual Representations . 28 2.3 Additional Topics of Relevance . 32 2.3.1 Interconnected Musical Networks . 32 2.3.2 Interactive Installations . 36 2.3.3 Responsive Environments . 37 2.3.4 Embodied Interaction . 40 2.4 User Involvement in Design . 43 2.4.1 User-Centered Design . 43 2.4.2 Participatory Design . 46 2.4.3 Music-oriented HCI . 47 2.5 System Evaluation: Going Beyond Usability . 50 2.5.1 Mixed Research . 51 2.5.2 The Qualitative Experiment . 56 2.5.3 Experience-Based Design . 59 Contents viii 2.6 Synthesis of Literature Review . 69 3 Preliminary Work and Motivation 73 3.1 The Videoconferencing Privacy System . 73 3.2 Design and Implementation . 74 3.2.1 Scene Rendering . 74 3.2.2 Gesture Detection . 75 3.3 From Videoconferencing to Network Musical Performance . 77 4 Understanding the Target User 80 4.1 User Observations . 80 4.1.1 What we noted . 81 4.1.2 What the conversation analysts noted . 83 4.2 Personas . 86 4.2.1 Listing the Distinguishing Behaviours . 87 4.2.2 Grouping the Behaviours . 89 4.2.3 Grouping the Performers .