Computer mediated music production: A study of abstraction and activity by Matthew Duignan A thesis for the degree of Doctor of Philosophy in Computer Science. Victoria University of Wellington 2008 Abstract Human Computer Interaction research has a unique challenge in under- standing the activity systems of creative professionals, and designing the user-interfaces to support their work. In these activities, the user is involved in the process of building and editing complex digital artefacts through a process of continued refinement, as is seen in computer aided architecture, design, animation, movie-making, 3D modelling, interactive media (such as shockwave-flash), as well as audio and music production. This thesis exam- ines the ways in which abstraction mechanisms present in music production systems interplay with producers’ activity through a collective case study of seventeen professional producers. From the basis of detailed observations and interviews we examine common abstractions provided by the ubiqui- tous multitrack-mixing metaphor and present design implications for future systems. ii Acknowledgements I would like to thank my supervisors Robert Biddle and James Noble for their endless hours of guidance and feedback during this process, and most of all for allowing me to choose such a fun project. Michael Norris and Lissa Meridan from the Victoria University music department were also invaluable for their comments and expertise. I would also like to thank Alan Blackwell for taking the time to discuss my work and provide valuable advice. I am indebted to all of my participants for the great deal of time they selflessly offered, and the deep insights they shared into their professional world. Thank you to all of the members of the Elvis research group for the support and motivation your provided, particularly to Pippin and Rilla who helped me carry this thesis to completion. My thanks go to the members of the Carleton HOT Lab for hosting me so generously in Canada, and to Judy in particular for sharing your knowledge of activity theory. The members of my band Attik — Mike, Christian, Duncan, and Rob; you fuelled me with the passion and motivation to undertake this research project. Thank you for encouraging me to experiment with new technologies in rehearsals, and for understanding my move to Canada. Additional thanks to Duncan from Experimenta Ltd. for providing your experience and examples from your graphic design work. Thanks to Richard Procter for taking such an active interest in my work and for our long conversations. My thanks go out to my family for your love, and for creating in me the thirst for learning that has taken me this far. Lastly, thank you to Juls for your unwavering care and love. This research was made possible though the assistance of Victoria Uni- versity postgraduate scholarships, and a Royal Society of New Zealand travel grant. Contents 1 Introduction 1 2 Background 7 2.1 Introduction . 7 2.2 Computer mediated music production . 8 2.2.1 Recording, reproduction and synthesis of sound . 9 2.2.2 Capture and reproduction of musical gesture . 11 2.2.3 Editing audio and gesture . 11 2.2.4 The producer . 12 2.2.5 The Digital Audio Workstation . 13 2.3 Human Computer Interaction . 16 2.4 Digital craft and notational systems . 17 2.5 Abstraction . 19 2.5.1 Types of abstraction . 21 2.5.2 Conceptual models . 24 2.5.3 The cognitive dimensions of notations . 26 2.6 Activity theory . 28 2.6.1 Activity theory in HCI . 30 2.7 User-interface metaphor . 32 2.7.1 Usability and metaphor . 34 2.8 Summary . 38 iii iv CONTENTS 3 Models and Abstractions for Music 39 3.1 Introduction . 39 3.2 Models of music . 40 3.2.1 Psychological perception of music . 41 3.2.2 Common Music Notation and theory . 45 3.2.3 Digital representations of music . 47 3.2.4 Representations in software for experimental and gen- erative music . 51 3.3 The multitrack-mixing metaphor . 51 3.3.1 Multitrack tape recorders and mixing consoles . 52 3.3.2 From physical hardware to user-interface metaphor . 56 3.3.3 Multitrack-mixing model critique . 64 3.3.4 Abstraction mechanisms atop the multitrack-mixing model . 66 3.4 Summary . 72 4 Methodology 75 4.1 Introduction . 75 4.2 An introduction to case study . 76 4.3 The researcher . 78 4.4 Research tools . 80 4.4.1 Activity theory . 80 4.4.2 Cognitive dimensions . 86 4.5 Data collection and analysis . 89 4.5.1 Ethical considerations . 89 4.5.2 Data collection and recording procedure . 89 4.5.3 Data analysis procedure . 93 4.5.4 Case selection and participants . 96 5 Activity Analysis 103 5.1 Introduction . 103 5.2 Computer mediated music production . 104 CONTENTS v 5.3 Contradictions . 105 5.4 Subject, tool, and object . 108 5.5 The subject and their roles . 113 5.6 Activity hierarchy analysis . 118 5.6.1 Actions to generate material . 120 5.6.2 Actions to refine material . 122 5.6.3 Actions to arrange material . 123 5.6.4 Actions for live performance . 123 5.6.5 Actions for housekeeping . 125 5.7 Internalisation and externalisation . 126 5.8 Abstraction and activity . 128 5.8.1 The four aspects of abstraction . 129 5.8.2 Rendering considered harmful . 132 5.8.3 Option dilemma . 134 5.8.4 Conceptual burden . 136 5.9 Summary . 137 6 Process Abstraction 139 6.1 Introduction . 139 6.2 Processing power . 140 6.3 Viscosity caused by processing . 142 6.3.1 Moving on the time-line . 142 6.3.2 Moving to non-processed context . 146 6.4 One-off effects . 147 6.4.1 Effect on a single track . 147 6.4.2 Temporary diversion of shared effects . 149 6.5 Editing effected results . 149 6.6 Rendering for live performance . 151 6.7 Hiding edits . 152 6.8 Real-time processing . 154 6.9 Summary . 155 vi CONTENTS 7 Voice Abstraction 157 7.1 Introduction . 157 7.1.1 Stream perception . 158 7.2 Voice duration . 160 7.3 Organising voices . 161 7.4 Amalgamating voices . 162 7.5 Separating grouped material . 173 7.6 Grouped voices and context . 175 7.7 Summary . 177 8 Temporal Abstraction 179 8.1 Introduction . 179 8.2 Hierarchical temporal grouping . 179 8.2.1 Ambiguous temporal structure . 188 8.3 Linearisation . 189 8.3.1 Forming arrangements . 190 8.3.2 Material in arrangement context . 192 8.3.3 Flexibility . 195 8.3.4 Lead-ins . 198 8.4 Rhythmic structure . 200 8.4.1 Tempo and Time Signatures . 200 8.4.2 Rhythmic effects . 203 8.4.3 Event and metrical structure relationships . 204 8.5 The editing / performance divide . 209 8.5.1 Performance gesture . 210 8.5.2 Editing notation . 212 8.6 Summary . 214 9 Reuse and Versioning Abstraction 217 9.1 Introduction . 217 9.2 Copying versus referencing . 218 9.2.1 Copying . 218 CONTENTS vii 9.2.2 Referencing . 221 9.3 Variation . 223 9.4 History . 226 9.4.1 Versioning history . 227 9.4.2 Take management . 232 9.4.3 Material history . 233 9.4.4 Dependencies . 236 9.5 Library management . 237 9.5.1 Archiving previous work . 237 9.5.2 Locating material . 239 9.5.3 Local libraries and working space . 243 9.6 Summary . 247 10 Evaluation and Discussion 249 10.1 Introduction . 249 10.2 Evaluation of systems . 249 10.2.1 Apple Logic 7 . 250 10.2.2 Ableton Live 6 . 257 10.3 Implications for design . 265 10.3.1 Designing for processing abstraction . 265 10.3.2 Designing for voice abstraction . 270 10.3.3 Designing for temporal abstraction . 273 10.3.4 Designing for reuse and versioning abstraction . 276 10.4 Wider applications . 282 11 Conclusion 291 11.1 Thesis contributions . 292 11.2 Generalisability . 295 11.3 Future work . ..