Making Musical Magic Live Inventing modern production technology for human-centric music performance Benjamin Arthur Philips Bloomberg Bachelor of Science in Computer Science and Engineering Massachusetts Institute of Technology, 2012 Master of Sciences in Media Arts and Sciences Massachusetts Institute of Technology, 2014 Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Media Arts and Sciences at the Massachusetts Institute of Technology February 2020 © 2020 Massachusetts Institute of Technology. All Rights Reserved. Signature of Author: Benjamin Arthur Philips Bloomberg Program in Media Arts and Sciences 17 January 2020 Certified by: Tod Machover Muriel R. Cooper Professor of Music and Media Thesis Supervisor, Program in Media Arts and Sciences Accepted by: Tod Machover Muriel R. Cooper Professor of Music and Media Academic Head, Program in Media Arts and Sciences Making Musical Magic Live Inventing modern production technology for human-centric music performance Benjamin Arthur Philips Bloomberg Submitted to the Program in Media Arts and Sciences, School of Architecture and Planning, on January 17 2020, in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Media Arts and Sciences at the Massachusetts Institute of Technology Abstract Fifty-two years ago, Sergeant Pepper’s Lonely Hearts Club Band redefined what it meant to make a record album. The Beatles revolution- ized the recording process using technology to achieve completely unprecedented sounds and arrangements. Until then, popular music recordings were simply faithful reproductions of a live performance. Over the past fifty years, recording and production techniques have advanced so far that another challenge has arisen: it is now very difficult for performing artists to give a live performance that has the same impact, complexity and nuance as a produced studio recording. Live performance production technology is now used almost exclusively to recreate studio albums exactly as they were recorded. Recently, this approach has dominated the entertainment industry. In an attempt to reach superhuman levels of perfection and complexity, many elements that make live performances emotionally meaningful for audiences have been given less priority — or lost altogether. The mission of the work described in this dissertation is to reverse this trend by investigating methods of integrating technology and live music performance in such a way that the technology allows for flexible musical expression, sound and connection to the audience, while still enabling exciting, sophisticated and “magical” production values. This dissertation identifies six objectives for the human-centric design and integration of technology in musical performance, and a methodology to support each objective. These have been developed, refined and tested with artists and performers through a series of ten large-scale projects and approximately 300 individual performances. Through this work, I demonstrate that it is possible to combine high-value production with interactive musical performance. We are now on the cusp of redefining live musical performance production as an art form just as Sergeant Pepper’s Lonely Hearts Club Band redefined studio album production as an art form fifty years ago. Thesis Supervisor: Tod Machover Muriel R. Cooper Professor of Music and Media Program in Media Arts and Science, Massachusetts Institute of Technology Making Musical Magic Live Inventing modern production technology for human-centric music performance Benjamin Arthur Philips Bloomberg The following person served as a reader for this thesis: Thesis Reader: Joseph Paradiso Alexander W. Dreyfoos Professor Program in Media Arts and Sciences Massachusetts Institute of Technology Making Musical Magic Live Inventing modern production technology for human-centric music performance Benjamin Arthur Philips Bloomberg The following person served as a reader for this thesis: Thesis Reader: Robert Rowe Director of Music Technology Department of Music and Performing Arts Professions Steinhardt School at New York University Acknowledgments Tod Machover I am indebted to you for looking after me for the last 12 years, inspiring Family & Friends and mentoring me, and teaching me how to listen. Lisa Philips, Owen Hooks Davis, John and Julie, Ari, Rose, Kristen Jacob Collier My brother and collaborator, with whom I get to imagine, adventure and Evan, Anthony Thomas, Tommy and create. Thank you for your trust and friendship. Moriarty, Nicole L’Huillier and Juan Necochea, Iris Fung, Julia Rue Robert Rowe My dissertation committee, whose guidance and patience were critical Joseph Paradiso in the formulation of this dissertation, and whose work inspired me MIT & Beyond significantly in this journey. Fred Harris, John & Rosie Harbison, Beth Singer, Beau Flahive, Ash Neh- Brenda Philips My family who has supported me infinitely through thick and thin, ru, Imogen Heap, Ariana Grande, David Bloomberg helping me to find, follow and grow my passion into something real. Kelly Snook, John Taylor, Shawn Mikaela Bloomberg My parents and Alex especially, who contributed significantly to the Duncan, Patrick Ostwald, Michael Alex Turner editing of this document and know my voice and what I want to say. Eisenberg, Luke Plummer, Ben Glenda & Gregory Philips Maron, Abe Jacob, Chris Full, Cuco Louise & Sanford Bloomberg Daglio, Maxime Le Saux, Antonio Susan Fink Silva, Scott McArthur, Dan Richard- son, John Bechtold, Rob Scalea, The Peter Torpey My Media Lab co-conspirators in all things, thank you for being my MIT Logarhythms Brian Mayton role models and confidants, teaching me to be thoughtful, and pushing Charles Holbrow me to think boldly. Additional Editing Rebecca Kleinberger Sara Amâncio & Bryan Reese Michael Miller Garrett Parrish JC Collaborators Suzie, Sophie & Ella Collier, Jose Ortega, Quincy Jones, Adam Fell, Michael Peha, Claudio Somigli, Rob Fisher, Luke Edwards, Louis Mustill, Arron Smith, Will Young, Jules Buckley, Maro Secca, Robin Mullarkey, Christan Euman, Nick Groff Media Lab Collaborators Simone Ovsey, Akito Van Troyer, Gershon Dublon, David Kong, Nikhil Singh, Sizi Chen, Alexandra Rieger, Thom Howe, Priscilla Capistrano, Kevin Davis, Jessica Tsymbal, Dan Pfeiffer, Ethan Nevidomsky, Peter Colao The Opera of the Future Group - Past, Present and Future Table of Contents 0 Introduction 15 1 The Evolution of Instruments, Performance, Production and Hyperproduction 19 2 Background 21 2.1 Existing Production 22 2.1.1 Techniques 22 2.1.2 On the pursuit of “Liveness” 26 2.1.3 Roles 27 2.1.4 Process 28 2.2 Composition and Sound Design 29 2.3 Hyperinstruments 32 2.4 Illustrative Production Examples 32 2.4.1 Broadway Theatre 33 2.4.2 Orchestra 36 2.4.3 Immersive Theater 37 2.4.4 Pop Tour 39 2.4.5 Indie Artist Performance 41 2.4.6 Jam Band 42 2.4.7 Live Television Broadcast 43 2.4.8 DJ Set 44 3 The Opera of the Future 47 3.1 Early Projects 47 3.2 Skellig 51 3.2.1 Electronics 52 3.2.2 Live Production 54 3.3 Death and the Powers 56 3.3.1 Initial Prototyping 57 3.3.2 A.R.T. Workshop 60 3.3.3 Bringing the Full-Scale Production to Life 61 3.3.4 Rehearsals and Development 70 3.3.5 Premiere in Monaco 74 3.3.6 Touring 76 4.1 The Beginning of the Harmonizer 162 3.3.7 Remote Theatrical Experience with Powers Live 78 4.2 Live Looping and Mixing 164 3.3.8 Reflecting onPowers 89 4.3 Refining the Harmonizer for Touring 166 3.4 Extending Sleep No More 91 4.4 Show Control with Ableton Live 168 3.4.1 System Overview 92 4.5 Shrinking the One-Man Show 177 3.4.2 Media Distribution 93 4.6 Harmonizer Hardware 2.0 181 3.4.3 Video Systems 94 4.7 Video Looping System 183 3.4.4 Audio Systems 96 4.8 Lessons from the One-Man Show 184 3.4.5 Control Middleware 98 4.8.1 Timing 184 3.4.6 Telephone Systems 99 4.8.2 Communication 186 3.4.7 Network Infrastructure 101 4.8.3 Improvisation 188 3.4.8 Design Process 101 4.9 Creative Goals for the Djesse Tour 189 3.4.9 Conclusions 102 4.10 Technical Goals for the Djesse Tour 191 3.5 Hyperproduction 104 4.10.1 Connectors 191 3.5.1 Data Architecture 106 4.10.2 Redundancy and Notable Failures 192 3.5.2 Advanced Features 109 4.11 Development of the Djesse Album Tour 194 3.5.3 The Process of Building Hyperproduction 111 4.11.1 The BBC Proms 197 3.6 Fensadense 113 4.11.2 Finding the Band 202 3.6.1 Initial Rehearsal Period 114 4.11.3 Djesse Album Release Celebration at MIT 203 3.6.2 Developing the Analysis and Production 124 4.11.4 London Rehearsals 208 3.6.3 Final Rehearsals 126 4.12 Touring Djesse 219 3.6.4 Performance and Touring 133 4.12.1 Hardware Revision 223 3.6.5 Reflections 134 4.12.2 Show Documentation 227 3.7 City Symphonies and Orchestra Imaging 134 4.12.3 Rehearsal Process in Sound Checks 227 3.7.1 City Symphonies Around the World 139 4.12.4 Flying the Full Show 227 3.7.2 Symphony in D 140 4.13 What Lies Ahead 235 3.7.3 Philadelphia Voices 143 3.8 Schoenberg in Hollywood 146 5 Production Infrastructure for Everyday Life 237 3.8.1 Orchestra as Character 148 5.1 Gamma Home 237 3.8.2 Sound Design 150 5.1.1 Lighting Implementation 238 3.8.3 Evaluation 155 5.1.2 Sound Implementation 241 3.9 The Rhythm of Production: One-offs vs. Mainstream Touring 157 5.1.3 Results in Practice 241 5.2 The Future of Hyperproduction... Borrowing from Social Apps? 243 4 Jacob Collier and the Djesse World Tour 159 5.3 Proliferation of Home Production 244 6 Conclusion 245 6.1 Methodologies for Hyperproduction