Tools and Techniques for Rhythmic Synchronization in Networked
Total Page:16
File Type:pdf, Size:1020Kb
Toolsand Techniques for Rhythmic Synchronization in Networked Musical Performance Reid Kei Oda a dissertation presented to the faculty of princeton university in candidacy for the degree of Doctor of Philosophy recommended for acceptance by the Department of Computer Science Advisers: Rebecca Fiebrink Adam Finkelstein June 2017 © Copyright by Reid Kei Oda, 2017. All rights reserved. Abstract This thesis examines the Internet as a space for playing collaborative, rhythmically synchro- nized music among artists who are in different physical locations. The Internet has revolu- tionized many aspects of life, but has yet to become a frequently-used venue for music per- formance. We take the position that in order for this to occur, we must develop a new class of musical instruments that are specifically designed for the web. In particular, they should compensate for the delay that exists in networked communication. We explore two differ- ent approaches to this problem. The first is a method for tempo synchronization across large distances. The second is a method for reducing delay by predicting notes before they are played, sending the note information across the Internet, and scheduling the note to be synthesized into audio at the same time in multiple locations. Our method for tempo synchronization, the Global Metronome, delivers timing er- ror comparable to or less than the industry standard method. However, our method can synchronize tempo between devices located anywhere on earth, while the industry stan- dard method requires devices to be connected via a physical cable. Our approach depends on synchronized clocks. Therefore this thesis presents an inexpensive, portable and accu- rate time server device to grant users the ability to use the Global Metronome. This device, called PIGMI, is a software suite that runs on a popular single-board computing platform. The software is published as a freely-available, open source software project. This thesis also presents a musical instrument that uses prediction to reduce network latency, called MalLo. We present a series of studies that explore the sensor requirements, viability, and usability of the system. We also use MalLo in concert performance and study musicians as they learn to use the instrument over 3 rehearsals and the concert. The primary contributions of this work include (1) a method to synchronize tempo be- tween devices that are separated by great distance; (2) a small, inexpensive timeserver that facilitates this approach; (3) a better understanding of the advantages and disadvantages of tempo synchronization via this approach; (4) a method for latency reduction via per- note prediction including simulations, a usability study, and a study of live performance exploring its viability. This work enables new methods of remote musical collaboration and moves us towards realizing the dream of a global community of collaborating musicians. iii Acknowledgments I am deeply grateful to the many people who have helped me throughout my academic career. First and foremost I would like to thank my advisor, Rebecca Fiebrink, for her guid- ance throughout my graduate study at Princeton. Your expertise, broad vision, and skillful mentoring has been truly inspiring. There is no way to thank you enough for all of the time and support you’ve given to me. I am also deeply indebted to Adam Finkelstein who provided much needed in-person support and guidance. You helped me to feel at home at Princeton. I learned a lot from you and I had fun doing it. I would like to humbly thank Lynn Shostack and the Princeton Project X grant for choosing to fund this work. I am privileged to be one of the many “out-of-left-field” lines of research you have supported. My favorite part of Princeton has been working with the faculty of Department of Com- puter Science. In particular I would like to thank Szymon Rusinkiewicz for helping to build the initial prototypes of the MalLo system. I would like to thank Thomas Funkhouser for thinking deeply about my work and giving honest, insightful feedback. I’d like to thank the graphics group for allowing us audio people to tag along. I am very impressed by the well-oiled research machine you have created, and I am grateful to have benefited from it. I would also like to thank Jennifer Rexford who, despite her multitude of obligations, always made time to answer my questions. Thank you to my committee members. Marshini Chetty, your skillful advice on human- computer interaction topics have been a breath of fresh air. Chris Chafe, I could not have done this research without your foundational work on networked music performance. I feel fortunate that when I needed guidance in my thesis work I could go straight to the source. Dan Trueman, your seminar during my first semester at Princeton helped estab- lish my belief that the relationship between music and technology can be symbiotic, but we have to put some effort into it. Now, six years later, I’m honored to have you on my com- mittee. I am extremely grateful for my time with Jeff Snyder spent exploring a vast range of top- ics. Through our conversations I gained a broader understanding of music and technology than I thought possible. I’d also like to thank Jeff for fostering a thriving computer music community at Princeton. You are the beating heart of the beast. I would have been lost without our graduate coordinators, Melissa Lawson and Nicki Gotsis. Thank you for helping me to navigate the complex world of Princeton academics. This work would not have been possible without the many volunteers who helped to test and develop the systems described in this thesis. I’d like to thank my fellow Internet music adventurers, Olivier Ozoux, Mike O’Connor, and Mason Malvini. Your eagerness iv made the experience fun, and your patience made it possible. To the musicians who per- formed in the Numerical Experience concert I send my heartfelt thanks. Chris Douthitt, Nikitas Tampakis, Joshua Becker, and Josh Collins, you brought your enthusiasm and made the event a success. My Sound Lab colleagues made my experience at Princeton fun. To Katie Wolf, thank you for the many conversations and insights. Nearly everything that I produced in graduate school has your mark on it. I am proud to call you a friend and colleague. I genuinely could not have made it to this point without your help. I also could not have made it without Zeyu Jin. Your help and expertise made MalLo a reality. Your humor and zest for life made the Sound Lab a fun and silly place. I’m eager to see what you two do next. I would like to thank my past mentors and colleagues. My time with you formed the basis for this thesis. Jim Hollan, Gert Lanckriet, Luke Barrington, Brian McFee, Ery Arias- Castro, Dr. Willie Brown; I am grateful for you. In particular, I’d like to thank Dr. Willie Brown for believing in me. Finally I’d like to thank those that are closest and dearest to my heart. Thank you to my friends; you all are the best. Thanks to my family: Dad, Mom, Nalani, Kim. And of course thank you to Erin. Your loving support, conviction, and technical mastery helped me through many challenges to the finish line. v To my parents, Francis and Caroline Oda. Thank you for your never-ending love and support. vi Contents Abstract ...................................... iii Acknowledgments ................................. iv 1 Introduction 2 1.1 Internet as a Performance Space ....................... 2 1.2 Goals and Contributions .......................... 3 1.3 Outline ................................... 5 2 Background and Motivation 7 2.1 Summary .................................. 7 2.2 The Internet ................................. 8 2.3 Networked Musical Performance ...................... 11 2.4 Gestural Control of Sound Synthesis .................... 16 2.5 Clock Synchronization ........................... 17 2.6 Conclusions ................................. 21 3 The Global Metronome 23 3.1 Introduction ................................. 23 3.2 Background ................................. 25 3.3 The Global Metronome Phase ........................ 28 3.4 Tempo Changes ............................... 29 3.5 PIGMI: The $99 Global Metronome .................... 32 3.6 Evaluation .................................. 33 3.7 Discussion .................................. 38 3.8 Conclusion ................................. 39 4 The Global Metronome in Use 42 4.1 Introduction ................................. 42 4.2 Study Sessions, in Practice .......................... 51 4.3 Results .................................... 56 vii 4.4 Lessons from Participant Experiences .................... 63 4.5 Conclusion ................................. 68 5 MalLO: A Distributed, Synchronized Musical Instrument 69 5.1 Introduction ................................. 69 5.2 Initial Study ................................. 72 5.3 MalLo: Proof of Concept .......................... 82 6 Usability of MalLo 100 6.1 Introduction ................................. 100 6.2 Experiment Design ............................. 101 6.3 Results .................................... 111 6.4 Discussion .................................. 114 6.5 Conclusion ................................. 118 7 MalLo in Use 119 7.1 Introduction ................................. 119 7.2 The Numerical Experience Performance . 120 7.3 Internet Pilot Study ............................. 135 7.4 Conclusion ................................. 138 8 Conclusion 140 8.1 Summary and Contributions . 140 8.2 Conclusion ................................