The Effects of Latency on Ensemble Performance by Nathan Schuett May, 2002 Advisor: Chris Chafe Research sponsored by National Science Foundation CCRMA DEPARTMENT OF MUSIC Stanford University Stanford, California 94305 THE EFFECTS OF LATENCY ON ENSEMBLE PERFORMANCE Nathan Schuett Stanford University, 2002 Abstract Methods for audio transport over networks have developed to a point where experiments with professional-quality musical collaboration are now possible. Low-latency, low-jitter, next-generation networks have been tested in a variety of musical scenarios including performers playing together across continental distances. The latency problem inherent in long-haul network paths is well known but is less well understood in terms of its effect on real-time musical collaboration. We have begun a series of experiments testing the effect of latency on ensemble performance. The performances will be evaluated based on their tempo direction, average beat duration, and standard deviation of their beat durations. The goal is to define the Ensemble Performance Threshold (EPT), or the level of delay at which effective real-time musical collaboration shifts from possible to impossible. Our motivation is the need for a “latency design spec" (in msec) for engineering new systems that support truly natural feeling audio collaboration environments. This study served as a pilot study to investigate whether an EPT exists and to determine the general effects of latency on musical performance. Conclusions were as follows: (1) The direction of the tempo was a very useful indicator of whether a performance was being hindered by the effects of latency. If the delay was greater than 30 msec, the tempo would begin to slow down. This gives a solid indication that EPT for impulsive, rhythmic music lies between 20-30 msec. (2) A coping strategy was discovered that allowed the performers to maintain a solid tempo up to 50-70 msec of delay. The strategy can be quickly summarized as a leader - follower relationship. Unfortunately, this strategy results in a severe decrease of synchrony on the leader’s end. (3) It is most likely that EPT varies depending on the type of music (speed, style, attack times of instruments, etc). (4) When delay is between 10-20 msec each way, it may be providing a stabilizing effect on the tempo. 10-20 msec of delay may be better for ensemble performance than 0 msec of delay. (5) The EPT determined in the electronic delay tests was much lower than the EPT estimated in the outdoor delay tests. This is predicted to be due to the lack of auditory cues in the electronic tests such as reverb and variable amplitude which were present in the outdoor tests. 2 Table of Contents Chapter I. Introduction 1.1. Summary 1.2. Internet Performance 1.3. Thesis Scope Chapter II. Historical Review of Relevant Research 2.1. Latency 2.1.1. Effects on Telephone Conversation 2.1.2. Effects on Ensemble Performance Large Ensemble Small Ensemble 2.1.3. Electronically Manipulated Delay Experiments 2.1.4. Spatially Manipulated Delay Experiments 2.2. Tempo 2.2.1. Tempo Studies 2.2.2. Tempo During a Performance 2.2.3. Tempo Evaluation 2.3. Rhythmic Synchronization Chapter III. A Tool for Tempo Analysis 3.1. Local Maximums 3.2. Surfboard Method 3.3. Determining Events 3.3.1. Error 3.4. Tracking the Tempo 3.5. Analyzing the Data Chapter IV. Empirical Research 4.1. Method for Electronically Manipulated Delay Experiments 4.2. Scenario 1 4.2.1. Experiment 4.2.2. Results 4.3. Scenario 2 4.4. Scenario 3 4.5. Scenario 4 4.6. Scenario 5 4.7. Summary of Suggested EPTs Chapter V. Discussion / Conclusions 5.1. General Effects 5.2. Swinging Beats 3 5.3. Two Coping Strategies and their Respective EPTs 5.4. Quantitative Performance Analysis 5.4.1. Tempo Direction Measure 5.4.2. Standard Deviation Measure 5.4.3. Synchrony Measure 5.5. Adding Limited Delay may Actually Improve Performance 5.6. Reverberation 5.6.1. A Preliminary Test 5.6.2. A Proposed Solution 5.7. Future Study 5.7.1. Multiple Performers 5.7.2. Real Music vs. Clap Tests 5.7.3. Different EPT’s for Different Types of Music 5.7.4. Improvisation 5.7.5. Shifting Between Coping Strategies 5.8. Application 5.8.1. Long-Distance Sessions over the Net 5.8.2. Design Specs 5.8.3. A Little Latency Could be Good Latency 5.9. Conclusions Appendix A. Amplitude Envelope Code Appendix B. Event Detector / Tempo Analyzer Code 4 Chapter I Introduction 1.1. Summary High-bandwidth audio streaming in real-time has recently become feasible. This is a result of the emergence of low-latency, low-jitter, next-generation networks. Collaborative musical performance is now possible over the Internet with professional-quality sound. The latency problem inherent in long-haul network paths is well known, but is less well understood in terms of its effect on real-time musical collaboration. This study served as a pilot study to investigate the effects of latency on collaborative performance. 1.2. Internet Performance Latency between New York and California measured over the Abilene network (Internet2's next-generation test bed) is 33 msec (which is within a factor of 2 of the speed of light). Network jitter has been measured on the order of 4%. Sound travels at around 345m/s, so the equivalent distance acoustically is around 12m, well within the dimensions of a large concert stage. Since musicians can effectively perform together at this distance, it is hypothesized that they should also be able to perform together if their signals are delayed electronically by a similar amount. The assumption that electronic latency correlates with a physical distance between players was tested in a preliminary study at Stanford involving two drummers outside. This preliminary study will be described in more detail in Section 2.1.4. 1.3. Thesis Scope 1. Determine and document the effects of delay on two-way musical performance. 2. Attempt to isolate a critical delay "comfort level" for playing rhythmic music under delay constraints. This will be called the Ensemble Performance Threshold (EPT). 5 3. Identify any other differences that distinguish a "telejam" from an ensemble performance in the same acoustic space. 4. In order to answer the above three questions, a quantitative method must be developed for analyzing ensemble performance. The method must be accurate and repeatable. 6 Chapter II Historical Review of Relevant Research 2.1. Latency There is a curious lack of research that deals with delay and its effects on music. Perhaps this is because of the hard to quantify characteristic of music and the lack of technology capable of performing such analyses. Dave Phillips, who maintains the Linux Music & Sound Applications Website, writes that, “studies have indicated that the ear is sensitive to timing differences at the millisecond level, perhaps even down to a single millisecond.” He also claims that latencies under 7 msec are not typically perceptible and should be considered acceptable for desktop and semiprofessional audio applications [1]. 2.1.1. Effects on Telephone Conversation The general consensus of much study on voice transmission is that one-way delay of less than 100 milliseconds (msec) is imperceptible to most users. Delays in the range of 100 – 300 msec are considered to be noticeable, but tolerable. Latencies greater than 300 msec are not tolerable, as they result in a speak-and- wait conversation [2]. 2.1.2. Effects on Ensemble Performance Jeremy Cooperstock, from McGill University claims that there are two EPTs for ensemble performance based on size of the ensemble alone. He claims that based on research studies, large ensembles can tolerate up to 40 msec of latency while small ensembles can tolerate only up to 5 msec [3]. Large Ensemble (8 or more players) Cooperstock points to two studies by Rasch which showed that a typical delay between the first and the last attack between performers who are playing a single note was approximately 40 msec for large ensembles [4]. For example, if a symphony orchestra were to play a single note simultaneously, the time between the earliest musician’s and the latest musician’s entries on that note will be approximately 40 msec. This is not surprising, since many stages have dimensions as large as 40 ft - the distance traveled by a sound wave in air in 40 7 msec. Cooperstock then concludes that a two-way latency of up to 40 msec is an acceptable maximum delay for large ensemble performance. Small Ensemble Cooperstock’s estimate of a 5 msec EPT for small ensembles is based largely on practical experience. Looking at the evolved seating arrangements of string quartets and trios, it is easy to see that the players try to sit very close to one another. He claims that when musicians in such a group are separated by more than roughly 2 m, difficulties in the ensemble are incurred [3]. Thus, he sets the EPT at 5 msec. 2.1.3. Electronically Manipulated Delay Experiments The first such experiment involved placing two trumpeters in separate rooms. The experiments were run at the Banff Center for the Arts. Microphones and headphones allowed the players to hear each other. A TCP-based, 1-channel, bi- directional application was tuned to provide delays of about 200 msec. The musicians were initially mystified by trying to perform in such a situation (especially with no visual cue for starting together). It only became possible to avoid recursive tempo slowing when one player agreed to play behind the other [6]. There were no tools available to analyze the above experiment.