Henkjan Honing MMM Group, NICI, University of Nijmegen From Time to Time: P.O. Box 9104, 6500 HE Nijmegen, The Netherlands Music Department, University of Amsterdam The Representation Spuistraat 134, 1012 VB Amsterdam, The Netherlands [email protected] of Timing and Tempo Timing plays an important role in the performance deviations. Furthermore, in some musical situa- and appreciation of almost all types of music. It tions or styles of music, where the global tempo is has been studied extensively in music perception mostly constant, event-shift (Bilmes 1993)—mea- and music performance research (see Palmer 1997 sured as the deviation with respect to a fixed beat for a review). The most important outcome of this or pulse in a constant tempo—offers a more natu- research is that a large part of the timing patterns ral way of representing timing. found in music performance—commonly referred First, this article will review existing representa- to as expressive timing—can be explained in terms tions of timing and tempo common in computa- of musical structure, such as the recurrent pat- tional models of music cognition and in terns associated with the metrical structure that programming languages for music. Their differences are used in jazz swing, or the typical slowing down are discussed, and some refinements will be pro- at the end of phrases in classical music from the posed (referred to as time maps, or TMs). The second Romantic period. These timing patterns help in part presents an alternative representation and communicating temporal structure (such as model for time transformation: so-called timing rhythm, meter, or phrase structure) to the listener. functions (TIFs, an acronym chosen to distinguish Furthermore, timing is adapted with regard to the them from TFs, or time functions, described by global tempo: at different tempi, other structural Desain and Honing 1992). This knowledge represen- levels of the music are emphasized, and the ex- tation differs in two important aspects from earlier pressive timing is adapted accordingly. In short, in proposals. First, expressive timing is seen as a com- music performance there is a close relationship be- bination of a tempo component (expressing the tween expressive timing, global tempo, and tem- change of rate over a fragment of music), and a tim- poral structure. One cannot be modeled without ing (or time-shift) component that describes how the other (see Figure 1). events are timed (e.g., early or late) with respect to Existing computational models of expressive this tempo description. Second, expressive timing timing (e.g., Clarke 1999; Gabrielsson 1999) are can be specified in relation to the temporal structure primarily concerned with explaining tempo varia- (e.g., position in the phrase or measure), as well as in tions, using tempo curves (specifying tempo or the terms of performance-time, score-time, and global reciprocal of duration as a function of the position tempo. In addition, TIFs support compositionality in the score) as the underlying representation. Al- (how simple descriptions can be combined into more though a useful means of measuring tempo pat- complex ones) and maintain consistency over musi- terns in a performance, tempo curves have been cal transformations (how these descriptions of tim- shown to fall short as an underlying representation ing and tempo should adapt when other parts of the of timing from a musical perspective (Desain and representation change), both important design crite- Honing 1991, 1993) and a psychological perspec- ria of the formalism. tive (Desain and Honing 1994). For instance, some Another design criterion is that timing transfor- types of timing, like chord spread (the asynchrony mations (e.g., the application of an expressive tim- in performing a chord), ornaments (like grace ing model to a score representation) are part of the notes), or the timing between parallel voices sim- representation, instead of only acting on a score— ply cannot be measured or represented as tempo the difference between a knowledge representation and a data representation. To realize this, it is cru- Computer Music Journal, 25:3, pp. 50–61, Fall 2001 cial to have access to the timing transformations © 2001 Massachusetts Institute of Technology. themselves, not only to the result of their applica- 50 Computer Music Journal Figure 1. Three aspects of tempo (or rate), and tem- music performance that poral structure (such as are closely related: ex- rhythm, meter, or phrase pressive timing, global structure). Temporal mance—the difference between a knowledge repre- structure sentation and a data representation (cf. ”The Vi- brato Problem” described in Honing 1995; Dannenberg, Desain, and Honing 1997). Tempo Timing The Representation of Tempo and Timing There are a number of ways of representing expres- sive timing. The three most frequently encoun- tion (as is the case in most music representation tered ones will be described here. Tempo functions systems, e.g., Dannenberg 1993). In other words, to are primarily used in music psychology research. be able to perform multiple transformations (as re- They form the output of several generative models quired by music sequencers and expression edi- of expressive timing (e.g., Clynes 1995; Todd 1992; tors), or compose a number of transformations into Sundberg 1988). Most of this research is concerned a complex transformation (essential in program- with keyboard music from the Baroque and Ro- ming languages for music or in combining partial mantic periods, where indeed tempo rubato (ex- computational models of expressive timing), the pressive tempo fluctuations) serves an important transformations themselves should be an object expressive function. In some more recent studies, within the representation, not just functions ap- time-shift (or event-shift) patterns (i.e., timing plied to it. A practical example will clarify this. measured as deviations from a regular pulse) are Imagine one applies a jazz-swing transformation analyzed, for example, in studies of timing in jazz to a score representation in a sequencer, resulting ballads (Ashley 1996) or in Cuban percussion mu- in a performance with some expressive timing sic (Bilmes 1993). In computer music, time- added. Next, the user applies a global tempo trans- maps—also referred to as time-deformations formation to this result, simply speeding up the (Anderson and Kuivila 1990) or time-warps performance. The result of this second transforma- (Dannenberg 1997)—are primarily used. They ex- tion, however, will sound strange. This is because press performance-time directly as a function of the swing pattern is closely related to the beat at score-time, and can, in principle, describe both the original tempo, which is now changed by the time-shift and tempo-change. tempo transformation. In order to obtain the de- Mathematically, tempo changes can be ex- sired result, the swing pattern transformation pressed as time-shifts and vice versa: they are must adapt itself, in retrospect, to the new tempo. equivalent under some constraints that will be It must, in fact, be a function of tempo not known mentioned later (see Figure 2). However, they are at the time of application. In general, this means musically very different notions. Tempo change is that previously applied transformations (or nested associated with, for example, rubato, accelerando transformations, for that matter) must sometimes (speeding up), and ritardando (slowing down), adapt themselves when new transformations are while time-shift involves, for instance, accentuat- applied. (Note: an alternative could be to describe, ing notes by delaying them a bit or playing notes with every new transformation, how all the other ”behind the beat,” both apparently independently aspects of the representation should be kept con- of the tempo. Also, from a perceptual point of sistent. However, this is a virtually impossible view, it seems that listeners do perceive tempo task.) Defining this so-called ”behavior under relatively independently from timing, a point that transformation” is essential in situations where a will be discussed later in this article. representation of timing is actually used (such as Before proposing a formalism that takes these ob- in music editors or computational modeling), in- servations into account the three existing representa- stead of being a static description of a perfor- tions mentioned above (i.e., tempo curves, time-shift Honing 51 Figure 2. (a) Three arbi- tempo, g a linear tempo- (to which f, g, and h be- trary tempo functions, f, change (a ”give and take” long), one can freely con- g, and h, and their rubato; see Figure 3 for an vert from one equivalent representa- example in musical nota- representation to the tions as a time-shift func- tion), and h a sudden other (see text for details). tion (b) and a time-map tempo-change. For a re- (c). f depicts a constant stricted class of functions ® ® ® x f x g x h 1 1 1 a) Tempo s ® s ® s ® ® ® ® d d d 0 0 0 b) Time-shift s ® s ® s ® ® ® ® ’ ’ ’ t t t c) Time-map t ® t ® t ® functions, and time-maps) will be presented. After tempo function to that score-time and multiplying their formal definition (in a functional style, without (´ ) the results. any typing for simplicity), their composition (in the A time-shift function (or event shift) can be ex- mathematical sense) will be shown, compositionality pressed as a function of score-time s (a rational being an essential strength of all three alternatives. number denoting symbolic score-time) returning a A tempo function (or tempo curve) can be ex- deviation interval d (a real number), that is, the pressed as a function of score-time s (a rational amount of time an event is shifted with respect to number denoting symbolic score-time) returning a its score-time: tempo factor x (a real number): f (s) ® d .