Chapter 8 The Perception of Auditory Patterns Diana Deutsch University of California, San Diego 1 INTRODUCTION Over the last half-century a considerable body of knowledge has accumulated concerning visual shape perception, based on findings in experimental psychology, neurophysiology, artificial intelligence and related fields. These developments have not, however, been paralleled by analogous developments in audition. It is inter- esting, therefore, that the foundations of pattern perception were laid by scientists who were as much concerned with auditory and musical phenomena as they were with visual ones. Wertheimer (1924/1938) described the beginnings of Gestalt theory thus: 'Historically, the most important impulse came from von Ehrenfels who raised the following problem. Psychology has said that experience is a compound of elements; we hear a melody and then, upon hearing it again, memory enables us to recognize it. But what is it that enables us to recognize the melody when it is played in a new key? The sum of its elements is different, yet the melody is the same; indeed, one is often not even aware that a transposition has been made.' (p. 4) In another paper in which he proposed the Gestalt principles of perceptual organization, Wertheimer (1923/1955) frequently presented musical illustrations along with visual ones. For example, in order to illustrate that 'more or less dissimilarity operates to determine experienced arrangement' he wrote: 'With tones, for example, C, C~, E, F, G~, A, C, C~... will be heard in the grouping ab/cd... ; and C, C~, D, E, F, F~, G~, A, A~, C, C~, D ... in the grouping abc/def' (1923/1955, p. 76). Figure 8.1 shows visual arrays such as presented by Wertheimer as analogous to these musical ones. In contrast to work on visual perception, the development of psychoacoustics around the middle of this century was characterized by a strong focus on highly restricted perceptual phenomena: namely, those that could be produced by very simple stimuli and which lent themselves to interpretations based on the workings of the peripheral hearing apparatus. Issues explored in detail included absolute and differential thresholds for pitch and loudness, lateralization of sine waves and noise bands, and various masking functions. The reluctance over the past few decades to consider higher-level processing in audition c~n be attributed to a number of factors (Deutsch and Pierce, 1992). First, Handbook of Perception and Action: Volume 1 Copyright 1996 Academic Press Ltd ISBN 0-12-516161-1 All rights of reproduction in any form reserved 253 254 D. Deutsch (a) (b) O O0 O o~ O O0 O O0 @O Figure 8.1 Visual analogs of two pitch series, to illustrate perceptual grouping in the two modalities. Pattern (a) corresponds to the series C, C~, E, F, G~, A, C, C~, and pattern (b) to the series C, C~, D, E, F, F~, G~, A, A~, C, C~, D. [From Wertheimer, 1923/1955. 1923 Springer; reprinted with kind permission of the publisher.] it was very difficult until recently to generate complex auditory stimuli with sufficient precision to address higher-level issues in the laboratory. A second factor was the lack of awareness of musical phenomena on the part of many researchers in the field. In the case of vision, it is evident from everyday experience that we recognize shapes as equivalent when these differ in size, in orientation and location in the visual field. Analogous perceptual equivalences in hearing are not evident from consideration of extramusical phenomena; however, music provides us with a number of convincing examples. We recognize musical passages when these are transposed to different keys; we treat as harmonically equivalent tones that are related by octaves; we perceive patterns of duration as equivalent when these are presented at different tempi, and so on. We are at present experiencing a renewed interest in mechanisms of auditory shape perception, and it appears as no surprise that this is accompanied by a resurgence of interest in music. 2 PERCEPTUAL GROUPING PRINCIPLES Before embarking on a consideration of auditory phenomena, we first review two approaches to the issue of how elements of perceptual arrays are grouped together. The first approach, expounded by the Gestalt psychologists, assumes that the perceptual system forms groupings on the basis of a number of primitive organiza- tional principles (Wertheimer, 1923/1955). One principle is termed proximity and states that perceptual connections are formed between elements that are close together, in preference to those that are further apart. Figure 8.2a presents an The Perception of Auditory Patterns 255 b) O00O 0000 O000 0000 A 0 c Q DO00000 0 0 OB Figure 8.2. Illustrations of several principles of perceptual organization enunciated by the Gestalt psychologists: (a) proximity; (b) similarity; (c) good continuation; (d) closure. example of this principle in vision. An array of dots is here displayed, which are closer along the vertical axis than along the horizontal one. These dots are perceptually connected on the basis of spatial proximity, so that columns rather than rows are perceived. A second principle, termed similarity, states that connec- tions are formed between elements that are similar rather than between those that are dissimilar. Figure 8.2b presents an example from vision. Here, the open and closed circles form separate groupings, so that, in this case, rows rather than columns are perceived. A third principle, termed good continuation, states that connections are formed between elements that continue in the same direction. As a visual example, the dots in Figure 8.2c are perceived as forming the continuous lines AB and CD. A fourth principle, termed closure, states that we tend to perceive arrays as self-enclosed units. Thus, the pattern in Figure 8.2d is perceived as a circle that is partially occluded by a rectangle. A fifth principle, termed common fate, states that elements that move synchronously in the same direction are perceptually connected together. For example, two rows of dots moving in opposite directions are seen as belonging to two distinctly different sources. Another approach to perceptual organization, expounded by Helmholtz (1862/ 1954), argues that the observer employs perceptual principles that give rise to the most effective interpretation of the environment. This line of reasoning could explain why the Gestalt grouping principles operate in perception. In making sense of our visual environment, it is useful to group together elements that are in spatial proximity or that are similar to each other, since these elements are more likely to belong to the same object. Analogously, sounds that are proximal in frequency or that are similar in timbre are likely to be emanating from the same source. We now turn to the application of these organizational principles to sound perception, with an emphasis on music, and we also consider the relative contribu- tions of pitch, timbre, spatial location and time as bases for such organization. 256 D. Deutsch .~ ,f. w - Figure 8.3. A measure from Beethoven's Waldstein Sonata. The visual representation provided by the score mirrors the converging pitch lines heard in the passage. The most prominent characteristics of music consist of arrangements of pitches in time. As a result, the conventions of musical scores have evolved primarily to emphasize such arrangements. Generally speaking, in a written score, pitch and time are mapped respectively into the vertical and horizontal dimensions of visual space. An illustration of such a mapping is given in Figure 8.3, which reproduces a measure from Beethoven's Waldstein Sonata. Here, the converging lines in the score provide a good visual analog of the converging pitch movements that are heard in the passage. As we shall see, several principles of perceptual organization in music are evident in visual analogs provided by scores, although there are also important exceptions. 2.1 Pitch Proximity in the Grouping of Sound Patterns Proximity is a powerful organizational principle in the perception of pitch struc- tures. This is particularly manifest for series of tones which are presented in rapid succession. When such tones are drawn from different pitch ranges, the listener hears two melodic lines in parallel: one corresponding to the higher tones and the other to the lower tones. This perceptual phenomenon has frequently been ex- ploited by composers. Figure 8.4 provides, as an example, a passage from Tarrega's Recuerdos de la Alhambra. The musical score is here given together with a represen- tation in which pitch (or log frequency) and time are mapped into two dimensions of visual space. The two separate pitch lines are clearly reflected in the visual representation. There are a number of interesting consequences to this tendency to group rapid series of tones on the basis of pitch proximity. One concerns the perception of temporal relationships. These are well perceived between tones that are in the same pitch range, but poorly perceived between tones that are in different pitch ranges (Bregman, 1990; Bregman and Campbell, 1971; Dannenbring and Bregman, 1976; Fitzgibbon, Pollatsek and Thomas, 1974; van Noorden, 1975). As an example, Figure 8.5 shows the just noticeable temporal displacement of a tone in a continuous series of alternating tones, plotted as a function of the pitch separation between the alternating tones. As can be seen, there is a gradual breakdown of temporal resolution with an increase in the pitch separation between the alternating tones. Another interesting consequence of grouping by pitch range was demonstrated by Dowling (1973). He generated two well-known melodies such that the tones from each melody occurred in rapid alternation. When the pitch ranges of the melodies overlapped heavily, listeners perceived a single line which corresponded The Perception of Auditory Patterns 257 00 000 000 000 000 000 000 000 0 0 M time Figure 8.4.