Haverford College Haverford Scholarship Faculty Publications Psychology 2009 Audiovisual Interactions: The Impact of Visual Information on Music Perception and Memory Marilyn Gail Boltz Haverford College, [email protected] B. Ependorf B. Field Follow this and additional works at: https://scholarship.haverford.edu/psychology_facpubs Repository Citation Boltz, M.G., Ebendorf, B. & Field, B. (2009). Audiovisual interactions: The impact of visual information on music perception and memory. Music Perception, 27(1), 43-59. This Journal Article is brought to you for free and open access by the Psychology at Haverford Scholarship. It has been accepted for inclusion in Faculty Publications by an authorized administrator of Haverford Scholarship. For more information, please contact [email protected]. Audiovisual Interactions: the Impact of Visual Information on Music Perception and Memory Author(s): Marilyn G. Boltz, Brittany Ebendorf , and Benjamin Field Source: Music Perception: An Interdisciplinary Journal, Vol. 27, No. 1 (September 2009), pp. 43-59 Published by: University of California Press Stable URL: http://www.jstor.org/stable/10.1525/mp.2009.27.1.43 . Accessed: 22/03/2013 15:04 Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at . http://www.jstor.org/page/info/about/policies/terms.jsp . JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact [email protected]. University of California Press is collaborating with JSTOR to digitize, preserve and extend access to Music Perception: An Interdisciplinary Journal. http://www.jstor.org This content downloaded from 165.82.168.47 on Fri, 22 Mar 2013 15:04:20 PM All use subject to JSTOR Terms and Conditions Music2701_04 8/12/09 3:01 PM Page 43 Audiovisual Interactions 43 AUDIOVISUAL INTERACTIONS:THE IMPACT OF VISUAL INFORMATION ON MUSIC PERCEPTION AND MEMORY MARILYN G. BOLTZ,BRITTANY EBENDORF, the mood and format of videos influence how a musi- AND BENJAMIN FIELD cal composition is heard and remembered. Haverford College The Influence of Music on the Film Experience PREVIOUS RESEARCH HAS DEMONSTRATED THAT MUSICAL soundtracks can influence the emotional impact, inter- As noted by several music and psychological theorists pretation, and remembering of visual information. The (e.g., Juslin & Sloboda, 2001; Meyer, 1956), the mean- present research examines the reverse relationship and ing of music, to a large extent, arises from the moods whether visual information influences the perception and emotions it evokes within listeners. A number of and memory of music. In Experiment 1, listeners were studies have examined people’s subjective reports, presented with affectively ambiguous tunes paired with adjective ratings, physiological responses, or nonverbal visual displays varying in their affect (positive, nega- expressions as they listen to various musical composi- tive) and format (video, montage), or a control condi- tions and found a high degree of agreement on what tion (no visual information at all). After each, particular emotions are elicited by a given musical participants were asked to provide a set of perceptual selection (see Sloboda & Juslin, 2001, for a review). In ratings that evaluated different melody characteristics addition, particular emotions are reliably associated and qualities of the visual displays. Results showed that with particular acoustical characteristics that general- both the affect and format of visual information differ- ize across different compositions. For example, sad entially influenced the way a melody was perceived. melodies tend to be characterized by a slow tempo, Experiment 2 extended these findings by revealing that lower amplitude, minor mode, and a narrow pitch the affect of visual displays distorted melody recogni- range in the lower octaves. As one might expect, happy tion in a mood congruent fashion. These results are dis- melodies tend to display a reverse set of structural cussed in terms of their theoretical implications for characteristics (i.e., faster tempo, higher amplitude, audiovisual processing. wider pitch range, etc.; Gabrielsson & Lindstrom, 2001; Hevner, 1935a, 1935b; Juslin, 1997; Scherer & Received September 10, 2008, accepted February 27, 2009. Oshinsky, 1977). Some individuals have described this Key words: audiovisual, music videos, musical relationship between emotional expression and musi- soundtracks, video information, music/film cal structure as an iconic one in which the particular attributes corresponding to a given emotion bear a resemblance to those found in nonmusical entities. Given that the structural correlates of different emo- USIC IS AN INTEGRAL ASPECT OF daily life tions are remarkably invariant across a wide variety of and often is encountered within a visual con- everyday events, including those of music, the human Mtext such as film, advertising, or an artistic voice (Cosmides, 1983; Scherer, 1979; Williams & performance. During the past several decades, a sub- Stevens, 1979), and walking gaits (Montepare, stantial amount of research has revealed various ways Goldstein, & Clauser, 1987; Sloman, Berridge, in which music can influence both the perception and Homatidis, Hunter, & Duck, 1982), there is, in fact, memory of visual scenes (see Cohen, 2001, for a ample opportunity to learn these types of associations. review). However, surprisingly few studies have exam- One illustration of emotional iconics comes from a ined the reverse relationship and ways in which visual study by Cohen (1993). Participants were asked to pro- information may influence the cognitive processing of vide adjective ratings in response to musical patterns music. The present research addresses this issue that varied in pitch height and tempo, as well as scenes through the use of music videos and considers whether of a bouncing ball that also varied in height and Music Perception VOLUME 27, ISSUE 1, PP. 43–59, ISSN 0730-7829, ELECTRONIC ISSN 1533-8312 © 2009 BY THE REGENTS OF THE UNIVERSITY OF CALIFORNIA. ALL RIGHTS RESERVED. PLEASE DIRECT ALL REQUESTS FOR PERMISSION TO PHOTOCOPY OR REPRODUCE ARTICLE CONTENT THROUGH THE UNIVERSITY OF CALIFORNIA PRESS’S RIGHTS AND PERMISSIONS WEBSITE, HTTP://WWW.UCPRESSJOURNALS.COM/REPRINTINFO.ASP. DOI:10.1525/MP.2009.27.1.43 This content downloaded from 165.82.168.47 on Fri, 22 Mar 2013 15:04:20 PM All use subject to JSTOR Terms and Conditions Music2701_04 8/12/09 3:01 PM Page 44 44 Marilyn G. Boltz, Brittany Ebendorf, & Benjamin Field tempo. When each stimulus was judged independently, Music also guided attending toward certain aspects of a increases or decreases along both physical dimensions visual scene that were consistent with this interpretative led to the same emotional judgment (happiness and framework and thereby influenced which particular items sadness, respectively) and these ratings were either were later remembered. Related research has revealed enhanced or attenuated when the two stimuli were that music’s attentional highlighting of information paired in a congruent or incongruent fashion. leads to a more integrated memory code that enhances The concept of an iconic relationship becomes even the ability to later recall a film (Boltz, 2003; Boltz, more apparent in cinematic art in which musical Schulkind, & Kantra, 1991). soundtracks commonly appear within the background of the ongoing action. By using music that evokes a The Influence of Visual Information mood similar to the visual story, a film director can on the Processing of Music heighten the emotional impact of a particular scene and its effects upon listeners. Consider an example used In the early 1980’s, MTV heightened the popularity of in one study in which the aggressive or playful behavior a relatively new art genre, the music video, which of wolves (within a documentary film) was accompa- immediately generated a debate about its predicted nied by music that had been independently described effectiveness. Some argued that an accompanying by listeners as aggressive vs. playful, respectively. video would decrease the appeal of a song by dictating Relative to the same scenes viewed in the absence of a particular interpretation that limits a listener’s imag- music, mood-congruent soundtracks increased the ination (e.g., Zorn, 1984). Others claimed that videos magnitude of the perceived aggressiveness and playful- would enhance the aesthetic appeal of music by clari- ness exhibited by these animals (Bolivar, Cohen, & fying a song’s intended meaning (e.g., Sun & Lull, Fentress, 1994). Mood-incongruent music not only 1986). Although this debate is unresolved, there is evi- attenuates the emotional impact of a scene but often is dence indicating that videos can help to prevent “wear used to convey a more subtle meaning through “ironic out”—becoming tired of a song due to excessive expo- contrast.” For many, scenes from A Clockwork Orange sure. Goldberg, Chattopadhyay, Gorn, & Rosenblatt readily will come to mind, in which the acts of murder, (1993) investigated this phenomenon and found that rape, and mayhem acquire a surreal meaning from the videos are most likely to maintain listeners’ interest carefree beauty of Rossini’s The Thieving Magpie. when songs are relatively ambiguous and open to mul- In addition