Telling Stories with Soundtracks: An Empirical Analysis of Music in Film Jon Gillick David Bamman School of Information School of Information University of California, Berkeley University of California, Berkeley [email protected] [email protected] Abstract (Guha et al., 2015; Kociskˇ y` et al., 2017), natural language understanding (Frermann et al., 2017), Soundtracks play an important role in carry- ing the story of a film. In this work, we col- summarization (Gorinski and Lapata, 2015) and lect a corpus of movies and television shows image captioning (Zhu et al., 2015; Rohrbach matched with subtitles and soundtracks and et al., 2015, 2017; Tapaswi et al., 2015), the analyze the relationship between story, song, modalities examined are almost exclusively lim- and audience reception. We look at the con- ited to text and image. In this work, we present tent of a film through the lens of its latent top- a new perspective on multimodal storytelling by ics and at the content of a song through de- focusing on a so-far neglected aspect of narrative: scriptors of its musical attributes. In two ex- the role of music. periments, we find first that individual topics are strongly associated with musical attributes, We focus specifically on the ways in which 1 and second, that musical attributes of sound- soundtracks contribute to films, presenting a first tracks are predictive of film ratings, even after look from a computational modeling perspective controlling for topic and genre. into soundtracks as storytelling devices. By devel- oping models that connect films with musical pa- 1 Introduction rameters of soundtracks, we can gain insight into The medium of film is often taken to be a canon- musical choices both past and future. While a ical example of narrative multimodality: it com- great film score is in part determined by how well bines the written narrative of dialogue with the vi- it fits with the context of the story (Byrne, 2012), sual narrative of its imagery. While words bear we are also interested in uncovering musical as- the burden alone for creating compelling charac- pects that, in general, work better in support of a ters, scenes, and plot in textual narratives like lit- film. erary novels, in film, this responsibility is shared To move toward understanding both what by each of the contributors, including the screen- makes a film fit with a particular kind of song and writer, director, music supervisor, special effects what musical aspects can broadly be effective in engineers, and many others. Working together to the service of telling a story, we make the follow- support the overall story tends to make for more ing contributions: successful component parts; the Academy Award winner for Best Picture, for example, often also 1. We present a dataset of 41,143 films paired collects many other awards and nominations— with their soundtracks. Metadata for the films including acting, cinemotography, and sound de- is drawn from IMDB, linked to subtitle infor- sign. mation from OpenSubtitles2016 data (Lison While film has recently been studied as a tar- and Tiedemann, 2016), and soundtrack data get in natural language processing and computer is linked to structured audio information from vision for such tasks as characterizing gender rep- Spotify. resentation in dialogue (Ramakrishna et al., 2015; 2. We present empirical results demonstrating Agarwal et al., 2015), inferring character types the relationship between audio qualities of from plot summaries (Bamman et al., 2013), mea- the soundtrack and viewers’ responses to suring the memorability of phrasing (Danescu- 1We use the word film to refer to both movies and televi- Niculescu-Mizil et al., 2012), question answering sion shows interchangeably. 33 Proceedings of the First Workshop on Storytelling, pages 33–42 New Orleans, Louisiana, June 5, 2018. c 2018 Association for Computational Linguistics the films they appear in; soundtracks with dard for cinematic scoring practices and have been more instrumental or acoustic songs generate extensively analyzed in the film music literature higher ratings; “danceable” songs lower the (Slowik, 2012). Following this period, with the average ratings for films they appear in. rise of rock and roll, popular music began to make 3. We present empirical results demonstrating its way into film soundtracks in addition to the the relationship between the topics that make scores written specifically for the movie. As Rod- up a film script and the audio qualities of man(2006) points out, this turn coincided with di- the soundtrack. Films with settings in high rectors seeing the potential for songs to contribute school or college, for example, tend to have to the narrative meaning of the film: electric instrumentation and singing; sound- tracks with faster tempos appear both in films In The Blackboard Jungle, Bill Haley’s about zombies and vampires and in films in rock and roll anthem, ‘Rock Around the which the word dude appears frequently. Clock,’ was used in the opening cred- its, not only to capture the attention of 2 The Narrative Role of Music in Film the teenage audience, but also to sig- nify the rebellious energy of teenagers in The first films appeared around 1890, before the the 1950s. The Graduate relied upon development of technology that enabled synchro- the music and poetry of Simon and Gar- nization of picture with sound (Buhler et al., funkel to portray the alienation of Amer- 2010). While silent films featured no talking or ican youth of the 1960s. Easy Rider music in the film itself, they were often accom- took a more aggressive countercultural panied by music during live performances in the- stance by using the rock music of Hoyt atres. Rather than playing set scores, these live ac- Axton, Steppenwolf, The Byrds, and companiments were largely improvised; practical Jimi Hendrix to portray the youth rebel- catalogues for such performances describe the mu- lion in American society, complete with sical elements appropriate for emotions and narra- communes, long hair and drugs (Rod- tive situations in the film (Becce, 1919; Erdmann man, 2006, 123) et al., 1927). For example, Lang and West(1920) note that a string accompaniment with tremolo In recent years, the boundaries between popu- (trembling) effect is appropriate for “suspense and lar music and film music in the traditional sense impending disaster”; an organ tone with heavy have become increasingly blurred, pushed for- pedal is appropriate for “church scenes” and for ward especially by more affordable music produc- generally connoting “impressive dignity”; flutes tion technology including synthesizers and pre- are fitting for conveying “happiness,” “spring- recorded samples that allow a broad range of time” or “sunshine.” composers to use sounds previously reserved for With the rise of talkies in the late 1920’s those with access to a full orchestra (Pinch et al., (Slowik, 2012), music could be incorporated di- 2009). Though electronic music pioneers like rectly into the production of the film, and was of- Wendy Carlos have been composing for film since ten composed specifically for it; Gorbman(1987) the late 1960’s (Pinch et al., 2009), pop and elec- describes that in the classical model of film pro- tronic musicians have only gradually been recog- duction, scored music is “not meant to be heard nized as film composers in their own right, with consciously,” primarily acts as a signifier of emo- Daft Punk’s original score for Tron: Legacy in tion, and provides referential and narrative cues, 2010 marking a breakthrough into the mainstream such as establishing the setting or character. The (Anderson, 2012). use of Wagnerian leitmotif—the repeated associ- ation of a musical phrase with a film element, 3 Data such as a character—is common in original scores, In order to begin exploring the relationship be- especially in those for epic films (Prendergast, tween films and their soundtracks, we gather data 1992). from several different sources. First, we draw Works from the “Golden Age” of film mu- on the OpenSubtitles2016 data of parallel texts sic (the period between 1935–1950, shortly af- for film subtitles (Lison and Tiedemann, 2016); ter the rise of synchronized sound) set the stan- this dataset includes scripts for a wide variety of 34 movies and episodes of television shows (106,609 of Golden Slumbers by Jennifer Hudson (from the total in English) and contains publicly available movie Sing) were not in Spotify’s catalogue, it subtitle data. Each film in the OpenSubtitles2016 would match the performance by The Beatles on data is paired with its unique IMDB identifier; us- Abbey Road. ing this information, we extract IMDB metadata Spotify provides a number of extracted audio for the film, including title, year of release, av- features for each song; from a set of 13 we chose erage user rating (a real number from 0-10), and 5 that we hypothesized would be predictive of genre (a set of 28 categories, ranging from drama viewer preferences and whose descriptors are also and comedy to war and film-noir). interpretable enough to enable discussion. Those Most importantly, we also identify soundtrack that we include in our analysis are the follow- information on IMDB using this identifier; sound- ing, with descriptions drawn from Spotify’s Track tracks are listed on IMDB in the same form as they API:3 appear in the movie/television credits (generally also in the order of appearance of the song). A Mode. “Mode indicates the modality (major • typical example is the following: or minor) of a track, the type of scale from which its melodic content is derived. Major Golden Slumbers is represented by 1 and minor is 0.” Written by John Lennon and Paul McCartney Tempo. “The overall estimated tempo of a • Performed by Jennifer Hudson track in beats per minute (BPM).