Detection and Analysis of Emotional Highlights in Film
Total Page:16
File Type:pdf, Size:1020Kb
DETECTION AND ANALYSIS OF EMOTIONAL HIGHLIGHTS IN FILM h dissexhttion submitted ia fulf bent pf the x~q&cments for -the award of to the DCU ,Geqtxe fol--Digieal Video Processing Bcbaal ed Computing Ddblia City Un3vtrdty September, 5007 ~t~kk~m&,&d%rn,~dk dkmt ~WWQ~hu wkddd$~'d d-d q ad& ~&IJOR DECLARATION 1 hereby certify that this material, which 1 now submit fnr assessment on thc progmmme of study leacling to the award of Mnster of Science is eiltircly my own work md has not teen ~kcnfrom the work of others save and to thc extent that SUCIZ ~vorkI-tns bcetl cited and acknoxvivdged within the text of my worlc. ABSTRACT Thls work explores the emotional experience of viewing a film or movie. We seek to invesagate the supposition that emotional highlights in feature films can be detected through analysis of viewers' involuntary physiological reactions. We employ an empirical approach to the investigation of this hypothesis, which we will discuss in detail, culminating in a detailed analysis of the results of the experimentation carried out during the course of this work. An experiment, known as the CDVPlex, was conducted in order to compile a ground-truth of human subject responses to stimuli in film. This was achieved by monitoring and recordng physiological reactions of people as they viewed a large selection of films in a controlled cinema-like environment using a range of biometric measurement devices, both wearable and integrated. In order to obtain a ground truth of the emotions actually present in a film, a selection of the films used in the CDVPlex were manually annotated for a defined set of emotions. We examine how filmmakers use devices and techniques to stimulate viewers' emotions, particularly how music is used in film to intensify the impact of onscreen action. We also examine the different event types of which film scenes are comprised and how they can be detected using audio-visual analysis of the video content. Finally, we calculate and study the correlations between the emotions found in the annotated films, and "events" or highhghts in the viewers' biometric measurements, and also between the scene events and music occurring in the film and the emotions present. To Stephen 1uho got me through ACKNOWLEDGEMENTS This work is the culmination of many people's belief in me and my ability to acheve my goals, however they may have changed over the past couple of years. To my colleagues and friends of the CDVP; I feel privileged to have had the opportunity to work alongside such a talented group of researchers. My days in L1.31 have given me many enduring memories of friendship, comradeship. .and a certain wachness! My everlasting thanks to the gang (past and present): Sintad, Cathal, Pete, James, Paul F., Colum, Uke, Hyowon, Neil, Fabrice, Paul B., Jiamin, IQeran, Dave, Bart and 0rla. Most of all I wish to thank Georgina who, no matter what, is always there for me as my &end and counsellor. I wish to thank my supervisor Prof. Alan Smeaton for hs support in hfficult times. This list would not be complete without mentioning some friends who were always on hand to listen and help - for ths, I'd like to thank Annette E., D6nal and Jean. To my famdy, Ivan, Geraldine, Lorraine and Alison, who know me better than I know myself, goes my heartfelt gratitude. They have always encouraged and supported me in whatever I chose to do. My parents' good advice and their unwavering belief in me have been my bedrock. Thank you for always listening and so much more. Last, but far from least, to Steve - saylng "thank you" seems almost a paltry phrase to acknowledge his part in the completion of ths work. I could not have done this without him; he picked me up when I thought I couldn't go on, his brilliant mind was my soundtng board and inspiration, his constant support is the rock on which I lean. My love and gratitude always. CONTENTS Acknowledgements List of Tables List of Fignres xii 1 Introduction 1.I Multimedia In €01-nwtionRetrie17d 1.1 .I Vidco in thc Digit:ital Age 1.I .2 Video Tn formation Retrieval 3.1.3 Cliall~ngesin Viclco 1nforrnn tion Retricv~I 1.1.4 IndcxingWde~ I ,I .5 Autoima tic Summaiisation t .2 Emotion Dutcction in Film 1.2.1 Researcl~Questions hcldresscd in this 'Shcais 3.3 Tl~esisStruckme 2 The Film Scare and Cinematic Emotion 2.1 The Psycl~olngyof Music 2.1.1 Perception of Emotion in Music 2.2 Music and the Movies 2.2.1 Historical Perspective 2.2.2 The Effect of Filrn Music 2.3 The Classical Hollywood Model 2.3.1 The Five Principles of Classical Filrn Music 2.4 Studying the Effect Of Movie Music 3 The CDVPlex Biometric Cinema 3.1 Sensing Physiological Reactions 3.1.1 Biometric Measurements 3.1.2 Sensor Devices Used 3.2 The CDVPlex Experiment 3.2.1 The Cinema 3.2.2 The Movies 3.2.3 The Viewers 3.2.4 The Procedure 3.3 Biometric Data 3.3.1 GSR Peak Detection 3.3.2 Heart-Rate Peak Detection 3.3.3 Motion Peak Detection 3.3.4 Summary of Biometric Peak Detection 4 Audio-Visual Analysis of Feature Films 4.1 Film Grammar 4.1.1 Film Grammar and Events 4.1.2 Scene-Based vs. Event-Based Structures 4.2 Event Detection 4.2.1 Filrnrnakmg Techniques 4.2.2 Event Classes 4.2.3 Uses for Automatic Event Detection 4.3 Audio-Visual Analysis 4.3.1 Low-Level Features 4.3.2 Generating a Shot-Level Feature Vector 4.3.3 High-Level Event Detection 5 Manual Annotation of Feature Films 5.1 Emotion Tag Set 5.1.1 Sets of Emotion Types 5.1.2 Emotion Tag Set Chosen for Annotation 5.2 The Annotation Process 5.2.1 Music-Rich Movie Segments 5.2.2 The Manual Annotation Process 5.3 Manually Annotated Feature Films 5.3.1 Movie Genres 5.3.2 Feature Film Selection 5.4 Annotation Results 5.4.1 Music Detection 5.4.2 Emotion Annotation Results 6 Results and Analysis 6.1 Analysis of Film Viewings 6.1.1 Peak Detection and Sirmlarity Per Viewing 6.1.2 Comparison of Viewings Across All Movies 6.2 Annotation vs. Detection 6.2.1 Detection Rate per Emotion 6.2.2 Emotion Detection Using Norrnalised Results 6.3 The Effect of Music & Events 6.3.1 Music and Events 6.3.2 Music and Events' Influence on Emotional Responses 6.3.3 Degree of Emotional Influence of Music and Events 7 Conclusions and Future Work 7.1 Thesis Summary 7.2 Summary Of Findings 7.3 Future Work A .I Pitch A.2 .I I*irnbrc A,1.2 Scalcs,I<cysandModes A.1.3 FIa~mony 14.2 Rhythm A.2.1 Metrc A.2.2 Phrasing A.3 Melody A.4 Genre A.5 Narrative B CDVPlex Filmograpy C CDVPlex Documentation D Details of Analysed Feature Films Bibliogaphy LIST OF FIGURES figire 3.1(a): Polar SGl OiTM Components 30 1:igure 3.l(b): RodyMcclia SensetVearO PRO2 Armband 31 re2 Age Range of Viewers who Participated in the CDVI>/P-I- 34 Figure 3.2(b): Vielngs of Previously Seen or Unseen Films by J'olunteers in thc CDVPhx 34 Figuve 3.2(c): Aspects of r-he 1:ilrns in the CDV P/~xas Favoured by the Viewers 35 Fiprc 3.2(d): Typical Cinern~Attcndancc of CDVP/cxViewcrs 35 FiLgure3.3 (a): GSR Peak Detcctian per Film Viewing 42 Fk~re3.3(b): GSRPrnlrsKelati~~eT\lla~ni~idcpcrVi~~vin~ 46 T;igul-c 3.3(c): Hear-t-Rate Peak Dcrecti(~nper Vilicwing 51 k~ure3.3(d): TJenrt-ltatc l'cs ks Ttellntivu Magilituclc per VicwinR 54 figure 3.3(c): Mociot~T'cak Dctcction per Viewing 58 Fir3 Motion Illeaks Rchtke Magnitude pcr Relcvnnt Vir\Ying h0 Figure 3.3k;): Ovc~rvinvof No. Riametric 13eaks Detected Pcr Vidng 62 Figure 41 (n): Lchanc's Smicmre nf a Feature Film 120061 6fi Figure 4.2(a): Lehane's Event Detection System [2006] 72 Figure 5.1(a) : Ortony's Emotion Structure Framework 81 Figure 5.2(a) : Labelled Screen Shot of the Movie Annotation Spreadsheet Matrix 85 Figure 5.2@): Screen Shot of a Movie Annotation in Progress 86 Figure 5.3(a): Most Popular Film Genre Categories as per CDVl3lexParticipants 87 Figure 5.4(a): Legend for Emotion Types as depicted in Annotation Results 8 9 Figure 5.4@): Average Occurrence of Emotions over all Manually Annotated Fhs 90 Figure 5.4(c) : Distribution of emotions in Finding Nemo 9 1 Figure 5.4(d) : Distribution of emotions in Pirates ofthe Caribbean 9 1 Figure 5.4 (e) : Distribution of emotions in IndianaJones and the Last Cmade 91 Figure 5.4(9: Distribution of emotions in Warty Potter and the Prisoner ofApkaban 91 Figure 5.4(g) : Distribution of emotions in Casablanca 9 1 Figure 5.4@): Distribution of emotions in Mean Streets 9 1 Figure 5.4 (i) : Distribution of emotions, with Music-Rich Sections indicated, in Finding Nemo Figure 5.4(j): Distribution of emotions, with Music-Rich Sections indicated, in Pirdtes ofthe Cadbean 92 Figure 5.4(k): Distribution of emotions, with Music-Rich Sections indicated, in Indiana Jones and the Last Crusade 92 Figure 5.40: Distribution of emotions, with Music-Rich Sections indicated, in Hay Potter and the Ptisoner oJAxkaban Figure 5.4(m): Distribution of emotions, with Music-Rich Sections indicated, in Casablanca Figure 5.4(n): Distribution of emotions, with Music-Rich Sections indicated, in Mean Streets Figure 5.4(0): Frequency of emotions in Finding Nemo Figure 5.4(p): Frequency of emotions w.r.t.