CONNECTING THE PERCEPTION OF EMOTION TO MUSIC SIGNALS by PAMELA A. WOOD M.S., Computer Science, University of Colorado Colorado Springs, May 2001 B.A., Mathematics, State University of New York at Buffalo, February 1980 A dissertation submitted to the Graduate Faculty of the University of Colorado Colorado Springs in partial fufillmnent of the requirements for the degree of Doctor of Philosophy Department of Computer Science School of Engineering and Applied Science May 2016 ii ➞ by Pamela A. Wood 2016 All Rights Reserved. As is a common practice in Computer Science, journal and conference articles are published based on the MS/Ph.D thesis work, which means that wording, tables, figures, and sentence structure are sometimes identical in the dissertation document, and journal and conference articles co-authored by the student and their advisor. The author acknowledges that the following articles: [91, 92] have been published based on this thesis. iii This dissertation for Doctor of Philosophy degree in Engineering - Concentration in Computer Science by Pamela A. Wood has been approved for the Department of Computer Science School of Engineering and Applied Science University of Colorado Colorado Springs Dr. Sudhanshu K. Semwal, Advisor Dr. Albert Glock Dr. Gregory L. Plett Dr. Jonathan Ventura Dr. Kristen Walcott-Justice Date IRB protocol #15-226 Wood, Pamela A. (Ph.D., Engineering) Connecting the Perception of Emotion to Music Signals Dissertation directed by Dr. Sudhanshu K. Semwal Determining the emotional impact of a particular musical excerpt can be challenging when dealing with human perceptions. It is not possible to ensure that each person will have a comparable experience. In addition, cultural differences may influence musical preference. This dissertation studies the linear relationship between signal processing data ex- tracted from a musical excerpt and user perceived emotion data in order to classify the musical excerpt by the emotion it evokes. When using only perceived emotion ratings that fell within one standard deviation of the mean rating, we were able to identify a relationship between music signal data and perceived emotion within a single genre using a weighing algorithm generated using the Weka linear regression classifier. Once a relationship was identified, the same process was run on the marching band music in order to determine if the orchestral signal weightings would correctly predict the ground truth emotion data for the band music. It became apparent that neither the or- chestral nor the band signal weightings would correctly identify the other genre’s ground truth emotion. A combined signal weighing for the orchestra and band music was more suc- cessful but there was insufficient data to conclusively say whether or not a single weighing algorithm would be able to accurately predict perceived emotions in more than one genre. In comparing our results to that of Eerola, et al. [27], it is apparent that expanding the emotional content of the music studied significantly reduced the correlation coefficients to less than 40% for the eight emotion sliders and less than 50% for the energy level slider. This applied to both the Linear Regression and the Partial Least Squares classification algorithms. We believe that this exposes a flaw in the current methods being employed to classify music. By limiting the music to specific emotions, it creates an artificially linear v input dataset causing the linear based algorithms to appear to be more effective than they actually are. An important outcome of this research is the creation of a music signal database which includes the ground truth data collected for the music excerpts. DEDICATION This dissertation is dedicated to my husband, Jeff and daughters, Rachel and Michelle. Their love and support made my success possible. I would also like to dedicate this dissertation to the memory of Dr. Margaret Gram, who encouraged me to go to college, major in math (even though I was a girl) and, most of all, believe in myself. vii ACKNOWLEDGEMENTS I extend my heartfelt appreciation to my advisor and committee chair, Dr. Sudhanshu K. Semwal for his unwavering support and assistance in my journey toward my Ph.D. His guidance and patience made my success possible. I am also very grateful for the time that my committee members, Dr. Albert Glock, Dr. Gregory Plett, Dr. Kristen Walcott-Justice and Dr. Jonathan Ventura spent, speaking with me and providing constructive criticism on my research. I am especially grateful for the input I received from Dr. Jonathan Ventura, who’s insight into prior research, enabled me to significantly improve my dissertation. I am indebted to the members of the Graphic, Media and Integration Consortium members, Khalid Al Harbi, Hans Cox, Matthew Higgins, Irving Rynning, Bonnie Snyder and Dr. Dana Wortman for the feedback they provided on the “trial run” of both my topic and my dissertation presentations. Their helpful criticisms enable me to, um, get it right. I am also grateful to the following former and current staff in the Computer Science Office, for the various forms of support they provided me during my graduate study, Patricia Rea and Alessandra Langfels. Most importantly, I want to thank my husband, Jeff and my daughters, Rachel and Michelle for their emotional support as well as the invaluable help they provided proof reading my dissertation. viii TABLE OF CONTENTS CHAPTER 1 INTRODUCTION 1 1.1 ProblemStatement................................ 3 1.2 Impact....................................... 4 1.3 PerspectivefromotherDisciplines . 6 1.3.1 Neurobiological . 7 1.3.2 Psychological . 7 1.3.3 Musicology . 10 2 PREVIOUS RESEARCH 14 2.1 An Exploration of the Effects of Sound in Story Telling . 14 2.1.1 History . 15 2.2 Implementation.................................. 18 2.2.1 MusicSelectionProcess ......................... 19 2.2.2 Perceived Mood Identification . 20 2.2.3 PerceivedMoodRating ......................... 20 2.2.4 An Exploration of the Effects of Sound in Story Telling Results and Conclusion ................................ 23 3 BACKGROUND AND RELATED WORK 27 3.1 Electronic Music Emotion Classification . 27 ix 3.2 Automating Music Retrieval and Recommender Systems . 32 3.3 Signal Processing and Clustering Tools . 33 4 RUNNING AN EXTERNAL SURVEY ON AMAZON MECHANICAL TURK 36 4.1 Amazon Mechanical Turk Validity . 38 4.2 MTurkExternalSurveyDesign . .. 41 4.2.1 MTurk Interface . 41 4.2.2 TheExternalSurveyWebsite . 45 4.3 MTurkExternalSurveyTesting. 47 5 ALGORITHMS USED 49 5.1 Multiple Linear Regression . 49 5.2 Partial Least Squares . 50 5.3 Data Set Reliability Algorithms . 51 5.3.1 Calculating Internal Consistency - Cronbach’s Alpha . 51 5.3.2 Calculating Consensus - Krippendorff’s Alpha . 52 5.4 Calculating the Coefficient of Determination - R2 . 52 6 SINGLE GENRE RESEARCH DESCRIPTION 54 6.1 MusicSelectionandPreprocessing . 55 6.2 MusicSignalProcessing ............................. 55 6.3 OrchestraMusicSurvey ............................. 58 6.4 Data Analysis . 66 6.4.1 Clean and verify the survey data . 66 6.5 SingleGenreResults ............................... 70 7 CROSS GENRE RESEARCH DESCRIPTION 96 7.1 MusicSelectionandPreprocessing . 98 x 7.2 MusicSignalProcessing ............................. 99 7.3 BandMusicSurvey................................ 99 7.4 Data Analysis . 102 7.5 CrossGenreResults ............................... 102 8 COMPARING LINEAR REGRESSION CLASSIFIER TO PARTIAL LEAST SQUARES CLASSIFIER 174 8.1 Data Set Reliability . 175 9 CONTRIBUTIONS 252 10 FUTURE WORK 254 10.1 Emotion to Signal Mapping Library . 254 10.2 Emotion Vector Mapping Algorithm . 255 10.3 Cross Genre Music Weighing Algorithm . 255 10.4 Graphical User Interface for Music to Signal Mapping Functions . 256 10.5 Additional Survey Questions . 256 10.6 Cross Cultural Music Weighing Algorithms . 256 11 CONCLUSION 257 BIBLIOGRAPHY 260 APPENDICES A IRB APPROVAL LETTER 266 B DATABASE STRUCTURES 268 B.1 MYSQL Database Tables . 268 B.2 Access Database Tables . 271 xi B.3 Access Database Queries . 272 C WEBSITE SOFTWARE CODE 275 C.1 surveyInit.php .................................. 275 C.2 contactUs.php . 282 C.3 dbconnect2015.php . 282 C.4 displayForm.php ................................. 283 C.5 getNextMusicID.php ............................... 283 C.6 initUser.php.................................... 285 C.7 nextSurvey.php .................................. 285 C.8 RateMusicSliders.php . 287 C.9 saveSliderValues.php . 288 C.10sendEmail.php .................................. 289 C.11 ThankYou.php . 290 D DATA MANIPULATION SOFTWARE CODE 292 D.1 preprocessData .................................. 292 D.2 musicStats..................................... 302 D.3 RatingsToSignalMapping . 345 D.4 musicStatsBoth.................................. 366 D.5 ProjectEmotionRating . 383 D.6 CompareSignalWeighingToSurvey . 390 xii TABLES Table 1.1 Music’s impact on memory of an accompanying film clip [11] [10] [76]. 9 2.1 Musical Selections and Performers . 26 3.1 MIRtoolbox functions used, a description of the function, and the music features they are related to [53]. 33 3.2 MIRtoolbox functions used to electronically categorize the music excerpts [53]. 35 4.1 Amount of Pay vs Time to Complete MTurk Survey . 39 4.2 Number of Responses received for Hits in both Orchestra Survey and Band Survey....................................... 40 6.1 Symphonie Fantastique Movement and Associated Snippet Breakdown . 58 6.2 Breakdown of MTurk Workers Participating in Orchestra Survey by Country 59 6.3 Breakdown