Detecting Emotional response to music using near-infrared spectroscopy of the prefrontal cortex by Saba Moghimi A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Institute of Biomaterials and Biomedical Engineering University of Toronto ⃝c Copyright 2013 by Saba Moghimi Abstract Detecting Emotional response to music using near-infrared spectroscopy of the prefrontal cortex Saba Moghimi Doctor of Philosophy Graduate Department of Institute of Biomaterials and Biomedical Engineering University of Toronto 2013 Many individuals with severe motor disabilities may not be able to use conventional means of emotion expression (e.g. vocalization, facial expression) to make their emo- tions known to others. Lack of a means for expressing emotions may adversely affect the quality of life of these individuals and their families. The main objective of this thesis was to implement a non-invasive means of identifying emotional arousal (neutral vs. intense) and valence (positive vs. negative) by directly using brain activity. In this light, near infrared spectroscopy (NIRS), which optically measures oxygenated and deoxygenated hemoglobin concentrations ([HbO2] and [Hb], respectively), was used to monitor prefrontal cortex hemodynamics in 10 individuals as they listened to music ex- cerpts. Participants provided subjective ratings of arousal and valence. With respect to valence and arousal, prefrontal cortex [HbO2] and [Hb] were characterized and significant prefrontal cortex hemodynamic modulations were identified due to emotions. These mod- ulations were not significantly related to the characteristics of the music excerpts used for inducing emotions. These early investigations provided evidence for the use of pre- frontal cortex NIRS in identifying emotions. Next, using features extracted from [HbO2] and [Hb] in the prefrontal cortex, an average accuracy of 71% was achieved in identifying arousal and valence. Novel hemodynamic features extracted using dynamic modeling and template-matching were introduced for identifying arousal and valence. Ultimately, the ii ability of autonomic nervous system (ANS) signals including heart rate, electrodermal activity and skin temperature to improve the identification results, achieved when using PFC [HbO2] and [Hb] exclusively, was investigated. For the majority of the participants, prefrontal cortex NIRS-based identification achieved higher classification accuracies than combined ANS and NIRS features. The results indicated that NIRS recordings of the prefrontal cortex during presentation of music with emotional content can be automat- ically decoded in terms of both valence and arousal encouraging future investigation of NIRS-based emotion detection in individuals with severe disabilities. iii Dedication To Hope and Trinity for inspiring me to pursue this work. iv Acknowledgements I would like to thank my supervisor Dr. Tom Chau for his kind help and all his support throughout my work. I will be forever indebted to him for giving me the chance to be part of his dynamic research team. His mentorship has helped me develop skills that I will carry for the rest of my life. My special thanks to my co-supervisor Dr. Anne-Marie Guerguerian for sharing her knowledge and supporting me throughout the challenges I faced. Her unwavering care and concern for the patients has always been a source of inspiration to me. I would like to thank my committee members Dr. Maureen Dennis and Dr. Milos Popovic for sharing their insight, and guiding me with their suggestions. I would like to express my gratitude to Dr. Azadeh Kushki and Dr. Sarah Power for their kind help throughout my research. I am also grateful to Ka Lun Tam and Pierre Duez for their technical support. I would like to express my gratitude to Dr. Negar Memarian and Dr. Stefanie Blain-Moraes for helping me in developing my research skills. I would like to thank the participants who took the time to help me with this study, without whom this work would have not been possible. I acknowledge the financial support of the National Science and Engineering Research Council CREATE CARE program, and Holland Bloorview Kids Rehabilitation Hospital graduate scholarship. I would like to thank donors of the K.M. Peterborough Hunter graduate studentship for their financial support. Finally, I would like to express my gratitude to my family whose love and support has always embraced me although they are miles and miles away. I would like to thank my father for all his contributions. His interest in my work and our discussions truly motivated me in my research. I thank my mother and my aunt Ferreshteh who reminded me to be strong and determined throughout my work. Special thanks to my sister who helped me in so many ways from encouraging me in my work to sharing her technical insight. Finally, my special thanks to Amin Abdossalami for reminding me to never give v up. vi Contents 1 Introduction 1 1.1 Preamble . 1 1.2 Motivation . 1 1.3 Current clinical evidence for EEG-based BCIs, a literature appraisal . 3 1.3.1 BCI Development Using Electroencephalography . 5 1.3.2 Applications User Interface . 7 1.3.3 Controlling brain computer interfaces . 7 1.3.4 Evaluation Criteria . 12 1.3.5 Future Directions in BCI research . 13 1.3.6 Towards affective brain computer interfaces . 16 1.4 Neural correlates of emotion . 18 1.4.1 The role of prefrontal cortex in default, salient and executive con- trol networks . 20 1.5 Near-infrared spectroscopy of the brain . 21 1.6 Emotion induction via music . 22 1.7 Objectives . 25 1.8 Roadmap . 25 2 Experimental Protocol 29 2.1 Preamble . 29 vii 2.2 Introduction . 29 2.3 Participants . 29 2.4 Stimuli . 30 2.5 Signal acquisition . 31 2.6 Pre-processing . 31 2.7 Study design . 32 3 Characterizing PFC Hemodynamic changes due valence and arousal 35 3.1 Preamble . 35 3.2 Abstract . 36 3.3 Introduction . 36 3.4 Methods . 38 3.4.1 Procedures . 38 3.4.2 Wavelet-based peak detection . 40 3.4.3 Statistical analysis . 42 3.5 Results . 42 3.6 Discussion . 45 4 The Effect of Music Characteristics 47 4.1 Preamble . 47 4.2 Introduction . 47 4.3 Methods . 49 4.3.1 Music characteristic extraction . 49 4.3.2 Music database . 50 4.3.3 Statistical analysis . 50 4.4 Results . 51 4.5 Discussion . 52 4.5.1 Subject specific patterns . 53 viii 4.5.2 Temporal dynamics . 54 4.6 Conclusion . 54 5 Automatic Detection of Emotional Response to Music 55 5.1 Preamble . 55 5.2 Abstract . 56 5.3 Introduction . 57 5.4 Methods . 59 5.4.1 Stimuli . 59 5.4.2 Preprocessing . 61 5.4.3 Feature extraction . 61 5.4.4 Classification procedures . 62 5.5 Results . 63 5.6 Discussion . 66 5.6.1 Classification Accuracy . 66 5.6.2 Diversity in the music database . 69 5.6.3 Challenges . 69 6 Combining autonomic and central nervous system activity 71 6.1 Preamble . 71 6.2 Introduction . 72 6.3 Methods . 73 6.3.1 Procedures . 73 6.3.2 NIRS data . 74 6.3.3 ANS data . 75 6.3.4 Analysis . 75 6.3.5 Feature extraction . 76 6.3.6 Classification . 80 ix 6.3.7 Mixture of experts . 80 6.4 Results . 83 6.4.1 Dynamic model-based features . 84 6.4.2 Classification results . 84 6.5 Discussion . 85 6.6 Conclusion . 88 7 Concluding remarks 89 7.1 Summary of contributions . 89 7.1.1 A literature appraisal of the existing evidence for the use of BCI for individuals with disabilities [143] . 89 7.1.2 PFC [Hb] and [HbO2] patterns characterization using wavelet anal- ysis with respect to emotional arousal and valence [142] . 90 7.1.3 Identified emotional arousal and valence in response to dynamic emotion induction using PFC NIRS [144] . 90 7.1.4 Introduced features based on dynamic modeling for emotion iden- tification . 91 7.1.5 Multi-modal emotion identification using a mixture of classifier ex- perts . 91 7.2 Recommendation for future studies . 92 7.2.1 Assessing PFC hemodynamics for emotion identification in the pe- diatric population and individuals with severe disabilities . 92 7.2.2 Potential clinical implications . 93 7.2.3 Dynamic emotional rating paradigms . 94 7.2.4 Emotional sensitivity measures . 94 7.2.5 Individual specific analysis . 94 7.2.6 Inclusion of larger sample sizes . 95 x Appendix A: Open Challenges Regarding Control Mechanisms 96 Appendix B: Music Database 100 Appendix C: Music characteristic extraction using MIRTOOLBOX 103 Appendix D: Region specific analysis of [HbO2] and [Hb] with respect to music characteristics 104 Appendix E: Contributions from Systemic Blood Flow 105 Appendix.