The Behavioral and Neural Effects of Audiovisual Affective Processing
Total Page:16
File Type:pdf, Size:1020Kb
University of South Carolina Scholar Commons Theses and Dissertations Summer 2020 The Behavioral and Neural Effects of Audiovisual Affective Processing Chuanji Gao Follow this and additional works at: https://scholarcommons.sc.edu/etd Part of the Experimental Analysis of Behavior Commons Recommended Citation Gao, C.(2020). The Behavioral and Neural Effects of Audiovisual Affective Processing. (Doctoral dissertation). Retrieved from https://scholarcommons.sc.edu/etd/5989 This Open Access Dissertation is brought to you by Scholar Commons. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of Scholar Commons. For more information, please contact [email protected]. THE BEHAVIORAL AND NEURAL EFFECTS OF AUDIOVISUAL AFFECTIVE PROCESSING by Chuanji Gao Bachelor of Arts Heilongjiang University, 2011 Master of Science Capital Normal University, 2015 Submitted in Partial Fulfillment of the Requirements For the Degree of Doctor of Philosophy in Experimental Psychology College of Arts and Sciences University of South Carolina 2020 Accepted by: Svetlana V. Shinkareva, Major Professor Douglas H. Wedell, Committee Member Rutvik H. Desai, Committee Member Roozbeh Behroozmand, Committee Member Cheryl L. Addy, Vice Provost and Dean of the Graduate School © Copyright by Chuanji Gao, 2020 All Rights Reserved. ii DEDICATION To my family iii ACKNOWLEDGEMENTS I have been incredibly lucky to have my advisor, Svetlana Shinkareva. She is an exceptional mentor who guides me through the course of my doctoral research and provides continuous support. I thank Doug Wedell for his guidance and collaborations over the years, I have learned a lot. I am also grateful to other members of my dissertation committee, Rutvik Desai and Roozbeh Behroozmand for their invaluable feedback and help. I would also like to thank my labmates and friends in the department for their support and friendship. I also appreciate the help of all the research assistants. I’m also grateful to many colleagues, graduate student peers in department I cannot enumerate here. Finally, I want to thank my family and especially my fiancée Xi, for their endless love. iv ABSTRACT In everyday life, we receive affective information from a multisensory environment. What we see and what we hear jointly influence how we feel, think and act. Outstanding questions still remain about the essential behavioral and neural mechanism underlying how we combine visual and auditory affective signals. In this dissertation, I report a series of behavioral, EEG and fMRI experiments addressing this question. I found behaviorally there are congruency, visual dominance, and negativity dominance effects. Using ERPs, I showed that these behavioral effects can map onto different time course in audiovisual affective processing. Time-frequency analyses of EEG data showed that there are early sub-additive evoked theta, long-lasting supra-additive induced delta and beta activities. Meta-analysis of previous neuroimaging studies revealed the role of superior temporal cortex, amygdala, and thalamus in audiovisual affective processing. In an fMRI study, brain areas associated with audiovisual affective congruence and valence processing were identified, wherein superior temporal and anterior cingulate cortices have roles in both processes. Representational similarity analyses revealed modality- general brain areas that are sensitive to valence from both visual and auditory modalities; and modality-specific brain areas that are sensitive to either visual or auditory emotions. Together these convergent findings advance our understanding of behavioral and neural effects of audiovisual affective processing. v TABLE OF CONTENTS Dedication .......................................................................................................................... iii Acknowledgements ............................................................................................................ iv Abstract ............................................................................................................................... v List of Tables ..................................................................................................................... ix List of Figures ..................................................................................................................... x Chapter 1: Introduction ....................................................................................................... 1 1.1 Theoretical perspectives of emotion ........................................................... 1 1.2 The extant literature on audiovisual affective processing ........................... 3 1.3 Research gap in audiovisual affective processing ...................................... 7 1.4 Summary of following chapters ................................................................ 11 Chapter 2: Behavioral effects of audiovisual affective processing ................................... 16 2.1 Introduction ............................................................................................... 16 2.2 Experiment 1 ............................................................................................. 22 2.3 Experiment 2………………………………………………………….….33 2.4 Discussion………………………………………………………………..40 Chapter 3: Temporal dynamics of audiovisual affective processing ................................ 45 3.1 Introduction………………………………………………………………45 3.2 Method ...................................................................................................... 52 3.3 Results……………………………………………………………………61 3.4 Discussion………………………………………………………………..77 vi Chapter 4: EEG oscillations linked to audiovisual affective processing .......................... 85 4.1 Introduction………………………………………………………………85 4.2 Method…………………………………………………………………...89 4.3 Results……………………………………………………………………94 4.4 Discussion………………………………………………………………..98 Chapter 5: Meta-analysis of neuroimaging studies on audiovisual affective processing 102 4.1 Introduction……………………………………………………………..102 4.2 Method .................................................................................................... 107 4.3 Results…………………………………………………………………..113 4.4 Discussion………………………………………………………………117 Chapter 6: Decoding audiovisual affective congruence from fMRI data ....................... 125 6.1 Introduction……………………………………………………………..126 6.2 Method………………………………………………………………….129 6.3 Results…………………………………………………………………..137 6.4 Discussion………………………………………………………………143 Chapter 7: Representational similarity of audiovisual valence ....................................... 151 7.1 Introduction……………………………………………………………..153 7.2 Method………………………………………………………………….154 7.3 Results…………………………………………………………………..162 7.4 Discussion………………………………………………………………167 Chapter 8: General discussion ........................................................................................ 171 8.1 Summary of novel findings……………………………………………..171 8.2 Future directions ..................................................................................... 174 vii References ....................................................................................................................... 178 Appendix A: Permission to reprint ................................................................................. 194 viii LIST OF TABLES Table 3.1. Summary of effects from repeated measures ANOVAs on ERP data ..............65 Table 4.1. Studies investigating neural oscillations underlying audiovisual integration of emotion using EEG or MEG data ...........................................................87 Table 4.2. Valence and arousal ratings of the video and music stimuli .............................91 Table 5.1. Studies included in the meta-analysis .............................................................108 Table 5.2. General audiovisual affective processing network identified by ALE meta-analysis: Brain regions showing significant convergence across included studies examining neural correlates of audiovisual affective processing ...................114 Table 6.1. Mean valence and arousal ratings from prior norming studies. Means and standard deviations are shown ..............................................................................131 Table 6.2. Summary of multivariate searchlight results identifying brain regions that differentiate affectively congruent versus incongruent audiovisual trials ...........139 Table 6.3. Summary of multivariate searchlight results identifying brain regions that differentiate positive versus negative audiovisual trials ......................................140 Table 6.4. Summary of conjunction analysis results identifying brain regions associated with representations of both valence congruence across visual and auditory modalities and valence ..................................................................................141 Table 6.5. Post-scan mean valence ratings, with standard errors in parentheses .............142 Table 7.1. Result summary of the searchlight representational similarity analysis identifying model general affective brain regions.......................................................164 Table 7.2. Result summary of the searchlight representational similarity analysis identifying visual specific affective brain regions ......................................................165 Table 7.3. Result summary of the searchlight representational similarity analysis identifying auditory specific affective brain regions ..................................................166