The Perception of Emotions in Multimedia: An Empirical Test of Three Models of Conformance and Contest by KRITSACHAI SOMSAMAN Submitted in partial fulfillment of the requirements For the degree of Doctor of Philosophy Dissertation Advisor: Prof. Fred Collopy Department of Information Systems CASE WESTERN RESERVE UNIVERSITY May, 2004 CASE WESTERN RESERVE UNIVERSITY SCHOOL OF GRADUATE STUDIES We hereby approve the dissertation of ______________________________________________________ candidate for the Ph.D. degree *. (signed)_______________________________________________ (chair of the committee) ________________________________________________ ________________________________________________ ________________________________________________ ________________________________________________ ________________________________________________ (date) _______________________ *We also certify that written approval has been obtained for any proprietary material contained therein. Copyright © 2004 by Kritsachai Somsaman All right reserved ii To my beloved parents and my most beloved harmony of all – world peace. iii Table of Contents Page List of Tables ..................................................................................................................... 6 List of Figures.................................................................................................................... 8 Acknowledgements ......................................................................................................... 10 List of Abbreviations ...................................................................................................... 11 Abstract............................................................................................................................ 12 Chapter 1 Introduction................................................................................................... 14 Research Context .......................................................................................................... 14 Research Question ........................................................................................................ 16 Significance of research question ............................................................................. 17 Chapter 2 Theoretical Background............................................................................... 19 Definition of Emotions ................................................................................................. 19 Classification of emotions......................................................................................... 25 Perception of Emotions in Multimedia......................................................................... 27 The sources of emotion perception........................................................................... 30 Simultaneous Perception of Audio and Visual Information......................................... 34 Model 1: Integration ................................................................................................. 35 Model 2: Channel dominance ................................................................................... 38 Model 3: Individual differences................................................................................ 39 Three Conditions of Multimedia Combination............................................................. 40 Chapter 3 Framework and Research Models .............................................................. 44 Model 1: Integration ..................................................................................................... 48 1 Dimensional integration............................................................................................ 48 Model 2: Channel Dominance ...................................................................................... 51 Visual channel dominance ........................................................................................ 52 Auditory channel dominance .................................................................................... 52 Model 3: Individual Differences................................................................................... 53 Chapter 4 Research Design............................................................................................ 54 Measurement................................................................................................................. 55 Instrument development............................................................................................ 60 Design of Experiment ................................................................................................... 62 Participants................................................................................................................ 67 Procedure .................................................................................................................. 67 Control for potential influences ................................................................................ 69 Design of Stimuli.......................................................................................................... 70 Audio stimuli ............................................................................................................ 70 Visual stimuli............................................................................................................ 73 Chapter 5 Test of Measurement Model ........................................................................ 77 Calibration .................................................................................................................... 77 Evaluation of Scale Reliability ..................................................................................... 79 Evaluation of the Two-factor Model ............................................................................ 80 Two-factor model...................................................................................................... 81 Four-factor model ..................................................................................................... 83 Second-order model .................................................................................................. 84 Validity Assessment ..................................................................................................... 87 2 Discriminant validity ................................................................................................ 87 Convergence validity ................................................................................................ 88 Comparison of Two Instruments .................................................................................. 89 Chapter 6 Protocol Analysis Results............................................................................. 90 Evidence of the Two-factor Model of Emotion............................................................ 91 Sources of Emotion Perception..................................................................................... 95 Structural properties.................................................................................................. 95 Narrative elements .................................................................................................. 100 Comparison of the two sources of emotion perception .......................................... 101 Difficulties of interpretation ....................................................................................... 102 Multiple definitions................................................................................................. 102 Double negatives..................................................................................................... 104 Separation of perception and feeling ...................................................................... 105 Chapter 7 Hypothesis Testing Results ........................................................................ 107 Perception of Emotion in Individual Medium ............................................................ 107 Visual stimuli.......................................................................................................... 108 Audio stimuli .......................................................................................................... 110 Screening of multimedia combination .................................................................... 111 Assessment of the Individual Difference Model ........................................................ 111 Normality ................................................................................................................ 112 Gender..................................................................................................................... 113 Cultural background................................................................................................ 114 Musical expertise .................................................................................................... 116 3 Assessment of the integrative and channel dominance models.................................. 118 Conformance condition........................................................................................... 118 Contest condition .................................................................................................... 122 Summary of Results...................................................................................................