Vocalic Markers of Deception and Cognitive Dissonance for Automated Emotion Detection Systems Item Type text; Electronic Dissertation Authors Elkins, Aaron Chaim Publisher The University of Arizona. Rights Copyright © is held by the author. Digital access to this material is made possible by the University Libraries, University of Arizona. Further transmission, reproduction or presentation (such as public display or performance) of protected items is prohibited except with permission of the author. Download date 01/10/2021 09:59:37 Link to Item http://hdl.handle.net/10150/202930 VOCALIC MARKERS OF DECEPTION AND COGNITIVE DISSONANCE FOR AUTOMATED EMOTION DETECTION SYSTEMS by Aaron C. Elkins _____________________ Copyright © Aaron C. Elkins 2011 A Dissertation Submitted to the Faculty of the COMMITTEE ON BUSINESS ADMINISTRATION In Partial Fulfillment of the Requirements For the Degree of DOCTOR OF PHILOSOPHY WITH A MAJOR IN MANAGEMENT In the Graduate College THE UNIVERSITY OF ARIZONA 2011 2 THE UNIVERSITY OF ARIZONA GRADUATE COLLEGE As members of the Dissertation Committee, we certify that we have read the dissertation prepared by Aaron C. Elkins entitled Vocalic Markers of Deception and Cognitive Dissonance for Automated Emotion Detection Systems and recommend that it be accepted as fulfilling the dissertation requirement for the Degree of Doctor of Philosophy __________________________________________________________________ Date: 8/11/2011 Jay F. Nunamaker, Jr. __________________________________________________________________ Date: 8/11/2011 Judee K. Burgoon __________________________________________________________________ Date: 8/11/2011 Elyse Golob __________________________________________________________________ Date: 8/11/2011 Paulo B. Goes Final approval and acceptance of this dissertation is contingent upon the candidate’s submission of the final copies of the dissertation to the Graduate College. I hereby certify that I have read this dissertation prepared under my direction and recommend that it be accepted as fulfilling the dissertation requirement. ____________________________________________ Date: 8/11/2011 Dissertation Director: Jay F. Nunamaker, Jr. ____________________________________________ Date: 8/11/2011 Dissertation Director: Judee K. Burgoon 3 STATEMENT BY AUTHOR This dissertation has been submitted in partial fulfillment of requirements for an advanced degree at the University of Arizona and is deposited in the University Library to be made available to borrowers under rules of the Library. Brief quotations from this dissertation are allowable without special permission, provided that accurate acknowledgment of source is made. Requests for permission for extended quotation from or reproduction of this manuscript in whole or in part may be granted by the copyright holder. SIGNED: ________________________ Aaron C. Elkins 4 ACKNOWLEDGEMENTS The Department of Homeland Security’s National Center for Border Security and Immigration (BORDERS) and the National Center for Credibility Assessment (NCCA) provided significant support for this research. Similarly, funding from the Center for Identification Technology Research (CITeR), a National Science Foundation (NSF) Industry/University Cooperative Research Center (I/UCRC), supported portions of this dissertation. The vocal analysis software investigated in the research was provided for evaluation purposes by Nemesysco Ltd. I would like to thank my committee, Jay F. Nunamaker, Jr., Judee K. Burgoon, Elyse Golob, and Paulo Goes for their friendship and tireless efforts to help me become a better academic and researcher. I would also like to thank Jeff Stone for his mentorship and friendship. It was a privilege to learn and research his Self and Attitudes lab. Finally, I would like to acknowledge and thank Doug Derrick my friend and colleague from the doctoral program for developing the Embodied Conversational Agent investigated in this research. 5 DEDICATION I dedicate this dissertation to my mother Lourdes Elkins. None of my success or accomplishments, including this document, would have been possible without her unconditional love and limitless support. I would also like to thank my sister Hashaw Elkins and father David Elkins for their support and encouragement over the past four years. I would also like to thank both my Psychologist/Social Worker mother and Engineer father for cultivating my interest and curiosity for both the technical and behavioral sciences. 6 It may be said with some assurance that if no one has calculated the orbit of a fly, it is only because no one has been sufficiently interested in doing so. The tropistic movements of many insects are now fairly well understood, but the instrumentation needed to record the flight of a fly and to give an account of all the conditions affecting it would cost more than the importance of the subject justifies. Difficulty in calculating the orbit of the fly does not prove capriciousness, though it may make it impossible to prove anything else. The problems imposed by the complexity of a subject matter must be dealt with as they arise. Certainly no one is prepared to say now what a science of behavior can or cannot accomplish eventually. Advance estimates of the limits of science have generally proved inaccurate. The issue is in the long run pragmatic: we cannot tell until we have tried. -- B.F. Skinner, Science and Human Behavior (1953) 7 TABLE OF CONTENTS LIST OF TABLES .................................................................................................... 13 LIST OF FIGURES .................................................................................................. 15 ABSTRACT ............................................................................................................. 18 1 INTRODUCTION ................................................................................................ 20 1.1 Vocal Emotion and Deception Detection for National Security and Law Enforcement .................................................................................................... 21 2 VOCAL MEASUREMENTS OF EMOTION AND DECEPTION ......................... 25 2.1 Commercial Vocal Analysis Software ........................................................ 25 2.1.1 Vocal Deception .............................................................................. 26 2.1.2 Vocal Stress Analysis Software ...................................................... 27 2.1.3 Full Spectrum Vocal Analysis ........................................................ 28 2.2 Deception ................................................................................................... 31 2.3 Cognitive Dissonance ................................................................................ 32 2.3.1 Cognitive Dissonance and Arousal ................................................ 33 2.3.2 Cognitive Dissonance and Deception ............................................ 34 2.4 Vocalics and Linguistics ............................................................................ 35 2.4.1 Arousal and Cognitive Effort ......................................................... 35 2.4.2 Emotion ......................................................................................... 36 3 STUDY ONE – PREDICTING AND VALIDATING VOCAL DECEPTION AND EMOTION .......................................................................................................... 39 3.1 Introduction ............................................................................................... 39 8 TABLE OF CONTENTS - Continued 3.2 Deception Experiment ............................................................................... 40 3.3 Method ....................................................................................................... 42 3.3.1 Participants .................................................................................... 42 3.3.2 Procedures ..................................................................................... 42 3.4 Instrumentation ........................................................................................ 45 3.4.1 Self-Report Measures ..................................................................... 45 3.4.2 Vocal Analysis Software and Segmentation .................................. 47 3.4.3 Standardization ............................................................................. 48 3.5 Deception Experiment ............................................................................... 48 3.5.1 Results of Vocal Analysis Software Built-in Classifier .................. 48 3.5.2 Question Type ................................................................................ 51 3.6 Analysis of Vocal Measurements ............................................................... 52 3.6.1 Results of Experimental Treatment ............................................... 53 3.6.2 Moderators of Lying on Vocal Measurements .............................. 57 3.7 Vocal Measurements ................................................................................. 62 3.7.1 Multilevel Factor Analysis .............................................................. 62 3.8 Interpretation of Vocal Measurement Factors ......................................... 68 3.8.1 Lasso Regression ............................................................................ 68 3.8.2 Results of Lasso Regression and Factor Interpretation ............... 69 3.9 Predicting Deception ................................................................................