IDENTIFYING EMOTIONAL STATES THROUGH KEYSTROKE DYNAMICS A Thesis Submitted to the College of Graduate Studies and Research In Partial Fulfillment of the Requirements For the Degree of Master of Science In the Department of Computer Science University of Saskatchewan Saskatoon, CANADA By Clayton Epp Keywords: Affective computing, keystroke dynamics Copyright Clayton Epp, July, 2010. All rights reserved. PERMISSION TO USE In presenting this thesis in partial fulfillment of the requirements for a Postgraduate degree from the University of Saskatchewan, I agree that the Libraries of this University may make it freely available for inspection. I further agree that permission for copying of this thesis in any manner, in whole or in part, for scholarly purposes may be granted by the professor or professors who supervised my thesis work or, in their absence, by the Head of the Department or the Dean of the College in which my thesis work was done. It is understood that any copying or publication or use of this thesis or parts thereof for financial gain shall not be allowed without my written permission. It is also understood that due recognition shall be given to me and to the University of Saskatchewan in any scholarly use which may be made of any material in my thesis. Requests for permission to copy or to make other use of material in this thesis in whole or part should be addressed to: Head of the Department of Computer Science 176 Thorvaldson Building 110 Science Place University of Saskatchewan Saskatoon, Saskatchewan Canada S7N 5C9 i ABSTRACT The ability to recognize emotions is an important part of building intelligent computers. Extracting the emotional aspects of a situation could provide computers with a rich context to make appropriate decisions about how to interact with the user or adapt the system response. The problem that we address in this thesis is that the current methods of determining user emotion have two issues: the equipment that is required is expensive, and the majority of these sensors are invasive to the user. These problems limit the real-world applicability of existing emotion-sensing methods because the equipment costs limit the availability of the technology, and the obtrusive nature of the sensors are not realistic in typical home or office settings. Our solution is to determine user emotions by analyzing the rhythm of an individual‘s typing patterns on a standard keyboard. Our keystroke dynamics approach would allow for the uninfluenced determination of emotion using technology that is in widespread use today. We conducted a field study where participants‘ keystrokes were collected in situ and their emotional states were recorded via self reports. Using various data mining techniques, we created models based on 15 different emotional states. With the results from our cross-validation, we identify our best-performing emotional state models as well as other emotional states that can be explored in future studies. We also provide a set of recommendations for future analysis on the existing data set as well as suggestions for future data collection and experimentation. ii ACKNOWLEDGMENTS I would like to convey my appreciation to my supervisor Regan Mandryk for her support and guidance throughout my graduate career at the University of Saskatchewan. I would also like to thank the faculty and staff in the Department of Computer Science and the members of the Interaction Lab both past and present. In particular, I would like to express my gratitude to Mike Lippold, Andre Doucette, and Craig Yellowlees for their assistance during this process. Finally, I would like to thank Carrie Demmans Epp for all the support that she provided throughout my time as a graduate student. iii CONTENTS PERMISSION TO USE I ABSTRACT II ACKNOWLEDGMENTS III CONTENTS IV LIST OF TABLES VIII LIST OF FIGURES XI LIST OF ABBREVIATIONS XIII 1 INTRODUCTION .........................................................................................................1 1.1 Problem .............................................................................................................. 1 1.2 Solution .............................................................................................................. 3 1.3 Steps in the Solution .......................................................................................... 4 1.3.1 Experience Sampling Field Study ............................................................... 4 1.3.2 Data Collection Software ............................................................................ 4 1.3.3 Post Processing and Feature Extraction ...................................................... 5 1.3.4 Model Building ........................................................................................... 5 1.4 Contributions ...................................................................................................... 5 1.5 Thesis Outline .................................................................................................... 6 2 RELATED WORK .......................................................................................................8 2.1 Affect, Mood, and Emotion ............................................................................... 8 2.1.1 Terminology ................................................................................................ 8 2.1.2 Describing Emotion .................................................................................... 9 2.1.2.1 Discrete Categories .............................................................................9 2.1.2.2 Continuous Dimensions ....................................................................10 2.1.2.3 Using Discrete Categories and Continuous Dimensions ...................11 2.2 Recognizing Emotions ..................................................................................... 11 2.3 Emotional Experimentation ............................................................................. 13 2.3.1 Laboratory Settings ................................................................................... 13 2.3.2 Naturalistic Settings .................................................................................. 15 2.4 Keystroke Dynamics ........................................................................................ 17 iv 2.4.1 Pattern Recognition ................................................................................... 18 2.4.2 Keystroke Dynamics Background ............................................................ 19 2.4.2.1 Authentication & Intrusion Detection Systems .................................19 2.4.2.2 Commercial Products ........................................................................20 2.4.3 Terminology .............................................................................................. 21 2.4.3.1 Static and Dynamic Text ...................................................................21 2.4.3.2 Fixed and Free Text ...........................................................................22 2.4.4 Keystroke Features.................................................................................... 24 2.4.5 Classification............................................................................................. 25 2.4.5.1 Training Sample Size ........................................................................25 2.4.5.2 Model Validation ...............................................................................27 2.4.5.3 Classifiers ..........................................................................................27 2.4.6 Typing Errors ............................................................................................ 29 2.4.7 Novice & Experienced Keyboard Users ................................................... 30 2.5 Affective computing and keystroke dynamics ................................................. 32 3 DATA COLLECTION ................................................................................................34 3.1 Field study ........................................................................................................ 36 3.1.1 Getting Started .......................................................................................... 36 3.1.2 Restrictions ............................................................................................... 38 3.1.3 Maintaining Privacy .................................................................................. 39 3.1.4 Study Completion ..................................................................................... 40 3.2 Participant Demographics ................................................................................ 40 3.3 Field study software ......................................................................................... 43 3.3.1 Installation & Operation ........................................................................... 43 3.3.2 Keystroke Capture .................................................................................... 45 3.3.3 Questionnaire Interface ............................................................................. 47 3.3.4 Event Logs ................................................................................................ 52 3.3.5 Data Collection Server .............................................................................. 53 4 FEATURE EXTRACTION ..........................................................................................55 4.1 Data