Microexpression Spotting in Video Using Optical Strain
Total Page:16
File Type:pdf, Size:1020Kb
University of South Florida Scholar Commons Graduate Theses and Dissertations Graduate School 7-1-2010 Microexpression Spotting in Video Using Optical Strain Sridhar Godavarthy University of South Florida Follow this and additional works at: https://scholarcommons.usf.edu/etd Part of the American Studies Commons Scholar Commons Citation Godavarthy, Sridhar, "Microexpression Spotting in Video Using Optical Strain" (2010). Graduate Theses and Dissertations. https://scholarcommons.usf.edu/etd/1642 This Thesis is brought to you for free and open access by the Graduate School at Scholar Commons. It has been accepted for inclusion in Graduate Theses and Dissertations by an authorized administrator of Scholar Commons. For more information, please contact [email protected]. Microexpression Spotting in Video Using Optical Strain by Sridhar Godavarthy A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computer Science Department of Computer Science and Engineering College of Engineering University of South Florida Major Professor: Dmitry B. Goldgof, Ph.D. Sudeep Sarkar, Ph.D. Rangachar Kasturi, Ph.D. Date of Approval: July 1, 2010 Keywords: Computer vision, FACS, Optical flow, Facial deformation, Datasets Copyright © 2010, Sridhar Godavarthy DEDICATION To the Omnipotent, Omnipresent and Omniscient one... What are we, but for him? ACKNOWLEDGEMENTS First and foremost: God - for allowing me to tread this path. My parents Subrahmanya Sarma and Raja Rajeswari, brother Dr. Srinivasa Sastry and sister in law Sarada - but for whose support, trust and perseverance, I would have been just another baccalaureate. A special thanks to Dr. Dmitry Goldgof for accepting me as his student - I have never worked under a more understanding professor. My sincere gratitude goes to Dr. Rangachar Kasturi - the first person from USF I spoke to - if not for his encouraging words, I might have missed this golden opportunity of working with so many intelligent minds. I am also grateful to Dr. Sudeep Sarkar for his timely suggestions and ideas. I am greatly indebted to all three of them - each of them a brilliant mind overflowing with ideas, of which, I was able to glean a few- for their guidance and motivation. Dr. Vasant Manohar - whose spark started this whole work, Matthew Shreve for sharing his knowledge, his technical support and guidance. Joshua Candamo – my inspiration. I must thank my supervisor Daniel Prieto, an amazing person - for understanding my constraints and allowing me to work flexibly. Parameswaran Krishnan and Ramya Parameswaran - both unrelated to each other - my prime movers and pillars of support, who will never willingly accept my thanks. Ravikiran Krishnan- for being my constant companion, those midnight trips to Starbucks and brainstorming sessions. Lucy Puentes for her pep talks and motivation in my times of need. Ekta Karia for her delightful company and more importantly, sticking by me despite all the pestering I subjected her to. Pavithra Bonam for her confidence in me and all those delightful dinners. Ravi Panchumarthy, a gem of a person- for his support and company. Valentine Gnana Selvi for being “that” friend who blindly believes in you. Last but by no means the least, Aruna Pandian – for showing me what life is all about. There are several others who helped me to get to where I am. I am thankful to every one of them. Portions of the research in this paper use the Canal 9 Political Debates Database made available by A. Vinciarelli at the Idiap Research Institute, Switzerland. TABLE OF CONTENTS LIST OF TABLES iii LIST OF FIGURES iv ABSTRACT v CHAPTER 1 INTRODUCTION 1 1.1 Microexpression 1 1.2 Applications of Automatic Microexpression Detection 2 1.3 FACS 3 1.4 Expression Segmentation 4 1.5 Aim of This Research 5 1.6 Proposed Method 5 1.7 Thesis Organization 6 CHAPTER 2 PREVIOUS WORK 7 2.1 Microexpressions 7 2.2 Automatic Facial Expression Recognition Systems 8 2.2.1 Face Detection 9 2.2.2 Feature Extraction 10 2.2.3 Expression Analysis 10 2.3 Datasets 11 CHAPTER 3 BACKGROUND 14 3.1 Optical Flow 14 3.2 Stress and Strain 16 3.3 Strain Magnitude Characteristics 17 3.4 Computation of Strain 20 3.5 The OpenCV Haar Classifier 21 CHAPTER 4 ALGORITHM AND IMPLEMENTATION 24 4.1 Overview 24 4.2 Face Alignment 26 4.3 Optical Flow 28 4.4 Computation of Optical Strain 28 4.5 Split into Regions 33 4.6 Thresholding 34 CHAPTER 5 DATASETS 37 5.1 USF Dataset 37 5.1.1 Feigned Dataset 37 5.1.2 Live Questioning Dataset 37 5.2 Canal 9 Dataset 38 5.3 Found Videos 39 CHAPTER 6 EXPERIMENTAL RESULTS 41 6.1 Determination of Threshold 41 6.2 Results with Only Macroexpressions 43 i 6.3 Microexpression Spotting Results 44 CHAPTER 7 CONCLUSION AND FUTURE WORK 48 7.1 Conclusions 48 7.2 Future Work 49 REFERENCES 50 ii LIST OF TABLES Table 1.1 Comparison of various facial expression coding schemes 4 Table 2.1 Select automatic facial expression recognition systems in literature 13 Table 4.1 List of parameters used 24 Table 5.1 Characteristics of datasets used for evaluation 40 Table 6.1 Results of thresholding on the training set with 22 microexpressions 42 Table 6.2 Microexpression spotting results 44 iii LIST OF FIGURES Figure 1.1 Sample microexpressions 2 Figure 3.1 Errors in optical flow caused by change in illumination 15 Figure 3.2 Errors in optical flow due to rotation of the object 15 Figure 3.3 Types of motion in an object when acted upon by an external force 16 Figure 3.4 Sample frames from the stability test video 18 Figure 3.5 Optical flow and optical strain with out of plane movement 19 Figure 3.6 Representations of some Haar like features from the OpenCV library 22 Figure 4.1 Flowchart of proposed algorithm 25 Figure 4.2 Illustration of the face alignment process 27 Figure 4.3 Dense optical flow on two frames of the Alex Rodriguez video 29 Figure 4.4 Computed dense optical flow on two frames of the USF dataset 30 Figure 4.5 Strain maps for the optical flow computed in Figure 4.3 31 Figure 4.6 Strain map for the optical flow computed in Figure 4.4 32 Figure 4.7 Split of face into separate regions 33 Figure 4.8 Algorithm for thresholding strain magnitudes 35 Figure 4.9 Example strain thresholding 36 Figure 5.1 Canal 9 dataset 39 Figure 5.2 Found video dataset 40 Figure 6.1 ROC curve used for threshold determination 41 Figure 6.2 Typical strain pattern for a microexpression 42 Figure 6.3 Strain map and thresholding with only macroexpressions 44 Figure 6.4 Strain plots for select USF sequences 45 Figure 6.5 Microexpression spotting results 46 iv Microexpression Spotting in Video Using Optical Strain Sridhar Godavarthy ABSTRACT Microexpression detection plays a vital role in applications such as lie detection and psychological consultations. Current research is progressing in the direction of automating microexpression recognition by aiming at classifying the microexpressions in terms of FACS Action Units. Although high detection rates are being achieved, the datasets used for evaluation of these systems are highly restricted. They are limited in size - usually still pictures or extremely short videos; motion constrained; containing only a single microexpression and do not contain negative cases where microexpressions are absent. Only a few of these systems run in real time and even fewer have been tested on real life videos. This work proposes a novel method for automated spotting of facial microexpressions as a preprocessing step to existing microexpression recognition systems. By identifying and rejecting sequences that do not contain microexpressions, longer sequences can be converted into shorter, constrained, relevant sequences which comprise of only single microexpressions, which can then be passed as input to existing systems, improving their performance and efficiency. This method utilizes the small temporal extent of microexpressions for their identification. The extent is determined by the period for which strain, due to the non-rigid motion caused during facial movement, is impacted on the facial skin. The subject‟s face is divided into sub-regions, and facial strain is calculated for each of these regions. The strain patterns in individual regions are used to identify subtle changes which facilitate the detection of microexpressions. The strain magnitude is calculated using the central difference method over the robust and dense optical flow field of each subject‟s face. The computed strain is then thresholded using a variable v threshold. If the duration for which the strain is above the threshold corresponds to the duration of a microexpression, detection is reported. The datasets used for algorithm evaluation are comprised of a mix of natural and enacted microexpressions. The results were promising with up to 80% true detection rate. Increased false positive spots in the Canal 9 dataset can be attributed to talking by the subjects causing fine movements in the mouth region. Performing speech detection to identify sequences where the subject is talking and excluding the mouth region during those periods could help reduce the number of false positives. vi CHAPTER 1 INTRODUCTION Decision making is an integral part of our day to day life. Some of the decisions we make are based on discernible facts and are well thought out, whilst others are made instantaneously considering only the immediately available evidence. In the latter scenario, the human mind does not consciously weigh the various contributing factors; instead it just recognizes that there is something “right” or “wrong” with the situation. A significant number of these instantaneous decisions we make are related to humans and are made when in direct visual contact with them – most often when listening to them.