Computational Support for the Evaluation of Facial Expressions in Photographs
Total Page:16
File Type:pdf, Size:1020Kb
Computational Support for the Evaluation of Facial Expressions in Photographs by Rachel Klingberg Submitted in partial fulfillment of the requirements for the Master’s degree in Computer Science at The Seidenberg School of Computer Science and Information Systems Pace University May 2013 Abstract Computational Support for the Evaluation of Facial Expressions in Photographs by Rachel Klingberg Submitted in partial fulfillment of the requirements for the Master’s degree in Computer Science May 2013 Charles Darwin first proposed that certain facial expressions of spontaneous emotion are genetically determined and culturally universal in his 1872 monograph The Expression of Emotion in Man and Animals. His theory of evolution applied not only to physical qualities, but also to emotional expression, or behavioral attributes. The latter idea was not widely accepted until the 1960s, when Paul Ekman’s research in neuroscience, psychology, and anthropology validated Darwin’s theory of evolutionary behavior. Drawing on Ekman’s pioneering research and his comprehensive taxonomy of muscular actions of the face, this thesis describes research into the development of computer software aiding the evaluation of facial expressions in photographs. The software program described here reports the presence and intensity of seven universal emotions: anger, contempt, fear, happiness, sadness, and surprise. A Web form prompts the user to match the photo’s appearance with a simple visual lexicon of the various muscular actions of the brow, eyes, nose and mouth. This input is compared to a relational data set of all possible actions of the facial muscles for all seven universal emotions. Using a weighted system of scoring, output reveals the presence and intensity of each of the emotions expressed by the photographic subject. i Acknowledgements I express my sincere gratitude to Dr. Richard Nemes for his guidance with my thesis and with my prior coursework for the MS in Computer Science. I also wish to thank Dr. Catherine Dwyer and Dr. Howard Blum for serving on the thesis committee, as well as Pace University for sponsoring my education with the Staff Scholarship program. ii Table of Contents Abstract ………………………………………………………………………………………... i List of figures…………………………………………………………………………………... iv I. Introduction to Pathognomy …………………………………………………………...… 1 1.1 A history of the field …………………………………………………………………… 1 1.1 Ekman et. al.: FACS and FACE ………………………………………………………... 7 1.2 Affective computing ……………………………………………………………………. 9 1.3 Resolution of the universality debate …………………………………………………... 11 II. General Approach to the Problem of Evaluation ……………………………………… 13 2.1 FACS, Aranatomy, and commercial software …………………………………………. 13 2.2 Expressions as sets …………………………………………………............................... 15 2.3 A lexicon of expressions of basic emotions ……………………………………………. 16 III. SETL, a Procedural Approach ………………………………………………................ 19 3.1 SETL analysis with set operations …………………………………………………........ 19 3.2 Testing input and scoring results ………………………………………………….......... 20 3.3 User considerations …………………………………………………............................... 21 IV. SQL, a Relational Approach ………………………………………………................. 22 4.1 The table …………………………………………………............................................... 22 4.2 The application …………………………………………………..................................... 24 4.3 Testing with naïve users …………………………………………………....................... 27 4.4 Scoring …………………………………………………................................................. 27 4.5 Evaluation of results …………………………………………………............................. 35 V. Conclusion ……………………………….….….….….….……...………………………... 37 References …………………………….….….….….….……...……………………………… 69 Appendix A: Lexicon for SETL Analyzer ……………………………….….….….….……… 38 Appendix B: SETL Program to Analyze Facial Expressions…………………………….....… 45 Appendix C: Form to Accept Input for Analysis of Facial Expressions………………....…… 54 Appendix D: SQL/ColdFusion Program to Analyze Input and Report on Presence of Facial Expressions…………………………………………….....… 60 iii List of Figures Fig. 1 Duchenne using electrical stimulation to provoke a smile ……………………….. 2 Fig. 2 Paul Ekman demonstrating the Duchenne and non-Duchenne smiles …………… . 3 Fig. 3 "The Cartesian Theater" ……………...………………...………………...………. 6 Fig. 4 The seven universal emotions ……...………………...……………...…………… . 8 Fig. 5 The Artanatomy depiction of anger ……...………………...………………...…… 16 Fig. 6 Web form to accept input for the analysis of facial expressions ……...………...... 24 Fig. 7 Expression of anger ……...………………...………………...…………………… 28 Fig. 8 Expression of anger/contempt ……...………………...………………...………… 28 Fig. 9 Expression of contempt ……...………………...………………...……………….. 30 Fig. 10 Expression of disgust ……...………………...………………...…………………. 30 Fig. 11 Expression of fear ……...………………...………………...…………………...... 32 Fig. 12 Expression of happiness ……...………………...………………...……………..... 33 Fig. 13 Expression of sadness ……...………………...………………...…………………. 33 Fig. 14 Expression of surprise ……...………………...………………...………………… 34 iv I. Introduction to Pathognomy 1.1 A history of the field What causes us to feel emotions? Why do we feel certain emotions so strongly? And how do we cope with the intensity of our feelings? Throughout history, philosophers and physicians have debated these questions. Since the 19th century, the entire discipline of psychology has been devoted to them. Because of its association with our capricious feelings, the very word “emotion” implies the opposite of scientific rigor, yet facial expressions of emotion are as universal to humanity as any other aspect of muscular physiology. The study of facial expressions of emotion has been a well-established discipline since the 1960s, but it has no formal name in modern vernacular. The 18th-century term is pathognomy: Pathognomy (archaic): Expression of the passions; the science of the signs by which human passions are indicated. 1793 HOLCROFT Lavater’s Physiog. ii. 24: Pathognomy is the knowledge of the signs of the passions. [1] Pathognomy falls under the umbrella of non-verbal communication, a field of biology, psychology, and cognitive neuroscience that also includes gestures, tone of voice, posture, movement, positioning, and body cues, as well as changes in heart rate, temperature, pupil dilation, and other involuntary responses to intense feeling. The science of pathognomy is only concerned with the anatomical facial expressions of emotion, not with the stimulus or response, since the latter cannot be categorized. To understand how the expression of human emotion can be approached with scientific rigor, as a biological function universal across cultures and nearly identical for all of humanity, it is necessary to entirely disregard the stimulus and response to emotion that is the essence of psychology, and focus instead on the mechanics of facial expression, from gross motor action of the muscles to the subtlest trigger of the brain’s amygdala. From this perspective, emotional expression is a genetic attribute, subject to the same natural selection and inheritance as other biological attributes. To validate this concept of emotion expressed universally among all human beings, I wrote two software applications to demonstrate that facial expressions of seven basic emotions can be modeled as sets and manipulated computationally with set operations – a programmatic interpretation of ideas first espoused by eighteenth-century scientists. A brief survey of the field of pathognomy is necessary to lay the foundation for approaching emotional expressions algorithmically. Pathognomy is concerned only with the physical expression of emotion, because the human experience is far too diverse for the study of emotional stimulus or response to be a precise science, even at the most basic level of survival. For example, there are divers who can hold their breath for three minutes, although for most people, being unable to breathe for more than one minute incites distress. Some people find activities such as mountain-climbing or swimming exhilarating, while others are terrified by heights or unable to swim and find those same activities life-threatening. Books like Ben Sherwood’s The Survivors Club examine why some people endure extreme situations, such as plane crashes, while others in the same challenging environment perish. Even the simple pleasure we take in eating when hungry, an emotion that ensures our daily survival, is not elicited by the same stimuli across all cultures, across smaller groups of people within the same culture, or even within the same individual at different times. A familiar example is the family gathering attended by someone who has recently become a vegan, and is now disgusted by the 1 sight and smell of meat, although the very same person was once delighted at spareribs grilling on the barbeque. Pathognomy has for centuries been of interest to actors and artists seeking to accurately portray the human experience, and the Age of Reason brought the field close to the science it is today. An eighteenth-century French scientist, Duchenne de Boulogne, first advanced the notion that facial expressions were a biological component rather than a voluntary form of expression, nudging the field from the realm of artistic consideration into a subject to which rigorous scientific procedure could be applied. Using electrical stimulation on a patient with very little facial nerve sensation, Duchenne provoked muscular actions associated with recognizable emotions (see figure