Tilburg University The Power of Facial Expressions Latuny, Wilma Publication date: 2017 Document Version Publisher's PDF, also known as Version of record Link to publication in Tilburg University Research Portal Citation for published version (APA): Latuny, W. (2017). The Power of Facial Expressions. [s.n.]. General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal Take down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim. Download date: 27. sep. 2021 THE POWER OF FACIAL EXPRESSIONS Wilma Latuny The research reported in this thesis was sponsored by the Netherlands organisation for international cooperation in higher education (Nuffic) programme, dossier no. CF 7553; NFP-PhD.11/238 SIKS Dissertation series No. 2017-30. The research reported in this thesis has been carried out under the auspices of SIKS, the Dutch Research School for Information and Knowledge Systems TiCC Ph.D. Series No. 55. ISBN/EAN: 978-94-6295-730-5 Model of the cover and the back cover by courtesy of David Tuhurima. Cover design by: ProefschriftMaken.nl || www.proefschriftmaken.nl Printed by: ProefschriftMaken.nl || www.proefschriftmaken.nl Layout by: ProefschriftMaken.nl || www.proefschriftmaken.nl Published by: ProefschriftMaken.nl || www.proefschriftmaken.nl All rights. No part of this thesis may be reproduced, stored in a retrieval system, or transmit- ted, in any form or by any means, electronically, mechanically, photocopying, recording or otherwise, without prior permission of the author. ©2017 Wilma Latuny THE POWER OF FACIAL EXPRESSIONS PROEFSCHRIFT ter verkrijging van de graad van doctor aan Tilburg University op gezag van de rector magnificus, prof. dr. E.H.L. Aarts, in het openbaar te verdedigen ten overstaan van een door het college voor promoties aangewezen commissie in de aula van de Universiteit op vrijdag 29 september 2017 om 10.00 uur door WILMA LATUNY, geboren op 6 april 1978 te Ambon, Indonesië Promotores: Prof. dr. E. O. Postma Prof. dr. H. J. van den Herik Overige leden van de promotiecommissie: Prof. dr. ir. P. H. M. Spronck Prof. dr. B. A. Van de Walle Prof. dr. H. C. Bunt Dr. V. Feltkamp Dr. C-J. Tsay PerformingPREFACE Ph.D. research and writing a thesis is as a journey. At one day you start and at another day you hope to finish. In my case, the first day was in 2011. The research was really a journey. My first idea was using data mining for predicting seaweed selling prices. However, I did not get the project running. My main diffi- culty was collecting the appropriate data. Meanwhile, I was given to understand that data mining is closely related to pattern recognition. So, I looked for areas in which many data were available and where pattern recognition could be applied in a profitable way. During that period in 2012, I started to create my own database from publicly online sources, such as www.youtube.com. Thus, my journey was restarted by investigating video data of human performers. Thereafter the idea arose to do exploratory analysis of human facial expressions from video or photo data. By employing statistical analysis and data mining tech- niques I addressed the following problem statement: “What is the power of the facial expressions in competitive settings?”. Competitive settings appear, among others, in beauty contests and music competitions. I operationalised the research on four competition domains namely: Miss World International Competition, Mister World International Competition, International Piano Competition, and JKT48 (JaKarTa 48) Leadership Singing Competition. The findings of my research revealed the following four results: (1) facial expres- sions contribute to attractiveness ratings but only when considered in combination with sexual dimorphism (femininity), (2) thin slices of dynamic facial expressions contribute to the attractiveness of males in similar way as static images do, (3) facial expressions allow the identification of the winning musician, and (4) facial expres- sions can predict a direct relation to the assessment of leadership. In summary, the thesis contains the following three scientific contributions: (a) four new findings con- cerning the power of facial expressions in competitive settings, (b) a variety of small contributions to the existing literature on facial expressions, and (c) a measurement comparison of facial expressions as assessed by human beings and computers. I would like to recognise the help of many people and institutions. First of all, I would like to thank my supervisors Professor Eric. O. Postma and Professor H. Jaap van den Herik for their valuable guidance and encouragement they gave me. God bless you and your families. Second, I am grateful to the Netherlands organisation for international cooperation in higher education (Nuffic) for their sponsorship, which enabled me to undertake my Ph.D. project. I also thank the Tilburg University, the Maastricht School of Man- agement (MSM) the Netherlands, and the University of Pattimura Ambon Indonesia, for giving me the opportunity to study. Third, my sincere thanks go to all my friends and family members. In particular, to my parents Izaak Latuny and Monika Nussy, and to my brother Richard Latuny and his family (Felly, Abigail, and Zipora), I would like to express my sincere thanks for v their support that I received throughout the Ph.D. journey. To my aunts Josephina Sopacua and children, Kristin Toisuta and husband, to my uncles Johan Nunumete, wife, and children and Henoch Nunumete, wife, and children, I also would like to express my thanks for their support and their nurturance during my study in Maastricht and Tilburg. Lastly, I would also like to acknowledge the support of the staff of DCI and TiCC of the Tilburg School of Humanities, particularly my thanks go to Professor dr. A.A. Maes as the Chair of DCI. Also, I wish to acknowledge the excellent support received from the staff members, more specifically from Eva, Lauraine and the former staff members Jachinta and Joke. Finally, I wish to acknowledge the other excellent sup- port that I received, namely from the Research Department of the Maastricht School of Management (MSM). I would like to thank in particular the dean of MSM Prof. dr. W.A. Naudé as well as Patrick Mans and Sandra Kolkman as former manager and senior research staff in MSM and also Rocco Muhlenberg as a senior officer research and IT Education. Tilburg, April 17, 2017 Wilma Latuny vi PrefaceCONTENTS v Contents vii List of Figures x List of Tables xi List of Abbreviations xiii 1 1 1.1 The Influence of Facial Expressions . 1 1.2 The Effect of the Context . 2 1.2.1 Our Research Domain . 3 1.2.2 The Power of Facial Expressions . 3 1.3 The Facial Action Coding System . 4 1.4 Automatic Facial Action Coding . 4 1.4.1 Computer Expression Recognition Toolbox . 5 1.4.2 Intraface . 5 1.4.3 Face++ . 5 1.5 Problem Statement and Research Questions . 6 1.6 Research Methodology . 7 1.7 Structure of the Thesis . 8 2 11 2.1 Three Characteristics of Attractiveness . 11 2.1.1 Static versus Dynamic Attractiveness . 12 2.1.2 Facial Expressions and Attractiveness . 12 2.1.3 Facial Action Units and Emotional Expressions . 13 2.1.4 Previous Work on Facial Expressions and Attractiveness . 14 2.2 Research Methodology . 14 2.2.1 Video Collection . 15 2.2.2 Measuring Facial Expressions . 15 2.2.3 Automatic Expression Extraction . 16 2.2.4 Correlation . 17 2.2.5 Prediction . 18 2.3 Experimental Results . 18 2.3.1 Correlation Results . 19 2.3.2 Prediction Results . 20 2.3.3 Contribution of Facial Expression Features . 22 2.4 Discussions and Limitations . 24 2.4.1 Comparison Correlation Analysis with Static Images . 24 2.4.2 The Importance of Lip-related Action Units . 25 2.4.3 Smiling, Dynamics and Attractiveness . 25 2.4.4 Three Limitations . 25 vii viii 2.4.5 Answer to RQ1 ............................ 27 3 29 3.1 Three Visual Cues of Male Attractiveness . 30 3.2 Static and Dynamic Cues to Attractiveness . 30 3.3 Research Method Survey Study . 32 3.4 Results of the Survey Study . 33 3.5 Research Method Computational Study . 37 3.6 Results of the Computational Study . 39 3.7 General Discussion . 41 3.8 Answer to RQ2 ................................ 42 4 43 4.1 Tsay’s Experiments . 45 4.1.1 A Novice’s Assessment . . 45 4.1.2 An Expert’s Assessment . 47 4.1.3 Dominance of Visual-only Information . 47 4.2 Experimental Set-up . 47 4.2.1 Dataset . 47 4.2.2 Facial Expression Coding . 51 4.2.3 Training and Testing Procedure . 51 4.2.4 Classifier . 52 4.2.5 Evaluation . 52 4.3 Results . 53 4.3.1 Results of Classification . 53 4.3.2 Contribution of Individual Action Units . 53 4.4 Discussion . 56 4.5 Answer to RQ3 ................................ 59 5 61 5.1 Implicit Leadership Theories and Facial Expressions . 62 5.1.1 Implicit Leadership Theory . 62 5.1.2 Overview of the Trichas and Schyns (2012) Study . 63 5.1.3 Dynamic Facial Expression and Leadership .