DISI - University of Trento

PhD Dissertation International Doctorate School in Information and Communication Technologies DISI - University of Trento MINING HUMAN BEHAVIORS: AUTOMATED BEHAVIORAL ANALYSIS FROM SMALL TO BIG DATA Jacopo Staiano Advisor: Prof. Nicu Sebe Universita` degli Studi di Trento 28th of April, 2014 Nothing is mysterious, no human relation. Except love. Susan Sontag1 Quando penso a Pasolini, a come agiva rispetto alla societa,` alle cose, mi stimo molto poco. Massimo Troisi 1As Consciousness Is Harnessed to Flesh: Journals and Notebooks, 1964-1980. Acknowledgements First, I want to thank Nicu for all the support and for believing in me (sometimes even more than I myself did). It was, is, and will be, a great privilege to work under your guidance, to witness the birth and growth of the M-HUG group, and to have you as a great friend. I thank Prof. Theo Gevers, and the friends and former colleagues at the Intelligent Systems Lab of the University of Amsterdam: my experience working there is what actually brought me to push a PhD. Vladimir Nedovic, Roberto Valenti, Ivo Everts, Jose Alvarez and Hamdi Dibeklioglu deserve a special mention. I am grateful to Prof. Hamid Aghajan at Stanford, for having me as a visiting researcher in the Ambient Intelligence Lab at Stanford. To the 1737 crew, Scott Zimmerman, Handan Selcuk, Sujay and Naveem Vennam, Trip, Cemre and Yesim Ozkurt, Giuseppe “tac” and Flo Valente: no matter how far, we’re connected. Thanks to Prof. Alex “Sandy” Pentland, for giving me the opportunity of working at the Human Dynamics Lab at MIT in 2011 and 2013, and to Nuria Oliver for having me as intern at Telefonica I+D in Barcellona. I finally want to thank: Bruno Lepri and Fabio Pianesi for the fruitful (and ongoing) collabo- rations, Marco Guerini, Stefano Teso, Andrea Passerini, Leo Maccari, Elena Pavan, Elisa Ricci, Gabriele Catania, Alex Cappelletti, Maria Menendez, Antonella De Angeli, Matteo Bonifacio, Andrey Bogomolov along with colleagues at DISI and all my coauthors; Gloria, Jasper, Vika, Ram, Julian, and the entire M-HUG group; Manuel, Francesca, Andrea, Mario and Danilo at DISI; the friends at Mozart (RIP), Cafe’ de la Paix, Chinaski, Qube´ Cafe,´ RockInCapri, Pis- toiaBlues, NoGuRu; Anna & Mario at Castelli Romani; Riccardo Esposito at My Social Web; all lifelong friends DOP, Antonio “azazel”, Antonio “hope”, Antonio “goodbirds”, Vincenzo “the shell”, Cecco, Graziano, Marcella, Gabriella, Luigi, Ottavio, Colonnesi, Fausto, Mari- alaura, Cekkini, Palillo, Catuogno, the Guardino Bros. Last but not least, I am deeply grateful to my parents and family for their endless support and patience. And to Ilaria, for what we are and for what we’ll be. List of Publications • J Staiano, N Oliver, B Lepri, R de Oliveira, M Caraviello, N Sebe Money Walks: a Human-Centric Study on the Economics of Personal Mobile Informa- tion [258] Proceedings of the 2014 ACM Conference on Ubiquitous Computing, ACM UBICOMP 2014; • J Staiano, M Guerini Depeche Mood: a Lexicon for Emotion Analysis from Crowd Annotated News [254] The 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014; • R Subramanian, Y Yan, J Staiano, O Lanz, N Sebe On the relationship between head pose, social attention and personality prediction for unstructured and dynamic group interactions [268] 15th International conference on multimodal interaction, ACM ICMI 2013; • S Teso, J Staiano, B Lepri, A Passerini, F Pianesi Ego-centric graphlets for personality and affective states recognition [276] International Conference on Social Computing, 874-877, IEEE SocialCom 2013; • M Guerini, J Staiano, D Albanese Exploring Image Virality in Google Plus [105] International Conference on Social Computing, 874-877, IEEE SocialCom 2013; • MK Abadi, J Staiano, A Cappelletti, M Zancanaro, N Sebe Multimodal Engagement Classification for Affective Cinema [2] Humaine Association Conference on Affective Computing and Intelligent Interaction, IEEE ACII 2013; • G Zen, N Rostamzadeh, J Staiano, E Ricci, N Sebe Enhanced semantic descriptors for functional scene categorization [317] 21st International Conference on Pattern Recognition, ICPR 2012; • J Staiano, B Lepri, N Aharony, F Pianesi, N Sebe, A Pentland Friends don’t lie: inferring personality traits from social network structure [255] Proceedings of the 2012 ACM Conference on Ubiquitous Computing, 321-330, ACM UBICOMP 2012; • B Lepri, J Staiano, G Rigato, K Kalimeri, A Finnerty, F Pianesi, N Sebe, A Pentland The sociometric badges corpus: A multilevel behavioral dataset for social behavior in complex organizations [165] International Conference on Social Computing, 623-628, IEEE SocialCom 2012; • B Lepri, R Subramanian, K Kalimeri, J Staiano, F Pianesi, N Sebe Connecting Meeting Behavior with Extraversion – A Systematic Study [167] IEEE Transactions on Affective Computing, 2012; • J Staiano, M Menendez,´ A Battocchi, A De Angeli, N Sebe UX Mate: From facial expressions to UX evaluation [257] Proceedings of the Designing Interactive Systems Conference, 741-750, ACM DIS 2012; • J Staiano, B Lepri, R Subramanian, N Sebe, F Pianesi Automatic modeling of personality states in small group interactions [256] Proceedings of the 19th ACM international conference on Multimedia, 989-992, ACM MM 2011; • H Joho, J Staiano, N Sebe, JM Jose Looking at the viewer: analysing facial activity to detect personal highlights of multimedia contents [137] Multimedia Tools and Applications, 51(2):505-523, 2011; • B Lepri, R Subramanian, K Kalimeri, J Staiano, F Pianesi, N Sebe Employing social gaze and speaking activity for automatic determination of the extraversion trait [166] International Conference on Multimodal Interfaces and the Workshop on Machine Learn- ing for Multimodal Interaction, ACM ICMI 2010; • R Subramanian, J Staiano, K Kalimeri, N Sebe, F Pianesi Putting the pieces together: multimodal analysis of social attention in meetings [267] Proceedings of the international conference on Multimedia, 659-662, ACM MM 2010. Abstract This research thesis aims to address complex problems in Human Behavior Understanding from a computational standpoint: to develop novel methods for enabling machines to capture not only what their sensors are perceiving but also how and why the situation they are presented with is evolving in a certain manner. Touching several fields, from Computer Vision to Social Psychology through Natural Language Processing and Data Mining, we will move from more to less constrained scenarios, describing models for automated behavioral analysis in different contexts: from the individual perspective, e.g. a user interacting with technology, to the group perspective, e.g. a brainstorming session; from living labs, e.g. hundreds of people transparently tracked in their everyday life through smart-phone sensors, to the World Wide Web. Contents 1 Introduction 1 1.1 The Context ............................ 2 1.2 Structure of the Thesis ...................... 5 2 Deriving and Exploiting Behavioral Insights at Individual Level 11 2.1 Analysing Facial Activities to Detect Personal Highlights of Videos ............................... 12 2.1.1 Affective Video Summarisation ............. 14 2.1.2 Facial Expression Recognition System .......... 18 2.1.3 Analysis ......................... 23 2.1.4 Results and Discussion .................. 28 2.1.5 Conclusions and Future Work .............. 33 2.2 From Facial Expressions to UX evaluation ............ 34 2.2.1 State of the Art ...................... 35 2.2.2 UX Mate ......................... 38 2.2.3 Pilot Study ........................ 42 2.2.4 Validation Study ..................... 49 2.2.5 Conclusions ........................ 54 3 Human Interaction in Small Groups: What and How Influence Whom 57 3.1 Putting the Pieces Together: Multimodal Analysis of Social At- tention in Meetings ........................ 58 i 3.1.1 Related Work ....................... 59 3.1.2 ’Mission Survival’ Meeting Videos ........... 61 3.1.3 Inferences from Ground-Truth .............. 62 3.1.4 Automated Social Attention Estimation ......... 66 3.1.5 Conclusions ........................ 70 3.2 Automatic Modeling of Personality States in Small Group In- teractions ............................. 70 3.2.1 Data and Experimental Setup ............... 73 3.2.2 Feature Extraction .................... 74 3.2.3 Feature Selection ..................... 76 3.2.4 Automatic Recognition of Personality States ...... 77 3.2.5 Results and Discussion .................. 79 4 Into the Wild: Implicit Behavioral Patterns Emerging in a Technology- Mediated and Inter-Connected Society 81 4.1 Inferring Personality Traits from Social Network Structure ... 83 4.1.1 Related Works ...................... 85 4.1.2 Dataset .......................... 88 4.1.3 Extraction of Network Characteristics .......... 91 4.1.4 Automatic Prediction of Personality Traits ........ 97 4.1.5 Discussion and Comparison with Previous Works .... 105 4.1.6 Practical Implications and Limitations .......... 107 4.1.7 Conclusions ........................ 108 4.2 A Multilevel Behavioral Dataset for Social Behavior in Com- plex Organizations ........................ 109 4.2.1 Collection Methodology ................. 112 4.2.2 Data Collected: Personal and Situational Data ...... 114 4.2.3 Data Collected: Digital Data ............... 116 4.2.4 A Few Statistics ..................... 118 ii 4.2.5 Discussion and Future Works .............. 121 4.3 Ego-Centric Graphlets for Personality and Affective States Recog- nition ............................... 123 4.3.1 Related Works

Load more