Self-supervised Face Representation Learning zur Erlangung des akademischen Grades eines Doktors der Ingenieurwissenschaften von der KIT-Fakultät für Informatik des Karlsruher Instituts für Technologie (KIT) genehmigte Dissertation von Vivek Sharma aus Siliguri, India Tag der mündlichen Prüfung: 14. May 2020 Hauptreferent: Prof. Dr.-Ing. Rainer Stiefelhagen Karlsruher Institut für Technologie Korreferent: Prof. Dr.-Ing. Tamim Asfour Karlsruher Institut für Technologie KIT – Die Forschungsuniversität in der Helmholtz-Gemeinschaft www.kit.edu Abstract This thesis investigates fine-tuning deep face features in a self-supervised manner for discriminative face representation learning, wherein we develop methods to automatically generate pseudo-labels for training a neural network. Most importantly solving this problem helps us to advance the state-of-the-art in representation learning and can be beneficial to a variety of practical downstream tasks. Fortunately, there is a vast amount of videos on the internet that can be used by machines to learn an effective representation. We present methods that can learn a strong face representation from large-scale data be the form of images or video. However, while learning a good representation using a deep learning algorithm requires a large-scale dataset with manually curated labels, we propose self-supervised approaches to generate pseudo-labels utilizing the temporal structure of the video data and similarity constraints to get supervision from the data itself. We aim to learn a representation that exhibits small distances between samples from the same person, and large inter-person distances in feature space. Using metric learning one could achieve that as it is comprised of a pull-term, pulling data points from the same class closer, and a push-term, pushing data points from a different class further away. Metric learning for improving feature quality is useful but requires some form of external supervision to provide labels for the same or different pairs. In the case of face clustering in TV series, we may obtain this supervision from tracks and other cues. The tracking acts as a form of high precision clustering (grouping detections within a shot) and is used to automatically generate positive and negative pairs of face images. Inspired from that we propose two variants of discriminative approaches: Track-supervised Siamese network (TSiam) and Self-supervised Siamese network (SSiam). In TSiam, we utilize the tracking supervision to obtain the pair, additional we include negative training pairs for singleton tracks – tracks that are not ii Abstract temporally co-occurring. As supervision from tracking may not always be available, to enable the use of metric learning without any supervision we propose an effective approach SSiam that can generate the required pairs automatically during training. In SSiam, we leverage dynamic generation of positive and negative pairs based on sorting distances (i.e. ranking) on a subset of frames and do not have to only rely on video/track based supervision. Next, we present a method namely Clustering-based Contrastive Learning (CCL), a new clustering-based representation learning approach that utilizes automatically discovered partitions obtained from a clustering algorithm (FINCH) as weak super- vision along with inherent video constraints to learn discriminative face features. As annotating datasets is costly and difficult, using label-free and weak supervision ob- tained from a clustering algorithm as a proxy learning task is promising. Through our analysis, we show that creating positive and negative training pairs using clustering predictions help to improve the performance for video face clustering. We then propose a method face grouping on graphs (FGG), a method for unsupervised fine-tuning of deep face feature representations. We utilize a graph structure with positive and negative edges over a set of face-tracks based on their temporal structure of the video data and similarity-based constraints. Using graph neural networks, the features communicate over the edges allowing each track’s feature to exchange information with its neighbors, and thus push each representation in a direction in feature space that groups all representations of the same person together and separates representations of a different person. Having developed these methods to generate weak-labels for face representation learning, next we propose to learn compact yet effective representation for describing face tracks in videos into compact descriptors, that can complement previous methods towards learning a more powerful face representation. Specifically, we propose Temporal Compact Bilinear Pooling (TCBP) to encode the temporal segments in videos into a compact descriptor. TCBP possesses the ability to capture interactions between each element of the feature representation with one-another over a long-range temporal context. We integrated our previous methods TSiam, SSiam and CCL with TCBP and demonstrated that TCBP has excellent capabilities in learning a strong face representation. We further show TCBP has exceptional transfer abilities to applications such as multimodal video clip representation that jointly encodes images, audio, video and text, and video classification. Abstract iii All of these contributions are demonstrated on benchmark video clustering datasets: The Big Bang Theory, Buffy the Vampire Slayer and Harry Potter 1. We provide extensive evaluations on these datasets achieving a significant boost in performance over the base features, and in comparison to the state-of-the-art results. Acknowledgements Firstly, I would like to thank my supervisor Rainer Stiefelhagen for his guidance, support, advice and most importantly, encouragement. I also thank Tamim Asfour for kindly agreeing to be a reviewer and helping to improve the quality of this thesis. I thank my main collaborators Ali Diba, M. Saquib Sarfraz and Makarand Tapaswi. They have been an inspiration to me. Their daily and sometimes weekly conversations have proved to be one of my best learning experiences during my Ph.D. thesis. Special thanks to Corinna Haas-Hecker for her help with the university’s administra- tive things and making me feel welcome. I thank my previous supervisor, Luc Van Gool at KU Leuven/ETH Zürich. He is one of the most intelligent and inspiring people I have known. He taught me how to think outside of the box and introduced me to the field of computer vision. I would also like to thank my other advisors at Massachusetts Institute of Technology, Harvard Medical School and Harvard University. Ramesh Raskar is to be thanked for his brilliant ideas and inspiration. Rajiv Gupta inspired me to work harder and better on problems that can impact billions of lives. Mauricio Santillana inspired me to work on tracking epidemic outbreaks. I thoroughly enjoyed our weekly conversations to bounce ideas off of and debate with. I thank all other collaborators including Davy Neven, Michael S. Brown, Gabriela Csurka, Naila Murray, Diane Larlus, Ali Pazandeh, Hamed Pirsiavash, Veith Röth- lingshöfer, Mohsen Fayyaz, Manohar Paluri, Juergen Gall, Praneeth Vepakomma, Tristan Swedish, Ken Chang, Jayashree Kalpathy-Cramer, Congcong Wang, Faouzi Alaya Cheikh, Azeddine Beghdadi, Ole Jacob Elle, Jon Yngve Hardeberg, Sony vi Abstract George, ... for your support and help. The final three dots should be interpreted as the people I forgot to mention here and as an apology. Further, during my Ph.D. thesis, I was lucky and fortunate enough to find wonderful numerous friends. Without their company my social life would have been incomplete. I would also like to thank them. These include Keni Bernardin (the most available, reliable and lively party guy I have known. Without you, I would have never known the real music: James Brown, funky jazz, acid jazz, blues and many more, thanks for that!), Ertunc Ertekin (my unstoppable boxing coach), André Nordbø (my always reliable soundboard for discussions), Haroon Mughal, Constantin Seibold (after 4am we start working), Sebastian Bullinger, Piyush Verma, Abhishek Singh, Subhash Chandra Sadhu, Himi Mathur, Connor Henley (Boston nightlife belongs to you!), Samuel Schöb (The Muddy Charles Pub misses you!), Kevin Marty (my MIT social backbone), Fatima Villa (coffee tastes better when you’re around), Guy Satat, Tjada Schult, Hao Li (the after MIT tech review party stays memorable!), Backtosch Mustafa (you indeed have impressive social skills - Harvard is complementary!), César Roberto de Souza, Uta Büchler, Patrick Buehler, Biagio Brattoli, Alessandra Bernardi, Rafael Rezende, Lagnojita Sinha, Tomohiro Maeda, Arun Nair (we share the pride of ACMMM’19 best paper together!), Eric Marengaux (you own Mont Aiguille!), Christoph Feichtenhofer, David Ross, Du Tran, Sourish Chaudhuri, Georg Buchner, Anca Nicoleta Ciubotaru, Pablo Galindo Zapata, Bert De Brabandere, Daniel Koester, Elsa Itambo, Amey Chaware, Ankit Ranjan, Syed Qasim Bukhari (the guy who knows the A-Z of human connections and communication), Ewa Szyszka, Prithvi Rajasekaran, Andres Mafla Delgado, Martin Humenberger (I agree, you told me, graduating earlier is better), Ferran Hueto Puig, Leonie Malzacher, Eva Schlindwein (my first awesome unofficial Ph.D. examiner - secret stays between us!), ... Most importantly, I would like to thank my family for their inestimable support, understanding, love, care and believing in me. Contents 1 Introduction.........................1 1.1 Objective and motivation...................1 1.2 Key challenges