Activity Report 2017 Project-Team THOTH Learning visual models from large-scale data IN COLLABORATION WITH: Laboratoire Jean Kuntzmann (LJK) RESEARCH CENTER Grenoble - Rhône-Alpes THEME Vision, perception and multimedia interpretation Table of contents 1. Personnel :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 1 2. Overall Objectives :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 2 3. Research Program :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 3 3.1. Designing and learning structured models3 3.2. Learning of visual models from minimal supervision5 3.3. Large-scale learning and optimization6 3.4. Datasets and evaluation7 4. Application Domains ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::8 4.1. Visual applications8 4.2. Pluri-disciplinary research9 5. Highlights of the Year ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 9 6. New Software and Platforms :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 9 6.1. ACT-detector9 6.2. Joint object-action learning 10 6.3. BlitzNet 11 6.4. LCR-Net 11 6.5. CKN-seq 11 6.6. CKN-TensorFlow 12 6.7. Stochs 12 6.8. MODL 12 6.9. Loter 13 6.10. SPAMS 13 6.11. MP-Net 13 6.12. LVO 14 6.13. SURREAL 14 7. New Results ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 15 7.1. Visual recognition in images 15 7.1.1. Dynamic Filters in Graph Convolutional Networks 15 7.1.2. LCR-Net: Localization-Classification-Regression for Human Pose 15 7.1.3. Incremental Learning of Object Detectors without Catastrophic Forgetting 17 7.1.4. BlitzNet: A Real-Time Deep Network for Scene Understanding 17 7.1.5. SCNet: Learning semantic correspondence 18 7.1.6. Auxiliary Guided Autoregressive Variational Autoencoders 18 7.1.7. Areas of Attention for Image Captioning 18 7.1.8. Enhancing Energy Minimization Framework for Scene Text Recognition with Top-Down Cues 20 7.1.9. Learning deep face representations using small data 21 7.1.10. Invariance and Stability of Deep Convolutional Representations 21 7.1.11. Weakly-supervised learning of visual relations 21 7.1.12. Learning from Synthetic Humans 22 7.2. Visual recognition in videos 24 7.2.1. Detecting Parts for Action Localization 24 7.2.2. Learning from Web Videos for Event Classification 24 7.2.3. Learning Motion Patterns in Videos 24 7.2.4. Learning Video Object Segmentation with Visual Memory 26 7.2.5. Learning to Segment Moving Objects 26 7.2.6. Joint learning of object and action detectors 27 7.2.7. Action Tubelet Detector for Spatio-Temporal Action Localization 27 7.3. Large-scale statistical learning 28 2 Activity Report INRIA 2017 7.3.1. Stochastic Optimization with Variance Reduction for Infinite Datasets with Finite-Sum Structure 28 7.3.2. Catalyst Acceleration for First-order Convex Optimization: from Theory to Practice 29 7.3.3. A Generic Quasi-Newton Algorithm for Faster Gradient-Based Optimization 29 7.3.4. Catalyst Acceleration for Gradient-Based Non-Convex Optimization 29 7.4. Machine learning and pluri-disciplinary research 30 7.4.1. Predicting Transcription Factor Binding Sites with Convolutional Kernel Networks 30 7.4.2. Loter: Inferring local ancestry for a wide range of species 30 7.4.3. High Dimensional Classification with combined Adaptive Sparse PLS and Logistic Regression 31 7.4.4. Probabilistic Count Matrix Factorization for Single Cell Expression Data Analysis 32 8. Bilateral Contracts and Grants with Industry ::::::::::::::::::::::::::::::::::::::::::::: 32 8.1. MSR-Inria joint lab: scientific image and video mining 32 8.2. MSR-Inria joint lab: structured large-scale machine learning 34 8.3. Amazon 34 8.4. Intel 34 8.5. Facebook 34 8.6. Xerox Research Center Europe 34 8.7. Naver 35 9. Partnerships and Cooperations ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 35 9.1. Regional Initiatives 35 9.2. National Initiatives 35 9.2.1. ANR Project Macaron 35 9.2.2. ANR Project DeepInFrance 35 9.3. European Initiatives 36 9.3.1.1. ERC Advanced grant Allegro 36 9.3.1.2. ERC Starting grant Solaris 36 9.4. International Initiatives 36 9.4.1. Inria Associate Teams Not Involved in an Inria International Labs 36 9.4.2. Inria International Partners 37 9.4.3. Participation in Other International Programs 37 9.5. International Research Visitors 37 10. Dissemination :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: 37 10.1. Promoting Scientific Activities 37 10.1.1. Scientific Events Organisation 37 10.1.2. Scientific Events Selection 38 10.1.2.1. Member of the Conference Program Committees 38 10.1.2.2. Reviewer 38 10.1.3. Journal 38 10.1.3.1. Member of the Editorial Boards 38 10.1.3.2. Reviewer - Reviewing Activities 38 10.1.4. Invited Talks 38 10.1.5. Scientific Expertise 39 10.1.6. Research Administration 39 10.2. Teaching - Supervision - Juries 40 10.2.1. Teaching 40 10.2.2. Supervision 40 10.2.3. Juries 41 10.3. Popularization 41 11. Bibliography :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::41 Project-Team THOTH Creation of the Team: 2016 January 01, updated into Project-Team: 2016 March 01 Keywords: Computer Science and Digital Science: A3.4. - Machine learning and statistics A5.3. - Image processing and analysis A5.4. - Computer vision A5.9. - Signal processing A6.2.6. - Optimization A8.2. - Optimization A9.2. - Machine learning A9.3. - Signal analysis A9.7. - AI algorithmics Other Research Topics and Application Domains: B5.6. - Robotic systems B8.4. - Security and personal assistance B8.5. - Smart society B9.4.1. - Computer science B9.4.5. - Data science 1. Personnel Research Scientists Cordelia Schmid [Team leader, Inria, Senior Researcher, HDR] Karteek Alahari [Inria, Researcher] Julien Mairal [Inria, Researcher, “en détachement du corps des mines”, HDR] Grégory Rogez [Inria, Starting Research Position] Jakob Verbeek [Inria, Senior Researcher, HDR] Post-Doctoral Fellow Henrique Morimitsu [Inria] PhD Students Alberto Bietti [Univ. Grenoble Alpes, funded by MSR-Inria joint centre, from Sep 2016 to Sep 2019] Dexiong Chen [Univ. Grenoble-Alpes, from Sep 2017 to Sep 2020] Nicolas Chesneau [Univ. Grenoble-Alpes, funded by ERC Allegro] Thomas Dias-Alves [Univ. Grenoble-Alpes, from Sep 2014 until Sep 2017] Mikita Dvornik [Univ. Grenoble-Alpes, funded by ERC Allegro and ANR Macaron, from Sep 2016 to Sep 2019] Maha Elbayad [Univ Grenoble Alpes, funded by Persyval DeCoRe project, from Oct 2016 until Sep 2019] Roman Klokov [Inria, from Sep 2017 until Aug 2020] Andrei Kulunchakov [Inria, funded by ERC Solaris, from Sep 2017 to Sep 2020] Hongzhou Lin [Univ. Grenoble-Alpes, from Sep 2014 to Dec 2017] Pauline Luc [Univ. Grenoble, funded by Facebook, from Jan 2016 to Dec 2018] Thomas Lucas [Univ. Grenoble Alpes, from Feb 2016 to Sep 2019] Alexander Pashevich [Inria, from Sep 2017] 2 Activity Report INRIA 2017 Alexandre Sablayrolles [Univ. Grenoble, funded by Facebook, from Mar 2017] Konstantin Shmelkov [Inria] Vladyslav Sydorov [Inria] Pavel Tokmakov [Inria] Nitika Verma [Univ. Grenoble Alpes, from Sep 2017 to Sep 2020] Daan Wynen [Inria, from Sep 2015 to Sep 2017] Technical staff Ghislain Durif [Inria, from Feb 2017] Vasiliki Kalogeiton [Inria, until Nov 2017] Xavier Martin [Inria] Interns Dexiong Chen [Inria, from Apr 2017 until Aug 2017] Vasileios Choutas [Inria, from Apr 2017] Pau de Jorge Aranda [Inria, from Feb 2017 until Jun 2017] Andrei Kulunchakov [Inria, from Feb 2017 until Jun 2017] Erwan Le Roux [Inria, from Feb 2017 until Dec 2017] Alexander Pashevich [Inria, from Feb 2017 until Jun 2017] Sergey Rubtsovenko [Inria, from Feb 2017 until Aug 2017] Administrative Assistant Nathalie Gillot [Inria] Visiting Scientists Francisco Manuel Castro [University of Malaga, Spain, from Mar 2017 until Jun 2017] Courtney Paquette [University of Washington, Apr 2017] Juha Ylionas [Aalto University, from Jan 2017 until Mar 2017] Gunnar Atli Sigurdsson [Carnegie Mellon University, USA, from Jul 2017 until Nov 2017] External Collaborators Marco Pedersoli [École de technologie supérieure, Montreal, Canada] Danila Potapov [until Jan 2017] 2. Overall Objectives 2.1. Overall Objectives In 2018, it is expected that nearly 80% of the Internet traffic will be due to videos, and that it would take an individual over 5 million years to watch the amount of video that will cross global IP networks each month by then. Thus, there is a pressing and in fact increasing demand to annotate and index this visual content for home and professional users alike. The available text and speech-transcript metadata is typically not sufficient by itself for answering most queries, and visual data must come into play. On the other hand, it is not imaginable to learn the models of visual content required to answer these queries by manually and precisely annotating every relevant concept, object, scene, or action category in a representative sample of everyday conditions—if only because it may be difficult, or even impossible to decide a priori what are the relevant categories and the proper granularity level. This suggests reverting back to the original metadata as source of annotation, despite the fact that the information it provides is typically sparse (e.g., the location and overall topic of newscasts in a video archive) and noisy (e.g., a movie script may tell us that two persons kiss in some scene, but not when, and the kiss may occur off screen or not have survived the final cut). On the other hand, this weak form of “embedded annotation” is rich and diverse, and mining