Visual Object Recognition Kristen Grauman University of Texas at Austin Bastian Leibe RWTH Aachen University SYNTHESIS LECTURES ON COMPUTER VISION # 1 Abstract The visual recognition problem is central to computer vision research. From robotics to information retrieval, many desired applications demand the ability to iden- tify and localize categories, places, and objects. This tutorial overviews computer vision algorithms for visual object recognition and image classification. We introduce primary representations and learning approaches, with an emphasis on recent advances in the field. The target audience consists of researchers or students working in AI, robotics, or vision who would like to understand what methods and representations are available for these problems. This lecture summarizes what is and isn’t possible to do reliably today, and overviews key concepts that could be employed in systems requiring visual categorization. Keywords: More specifically, the topics we cover in detail within the above are: global representations versus local descriptors; detection and description of local invari- ant features; efficient algorithms for matching local features, including tree-based and hashing-based search algorithms; visual vocabularies and bags-of-words; methods to ver- ify geometric consistency according to parameterized geometric transformations; deal- ing with outliers in correspondences, including RANSAC and the Generalized Hough transform; window-based descriptors, including histograms of oriented gradients and rectangular features; part-based models, including star graph models and fully con- nected constellations; pyramid match kernels; detection via sliding windows; Hough voting; Generalized distance transform; the Implicit Shape Model; and the Deformable Part-based Model. v Contents Contents............................................ .........................v 1 Introduction........................................ .........................1 1.1 Overview......................................... ......................1 1.2 Challenges.............................................................3 1.3 The State of the Art...................................................5 2 Overview: Recognition of Specific Objects . ....................7 2.1 Global Image Representations . ...................7 2.2 Local Feature Representations..........................................9 3 Local Features: Detection and Description. ....................11 3.1 Introduction..................................... .....................11 3.2 Detection of Interest Points and Regions . 13 3.2.1 Keypoint Localization 13 3.2.2 Scale Invariant Region Detection 15 3.2.3 Affine Covariant Region Detection 21 3.2.4 Orientation Normalization 23 3.2.5 Summary of Local Detectors 23 3.3 Local Descriptors.....................................................24 3.3.1 The SIFT Descriptor 24 3.3.2 The SURF Detector/Descriptor 25 3.4 Concluding Remarks..................................................27 4 Matching Local Features............................... .....................29 4.1 Efficient Similarity Search.......................... ...................30 4.1.1 Tree-based Algorithms 30 4.1.2 Hashing-based Algorithms and Binary Codes 33 vi 4.1.3 A Rule of Thumb for Reducing Ambiguous Matches 36 4.2 Indexing Features with Visual Vocabularies ........................... 37 4.2.1 Creating a Visual Vocabulary 37 4.2.2 Vocabulary Trees 39 4.2.3 Choices in Vocabulary Formation 40 4.2.4 Inverted File Indexing 42 4.3 Concluding Remarks..................................................42 5 Geometric Verification of Matched Features . ..................44 5.1 Estimating Geometric Models........................ .................44 5.1.1 Estimating Similarity Transformations 45 5.1.2 Estimating Affine Transformations 45 5.1.3 Homography Estimation 46 5.1.4 More General Transformations 48 5.2 Dealing with Outliers.................................................49 5.2.1 RANSAC 50 5.2.2 Generalized Hough Transform 52 5.2.3 Discussion 53 6 Example Systems: Specific-Object Recognition . ..................55 6.1 Image Matching.................................... ...................55 6.2 Object Recognition ...................................................55 6.3 Large-Scale Image Retrieval...........................................56 6.4 Mobile Visual Search..................................................56 6.5 Image Auto-Annotation...............................................58 6.5.1 Conclusions 59 7 Overview: Recognition of Generic Object Categories. ..................61 8 Representations for Object Categories . .....................62 8.1 Window-based Object Representations . ................62 8.1.1 Pixel Intensities and Colors 62 vii 8.1.2 Window Descriptors: Global Gradients and Texture 63 8.1.3 Patch Descriptors: Local Gradients and Texture 64 8.1.4 A Hybrid Representation: Bags of Visual Words 67 8.1.5 Contour and Shape Features 68 8.1.6 Feature Selection 68 8.2 Part-based Object Representations....................................69 8.2.1 Overview of Part-Based Models 70 8.2.2 Fully-Connected Models: The Constellation Model 73 8.2.3 Star Graph Models 75 8.3 Mixed Representations................................................76 8.4 Concluding Remarks..................................................76 9 Generic Object Detection: Finding and Scoring Candidates . ................78 9.1 Detection via Classification . ...................78 9.1.1 Speeding up Window-based Detection 79 9.1.2 Limitations of Window-based Detection 80 9.2 Detection with Part-based Models.....................................82 9.2.1 Combination Classifiers 82 9.2.2 Voting and the Generalized Hough Transform 83 9.2.3 RANSAC 84 9.2.4 Generalized Distance Transform 85 10 Learning Generic Object Category Models . .................87 10.1 Data Annotation.................................. ....................87 10.2 Learning Window-based Models.......................................89 10.2.1Specialized Similarity Measures and Kernels 89 10.3 Learning Part-based Models ......................................... 100 10.3.1Learning in the Constellation Model 100 10.3.2Learning in the Implicit Shape Model 100 10.3.3Learning in the Pictorial Structure Model 101 11 Example Systems: Generic Object Recognition . ................104 viii CONTENTS 11.1 The Viola-Jones Face Detector.......................................104 11.1.1Training Process 104 11.1.2Recognition Process 106 11.1.3Discussion 107 11.2 The HOG Person Detector...........................................108 11.3 Bag-of-Words Image Classification ................................... 109 11.3.1Training Process 109 11.3.2Recognition Process 110 11.3.3Discussion 110 11.4 The Implicit Shape Model ........................................... 111 11.4.1Training Process 111 11.4.2Recognition Process 112 11.4.3Vote Backprojection and Top-Down Segmentation 112 11.4.4Hypothesis Verification 114 11.4.5Discussion 115 11.5 Deformable Part-based Models.......................................116 11.5.1Training Process 116 11.5.2Recognition Process 116 11.5.3Discussion 118 12 Other Considerations and Current Challenges . .................120 12.1 Benchmarks and Datasets........................... .................120 12.2 Context-based Recognition...........................................123 12.3 Multi-Viewpoint and Multi-Aspect Recognition . 124 12.4 Role of Video........................................................126 12.5 Integrated Segmentation and Recognition . 126 12.6 Supervision Considerations in Object Category Learning . 127 12.6.1Using Weakly Labeled Image Data 128 12.6.2Maximizing the Use of Manual Annotations 129 12.6.3Unsupervised Object Discovery 129 12.7 Language, Text, and Images ......................................... 130 CONTENTS ix 13 Conclusions......................................... .......................132 Preface This lecture summarizes the material in a tutorial we gave at AAAI 2008 (Grau- man & Leibe 2008)1. Our goal is to overview the types of methods that figure most prominently in object recognition research today, in order to give a survey of the concepts, algorithms, and representations that one might use to build a visual recognition system. Recognition is a rather broad and quickly moving field, and so we limit our scope to methods that are already used fairly frequently in the literature. As needed, we point the reader to outside references for more details on certain topics. We assume that the reader has basic familiarity with machine learning algo- rithms for supervised classification, and some background in low-level image processing. Outline: The text is divided primarily according to the two forms of recogni- tion: specific and generic. In the first chapter, we motivate those two problems and point out some of their main challenges. Chapter 2 then overviews the problem of specific object recognition, and the global and local representations used by current approaches. Then, in Chapters 3 through 5, we explain in detail how today’s state-of- the-art methods utilize local features to identify an object in a new image. This entails efficient algorithms for local feature extraction, for retrieving candidate matches, and for performing geometric verification. We wrap up our coverage of specific objects by outlining example end-to-end systems from recent work in Chapter 6, pulling together the key steps from the prior chapters on local features and matching. Chapter 7 introduces the generic (category-level) recognition problem, and