Superpixels and their Application for Visual Place Recognition in Changing Environments von der Fakult¨atf¨ur Elektrotechnik und Informationstechnik der Technischen Universit¨atChemnitz genehmigte Dissertation zur Erlangung des akademischen Grades Doktor der Ingenieurwissenschaften (Dr.-Ing.) vorgelegt von Dipl.-Inf. Peer Neubert geboren am 10.11.1984 in Karl-Marx-Stadt Eingereicht am 18. August 2015 Gutachter: Prof. Dr.-Ing. Peter Protzel Prof. Dr. Achim Lilienthal Tag der Verleihung: 1. Dezember 2015 Peer Neubert Superpixels and their Application for Visual Place Recognition in Changing Environments Dissertation, Fakult¨at f¨ur Elektrotechnik und Informationstechnik Technische Universit¨at Chemnitz, Dezember 2015 Keywords superpixel, visual place recognition in changing environments, image segmentation, segmen- tation benchmark, simple linear iterative clustering, seeded watershed, appearance change prediction, visual landmarks Abstract Superpixels are the results of an image oversegmentation. They are an established in- termediate level image representation and used for various applications including object detection, 3d reconstruction and semantic segmentation. While there are various ap- proaches to create such segmentations, there is a lack of knowledge about their proper- ties. In particular, there are contradicting results published in the literature. This thesis identifies segmentation quality, stability, compactness and runtime to be important prop- erties of superpixel segmentation algorithms. While for some of these properties there are established evaluation methodologies available, this is not the case for segmentation stability and compactness. Therefore, this thesis presents two novel metrics for their evaluation based on ground truth optical flow. These two metrics are used together with other novel and existing measures to create a standardized benchmark for superpixel algorithms. This benchmark is used for extensive comparison of available algorithms. The evaluation results motivate two novel segmentation algorithms that better balance trade-offs of existing algorithms: The proposed Preemptive SLIC algorithm incorporates a local preemption criterion in the established SLIC algorithm and saves about 80 % of the runtime. The proposed Compact Watershed algorithm combines Seeded Watershed segmentation with compactness constraints to create regularly shaped, compact super- pixels at the even higher speed of the plain watershed transformation. Operating autonomous systems over the course of days, weeks or months, based on vi- sual navigation, requires repeated recognition of places despite severe appearance changes as they are for example induced by illumination changes, day-night cycles, changing weather or seasons - a severe problem for existing methods. Therefore, the second part of this thesis presents two novel approaches that incorporate superpixel segmentations in place recognition in changing environments. The first novel approach is the learning of systematic appearance changes. Instead of matching images between, for example, sum- mer and winter directly, an additional prediction step is proposed. Based on superpixel vocabularies, a predicted image is generated that shows, how the summer scene could look like in winter or vice versa. The presented results show that, if certain assumptions on the appearance changes and the available training data are met, existing holistic place recognition approaches can benefit from this additional prediction step. Holistic approaches to place recognition are known to fail in presence of viewpoint changes. Therefore, this thesis presents a new place recognition system based on local landmarks and Star-Hough. Star-Hough is a novel approach to incorporate the spatial arrangement of local image features in the computation of image similarities. It is based on star graph models and Hough voting and particularly suited for local features with low spatial precision and high outlier rates as they are expected in the presence of appearance changes. The novel landmarks are a combination of local region detectors and descriptors based on convolutional neural networks. This thesis presents and evaluates several new approaches to incorporate superpixel segmentations in local region detection. While the proposed system can be used with different types of local regions, in particular the combi- nation with regions obtained from the novel multiscale superpixel grid shows to perform superior to the state of the art methods - a promising basis for practical applications. Contents List of Acronyms and Terms IX 1 Introduction and Motivation 11 1.1 Thesis outline, research questions and contributions . 14 1.2 Publications and collaboration . 16 1.3 Open source software releases. 17 1.4 My personal journey to this thesis....................... 17 I Superpixel Segmentation Algorithms and Benchmarks 19 2 An Introduction to Superpixel Segmentations 21 2.1 The problem of image segmentation. 21 2.2 Whataresuperpixels? ................................ 24 2.3 Creating superpixel segmentations . 26 2.3.1 Fundamentals of image (over-) segmentation techniques. 27 2.3.2 Algorithms . 31 2.3.3 Summary of the presented superpixel segmentation algorithms . 41 3 How to Compare Superpixel Segmentation Algorithms 45 3.1 Superpixel segmentation benchmarks. 46 3.2 Experimentalsetup................................. 49 3.2.1 Figure-ground segmentation dataset . 52 3.2.2 Using datasets with ground truth optical flow. 52 3.2.3 Compared algorithms . 56 3.3 Runtime .......................................... 58 3.4 Segmentation quality . 59 3.4.1 Boundary recall . 60 3.4.2 Undersegmentation error . 60 3.4.3 Maximum achievable segmentation accuracy. 61 3.4.4 Results on segmentation quality. 62 3.5 The stability of superpixel segmentations . 64 3.5.1 Benchmarking robustness to affine transformations . 64 3.5.2 Robustnesstonoise ................................. 66 3.5.3 A novel metric for stability based on ground truth optical flow . 66 3.5.4 Results on segmentation stability. 67 V Contents 3.6 Compactness properties of superpixel segmentations. 71 3.6.1 Compactness from geometric properties . 72 3.6.2 A novel metric measuring the accordance with motion disconti- nuities............................................ 74 3.6.3 Results on segmentation compactness . 75 3.7 Summary of the benchmark results ........................ 78 4 Two Novel Superpixel Algorithms: Preemptive SLIC and Compact Watershed 81 4.1 Trade-offs of superpixel segmentation algorithms . 81 4.2 Preemptive SLIC: Making a good algorithm faster . 84 4.3 ResultsofPreemptiveSLIC............................ 87 4.4 Compact Watershed: Getting from oversegmentations to superpixels. 89 4.5 Results of Compact Watershed. 96 4.6 Conclusion......................................... 100 II Superpixel for Place Recognition in Changing Environments 103 5 The Problem of Place Recognition in Changing Environments 105 5.1 Datasets comprising changing environments . 108 5.1.1 The Nordland dataset - A 3000 km journey through all seasons. 108 5.1.2 The Gardens Point dataset - Day and night with lateral shift . 110 5.1.3 The Alderley dataset - From sunny day to rainy night . 112 5.2 Related work and own contributions to the field . 112 5.3 Using precision-recall curves for evaluation of place recognition exper- iments............................................. 121 6 Superpixel-based Appearance Change Prediction 123 6.1 The concept of appearance change prediction for long term localization 123 6.1.1 Related work for appearance change prediction. 125 6.1.2 Overview of the main components . 125 6.2 Superpixel descriptors............................. 126 6.2.1 A gentle introduction to descriptors . 127 6.2.2 From keypoint descriptors to recognizing superpixels . 129 6.2.3 What can a descriptor tell about superpixel overlap? . 130 6.2.4 Experimental comparison . 132 6.3 Superpixelvocabularies ............................. 135 6.3.1 A gentle introduction to visual vocabularies . 135 6.3.2 Evaluating the semantics of superpixel words . 137 6.4 Algorithmic approach SP-ACP: Learning to predict scene changes. 140 6.4.1 Learning a vocabulary for summer and winter . 141 6.4.2 Learning a dictionary to translate between vocabularies. 142 6.4.3 Predicting image appearances across seasons . 142 VI Contents 6.5 Results on the Nordland dataset . 143 6.5.1 Applying SP-ACP: Predicting images of the Nordland dataset . 145 6.5.2 Experiments with FAB-MAP............................ 145 6.5.3 Extending and improving BRIEF-Gist .................. 145 6.5.4 Extending and improving SeqSLAM . 148 6.6 Current limitations of the approach and future work . 155 7 Landmark-based Place Recognition in Changing Environments 157 7.1 The components of place recognition based on local image features . 157 7.2 Local region descriptors based on Convolutional Neural Networks . 159 7.2.1 What is a Convolutional Neural Network? . 159 7.2.2 CNN for place recognition . 160 7.2.3 Remarks on the computational efforts for CNN-based descriptors 161 7.3 The novel Star-Hough matching scheme: From local regions to image similarities........................................ 162 7.3.1 The basic problem and the challenges of changing environments . 163 7.3.2 The novel Star-Hough approach. 164 7.4 Novel superpixel-based local region detectors . 166 7.4.1 Using stable superpixel algorithms: Plain superpixels and seg- mentsoups........................................... 168 7.4.2 Using compact superpixel algorithms: The multi scale superpixel