Introspective Classification for Robot Perception and Decision Making Hugo Grimmett New College Supervisor: Ingmar Posner Mobile Robotics Group Department of Engineering Science University of Oxford August 2016 Hugo Grimmett Doctor of Philosophy New College August 2016 Introspective Classification for Robot Perception and Decision Making Abstract In robotics, a classifier is often a core component of the decision-making framework. Precision and recall have been widely adopted as canonical metrics to quantify the performance of a classifier, but for applications involving mission-critical decision making, good performance in relation to these metrics is insufficient. The use of a classification framework which produces scores with inappropriate confidences will ultimately lead to the robot making bad decisions, thereby compromising robot or user safety. In order to select a classifier which will make decisions reflecting the nature of the costs, we should pay careful attention to the ways in which it generates scores. We introduce and motivate the importance of a classifier’s introspective capacity: the ability to give an appropriate assessment of confidence with any test case. Classification made confidently must be correct, and mistakes should be made with high uncertainty. A classifier’s capacity to do so must remain consistent despite unusual or surprising test cases. We propose that a key ingredient for introspection is a classifier’s potential to increase its uncertainty with the distance between a test datum and its training data. We define the ideal introspective behaviour, and derive idealised classifiers which serve to benchmark a number of commonly used classification frameworks in a vari- ety of decision-making tasks. We show that classifiers that offer predictive variance at test-time are more cautious and less over-confident than those which consider a single hypothesis or discriminant. However, in high-cost (or high-risk) decision making, none of the classifiers evaluated in this thesis are sufficiently introspective to prevent all potential catastrophic mistakes. We show that in sequential decision- making, when the mapping from score to class is explicitly stated, a classifier’s ability to behave consistently despite non-stationary test data is of primary importance. Statement of Authorship This thesis is submitted to the Department of Engineering Science, University of Oxford, in fulfilment of the requirements for the degree of Doctor of Philosophy. This thesis is entirely my own work, and except where otherwise stated, describes my own research. Hugo Grimmett, New College Funding The work described in this thesis was funded under the European Community’s Seventh Framework Programme (FP7/2007-2013) under Grant Agreement Number 269916 (V-CHARGE). Acknowledgements First and foremost I would like to thank my supervisor, Professor Ingmar Posner. His support, direction, and collaboration know no bounds, and I am immensely grateful for having had the opportunity to work together. I would also like to thank Professor Paul Newman, who inspired me to begin this journey and made me feel at home at MRG. Secondly, I would like to thank my thesis examiners Professors Mike Osbourne and Tom Duckett for their insightful questions and feedback during and after the viva examination. Their thorough treatment of my work has brought to light interesting aspects that I had not previously considered, and for that I have great appreciation. I give great thanks to my friends and coauthors, Drs Rohan Paul, Rudi Triebel, Lina Paz, and Pedro Pinies. The late nights writing papers together are my fondest and most formative memories from the past years. I owe a great deal to countless enlightening discussions with Dr Chi Tong, Ge- off Hester, and Dr Dominic Wang. Their generosity of their immense collective knowledge empowers all of those around them. The funding from the V-Charge project allowed me to pursue this work, and col- laborate and find friends in Paul Furgale, Wojciech Derendarz, Ulrich Schwesinger, and Matthias Buerki to name a few. It has been a pleasure to work with you all, and to contribute to the success that is the V-Charge autonomous valet car. My thanks to all of MRG, who have supported and inspired me throughout the years. To name but a few: Colin McManus, Terry Scott, Dan Withers, Peter Ondruska, Corina Gurau, Matt Gadd, Winston Churchill, Julie Dequaire, Chris Prahacs, Tom Wilcox, and Anita Hancox. Also thank you to those who have since left the group for your immeasurable wisdom, Alastair Harrison, Ashley Napier, Bonolo Mathibela, Ben Davis, and Ian Baldwin. My heartfelt gratitude also goes to my friend Amaury Dame, whose perspectives on life guide me always. I am indebted to those outside robotics who supported me throughout my time in Oxford, most notably but not limited to Philippa Harris, Leila Denniston, Michael West, Alex Nevitte, Jonny Sadler, Christine Moore, Jon Daly, Joanna Hamer, Isaac Black, and Rachel James. Thank you for believing in me. Finally I would like to thank my parents, Geoffrey and Rosine, for their over- whelming and unconditional love and support. Their selflessness and trust are both my inspiration and aspiration. Hugo Grimmett August 11, 2016 Contents 1 Introduction 1 1.1 Motivation................................. 1 1.2 Introspection ............................... 2 1.3 Contributions ............................... 3 1.4 ThesisStructure.............................. 4 1.5 Publications................................ 5 2 Semantic Mapping: A Case Study 7 2.1 TheV-ChargeVehicle .......................... 9 2.2 RelatedWorks............................... 10 2.3 SemanticMapping ............................ 11 2.3.1 TheRoadNetwork ........................ 12 2.3.2 ParkingSpaceLocations . 17 2.3.3 RecommendedDrivingSpeed . 22 2.4 IntegratingMetricandSemanticLayers. .. 23 2.5 Conclusions ................................ 29 3 Data Sets, Features, and Performance Metrics 31 3.1 TrafficLightsRecognition . 32 3.2 GTSRB .................................. 34 3.3 DaimlerPedestrian ............................ 34 CONTENTS 3.4 KITTI................................... 35 3.5 SyntheticData .............................. 35 3.6 Features .................................. 37 3.6.1 TemplateFeatures . 37 3.6.2 HistogramofOrientedGradients(HOG) . 40 3.7 PerformanceMetrics . 40 3.7.1 Precision, Recall, Accuracy, and F-measure . 40 4 Introspection 43 4.1 TheIdealClassifier ............................ 44 4.2 RelatedWorks............................... 48 4.3 IdealisedClassifiers ............................ 51 4.3.1 DeterminingtheDensityFunctions . 55 4.4 Summary ................................. 60 5 Introspection in Practice 61 5.1 Notation.................................. 63 5.2 MeasuresofUncertainty . 63 5.3 ADistance-BasedViewonIntrospection . 65 5.4 RelatedWorks............................... 67 5.5 Commonly-Used Classification Frameworks . .. 69 5.5.1 GaussianProcessesClassification . 70 5.5.2 SupportVectorMachine . 74 5.5.3 LogitBoost............................. 76 5.5.4 RandomForests.......................... 77 5.5.5 Kernels .............................. 79 5.6 AnalysisofNon-StationaryData . 81 5.6.1 SyntheticData .......................... 83 CONTENTS 5.6.2 RealData ............................. 88 5.6.3 Discussion............................. 94 5.7 UncertaintyinDetection . 95 5.8 Conclusions ................................106 6 IntrospectioninDecisionMaking 109 6.1 MakingErrorswithUncertainty . .110 6.2 ActiveLearning..............................113 6.2.1 RelatedWorks ..........................115 6.2.2 TheCross-OverExperiment . .116 6.2.3 Discussion.............................117 6.3 DecisionMakingwithCosts . .121 6.3.1 BayesianDecisionTheory . .122 6.3.2 Relating Costs to ǫ-bounds....................125 6.3.3 Experiments............................126 6.4 Conclusions ................................129 7 Introspection in Sequential Decision Making 133 7.1 The Observation Probability Function as a Classifier . .135 7.2 RelatedWorks...............................137 7.3 TestScenarios...............................139 7.3.1 GridWorld ............................139 7.3.2 WumpusWorld ..........................142 7.4 Entropy ..................................144 7.5 Experiments................................147 7.5.1 TheIllEffectsofNon-stationarity . 149 7.5.2 ConsistentandAppropriateSensors . 155 7.5.3 ChangingtheSizeoftheWorld . .161 CONTENTS 7.5.4 ChangingtheNumberofSensors . .163 7.6 Conclusions ................................165 8 Conclusions and Future Work 167 8.1 Conclusions ................................167 8.2 FutureWork................................170 8.2.1 Introspection . .170 8.2.2 SemanticMapping . .175 8.2.3 ActiveLearning. .175 8.2.4 ClassicalDecisionMaking . .176 8.2.5 SequentialDecisionMaking . .177 Appendices 180 ARoadGraphGenerationAlgorithms 181 B Classifier Probability Contours 184 C Idealised Classifier Error Functions 187 D Empirical Probability Density Functions of Real Classifiers 189 E MDPs and POMDPs 192 E.1 MarkovDecisionProcess(MDP) . .192 E.2 Partially Observable Markov Decision Processes (POMDP)......197 Bibliography 202 Chapter 1 Introduction 1.1 Motivation An autonomous robot operating in an environment such as a city street will see so many varied inputs that it is impossible to have prepared it by labelling each one. In practice we attempt to forewarn it by selecting training exemplars and generating a model, hoping that together they will be sufficient to generalise well to its future experiences.