DEGREE PROJECT IN COMPUTER SCIENCE AND ENGINEERING, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2019 A Novel SLAM Quality Evaluation Method FELIX CAESAR KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE A Novel SLAM Quality Evaluation Method FELIX CAESAR Master’s in System, Control and Robotics Date: July 1, 2019 Supervisor: Jana Tumova Examiner: John Folkesson School of Electrical Engineering and Computer Science Host company: ÅF Swedish title: En ny metod för kvalitetsbedömning av SLAM iii Abstract Autonomous vehicles have grown into a hot topic in both research and indus- try. For a vehicle to be able to run autonomously, it needs several different types of systems to function properly. One of the most important of them is simultaneous localization and mapping (SLAM). It is used for estimating the pose of the vehicle and building a map of the environment around it based on sensor readings. In this thesis we have developed an novel approach to mea- sure and evaluate the quality of a landmark-based SLAM algorithm in a static environment. The error measurement evaluation is a multi-error function and consists of the following error types: average pose error, maximum pose error, number of false negatives, number of false positives and an error relating to the distance when landmarks are added into the map. The error function can be tailored towards specific applications by settings different weights for each error. A small research concept car with several different sensors and an outside tracking system was used to create several datasets. The datasets include three different map layouts and three different power settings on the car’s engine to create a large variability in the datasets. FastSLAM and EKF-SLAM were to test the proposed SLAM evaluation method. A comparison to just the pose error was made to asses if our method can provide more information concern- ing establishing SLAM quality. Our results show that the pose error is often a good enough indicator of SLAM quality. However it can occasionally be mis- leading with errors related to mapping (location of landmarks, false negative and false positive landmarks). By using the method presented in this thesis, errors relating to the mapping will be more easily detected than by looking at the pose error. iv Sammanfattning Autonoma fordon har vuxit till ett viktigt ämne både inom forskning och in- dustri. För att ett fordon ska kunna köras autonomt behövs det att flera olika system fungerar korrekt. En av de viktigaste av dem är simultaneous localiza- tion and mapping (SLAM). Det används för att uppskatta fordonets position samt för att bygga en karta av miljön runt fordonet. I det här examensarbe- tet har vi utvecklat en ny metod för att mäta och utvärdera kvaliteten på en landmärkes-baserad SLAM-algoritm i en statisk miljö. Metoden består av föl- jande feltyper: medelvärdet av positionsfelet, det maximala positionsfelet, an- tal falskt negativa fel, antal falskt positiva fel samt ett fel relaterat till avståndet när ett landmärke läggs till i kartan. Genom att använda vikter för varje fel kan metoden skräddarsys till en specifik applikation. En liten konceptbil med flera olika sensorer och ett yttre spårningssystem användes för att skapa flera dataset. Dataseten innehåller tre olika kartlayouter och tre olika effektinställningar på bilen för att skapa stor variation i datase- ten. FastSLAM och EKF-SLAM användes vid testningen av den nya metoden för kvalitetsbedömning av SLAM. Den nya metoden jämfördes mot positions- felet för att analysera ifall den nya metoden är ett bättre sätt att mäta SLAM- kvalitet. Våra resultat visar att positionsfelet ofta är en tillräckligt bra indikator för SLAM-kvalitet, men det kan ibland vara vilseledande gällande fel i kart- läggningen (positioner av landmärken, falskt negativa fel och falskt positiva fel). Genom att använda metoden som presenteras i det här examensarbete är fel som är relaterade till kartläggningen lättare att upptäcka än om man enbart kollar på positionsfelet. v Acknowledgements I would like to thank my Supervisor at KTH Jana Tumova for her help dur- ing the thesis. Give a big thanks to ÅF, my supervisor Johan Olsén and my manager Mia Sköld for providing me with the material and help necessary for my thesis. I would also like thank my fellow thesis students: Hugo Skepp- ström, Richard Hedlund and Kristian Whalqvist for helping with different is- sues during the thesis. An extra thanks to Kristian for allowing me to use the EKF-SLAM he implemented. Contents 1 Introduction 1 1.1 Problem Definition . .2 1.2 Research Question and Hypothesis . .3 1.3 Scope . .4 1.4 Sustainability, Societal Impact & Ethics . .4 1.5 Outline . .5 2 Background 6 2.1 SLAM . .6 2.1.1 Formulation and Structure of the SLAM Problem . .7 2.1.2 Map Representations . .8 2.1.3 Data Association . 10 2.1.4 Solutions to the SLAM Problem . 10 2.2 Related Work . 12 2.2.1 Datasets & Creating Ground-Truth . 12 2.2.2 Evaluating the Quality of SLAM . 13 3 Methods 17 3.1 Multi-Error Function . 17 3.1.1 Pose Error . 19 3.1.2 Maximum Pose Error . 20 3.1.3 Map Error . 21 3.1.4 False Negative Error . 21 3.1.5 False Positive Error . 23 3.1.6 Distance Error . 23 3.1.7 Measurement Guidelines . 24 3.2 Experimental Setup . 25 3.2.1 Research Concept Car . 25 3.2.2 Ground-Truth Capturing System . 26 vi CONTENTS vii 3.2.3 Datasets Created . 27 3.2.4 SLAM algorithms Used During Testing . 28 4 Results & Discussion 32 4.1 Weights and Parameters Used in the Error Function . 32 4.2 Turn Datasets . 33 4.2.1 Turn Dataset Result Discussion . 35 4.3 S-curve Datasets . 36 4.3.1 S-curve Dataset Result Discussion . 38 4.4 Loop Datasets . 39 4.4.1 Loop Dataset Result Discussion . 41 4.5 All Results & Multi-Error Function Discussion . 42 5 Future Work & Conclusions 46 5.1 Conclusions . 46 5.2 Future Work . 47 Bibliography 47 A 55 B 56 C 58 Chapter 1 Introduction Automated, autonomous vehicles and mobile robots (AVs) may soon overtake conventional vehicles and dominate the automotive industry. Once AVs reach a state where they are sufficiently reliable and affordable, they will impact sev- eral markets in many industries. It is estimated that AVswill have an economic impact of $1.2 trillion on the American automotive industry [1]. The effects of AVs can already be seen today: autonomous trucks are used for transportation in mines [2], autonomous cars are being tested and evaluated in urban envi- ronments [3] and several consumer grade vehicles already contain rudimentary autonomous functions that reach the threshold of partial automation [4] [5]. The rise of AVs will not only impact conventional automotive industries, but can create new industries such as delivering packages with drones [6]. Strategy Analytic has predicted that fully AVs will create a "Passenger Economy" that will represent a $7 trillion global opportunity in 2050 [7]. The emergence of AVsare carried by the advancement in fields such as: perception, sensors, path planning and machine learning. A field of great importance is called Simulta- neous Localization and Mapping (SLAM). Specifically, SLAM is a problem that arises when an agent does not have access to a map of the environment and needs to locate itself within this unknown environment and simultaneously construct a map of it [8] [9]. Agent can refer to a car, a truck, a robot or some- thing else that is equipped with the necessary equipment to perform SLAM. SLAM as a research area has made great progress over the past 30 years, tran- sitioning to large-scale applications in several industries [10]. In [10] Cadena et al. tries to answer the question ”is SLAM solved?” They reach the conclu- sion that SLAM research is entering the robust-perception age. They argue that the robust-perception age and SLAM today is characterized by four key requirements: robust performance, high-level understanding, resource aware- 1 2 CHAPTER 1. INTRODUCTION ness and task-driven perception and that the question ”is SLAM solved?” can not be answered without specifying a robot/environment/performance com- bination. To really achieve autonomous robots, more research into SLAM is needed. As the number of SLAM algorithms and applications that employ SLAM increase, the importance of evaluating and measuring the quality of SLAM is paramount. In this thesis, the problem of evaluating the quality of a landmark-based SLAM algorithm in a static environment is addressed and a novel approach is developed that incorporates several important aspects of the localization and map creation. 1.1 Problem Definition SLAM can be divided into two different environmental types: static and dy- namic. Static means that all objects in the environment have a constant posi- tion except the agent. In a dynamic environment, besides static objects there are dynamic objects which move around as the mapping takes place. From an AV’s perspective a static environment could e.g., be buildings, intersections, road boundaries and street signs. All of these objects could represent potential landmarks for a SLAM algorithm. Hence, the focus in this thesis will be on static objects for landmark-based SLAM. Almost all evaluations of SLAM algorithms are based only on the pose errror [11] [12] [13]. Depending on the SLAM algorithm used, the map rep- resentation is different. We argue that to get a better measurement of quality the error measurement should be more rigorous and tailored towards specific map representations, as the choice of SLAM algorithm for a particular situa- tion requires in-depth knowledge about the advantages and disadvantages of it [14].