HW/SW co-design of a visual SLAM application Jonathan Piat, Philippe Fillatreau, Daniel Tortei, François Brenot, Michel Devy To cite this version: Jonathan Piat, Philippe Fillatreau, Daniel Tortei, François Brenot, Michel Devy. HW/SW co- design of a visual SLAM application. Journal of Real-Time Image Processing, Springer Verlag, 2018, 10.1007/s11554-018-0836-2. hal-02049364 HAL Id: hal-02049364 https://hal.laas.fr/hal-02049364 Submitted on 6 Mar 2019 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. HW/SW co‑design of a visual SLAM application Jonathan Piat1,2 · Philippe Fillatreau4 · Daniel Tortei1,2 · Francois Brenot1,3 · Michel Devy1 Abstract Vision-based advanced driver assistance systems (ADAS), appeared in the 2000s, are increasingly integrated on-board mass-produced vehicles, as off-the-shelf low-cost cameras are now available. But ADAS implement most of the time- specific and basic functionalities such as lane departure or control of the distance to other vehicles. Integrating accurate localization and mapping functionalities meeting the constraints of ADAS (high-throughput, low-consumption, and small- design footprint) would pave the way towards obstacle detection, identification and tracking on-board vehicles at potential high speed. While the SLAM problem has been widely addressed by the robotics community, very few embedded operational implementations can be found, and they do not meet the ADAS-related constraints. In this paper, we implement the first 3D monocular EKF-SLAM chain on a heterogeneous architecture, on a single System on Chip (SoC), meeting these constraints. In order to do so, we picked up a standard co-design method (Shaout et al. Specification and modeling of hw/sw co-design for heterogeneous embedded systems, 2009) and adapted it to the implementation of potentially any of such complex processing chains. The refined method encompasses a hardware-in-the-loop approach allowing to progressively integrate hardware accelerators on the basis of a systematic rule. We also have designed original hardware accelerators for all the image processing functions involved, and for some algebraic operations involved in the filtering process. Keywords FPGA · Co-design · SLAM · Machine-vision · ADAS 1 Introduction Advanced driver assistance systems (ADAS) appeared in the * Jonathan Piat 2000s and are now increasingly found on-board mass-pro- [email protected] duced vehicles. They involve various sensors (cameras, lidar, Philippe Fillatreau others) and signal or image processing functionalities (e.g. [email protected] extraction and analysis of features detected in acquired sig- Daniel Tortei nals or images), see [33]. ADAS-related concepts have been [email protected] notably validated by research on perception or navigation for Francois Brenot autonomous mobile robots since the 1980s. Nevertheless, [email protected] ADAS implement most of the time specific and independ- Michel Devy ent functionalities, such as the control of distance to other [email protected] vehicles, or the detection of driver hypovigilance or lane departure, see [27]. Functionalities such as accurate vehicle 1 CNRS, LAAS, 7 avenue du colonel Roche, 31400 Toulouse, France localization and sparse vehicle environment mapping would allow paving the way towards obstacle detection, identifica- 2 Univ de Toulouse, UPS, LAAS, 31400 Toulouse, France tion and tracking (notably thanks to the reduction of the 3 Institut National Polytechnique de Toulouse, complexity of the associated data processing) and towards INPT - University of Toulouse, 4 Allee Emile Monso, 31030 Toulouse, France autonomous navigation of road vehicles at high speed. Self-localization and environment modelling have been 4 Laboratoire Genie de Production de l’Ecole Nationale d’Ingenieurs de Tarbes (LGP-ENIT), Institut National central issues for research on mobile robots since the 1980s Polytechnique de Toulouse (INPT), University of Toulouse, [10], although the SLAM acronym appeared towards the end 47 avenue d’Azereix, BP1629, 65016 Tarbes Cedex, France of the 1990s [14]. Various sensors have been used for the Data Map associated perception tasks, e.g., ultrasonic sensors, laser Association range finders, and cameras. The emergence of off-the-shelf low-cost cameras allows using them in monocular, stereo- Image Features State vision, or panoramic vision-based ADAS systems, see [18]. detection Estimate state The simultaneous localization and mapping (SLAM) issue has now been widely explored by the robotics com- Map Update munity, and numerous academic solutions have been pro- posed. However, very few complete embedded operational chains meeting the constraints associated with ADAS (fast Fig. 1 Simultaneous localization and mapping—SLAM processing times, low-consumption, small-design footprint) can be found in the literature. This is partly due to the intrin- sic complexity of the advanced image processing algorithms developed here, and can be reused for any advanced sig- involved and to the large volume of data (number of pixels) nal or image processing chain. to be processed. • we have used this refined co-design method to integrate The computer vision and robotics perception communi- the first complete monocular 3D EKF-SLAM chain on a ties started in the late 1990s–early 2000s to study the design heterogeneous (hardware/software) architecture on a sin- of specific, embedded and real-time architectures, involv- gle SoC. The obtained performances meet the constraints ing programmable logics, for advanced image processing (processing times, power consumption, design footprint) functionalities, see [35]. Much progress remains to be done of the ADAS. about vision-based SLAM, a complex image processing • we have developed and validated original hardware accel- and numerical filtering chain. To embed such a complex erators (reusable for any embedded SLAM application functionality on an ADAS, one must explore the possibili- or vision application involving the same functionalities) ties offered by available execution resources, to take advan- of all the image processing and of some data process- tage of their specific properties. This leads the designer to ing functions involved in the numerical filtering steps formulate a heterogeneous (or not) processing architecture (Fig. 1). involving both general purpose processing units and appli- cation-specific hardware accelerators. To define an efficient This work was conducted in the context of the Develop- distribution of the computational load over these two kinds ment of an Integrated Camera for Transport Applications of operators, the design needs to be undertaken following (DICTA) cooperative project. The goal of this project is to joint design (called co-design) approaches. integrate advanced ADAS functionalities on the iCam smart This paper deals with the definition of a heterogeneous camera developed by the Delta Technologies company, see architecture for the integration of a complete monocular 3D [13]. extended Kalman filter (EKF) vision-based SLAM process- This paper is organized as follows. In Sect. 2, we present ing chain meeting the constraints of an ADAS. To achieve a state-of-the-art spanning over the basic SLAM principles that, we have studied the introduction of a suitable co-design and different approaches found in the literature, the basics methodology associated with a hardware-in-the-loop (HIL) of extended Kalman filtering, the operational vision-based approach allowing to progressively integrate specifically SLAM implementations in the community and at the LAAS designed hardware accelerators. laboratory, and the hardware/software co-design method- The contribution is threefold : ologies; this section ends by a synthesis of the working hypothesis, the scientific approach followed and the main • we have selected in the literature a standard and generic contributions of this work. Section 3 presents the refined co-design approach [38] which we have (1) refined to and adapted generic co-design methodology proposed here, allow the integration of a complex processing chain such and details the successive steps of the development of our as a vision-based monocular SLAM and (2) adapted to heterogeneous monocular 3D EKF-SLAM chain. The rest encompass a hardware-in-the-loop approach allowing of the paper is organized according to these steps. Section 4 to progressively integrate hardware accelerators; in our deals with step 1 of our co-design methodology, which approach, the hardware accelerators are integrated one defines the architecture model, the application model, and by one, based on the use of a systematic rule based on the constraints on the embedded system. Section 5 presents the results of the profiling of the heterogeneous SLAM iteration 1, performing the initial platform specification, the chain under construction. The refinement and adapta- application model refinement, and the implementation of tion brought to the state-of-the-art method we picked up a software-only prototype. Section 6 deals with iterations [38] are