Public Driving Datasets and Toolsets for Autonomous Driving Virtual Test
Total Page:16
File Type:pdf, Size:1020Kb
Perspective, Survey and Trends: Public Driving Datasets and Toolsets for Autonomous Driving Virtual Test Pengliang Ji, Li RuanB, and Limin Xiao Yunzhi Xue, Qian Dong State Key Laboratory of Software Development Environment State Key Laboratory of Computer Science School of Computer Science and Engineering Institute of Software & Chinese Academy of Sciences Beihang University Beijing, China Beijing, China [email protected], [email protected] [email protected], [email protected], [email protected] Abstract—Owing to the merits of early safety and reliability first serious fatal accident of an autonomous driving car [5]. guarantee, autonomous driving virtual testing has recently gains Extensive research [3] [4] shows that autonomous driving increasing attention compared with closed-loop testing in real faces more demanding safety and reliability requirements scenarios. Although the availability and quality of autonomous driving datasets and toolsets are the premise to diagnose the than traditional automotive. autonomous driving system bottlenecks and improve the system A. Challenges and Problems performance, due to the diversity and privacy of the datasets and toolsets, collecting and featuring the perspective and quality With the rapid development of deep learning that covers of them become not only time-consuming but also increasingly end-to-end learning, owing to the merits of early safety and challenging. This paper first proposes a Systematic Literature reliability guarantee, autonomous driving virtual testing has review approach for Autonomous driving tests (SLA), then recently gains increasing attention compared with closed- presents an overview of existing publicly available datasets and toolsets from 2000 to 2020. Quantitative findings with loop testing in real scenarios [6] [7]. With the rapid de- the scenarios concerned, perspectives and trend inferences and velopment of big data-driven, extensive research shows the suggestions with 35 automated driving test tool sets and 70 test availability and quality of autonomous driving datasets and data sets are also presented. To the best of our knowledge, we toolsets are the premise to diagnose the autonomous driving are the first to perform such recent empirical survey on both the system bottlenecks and improve the system performance [8] datasets and toolsets using a SLA based survey approach. Our multifaceted analyses and new findings not only reveal insights [9] [10]. However, due to the diversity and privacy of the that we believe are useful for system designers, practitioners and datasets and toolsets, collecting and featuring the perspective users, but also can promote more researches on a systematic and quality of datasets and toolsets for virtual testing of survey analysis in autonomous driving surveys on dataset and autonomous driving become not only time-consuming but toolsets. also increasingly challenging. However, the recent research of autonomous driving I. INTRODUCTION datasets and toolsets in autonomous driving virtual testing faces two main challenges as follows: arXiv:2104.00273v4 [cs.RO] 30 Jun 2021 Safety is the most critical requirement in autonomous driv- ing test at the era when the automation level of autonomous • How to survey the recent increasing datasets and driving cars reaches SAE3 or higher where the driving toolsets for autonomous driving virtual test using a responsibility is gradually shifting from traditional drivers systematic method? Recently there are some surveys, for to autonomous driving cars [1] [2]. Recent safety research example [11] and [12], show the urgent demands about [1] [3] [4] shows that autonomous driving still faces huge the datasets and environments of autonomous driving challenges in complex interactive scenarios and unstructured test. However, they paid more attention to the results environments, such as campuses with surging people and analysis and a formal systematic survey process for high-speed fog on highways. For example, Uber recorded the autonomous driving test still lacks. • What are the recent findings and trends of both toolsets This work is by supported by the National Key R&D Program of China and datasets with scenario concerned so far? Although under Grant No. 2017YFB0503400, the National Science Foundation of China under Grant No. 11701545, the fund of the State Key Laboratory datasets and toolsets are the foundations of autonomous of Software Development Environment under Grant No. SKLSDE-2020ZX- tests and there are some recent survey papers on datasets 15. [13] [14], only [15] recently reviewed the virtual test Duing to the page limitations, we have not released detailed information on the toolsets and datasets in this paper. For specific open source datasets platforms. However, [15] paid more attention on explor- and toolsets, please contact [email protected]. ing the simulator of autonomous driving, but lacks tools autonomous driving virtual test specific survey rules to help SLA AuDrTeSurvey Review output Findings and Trends researchers identify and explain the available research related RQ1 RQ1 Answer to collecting datasets and toolsets for autonomous driving vir- GQ. How to benefit the virtual test IEEE Explore, process of autonomousdriving Extract Engineering Village, Paper Count=5285 Search String tual test questions. AuDrTeSurvey mainly includes searching, based on the datasets and toolsets? ACM Library Studies duplicates evaluating, explaining, and summarizing findings and trends Challenging Challenging 3056 Scenesinput First Category inference from autonomous driving research. Based on the input Studies excluded based Toolsets on title and keyword 70 Dataset 1825 Knowledge Map Mapping RQ1 Literature Datasets input Datasets RQ1 RQ1-RQ6 Input Dictionary scientific process, AuDrTeSurvey can provide clear moti- 35 Atomic Variable Toolset Studies excluded based on abstract vation for new work, and new evidence to guide decision- Second 201 Vehicle Settings 98 Other Category Response Variable Paper Count=203 making using autonomous driving datasets and toolsets [18]. Functional Relationship B. Process Fig. 1: Autonomous Driving Virtual Test Systematic Litera- ture Review framework. Based on the framework in Fig.1, we now present the details of our analysis process. The key steps are highlighted as follows: to establish other testing processes. Moreover, they are 1) Design research questions in autonomous driving vir- based on the papers before 2018 which lacks those of tual test. 2020. 2) Perform a knowledge map based test influencing fac- B. Main Contributions tors analysis. First, the most critical papers are clas- To solve these essential challenges in autonomous driving sified into sub-categories which will be denoted as virtual test, we presents a survey of datasets and toolsets Atomic Variable, Response Variable, Vehicle Settings for autonomous driving virtual test from 2000 to 2020. The and the relationship between variables. Then based major contributions of this paper are summarized as follows: on the first-level categories, sub-categories (or atomic factors) are derived. After these steps, a knowledge map • To the best of our knowledge, we are the first propose can be formed. a Systematic Literature Review (SLR) approach for 3) Perform a review of datasets and toolsets in au- autonomous driving tests. tonomous driving test. • We present a review of the datasets and toolsets for 4) Analyze new findings and give research trend infer- automated driving virtual tests based on the investiga- ences with the scenes, influence factors concerned. tion research from 2000 to 2020. Quantitative findings with the scenarios concerned, perspectives and trend Using the literature from 2000 to 2020 as an example, the inferences and suggestions with the most important 35 analysis process and its results will be shown in the following automated driving test tool sets and 70 test data sets SectionIII. are also presented. To the best of our knowledge, we are the first to perform such recent empirical systematic III. CASE STUDY DESIGN survey on both the datasets and toolsets using a SLA based survey approach. A. Design of Research Questions C. Organization As our method is driven by research questions, first the general question (GQ) and the sub-questions are designed. The remainder of this paper is organized as follows: Section II introduces the SLA based autonomous driving • GQ. How to benefit the virtual test process of au- research survey approach. SectionIII introduces results and tonomous driving based on the datasets and toolsets? trends inferences. The paper is concluded in SectionV. Next, GQ was divided into the following six research II. AUDRTESURVEY:SYSTEMATIC LITERATURE REVIEW questions (RQ), and we will mark at the answer to them APPROACH FOR AUTONOMOUS DRIVING VIRTUAL TEST in full text. In this section, we introduce the overall design and process • RQ1. Research from which perspective has an impact of our systematic literature review approach for autonomous on autonomous driving testing? driving virtual test. • RQ2. What data sets and tools for automated driving tests are involved in the research? A. Overall Design • RQ3. Which test scenario variables are mainly involved As is introduced in [16], Systematic Literature Review in the research? (SLR) is a mainstream means of understanding the latest • RQ4. Which technique was used in the test? research tools in a specific field [17]. We introduce SLA into • RQ5. What innovative discoveries or unique features did autonomous driving survey process. Therefore,