Video-Based Lane Estimation and Tracking for Driver Assistance: Survey, System, and Evaluation
Total Page:16
File Type:pdf, Size:1020Kb
UC San Diego UC San Diego Previously Published Works Title Video-based lane estimation and tracking for driver assistance: Survey, system, and evaluation Permalink https://escholarship.org/uc/item/1bg5f8qd Journal IEEE Transactions on Intelligent Transportation Systems, 7(1) ISSN 1524-9050 Authors McCall, J C Trivedi, Mohan Manubhai Publication Date 2006-03-01 DOI 10.1109/TITS.2006.869595 Peer reviewed eScholarship.org Powered by the California Digital Library University of California 20 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 7, NO. 1, MARCH 2006 Video-Based Lane Estimation and Tracking for Driver Assistance: Survey, System, and Evaluation Joel C. McCall and Mohan M. Trivedi Abstract—Driver-assistance systems that monitor driver intent, emerging [7]. With such a wide variety of system objectives, warn drivers of lane departures, or assist in vehicle guidance are it is important that we examine how lane position is detected all being actively considered. It is therefore important to take and measure performance with relevant metrics in a variety of a critical look at key aspects of these systems, one of which is lane-position tracking. It is for these driver-assistance objectives environmental conditions. that motivate the development of the novel “video-based lane There are three major objectives of this paper. The first is estimation and tracking” (VioLET) system. The system is designed to present a framework for comparative discussion and devel- using steerable filters for robust and accurate lane-marking detec- opment of lane-detection and position-estimation algorithms. tion. Steerable filters provide an efficient method for detecting cir- The second is to present the novel “video-based lane estimation cular-reflector markings, solid-line markings, and segmented-line markings under varying lighting and road conditions. They help and tracking” (VioLET) system designed for driver assistance. in providing robustness to complex shadowing, lighting changes The third is to present a detailed evaluation of the VioLET from overpasses and tunnels, and road-surface variations. They system by performing an extensive set of experiments using are efficient for lane-marking extraction because by computing an instrumented-vehicle test bed. To this end, the paper is only three separable convolutions, we can extract a wide variety arranged in the following manner. In Section II, we will first of lane markings. Curvature detection is made more robust by in- corporating both visual cues (lane markings and lane texture) and explore the system objectives, environmental variations, and vehicle-state information. The experiment design and evaluation sensing modalities involved in creating a lane-position tracker. of the VioLET system is shown using multiple quantitative metrics In Section III, we will introduce a common framework for over a wide variety of test conditions on a large test path using a lane-position-tracking systems, which we will use to provide unique instrumented vehicle. A justification for the choice of met- comparisons between existing systems based on the objectives, rics based on a previous study with human-factors applications as well as extensive ground-truth testing from different times of day, conditions, and sensing systems described in the introduction. road conditions, weather, and driving scenarios is also presented. Next, in Section IV, we will present the VioLET system, a In order to design the VioLET system, an up-to-date and compre- lane-position detection and tracking system with its design hensive analysis of the current state of the art in lane-detection based upon a driver-assistance system for use in a highway research was first performed. In doing so, a comparison of a wide road environment. Finally, in Section V, we will evaluate the variety of methods, pointing out the similarities and differences between methods as well as when and where various methods are VioLET system with both: a) a wide variety of performance most useful, is presented. metrics that are relevant to the system objectives; and b) a wide range of environmental variations and driving contexts. Index Terms—Active safety systems, intelligent vehicles, lane detection, lane-departure warning, machine vision, performance The contributions of this research extend to five areas. metrics. 1) The introduction of a fully integrated lane-estimation- and-tracking system with specific applicability to I. INTRODUCTION driver-assistance objectives. By working closely with ITHIN the last few years, research into intelligent ve- human-factors groups to determine their needs for lane W hicles has expanded into applications that work with or detection and tracking, we developed a lane-tracking for the human user. Human-factors research is merging with system for objectives such as driver-intent inferencing intelligent-vehicle technology to create a new generation of [8] and behavioral analysis [9]. driver-assistance systems that go beyond automated control 2) The introduction of steerable filters for robust and ac- systems by attempting to work in harmony with a human op- curate lane-marking extraction. As will be described in erator. Lane-position determination is an important component Section IV, steerable filters provide an efficient method of these new applications. Systems that monitor the driver’s for detecting circular-reflector markings, solid-line mark- state [1], predict driver intent [2], [3], warn drivers of lane ings, and segmented-line markings under varying lighting departures [4], and/or assist in vehicle guidance [5], [6] are all and road conditions. They help to provide robustness to complex shadowing, lighting changes from overpasses and tunnels, and road-surface variations. Steerable filters Manuscript received December 28, 2004; revised April 27, 2005, July 26, 2005, and September 14, 2005. The Associate Editor for this paper was are efficient for lane-marking extraction because by com- N. Papanikolopoulos. puting only three separable convolutions, we can extract The authors are with the Computer Vision and Robotics Research Lab- a wide variety of lane markings. oratory, University of California, San Diego, CA 92093 USA (e-mail: [email protected]; [email protected]). 3) The incorporation of visual cues (lane markings and Digital Object Identifier 10.1109/TITS.2006.869595 lane texture) and vehicle-state information to help 1524-9050/$20.00 © 2006 IEEE McCALL AND TRIVEDI: VIDEO-BASED LANE ESTIMATION AND TRACKING FOR DRIVER ASSISTANCE 21 Fig. 1. Illustrations of systems that require lane position, and key performance metrics associated with the system objectives. (a) Lane-departure warning, (b) driver-attention monitoring, and (c) vehicle control. generate robust estimates of lane curvature, as described 2) Driver-Attention Monitoring Systems:Foradriver- in Section IV-C. By using the vehicle-state information attention monitoring system, it is important to monitor to detect an instantaneous road curvature, we can detect the driver’s attentiveness to the lane-keeping task. Mea- a curvature in situations where roadway lookahead is sures such as the smoothness of the lane following are limited. important for such monitoring tasks [1]. 4) The experiment design and evaluation of the VioLET sys- 3) Automated Vehicle-Control Systems: For a vehicle- tem. This experimentation was performed using multiple control system, it might be required that the lateral- quantitative metrics over a wide variety of test conditions position error at a specific lookahead distance, as shown on a large test path using a unique instrumented vehicle. in Fig. 1(c), be bounded so that the vehicle is not in danger We also present a justification for our choice of met- of colliding with any object [12]. rics based on our work with human-factors applications For each objective, it is important to examine the role that the as well as extensive ground-truth testing from differ- lane-position sensors and algorithms will take in the system and ent times of day, road conditions, weather, and driving design the system accordingly. Also, the evaluation of these scenarios. sensors and algorithms must be performed using the proper 5) The presentation of an up to date and comprehensive metrics. Components of lane-position sensors and algorithms analysis of the current state of the art in lane-detection that work well for certain objectives and situations might not research. We present a comparison of a wide variety necessarily work well in others. Examples of these situations of methods, pointing out the similarities and differences will be shown in Section III. between methods as well as for what objectives and envi- ronmental conditions various methods are most useful. B. Environmental Variability In addition to the system objective in which the lane-position II. LANE-POSITION DETECTION:OBJECTIVES, detection will be used, it is important to evaluate the type of ENVIRONMENTS, AND SENSORS environmental variations that are expected to be encountered. A. System Objectives Road markings and characteristics can vary greatly not only between regions, but also over nearby stretches of road. Roads In this paper, we will look at three main objectives of lane- can be marked by well-defined solid lines, segmented lines, position-detection algorithms, as illustrated in Fig. 1. These circular reflectors, physical barriers, or even nothing at all. three objectives and their distinguishing characteristics are the The road surface can be comprised of light pavement, dark following. pavement, or even combinations of different pavements. An ex- 1) Lane-Departure-Warning Systems: For a lane-departure- ample of the variety of road environments can be seen in Fig. 2. warning system, it is important to accurately predict the All of the images in the figure were taken from roads within a trajectory of the vehicle with respect to the lane boundary few miles of each other to show the environmental variability [10], [11]. even within small regions. Fig. 2(a) shows a relatively simple 22 IEEE TRANSACTIONS ON INTELLIGENT TRANSPORTATION SYSTEMS, VOL. 7, NO. 1, MARCH 2006 Fig. 2. Images depicting the variety of road markings and conditions for lane-position detection and tracking.