Terramax Vision at the Urban Challenge 2007
Total Page:16
File Type:pdf, Size:1020Kb
1 TerraMax Vision at the Urban Challenge 2007 Alberto Broggi, Andrea Cappalunga, Claudio Caraffi, Stefano Cattani, Stefano Ghidoni, Paolo Grisleri, Pier Paolo Porta, Matteo Posterli, Paolo Zani VisLab – Dipartimento di Ingegneria dell’Informazione Universita` degli Studi di Parma, ITALY www.vislab.it fbroggi,kappa,caraffi,cattani,ghidoni,grisleri,portap,posterli,[email protected] Abstract—This paper presents the TerraMax vision systems used during the 2007 DARPA Urban Challenge. First a descrip- tion of the different vision systems is provided, focusing on their hardware configuration, calibration method and tasks. Then each componentis described in details, focusing on the algorithms and sensor fusion opportunities: obstacle detection, road markings detection and vehicle detection. The conclusions summarize the lesson learned from the developing of the passive sensing suite and its succesful fielding in the Urban Challenge. I. INTRODUCTION The Challenges organized by DARPA are considered some of the most important milestones in the field of vehicular Fig. 1. The TerraMax vehicle. robotics. Not only did DARPA provide a vigorous help in the flourishing of new technological solutions, but the many researchers who accepted these challenges demonstrated that in order to decide on the action to be taken, it is not sufficient the current technology is indeed mature enough to drive to model the vehicle itself like it happens on the control side, vehicles in off-road and urban-like environments. but it is mandatory to model the real world as well; therefore, The design and realization of autonomous vehicles capable a huge number of situations must be taken into account. of handling unknown and dynamic situations involve different Different technologies have been tested and implemented aspects, spanning from new technologies to implementation, on the robotic vehicles that participated in the DARPA Urban from concept to hardware issues, challenging many different Challenge, ranging from conservative products up to com- research fields. pletely innovative solutions. Based on the teams’ capabilities, A completely autonomous vehicle needs to perceive the budget, and –most importantly– vision, the various vehicles environment, assess the situation and react to it, and finally have tackled the perception problem in different ways. control the actuators to implement a correct behavior. In other Team Oshkosh’s vehicle, TerraMax, was designed to operate words, the vehicle must have perception, decision, and control with conservative technology so that the integration of the final capabilities. The DARPA Grand Challenge –in particular the sensing suite on new vehicles and/or its retrofit to existing second one in 2005 when 5 vehicles were able to reach the vehicles could be straightforward. Furthermore the choice of end of the 130 miles course,– the DARPA Urban Challenge using passive technology (sensors not emitting any signal) is –which witnessed the completion of the course by 6 vehicles,– generally preferred in the military environment and is also and other public demonstrations –such as the PReVENT dictated by the will of producing vehicles able to operate in demonstration at the end of 2007 [1]– showed that the control close proximity, namely in situations in which active sensors side has already reached an advanced development stage. may interfere with each other. Some advanced control systems (such as ABS, EBD, ESP,...) The final solution was then to adopt cameras as the main have in fact already reached the market. sensing device and develop a 360◦ sensing suite based on Conversely, developing effective and reliable perception artificial vision. Eleven cameras have been mounted on Ter- and decision systems is definitely more challenging. The raMax to perceive all possible events around the vehicle that perception problem is very complex and even not well defined: may need to be considered by the decision system: presence a perception system must deal with a high number of different of obstacles, moving vehicles, lane markings, horizontal road situations and unknown scenarios. Objects in a scene have to signs. To improve system robustness, three laserscanners have be perceived and classified; a cluttered background –typical of also been installed –two in the front and one on the back– a urban environment and therefore of a common situation– can to complement vision perception. Besides providing their own lead to misdetections and introduce a high level of noise in the information to the decision system, these 3 LIDARs supply results. Similar considerations also apply to decision systems: raw data to the vision systems, which perform data fusion to 2 T2500 CPU, 2 GB of RAM and an Intel 945GM chipset. Two FireWire 800 boards, one PCI, and one PCI Express provide 4 connectors on the back. All the PCs are placed under the passenger seat inside the cabin, together with LIDAR, INS/GPS, and vehicle control units, as shown in Fig. 3. The Fig. 2. The Terramax vision systems (TerraMax facing right) and their placement: orange – trinocular system; purple – stereo systems; green – lateral system; cyan – rear view system. fine tune their final results. This paper is organized as follows: Section II presents the vision systems and provides details on the architectural and hardware solutions; Section III presents the Obstacle Detection functionality and shows how it is implemented using the different vision systems; Section IV and Section V describe the solutions adopted for Lane Detection and Vehicle Detection respectively; Section VI concludes the paper with a discussion on the lessons learned about the use of a vision system onboard an autonomous vehicle. II. TERRAMAX VISION SYSTEMS Fig. 3. Hardware arrangement inside the passenger seat for the various TerraMax perception is provided by 11 cameras grouped systems. into 4 different systems, each supplying information about the environment in a specific area around the vehicle. Fig. 2 shows imaging hardware for the trinocular, stereo and rear systems a sketch of the coverage of the vision systems. is based on Point Grey Research4 Flea2 industrial cameras, A trinocular vision system (orange in Fig. 2) is used to equipped with a 1/3 ”, 1024×768 pixels Sony CCD sensor, perceive information about the road in front, including lane while the lateral one features two full HD (1920×1080), markings, road borders, and possible obstacles/vehicles on the FireWire B, AVT5 Pike2 industrial cameras; the exposure driving path. Two stereo vision systems (purple in Fig. 2) parameters, like gain and shutter speed, can be adjusted monitor the close proximity of the vehicle’s front and back through the FireWire interface, while syncronization at 12.5 Hz in order to locate obstacles and road signs. The precise with the three IBEO6 AlascaXT LIDARs is provided by a localization of road signs (lane markings and stop lines) is dedicated SyncBox. Cameras are connected to the processing essential due to the vehicle’s abnormal size. Two monocular system using 7.5 m cables from AVT, specifically designed vision systems (green in Fig. 2) are used to surveil the sides for industrial use in noisy environments. Cables, cameras and of the vehicle when approaching intersections; normally off, boards are also equipped with locking screws to keep the these two high resolution cameras start to process images connectors in a fixed position, even in critical environments as soon as the vehicle stops at an intersection, and look for with strong vibrations; all cameras placed outdoor are enclosed oncoming traffic in any possible direction (not only orthogonal in sealed boxes, in order to preserve them from dust and rain. to the vehicle direction). Finally, two other monocular vision Polarizers have been used to minimize the reflections. systems (cyan in Fig. 2) keep track of overtaking traffic in adjacent lanes; their output is required when TerraMax needs The following sections present each single system in detail to change lane. together with the solution adopted for their calibration, consid- All the vision systems share a common hardware platform ering that calibration has to be performed on-the-fly, in many consisting of a rugged PC from SmallPC1, equipped with a different places, without any static calibration environment. mini-ITX board from Mini-Box.com2, an Intel3 Core 2 Duo 1http://www.smallpc.com 4http://www.ptgrey.com/ 2http://www.mini-box.com 5http://www.alliedvisiontec.com 3http://www.intel.com 6http://www.ibeo-as.com 3 Fig. 4. The trinocular portable calibration grid. A. Trinocular The trinocular system has been designed to provide robust Fig. 5. Stereo calibration is performed by means of a grid painted on a obstacle and lane detection capabilities in the long range (7– portable tarp. The picture on the top shows TerraMax while calibrating the 50 m, with the lower bound determined by the presence of stereo back, on the bottom is shown the acquired image and the result after the hood in the cameras field of view). Obstacle detection the pixel remapping using a lookup table. makes it possible to deal with certain kinds of situations that LIDARs can hardly (if at all) handle correctly, such as multiple extracting the correct values from the collected data is a non- vehicles queued at crossings, poles, and dust in unpaved areas trivial, time-consuming activity, which needs to be performed (which can be perceived as a solid obstacle by these sensors, each time the cameras are serviced, and as such must be while stereovision tends to filter it out since it corresponds scheduled properly, in order to avoid making the system to a textureless region originating no reliable matches); it also unavailable for some time provides additional information on the detected obstacles (such as height), and constitutes a reliable solution even in case of complete LIDAR sensors failures. Finally, it improves the B. Stereo abilities of the lane detection system by masking image areas The trinocular system described in section II-A can only corresponding to objects.