TEXTURE BASED ROAD SURFACE DETECTION
by
GUANGYU CHEN
Submitted in partial fulfillment of the requirements
For the degree of Master of Science
Thesis Advisor: Dr. Guo-Qiang Zhang
Department of Electrical Engineering and Computer Science
CASE WESTERN RESERVE UNIVERSITY
August, 2008 CASE WESTERN RESERVE UNIVERSITY
SCHOOL OF GRADUATE STUDIES
We hereby approve the thesis/dissertation of
______Guangyu Chen______
candidate for the __Master of Science______degree *.
(signed) Guo-Qiang Zhang______(chair of the committee)
H.Andy Podgurski ______
Francis Merat______
______
______
______
(date) 06/16/2008______
*We also certify that written approval has been obtained for any proprietary material contained therein. To my parents with Love and Gratitude Table of Contents
1. Introduction ...... 10
1.1Background...... 10
1.2DEXTER...... 13
1.3Motivation...... 15
1.4 Thesis Outline ...... 15
2. Image Sequence Time filter ...... 17 2.1 Image Sequence Time Filter ISTF ...... 17
2.2 ISTF effect Number One: connecting the dash lane marks ..... 18
2.3 ISTF effect Number Two: smooth the road images ...... 21
3. Color Texture Analysis ...... 27
3.1 Texture Analysis of Road Images ...... 27
3.2CovarianceMatrixofColorValueDifferential...... 28
3.3Thestrengthoftextureanisotropy...... 32
3.4TextureAnalyzingResultsandBenefits...... 33
3.4.1 ResultImages...... 33
3.4.2 Benefits of Color Texture Analysis Comparing to Other Ap-
proaches...... 33
4. Image Pixel Scale Problem ...... 40 4.1VisionPerceptualdistortion...... 40 4.2VisionPerceptualdistortioncorrection...... 40
5. Implementation ...... 49 5.1OrganizationoftheAlgorithm...... 49 5.2SegmentationAlgorithm...... 52
4 6. Results and Discussion ...... 57 6.1 Results of Road Segmentation Algorithm ...... 57
6.2FutureWork...... 61
7. Appendix A: Road image samples from NASA Plum Brook and Case Farm ...... 63
8. Appendix B: Documentation of Road Segmentation Software ... 66
9. Appendix C: C code of DLL file embedded in Road Segmentation
Software ...... 69
9.1TimeClosingFunction...... 69
9.2TextureAnalysis...... 72
9.3 Segmentation Algorithm–To Draw Horizontal White Lines and Output
Road Edge Points ...... 86
10. BIBLIOGRAPHY ...... 92
5 List of Figures
1.1 DEXTER with road detection sensors highlighted. blue = color cam- eras;green=LIDAR;red=IRcamera(22)...... 13
1.2DEXTER’sSoftwareArchitecture(22)...... 14
2.1 15 continuous road images from a front mounted vehicle camera . . . 19 2.2TimeDilationResults...... 20 2.3TimeErosionResults...... 22 2.4 Time Opening Results ...... 23 2.5TimeclosingResults...... 24
2.6 Original Input Image ...... 26
2.7 Noise Reduced Image by ISFT ...... 26
3.1 Road Images ...... 29
3.2TextureResults...... 34
3.3 Input Image (1) ...... 36
3.4 Color Texture Results of Input Image (1) ...... 36
3.5 Gray Texture Results of Input Image (1) ...... 36
3.6 Edge Detection Results of Input Image (1) ...... 37
3.7 Input Image (2) ...... 38 3.8 Color Texture Results of Input Image (2) ...... 38 3.9 Gray Texture Results of Input Image (2) ...... 38
3.10 Edge Detection Results of Input Image (2) ...... 39
4.1 Image of Road ...... 41 4.2 part of calibration matrix (meters) ...... 42 4.3 Horizontal (left) and Vertical(right) distance to center line (X=0) and
bottom line (Y=0) respectively ...... 42
6 4.4 Coordinate of an image ...... 43 4.5 relationship between pixel’s Real World Vertical Size and Vertical Po-
sition...... 44 4.6 relationship between pixel’s Real World Horizontal Size and Vertical Position...... 45
4.7 The distribution of pixel’s Real World Vertical Size to its Horizontal Position...... 45 4.8 The distribution of pixel’s Real World Horizontal Size and its Horizon-
talPosition...... 46
5.1 The front panel (a) and program blocks (b) of Road Segmentation System 50
5.2OrganizationchartoftheAlgorithm...... 51
5.3 Color Texture Image for segmentation (left) and Segmented Road Im-
age(Right)...... 52
6.1 Results of Road Segmentation ...... 58
6.2 Results of Road Segmentation ...... 59
7.1 Road Image Samples of NASA Plum Brook ...... 64
7.2 Road Image Samples of Case Farm ...... 65
8.1 User Interface Of Road Segmentation Software ...... 67
7 Acknowledgement
I would like to thank my supervisors, Dr. Guo-qiang Zhang and Dr. Frank Merat, for their support and review of this thesis. Their wide knowledge and rigorous way of thinking have been of great value to me. Their understanding and guidance have provide a good basis for this thesis. My appreciation as well to Scott McMichael for his help in understanding the problems and using Labview at the beginning of the project and his help in debugging the code. Special thanks to Fangping Huang who provide his experience on Latex.
8 Texture Based Road Surface Detection
Abstract by
GUANGYU CHEN
Using computer vision techniques to identify drivable areas for autonomous ve- hicles is a challenging problem due to unpredicted situations of the road, different road types, rough road surface and image noises. This thesis describes a system to recognize and segment road surfaces in real-world road images. This system uses a new-technique called ISTF–Image Sequence Time Filtering to reduce the noise in road images. To calculate the strength of anisotropy of each pixel, texture analysis based upon the covariance matrix of color changes in the image is used. A threshold segmentation algorithm is then applied to segment the road surface in the resulting texture image. This system was developed in LabView and C. It operates in real-time and has been extensively tested with real road image sequences. It has been deployed into DEXTER, the autonomous vehicle built by Team Case for the 2007 DARPA Ur- ban Challenge. Empirical comparative evaluations were also performed with similar approaches and the results of color texture analysis are significantly better than those using gray-scale texture analysis and edge detection methods.
9 1. INTRODUCTION
This thesis addresses vision-based road detection. The road scenes were obtained during development of DEXTER, an autonomous vehicle built by Team Case for the 2007 DARPA Urban Challenge(26).
1.1 Background
There are more than 200 million vehicles on the roads of North America and many people have been killed in car accidents because of the failure to follow a road or stay on a road. Driving a long distance is also a challenge for individuals. As a result driver assistance systems and even unmanned vehicles have been researched for many years to help drivers drive safely and comfortably. The first requirement of such a system is to detect other vehicles and obstacles. Many former driver assistance systems are gradually appearing in new vehicles. Recently, luxury cars have been equipped (23) with front and rear facing radars for detecting objects in front of and behind the vehicle. The new VolvoS80 has Blind Spot Detectors (24) which use cameras positioned in both side rear view mirrors. This system can detect vehicles in the blind spot humans encounter when driving and is crucial to safe lane changes.
The next step in driving system development is to track and follow other vehicles and the road. The ultimate goal is autonomous vehicles which can operate in all environments without drivers. Transportation is crucial to the development and defense of societies and nations.
This is the reason why many countries are developing unmanned systems such as unmanned fighter jets to replace piloted fighter jets, robots to replace human soldiers
10 and autonomous vehicles to transport materials. Carnegie Mellon University has been a leader in autonomous vehicle research.
Their most famous unmanned vehicle is ALV INN which is a vision-only based au- tonomous driving vehicle. ALVINN (25)used relatively primitive vision technology (30*32 pixels) images processed by a neural network to control vehicle steering. This system worked very well in spite of the low resolution image used. In November 2007, DARPA (Defense Advanced Research Projects Agency ) held a competition called the DARPA Urban Challenge to select the autonomous vehicle which could best navigate urban streets. Unlike the DARPA Grand Challenge which was set in the desert, this competition was held in an urban environment containing asphalt and concrete road surfaces with many other vehicles also using the same roads. The environment in which vehicles had to operate for the Urban Challenge was much more complex than that of the Grand Challenge. The vehicle was required to drive roads bordered by concrete walls or grass fields. The vehicle had to properly recognize stop signs and navigate intersections. The vehicle had to park in a parking lot and perform a U-Turn when the road was blocked. Although the competition was complicated, many teams competed in this event including Team Case. In this competition, Team case placed in the top 15 out of 55 teams.
Most teams used LIDAR to measure the distance to surrounding obstacles for the
Urban Challenge. But LIDAR has its limitations; the measured distance can not be larger than 30 meters when the vehicle is driving more than 40mph (27). A powerful LIDAR which can detect objects at larger distances costs many times the cost of a SICK LIDAR which costs only 3000 dollars. And the data from a powerful LIDAR needs a more powerful computer system to process it. That means that a LIDAR system to detect distant obstacles might cost more than 100 thousand dollars, but a good camera for image detection might only cost 500.00 dollars.
Because the cost of a camera is much cheaper than LIDAR, many methods for
11 driver assistance based upon vision have been developed in the past 20 years. Recent vision-based road vehicle detection system typically use cameras mounted on the front, rear or both sides of a vehicle. Processing these images is a challenging task because the camera has to operate in an environment where there may be dirt on the lens, water on the lens when it is raining and glare in the image when the lens faces the sun. There is no simple method to deal with all these situations. The outdoor environment is very complex and traffic situations are hard to predict. These facts make vehicle systems hard to design. And the biggest issue of an automatic vehicle system is that it must operate in real-time. The system must operate faster than the speed of the vehicle. For example, the camera might take 30 images per second when the vehicle is driving 20 meters per second. A slow-moving car 10 meters ahead requires a system that can detect the car ahead and stop the vehicle in less than a half second. This means the system must detect the car in less than one fifth of a second. This requires a system which can process at least 5 images per second, and for safety reasons, 10 images per second would be better.
Currently, the normal image size from a camera is 640 * 480 = 307200 pixels. This is too large for real-time processing using common computers. This is the reason why much research in this field focuses on object detection such as vehicle detection, road sign, traffic signal detection, etc. Because these objects occupy only a small region of a large image, the calculation time for image processing can be reduced considerably, allowing more complicated and precise algorithms to be used. An automatic driving system is very much related to intelligent systems. Overviews of relevant intelligent systems can be found in (1) (2) (3) (4) (5) (6) (7) and vision based systems in(8) (9). For vision based systems, three kinds of cameras are typically used: single lens cameras, infrared cameras and stereo cameras. Many vision-based autonomous vehicle systems have been developed based upon detecting objects on the road. Only a few systems have been developed which detect the drivable surface
12 Fig. 1.1: DEXTER with road detection sensors highlighted. blue = color cameras; green = LIDAR ; red = IR camera (22) of a road. (10) developed a system to reconstruct a 3-D shape of a road based upon
fine edge detection and statistical testing. But this system only finds the two side edges of a road and not the drivable surface of a road. And the computation time was
150ms using a HP workstation which is not fast enough for real-time applications.
(11) developed a system called 6D SLAM which uses a 3-D laser scanner to build a 3-D environment around the vehicle.Then it checks this environment for drivable areas. This system works well. But the 3-D laser scanner and onboard computers are far more expensive than a camera-based vision system.
1.2 DEXTER
DEXTER is an autonomous robot built by Team Case and ENSCO Inc. to compete in the DARPA 2007 Urban Challenges. DEXTER integrated cameras, LIDARS, radar, and GPS to monitor its environment and enable it to travel on urban roads. Figure 1.1 shows the DEXTER’s sensors, there are several cameras, several LIDARS, one IR camera , one 3-D camera and two GPS receiver devices on DEXTER. Figure 1.2 is the system architecture of DEXTER (26). It enables DEXTER to drive on
13 Fig. 1.2: DEXTER’s Software Architecture(22) the road without the human control. The road surface detection of this thesis is the
Road Detection Module of the Lane Observer which gives the representation of the current lane in global coordinates. The Road Detection Module reports 3 parameter polynomial fits of the detected lane edges. The task of road segmentation is to detect the road surface and report the road edges.
Most of DEXTER’s components were written in LabVIEW primarily because of the ease with which multi-threaded processes can be developed in LabVIEW. Al- though LabVIEW is not good for image processing, its ability to accept embedded
C programs makes it good for real-time control. LabVIEW can embed an compiled
DLL file. The image processing software is in a DLL file which was written in the C language. Input and output for the DLL file can be implemented through LabVIEW.
14 1.3 Motivation
The purpose of this thesis was to build a system to segment the road surface for
DEXTER. There were four single lens cameras on the front and one single lens camera on each side of the vehicle. The task of this system was to find the drivable surface using the images from both front and side cameras.
The goals of this system were:
• Front Road Segmentation—For the images from the front wide angle lens cam-
era, it was to segment the road surface in the front, to detect both side bound-
aries of the front road, and to measure the largest distance that the vehicle can
drive. And, if the road has painted lane marks, the system has to recognize the
lane marks as the boundary of the road.
• Side Road Segmentation—For the images from both side cameras, it was to
detect the boundary of the road and to measure the distance between the vehicle
and the road boundary.
• Detect Road Edges—To get enough points along the left and right road bound-
aries to construct the estimated road edge.
• Real Time—The most important requirement is that the system should be real time, which means that it should process 10 or more frames per second.
1.4 Thesis Outline
The contents of chapters are organized by the order in which they are used by the road segmentation algorithm. Chapter 2 describes a filter called the Image Sequence Time Filter (ISTF) which is used to connect the white painted dashed lane marks and enhance the signal to noise ratio of the image.
15 Chapter 3 presents the detailed texture analysis. Chapter 4 introduces an image pixel scale problem which is caused by perceptual distortion. Correcting this distortion improves the texture analysis results. Chapter 5 describes the organization of the current vision-based system -Road Segmentation Version 2
Chapter 6 presents results from the vision system for real roads and describes future work.
16 2. IMAGE SEQUENCE TIME FILTER
2.1 Image Sequence Time Filter ISTF
In this chapter, we present a filter called the Image Sequence Time Filter which can connect lane marks or erase disconnected lane marks, and can also smooth the road edges and road surface to reduce noise. This filter is used to pre-process road images for subsequent texture analysis.
The typical urban scene road surface consists of asphalt or concrete and painted lane marks which may or may not be connected. The task of road surface segmen- tation is to find the possible road edges to guide the vehicle. The texture analysis described in chapter 3 will consider a short disconnected lane mark as an object which does not belong to the road surface unless the lane mark belongs to the road edge.
But the road edge is required to be continuous which is why the Image Sequence
Time Filter(ISTF) must be applied before texture analysis. There are three sets of output points when the vehicle stops on a disconnected lane marked road. One set corresponds to the left road edge points, the second set corresponds to the dashed lane markers and the third set consists of left road edge points. The current driv- ing system can only use two sets of road edges points, so, if there are three sets of road edge points, it is hard to determine the correct road edges for driving. This is the reason why Image Sequence Time Filter(ISTF) must be applied before texture analysis. The idea for the Image Sequence Time Filter comes from 3D image morphology. Similar to a 2D image, each pixel has two spacial coordinates plus a third time
17 coordinate which provides the 3D image.Paper (20) gives a definition which we use for Time Morphological operations. By choosing a certain size structure along the time axis, one can define morphological operations for image sequences.
For each single frame {fi}:
Time Dilation operation (TD (fi)):
The value of pixel (x,y) in the new frame fi (x, y) after dilation is:
fi (x, y) = max (fj(x, y)) (2.1) (i−P Time Erosion operation (TE (fi)): The value of pixel (x,y) in the new frame fi (x, y) after erosion is: fi (x, y) = min (fj(x, y)) (2.2) (i−P where P is the length of the kernel along the time axis. We similarly define the Time opening (TO) and closing (TC) operations: TO(fi)=TD(TE(fi)) (2.3) TC(fi)=TE(TD(fi)) (2.4) 2.2 ISTF effect Number One: connecting the dash lane marks Figure[2.1] is a typical road image sequence consisting of 15 images acquired from one of DEXTER’s front mounted cameras. Figure [2.2] is the result of time dilation applied to each of the three color planes (Red, Green and Blue) of the 15 images road image sequence with a time kernel that is 7 frames in length. From the images, we see that the filtered road sequence is very similar to the original sequence except for the painted dashed lane markers. 18 Fig. 2.1: 15 continuous road images from a front mounted vehicle camera 19 Fig. 2.2: Time Dilation Results 20 The dashed lane markers look continuous in the images of the time dilated sequence. Figure 2.2 reveals a problem for road segmentation. The horizontal white painted line corresponding to a cross walk is not removed. For texture analysis, the horizontal line would be considered as an object in the front of the vehicle, but obviously it is not. Figure [2.3] is the result of applying time erosion on the three color planes (Red, Green and Blue) of a 15 images road image sequence with a time kernel that is 7 frames in length. From the filtered images, we see that the road images are very similar to the original images except that the painted dashed lane markers have disappeared. Figure [2.4] is the result of applying time opening on the three color planes (Red, Green and Blue) of a 15 images road image sequence with a time kernel that is 7 frames in length. From the images, we see the the road images are very similar to results of time erosion. Figure [2.5] is the result of applying time closing on the three color planes (Red, Green and Blue) of a 15 images road image sequence with a time kernel that is 7 frames in length. The road images are very similar to the original road scene. However, time closing can not only connect the dashed lane markers but can also remove the white painted horizontal line which is necessary for subsequent texture analysis. 2.3 ISTF effect Number Two: smooth the road images The Image Sequence Time filter can not only connect the dashed lane markers but also reduce the noise of an image. Normally, the road surface is not uniform. There are many spots and holes whose color is much different from the color of the asphalt or concrete which is the main material of the road surface. If the difference between the color of a spot and the color of the road surface is too large, the spot will affect subsequent texture analysis significantly. Just like a 2D morphological operation is used to reduce image noise, the ISTF can greatly increase the signal to noise of the 21 Fig. 2.3: Time Erosion Results 22 Fig. 2.4: Time Opening Results 23 Fig. 2.5: Time closing Results 24 image. The original input road image in Figure [2.6] shows a long black diagonal tire track line which is eliminated by ISTF in the process image Figure [2.7]. The grainy road surface is visibly smoother in the ISTF processed image figure [2.7] compared to the input in Figure [2.6] Using of the ISTF is required only when the vehicle is moving. If the vehicle is not moving, all images in the image sequence are the same and the ISTF will do nothing. The speed of the vehicle should be monitored to disable ISTF when the speed of the vehicle is zero to minimize computational requirements. 25 Original Image Fig. 2.6: Original Input Image Noise Reduced Image by ISTF Fig. 2.7: Noise Reduced Image by ISFT 26 3. COLOR TEXTURE ANALYSIS 3.1 Texture Analysis of Road Images Texture analysis is a very useful image processing technique and many approaches (13)(14)(15)(16) (17)have been developed to deal with road scene recognition and detection. However, texture analysis is computationally intensive and a challenge for real time image processing. That is the reason why most researchers (13)(14)(15) only use texture to detect a vehicle or obstacle which occupies only a very small region in a much larger road image but not to detect the road surface which is typically almost one fourth of a road image.There are fundamentally two kinds of texture analysis: statistical (19) and structural (18). Human psychophysicists have been studying human vision for years. Human eyes are very sensitive to high frequency areas and large orientation. For example, children always draw the contour first when they are drawing a picture because they can see the edges and the direction of edges very easily. This is one reason for the wide-spread use of the Fourier Transform which can transfer an image from the spatial domain to the frequency domain. The area where an edge appears has very high frequency components which are clearly shown in the Fourier transform of the original image. Several approaches have been developed in analyzing texture orientation.(21)and (16) give an approach to the analysis of oriented textures. They propose a measure of the strength of the texture anisotropy from the eigenvalues of the covariance matrix of gray value differences. This thesis demonstrates a new method which extends the texture analysis of 27 [J.Zhang](16) to handle color-images. 3.2 Covariance Matrix of Color Value Differential An input RGB image I from a camera has a Red plane (R), a Green Plane (G) and a Blue Plane (B). Each plane is a gray-scale image where each pixel is represented by a value from 0 to 255. We use a 3 by 3 Gaussian low pass filter to smooth the image thereby reducing the high frequency noise. The reason why we use the small filtersize (3 by 3) is to decrease the computation time. The image resolution has been reduced to 160 * 120 from 640 * 480 to reduce the processing time of each frame. And because the input image’s size has been reduced to 160*120, a 3 by 3 sized low-pass filter is adequate. We convolve each image planes (R, G, B) with a Gaussian Low pass filter Ga, e.g F = Ga ∗ I (3.1) In edge detection, the magnitude and orientation of the gray value gradients are very useful, but they represent magnitudes and orientation at a single pixel. Texture is described for a local region that consists of many pixels. The covariance matrix is introduced to represent the magnitudes and orientation of a texture region. The covariance of two different variables estimates their the statistical relationship. In an image, there are two dimensions x and y.Letdf x and df y be the first derivatives in the horizontal and vertical directions of a color image. However, df x or df y can not be calculated by simply subtracting two adjacent columns or rows in each color plane (R or G or B). Instead, df x and df y should be calculated using all the color information. Before presenting this further, we first consider several color road scenes. [Figure 3.1] illustrates three typical road images taken by the front cameras mounted on Dexter: Although the brightness varies substantially, we notice that all the road surfaces 28 Fig. 3.1: Road Images are basically gray in color which means that the Red, Green and Blue values are all the same or very close. But non-road image pixels are not gray which means that the pixels in that part of image have Red, Green, and Blue values which differ from each other. In order to highlight road edges and fields which do not belong to the road surface, we need to analyze the color of non-road fields. Because most of the non-road scene is not gray in color, these fields should have different red, green and blue values. Basically there will be three cases: • case 1: Red has the largest value among Red, Green and Blue values e.g: (R ≥ G>B)or(R ≥ B>G). • case 2: Green has the largest value among Red, Green and Blue values e.g (G ≥ R>B)or(G ≥ B>R). • case 3: Blue has the largest value among Red, Green and Blue values e.g (B ≥ R>G)or(B ≥ G>R). These three cases can be extended to another three cases. For example, if the Red value is equal to the Green value (R ≥ G>B), this case is same as case 1 or case 2. 29 Because the road surface is gray in color, the difference between R, G and B is equal or very close to zero. But the color difference for most non-road image components (not gray color) is very far from zero. Then we can easily separate the road surface. And we can not only calculate the color differential, but we can also calculate the spatial differential information. Now, we can define df x and df y: Consider a field with Red values which are significantly larger than the Green or Blue Values, For this field, df x and df y can be calculated by the following: R(x +1,y) − G(x, y) R(x, y +1)− G(x, y) df x(x, y)= df y(x, y)= (3.2) G(x, y) G(x, y) or R(x +1,y) − B(x, y) R(x, y +1)− B(x, y) df x(x, y)= df y(x, y)= (3.3) B(x, y) B(x, y) The second type of field has Green values which are significantly larger than the Red or Blue Values. For this field, df x and df y can be calculated by the following: G(x +1,y) − R(x, y) G(x, y +1)− R(x, y) df x(x, y)= df y(x, y)= (3.4) R(x, y) R(x, y) or G(x +1,y) − B(x, y) G(x, y +1)− B(x, y) df x(x, y)= df y(x, y)= (3.5) B(x, y) B(x, y) The third type of field has Blue values which are significantly larger than Red or Green Values. For this field, df x and df y can be calculated by the following: B(x +1,y) − R(x, y) B(x, y +1)− R(x, y) df x(x, y)= df y(x, y)= (3.6) R(x, y) R(x, y) or B(x +1,y) − G(x, y) B(x, y +1)− G(x, y) df x(x, y)= df y(x, y)= (3.7) G(x, y) G(x, y) Thecovarianceoftwovariance(df x and df x or df x and df y or df y and df y)is: n n σxx = ω(i, j) ∗ df x(x + i, y + i) ∗ df x(x + i, y + i) − μx ∗ μx (3.8) i=−n j=−n 30 n n σxy = ω(i, j) ∗ df x(x + i, y + i) ∗ df y(x + i, y + i) − μx ∗ μy (3.9) i=−n j=−n n n σyy = ω(i, j) ∗ df y(x + i, y + i) ∗ df y(x + i, y + i) − μy ∗ μy (3.10) i=−n j=−n where the local mean value of df x and df y are n n μx = ω(i, j) ∗ df x(x + i, y + j) μy = ω(i, j) ∗ df y(x + i, y + j) (3.11) i=−n i=−n ω(i, j) is a Gaussian matrix which is used for normalization and will be discussed in more detail in Chapter 4. 2 2 − 1 ∗[ i + j ] 1 2 σ2 σ2 ω(i, j)= ∗ e i j (3.12) 2πσiσj In order to keep the intensity value of the image, ω(i, j) has been used for nor- malization because the summation of matrix w(i,j) is 1. The size of matrix w(i,j) is 3by3. These variances can be used to define a color value gradient covariance matrix: σxx σxy =[σyx σyy ] (3.13) This covariance matrix is symmetric because σyx = σxy. It can be transformed to the matrix: λ1 0 =[0 λ2 ] (3.14) where the λ1 and λ2 can be calculated as follows: σxx σxy Let A =[σyx σyy ] and define: λ1 0 λ =[0 λ2 ] (3.15) A ∗ X = λ ∗ I ∗ X where X is a random 2*2 matrix (A − λ ∗ I) ∗ X = 0 (3.16) Then the determinant of (A − λ ∗ I)m: det(A − λ ∗ I)=0 2 det(A − λ ∗ I)=(σxx − λ) ∗ (σyy − λ) − σxy = 0 (3.17) 31 where 1 2 2 λ1,2 = ∗ (σxx + σyy ± (σxx − σyy) +4∗ σ ) (3.18) 2 xy From our experience, there are three kinds of image covariance matrices which can be summarized as follows: In a region which has oriented texture, there is one larger eigenvalue and one smaller eigenvalue. In a region which is homogeneous and has no oriented texture, both eigenvalues are very small( close to zero). In a region which has two orientations such as a corner, there are two large eigenvalues. The first case represents an oriented texture field. But the other two cases have weak anisotropy because they have no strong single direction. 3.3 The strength of texture anisotropy The task of image segmentation for road surface detection is to highlight areas such as road edges, side walks, buildings and cars on the road which do not belong to the road surface. This task can be done by calculating the texture anisotropy which is very important for characterizing oriented textures such as encountered in road images. The oriented texture measures significant orientation which can be described by the two situations described in section 3.2: the covariance matrix has two very different eigenvalues or two large eigenvalues. For the first case, the larger eigenvalue may be very small, but still significantly larger than the smaller one. (16) defines an equation to calculate the strength of the texture anisotropy which can not only find areas which have two large eigenvalues or one larger and one smaller eigenvalues, but also can highlight areas which have two significantly different eigen- values even when the larger one is very small. The strength of the texture anisotropy may be defined as: 32 2 2 1 λ2 1 2 1 λ1 − λ2 s = ∗ (1 − )trace(Σ) = ∗ λ1 ∗ e = ∗ (3.19) c λ1 c c λ1 with 2 2 λ1 − λ2 e = (3.20) λ1 where λ1 is larger than λ2 and c is a normalization constant. The final equation (3.3.19) is simple but its evaluation requires calculating (e) and (trace(Σ)). e is the eccentricity of an ellipse whose major axis is λ1 and whose minor axis is λ2, the smaller eigenvalue. The paper (16) introduced trace(Σ) which is equal to λ1 +λ2 to represent a strong anisotropy area which is distinguished from the homogeneous areas. The value of s ranges from 0 which is the weakest texture anisotropy to 1 which is the largest texture anisotropy. 3.4 Texture Analyzing Results and Benefits 3.4.1 Result Images In this section, some results of texture analysis are shown. The result of texture analysis is a pseudo black and white image with the value of pixels in the range of 0 to 1 corresponding to the strength of the texture anisotropy defined above. A threshold is set to determine whether a pixel belongs to the drivable surface or not. In practice, a controller adjusts the threshold over the range is 0.01-1. From experience, the threshold can set to 0.5 because the result is not strongly dependent upon the threshold value. This threshold is used in the segmentation algorithm which is described in chapter 5 Implementation. 3.4.2 Benefits of Color Texture Analysis Comparing to Other Approaches This section shows that color texture analysis has significantly better result than gray texture analysis and edge detection methods. 33 Fig. 3.2: Texture Results Figure [3.3] and [3.7] are similar road images taken by a camera mounted in DEX- TER at the case farm. The left image of figure [3.4] is the texture anisotropy image of road image Figure [3.3] generated by color texture analysis algorithm described in section 3.2 and 3.3. The left image of figure [3.5] is the texture anisotropy image of road image 3.3 generated using only gray scale texture analysis (16). When the gray scale texture analysis is applied to a color image, the color image is converted to a gray scale image whose pixel value is the average of the corresponding R, G and B values. The first derivatives of gray texture analysis are the differences between adjacent columns and rows. The road surface in the texture image (left of figure 3.4 ) generated by color texture analysis is much smoother than that in the texture image (left of figure 3.5) generated by gray scale texture analysis. Because there is a water flooded area on the road surface and gray texture analysis only analyzes the spatial texture information, it will high light the area which has large spatial variances ig- noring the water flooded area. However, the water flooded area is gray in color and the road surface is also gray in color. The color texture analysis which uses both spatial and color information will reduce the difference between the water flood area and the road surface in the texture anisotropy image. That is the reason why the 34 water flooded area appears darker in the color texture result image (left in figure 3.3) than that shown in the gray texture result image (left in figure 3.4). The texture result image’s ratio of signal to noise is very important to the subsequential segmen- tation algorithm. The right image of figure 3.4 is the road segmentation result using the color texture result image (left). The horizontal white lines in the right image of 3.4 or 3.5 show the segmented road surface. Obviously, the segmentation results (right in figure 3.4) using color texture analysis are much better than the segmen- tation results (right in figure 3.5) using gray scale texture analysis. The horizontal white lines in left image of figure 3.4 cover all the road surface. But the horizontal white lines in left image of figure 3.5 only covers 50% of the road surface. Another example is illustrated in figures 3.8 and 3.9. Figures 3.8 and 3.9 display the texture and segmentation results from road image 3.7 using color texture analysis and gray scale texture analysis respectively. The road surface in the left image of figure 3.8 is much smoother than the left image of figure 3.9. The segmentation results (right in figure 3.8) using color texture results is much better than that (right in figure 3.9) using the gray scale texture analysis. Color texture analysis also has significantly better results than conventional edge detection methods. Figure 3.6 shows the edges computed in each color plane (Red, Green, Blue) of Figure 3.3 using Sobel and Canny edge detection methods. Figure 3.10 shows the edges in each color plane (Red, Green, Blue) of Figure 3.10 by the Sobel and Canny edge detection methods. These black and white edge images can not show the road edge clearly. The Sobel edge detection method can only find a small part of the road edges. The Canny edge detection method shows much noise on the road surface. These results can not be accurately segmented by the segmentation algorithm of chapter 5. In summary, color texture analysis is the best approach for road surface detection. 35 Fig. 3.3: Input Image (1) Fig. 3.4: Color Texture Results of Input Image (1) Fig. 3.5: Gray Texture Results of Input Image (1) 36 Sobel on Red Canny on Red Sobel on Green Canny on Green Sobel on Blue Canny on Blue Fig. 3.6: Edge Detection Results of Input Image (1) 37 Fig. 3.7: Input Image (2) Fig. 3.8: Color Texture Results of Input Image (2) Fig. 3.9: Gray Texture Results of Input Image (2) 38 Sobel on Red Canny on Red Sobel on Green Canny on Green Sobel on Blue Canny on Blue Fig. 3.10: Edge Detection Results of Input Image (2) 39 4. IMAGE PIXEL SCALE PROBLEM 4.1 Vision Perceptual distortion Perceptual distortion is very common. You can see that the further an object is from your eyes, the smaller it appears. This is the same perspective distortion seen in camera images. Dexter’s road images are acquired by cameras with wide-angle lenses which accen- tuate distortion. In the real world, the width of the road is constant, but in images of the road, a straight road surface appearing as a triangle with the width of the road appearing smaller with distance. This is a perceptual distortion problem. The point where the width of the straight road goes to zero is called the vanishing point. From this, it is clean that the pixels in different positions of an image have different real world physical sizes. This is why the width of the road appears wider at the bottom of the image than in the middle of the image. Figure 4.1 shows that the width of the road surface is not constant in the image because of the effect of perspective distortion. 4.2 Vision Perceptual distortion correction Because of the different physical areas corresponding to the pixels in an image, the low pass filtering and ω(i, j) described in Chapter 3 for texture analysis has to be dynamically changed according to pixel location. The actual real world physical size represented by each pixel can be obtained from the camera calibration. Figure [4.2] is a snapshot of the calibration matrices. Assume the location of the camera mounted on the vehicle is the origin of an X-Y coordinate. 40 Fig. 4.1: Image of Road Each entry of the upper matrix represents the real world horizontal distance (meters) between the corresponding pixel and the center line X=0. And each entry of the lower matrix in figure 4.2 represents the real world vertical distance (meters) between the corresponding pixel and the bottom line Y=0. To conveniently visualize the calibration matrix, we can generate pseudo color images as shown in figure 4.3. The left image shows the real world horizontal distance from each pixel to the center line X=0; the right image shows the real world vertical distance from each pixel to the bottom line Y=0. The different colors represents the different real world distances. The value of Green color is less than that of Red color. We define the coordinate of an image as shown in figure 4.4. The y-axis is straight down the left side of the image. From the two calibration matrices, we can determine four relationships bwteen image pixels and their real-world values: • The relationship between each pixel’s real-world vertical size and its vertical position in the road image. Obviously, it is nonlinear. Figure [4.5] shows the relationship between the real world vertical size of each pixel and its y-position (vertical position) in a road image. The dark line in figure 4.5 is a very close fitting function which can represent this relationship 41 Horizontal Distance from a pixel to Center Line X=0 Vertical Distance from a pixel to Bottom Line Y=0 Fig. 4.2: part of calibration matrix (meters) 50 50 100 100 150 150 200 200 250 250 100 200 300 400 500 600 100 200 300 400 500 600 0 2 4 6 8 10 12 14 16 18 20 22 24 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 meters meters Fig. 4.3: Horizontal (left) and Vertical(right) distance to center line (X=0) and bottom line (Y=0) respectively 42 Fig. 4.4: Coordinate of an image mathematically. The function of the dark line is : A YFit1 = + D; (4.1) (B ∗ X − C)E where A =1.8; B =0.1; C = 222 ∗ B − 1; D = −0.1; E =0.95; These values correspond to actual experimental data from the calibration matrices. • The relationship between each pixel’s real-world horizontal size and its vertical position in the road image. Figure [4.6] shows the relationship between the real world horizontal size of each pixel and its y-position (vertical position) in a road image. The dark line in figure 4.6 is a very close fitting function which can represent this relationship mathematically. The function of the dark line is : A YFit2 = + D; (4.2) (B ∗ X − C)E where A =0.1; B =0.1; C = 222 ∗ B − 1; D =0;E =0.95; These values correspond to actual experimental data from the calibration matrices. 43 Fig. 4.5: relationship between pixel’s Real World Vertical Size and Vertical Position We can observe that (4.2.1) and (4.2.2) are the same except for the values of A and D. • The relationship between each pixel’s real-world vertical size and its horizontal position (x-position) in the road image. Figure [4.7] shows the distribution of the real world vertical size of each pixel to its x-position (horizontal position) in a road image. There are many lines in Figure 4.7. Each line represents the distribution of the real world horizontal size of the pixel in a single line of an road image and its x-position. Each line can be fit to a function. There are 260 lines in the vertical and horizontal calibration matrices. So there are 260 lines in Figure 4.7. It is difficult to find a function to fit this distribution. • The relationship between each pixel’s real-world horizontal size and its horizon- tal position (x-position) in the road image. Figure [4.8] shows the distribution of the real world horizontal size of each pixel to its x-position (horizontal position) in a road image. It is also difficult to build 44 Fig. 4.6: relationship between pixel’s Real World Horizontal Size and Vertical Position Fig. 4.7: The distribution of pixel’s Real World Vertical Size to its Horizontal Position 45 Fig. 4.8: The distribution of pixel’s Real World Horizontal Size and its Horizontal Position a function to fit this distribution. Actually, there are two more relationships: one is between the pixel’s vertical real-world size and its position (both X and Y) in an image; the second is between the pixel’s horizontal real-world size and its position (both X and Y) in an image. Each would be a 3-D plot associated with three axes: X axis is the pixel’s horizontal position in the image, Y axis is the pixel’s vertical position in the image, Z axis is either its vertical or horizontal real-world size. It is possible to find these two functions, but that is future work. The distribution of either real-world vertical size or horizontal size of each pixel to its vertical image position (y-position) can be fit to the same function with different values. Although there is no easy way to fit the relationship between either real-world vertical size or horizontal size of each pixel and its horizontal position (x-position), equations (4.2.1) and (4.2.2) are still useful for distortion correction. A Gaussian low pass filter is used before texture analysis. And another Gaussian matrix (3.2.12) is used in texture analysis. The shape of these Gaussian matrices is determined by two factors σx and σy. Because of the different real-world size of 46 each pixel, the shape of the Gaussian low pass filter and the Gaussian matrix used in texture analysis should be adjusted by the size of the pixel. For example, let pixel P1’s real-world size be 5 cm horizontally and 0.5 m vertically and pixel P2’s real-world size be 10 cm horizontally and 1 m vertically. Then the σx and σy of the Gaussian low pass filter for P2 should be larger than that of the Gaussian low pass filter for P1. Similarly, the Gaussian matrix ω(i, j) used in texture analysis also should be different for P1 and P2. We write the Gaussian filter as: 2 2 − 1 ∗[ x + y ] 1 2 2 2 Ga(x, y)= ∗ e σx σy (4.3) 2πσxσy x,y x,y and adjust (σX ,σY ) of the Gaussian matrix as follows: x,y x,y 0,0 0,0 First, set the default σX and σY for a user defined initial position as σX ,σY . x,y This can be adjusted in the front panel of the software as an input. New σX and x,y σY are then calculated based on the position (x, y) of each pixel and using (4.2.1) and (4.2.2) as: A x,y 0,0 (B∗N−C)E + D σX = σX ∗ A (4.4) (B∗(N−y)−C)E + D where A =1.8; B =0.1; C = 222 ∗ B − 1; D = −0.1; E =0.95; and A x,y 0,0 (B∗N−C)E + D σY = σY ∗ A (4.5) (B∗(N−y)−C)E + D where A =0.1; B =0.1; C = 222 ∗ B − 1; D =0;E =0.95; N is the number of horizontal lines in an image. But because there are only 260 lines in the calibration matrices rather than 480 lines as in the in original road images, we do not calculate σx and σy for those pixels which are not calibrated. The values of the calibration matrices are measured manually. We put many spots on the ground in the front of camera, and the coordinates of those spots are pre-determined. Then the camera takes an image of those spots. Those pixels which show the spots are 47 calibrated and the other pixels are calibrated by interpolation. 48 5. IMPLEMENTATION The algorithms described in this thesis were implemented in C (all C code is in Appendix C) using LABVIEW from National Instruments. LabView has a good user interface for powerful graphical programming and is built on a real-time operating system for applications such as DEXTER. Road Segmentation.vi is the final program which was integrated into the DEXTER’s software. Figure 5.1 shows the front panel (a) and program blocks (b) of the Road Segmen- tation system. From the front panel in 5.1(a)—the document of the front panel is in Appendix B,it is easy to adjust all image processing parameters such as the thresh- old for the segmentation algorithm, the number of images used for ISTF, the image source (from camera or image files), whether there is brown grass or green grass on the side of road, etc. The user can also view the input image, texture image and segmented road image, processing time of each frame, number of output road edge points along the road edge, the current fit to the road edge, the accuracy (confidence value) of the curve etc. Figure 5.1 (b) shows the actual program blocks and data flow used to generate Figure 5.1 (a). 5.1 Organization of the Algorithm Figure 5.2 shows the basic organization of the Road Segmentation algorithm. The user chooses the number of images to be processed between 10 and 20 with 15 images being typically used. When the vehicle is moving, the algorithm enables ISTF with the appropriate kernel, 7 for example. Although there are many output images, only the most recent image (F) is output. F is then low-pass filtered by 49 a b Fig. 5.1: The front panel (a) and program blocks (b) of Road Segmentation System 50 Fig. 5.2: Organization chart of the Algorithm 51 a b Fig. 5.3: Color Texture Image for segmentation (left) and Segmented Road Image (Right) an appropriately pixel-scaled low pass Gaussian Filter. Texture analysis is used to generate the pseudo black and white image (F’) for segmentation. The road surface shows as black and the non-road surface area shows as white in F’. Once we have the texture image F’, a threshold algorithm is applied to identify the road surface in the pseudo black and white image F’. This threshold algorithm will be described in section 5.2. The horizontal white lines in the upper right image of Figure 5.1 (a) show the segmented road surface. The end points of these horizontal lines on the road surface define the edges of the road and are output for use by the driving algorithm. 5.2 Segmentation Algorithm This algorithm is used to segment the road surface area in the texture image. Figure 5.3 (a) is a texture image where the road surface shows as black. Figure 5.3 (b) is the segmented road surface image. The horizontal white lines are drawn by the segmentation algorithm described in this section. The segmentation algorithm draws horizontal white lines on the road surface area 52 every 3 lines of the road image. The initial search position is calculated by an algo- rithm described by the following pseudo code: Find the initial search position for the subsequent segmentation algorithm Input: Threshold:T Initial position of road bottom: Y_bottom. Y_bottom is an input of Segmentation system It can be adjusted by the user. Y_bottom is the vertical position of initial searching coordinate. Texture Image: F We assume that the width of road surface is larger than a certain number N_width of pixels immediately in front of the vehicle Output:X_init,Y_init; X_init=1; Y_init=Y_bottom; count=1; int Find=0; while(Find==0) { searching the line Y_init in F if (there is a (N_width) length continuous pixels sequence whose center is (X, Y_init) and the values of all those pixels in this sequence is less than T} 53 {Find =1; X_init=X} else { if (Y_init > 60) then {Find = 0; break;} else {Y_init--}; } } if (Find==0) { No road in the front of vehicle X_init=80;% No road surface Y_init=160;% set initial searching position to %the bottom center of image } output X_init and Y_init to subsequent segmentation algorithm This algorithm determines if there is a road in the front of vehicle by searching for a region large enough to be the road surface in the texture result image. The width of the road surface should be larger than a number N−width of continuous pixels where N−width can be adjusted by the user. The above code searches an N−width length sequence of contiguous pixels whose values are all less than threshold T from bottom to top and left to right in texture result image F. It expands the current white line to both right and left until the white line hits the right and left road edges respectively. The non-road pixels’ values are larger than the threshold (T); the road surface’s pixels values are smaller than the threshold (T). The coordinates of the two end points’ coordinates are put into the output road endpoints array. Next draw the next horizontal white line whose center point is 2 pixels above the center point of the current horizontal white line. The segmentation algorithm 54 keeps incrementing the horizontal white lines vertically until the value of the pixel which is 2 pixels above the center pixel of the current horizontal white line is larger than the threshold. The pseudo code for the threshold segmentation algorithm is given as follows: Input: Threshold:T Initial Search Position: X_init,Y_init Texture Image: F Output: array: LeftendPoints(n) array: RightendPoints(n) X=X_init; Y_left=Y_init; Y_right=Y_init; center=(Y_left+Y_right)/2; n=0; while (F(X,center)<=T) { Search for the left first pixel F(X,Y_left’) in F whose value is larger than T; Search for the right first pixel F(X,Y_right’) in F whose value is larger than T; Put (X,Y_left’) into LeftendPoints(n) array; Put (X,Y_right’) into RightendPoints(n) array; n++; X=X-2; Update Y_left = Y_left’; Update Y_right = Y_right’; 55 Update center=(Y_left+Y_right)/2; } 56 6. RESULTS AND DISCUSSION 6.1 Results of Road Segmentation Algorithm The success of the road segmentation system was demonstrated by successfully being used to drive Dexter on roads at the Case farm and NASA Plum Brook station. Typ- ical road images of the Case farm and NASA Plum Brook are presented in Appendix A. The average processing time of each frame was less than 100ms when run on a Pentium D 2.66Ghz processor with 2 GB memory. This processing time satisfied the real-time requirement. Figure 6.1 and 6.2 shows the results of the road segmentation algorithm for dif- ferent road images acquired by DEXTER on real roads. The left column of figure 6.1 and 6.2 shows six representive input images. The actual input to the algorithm is a sequence of 10 to 20 images, but, for convenience, only the latest image from the camera is shown. The second column of Figure 6.1 shows the segmented drivable surface as represented by horizontal white lines. The black points at the left and right end of the horizontal white lines are the output points to the driving software. The outputs are (x,y) coordinate pairs and are used for road curve reconstruction. The red lines shown in Figure 6.2 (D), (E) and (F) are the modeled road edges which were used to control the steering wheel of the vehicle. The first picture (A) is a typical road scene. The dashed lane markers are con- nected by the ISTF filter in order to correctly segment the lane in which the vehicle is driving. The second picture (B) shows a tree-lined road with tree shadows on the road 57 (A) (A−1) (A−2) (B) (B−1) (B−2) (C) (C−1) (C−2) Fig. 6.1: Results of Road Segmentation 58 (D) (D−1) (D−2) (E) (E−1) (E−2) (F) (F−1) (F−2) Fig. 6.2: Results of Road Segmentation 59 surface. Such shadows are a significant problem for the vision system which considers the different light intensities to correspond to differently textured regions. As a result there are fewer horizontal white lines shown in the segmented image (B-1). Pictures (C), (D) and (E) are from similar urban road scenes. The road surface in (E) has no painted lane markers, but (C) and (D) do. (C) was taken when the vehicle was stopped, but (D) was taken when the vehicle was driving along the road. From these pictures, we can see that the system correctly segments the lane in which the vehicle is driving. Figure (D-1) shows that the segmentation is very good, but in figure (C-1) many white lines go through the gaps between unconnected lane marks to include the opposite lane because the vehicle is stopped and the ISTF is turned off. There are three sets of output points when the vehicle is in a lane marked road. One set corresponds to the right road edge points, the second set corresponds to the painted lane markers, and the third set consists of the left road edge points. The current driving system can only use two sets of road edges points, so if there are three sets of road edge points, it is hard to determine the correct road edges for driving. A confidence value is used to measure the accuracy of each road edge. The confidence value is measured by the sum of square of fitting errors. If the sum of square of fitting errors is 0, the confidence value is 1. If the sum of square of fitting errors is significantly large (close to infinite), the confidence value is 0. In figure [C-1] the confidence value of the left edge would be very low because of the large variance in road edge position, but the confidence value of the right edge would be high because most of edge points are along the same straight line allowing the vehicle control system to more highly trust the right modeled edge. Figure 6.2 (E) shows a road with no painted lane markers on it where the seg- mentation algorithm should segment the entire road surface. The output in this case consists of two sets of road edge points: one from the left road boundary and the 60 other from the right road boundary. The result (E-1) is very good as seen by the calculated road edges which show as the two red curves in (E) along the actual road edges. The road in figure (F) has a rough asphalt surface and no clear road edges which are covered by many brown leaves. On both sides of the road, there are large trees which cause shadows on the road surface. The ISTF and texture segmentation deal with this situation very well. Figure (F-1) shows the segmentation results with the horizontal white lines correctly identifying the whole drivable surface. The red lines shown along the road edge in figure (F) represent the road edge calculated from the segmentation results and are quite good. (F-2) shows the texture image corresponding to input image (F). There is a small amount of white noise in the black road surface area, but this is minor compared to the white areas which do not belong to the road surface. It is not large enough to effect the segmentation results. As shown in Figure 6.1 and 6.2, the system provides stable output road edge points in most urban drving situations except for handling multiple lanes when the vehicle stops or is moving very slowly as described below. 6.2 Future Work Although this system can correctly identify most urban road surfaces, there are some situations which it can not currently handle. For example, when the vehicle stops on a road with multiple lanes, the system will output more than two sets of road edge points, but the system can not distinguish which sets correctly correspond to the driving lane. Segmenting multiple-lane road surfaces is a challenging problem. This problem might be treated as a classification problem. The coordinates of the road edge points can be classified into several different sets. There are many algorithms for potentially solving this problem such as artificial neural networks, decision trees and linear classifiers. Because of the real time processing requirement, artificial neural 61 net works would probably be the first choice because of the small number of inputs (less than 30 pairs of points) and outputs (3 outputs:single lane, two lanes and three lanes). There is a control in the front panel Figure 5.1 (a) which can be used to optimize the system for different environments such as green grass on the road sides, brown grass on the road sides, city road with many tall buildings, or dirty roads. These road environments need different segmentation algorithms for optimum performance. A self-adaptive vision system which automatically chooses the best segmentation algorithm for the current driving environment would improve the performance and reliability of the system. In the future, neural networks and a self-adaptive algorithm to optimize the system for different situations would be the most important improvements 62 7. APPENDIX A: ROAD IMAGE SAMPLES FROM NASA PLUM BROOK AND CASE FARM 63 Fig. 7.1: Road Image Samples of NASA Plum Brook 64 Fig. 7.2: Road Image Samples of Case Farm 65 8. APPENDIX B: DOCUMENTATION OF ROAD SEGMENTATION SOFTWARE Below is the description of items in the interface of the Road Segmentation Software: Input image: The current input image. Output: The image which is drawn white lines on road surface. Texture: The texture image of the input image. Error out: identify the error when error occurs. Camera configures: • Camera Type: the identification of camera. • Resolution: size (640*480) of the input image . • Frame rate: the number of frames that the camera records in one second. • Color space: shows the color space of the input image. File path: choose the input images’ file location. Calibration name: different cameras have different calibration matrices. Input the name of the appropriate calibrate matrix before running software. Acquisition Mode: • Read from camera: the input images are from camera. • read from jpeg sequence: the input images are stored in a fold and ordered by their names. 66 Fig. 8.1: User Interface Of Road Segmentation Software 67 • read from timestamped jpegs: the input images are stored in a fold and ordered by their names. The name of each image is its a string of time value. Road Scene: • Brown Grass: when most of the non-road areas are covered by green grass. • Brown Grass: when most of the non-road areas are covered by brown grass. • Lane Markers: when the road surface has lane markers. # of images: the value of time kernel for ISTF filtering. Morphological Closing: When the button is green, perform spatial closing operation for each image frame. Road Shape: When the value is straight, use linear function for road edge fitting. When the value is curve, use polynomial fitting to fit road edges. Search Length: the width of initial road surface searching. Start Line Position: the road searching initial position in the road texture image. Threshold: adjust the value of threshold for the function of drawing horizontal lines. Is CrossWalk: Green indicates there is cross walk ahead. Two green pillars above the input image: shows the confidence value of each calculated road edge curve. 68 9. APPENDIX C: C CODE OF DLL FILE EMBEDDED IN ROAD SEGMENTATION SOFTWARE 9.1 Time Closing Function double TimeOpening(double a[],double N_Of_Images,short int Time_Dilation_Number);// Time Opening Operation extern "C" _declspec(dllexport) void TextureDll(double *Number_Of_Images, double *input, double *output, double *output2, short int *Length, double *Rectangle, double *Rect_Mean, double *Rect_Max, double *Rect_Min, double *Pre_Var, double *Pre_Rect, double *outputLeftX, double *outputLeftY, double *outputRightX, double *outputRightY, short int *NP_of_left, short int *NP_of_right, short int stand_Y, double stand_sigmax, double stand_sigmay, double stand_sigma, short int Start_Line_Position, short int Time_Dilation_Number, long *Iterations, unsigned short int Road_Scene, double *Threshold, double Search_Length, short int *Cross_Walk_Indicator) { ...... ...... ...... ///////////////////////////////////////////////////// // ISTF filtering ///////////////////////////////////////////////////// 69 for (i=0;i<57600;i++) { for (j=0;j Image_temp[j]=input[i+57600*j]; } int count=int(N_Of_Images)-Time_Dilation_Number+1; Image1[i]=TimeClosing(Image_temp,N_Of_Images,Time_Dilation_Number); } ...... ...... ...... } double TimeClosing(double a[],double N_Of_Images,short int Time_Dilation_Number) { int i=0; int j=0; double Erosion_Temp=255; int count=int(N_Of_Images/Time_Dilation_Number); for (i=0;i double Dilation_Temp=0; 70 for (j=0;j if(a[i*Time_Dilation_Number+j]>Dilation_Temp) { Dilation_Temp=a[i*Time_Dilation_Number+j]; } } if(Dilation_Temp } } double Closing_Temp=0; for (i=0;i { for (j=0;j { if(a[i*Time_Dilation_Number+j] { Closing_Temp=a[i*Time_Dilation_Number+j]; } } if(Erosing_Temp>Closing_Temp) { Closing_Temp=Erosing_Temp; } } 71 return Closing_Temp; } 9.2 Texture Analysis void RGB_Texture_Lanemark(double I[][160], double IT[][160],int stand_Y,double stand_sigma) { double Fx[120][160];//first derivative in x double Fy[120][160];//first derivative in y //double S[120][160]; double W[3][3]={{0.0113,0.0838 , 0.0113}, {0.0838 ,0.6193 , 0.0838},{ 0.0113 , 0.0838 , 0.0113}}; int i=0; int j=0; for (i=19;i<120;i++) { for (j=0;j<160;j++) { if (j==159) { Fx[i][j]=fabs(I[i][j]-I[i][j-1])/I[i][j-1]; //IT[i][j]=I[i][j]-I[i][j-1]; } else { Fx[i][j]=fabs(I[i][j+1]-I[i][j])/I[i][j]; 72 //IT[i][j]=I[i][j]-I[i][j-1]; } if (i==119) { Fy[i][j]=fabs(I[i][j]-I[i-1][j])/I[i-1][j]; } else { Fy[i][j]=fabs(I[i+1][j]-I[i][j])/I[i][j]; } } } double Mux=0; double Muy=0; double Sigmaxx=0; double Sigmaxy=0; double Sigmayx=0; double Sigmayy=0; double L; double Lenda1; double Lenda2; double E; for (i=19;i<120;i++) { //if(i>56) 73 //{ Recal_W(W,i,stand_sigma,stand_sigma,stand_Y); //} for (j=0;j<160;j++) { Mux=0; Muy=0; Sigmaxx=0; Sigmaxy=0; Sigmayx=0; Sigmayy=0; Lenda1=0; Lenda2=0; E=0; Mux=W[0][0]*Fx[i-1][j-1]+W[0][1]*Fx[i-1][j]+ W[0][2]*Fx[i-1][j+1]+ W[1][0]*Fx[i][j-1]+W[1][1]*Fx[i][j]+W[1][2]* Fx[i][j+1]+W[2][0]* Fx[i+1][j-1]+W[2][1]*Fx[i+1][j]+W[2][2]*Fx[i+1][j+1]; Muy=W[0][0]*Fy[i-1][j-1]+W[0][1]*Fy[i-1][j]+ W[0][2]*Fy[i-1][j+1]+ W[1][0]*Fy[i][j-1]+W[1][1]*Fy[i][j]+W[1][2]* Fy[i][j+1]+W[2][0]* Fy[i+1][j-1]+W[2][1]*Fy[i+1][j]+W[2][2]*Fy[i+1][j+1]; Sigmaxx=W[0][0]*Fx[i-1][j-1]*Fx[i-1][j-1]+ W[0][1]*Fx[i-1][j]* Fx[i-1][j]+W[0][2]*Fx[i-1][j+1]*Fx[i-1][j+1]+ 74 W[1][0]*Fx[i][j-1]* Fx[i][j-1]+W[1][1]*Fx[i][j]*Fx[i][j-1]+W[1][2]* Fx[i][j+1]*Fx[i][j+1]+ W[2][0]*Fx[i+1][j-1]*Fx[i+1][j-1]+W[2][1]* Fx[i+1][j]*Fx[i+1][j]+ W[2][2]*Fx[i+1][j+1]*Fx[i+1][j+1]-Mux*Mux; Sigmaxy=W[0][0]*Fx[i-1][j-1]*Fy[i-1][j-1]+ W[0][1]*Fx[i-1][j]* Fy[i-1][j]+ W[0][2]*Fx[i-1][j+1]*Fy[i-1][j+1]+W[1][0]* Fx[i][j-1]*Fy[i][j-1]+ W[1][1]*Fx[i][j]*Fy[i][j-1]+W[1][2]*Fx[i][j+1]* Fy[i][j+1]+W[2][0]* Fx[i+1][j-1]*Fy[i+1][j-1]+W[2][1]*Fx[i+1][j]* Fy[i+1][j]+ W[2][2]*Fx[i+1][j+1]*Fy[i+1][j+1]-Mux*Muy; Sigmayx=Sigmaxy; Sigmayy=W[0][0]*Fy[i-1][j-1]*Fy[i-1][j-1]+ W[0][1]*Fy[i-1][j]* Fy[i-1][j]+W[0][2]*Fy[i-1][j+1]*Fy[i-1][j+1]+ W[1][0]*Fy[i][j-1]* Fy[i][j-1]+W[1][1]*Fy[i][j]*Fy[i][j-1]+W[1][2]* Fy[i][j+1]*Fy[i][j+1] +W[2][0]* Fy[i+1][j-1]*Fy[i+1][j-1]+W[2][1]*Fy[i+1][j]* Fy[i+1][j]+W[2][2]* Fy[i+1][j+1]*Fy[i+1][j+1]-Muy*Muy; 75 L=(Sigmaxx-Sigmayy)*(Sigmaxx-Sigmayy)+4*Sigmaxy*Sigmaxy; Lenda1=0.2*(Sigmaxx+Sigmayy+sqrt(L)); Lenda2=0.2*(Sigmaxx+Sigmayy-sqrt(L)); if (Lenda1==0) { E=10000; } else { E=sqrt(Lenda1*Lenda1-Lenda2*Lenda2)/Lenda1; } if (i<20) { IT[i][j]=int(0.4*Lenda1*E*E*200000)*(20-i); } else { IT[i][j]=int(0.4*Lenda1*E*E*200000); } } } } 76 void RGB_Texture_GreenGrass(double G[][160],double B[][160], double IT[][160],int stand_Y,double stand_sigma) { double Fx[120][160];//first derivative in x double Fy[120][160];//first derivative in y //double S[120][160]; double W[3][3]={{0.0113,0.0838 , 0.0113},{0.0838 ,0.6193 , 0.0838}, { 0.0113 , 0.0838 , 0.0113}}; int i=0; int j=0; for (i=19;i<120;i++) { for (j=0;j<160;j++) { if (j==159) { Fx[i][j]=fabs(G[i][j]-B[i][j-1])/B[i][j-1]; //IT[i][j]=I[i][j]-I[i][j-1]; } else { Fx[i][j]=fabs(G[i][j+1]-B[i][j])/B[i][j]; //IT[i][j]=I[i][j]-I[i][j-1]; } if (i==119) 77 { Fy[i][j]=fabs(G[i][j]-B[i-1][j])/B[i-1][j]; } else { Fy[i][j]=fabs(G[i+1][j]-B[i][j])/B[i][j]; } } } double Mux=0; double Muy=0; double Sigmaxx=0; double Sigmaxy=0; double Sigmayx=0; double Sigmayy=0; double L; double Lenda1; double Lenda2; double E; for (i=19;i<120;i++) { ///if(i>55) //{ Recal_W(W,i,stand_sigma,stand_sigma,stand_Y); //} 78 for (j=0;j<160;j++) { Mux=0; Muy=0; Sigmaxx=0; Sigmaxy=0; Sigmayx=0; Sigmayy=0; Lenda1=0; Lenda2=0; E=0; Mux=W[0][0]*Fx[i-1][j-1]+W[0][1]*Fx[i-1][j]+ W[0][2]*Fx[i-1][j+1] +W[1][0]*Fx[i][j-1]+W[1][1]*Fx[i][j]+W[1][2]* Fx[i][j+1]+W[2][0]* Fx[i+1][j-1]+W[2][1]*Fx[i+1][j]+W[2][2]*Fx[i+1][j+1]; Muy=W[0][0]*Fy[i-1][j-1]+W[0][1]*Fy[i-1][j]+ W[0][2]*Fy[i-1][j+1] +W[1][0]*Fy[i][j-1]+W[1][1]*Fy[i][j]+W[1][2]* Fy[i][j+1]+W[2][0]* Fy[i+1][j-1]+W[2][1]*Fy[i+1][j]+W[2][2]*Fy[i+1][j+1]; Sigmaxx=W[0][0]*Fx[i-1][j-1]*Fx[i-1][j-1]+ W[0][1]*Fx[i-1][j]* Fx[i-1][j]+W[0][2]*Fx[i-1][j+1]*Fx[i-1][j+1]+ W[1][0]*Fx[i][j-1]* Fx[i][j-1]+W[1][1]*Fx[i][j]*Fx[i][j-1]+W[1][2]* Fx[i][j+1]* 79 Fx[i][j+1]+ W[2][0]*Fx[i+1][j-1]*Fx[i+1][j-1]+W[2][1]*Fx[i+1][j]* Fx[i+1][j]+W[2][2]* Fx[i+1][j+1]*Fx[i+1][j+1]-Mux*Mux; Sigmaxy=W[0][0]*Fx[i-1][j-1]*Fy[i-1][j-1]+ W[0][1]*Fx[i-1][j]* Fy[i-1][j]+W[0][2]*Fx[i-1][j+1]*Fy[i-1][j+1]+ W[1][0]*Fx[i][j-1]* Fy[i][j-1]+W[1][1]*Fx[i][j]*Fy[i][j-1]+W[1][2]* Fx[i][j+1]* Fy[i][j+1]+W[2][0]*Fx[i+1][j-1]*Fy[i+1][j-1]+ W[2][1]*Fx[i+1][j]* Fy[i+1][j]+W[2][2]*Fx[i+1][j+1]*Fy[i+1][j+1]- Mux*Muy; Sigmayx=Sigmaxy; Sigmayy=W[0][0]*Fy[i-1][j-1]*Fy[i-1][j-1]+ W[0][1]*Fy[i-1][j]* Fy[i-1][j]+W[0][2]*Fy[i-1][j+1]*Fy[i-1][j+1]+ W[1][0]*Fy[i][j-1]* Fy[i][j-1]+W[1][1]*Fy[i][j]*Fy[i][j-1]+ W[1][2]*Fy[i][j+1]*Fy[i][j+1]+ W[2][0]*Fy[i+1][j-1]*Fy[i+1][j-1]+ W[2][1]*Fy[i+1][j]*Fy[i+1][j]+ W[2][2]*Fy[i+1][j+1]*Fy[i+1][j+1]-Muy*Muy; L=(Sigmaxx-Sigmayy)*(Sigmaxx-Sigmayy)+4*Sigmaxy*Sigmaxy; Lenda1=0.2*(Sigmaxx+Sigmayy+sqrt(L)); Lenda2=0.2*(Sigmaxx+Sigmayy-sqrt(L)); 80 if (Lenda1==0) { E=10000; } else { E=sqrt(Lenda1*Lenda1-Lenda2*Lenda2)/Lenda1; } //IT[i][j]=int(0.4*Lenda1*E*E*20000*2); if (i<20) { IT[i][j]=int(0.4*Lenda1*E*E*20000)*(20-i)*4; } else { IT[i][j]=int(0.4*Lenda1*E*E*20000); } } } } void RGB_Texture_BrownGrass(double R[][160],double B[][160], double IT[][160],int stand_Y,double stand_sigma) { double Fx[120][160];//first derivative in x double Fy[120][160];//first derivative in y //double S[120][160]; 81 double W[3][3]={{0.0113,0.0838 , 0.0113}, {0.0838 ,0.6193 , 0.0838}, { 0.0113 , 0.0838 , 0.0113}}; int i=0; int j=0; for (i=19;i<120;i++) { for (j=0;j<160;j++) { if (j==159) { Fx[i][j]=fabs(R[i][j]-B[i][j-1])/B[i][j-1]; //IT[i][j]=I[i][j]-I[i][j-1]; } else { Fx[i][j]=fabs(R[i][j+1]-B[i][j])/B[i][j]; //IT[i][j]=I[i][j]-I[i][j-1]; } if (i==119) { Fy[i][j]=fabs(R[i][j]-B[i-1][j])/B[i-1][j]; } else { Fy[i][j]=fabs(R[i+1][j]-B[i][j])/B[i][j]; 82 } } } double Mux=0; double Muy=0; double Sigmaxx=0; double Sigmaxy=0; double Sigmayx=0; double Sigmayy=0; double L; double Lenda1; double Lenda2; double E; for (i=19;i<120;i++) { //if(i>55) //{ Recal_W(W,i,stand_sigma,stand_sigma,stand_Y); //} for (j=0;j<160;j++) { Mux=0; Muy=0; Sigmaxx=0; Sigmaxy=0; 83 Sigmayx=0; Sigmayy=0; Lenda1=0; Lenda2=0; E=0; Mux=W[0][0]*Fx[i-1][j-1]+W[0][1]*Fx[i-1][j]+ W[0][2]* Fx[i-1][j+1]+W[1][0]*Fx[i][j-1]+W[1][1]*Fx[i][j]+ W[1][2]*Fx[i][j+1]+W[2][0]*Fx[i+1][j-1]+W[2][1]* Fx[i+1][j]+W[2][2]*Fx[i+1][j+1]; Muy=W[0][0]*Fy[i-1][j-1]+W[0][1]*Fy[i-1][j]+ W[0][2]* Fy[i-1][j+1]+W[1][0]*Fy[i][j-1]+W[1][1]*Fy[i][j]+ W[1][2]* Fy[i][j+1]+W[2][0]*Fy[i+1][j-1]+W[2][1]*Fy[i+1][j]+ W[2][2]* Fy[i+1][j+1]; Sigmaxx=W[0][0]*Fx[i-1][j-1]*Fx[i-1][j-1]+W[0][1]* Fx[i-1][j]*Fx[i-1][j]+W[0][2]*Fx[i-1][j+1]* Fx[i-1][j+1]+ W[1][0]*Fx[i][j-1]*Fx[i][j-1]+W[1][1]*Fx[i][j]* Fx[i][j-1]+ W[1][2]*Fx[i][j+1]*Fx[i][j+1]+W[2][0]*Fx[i+1][j-1]* Fx[i+1][j-1]+ W[2][1]*Fx[i+1][j]*Fx[i+1][j]+W[2][2]*Fx[i+1][j+1]*F x[i+1][j+1]- Mux*Mux; 84 Sigmaxy=W[0][0]*Fx[i-1][j-1]*Fy[i-1][j-1]+W[0][1]* Fx[i-1][j]* Fy[i-1][j]+W[0][2]*Fx[i-1][j+1]*Fy[i-1][j+1]+W[1][0]* Fx[i][j-1]* Fy[i][j-1]+W[1][1]*Fx[i][j]*Fy[i][j-1]+W[1][2]* Fx[i][j+1]* Fy[i][j+1]+W[2][0]*Fx[i+1][j-1]*Fy[i+1][j-1]+W[2][1]* Fx[i+1][j]* Fy[i+1][j]+W[2][2]*Fx[i+1][j+1]*Fy[i+1][j+1]-Mux*Muy; Sigmayx=Sigmaxy; Sigmayy=W[0][0]*Fy[i-1][j-1]*Fy[i-1][j-1]+W[0][1]* Fy[i-1][j]* Fy[i-1][j]+W[0][2]*Fy[i-1][j+1]*Fy[i-1][j+1]+ W[1][0]*Fy[i][j-1]* Fy[i][j-1]+W[1][1]*Fy[i][j]*Fy[i][j-1]+W[1][2]* Fy[i][j+1]* Fy[i][j+1]+W[2][0]*Fy[i+1][j-1]*Fy[i+1][j-1]+ W[2][1]*Fy[i+1][j]* Fy[i+1][j]+W[2][2]*Fy[i+1][j+1]*Fy[i+1][j+1]-Muy*Muy; L=(Sigmaxx-Sigmayy)*(Sigmaxx-Sigmayy)+4*Sigmaxy*Sigmaxy; Lenda1=0.2*(Sigmaxx+Sigmayy+sqrt(L)); Lenda2=0.2*(Sigmaxx+Sigmayy-sqrt(L)); if (Lenda1==0) { E=10000; } else 85 { E=sqrt(Lenda1*Lenda1-Lenda2*Lenda2)/Lenda1; } IT[i][j]=int(0.4*Lenda1*E*E*20000*2); } } } 9.3 Segmentation Algorithm–To Draw Horizontal White Lines and Output Road Edge Points ////////////////////////////////////////////////////////////////// // Road Segmentation Function: // Drawing White Lines on the road surface to indicate the drivable // surface from the initial position which is close to the bottom to // the vanishing points or the obstacle in font of vehicle shown in // the image. // Those white lines expand to both left and right road boundaries // from the center of the road shown in image. // The end points’ coordinates of those white lines are output for // simulate the road edges by fitting a curve based on those output // points along the road boundaries /////////////////////////////////////////////////////////////////// int Draw_Line_LaneMarks(int Rectangle[],double 86 k[][160],double R[][160], double G[][160],double B[][160],double S[][160],int LeftPoints[],int RightPoints[],int Points[][2],int HeightOfPoints[],short int Start_Line_Position,double *Threshold) { //Defint the initial position which always is belong to the road //surface base on pre-knowledge. int X_Vpoints=80; int X_Init1=80; int Y_Init1=Start_Line_Position; if (fabs(double(Rectangle[0]-Rectangle[2]))<=3) { Rectangle[0]=75; Rectangle[2]=85; } if (fabs(double(Rectangle[1]-Rectangle[3]))<=3) { Rectangle[1]=98; Rectangle[3]=108; } //Get a square sample of a road surface to collect the //intensity values information of road surface int size_of_box=0; size_of_box=int(fabs(double(Rectangle[2]-Rectangle[0])) *fabs(double(Rectangle[3]-Rectangle[1]))); int row_of_box=0; row_of_box=int(fabs(double(Rectangle[3]-Rectangle[1]))); 87 int col_of_box=0; col_of_box=int(fabs(double(Rectangle[2]-Rectangle[0]))); double WindowB1[19200];//window 1 of Blue double WindowS[19200]; for(int i=0;i for (int j=0;j { WindowB1[j+i*col_of_box]=k[Rectangle[1]+i][Rectangle[0]+j]; WindowS[j+i*col_of_box]=S[Rectangle[1]+i][Rectangle[0]+j]; } } double VarB1=Variance(WindowB1,size_of_box); double MeanBlue=Mean_Array(WindowB1,size_of_box); double MinBlue=Min_Array(WindowB1, size_of_box); double MaxBlue=Max_Array(WindowB1, size_of_box); for (int i=0;i { k[Rectangle[1]][Rectangle[0]+i]=0;//MaxBlue; k[Rectangle[3]][Rectangle[0]+i]=0;//MinBlue; } for (int i=0;i { k[Rectangle[1]+i][Rectangle[0]]=0;//Threshold; 88 k[Rectangle[1]+i][Rectangle[2]]=0;//Threshold; } //Drawing White Lines int Y_up=Y_Init1; int X_right=X_Init1+1; int X_left=X_Init1; int PreXRight=X_Init1+1; int PreXLeft=X_Init1; int X_R=X_Vpoints; int X_L=X_Vpoints; int i=0;//The length of the points array. int Len=0; int X_center_AVE=0; while ((S[Y_up][X_Init1]<=*Threshold && S[Y_up+1][X_Init1]<= *Threshold&& S[Y_up+2][X_Init1]<*Threshold)&& (Y_up>55)) { int Decrease_Count=0; if (Y_up<65) { Decrease_Count=2; } else { if(Y_up<75) 89 { Decrease_Count=2; } else { Decrease_Count=3; } } while ((X_left>=0 && S[Y_up][X_left]<=*Threshold)) { k[Y_up][X_left]=255; X_left--; } while ((S[Y_up][X_right]<=*Threshold && X_right<=159)) { k[Y_up][X_right]=255; X_right++; } i++; X_center_AVE+=int((X_left+X_right)/2); if (Y_up>55&&Len<40&&(X_left<(X_right-2))) { LeftPoints[Len]=X_left; RightPoints[Len]=X_right; 90 HeightOfPoints[Len]=Y_up; Len++; } Y_up-=Decrease_Count; X_Init1=int(X_center_AVE/i); PreXRight=X_right; PreXLeft=X_left; X_right=X_Init1+1; X_left=X_Init1; } 91 BIBLIOGRAPHY [1] W.Jones, ”Building Safe Cars,” IEEE Spectrum, vol 39, no.1, pp.82-85,2002. [2] E. Dickmanns, ”The Development of Machine Vision For Road Vehicles in the Last decade,” Proc. IEEE Intelligent Vehicle Symp., 2002. [3] M. Bertozzi, A. Broggi, and A .Fascioli, ”Vision -Based intelligent Vehicles: State of Art and Perspectives,” Robotics and Autonomous System, vol. 32, pp. 1- 16,2000. [4] M.Bertozzi, A. Broggi, M. Cellario, A. Fascioli, P. Lombardi, and M.Porta, ” Artificial Vision in Road Vehicles,” Proc.IEEE, vol.90, no.7, pp.1258-1271,2002. [5] R. Bishop, ” Intelligent Vehicle Applications Worldwide,” IEEE Intelligent Sys- tems,2000 [6] D.Gavrila and U.Franke, and C. Wohler, and S. Gorzig, ” Real-Time Vision for Intelligent Vehicles,” IEEE Instrumentation and Measurement Magazine, pp.22- 27, 2001. [7] Z. Sun, G.Bebis and Ronald Miller, ”On-Road Vehicle Detection: A review,” IEEE Transaction on Pattern Analysis and Machine Intelligence, vol.28, N0.5, may 2006. [8] Image and vision computing, special issue on applications of computer vision to intelligent vehicles, vol.18, 2000. 92 [9] IEEE Trans.Vehicular Technology, special issue on in-vehicle computer vision system, vol.53, 2004. [10] R.Chapuis, R. Aufrere, and F. Chausse,” Accurate Road Following and Recon- struction by Computer Vision,” IEEE Transaction on Intelligent Transportation System, vol.3, No.4, 2002. [11] A. Nuchter, K. Lingemann, and J.Hertzberg, ” Extracting Drivable Surfaces in Ourdoor 6D Slam,” Institute for Computer Science, knowledge-based System Research Group, Germany. [12] Christopher Rasmussen,” Chonining Laser Range, Color, and Texture Cues for Autonomous Road Following.” National Insititute Of Standards and Technology, Gaithersburg, MD 20899. [13] T.kalinke, C.Tzomakas, and W.von SeelEn, ” A Texture-Based Object Detection and an Adaptive Model-Based Classification,” Proc.IEEE Int’l Conf.Intelligent Vehicles,pp.143-148,1998. [14] R.Haralick, B.Shanmugam, and I.Dinstein,”Texture Features for Image Classifi- cation,” IEEE Trans.System, Man, and Cybernetics,pp.610-621,1973. [15] R.Cucchiara and M.Piccardi,”Vehicles Detection under Dat and Night Illumina- tion,” Proc.Int’l ICSC Sysmp.Intelligent Industrial Automation,1999. [16] J.Zhang and H.Nagel,”Texture-Based Segmentation of Road Images.” Fraunhofer-Institue fur Information und Datenverarbertung (IITB), Fraunhofer- str.1, D-76131 Karlsruhe, Germany. [17] Q.Zhang, J.Wang, P.Gong and P.Shi,”Texture analysis for urban spatial pattern studt using SPOT imagery.” 0-7803-7031-7/01 IEEE 2001. 93 [18] D.Blostrin and N.Ahuja, :A MultiScale Region Detector” Computer Vision, Graphics, and Image Processing 45 22-41,1989. [19] R.W.Conners and C.A.Harlow, ”A Theoretical Comparison of Texture Algo- rithms.” IEEE Trans. Pattern Analysis and machine Intelligence PAMI-2:3 204- 222, 1980. [20] S.Beucher and X.Yu,”Road Recognition in Complex Traffic Situations.” Ecole Des Mines De Paris, Centre De morphologie Mathematique, Fontainebleau,France. [21] M.Kass and A.P.Witkin, ”Analyzing Oriented Patterns.” Computer Vision, Graphics, and Image Processing 37 362-385, 1987. [22] Amaury Rolin,”An Overview of Sensor Processing Challenges and Strategies for DEXTER, an Autonomous Robot, in the URBAN Challenge” EECS Case Western Reserve University Cleveland, OH, USA. [23] http://www.worldcarfans.com/2050616.005/new-s-class-to-debut-pre-safe- radar/. [24] http : //en.wikipedia.org/wiki/V olvoS80 [25] Baluja, S.Sch. of Comput. Sci., Carnegie Mellon Univ., Pittsburgh, PA;”IEEE Transactions, Volume 26, Issue 3, Jun 1996 Page(s):450 -463 [26] SCOTT MCMICHAEL,”Lane Detection for DEXTER, an Autonomous Robot, in the Urban Challenge” Case Western Reserve University, M.S.Thesis, May 2008 [27] Gary A. Carr Team ENSCO Leader,Brian Mee Co-Team Leader,”DARPA Grand Challenge 2005 Team ENSCOs DEXTER” 2005. 94