STATE-OF-THE-ART REMOTE SENSING GEOSPATIAL TECHNOLOGIES IN
SUPPORT OF TRANSPORTATION MONITORING AND MANAGEMENT
DISSERTATION
Presented in Partial Fulfillment of the Requirements for
The Degree Doctor of Philosophy in the Graduate
School of The Ohio State University
By
Eva Petra Paska, M.S.
*****
The Ohio State University 2009
Dissertation Committee: Approved by
Dr. Dorota Grejner-Brzezinska, Adviser
Dr. Mark McCord ______
Dr. Alper Yilmaz Adviser
Dr. Charles K. Toth, Co-Adviser Geodetic Science and Surveying
Graduate Program
ABSTRACT
The widespread use of digital technologies, combined with rapid sensor advancements resulted in a paradigm shift in geospatial technologies the end of the last millennium. The improved performance provided by the state-of-the-art airborne remote sensing technology created opportunities for new applications that require high spatial and temporal resolution data. Transportation activities represent a major segment of the economy in industrialized nations. As such both the transportation infrastructure and traffic must be carefully monitored and planned. Engineering scale topographic mapping has been a long-time geospatial data user, but the high resolution geospatial data could also be considered for vehicle extraction and velocity estimation to support traffic flow analysis.
The objective of this dissertation is to provide an assessment on what state-of-the- art remote sensing technologies can offer in both areas: first, to further improve the accuracy and reliability of topographic, in particular, roadway corridor mapping systems, and second, to assess the feasibility of extracting primary data to support traffic flow computation. The discussion is concerned with airborne LiDAR (Light Detection And
Ranging) and digital camera systems, supported by direct georeferencing.
The review of the state-of-the-art remote sensing technologies is dedicated to address the special requirements of the two transportation applications of airborne remotely sensed
ii data. The performance characteristics of the geospatial sensors and the overall error budget are discussed. The error analysis part is focused on the overall achievable point positioning accuracy performance of directly georeferenced remote sensing systems.
The QA/QC (Quality Assurance/Quality Control) process is a challenge for any airborne direct georeferencing-based remote sensing system. A new method to support
QA/QC is introduced that uses the road pavement markings to improve both sensor data accuracy as well as the position of road features. The identification of the pavement markings is based on LiDAR intensity data and is guided by the ground control information available. The centerline of the markings in LiDAR data is modeled and matched to the reference data, providing the observation to the QA/QC process.
The discussion on the innovative use of remote sensing technologies investigates the feasibility of providing remotely sensed traffic data for monitoring and management.
An advantage of air-based platforms, including manned and unmanned fixed-wing aircraft and helicopters, is that they can be rapidly deployed to observe traffic incidents that occur in areas where there are no ground-based sensors. To support vehicle extraction from airborne imagery, a method was introduced that provides a true object scale data representation that can facilitate the vehicle extraction. The vehicle extraction from LiDAR data was followed by coarse classification of the extracted vehicles to support coarse velocity estimation; basically, grouping the vehicles into three major categories based on their size. Finally, a novel method was introduced for simultaneously acquired LiDAR and image data, which can combine the advantages of the two sensors for obtaining better velocity estimates of LiDAR-extracted vehicles.
iii
Dedicated to my family and Babóka
iv
ACKNOWLEDGEMENTS
I wish to thank my adviser, Dr. Dorota Grejner-Brzezinska for her support and
encouragement during the years of my doctoral studies. She was my role model during
my studies, helping me to develop my professional identity and provided considerable
advice regarding the requisites.
I am enormously indebted to Dr. Charles Toth without whose continuous support, guidance and encouragement throughout the years of my studies this dissertation would not have been possible. His passion and enthusiasm in science and engineering provided me with motivation to improve my problem solving skills and creativity. The technical advice he offered during our daily discussions over the past several years has proven to be invaluable.
I wish to thank Dr. Mark McCord for the knowledge he imparted, the valuable discussions about the transportation discipline, and the careful and thorough review of my dissertation.
I also wish to thank Dr. Alper Yilmaz for his valuable comments and suggestions on the dissertation draft.
Additionally, I would also like to thank The Ohio Department of Transportation, in particular, John Ray, for providing the test datasets for my research.
v
VITA
March 14, 1977………...... Born in Eger, Hungary
2001……………………… M.S., Surveying and Geoinformatics Engineering, Budapest University of Technology and Economics, Budapest, Hungary
2007……………………… M.S., Department of Geodetic and Geoinformation Science, The Ohio State University
Oct 2000 – April 2002…… Intern, Center for Mapping, The Ohio State University
June 2002 – Feb 2008……. Graduate Research Associate, The Ohio State University
Feb 2008 – present……….. Photogrammetrists/LiDAR Processing Specialist, Kucera International Inc., Willoughby, OH
PUBLICATIONS
PEER-REVIEWED PUBLICATIONS:
1. Grejner-Brzezinska, D.A., C. K. Toth, Shahram Moafipoor, Eva Paska, Nora Csanyi, 2007, Vehicle Classification and Traffic Flow Estimation from Airborne LiDAR/CCD Data, IAG Symposia, Monitoring and Understanding a Dynamic Planet with Geodetic and Oceanographic Tools, Springer Berlin-Heidelberg.
2. Grejner-Brzezinska, D. A, Toth, C. and Paska, E., 2007. Airborne Remote Sensing Supporting Traffic Flow Estimation, Advances in Mobile Mapping Technology, ISPRS Book Series, editors: C. V. Tao and J. Li, Taylor and Francis, pp. 51-60.
3. Toth, C. K., Paska, E., Chen, Q., Zhu, Y., Redmill, R. and Ozguner, U., 2006: Mapping Support for the OSU DARPA Grand Challenge Vehicle, Proceedings of the 2006 IEEE Intelligent Transportation Systems Conference, Toronto, Canada, September 17-20, pp. 1580-1585.
vi
PROCEEDINGS PAPERS:
1. Toth, C., E. Paska and D. Brzezinska, (2008): Quality Assessment of LiDAR Data by Using Pavement Markings, ASPRS Annual Conference, Portland, Oregon, April 28 – May 2.
2. Paska, E. and Ray, J., 2007. Influence of Various Parameters on the Accuracy of Lidar Generated Products for Highway Design Applications, ASPRS Annual Conference, Tampa, Florida, May 7-11, 2007.
3. Toth, C., Paska, E., and Grejner-Brzezinska, D.A., 2007. Using Pavement Markings to Support the QA/QC of LiDAR Data, Photogrammetric Image Analysis (PIA 2007), International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 36 (3/W49B), pp. 173-178.
4. Toth, C. K. and Paska, E., 2006. Mobile Mapping and Autonomous Vehicle Navigation, ISPRS Commission I Symposium, Paris, France, July 4-6.
5. Toth C., D. Grejner-Brzezinska, N. Csanyi, E. Paska, S. Moafipoor, 2006. Airborne Platform Navigation Using a Closed Feedback Loop Between GPS/IMU and LiDAR Systems, IEEE/ION PLANS, San Diego, April 25-27.
6. Bevis, M., Grejner-Brzezinska, D. A., Toth, C. K., Paska, E. et al. (2005). The B4 Project: Scanning the San Andreas and San Jacinto Fault Zones, American Geophysical Union Fall Meeting, San Francisco, CA, December 5-9. (invited).
7. Paska, E. and Toth C.K., 2005. Vehicle Velocity Estimation by Combining LiDAR and Airborne Imagery, ISPRS WG I/2 Workshop on 3D Mapping from InSAR and LIDAR, Banff, Alberta, Canada, June 7-10, CD-ROM.
8. Paska, E. and Toth, C., 2005. Vehicle Velocity Estimation from Airborne Imagery and LiDAR, ASPRS Annual Conference, Baltimore, MD, March 7-11, 2005.
9. Toth, C. and Paska, E., 2005. Mapping Support For The Terramax Osu/Oshkosh Darpa Grand Challenge Team, ASPRS Annual Conference, Baltimore, MD, March 7- 11, 2005.
10. Grejner-Brzezinska, D., Toth, C., Paska, E. and Moafipoor, S., 2004. Precise Vehicle Topology and Road Surface Modeling Derived from Airborne LiDAR Data, ION GNSS 2004, Long Beach, California, September 21-24, 2004, CD-ROM.
11. Paska, E. and Toth, C., 2004. A Performance Analysis on Vehicle Detection from Remotely Sensed Imagery, Proceedings of the ASPRS Annual Conference, Denver, Colorado, May 23-28, CD-ROM.
vii
12. Grejner-Brzezinska, D., Toth, C. and Paska, E., 2004. Airborne Remote Sensing Supporting Traffic Flow Estimates, Proc. of 3rd International Symposium on Mobile Mapping Technology, Kunming, China, March 29-31, 2004, CD-ROM.
13. Csanyi, N., Paska, E. and Toth, C., 2003. Comparison of Various Surface Modeling Methods, ASPRS Fall Conference, Charleston, South Carolina, October 27-30, 2003, CD-ROM.
14. Grejner-Brzezinska, D., Toth, C. and Paska, E., 2003. Airborne Remote Sensing: Redefining a Paradigm of Traffic Flow Monitoring, ION GPS 2003, Portland, Oregon, September 24-27, 2003, CD-ROM.
15. Paska, E. and Toth, C., 2003. Object Space Segmentation Supported By LiDAR Intensity Data, Joint Workshop of ISPRS WG I/3 & II/2, Portland, Oregon, June 17- 19, 2003, CD-ROM.
16. Paska, E. and Toth, C., 2003. LIDAR Data Segmentation Based on Morphologic Filtering, Proceedings of the ASPRS Annual Conference, May 5-9, Anchorage, Alaska, CD-ROM.
FIELDS OF STUDY
Major Field: Geodetic Science
viii
TABLE OF CONTENTS Pages Abstract………………………………………………………………………………. ii
Dedication……………………………………………………………………………. iv
Acknowledgments…………………………………………………………………… v
Vita………………………………………………………………………………...... vi
List of Tables……………………………………………………………………...... xiii
List of Figures………………………………………………………………………... xvi
Abbreviations………………………………………………………………………… xx
Chapters:
1. Introduction……………………………………………………………………… 1 2. Transportation applications of remote sensing…………………………………. 6 2.1. Airborne remote sensing………………………………………………….. 6 2.1.1. State-of-the-art geospatial technologies…………………………….. 6 2.1.2. Topographic and infrastructure mapping…………………………… 8 2.1.3. Emerging applications: emergency/rapid mapping, traffic flow extraction……………………………………………………………… 10 2.2. Innovative use of remote sensing in transportation………………………. 14 2.2.1. Motivation for vehicle extraction: the mapping perspective……….. 14 2.2.2. The growing need for traffic data to support transportation management………………………………………………………… 16 2.2.3. Traffic flow extraction from remote sensed data…………………… 18 2.2.4. Ongoing research, background, literature review, concept………… 20
ix
3. Sensor characteristics…………………………………………………………… 26 3.1. Sensor technology developments, enabling sensors………………...... 26 3.2. Imaging sensors…………………………………………………………… 27 3.2.1. Optical imagery: panchromatic and multispectral sensors…………. 28 3.2.2. LiDAR sensors……………………………………………………… 38 3.2.3. IfSAR sensor………………………………………………………... 54 3.2.4. Hyperspectral sensor………………………………………………... 56 3.3. Navigation sensors………………………………………………………... 59 3.3.1. GPS and GNSS systems……………………………………………. 59 3.3.2. Inertial systems……………………………………………………... 64 3.3.3. Integrated navigation systems………………………………………. 67 3.3.4. Mission planning……………………………………………………. 69 3.3.5. Airborne/spaceborne platforms……………………………………... 74 3.4. Error characteristics………………...... 77 3.4.1. Direct/indirect georeferencing……………………………………… 78 3.4.2. The achievable accuracy of GPS/INS-based remote sensing systems……………………………………………………………… 83 4. Concept and methods for improving the accuracy of road infrastructure mapping using pavement markings………………...... 88 4.1. LiDAR data validation and performance analysis………………………... 88 4.1.1. Existing methods: review, natural surfaces and LiDAR targets…..... 90 4.1.2. Concept of using pavement markings (intensity and technology)….. 93 4.2. QA/QC method using feature-based matching of pavement markings extracted from LiDAR with ground control……………………………… 97 4.2.1. Extraction of pavement markings from LiDAR intensity signal…… 98 4.2.2. Curve fitting-based modeling of pavement markings………………. 111 4.2.3. Matching pavement mark with reference……………………...... 121 4.2.3.1. ICP performance test with simulated data……………………. 124 4.2.3.2. ICP performance test with real data………………...... 130 4.2.4. Data validation and corrections………………….…………………. 135
x
5. Concept and methods to improve traffic flow extraction from remote sensing data ……………………………………………………………………………... 141 5.1. Motivation and concept……………….……...... 141 5.2. Road extraction………………………………….………………………... 142 5.2.1. Road surface extraction from LiDAR………………..……………... 144 5.2.2. Road extraction from optical imagery……………………………… 148 5.3. Vehicle extraction (static and mobile platforms)…………………………. 149 5.3.1. Vehicle extraction from optical imagery…………………………… 149 5.3.2. Vehicle extraction from LiDAR……………………………………. 157 5.3.3. Classification of extracted vehicles………………………………… 160 5.4. Vehicle tracking…………………………………………………………... 163 5.5. Vehicle velocity estimation………………………………………………. 165 5.5.1. Vehicle velocity estimation from optical imagery…………………. 165 5.5.2. Vehicle velocity estimation from LiDAR………………………….. 166 5.5.3. Vehicle velocity estimation from the combined dataset (optical imagery and LiDAR)……………………………………………...... 170 5.5.4. Performance evaluation…………………………………………….. 173 5.6. Accuracy assessment of the vehicle velocity estimation from the combined dataset………………………………………………………….. 177 5.6.1. Accuracy of single ground point determination and vehicle size estimation from optical imagery………………………………...... 178 5.6.2. Uncertainties in size-parameter estimation from LiDAR data……... 183 5.6.3. Accuracy of the vehicle velocity estimation from the combined dataset……………………………………………………………….. 184 5.7. Traffic flow computation…………………………………………………. 187 5.7.1. Traffic flow parameters……………………………………...... 187 5.7.2. Platforms and sensors………………………………...... 189 5.7.2.1. Fixed-wing aircraft with high-performance imaging sensor…………………………………………………………. 191 5.7.2.2. Helicopter with medium format camera and LiDAR…………. 193
xi
5.7.2.3. UAV with small-format camera ……………………………… 195 5.7.2.4. Fixed pole with medium-format camera installation…………. 196 5.7.3. Performance comparison……………………………………………. 198 6. Summary, contribution, and future recommendations………………………….. 201 6.1. Summary and contributions…………………………………..…………... 201 6.2. Future recommendations………………………………………………….. 206 6.3. Emerging trend: terrain-referenced navigation…………………………… 210 List of References…………………………………………………………………... 213 Appendices A.1. Matlab code for pavement extraction…………………………………………. 223 A.2. Matlab code for for ICP-based matching, curve fitting and QA/QC computation….. 224
xii
LIST OF TABLES
Table Page
2.1 Paradigm shift and conceptual developments in geospatial sciences……….. 7
2.2 Technological developments relevant to geospatial technologies…………. 7
2.3 Milestones of airborne remote sensing developments ……………………... 8
3.1 Digital camera categories and major camera systems ……………………… 31
3.2 Manufacturer’s accuracy specification (RMSE) of LiDAR height, as a function of altitude…………………………………………………………. 42
The characteristics of selected commercial airborne laser scanning 3.3 systems…………………………………………………………………….... 50
3.4 HSI sensor categories and major systems.…………………………………. 57
3.5 Major pushbroom HSI systems…………………………………………….. 58
3.6 Specification of CASI-1500………………………………………………… 59
3.7 Comparison of various kinematic GPS methods…………………………… 62
3.8 Operational and planned GNSS systems…………………………………… 64
3.9 Manufacturer’s specifications for the inertial sensors; sf: scale factor; ma: 66 misalignment; wn: white noise; rw: random walk; bw: bandwidth…………
3.10 Advantages and disadvantages of GPS and IMU systems in standalone and 67 integrated modes…………………………………………………………….
3.11 POS/AV absolute accuracy specifications; post-processed RMSE values….... 68
3.12 Specification and accuracy values for the Leica IPAS10 system………..…. 69
xiii
3.13 Comparison of GeoEye-1 vs. Worldview-1 satellite sensors……………….. 75
3.14 UAV classification………………………………………………………….. 76
3.15 Comparison of the three major georeferencing techniques……………….. 85
3.16 Different groups of the parameters with typical error values………………. 84
4.1 Road pavement sample statistics of LiDAR intensity values ……………… 101
4.2 Grassy and soil area sample statistics of LiDAR intensity values …………. 103
4.3 Road pavement marking sample statistics of LiDAR intensity values for 105 points with intensity value larger than 180………………………………….
4.4 Least squares parameter estimation of a third-order polynomial …………... 118
4.5 Transformation parameter recovery based on ICP matching ……………… 128
4.6 Coordinate difference statistics at sample points…………………………… 128
4.7 ICP performance with respect to random noise, straight line case ………… 130
4.8 Transformation results (2D)………………………………………………… 133
4.9 Original differences and residuals after ICP (2D) between corresponding LiDAR and reference points………………………………………………… 133
4.10 The initial and final residuals for all the points and by features…………… 135
5.1 Misclassification rate of the training set…………………………………… 162
5.2 LiDAR-sensed lengths of passenger cars traveling at typical freeway speed.………………………………………………………………..………. 168
5.3 The velocity error estimates based on measured vehicle length……………. 168
5.4 Vehicle length and width measurement from LiDAR and length estimation based on combined LiDAR and image data (‘+’: marks vehicle moving in opposite direction with respect to LiDAR sensor, ‘0’: vehicle is not moving, and ‘-‘: vehicle moves in the same direction as the LiDAR sensor). 176
xiv
5.5 Velocity estimation performance for various sensor settings for LiDAR- only and for combined LiDAR and image data. Flight lines where vehicle and LiDAR move in the opposite directions are marked by white cells, while where vehicle and LiDAR move in the same directions are marked by shaded cells...... 177
5.6 The required parameters to describe the mathematical relation between image and ground coordinates by single photo resection ………………… 179
5.7 Accuracy of the orientation parameters and image coordinate measurements………………………………………………………………. 181
5.8 The accuracy estimates of ground coordinates of a center image point, and the length parameter estimation for different scenarios…………………… 182
5.9 The accuracy estimates of ground coordinates of an image point that is at the edge of the image, and the length parameter estimation for different scenarios………………………………………………………………..…… 183 … 5.10 Summary of parameters considered in the computation of the standard deviation of vehicle velocities at different LiDAR-sensed lengths………… 185
5.11 Feasible platform and sensor combinations………………………………… 190
5.12 Derived traffic flow parameters with error estimates; overall performance results for the Toronto, Ontario, tests; P- passenger car, M- multipurpose utility vehicle, T-truck……………………………………………………… 193
5.13 Sensor performance metrics with respect to the different processing steps of traffic flow extraction (shaded areas indicate cases when one sensor significantly outperforms the other one)……………………………………. 200
xv
LIST OF FIGURES
Figure Page
3.1 Projection between object and image space; see notation below …………. 35
3.2 Main components of LiDAR systems.…………………………………….. 39
3.3 (a) perpendicular return, no changes in shape; (b) deformed shape at sloped surface.……………………………………………………………... 43
3.4 Multiple reflections in vegetated areas; cyan: only one return pulse, red: first of multiple returns, orange: intermediate of multiple returns, blue: last of multiple returns.…………………………...... 44
3.5 Waveform example of two returns from close objects; blue lines mark detected returns …………………………………………………………… 45
3.6 Vertical shift in LiDAR data caused by retro-reflective materials: (a) elevation data, (b) contours, and (c) aerial imagery; note that the text on the taxiway is quite readable, although the thickness of the pavement markings is in the mm range (definitely way below the cm-level laser ranging accuracy).……………...... 47
3.7 (a) LiDAR height and (b) intensity data………………………………….. 48
3.8 Velodyne scanner; (a) system and (b) concept (Courtesy of M. Shand)…. 52
3.9 Terrestrial LiDAR data, acquired by the Lynx system from Optech…….. 53
3.10 Accuracy vs. swath width tradeoff of IfSAR……………………….……… 54
3.11 Direct georeferencing with optical imagery………………………………. 80
3.12 Accuracy analysis bar chart, for large format camera with high- performance georeferencing system for typical flying heights: (a) H=300 m, (b) H=600 m, (c) H=1500 m, (d) and H=3000 m ……………………… 85
xvi
3.13 Accuracy analysis bar chart for high-end LiDAR system with high- performance georeferencing system for typical flying heights: (a) H=300 m, (b) H=600 m, (c) H=1500 m, (d) and H=3000 m …………………….. 86
4.1 LiDAR-specific ground target…………………………………………….. 92
4.2 Pavement markings at a road intersection; (a) 4k by 4k digital image (orthorectified), (b) LiDAR intensity (gridded), and (c) LiDAR elevation (gridded)……………...... 96
4.3 General procedure for using linear features for improving the topographic and infrastructure LiDAR mapping (corridor mapping)…………………… 98
4.4 Using a locally optimal threshold in a nearly ideal situation; (a) thersholded image, and (b) the histogram of the original LiDAR intensity data…………………………………………………………………………. 100
4.5 Histograms of various pavement areas (asphalt).……………….………… 102
4.6 Histograms of various grassy and soil areas……………...... 104
4.7 Histograms of various pavement markings, including pavement areas; points with intensity value larger than 180 are considered as pavement marking points…………………………………………………………….. 105
4.8 LiDAR points (blue) and road centerline (green); intensity values are shown numerically ………………………………………………………… 106
4.9 Data processing block diagram of extracting LiDAR points of pavement markings …………………………………………………………………… 108
4.10 Mean residuals as a function of the intensity threshold…………………… 109
4.11 Number of selected points vs. the intensity threshold …………………… 110
4.12 Extracted pavement marking LiDAR points …………………………….. 111
4.13 The curve fitting in the local coordinate systems; the local coordinate system is oriented to the main direction of the segment………………...... 113
4.14 Piecewise weighted least squares curve fitting method …………………. 115
4.15 Transfer of slope at connection points ……………………………………. 117
xvii
4.16 Curves fitted to LiDAR (blue) and GPS reference (red) points; the length of the curve is about 15 m...... 121
4.17 Matching straight lines; yellow: reference, blue: points matched to the curve, red: results of the ICP, and green: iterated points ………………… 125
4.18 Matching third-order curve from two different initial positions, (a) and (b); yellow: reference, blue: points matched to the curve, red: results of the ICP, and green: iterated points……………………………………………. 126
4.19 Matching a sine wave; yellow: reference, blue: points matched to the curve, red: results of the ICP, and green: iterated points………………… 127
4.20 ICP result when noise was added to the data; yellow: reference, blue: points matched to the curve, red: results of the ICP, and green: iterated points.…………………………………...... 129
4.21 The effect of point sampling on ICP………………………………………. 131
4.22 ICP matched curves; magenta: curves fitted to control points, red: GPS control points, cyan: LiDAR point and curves fitted, and blue: matched points.……...... 132
4.23 Curve matching based on ICP in a 50 m by 50 intersection area; magenta: curves fitted to control points, red: GPS control points, cyan: curve points derived from LiDAR, and blue: transformed curve points (derived from LiDAR)…...... 134
5.1 The road boundary extraction process...... 145
5.2 The road cross-profiles (top) and the computed ASF showing surface roughness (bottom)………………………………………………………… 147
5.3 Road estimation based on intensity segmentation ……………………….. 147
5.4 Road edge delineation in LiDAR data ……………………………………. 148
5.5 (a) 1-m satellite and(b) 15-cm airborne digital camera images, acquired over the campus of University of Arizona, Tucson………………………. 150
5.6 (a) Vehicles extracted from 1-m satellite and (b) 15-cm airborne images…. 151
5.7 SIFT features extracted and matched from aerial imagery (Courtesy of N. Markiel)…………………………………………………………………….. 152
xviii
5.8 Original images (top row) and their orthorectified counterparts (bottom row)………………………………………………………………………… 155
5.9 Differencing orthorectified images reveals moving vehicles ……………. 156
5.10 Vehicle footprints projected onto the road surface; white circles mark the GIS centerline data ………………………………………………………… 158
5.11 Automatically extracted LIDAR points of a truck……………………….. 159
5.12 The classification of test data using the class boundaries determined in a PCA analysis on a training dataset (blue cross: car, light blue circle: trucks, green circle: other vehicles)……………………………………….. 161
5.13 (a) Reference image and reference point (red rectangle), (b) search window (red), and the corresponding position (blue) with a maximum similarity value of 0.96………………………………………………...... 164
5.14 Vehicle tracking using image sequences, acquired at 6 s intervals……… 164
5.15 Motion artifact: laser image of a truck……………………………………. 167
5.16 Elliptical ground scanning pattern and coverage of an airborne laser scanning system utilizing progressive Palmer scans (Drawn by M. Shand)…………………………………………………………...... 169
5.17 Vehicles extracted from the LiDAR data and overlaid on the orthoimage; (a) match of corresponding vehicles in the two datasets is marked with identical colors. Also shown are (b) vehicle elongation and (c) vehicle shortening ……...... 171
5.18 LiDAR target and the GPSVan ……………………………………………. 174
5.19 Limitations in accurate parameterization of LiDAR-sensed vehicles: (a) data 184 density and footprint size, (b) shadow effect…………………………….. 5.20 Accuracy of vehicle velocity estimation at different LiDAR-sensed vehicle length for scenarios summarized in Table 5.10…………………………… 186
5.21 LiDAR data acquired over a busy freeway in the Toronto downtown area; (a) road surface and objects extracted, and extracted and (b) extracted and classified vehicles …………………………………………………………. 192
5.22 Intersection flow count by direction………………………………………. 194
xix
5.23 Intersection flow count … ………………………………………………… 195
5.24 Flow balance at the intersection…………………………………………… 195
5.25 Medium-format imagery acquired from a tall (10-storey) building, Colombo, Sri Lanka; (a) near vertical orientation and (b) oblique view … 196
5.26 Medium-format oblique imagery acquired from a building; (a) original image and (b) orthorectified image...……………………………………… 197
xx
LIST OF ABBREVIATIONS
2D 2-Dimensional 3D 3-Dimensional AADT Average Annual Daily Traffic AGC Automated Gain Control ASPRS American Society of Photogrammetry and Remote Sensing CAD Computer-Aided Design CCD Charge-coupled device CMOS Complementary metal–oxide–semiconductor CORS Continuously Operating Reference Stations DEM Digital Elevation Model Deflections of the Vertical DOV Deflections of the Vertical DSM Digital Surface Model DTM Digital Terrain Model EKF Extended Kalman Filter EO Exterior Orientation FOV Field Of View FPS Frame per second FWHM Full Width at Half Maximum GIS Geographic Information Systems GNSS Global Navigation Satellite System GPS Global Positioning System GSD Ground Sampling Distance HSI Hyperspectral Imagery ICP Iterative Closest Point IFOV Instantaneous Field Of View
xxi
IfSAR Interferometric Synthetic Aperture RADAR IMU Inertial Measurement Unit INS Inertial Navigation System IO Interior Orientation ITS Intelligent Transportation Systems KF Kalman Filter LBS Location-Based Service LiDAR Light Detection And Ranging NIR Near-Infrared O-D Origin-Destination PCA Principal Component Analysis PCF Piecewise Curve Fitting PDOP Positional Dilution Of Precision QA/QC Quality Assurance/Quality Control RADAR RAdio Detection And Ranging RGB Red Green Blue RPC Rational Polynomial Coefficients SAR Synthetic Aperture RADAR SIFT Scale Invariant Feature Transform SNR Signal to Noise Ratio SRTM Shuttle Radar Topography Mission UAV Unmanned Aerial Vehicles VMT/VKT Vehicle Miles/Km Traveled VRS Virtual Reference Station
xxii
CHAPTER 1
INTRODUCTION
Transportation activities represent a major segment of the economy in industrialized nations. As such both the transportation infrastructure and traffic must be carefully monitored and planned. This monitoring and planning require up-to-date, accurate and continuous methods of screening, mapping, modeling, and managing.
Modern geospatial technologies can effectively support both the monitoring and planning tasks, with the key difference being that infrastructure mapping is only concerned with the static components of object space, basically the transportation corridor infrastructure, while monitoring traffic, the flow of the vehicles, requires addresses the dynamic part of the observation space. Since sensors deployed on different platforms cannot differentiate between the static and moving objects on the data acquisition level, data collected for infrastructure mapping inherently contain information about the non-stationary objects. For example, the road surface must be determined at sub-decimeter level accuracy for engineering purposes. Such accuracy is feasible with modern LiDAR (Light Detection and Ranging) systems. However, vehicles on the road represent obstructions to the LiDAR pulses sent to reflect off the pavement.
1
Therefore, a substantial amount of processing must be devoted to “removing the vehicle
signals.” Rather than removing and discarding the signals, they can be turned into
traffic flow information. This way, LiDAR surveys devoted to surface extraction are
able to provide a valuable byproduct with little additional effort.
Conventional topographic mapping, such as infrastructure mapping of the
transportation network, has been traditionally focused on extracting surface data, such as
DEM/DSM (Digital Elevation Model/Digital Surface Model), and natural and man-made objects. Consistent with the increased urbanization trend, the focus of mapping has clearly shifted toward extracting and positioning of man-made objects in urban areas, such as buildings and roads. Furthermore, as the technology advances, the growing demand for complete 3D city models can be more easily met, as more data with improved information content can be rapidly acquired at lower cost. The proliferation of city models in the consumer market is well-illustrated by the phenomenal success of Internet mapping, introduced primarily by Google Earth and Microsoft Virtual Earth in the past few years. The ongoing research in geospatial sciences is concentrated on improving feature extraction performance and on better supporting QA/QC (Quality
Assurance/Quality Control) processes.
Traffic flow extraction was not associated with mapping in the past. In fact, it is a
rather novel idea to exploit modern geospatial sensors for traffic flow extraction. The
traditional technologies for traffic sensing include inductive loop detectors, tubes, video
cameras, roadside beacons, and travel probes. To improve the efficiency of the
transportation systems, the demand is growing for finding new data sources to support
2 and advance the monitoring and management of traffic flows. This provides the motivation for the research on using remotely sensed data for transportation applications.
Traditional detectors, which are fixed at one location in a transportation network, provide useful information and data on the traffic flows at particular locations. Due to limited spatial distribution, however, they generally cannot provide detailed data for traffic flows over larger areas. Normally, they are not capable of providing data on vehicle trajectories through the network, or velocity, acceleration/deceleration, and routing information of individual vehicles. One of the main advantages of applying remote sensing technologies to traffic monitoring is that sensors can be carried on mobile platforms and they are, therefore, not attached to one location. Remotely sensed imagery provides much better spatial coverage, can be deployed during special events (e.g., evacuation) and is able to provide up-to-date traveler information, if applied in real time. In recent years, remote sensing technology has made remarkable progress and expanded into several application fields. Technological advances give the potential to widen the range of applications and to go beyond conventional topographic mapping. On one hand, the improving spatial and temporal resolution of satellite and airborne imagery, along with denser and more accurate LiDAR and SAR (Synthetic Aperture RADAR) datasets, provide more information-rich data for the feature extraction tasks, including the extraction of moving objects. On the other hand, assuming continuing progress in computer vision, more advanced and robust techniques could be developed following the ongoing research. The novel use of remote sensing technologies for traffic flow data extraction could efficiently support traffic monitoring and, ultimately, real-time traffic management.
3
The objective of this dissertation is to assess the potential of using remote sensing
technologies to support both objectives of improving transportation infrastructure
mapping and providing traffic flow information. This research is concerned with improving QA/QC of infrastructure mapping to support extraction of road surfaces, and traffic flow information, such as vehicle identification, vehicle tracking, and vehicle velocity estimation, based on airborne digital data collected by frame cameras and
LiDAR systems. For highway planning and traffic management purposes, each road segment can be characterized by its traffic flow. There are various flow parameters derived with respect to time, such as flow by hour, day, week, or year over a given road segment. The precondition of traffic flow computation is the availability of vehicles extracted, adequately described, and tracked in order to estimate their velocities.
Identified vehicles then can serve as input to more complex processing tasks, such as estimating velocity profiles of individual vehicles, or monitoring traffic dynamics or traffic pattern modeling.
The dissertation is organized in the following way. Chapter 2 reviews the specifics of the transportation applications of modern remote sensing technologies. The performance characteristics of the geospatial sensors and the overall error budget are discussed in Chapter 3. First, the various imaging sensor technologies are reviewed and analyzed with respect to the selected research objectives. Next, the georeferencing process is studied, as it has a strong impact on the feature extraction processes, and, in fact, it should be included in these processes to achieve a robust solution by maintaining good QA/QC. The error analysis part is focused on the overall achievable accuracy
4 performance of directly georeferenced remote sensing systems. Chapter 4 addresses the objectives of transportation infrastructure mapping and introduces a new method of using the road pavement markings to improve sensor data accuracy and better position road features. The identification of the pavement markings is based on LiDAR intensity data and guided by the ground control information available. The centerline of the markings in
LiDAR data is modeled and matched to the reference data. The improved LiDAR data, including the road surface and pavement markings, could reduce the search space for the vehicle extraction tasks. Chapter 5 is devoted to traffic flow extraction from airborne imagery, including the introduction of new methods to improve data extraction, which serves as input data for the traffic flow estimation. The extraction of moving and stationary vehicles from optical imagery is an additional component of this research; various computer vision techniques are considered in this effort. The velocity estimates are essential for flow computation and the capabilities of both sensors (airborne digital camera and LiDAR) are analyzed in that regard. An analysis to validate the performance of the developed methods based on empirical data, including system level performance metrics and error budget are presented. Finally, Chapter 6 provides a summary, conclusion, recommendation and a future outlook.
5
CHAPTER 2
TRANSPORTATION APPLICATIONS OF REMOTE SENSING
This chapter reviews the transportation applications of remote sensing with the
emphasis on airborne platforms.
2.1. Airborne remote sensing
2.1.1. State-of-the-art geospatial technologies
The widespread use of digital technologies, combined with rapid sensor
advancements resulted in a paradigm shift in geospatial technologies around the end of
the last millennium (Grejner-Brzezinska et al., 2004a). The major trends are described in
Table 2.1, clearly indicating that the tendency is to use multiple sensor-based direct georeferenced imaging systems to collect data in a highly redundant manner and then apply robust and highly automated methods to extract geospatial information, including proper error characterization. In addition, it is also clear that hardware developments, such as imaging sensor advancements and improving generic computer power, drive spatial information research, with algorithmic progress typically lagging behind system developments. Recent significant software and hardware technological developments are
6 listed in Table 2.2. The developments specific to remote sensing on airborne platforms are listed in Table 2.3.
Traditional State-of-the-art Platform dynamics Static/Kinematic Kinematic Observations Point measurement Image acquisition Imaging systems Analog Digital Georeferencing Indirect Direct Sensor type Passive Passive /Active Sensor(s) Single Multisensor suite Sensor models Physical Physical / Non-physical Image processing Manual Highly automated Navigation solution Post-processed (Near) real-time
Table 2.1: Paradigm shift and conceptual developments in geospatial sciences
Significant hardware developments
Almost unlimited computing power, storage and bandwidth New sensors for navigation-based georeferencing and imaging (2D / 3D) “Sensor web”, mobile devices, wireless communication
Significant software developments Standard operating systems, open source software Networks, the web - “the net is the computer” Standard GUIs, speech recognition, voice synchronization Virtual and augmented reality images everywhere – real and virtual Geo-spatial data infrastructure, GIS spatial databases, interoperability Location based services Web-mapping (Internet giants: Google Earth, Microsoft Virtual Earth, Yahoo)
Table 2.2: Technological developments relevant to geospatial technologies
7
Mobile mapping technology Fully operational GPS Early/mid 90s’ Navigation sensors Mid/late 90s’ for GPS/IMU commercial use High-resolution CCD sensors Late 90s’ Large-format digital cameras Since 2002 Imaging sensors LiDAR Late 90s’ SAR Early 90s’ Wireless communication Since 2000 Explosion of Internet Mid 90s’ Computer technology Mobile computing Since 2000 Unprecedented computer power Since early 90s’ Building recognition Since early 90s’ Image understanding / Scene interpretation Recently computer vision Automation (feature extraction, object Since early 90s’ identification and positioning)
Table 2.3: Milestones of airborne remote sensing developments
In summary, developments in geospatial technologies, in particular in sensing, are
strong, with significant performance improvements expected in the near future that will
result in better spatial and temporal resolution of data. In addition, a higher redundancy provided by simultaneous coverage by different sensors, as well as multiple or repeated observation from identical sensor(s), is anticipated.
2. 1.2. Topographic and infrastructure mapping
Topographic mapping is primarily concerned with obtaining a 3D representation of the surface of the Earth. Historically, airborne analog imagery was the main source of
8 geospatial information, and the information extraction process included: flight planning, establishing ground control for image georeferencing, acquisition of photography, film development, operator based photo evaluation and 3D data extraction on analog opto- mechanical, analytical opto-mechanical-electronic, and later, fully digital visualization and measurement instruments. Obviously, every natural and man-made feature that is visible in the imagery can be extracted, so, over time the number of geospatial products has increased. For example, city modeling, a high-demand and major current application, requires much more than the terrain surface. It also includes building models, road network, or even virtual visualization of the objects in 3D. In the following, the main geospatial products are listed:
Surface models (DEM/DSM), in both grid and contour representations, are the
most widely used baseline product and also represent the foundation for most
derived products.
Orthoimagery is the image product where any surface-related image deformation
has been removed; in other words, an orthoimage has the geometrical
characteristic of a map, it is North oriented, and has a scale, but in addition, has
the same visual appearance. Orthoimage products are formed by mosaicing
orthoimages. They are probably the most widely used mapping products.
Vector mapping refers to the GIS/CAD-type of representation of typically man-
made products, such raod centerline or edge lines, building skeletons, etc.
Classification maps are aimed at providing a direct representation of land use,
including various categories such as urban or rural areas, vegetation types, etc.
9
City models represent the most sophisticated geospatial products, and are
frequently considered as the state-of-the-art in mapping. A complete city model
contains all the information necessary to create a 3D view from practically any
location, such as an orthoimage representation or an oblique view. In addition,
many time it is combined with non-traditional geospatial information, such as
population density data, or typical traffic data, etc.
Obviously, the above discussion is far from being complete, as there are many
applications with their specific geospatial data/information needs. Many times, geospatial
data are categorized by the platform and/or sensor types used to acquire them. For
example, besides the fairly standard airborne and satellite platforms, UAV’s (Unmanned
Aerial Vehicle) are used in growing numbers for surveillance, and thus, there is a distinction for UAV imagery. Similarly, terrain data can be obtained by optical and laser sensors, so there is a distinction between photogrammetrically (or stereoscopiscally) obtained and LiDAR surface data. This distinction is logical, as it reflects the characteristics of the different sensors. For example, photogrammetrically-derived data have better horizontal and weaker vertical accuracy terms, which is just the opposite to
LiDAR point cloud, where the range data is significantly more accurate than the horizontal component.
2. 1.3. Emerging applications: emergency/rapid mapping, traffic flow extraction
10
The need for accurate and current geospatial data has grown tremendously in recent years. The best example is probably the skyrocketing use of location-based services (LBS) whose development is clearly due to the recent availability of inexpensive
2D/3D data (Kolodziej and Hjelm, 2006). As described earlier, the rapid developments in geospatial technologies led to a total paradigm shift in mapping, and resulted in more efficient map production, which ultimately stimulated the increasing use of geospatial data in several applications, including un-conventional uses. In addition, recent natural and human-induced disasters, such as hurricanes, earthquakes and tsunamis, or terrorist attacks, directed significant attention to near real-time mapping capabilities, as the availability of current geospatial data clearly is essential to support emergency operations. In particular, as a result of government and private sector cooperation, plans were created after 9/11 to build quickly deployable airborne sensor systems with ground- based processing capacity to meet the requirements of emergency mapping. The most notable first experiment was the ARIES project organized by EarthData Group with assistance of several government agencies (Schuckman and Hoffman, 2005). Despite these efforts, the mapping operations after Hurricanes Katharina and Rita in New Orleans were disappointing, and thus, more extended programs were undertaken to improve preparedness for future emergencies. More recently the Sechuan earthquake in China proved that the immediate availability of geospatial data not only can support rescue operations, but is also indispensible to prepare preventive steps needed to avoid further loss of human life, threatened by landslides or collapse of damaged dams (Shao and
Scheuchl , 2008).
11
The demand for real-time or near-real time mapping, needed for emergency, is clearly in sharp contrast to past map production practice, when the delivery time was measured in months at best. However, the paradigm shift in geospatial science, described earlier, clearly provides the technological foundation to move toward a significantly faster production and, in fact, has just recently approached near real-time performance. In other terms, the totally digital system design combined with networked sensors and systems form the basis, and the remaining task is the algorithmic developments and the software implementation. From the last two tasks, undoubtedly the first one is the really difficult one. Despite many years of research effort to create “the ultimate map machine”, progress is still limited to some specific areas and significant research is still needed to achieve near real-time overall performance in airborne mapping. For example, mapping flat or smoothly rolling terrain in rural areas can be almost totally automated, while city mapping of densely built-up urban areas, including terrain and building extraction, at present poses practically unsurpassable difficulties in terms a full automation. This statement is definitely valid for optical imagery, and applies to an extent to laser data, too. In the following, the challenging aspects of city mapping are analyzed:
Object scene complexity. Dense urban areas represent the highest object space
complexity for mapping. This environment is rich in man-made objects of various
sizes and shapes. Although man-made objects tend to have shapes of simple
geometry, such as straight lines, planes or cylindrical surfaces, the additional
details of the structures makes the extraction and proper description of these
objects rather difficult. In addition, the brightness conditions vary over a large
12
range, from sunlit to dark shadow areas, and the large variations in image
radiometry cause numerous problems in the processing, such as image matching
(which is very sensitive to changing light conditions). Vegetation is always a
problem for mapping, although the canopy thickness is less frequent in urban
areas.
Occlusions. Densely packed buildings in urban areas typically introduce a lot of
occlusions. In addition, the reliable mapping of the ground surface level in urban
canyons even with a linescanner camera is nearly impossible. The main problem
with occlusions and limited ground level image coverage is that most of the
optical image-based 3D location extraction is based on using overlap imagery, at
minimum stereo coverage, and maintaining this condition is nearly impossible in
dense urban areas.
Moving objects. Traditional mapping is concerned with the static part of the
object space, including terrain surface and man-made objects, such as building
and roads. Urban areas, however, represent rather dynamic environments, as
moving vehicles and people form a relatively large part of the object space. The
identification of moving objects requires either direct velocity sensing capability,
such as RADAR (RAdio Detection And Ranging), or sensor data from where
velocity can be estimated. Imaging sensors provide rich information, and multiple
images captured in short time over the same area can provide for velocity
estimates. Once moving objects are identified, they can be ignored or removed
from the data and conventional mapping products can be derived.
13
These three challenging aspects are related to each other in a sense that acquiring better spatial and temporal resolution sensory data can help deal with these difficulties better. Note that the various spatial extraction techniques used to mitigate the impact of the three difficulties have several common algorithmic components. Nevertheless, the last item, the separation of the static and moving objects is generally viewed as the relatively least complicated one. Since the great majority of the moving objects are vehicles that should be identified and removed, as the objective of the mapping process is the description of the static object space, instead of discarding these objects they can be turned into valuable traffic flow information. This is one of the overarching ideas of this dissertation.
2.2. Innovative use of remote sensing in transportation
2.2.1 Motivation for vehicle extraction: the mapping perspective
Most applications that use remotely sensed data require the separation of static and non-static components of the object space. In general terms, topographic mapping is primarily concerned with the mapping of the static part of the world, including the surface of the earth and natural and man-made objects. Non-static parts pose difficulties for the traditional mapping process. For example, moving vehicles can significantly decrease the surface extraction performance of stereo image-based methods. However, as general mapping technology continues to advance and temporal resolution of geospatial data improves, the extraction of non-static components is becoming increasingly feasible: objects can be excluded from mapping or can be captured by their
14 dynamics to describe the objects not only in space, but in time, too. In other words, instead of being a noise for the mapping process, moving objects are now part of 4D data acquisition and information extraction.
The research on how to extract static and non-static objects from remotely sensed data was previously prompted by robotics applications, where collision avoidance and close-range vehicle navigation required the timely and accurate reconstruction of the object space around the vehicle, including the dynamics of the object space. Sensor performance in the past imposed limitations on observability; for example, digital cameras provided moderate resolution and noisy imagery compared to current systems or there were no laser scanners available. A good initial discussion on the topic can be found in (Saufel, 2001).
Airborne mapping case is both similar and very different from the terrestrial case. The object distance is larger, and the spatial and temporal resolution of the sensor data is lower compared to the terrestrial case. Static and non-static components of the object space are of great importance, and thus, should be considered with respect to the sensor motion. Since the aircraft motion is typically simple, mostly including straight flight lines, the relative position of the sensor platform and object space is less complex for the airborne case, as the trajectory cannot change dramatically in a short time. The effect of the platform motion and moving objects, however, varies by the type of remote sensory data. Frame model-based cameras take snapshots of the object space as the exposure time is very short, and thus, the motion of both the platform and objects can be ignored. In contrast, scanning sensors, such as line-scanners, LiDAR, RADAR, are
15 continuously collecting data, and therefore, objects that are moving with respect to the sensor platform will suffer from motion artifacts in the data. This phenomenon can be exploited, as detecting the amount of motion artifacts can provide information about the dynamics of the object scene.
2.2.2 The growing need for traffic data to support transportation management
“Transportation has a high importance in our lives. In the United States, about 20 percent of Gross National Product (GNP) is spent on transportation, of which about 85 percent is used on highway transportation (passenger and freight). The population owns and operates 150 million automobiles and an additional 50 million trucks, bringing car ownership to 56 per hundred population (highest in the world). These vehicles are driven an average of 10,000 miles per year for passenger cars and 50,000 miles per year for trucks on a highway system that comprises more than 4 million miles. The indices in other countries may be somewhat different, but the importance of the transportation system, and especially the highway component of it, is just the same or even greater.
While car ownership in some countries may be lower, the available highway network is also smaller leading to similar or more severe congestion problems” (Gartner et al.,
1997).
As the quote above states, transportation represents a major pillar of any developed economy. Road traffic, measured in various parameters of passenger and cargo volume, has been steadily increasing worldwide for several decades. For example, the total vehicle-miles traveled in a year, the most frequently quoted traffic parameter
16
that includes traveled distances for all vehicle categories, has almost doubled in the US
in the last 15 years (US Bureau of Transportation Statistics, http://www.bts.gov).
Although recent data indicate that for the first time the trend is changing, as lower traffic
numbers are predicted for 2008 due to higher gas prices and financial problems
worldwide (USA Today, 11/20, 2008). Another important fact is that the continuous
increase of traffic is much faster than the population growth in the US, which reached
about 20% during the same 15-year period. Unfortunately (from a transportation
perspective), the paved road network grew only a modest 10% for the same period, and
thus the increasing traffic on the practically stagnating road infrastructure results in more
severe traffic congestion.( http://www.bts.gov/).
Federal and local government transportation management services monitor and
control the traffic over the urban road network and the nation’s highway system. These
agencies collect data for both long-term planning and real-time traffic control. Real-time
information is usually gathered from many sources, such as electronic sensors in the
pavement (loop detectors), road tubes, ramp meter sensors, and video and digital
cameras, and sent to traffic management centers at various times. Commonly, density and flow of traffic are the two main parameters for describing the traffic stream.
Namely, the density is the number of vehicles occupying a road lane per unit length at a
given time, while traffic flow represents the number of vehicles travelling over a road
segment in a given time period.
With the increasing number of vehicles entering the existing transportation
network annually, the effectiveness of traffic management is becoming more crucial,
17 simply because new road construction does not keep up with the growth of the traffic volume. The key to better traffic management is access to better and more complete data and, of course, the capability for immediate processing of these data to provide a real- time response. Hence the interest in the new sensors that can provide large volumes of data in (near) real-time is steadily growing. Currently, loop detectors produce the largest amount of traffic data (Burns and Wendt, 2003). Although, this sensor technology is well established, the installation and maintenance are not simple, and the associated cost can be also high. Recently, more and more imaging-based systems have been introduced. These pole- or bridge-mounted cameras and ranging sensors have shown significant performance improvements. The trend of switching toward imaging technologies is expected to increase.
2.2.3. Traffic flow extraction from remotely sensed data
Transportation is a broad subject and involves several disciplines with their own terminology and methods. For highway planning and traffic management purposes, each road segment is characterized by its traffic flow (Mannering et al., 2009), which is generally defined as the number of vehicles passing a given point on a highway per unit of time, such as the number of vehicles per hour. Here our interest is limited to traffic flow and only a short review of basic terms is provided below. Flow, q, given by equation
(2.1), is defined as the number of vehicles, n, passing a given point on a highway over a time interval, t. Flow is generally expressed by the number of vehicles per unit time.
n q (2.1) t
18
The average traffic speed can be defined in two ways. The first is the time mean
speed,ut , expressed in equation (2.2), is the arithmetic mean of speeds observed at one point.
1 n ut ui (2.2) n i1
The second is the space mean speed, u, which is defined on the basis of the time necessary for a vehicle to traverse some known length of roadway, l. Space mean speed is computed as:
1 n l n i u i1 (2.3) t
where t is the average of time intervals, t1...tn , that are necessary for vehicles i 1...n to
traverse a section of roadway of length, l1...ln . Finally, traffic density, k, refers to the number of vehicles, n, occupying a length of roadway, l, at a specified time,
n k (2.4) l
The basic relationship of the three elements, traffic flow, speed (space mean speed) and density is given in equation (2.5); the flow is the product of speed and density.
q uk (2.5)
The units are typically expressed in units of vehicles per hour, miles per hour and vehicles per mile, respectively, or vehicles per hour, kilometers per hour and vehicles per kilometer, respectively.
19
The two basic types of mathematical models for describing traffic flow are the macroscopic and microscopic models. While macroscopic models are concerned with describing the flow-density relationship for a group of vehicles, microscopic models describe the flow by tracking individual vehicles.
Two of the most important traffic measures produced by state departments of transportation and other transportation agencies around the world are AADT and
VMT/VKT (Pline, 1992). Average annual daily traffic (AADT) is produced to represent the vehicle flow over a highway segment on an average day of the year. Vehicle miles/kilometers traveled (VMT/VKT) aggregates travel over the entire highway system and is used to indicate mobility patterns and travel trends. VMT/VKT is also used as an indicator for allocation of highway resources. VMT/VKT is typically estimated from
AADT and length of highway segments; AADT is generally obtained from data collected with ground-based equipment, such as loop detectors, which are fixed to a location, or road tubes, which are portable and can be deployed at different locations as needed.
2.2.4. Ongoing research, background, literature review, concept
Just as transportation is a broad subject, transportation research contains a broad spectrum of disciplines; see, for example, (http://www.trb.org). Here our research effort is focused on improving transportation infrastructure mapping and investigating the feasibility of traffic flow extraction from remotely sensed data. Mapping transportation infrastructure is much more developed than extracting traffic flow from remotely sensed
20 data and is already widely used in applications. Therefore, only the research related to the traffic flow extraction is reviewed below.
Research efforts in exploiting remotely sensed data from various platforms, ranging from ground to airborne and spaceborne systems, for extracting traffic flow data is a fairly recent topic. Although there was some work before, the US Department of
Transportation (DOT) and the National Aeronautics and Space Administration (NASA) joined to support a concentrated research program in this area (http://www.ncrst.org). In the context of the multi-tier nationwide effort, airborne and satellite optical imagery,
LiDAR, airborne and spaceborne radar data were studied, and new methods were developed and subsequently evaluated for emerging application fields in transportation.
The European Union followed suit and under the leadership of DLR (German Aerospace
Center), a similar program was established. At the local level, a good example is the
Transportation Application of Restricted Use Technology (TARUT) study, initiated by the Michigan Department of Transportation (MDOT), that addressed the use of high- resolution remote sensing systems and geospatial analysis (http://www.tarut.org/). It should be noted that there are a few transportation applications that already use airborne remote sensing technology to serve specific transportation fields, see, for example,
(http://skycomp.com/).
One way of characterizing these research efforts is by the data acquisition platform and the imaging sensor type. The platforms considered are (1) spaceborne, (2) airborne, including manned and unmanned aircraft both fixed-winged and helicopter, and
(3) deployable ground sensor assemblies, such as pole-mounted cameras and mobile
21 mapping systems. The platforms provide a wide range of object distances for the imaging sensors. In addition, the captured imagery differs greatly in temporal resolution, depending on the platform motion. On the imaging sensor side, optical imaging in the visible range is the most widely used sensor on all the platforms, offering both panchromatic and multi-spectral imagery. Thermal (IR) and hyperspectral sensors (HSI) are of growing interest, although their availability is still limited. Active sensors, such as
LiDAR and SAR technologies have recently shown significant expansion and are now available on spaceborne and airborne platforms.
Optical imagery is the primary source of geospatial information acquired from airborne platforms. More than 90% of all the airborne data collected was from optical imagery until the late 1990’s. With the introduction of active sensors, such as LiDAR and radar, the percentage changed in the past decade; yet optical imagery is still the main source of geospatial data. The benefit of using optical data from both the visible (Shastry and Schowengerdt, 2002; Stilla et al., 2004; Toth and Grejner-Brzezinska, 2004a;
Reinartz et al., 2006) and thermal infrared (Ernst et al., 2003; Hinz and Stilla, 2006;
Kirchhof and Stilla, 2006) part of the spectrum for vehicle detection has been extensively studied, and various methods for the detection of moving objects have been developed.
Several methods are based on the computation of the entire optical flow, such as the methods discussed in (Haag and Nagel, 1999; Ogale et al., 2005; Woelk and Koch, 2004;
Nejadas et al., 2006). Optical flow techniques mostly rely on the constraint that objects in the scene have a temporally constant gray value. The methods for vehicle detection from airborne optical imagery based on knowledge- and structure-based techniques are
22 presented by Zhao and Nevatia (2001), who describe a Bayesian network algorithm to detect the vehicles from low-resolution aerial imagery (knowledge-based method), and by Hinz (2003), who introduces a feature detection algorithm for use with high-resolution imagery (structure-based model). Pantavungkour and Shibasaki (2002) discuss an vehicle extraction algorithm for feature detection of urban road surface by using three-line scanner imagery. Work on monitoring traffic flow from aerial video is discussed in
(Mirchandani et al., 2003). A general observation is that the availability of accurate georeferencing of the imagery has not been fully exploited in the majority of the methods that have been developed. The use of satellite imagery for traffic monitoring purposes is discussed by Merry et al. (1999). Satellite imagery allows wide spatial coverage, usually unavailable from ground-based and aerial sensors. Continuous observation of traffic volume based on satellite data, however, is impossible due to its limited temporal resolution. Nevertheless, the large area coverage of satellite imagery can effectively support local flow measurement in some cases. For instance, McCord et al. (2002) proposes a method that combines traffic information in satellite imagery and ground- based data to improve AADT estimates.
Several theoretical and practical studies were carried out on the feasibility of using airborne LiDAR data, collected over transportation corridors for estimation of traffic flow parameters (Toth and Brzezinska, 2006). Various road outline extraction and vehicle modeling techniques were developed (Toth et al., 2004b; Moafipoor et al., 2005); in addition, different parameterization and classification techniques were studied (Toth et al., 2003a). The preliminary results of using LiDAR for vehicle classification and
23 velocity estimates are presented in (Grejner-Brzezinska and Toth, 2003a; Toth et al.,
2003b).
SAR Along-Track Interferometry (ATI) is sensitive to ground motion and was recently proposed for traffic monitoring applications. Until now, Shuttle Radar
Topography Mission (SRTM) is the only spaceborne SAR system that provided data containing ATI signatures of moving objects. This makes it a unique data source for
Ground Moving Target Indication (GMTI) research. Breit et al. (2003) demonstrated the possibility of detecting a moving car in SRTM/X-SAR data for the first time. GMTI research and development are currently heavily based on the experience gathered from airborne SAR (ATI) campaigns. Currently, two operational multi-channel spaceborne
SAR systems are available for civilian purposes, which are the German TerraSAR-X and the Canadian Radarsat-2 satellites, launched in 2007, and both are capable of along-track multi-channel modes (Meyer and Hinz, 2004; Chiu and Gierull, 2006; Runge et al., 2006;
Meyer et al., 2006). Radar sensors mounted on the German TerraSAR-X satellite allow for monitoring traffic situations worldwide and independent of weather and illumination
(day and night operation). However, severe problems are expected in the inner city areas due to the inherent side-looking geometry of radar.
No significant data acquisition platform developments are expected in the near future. UAVs (Unmanned Aerial Vehicles) have shown a remarkable progress in recent years. They represent a mature technology, but institutional difficulties appear to be limiting the wide acceptance of UAVs in the foreseeable feature. Most of UAV related research is focused on higher altitude deployable systems, as this technology combines
24 the advantages of satellites and aircraft for traffic monitoring. These High Altitude Long
Endurance (HALE) platforms include the ability to provide superior remote sensing performance as compared to satellite systems for a wide range of civil applications involving time-varying or emerging phenomena. These platforms are more flexible than satellites and allow permanent observation of large areas on a regional scale
(http://ww3.psepc-sppcc.gc.ca/research/resactivites/CI/1999-D002_e.asp).
25
CHAPTER 3
SENSOR CHARACTERISTICS
In order to assess the feasibility of extracting static and dynamic data from modern mapping imaging sensors installed in airborne platforms, the main characteristics of the sensors, including the image formation method and data composition are considered first. Next, the implication of using integrated GPS/INS systems, as an enabling technology, which makes direct georeferencing feasible, is discussed. Finally, the performance analysis is focused on the specifics of directly oriented imagery.
3.1. Sensor technology developments, enabling sensors
The overall sensor developments that triggered the paradigm shift in geospatial sciences were described earlier, here the focus is on the technological developments that are essential to achieve high sensor performance. The introduction of a worldwide all- weather, anytime, absolute satellite-based radionavigation Global Positing System (GPS) in the late 80’s was probably the most significant milestone in modern remote sensing, as it enabled for a unified referencing on the ground, in air and space (Hofmann-Wellenhof et al., 2001). By the mid 90’s, the GPS satellite constellation was fully operational, and by now, the technology became a utility, which has penetrated practically all applications
26 that are related to mobility. Combined with inertial sensors, integrated GPS/INS systems can provide accurate platform georeferencing in air, space and on the ground. This enabling technology has made airborne surveying more economic by reducing the need for aerial triangulation and allowed for the introduction of new technologies, such as
LiDAR, IfSAR and hyperspectral imagery that require accurate sensor georeferencing; in fact, without it, these relatively new remote sensing technologies would not be feasible at all.
Besides the imaging sensor technology developments, generic systems developments, including embedded systems, micro-electro-mechanical systems, storage technologies, etc., played an essential role as enabling technologies in implementing the current generation of state-of-the-art remote sensing systems (Schowengerdt, 1997). For example, laser systems include high-precision moving parts, such as oscillating or rotating mirrors, encoders; powerful laser supply; high-precision optics; and supporting electronics that is capable to measure at ten picosecond range. IfSAR systems have no moving part as the antenna is synthesized from small elements under total electronic control and these systems are extremely complex with respect to mainstream mapping technologies. Modern hyperspectral sensors typically have no moving parts and use very sophisticated optics and supporting electronics.
3.2. Imaging sensors
The term of imaging sensor primarily refers to a sensor that acquires optical imagery, and was introduced when the first cameras with semiconductor sensors became available, and it simply indicated a non-film-based camera system and emphasized the
27
fact of the digital image acquisition. Later in mapping, this term was generalized to cover
any sensor that remotely acquires an image of the object space, including non-optical
sensors, such as LiDAR and IfSAR. In the following, the major imaging sensor groups
will be reviewed with the emphasis on the transportation applications’ requirements.
3.2.1. Optical imagery: panchromatic and multispectral sensors
High-performance digital aerial camera systems, meeting the requirements of
large-scale topographic mapping as well as providing quality thematic and visual information have been only used for about five years. Despite this short period, there are already nearly 200 systems installed worldwide and developments are still strong, as
existing systems are constantly improved and new systems are introduced (VEXCEL
Users Group meeting, ASPRS conference, Portland OR, 2008). In fact, the number of
camera systems has increased, and more importantly there is a growing number of one-
of-a-kind or custom-built systems.
The introduction of metric quality digital aerial cameras at the beginning of the
new millennium completed the decade-long transition process of moving from analog to
a totally digital technology in airborne surveying (Chapter 8.2.1, Manual of
Photogrammetry, 2004). To approach and to eventually surpass the high performance of
analog cameras was a difficult task, as these aerial cameras were absolutely perfected
masterpieces of their class. The fundamental difference between analog and digital
cameras is that a solid-state sensor, rigidly installed in the camera focal plane, replaces the film. Digital camera systems, dedicated to high performance topographic mapping have experienced significant advancements in recent years. While the introduction of the
28 first large format airborne digital photogrammetric cameras took a longer time due to technological difficulties and lack of experiences in that field of the main aerial camera manufacturers in the early 2000’s, the developments of both commercial and custom- made airborne digital camera systems are currently flourishing worldwide, fueled by high demand for imagery to support spatial data acquisition applications and by rapidly advancing technology. Improving CCD (Charge-Coupled Device) and CMOS
(Complementary Metal–Oxide–Semiconductor) imaging sensor characteristics as well as rapidly growing computer power, combined with decreasing hardware cost makes the development of custom-designed systems feasible.
The introduction of digital imaging sensors dramatically changed the former single-camera configuration model and allowed for both multiple camera head configurations as well as non nadir looking camera orientation on airborne platforms. In fact, the size limitation of the CCD and CMOS sensors with respect to the equivalent focal plane image size of the analog camera necessitated the innovative use of multiple imaging sensors to provide the camera system with nearly comparable ground coverage of that of the analog camera. Except for the line sensor-based high-performance airborne digital cameras, all the other camera systems are based on integrating several individual cameras to achieve the required ground coverage at the required image resolution. As a result of the described camera integration technology, the stage was set to build systems to collect non vertical aerial imagery, frequently called oblique imagery. Oblique camera systems are typically custom-made proprietary systems; notably Pictometry is the largest oblique image data provider, having 100+ complete systems used worldwide (2008).
29
Large-format, multihead, frame cameras Image CCD Number Pixel Dynamic Maximum Field of System Size Sensor Size of Size Range Frame Rate View GPS/IMU Software [pixel] [pixel] Sensors [micron] [bits] [image/sec] (FOV) DMC 7,000 x 4,000 Any system Digital 13,824 (pan) Optional (frame Mapping x 4 + 4 12 12 2.1 x 42 3,000 x 2,000 Integrated camera Camera 7,680 (multispectral) model) Intergraph 14,430 x Any system UltraCam 9,420 (pan) Optional (frame X 4,008 x 3,680 x 2,400 9 + 4 7.2 14 1 55 x 37 Integrated camera Vexcel 2,672 model) (MS) 17,310 x 11,310 Any system UltraCam (pan) Optional (frame XP 5,770 x 5,570 * 3,770 9 + 4 6 14 2 55 x 37 Integrated camera Vexcel 3770 model) (RGB & NIR Any system DiMAC 34 x 26 10,500 x Optional (frame DIMAC 7,216 x 5,412 2-4 6.8 16 2.1 or 7,200 Integrated camera Systems 66 x 48 model) 60/72/100 mm lens Any system RolleiMetri 13,000 x Optional (frame 7,228 x 5,428 4 6.8 16 3 80 x 65 c AIC x4 10,000 Integrated camera 70 x 45 model) 50 x 30 Large-format, linescanner cameras Number Pixel Dynamic Maximum Field of Image CCD System of Size Range Frame Rate View GPS/IMU Software Size Sensor Size Sensors [micron] [bits] [image/sec] (FOV) ADS40 GPro, Airborne ORIMA, Digital 12,000 SocetSet, 6.5 Mandatory Sensor x 12,000 (2x) 3 (2x) + 4 14 n/a 64 Virtuozo, (3.25) Integrated Leica any KLT Atlas. GeoSystem DIGI3, s ImageStation ADS80 Leica 12,000 x Mandatory As for 12,000 3 + 5 6.4 12 n/a 64 GeoSystem any Integrated ADS40 s JAS150 12,000 (HRSC) Mandatory JenaStereo, x 12,000 5 + 4 6.5 16 n/a 30 Jena- Integrated SocetSet any Optronik 3-DAS-1 8,002 x Mandatory Wehrli 8,002 3 (x3) 9 14 n/a 36 Proprietary any Integrated Associates SI-250 14,400 x 17, 23, Mandatory Starlabo 14,400 10 5 9 n/a Proprietary any Integrated Startimager 40 Continued.
30
Medium-format, multihead, frame cameras Number Pixel Dynamic Maximum Field of Image CCD System of Size Range Frame Rate View GPS/IMU Software Size Sensor Size Sensors [micron] [bits] [image/sec] (FOV) SpectraVie Any system w8 8,000x 8,000x 2,672 Optional (frame Airborne 2 + 4 9 12 n/a 64 2,672 4,000x 2,672 Integrated camera Data model) Systems Medium-format, singlehead, frame cameras Number Pixel Dynamic Maximum Field of Image CCD System of Size Range Frame Rate View GPS/IMU Software Size Sensor Size Sensors [micron] [bits] [image/sec] (FOV) 40/60 mm Any system DSS 5,436 x lens (frame 5,436 x 4,092 1 9 12 2.5 Built-in Applanix 4,092 62 x 49 camera 44 x 34 model) 35/40/80 5,440 x mm lens Any system DigiCAM 4,080 5,440 x 4,080 9 Optional (frame 1 16 2.5 69 x 55 IGI mbH 7,216 x 7,216 x 5,428 6.8 Integrated camera 52 x 40 5,428 model) 33 x 25 50/80/120 5,440 x Any system mm lens AIC 4,080 5,440 x 4,080 1.7 Optional (frame 1 9 16 69 x 55 Rollei 7,228 x 7,228 x 5,428 1.9 Integrated camera 6.8 52 x 40 5,428 model) 23 17 50/90 mm Any system NexVue 4,080 x lens Optional (frame Spectrum 4,080 x 4,080 1 9 12 2.5 4,080 23 x 23 Integrated camera Imaging 42 x 42 model) 35/60/100 mm lens RCD105 69.7 x Any system Leica 7,162 x Optional (frame 7,162 x 5,389 1 6.8 12 0.49 55.3 GeoSystem 5,389 Integrated camera 44.2 x 34 s model) 27.4 x 20.8 Any system 45 RMK D 6096 x Optional (frame 6096 x 6500 1 7.2 14 1 mm lens Intergraph 6500 Integrated camera 52 x 55 model)
Table 3.1: Digital camera categories and major camera systems
Notes:
31
Image Size – defines the size of the output image of the digital camera system in
pixels. The output image may have the size of the imaging sensor for a singlehead
camera system, or it could be the size of a virtual image composed from several
images captured by different cameras in a multihead camera system.
CCD Sensor Size – provides the size of the imaging sensor in pixels. There are a
few varieties of the sensor size, such as the total number of pixels, the number of
effective pixels, and finally the number of active pixels. Obviously, the last one is
the most important for the user. Charge Coupled Device (CCD) is one of the two
main imaging sensor types used in digital cameras and is the equivalent of film in
digital cameras. Implemented in silicon semiconductor, the individual CCD sensor
elements, pixels, convert the light during exposure into electric charge that is stored
and then read out in a subsequent process.
The Number of Sensors – provides the number of the CCDs used in a digital camera
system. There could be several sensors in a single camera head, such as several
linear CCDs in a linescanner or several area CCDs in beam splitter-based
multispectral cameras. For multihead cameras, this number is typically equivalent
with the number of cameras used in the system.
Pixel Size – defines the physical size of a CCD sensor element. The smaller the size
the more pixels can be placed on a CCD sensor but as the size decreases the noise
level increases as well as diffraction becomes a setback.
Dynamic Range – defines the intensity resolution of the output image in bits. The
number of bits of the D/A converter connected to the CCD sensor could be different
32
from the actual intensity resolution. In addition, imaging systems typically work
with 8 or 16 bit image intensity data (per band). For example, a digital camera may
use a 14 bit converter and can provide the output data in as a 16 bit number while
the actual dynamic range of the image is the typical 10-12 bits. In comparison,
scanned analog film has about a 6-8 bit dynamic range.
Maximum Frame Rate – defines the maximum frequency images can be captured
by a system. In practice, however, the inverse of this parameter is used in mapping,
which simply states the minimum time needed between two image captures.
All the major large-format cameras create vertical imagery, although there is a growing number of oblique camera systems, which typically acquire vertical imagery too; a list of the current major systems is included in Table 3.1, which is based on http://www.eijournal.com/images/articles/chart_digitalcameras.pdf).
The main advantages of the state-of-the-art digital large-format frame cameras over film-based analog cameras include:
An inherent geometric stability because of the absence of moving elements and of
photo laboratory processes. The camera sensor model, called interior orientation has
excellent long-term stability (Chapter 8.2.1, Manual of Photogrammetry, 2004).
An important advantage of the digital camera systems over analog cameras is that
they offer a much better radiometric resolution, and consequently, the improved
intensity data can better support most of the automated processes. The dynamic
range is typically 12-16 bits per pixel with about 10-12 bits information contents,
33
which is in sharp contrast to the 6-7 bits information normally obtained with
scanned film imagery. Perko and Gruber (2002) have shown that the performance
of stereo matching could be by a factor of 2.5 better with a digitally sensed image
pair than with scanned film. This is primarily due to the increased radiometric
resolution combined with higher SNR (Gruber et al., 2003).
Additional advantage of digital sensors is the absence of grain noise of the film.
Higher image acquisition rate of the digital sensors allows for more redundant data
at virtually no additional cost (e.g., the UltraCamX from Vexcel can easily maintain
1 frame/second image capture rate). Therefore, the routine use of 70% - 90%
forward overlap, instead of the traditional 60% forward overlap, and furthermore, a
multi-ray photogrammetric processing are feasible. The number of images is no
longer a factor of the project costs, as long as the processing is automated.
The digital camera systems offer the feasibility for real-time applications, as images
can be instantly displayed and processed provided adequate computer power is
available on the data acquisition platform.
Digital imaging sensors provide improved capabilities for acquiring multi-spectral
imagery compared to analog film, as spectral bands can be individually imaged at a
variety of spatial resolution.
Although, it possible to build camera systems with multiple views based on analog
cameras, in practice, it is only feasible with digital cameras, which are smaller in
size and can be easily synchronized and thus can offer multiple viewing angles with
simultaneous image acquisition.
34
From the point of view of extracting transportation network infrastructure and
traffic flow data, the following aspects, in particular, are relevant:
Shorter exposure times translate into less motion blur, and thus less image noise and
artifacts that could burden the feature extraction processes.
Better acquisition rate means better sampling of moving vehicles, which results in a
better, more continuous velocity approximation.
The better radiometric resolution improves in general the feature extraction
performance.
Multiple image coverage improves overall robustness of the processing.
Images acquired by digital cameras are typically modeled by the well-known
collinearity equations, which provides a connection between image and object space
(ground) coordinates. The projection is shown in Fig. 3.1, and the equations (3.1) and
(3.2) are derived accordingly.
z y x vi
Z V0
Y Vi
V P
X
Figure 3.1: Projection between object and image space; see notation below
35
vi s M Vi s M (V V0 ) (3.1) where, s is the scale, M is the rotation matrix between camera and mapping frames, V, V0 and Vi are vector to object point, camera center and the difference, respectively.
which can be extended in to the following two equations:
m11,i (X j X 0i ) m12,i (Y j Y0i ) m13,i (Z j Z 0i ) xij f i m31,i (X j X 0i ) m32,i (Y j Y0i ) m33,i (Z j Z 0i ) c c m21,i (X j X 0i ) m22,i (Y j Yi ) m23,i (Z j Z i ) yij f i (3.2) m31,i (X j X 0i ) m32,i (Y j Y0i ) m33,i (Z j Z 0i )
lens refraction lens refraction xij xij xP dxij dxij , yij yij yP dyij dyij
where the parameters are
xij, yij Raw image coordinates, point Pj in image Ii
x’ij, y’ij Corrected image coordinates, point Pj in image Ii
Interior orientation of image (camera) Ii:
fi,x0i,y0i Focal length, principal point coordinates lens lens dxij ,dyij Lens distortion refraction refraction dxij ,dyij Atmospheric distortion
Exterior orientation of image Ii:
X0i, Y0i, Z0i Projection center coordinates
m11,i, …,m33,i Rotation matrix components
Xj,Yj,Zj Object coordinates of point Pj
36
The transformation parameters can be divided into two groups. The parameters in the first one are specific to the sensor’s internal behavior, such as the projection from object space to image domain for a camera, and these parameters, jointly called interior orientation (IO), in general, should not change for subsequent image captures. The parameters in the second group are concerned with the georeferencing of the sensor, basically, the position and attitude of the camera in a mapping frame, and therefore, it is called the exterior orientation (EO). Note that other geometrical models are also available, RPC (Rational Polynomial Coefficients) is the probably the most notable one, which is frequently used as a generic model for satellite imagery.
For any single image, the ground to image transformation always works but the inverse transformation is not. For conventional overlapped imagery (stereo), both forward and backward transformation exists. Multiple image overlap can improve robustness and accuracy to a certain point (Csanyi May, 2008).
The exterior orientation can be established by both basic ways: direct and indirect orientation; the direct orientation will be discussed at detail in a subsequent section. The indirect method refers to the fact that some object space information is used to estimate the transformation parameters. Typically, ground control points are measured in image(s) and then using the collinearity or another model, the transformation parameters are adjusted in an iterative least squares solution. Instead of points, other geometrical primitives such as linear features can be used too. Regardless of the features selected, the extraction and matching of features are the difficult process of determining the transformation parameters. The topic of extracting features from images belongs to the computer vision discipline and there is no attempt to address this subject in this work,
37 except using existing methods or assuming the availability of adequate features extraction techniques as well as certain performance levels of the methods.
In summary, digital camera developments are currently very strong and significant performance improvements are expected in the near future that will further support the application of remote sensing in transportation, including road network infrastructure mapping and traffic flow extraction, on the data acquisition level.
Similarly, advancements in computer vision are likely to provide the required capabilities for highly automated information extraction processes that is essential to achieve real- time or near real-time traffic management.
3.2.2. LiDAR sensors
Since its invention in the 1960s, laser technology has made remarkable progress and proliferated to a broad range of applications. Airborne LiDAR is probably the most significant technology introduced in mainstream topographic mapping in the last decade
(Shan and Toth, 2008). The main advantage of the technique is that it provides a method for direct 3D data collection. Furthermore, it is highly accurate because of the mm- and cm-level laser ranging accuracy and precise sensor platform orientation supported by an integrated positioning and orientation system (POS). Unlike traditional image-based photogrammetric methods, LiDAR directly collects an accurately geo-referenced set of dense point clouds, which can be almost directly used in basic applications. Obviously, the full exploitation of LiDAR’s potentials and capabilities presents challenges and requires new data processing methods that are fundamentally different from those used in traditional photogrammetry. Over the last decade, there have been many significant
38
developments in this field, mainly resulting from multi-disciplinary research, including computer vision, computer graphics, electrical engineering, and photogrammetry.
Figure 3.2: Main components of LiDAR systems
Airborne LiDAR technology utilizes integrated GPS/INS systems and laser-range
finders to determine surface coordinates of objects resulting in a point cloud, see Fig. 3.2.
To obtain two-dimensional ground coverage, the platform motion typically provides one
dimension, while the across-track dimension is achieved by using moving optics, such as
oscillating or nutating mirrors or rotating polygons that measure a profile. LiDAR uses
pulses of laser light directed toward the ground and measures the time of pulse return.
From the traveling time of the laser pulse, the distance between the sensor and the ground
object from where the laser pulse was reflected is determined. Note that LiDAR is an
39
active sensor and thus offers day/night surveying capability. The geodetic coordinates of a ground point are computed from the geodetic position of the aircraft/sensor, known from the integrated GPS/INS solution, the scan angle of the emitted laser beam and the distance (range) between the sensor and the ground point as described in equation (3.3).
0 r r R M (R INS sin d b ) (3.3) M M ,INS INS L L INS cos
where
rM 3D coordinates of the laser point in the mapping frame
3D INS coordinates (origin) in the mapping frame, provided by
rM ,INS GPS/INS (the navigation solution typically refers to the origin of the INS body frame)
M RINS Rotation matrix between the INS body and mapping frame
INS RL Boresight matrix between the laser frame and INS body frame Range measurement (distance from laser sensor reference point to d L object point) Boresight offset vector (vector between laser sensor reference bINS point and the origin of INS) defined in the INS body frame
Scan angle defined in the laser sensor frame (xL is flight direction, yL to the right, and zL goes down)
Based on equation (3.3), the laser point accuracy in a mapping frame can be
determined by applying the law of error propagation. Earlier discussions on the errors and
40 model parameter recovery can be found in (Schenk, 2001; Filin, 2003a and 2003b;
Baltsavias, 1999) and a recent comprehensive discussion on the various terms of the error budget, including analytical and simulation results can be found in (Csanyi May, 2007).
The difficulty of applying the theory in practical cases, however, is the time dependency of some of the parameters (note that several parameters are modeled as a function of time). In a simplified interpretation, there is no clear separation between the systematic and stochastic error terms for longer periods of time, as several components could have non-stationary behavior, in terms of changing over intervals ranging from 10-30 minutes or even hours. For instance, the navigation part, which typically represents the largest terms in the error budget, may have components changing slowly in time, and could be considered as drifts with respect to the other stochastic components with higher dynamics. These could subsequently be modeled as systematic error terms for shorter time periods, such as for the time it takes to fly a strip. In fact, these phenomena form the basis for the need for LiDAR strip adjustments, as these error terms could change significantly between surveys as well as within surveys, similar to experiences with the
GPS-assisted aerial triangulation with longer flight lines (Ackerman, 1994). This is also the reason why it is difficult to obtain reliable error term estimates. Note that the accuracy discussed above relates only to the sensor side of the LiDAR error budget and the object space conditions, in particular, the surface micro characteristics should be also considered. For example, so-called hard surfaces provide good reflections and thus there is no significant amount of error introduced, see Table 3.2 for the typical overall sensor performance. In contrast, vegetated surfaces, such as tree canopies or tall and thick grass can produce uncertainties in terms when and how the reflection takes place and this may
41
be on the level of the combined sensor errors. Fortunately, man-made features
characterize the transportation network and thus strong laser reflections can be generally
assumed. Furthermore, due to the lack of dense vegetation, multiple returns (discussed
later) are rare and typically single returns form the bulk of the point cloud.
Laser Repetition Rate 500m 1000m 2000m 3000m 4000m (kHz) Altitude Altitude Altitude Altitude Altitude 33 <5 cm <10 cm <15 cm <20 cm <25 cm 50 <5 cm <10 cm <15 cm <20 cm N/A 70 <5 cm <10 cm <15 cm N/A N/A 100 <10 cm <10 cm <15 cm N/A N/A
Table 3.2: Manufacturer’s accuracy specification (RMSE) of LiDAR height, as a function
of altitude
N.B. – The quoted accuracies do not include GPS errors (Source: Optech).
Airborne LiDAR is based on time-of-flight measurement of a short laser pulse,
but the returning pulse at the sensor could be quite different from the one that was
emitted. Consequently, besides a single time value that is related to the sensor and object
distance, as shown in equation (3.4), there could be additional parameters of the
measurements that can characterize the object space conditions.
1 d c dt (3.4) 2 where d is the distance between the sensor and object, c is the speed of light, and dt is the traveling time of the pulse (to and from).
42
There are several reasons why the shape of the returning laser pulse could be different. First the beam divergence should be mentioned, which practically means that the pulse is reflected back from a relatively larger area, called footprint, which could be typically in the 10-50 cm range in practice. Depending on the surface orientation with respect to the direction of the incoming pulse, the reflection won’t happen instantaneously, as reflection will occur at different locations at different times, resulting in a “stretched” return pulse shape. Fig. 3.3 illustrates the shape formation or the return pulse at different surface directions. The spread of the laser beam causes an uncertainty in the computed coordinates of the objects within the footprint since the coordinates of each return from the same footprint are derived from the nominal beam direction, which is not the actual direction of the laser beam for each return. The possible horizontal displacement of the LiDAR point is not more than half of the footprint diameter.
(a) (b)
Figure 3.3: (a) perpendicular return, no changes in shape; (b) deformed shape at sloped surface
43
Another consequence of the beam divergence is that it provides for penetration in low density object domains, such as tree canopies. Depending on the pulse energy and divergence angle, plus the canopy parameters, there could be several reflections created by a single pulse. In a typical forest with medium density canopies, 2-3 returns from one emitted laser pulse are frequently generated and, in overall, a good number of returns actually coming from the ground. This is a significant advantage of LiDAR in forestry application, as it provides a direct measurement of the tree canopy height, which is formed from first return pulses, and ground level, which is a subset of the last return pulses, and accordingly, biomass can be estimated. Fig. 3.4 shows a typical reflection pattern.
Figure 3.4: Multiple reflections in vegetated areas; cyan: only one return pulse, red: first of multiple returns, orange: intermediate of multiple returns, blue: last of multiple returns
Although the concept of multiple returns is simple, there is a condition for obtaining subsequent returns, namely, the returns should be separated in space; in other words, they should be apart from each other at least as much as the width of the laser pulse. Given the about 10 ns pulse width, the minimum distance required to distinguish returns is 3 m, which is probably acceptable to separate tree canopy from the ground, but, in general, it is too big to properly describe the tree structure, including a large number of
44 limbs in close proximity. This leads to the concept of recording the complete return signal as opposed to identifying individual return pulses. The technique, called waveform analysis, records the time interval, sampled at sufficient rate, around the expected return pulse (which is based on the approximate object distance). The benefit of full waveform analysis is that return pulses coming from close objects can be identified with quite good success rate. Furthermore, more complex object space situations can be detected and parameterized. Fig. 3.5 shows a waveform with two “overlapping” return pulses.
Figure 3.5: Waveform example of two returns from close objects; blue lines mark detected returns
The pulse detection solution, implemented in the sensor, has also some implications on accuracy and separation capabilities. Earlier systems used a rather rudimentary threshold-based technique that, obviously, did not account for any shape deformation. Later a coarse measurement of the pulse allowed determining the “center” of the pulse. Eventually, this has lead to the waveform-based solution, where a short sequence of samples is kept in a circular buffer to support the pulse detection and parameterization.
45
The atmospheric conditions are somewhat less critical to the measurement accuracy, as the object distance for airborne LiDAR is not that large, typically less than
1,000 m, and therefore, the refraction is usually negligible.
The surface microstructure, such as diffuse or reflective types, has a strong impact on the return pulse shape, although it is rather difficult to calibrate a system for various materials, as the detection of the material type is practically not feasible in the general case. A good example, which is closely related to transportation applications, is that pavement markings, which are made of highly reflective materials, cause a bias in the measurements, as illustrated in Fig. 3.6.
Since most of the newer systems are based on waveform-based pulse detection, it is rather easy to extract an additional measurement parameter, which is related to the pulse and can express the shape and/or energy content of the pulse. This so-called
“intensity” signal is widely used in modern systems, and is the source of essential information that is used in one of the techniques introduced later in this dissertation. The intensity signal is an excellent visualization tool, as it provides for a panchromatic image, which closely resembles optical imagery. However, it is important to note that the
LiDAR intensity is a relative signal, and thus, extreme care should be exercised when used in computations. For example, identical intensity values could be obtained from different flying heights and different objects. Fig. 3.7 shows an elevation and intensity
LiDAR data pair. One early application of the intensity data was the introduction of intensity-based range correction, which, in theory, could eliminate the range differences shown in Fig. 3.6.
46
(a) (b)
(c) Figure 3.6: Vertical shift in LiDAR data caused by retro-reflective materials: (a) elevation data, (b) contours, and (c) aerial imagery; note that the text on the taxiway is quite readable, although the thickness of the pavement markings is in the mm range
(definitely way below the cm-level laser ranging accuracy)
47
(a)
(b)
Figure 3.7: (a) LiDAR height and (b) intensity data
48
LiDAR technology has advanced significantly in the last four-five years. The ranging accuracy of LiDAR systems has improved substantially, now it is at the 1-2 cm level (1) for hard surfaces. Besides obtaining elevation data, the interpretation of the returned signal is now feasible with the appearance of full-waveform LiDAR systems.
Point density has improved, as the pulse repetition frequency (PRF) advanced to the 100-
200 kHz range, which enables increased point density. In late 2006, Optech introduced its multiple pulse technology, which allows the firing of a second laser pulse by the rangefinder before the reflected signal from the previous pulse has been received and detected by the system (Toth, 2004). This has allowed the use of a much higher pulse repetition rate - in this case, 167 kHz - to be reached in the latest ALTM Gemini model.
The other market leader, Leica followed suite and the multipulse technology based Leica
ALS60 system is featuring data capture rate up to 200 kHz at up to 6000 m AGL. Table
3.3 summarizes the characteristics of some commercial airborne laser scanning systems based on (Kukko and Hyyppä, 2007).
With current LiDAR systems, the achievable vertical accuracy is within 15 cm up to 2000 m AGL. The horizontal accuracy varies over a larger range, but mostly characterized by the footprint size. The accuracy of the point cloud acquired by the
LiDAR sensor depends on various factors, including the accuracy of individual sensors, such as the georeferencing accuracy of the GPS/INS system, calibration quality of the
LiDAR sensor, and inter-sensor calibration, such as misalignment and time synchronization quality between the LiDAR and navigation sensors. Note that the GPS component plays an important role in achieving a high georeferencing performance, as it
49 defines the basic positioning error budget. The object space content has also an impact on the achievable accuracy, but it is difficult to characterize it in a general way. The accepted practice is to define the accuracy for hard surfaces, such as roads. The calibration of the LiDAR sensor includes scan angle and range calibration, and intensity- based range correction. The relation between the LiDAR and navigation (IMU) frames is described with the offset between the origins of the two frames and misalignment angles.
Misalignment angles give the angular differences between the two coordinate systems.
An overview of the basic relations and formulas regarding airborne laser scanning is presented in (Baltsavias, 1999).
Scan Pulse Scanning Beam Pulse Range Pulse Digitizer Sensor Mode Freq. Freq. Angle Diverg. Energy Resolution Length [ns] [Hz] [kHz] [º] [mrad] [ µJ] [cm] [ns]
Optech 2033 Oscillating 0-70 33 ±20 0.2/1.0 N/A 1.0 8.0 N/A
Optech 3100 Oscillating 0-70 33-100 ±25 0.3/0.8 <200 µJ 1.0 8.0 1
Optech 0.15/0.25/ Oscillating 0-70 167 ±25 <200 µJ 3.0 7.0 N/A Gemini 0.8
Optech Orion Oscillating 0-100 167 ±25 0.25 <200 µJ 2.0 7.0 N/A
TopEye MkII Conic 35 5-50 14,20 1.0 N/A <1.0 4.0 0.5
TopoSys I Line 653 83 ±7.15 1.0 N/A 6.0 5.0 N/A TopoSys II Line 630 83 ±7.15 1.0 N/A 2.0 5.0 1 Falcon Leica Oscillating 25-70 83 ±37.5 0.33 N/A N/A 10 N/A ALS50 Leica Oscillating 35-90 150 ±37.5 0.22 N/A N/A 10 1 ALS50-II Leica ALS60 Oscillating 0-100 200 ±37.5 0.22 N/A 3.0-4.0 5.0 1 Riegl Line 160 240 ±30.0 0.3 8 2.0 4.0 1 LMS-Q560 Riegl Line 200 240 ±30.0 0.4 8 2.0 4.0 1 LMS-Q680
Table 3.3: The characteristics of selected commercial airborne laser scanning systems
50
Over the past years LiDAR has become the preferred method for the acquisition of digital terrain models (DTMs) at local scale. The acquired unfiltered LiDAR point cloud provides the surface, the upper envelope of the objects (DSM) that can include natural and man-made objects, such as terrain, trees, buildings, power lines, vehicles. The points that are not part of the bare Earth surface should be filtered out to obtain terrain models, frequently called bald earth data. Several filtering algorithms have been developed for this purpose (Sithole and Vosselman, 2004).
Terrestrial and mobile laser scanning
The main focus of the discussion is on sensors installed on airborne platforms.
However, we should note that rapid developments have happened in terrestrial laser scanning recently, and more importantly, terrestrial systems now can be installed in mobile platforms, such a surveying vans moving at highway speed. These systems can be deployed on need, as much as airborne systems, and because of their productivity can compliment airborne data acquisition. Terrestrial laser scanners can be pulsed or continuous wave-based systems, although the vehicle mounted systems, which are typically pulse-based, are of great interest to transportation applications. Recent developments in vehicle mounted systems laser scanners were driven by DARPA Grand and Urban Challenges (Toth and Paska, 2006; Toth et al., 2006). Recognizing the potential of laser scanning, manufacturers developed dedicated laser scanners for the
2007 race. The Ibeo system (http://www.ibeo-as.com) provided laser profiling capabilities in four planes, thus replacing four SICK units, widely used laser profilers
(http://www.sick.com). Then, more importantly, the introduction of the Velodyne laser
51 scanner, which was used by the winner, the Boss, a fully autonomous Chevy Tahoe vehicle from Carnegie Mellon University, and by most of the top ten participants, represented a major technological breakthrough in 2007. The Velodyne HDL-64E High
Definition LiDAR, shown in Fig. 8, is based on using 64 laser units covering a 26.8 vertical spread, thus eliminating the need for any vertical mechanical motion
(http://www.velodyne.com/lidar/). The system sports high horizontal rotation rates of the laser sensors around the vertical axis of the unit, at up to 15 Hz, with an angular resolution of 0.09. The Class 1 laser operates at the wavelength of 905 nm with a 10 ns pulse width. The ranging accuracy is claimed to be less than 5 cm for 50 m and 120 m with reflectivities of 10% and 80%, respectively. The data collection rate of more than one million points per second is simply superb. Although the Velodyne mobile laser scanner that was specifically developed to support the DARPA Urban Challenge, since then it has been used in conventional mapping. A LiDAR data obtained by a mobile
LiDAR scanner, Lynx from Optech, shows a road segment in the front of the OSU
Stadium in Fig. 3.9.
Figure 3.8: Velodyne scanner; (a) system and (b) concept (Courtesy of M. Shand)
52
Figure 3.9: Terrestrial LiDAR data, acquired by the Lynx system from Optech
In summary, LiDAR and, in general, laser scanning technology continues to advance, following similar trends to the remote sensing imagery developments, where the past gaps between terrestrial, airborne and spaceborne imagery is not only disappearing, but the overlap between the imagery acquired from various platforms is growing.
53
3.2.3. IfSAR sensor
Interferometric Synthetic Aperture RADAR (IfSAR) has been a powerful remote sensing technology to capture 3D spatial information at a global scale (Manual of Remote
Sensing, 1998). In comparison to LiDAR, IfSAR is extremely productive; the area coverage is substantially larger, typically by two orders at the same spatial sampling rate.
Note, however, that accuracy of IfSAR falls way below that of LiDAR, and the accuracy is strongly affected by atmospheric conditions, in particular, by moisture content. In addition, occlusions are also causing problems, as the technique is inherently based on side-looking sensor configuration. The accuracy vs. swath width tradeoff of IfSAR is shown in Fig. 3.10
Figure 3.10: Accuracy vs. swath width tradeoff of IfSAR
Although, IfSAR has been used for a long time compared to digital cameras and
LiDAR, the number of systems is still limited, as this technology has never really
54 penetrated mainstream mapping. Recently, airborne IfSAR has gained significant market share in DEM/DSM surface model creation, which was primarily fueled by increased demand for medium accuracy statewide coverage in the industrialized world. Intermap, the largest IfSAR data provider’s MapUK program turned out to be such a success that a program was initiated to cover the US and EU (http://www.intermap.com). The primary users of such global surface models are government agencies and insurance companies.
The increasing commercial satellite market is also a main user of this data, as the accuracy and spatial resolution are comparable to the satellite data requirements. Finally, car navigation systems should be mentioned as another important user, as they are increasingly updating the earlier 2D maps to 3D models.
RADAR and IfSAR have been long used on satellite platforms. The most notable first system was the RADARSAT-1, Canada's first commercial Earth observation satellite, launched in 1995. RADARSAT-1 offer three different resolutions, 10 meters, 30 meters and 100 meters. In February of 2000, the 11-day Shuttle Radar Topography
Mission (SRTM) onboard the Space Shuttle Endeavour obtained elevation data on a near- global scale to generate the most complete high-resolution digital topographic database of the Earth. SRTM consisted of a specially modified radar system
(http://www2.jpl.nasa.gov/srtm/). The objective of the mission was to produce digital topographic map products with 30 m x 30 m spatial sampling with ≤ 16 m absolute vertical accuracy, ≤ 10 m relative vertical accuracy and ≤ 20 m absolute horizontal circular accuracy. SRTM was launched in an orbit with an inclination of 57 degrees. This allowed all of the Earth's land surface that lies between 60 degrees north and 56 degrees south latitude to be covered by the SRTM (this is about 80 percent of the Earth's surface).
55
More recently, RADARSAT-2, Canada's next-generation commercial SAR satellite, the follow-on to RADARSAT-1 was successfully launched in December, 2007
(http://www.radarsat2.info). The geolocation accuracy of RADARSAT-2 products varies with product type; it is estimated at +/- 30 m for standard beam products. Ultra-Fine beam provides 3-100 m resolution, which improves object detection and classification.
The successful launch of the German radar satellite TerraSAR-X in 2007 started a new era to use radar to collect new, high-quality and high spatial resolution data of Earth's surface.
3.2.4. Hyperspectral sensor
Hyperspectral imaging (HSI) is a relatively new technology; more precisely, the first experimental systems go back several decades, but the commercial use of this technology started only in the late 1990’s, when hardware and sensor developments reached the point that the systems became affordable at an acceptable performance level.
Note that the introduction of direct, GPS/INS-based georeferencing of airborne platforms was an essential enabling technology for HSI. In contrast to multispectral imagery, which has only a limited number of bands, the pixels of HSI contain dozens or hundreds of spectral bands, and thus, airborne HSI sensors can measure the detailed spectrum of the light from the earth’s surface to identify and classify many types of materials, such as minerals, vegetations, artificial materials, and water. In a way, HSI represents an ideal sensor, as from the spectral bands, both multispectral and panchromatic imagery can be easily derived. Unfortunately, the spatial resolution of HSI sensors is quite limited as
56 compared to that of digital cameras. Generally, the IFOV is in the 0.03-0.05 range, while digital cameras have typically a 0.01 or better parameter; which means that almost two orders more pixels cover the area of an HSI pixel on the ground; therefore, it is a trade- off whether better spectral or spatial resolution is more important in an application.
Fortunately, transportation applications could benefit from both, as the shape of the objects are better described in high spatial resolution, while the classification would gain a lot from a high spectral resolution data. Although airborne HSI technology is currently used only in a few commercial and defense applications, but is expected to grow rapidly in the future, as technology keeps advancing, and thus, HSI sensors will soon be feasible for transportation applications. Table 3.4-5 show the known HSI systems, grouped according to the sensor type.
Whiskbroom Hyperspectral/Multispectral Sensors
Dynamic FOV IFOV System Spectral Range [nm] Range GPS/INS Software [deg] [mrad] [bit] 450 – 2,500 Hymap Mandatory/ options for extended range 35-60 1 12-16 Proprietary Integrated Spectronics Integrated available Daedalus Daedalus AHS-75 430 – 13,000 1.25 or Available/ 90 16 AHS with Argon ST Up to 64 multispectral bands 2.5 Various Rapid Mapper Daedalus Daedalus AHS-160 450 – 13,000 Available/ 90 2.5 16 AHS with Argon ST Up to 128 multispectral bands Various Rapid Mapper Daedalus Daedalus MIVIS 433 – 12,700 Available/ 90 2.0 16 AHS with Argon ST Up to 128 multispectral bands Various Rapid Mapper
Table 3.4: HSI sensor categories and major systems
57
Pushbroom Hyperspectral Sensors
Across-Tacks Dynamic Spectral Range Pixels x No. of FOV IFOV System Range GPS/INS Software [nm] Spectral [deg] [mrad] [bit] Channels CASI-1500 and 380 – 1,500 CASI-1500 UV Mandatory/ 365 – 1,035 1,500 x 288 40 0.49 14 Proprietary Compact Airborne Various Stereographic Imager (UV) ITRES Research 400 – 1,000 CASI-550 Mandatory/ (545 nm free 550 x 288 40.4 1.34 14 Proprietary ITRES Research Various spectral range) SASI-600 Shortwave Infrared Mandatory/ Airborne 950 – 2,450 600 x 100 40 1.2 14 Proprietary Various Stereographic Imager ITRES Research TABI-320 Thermal Airborne Mandatory/ Stereographic 8,00 – 12,000 320 x 1 48 2.8 12 Proprietary Various Imager ITRES Research TASI-600 Thermal Airborne Mandatory/ Stereographic 8,000 – 11,500 600 x 32 40 1.25 14 Proprietary Various Imager ITRES Research Daedalus SWIR Daedalus Mandatory/ HSI 900 – 1,700 640 x 80 35 1.0 14 AHS with Various Argon ST Rapid Mapper Daedalus E-SWIR Daedalus Mandatory/ HSI 900 – 2,400 640 x 150 35 1 14 AHS with Various Argon ST Rapid Mapper Daedalus VNIR Daedalus Mandatory/ HSI 400 – 1,000 1,600 x 288 46 0.5 14 AHS with Various Argon ST Rapid Mapper
Table 3.5: Major pushbroom HSI systems
Among the pushbroom hyperspectral sensors, CASI-1500 by ITRES Research, and Daedalus VNIR HSI by Argon ST have the highest spatial resolution; CASI-1500
58 has 1,500 pixels per CCD line and Daedalus VNIR HSI has 1,600 pixels. ITRES
Research, is the largest commercial data provider of the HSI imagery, and therefore, the general specification of their flagship system is listed in Table 3.6; the CASI-1500 system has an operating range up to 10,000 ft (ASL).
# Spectral Channels 288 # Spatial Pixels 1500 Total Field of View 40° IFOV 0.49 mRad f/# f/3.5 Spectral Width Sampling/Row 2.4 nm Spectral Resolution 2.4 nm (Full-Width Half-Maximum) Pixel Size 20 x 20 microns Dynamic Range 14-bits (16384:1) Burst Data Rate 5 Megapixel/sec Spectral Smile ±0.25 pixels Keystone Distortion ±0.25 pixels Peak Signal to Noise Ratio (SNR) SNR models for various radiance conditions are available from ITRES Research ITRES Research provides three kinds of CASI-1500 image products
Table 3.6: Specification of CASI-1500
3.3. Navigation sensors
3.3.1 GPS and GNSS systems
The purpose of the Global Positioning System is to provide highly accurate positioning, navigation and timing data globally, 24 hours a day, and in any type of weather to civilian and authorized military users (Hofmann-Wellenhof et al., 2001). GPS
59 consists of three major segments: space segment, control segment and user segment. The current constellation of the space segment includes 31 satellites in six orbital planes, traveling in semi-synchronous (12-hour) orbits around the earth. These satellites broadcast navigation and timing signals on two frequencies (L1 and L2). The control segment consists of a master control station, six monitor stations, and five ground antenna stations. The monitor stations track the navigation signals and send their data back to the master control station. There, the controllers determine any adjustments or updates to the navigation signals needed to maintain precise navigation and update the satellites through the ground antennas. The user segment includes civil and military GPS receivers used for air, land, sea and space applications. A single GPS receiver calculates its antenna’s position by measuring the distance between itself and four or more GPS satellites. Measuring the time delay between transmission and reception of each GPS microwave signal gives the distance to each satellite, since the signal travels at a known speed in vacuum (path and speed are distorted in the atmosphere). The signals also carry information about the satellites’ location. By determining the position of, and distance to, at least four satellites, the receiver can compute its position using trilateration. The minimum error in range measurement is 3 meters using only C/A code, and about 30 centimeters with P(Y) signal. Different error sources, including atmospheric effects on the GPS signal, errors in the satellite position ephemeris data, clock drift and multipath, can significantly reduce the accuracy of point positioning. The combined effect of these errors could cause a deviation of ±15 meters from the actual GPS receiver position in extreme situations. To eliminate or reduce the effect of these error sources, different techniques were developed and various implementations exist, such as differential
60 correction techniques. Differential corrections can be applied in real-time in the field or in post-processing in the office. When applied in real-time, a base station or a network of base stations compute and transmit corrections to the roving receiver via a radio signal or orbiting satellites. Wild Area Augmentation System (WAAS), available in North-
America, computes corrections based on 25 ground reference stations positioned across the United States, and broadcasts those through one of two geostationary satellites.
Differential corrections, provided by WAAS, improve the accuracy of point positioning from 10-15 meters to 1-2 meters. OmniSTAR offers GPS correction services that can improve the accuracy of a GPS receiver by more than 100 times
(http://www.omnistar.com/). Currently, three levels of service are offered: "VBS" offers sub-meter positioning, "XP" achieves better than 20 centimeters, and "HP" is better than 10 centimeters. OmniSTAR is a wide-area differential GPS service, using satellite broadcast techniques. For the sub-meter service, data from many widely-spaced
Reference Stations is used in a proprietary multi-site solution over most land areas worldwide. The high accuracy HP solution uses more sophisticated data from these reference sites and XP uses satellite orbit and clock correction data which is independent of reference site location.
A new method of RTK-GPS (real-time kinematic GPS) is the network-based method, such as, for example, VRS (Virtual Reference Station) system, which enables instantaneous precise (cm-level accuracy) positioning in a wide area just by receiving the correction data from a LBS (Location Based Services). The VRS concept is based on a wide network covered by multi-reference stations which is used to create a virtual
61 reference station near the rover. An overview of various kinematic GPS positioning solutions is presented in Table 3.7.
Accuracy Real-time/Post-processed Pseudorange 10 m-level Post-processed Pseudorange-based differential m-level Post-processed WAAS pseudorange m-level Real-time Differential with base station cm-level (*) Post-processed/Real-time Differential with network solution cm-level Real-time/ Post-processed Satellite based differential correction sub-m level Real-time RTK cm-level Real-time VRS cm level Real-time PPP sub-dm level Post-processed (*) baseline-dependent
Table 3.7: Comparison of various kinematic GPS methods
Several improvements have been implemented to the GPS service over the last decade, including new ground stations, new satellites, and the future introduction of additional navigation signals for both civilian and military users in order to increase accuracy and integrity of the system. A significant improvement for civil and commercial users of GPS was the decision on 1 May 2000 to permanently turn off SA (Selective
Availability), the capability to degrade the civil signal. The 2008 GPS constellation consists of 31 Block II/IIA/IIR/IIR-M satellites. A newer generation of satellites, Block
IIR and IIR-M began replacing older Block II and IIA satellites in 1997. Block IIR-M satellites are providing a new civil signal (L2C) on the L2 channel and a new military
62 signal, M-code, transmitted in L1 and L2 frequencies. With the launch of the next generation of GPS satellites, Block IIF, civilian signal on a third frequency (L5) will be available (beginning ~2011). Furthermore, the L1C civilian signal broadcast on the L1 frequency will be offered with the first Block III launch, currently scheduled for 2013.
In parallel to the GPS modernization program in the US, several countries announced the implementation of their own global or local satellite based navigation systems, generally referred to as GNSS (Global Navigation Satellite System). From a technical point of view, the competing systems, if they are designed for interoperability, they could improve the availability of navigation satellite signals in the future, as there provide for a denser satellite constellation. For example, more satellite signals could be received in urban canyons, which is currently impossible. However, it should be noted that the sheer fact that more satellite signals are available doesn’t guarantee that more accurate results can be obtained. For example, in the urban canyons, the geometry of the satellites will be poor, as they are close and don’t provide a measurable spatial separation, which is important to achieve good positioning accuracy (low PDOP, explained Section
3.3.4). So in this case, the benefit could be to have a solution with modest or coarse accuracy and that the lock to the satellite signals is maintained. Table 3.8 lists the already existing and planned GNSS systems. As of 2009, the United States NAVSTAR Global
Positioning System is the only fully operational GNSS. The Russian GLONASS is a
GNSS in the process of being restored to full operation. China has indicated it will expand its regional Beidou navigation system into the global COMPASS navigation system by 2015. The European Union's Galileo positioning system is a GNSS in its initial deployment phase, scheduled to be operational by 2013.
63
Current global navigation satellite systems: US NAVSTAR Global Positioning System (GPS) Russian GLONASS (not fully operational) Proposed global navigation satellite system: European Galileo Chinese Compass Regional navigation systems: Japanese Quasi-Zenith Satellite System (QZSS) Indian Regional Navigational Satellite System (IRNSS) French Doppler Orbitography and Radio-positioning Integrated by Satellite (DORIS)
Table 3.8: Operational and planned GNSS systems
In summary, satellite-based navigation will remain the primary worldwide navigation and positioning technology, and improvements in both robustness and accuracy are expected. From the perspective of the transportation applications, from the above listed developments the network-based differential GPS, in particular the real-time
VRS are of high importance in both terrestrial and airborne remote sensing, as it provides for real-time traffic management.
3.3.2 Inertial systems
An INS (Inertial Navigation System) includes an embedded computer system and the sensor module, the IMU (Inertial Measurement Unit), including three orthogonal accelerometers and three orthogonal gyroscopes, used to detect and measure translational
64 and rotational motions (linear accelerations sensed by accelerometers and angular rates sensed by gyroscopes) based on the physical laws of nature (see, for example, Titterton &
Weston, 1997; Farrell & Barth, 1999; Jekeli, 2001). Accelerations, including both linear and angular, are integrated by the embedded computer to deduce the position, velocity and attitude of the platform. The INS principles require an accurate gravity model to compensate the local gravity vector g in order to produce an accurate kinematic acceleration; conventionally, a normal gravity model or a higher-order spherical- harmonics gravity model is used, while accurate Deflections Of the Vertical (DOVs) can also be included in the navigation equations to improve the INS navigation performance
(Grejner-Brzezinska & Wang, 1998; Jekeli, 2001).
Although, there are several INS mechanization methods, the most widely used
INS mechanization today is the strapdown system where the IMUs are firmly mounted to the navigation platform, and the inertial measurements are mathematically transformed to the navigation frame. Consequently, the IMUs are subjected to the entire range of the platform dynamics. The advantage of the strapdown inertial navigation system, compared to other existing mechanizations (such as the gimbaled inertial system), lies in the smaller size, less weight, less power consumption and lower costs. The majority of the currently implemented GPS/INS-integrated system for direct sensor georeferencing is based on the strapdown mechanization (reference for strapdown can be found in Jekeli, 2001).
The INS is a relative positioning system, as it can only compute position and attitude changes between two epochs. Because the current position of the unit is computed by continually adding detected changes to its previously-calculated positions, any errors in measurement, however small, are accumulated in the final calculation; in
65 particular, because of the position is computed by twice integrating the measured acceleration data. This leads to drift, an increasing difference between where the system
thinks it is located and the actual position. To correct positional inaccuracies of an INS,
external information, such as GPS is used to correct for long-term drift in position. Table
3.9 lists major commercially available INS systems; a more exhaustive discussion on INS
error modeling can be found in (Yi, 2007).
Sensor grade Sensor name Type Characteristics bias = 0.003º/h LN100 Non-dithered 18cm Zero rw = 0.001º/h½ Gyroscope LockTM Laser Gyro sf < 1ppm navigation bias = 25μg LN100 sf = 40ppm Miniature Accelerometer A4 Accelerometer ma = 2arcsec wh = 5μg/Hz½ bias = 0.0035º/h H764G Dithered GG1320AN RLG rw = 0.0035º/h½ Gyroscope sf = 5ppm navigation bias =25μg H764G QA2000 wh = 8.3μg (100Hz bw) Accelerometer sf =100ppm bias = 2.0º/h HG1700 Dithered GG1308 RLG rw = 0.125~0.3º/h½ Gyroscope sf =150ppm tactical bias = 1.0mg HG1700 wh = 0.2mg (100Hz bw) RBA500 Accelerometer sf = 300ppm ma = 12m-rad bias = 1º/sec IMU400CC Non-dithered Silicon MEMS rw = 2.25º/h½ Gyroscope Gyro sf = 1% consumer bias = 8.5mg IMU400CC Silicon MEMS Accelerometer sf = 1% Accelerometer rw = 0.1m/s/h½
Table 3.9: Manufacturer’s specifications for the selected inertial sensors; sf: scale factor; ma: misalignment; wn: white noise; rw: random walk; bw: bandwidth
66
3.3.3 Integrated navigation systems
With the integration of differential-GPS (DGPS) and INS, the advantages of the two systems are combined providing a highly accurate and highly sampled in time position and attitude of a platform. Table 3.10 summarizes the advantages and disadvantages of the two systems, individually and integrated.
Advantage Disadvantage Losses of lock causing gaps in positioning High accuracy of position and Low data sampling rate (1-10 DGPS velocity estimation Hz) Time-independent error model Slow ambiguity resolution time over long baseline Continuous data acquisition Three positioning and three Sensor errors grow with time INS attitude components causing positioning error High data sampling rate (up to divergence 256 Hz) Combine all advantages of both systems Redundant and complementary Precise time synchronization DGPS/INS data needed Navigation through GPS outages GPS fixes allow INS error estimation
Table 3.10: Advantages and disadvantages of GPS and IMU systems in standalone and integrated modes
The GPS/INS is typically achieved by using an Extended Kalman Filter (EKF), although alternative filtering techniques such as Unscented KF or Particle Filter are also strongly investigated in the research community (Jazwinski, 1970; Julier & Uhlmann,
67
1997; Liu and Chen, 1998; Doucet et al., 2001). Conceptually, there are two basic solutions implemented in the EKF: loosely-coupled and the tightly-coupled GPS/INS integrations, where the difference is whether two navigation solutions are blended
(loosely-coupled) or the sensor level data are directly integrated (tightly-coupled); additional details can be found in (Gautier and Parkinson, 2003).
Commercially available GPS/INS-based georeferencing systems to support airborne sensor orientation were introduced in the mid-90’s but eventually the technology got accepted by the late 90’s. The pioneer and for a long time the only provider was
Applanix Inc., and in fact, their POS AV product is still the most widely used integrated
GPS/INS system. Recently, new systems became available from IGI, Leica and Novatel
(AEROcontrol™, IPAS20 and SPAN™ Technology, respectively). For references,
Tables 3.11 and 3.12 list the manufacturers supplied performance parameters of the POS
AV and IPAS products.
Model No. 210 310 410 510 610 Position (m) 0.05 – 0.3 0.05 – 0.3 0.05 – 0.3 0.05 – 0.3 0.05 – 0.3 Velocity (m/s) 0.01 0.0075 0.005 0.005 0.005 Roll & Pitch () 0.04 0.015 0.008 0.005 0.0025 True Heading () 0.08 0.035 0.015 0.008 0.005
Table 3.11: POS/AV absolute accuracy specifications; post-processed RMSE values
Note - The lower-end POS/AV 210, 310 and 410 systems all use MEMS quartz gyros; the
POS/AV 510 uses fibre-optic gyros; while the POS/AV 610 uses ring laser gyros. (Source:
Applanix)
68
NUS4 DUS5 NUS5 CUS6 Position 0.05 - 0.3 m 0.05 - 0.3 m 0.05 - 0.3 m 0.05 - 0.3 m Absolute Accuracy After Velocity 0.005 m/s 0.005 m/s 0.005 m/s 0.005 m/s Post-processing Roll & Pitch 0.008 deg 0.005 deg 0.005 deg 0.0025 deg (RMS) Heading 0.015 deg 0.008 deg 0.008 deg 0.005 deg
Angular <0.05 <0.01 <0.01 <0.01 Relative Random deg/sqrt deg/sqrt deg/sqrt deg/sqrt Accuracy Noise (hour) (hour) (hour) (hour)
Drift <0.5 deg/hour <0.1 deg/hour <0.1 deg/hour <0.01 deg/hour 200 Hz 200 Hz 500 Hz 200 Hz High IMU Performance Fibre Optic Fibre Optic Dry Tuned Ring Laser Gyros Gyro Gyro Gyro Gyro
Internal in 54-channel Dual Frequency Receiver (L1/L2) Low Noise, GPS Receiver IPAS10 Control Unit 20 Hz Raw Data, DGPS Ready
Table 3.12: Specification and accuracy values for the Leica IPAS10 system
Note. – The NUS4 is the iMAR FSAS unit; DUS5 is the Litton LN-200 unit; NUS5 is a
Sagem unit; and CUS6 is the Honeywell MicroIRS unit. (Source: Leica Geosystems)
3.3.4 Mission planning
Mission planning is an essential step in acquiring quality airborne remote sensed data and it can include several tasks depending on resource availability and project objectives or requirements. Ideally, the sensor selection and configuration could be the first step to achieve an optimal performance, defined as the project requirements are met at the lowest cost possible. In real life, however, the process is limited to flight planning; in other words, for the given sensor suite, an optimal trade-off should be found for all the flight and sensor parameters to achieve the same objective. A carefully planned mission
69 for topographic mapping, in general, satisfies the requirements of typical transportation applications, and there are only a few areas where an additional attention must be paid to better serve the objectives of extracting traffic flow data from remote sensed imagery.
The following list provides a brief overview on the most important items of mission planning.
Georeferencing component
o A mission, preferably, should begin and end with a static GPS/IMU data
acquisition of the aircraft platform, each lasting a minimum of 5 minutes.
The static data helps the GPS post-processing software to obtain the
correct initial and final ambiguities with a high probability of success.
Obviously, there is no need for this step if VRS is used, and there is no
need for a post-processed solution.
o For single baseline solution, the baseline length should be limited to 10 to
50 km. This allows the GPS processing software to recover fixed integer
ambiguities after cycle slips or loss of phase lock at any time during the
mission. If baseline separation is greater than 100 km, then multiple base
station approach or network-based solution in post-processing mode must
be considered. Again, if VRS or another network-based approach is used,
there is no restriction on the flight lines.
o Since, the GPS constellation has been completed in the mid-90’s, there is
less need for planning of the time window of the mission, in general.
However, for applications with high accuracy requirements, planning for
70
PDOP (Positional Dilution of Precision) is important, as the GPS quality
sets the lower boundary for the navigation solution in most cases. The
PDOP and the number of satellites visible above a specified mask angle
should be examined prior to the flight as an indication of the potential
quality of the GPS/INS solution. The smaller the PDOP value (Leick,
1995) the better geometric strength of the satellite constellation, and
ultimately, better georeferencing solution accuracy. Despite careful
planning, the PDOP may change during the flight, and therefore, PDOP, if
possible, should be monitored during the mission to ensure a quality
solution. While it is typically easier to track more satellites in the air
versus on the ground (at a base station), aircraft maneuvers while
performing flight operations can block the view of satellites (depending
on the location of the GPS antenna) and instantaneously increase the
PDOP. The most common situation is for the wing of the plane to block
the signal of a satellite when making a turn involving banking. For this
reason, the banking angle is normally limited. o The INS is a self-contained sensor and thus less dependent on
environmental conditions. The only important task need for an INS is the
proper initialization/alignment, which is essential to achieve good
performance. Note that it can be performed either on ground or in air. The
in-air INS initialization involves flying straight and level for at least three
minutes to align roll and pitch, and then performing a series of turns to
align the heading. In normal operations, long straight segments should be
71
avoided, as it could degrade the attitude data, in particular, the heading
component. Therefore, during the mission every maneuver helps
maintaining and improving the quality of the navigation solution.
Digital camera
o The planning for acquiring airborne imagery was traditionally concerned
with deciding on the flying height and then assuring the images with
proper forward- and side-overlap will be acquired. The flying height was
primarily determined by the required image scale, in other words, what
accuracy is needed, which is mainly defined by the ratio of the focal
length and the flying height. Another aspect to achieve the required
accuracy is to have a good base/height ratio, which depends on the overlap
between consecutive images; smaller values results in less accurate feature
positioning accuracy. Besides that, a good image resolution is assumed, so
the measurement error in the image domain is relatively small with respect
to the other terms in the overall error budget.
o For indirect sensor orientation, the flight lines must satisfy certain
conditions to form an image block that provides a strong geometry for
reconstructing the camera orientation parameters. With the introduction of
GPS/INS-based direct georeferencing, there is no need to impose on any
pattern on the flight lines.
o Digital cameras have several features that are superior to that of the film-
based cameras. Since the spatial data acquisition over the transportation
72
network needs to handle rather dynamic object scenarios, the temporal
aspect of the acquired data is equally important to the spatial resolution,
therefore, the higher image capture rate must be exploited as much as
possible. The topographic data processing may not require the multiple
image coverage of the object space, but the vehicle extraction process will
definitely benefit from the availability of the better temporal resolution
image data. Similarly, the better spectral capabilities of the digital camera
system provides for better classification.
LiDAR system
o The key flight planning task for a LiDAR system is to achieve nearly even
spatial sampling in both directions, which requires the coordination of the
flying speed with the across-track scanning frequency (scan rate). Then
the flying height, scan rate, scan angle (FOV) and pulse rate should jointly
satisfy some conditions to achieve near even point spacing. Although the
even sampling distance is preferred from geospatial processing, for vehicle
extraction the higher sampling rate in one direction could offer
advantages, so the maximum pulse rate is preferred.
o LiDAR systems are typically used with some optical imagery to provide a
basic visual coverage of the mapped area. Over time, the requirements for
the companion digital cameras have gradually increased and now an ideal
sensor configuration includes a high-performance large-format digital
camera system. Therefore, the coordination with the digital camera, such
73
as FOV selection is getting less critical, and as a general rule, the
requirements discussed above at the digital cameras section should be
generally considered.
3.3.5 Airborne/spaceborne platforms
In the past, the great majority of the geospatial data was acquired from airborne platforms using large-format analog film-based aerial cameras. The dominance of aerial photography started to slowly change when technology, both sensors and general computer power, started to improve, and digital technique was gradually introduced in the data acquisition systems translating to affordable mapping applications. First, remote sensing satellites, introduced about three decades ago provided global multispectral coverage at modest resolution that posed no competition to airborne surveying, as the image resolution was so coarse that it did not allow for medium- and large-scale mapping. The last decade, however, has seen phenomenal developments in both sensor technology and increasing availability of a variety of remote sensing platforms. Most importantly, the large difference in airborne and satellite imagery has started to disappear, as the current commercial satellite systems offer 0.5 m GSD (Ground Sampling Distance) or even better spatial resolution that falls in to the range of airborne remote sensed imagery, which is mostly in the 0.1-1.0 m range. The new generation of commercial satellite systems under deployment will feature shorter repeat times, so they can better compete with anytime-deployable airborne platforms. Table 3.13 lists the specifics of the two market leader high-resolution satellite imagery providers, DigitalGlobe
(http:\www.digitalglobe.com) and GeoEye (http:\www.geoeye.com). In addition, there
74 have been remarkable developments in airborne platforms, as not only aircraft designs but a large number of UAV (Unmanned Airborne Vehicles) have been introduced in the last decade, and the use of these system is under intensive research in many applications;
Table 3.14 lists a representative sets of UAV categories (Gruen, 2007).
Satellite Feature GeoEye-1 Worldview-1 Pan Resolution at nadir .41 meters .50 meters Pan Resolution at 60 elevation .50-meters .59 meter Multi-spectral Resolution at 1.64 meters 2.4 meters nadir Swath width at nadir 15.2 km 17.6 km Spectral range (pan) 450-800 nm 400-900 nm Blue 450-510 nm 450-520 nm Green 510-580 nm 520-600 nm Red 655-690 nm 630-690 nm Near IR 780-920 nm 760-900 nm Launch date 06-Sep-08 18-Sep-07 Revisit Time 3 days at 40° latitude with 1.7 days at 1 meter GSD or less elevation > 60° 5.9 days at 20° off-nadir or less Orbital Altitude 681 km 496 km Nodal Crossing 10:30 AM 10:30 AM
Table 3.13: Comparison of GeoEye-1 vs. Worldview-1 satellite sensors
75
UAV classification Remotely Piloted Vehicle (RPV), Remotely Operated Aircraft (ROA) (a) Remotely Controlled Helicopter (RC-Helicopter) Unmanned Vehicle Systems (UVS) Fixed wing aircrafts (b) Helicopters Airships With subcategories depending on type, size, flying time, altitude of the aircraft
Table 3.14: UAV classification
The availability of various platforms is absolutely essential to better support traffic flow extraction for several reasons. Most importantly, inexpensive UAVs make the deployment of missions dedicated strictly to transportation surveying economic; in comparison, traffic flow data extraction from conventional platforms is only feasible as an opportunity data acquisition. For example, small UAV’s can be kept airborne during rush hours, and easily directed to areas of interest, such as accidents or areas experiencing unusual traffic patterns. Moreover, UAV can fly along highway corridors and quickly respond to requests to monitor traffic-related situations as they develop. UAV technology has recently significantly improved, yet the full exploitation of this technology depends on resolving legal issues associated with liabilities and the availability of small
(miniature) yet powerful remote sensing technologies.
76
3.4 Error characteristics
The error characterization of spatial data acquisition and information providing systems is an extremely important task, as it reflects the confidence in the data, which is essential for any application. In the past mapping practice, when a single camera was the typical sensor of the data acquisition system, the calibration of the imaging sensor provided a baseline for the accuracy along with the parameters of the flight and the processing system. Recent advancements in terrestrial, airborne and spaceborne technologies have not only drastically increased the amount of geospatial data, but improved the potential to obtain better spatial and temporal resolution and at higher accuracy. With the paradigm shift in telegeoinformation systems, described earlier, the data acquisition systems became multisensory, and, obviously, the old single sensor calibration model could not be applied anymore. Of course, the individual calibration is still important, in fact, in many cases, it is even more important, yet due to sensor interactions and the nature of the GPS-based georeferencing system and finally, with the overall system complexity, the generally accepted new model is the output data characterization. Although, the performance of the sensor components can provide an initial estimate of the overall error budget, the accepted approach is based on using a statistically significant amount of samples that are independently validated. It is important to note that in the past, there was always a human operator in the loop, who performed the image measurements, and thus, the operator measurement errors were known. For a large number of automatically extracted points, there is no general error model, as depending on the object scene, such as smooth rolling or hilly terrain, or grassy or densely vegetated areas, or rural or urban scenes, the data points will have quite
77 different error characteristics. The data/product characterization is subject on its own, here only a basic review is provided to briefly address important aspects.
3.4.1 Direct/indirect georeferencing
Traditionally, in airborne photogrammetry the orientation task is solved indirectly using the well-known method of aerial triangulation (AT), which is based on bundle block adjustment (Kraus, 1993). With the introduction of systems capable of directly measuring the position and orientation of the imaging sensor, the direct georeferencing of the image orientation, and subsequently object locations, are possible without the use of ground control points and image tie/pass points. In fact, direct georeferencing, based on integrated GPS/INS, is mandatory for active and passive sensors that are based on line sensor models, such as airborne laser scanners, IfSAR, optical line scanners, etc.
Obviously, direct georeferencing can provide superior economy for frame cameras, too.
The geolocation, i.e., 3D positioning of features extracted from optical images and active sensors requires the knowledge of the sensor model, i.e., the camera and scanner calibration parameters and the exterior orientation values. For classical object point determination, the bundle adjustment represents a very robust method over a block of images. In the least squares adjustment criterion, the residuals of the observations are minimized, and the bundles are optimally fitted to the given control points in the object space. Therefore, uncorrected systematic effects, such as sensor modeling errors, are shifted into the estimated orientation parameters. Consequently, the estimated orientation parameters might be physically wrong, in other words the reconstructed position and attitude of the airborne platform at the moment when the image was taken, but they are
78 optimal in terms of object space reconstruction on the ground. In contrast, direct orientation generally provides the orientation parameters (exterior orientation) with high absolute accuracy, but systematic errors, such as boresight misalignment errors and uncorrected image distortions (sensor modeling discrepancies), could produce errors in object space, as this solution does not compensate for them. Therefore, high-quality sensor models are the necessary prerequisites for high-accuracy geolocation and high quality feature extraction in general, including the capability to determine precise velocities and other parameters from an image sequence. Fig. 3.11 and equations (3.5) and (3.6) illustrate the direct georeferencing concept (Section 14.3, Manual of
Photogrammetry, 2004). Note that the determination of the exterior orientation parameters (position and attitude) of an image at the time of exposure can provide essential support for real-time feature extraction processes and object reconstruction from optical imagery from terrestrial, airborne or satellite platforms, as the search space for image primitive or feature matching can be substantially reduced in contrast when the orientation parameters should be recovered in the same process. The most typical example is the use of the epipolar line constraint, which means that if the orientation parameters of two overlapping images are known, the matching point of any point in image one should fall on a line in image two (obviously, the point should be in the overlap area).
79
GPS antenna
GPS ab
pitch C C ab INS p xb x rc b-frame c c-frame y yaw roll b p zb yc
INS rm p sp· Rc, m · rc C rm
Ym Zm P
P rm
m -frame
Xm
Figure 3.11: Direct georeferencing with optical imagery
r P r C t s R t r p t m m p c, m c (3.5)
r P r INS t R t a C s R t R r p t (3.6) m m b, m b P b, m c, b c
80 where subscripts m, c and b denote the mapping, camera and INS body frames, respectively; points P, p, C and INS denote an object point in the object space, its projection to the image plain, the projection center of the camera, and the origin of the
P C INS body frame, respectively. rm is the vector of the object point P, rm is the vector of
INS the projection center C, and rm is the vector of the INS body center INS in the
C mapping frame, m. ab is an offset vector in the b-frame between the INS body center (b-
GPS frame origin) and the camera projection center, C; ab is an offset vector in the b-frame between the INS body center and the GPS antenna phase center that is measured and
P compensated for during the GPS/INS data processing. rc is the vector of the image point p in the camera coordinate system with its coordinates corrected for the principal point offsets and lens distortion; sp is the scale factor of point p. Rc,m is a rotation matrix from the camera coordinate system to the mapping coordinate system; Rb,m is a rotation matrix from the INS body frame to the mapping coordinate system; furthermore, Rc,b is a rotation matrix from the camera coordinate system to the INS b-frame, known as boresight matrix. The model shown in Fig. 3.11 can be simplified for the LiDAR case, when the scanner parameters provide the image coordinates plus the range data delivers the object space location unlike the image case where only a line is defined. The LiDAR equation, (3.3), was already introduced. An overview comparison of various georeferencing methods is shown in Table 3.15.
81
Georeferencing method
Indirect (AT) GPS-aided AT Direct (GPS/INS) Bundle block Direct physical adjustment process with Bundle block adjustment measurements of the GPS-defined photo platform motion Sensor centers positioning EO parameters could be The position part of the and different from the true EO parameters should EO parameters should orientation physical parameters be nearly identical to be identical to the true (platform orientation the true physical physical parameters accuracy is not of parameters interest) Laboratory Camera (AT with self- Laboratory Laboratory and in-situ calibration calibration) Only for QA/QC Only for QA/QC GCPs Needed + Base station + Base station Compensated Uncompensated Interior orientation IO errors errors are absorbed by Boresight misalignment (systematic AT and uncorrected image and random) Wrong IO and AT will distortions provoke result in optimal EO errors in object space reconstruction Object space High object space Larger variances of the
accuracy accuracy reconstructed points Error propagation has Error Error propagation has an an extrapolation behavior interpolation character character No need for block structure Ideal for corridor mapping like mapping of power lines or Other Block structure transportation corridors Mandatory for digital line sensor technology (laser scanners, airborne line scanners).
Table 3.15: Comparison of the three major georeferencing techniques
82
In summary, it is important to understand that due to the lack of feedback in the direct georeferencing model, any sensor calibration error will directly introduce error in the geolocation error budget; most of them are amplified by flying height, such as the attitude errors.
3.4.2 The achievable accuracy of GPS/INS-based remote sensing systems
The overall accuracy of a geospatial data acquisition system depends on the individual sensor error characteristics and the interaction among the various system components. In a sense, the platform orientation accuracy sets the baseline as a lower bound. GPS/INS-based systems were discussed in a previous section, where performance tables were provided, and manufacturers’ specifications listed. In practice, however, these values seem to be frequently optimistic, as such specifications may not be available under real surveying conditions. One reason is that the accuracy terms are only derived internally; in other words, the EKF estimated variances are used, which are partially based on the sensor components’ physical parameters, which mostly refer to laboratory or test conditions. Furthermore, this accuracy strictly refers to the airborne remote sensing platform accuracy, which, as described earlier, is just one component of the overall error budget of the geolocation performance. As a general rule to achieve the highest possible accuracy, the GPS component should be as good as possible, including the ground component (single base station or network solution). Then if the high-quality GPS data are available throughout the whole flight, even a medium-range INS sensor can provide a good platform solution (provided the GPS data rate is sufficiently high). The spatial relationship between the georeferencing platform reference, the INS sensor, and any
83 onboard imaging sensors should be calibrated and the conditions reflecting the calibration maintained. For example, any disturbance to the sensor or mechanical flex could introduce significant errors that can be only detected with independent validation.
The error characteristics and the general performance of the imaging sensors have been of interest, and in particular, both digital cameras and LiDAR systems are extensively researched regarding their ultimate limits, as well as what can be realistically achieved in production. For digital cameras, there is an additional distinction between manual and automated processes with respect to feature extraction accuracy. In the first case, a typical measurement error, usually measured in image pixels, is used, while there is no generic characterization for automatically extracted points. A recent comprehensive analysis of digital camera and LiDAR systems can be found in (May Csanyi, 2007), and performance results for typical airborne surveys are illustrated in Figs. 3.12 and 3.13 for digital cameras and LiDAR, showing the contributions of the different sensors and system components. The assumed parameter error budgets are listed in Table 3.16.
Group Variables Error Terms Typical Values
1 Image coordinate measurements σx1, σy1, σx2, σy2 5 µm
2 Principal point shift σxo, σyo 4.5 µm
3 Focal length σc 9 µm Position σ , σ , σ 5-30 cm 4 Xo1 Yo1 Zo1 σXo2, σYo2, σZo2
5 Boresight misalignment σvb, σφb, σkb 7.5, 7.5, 15 arcsec
Attitude angle σ 1, σ , σ 1 10, 10, 20 arcsec 6 v φ1 k σv2, σφ2, σk2
Table 3.16: Different groups of the parameters with typical error values
84
(a) (b)
(c) (d) Figure 3.12: Accuracy analysis bar chart, for large format camera with high-performance georeferencing system for typical flying heights: (a) H=300 m, (b) H=600 m, (c) H=1500 m, (d) and H=3000 m
85
σXI σYI σZI σv σφ σk σv σφ σk σr σβ σXI σYI σZI σv σφ σk σv σφ σk σr σβ b b b b b b (a) (b)
σXI σYI σZI σv σφ σk σv σφ σk σr σβ σXI σYI σZI σv σφ σk σv σφ σk σr σβ b b b b b b (c) (d)
Figure 3.13: Accuracy analysis bar chart for high-end LiDAR system with high- performance georeferencing system for typical flying heights: (a) H=300 m, (b) H=600 m, (c) H=1500 m, (d) and H=3000 m
86
The overall feature positioning performance is an area where it is rather difficult to provide generic numbers, as performance is usually defined as a combination of the robustness and accuracy for the methods, and thus, there is quite a variety among different techniques. For example, image matching, based on points, is relatively well- developed and many commercially available products are available for automated AT
(Toth and Krupnik, 1996). However, it must be noted that validation of the results is not obvious, as matching methods usually do not pick the same points that a human operator would do; furthermore, points may not represent any object points, such as corner point or points may not sit on the ground, etc.
87
CHAPTER 4
CONCEPT AND METHODS FOR IMPROVING THE ACCURACY OF ROAD
INFRASTRUCTURE MAPPING USING PAVEMENT MARKINGS
4.1 LiDAR data validation and performance analysis
The horizontal accuracy of LiDAR data was not a concern in the early use of this technology. The fact that unprecedented vertical accuracy (dm-level) could be obtained relatively easily satisfied the mapping market for a long time. In addition, the applications that fueled the LiDAR technology, such as telecommunications, did not even require the accuracy that was achievable with those early systems. In mapping, orthophoto production was the primary beneficiary of the surface data provided by the new sensor and the requirements for accuracy were not that stringent. The introduction of the LiDAR data created a few quality control and even service/product contracting issues.
To address these subjects, the American Society of Photogrammetry and Remote Sensing
(ASPRS) initiated an effort to create a recommendation document on LiDAR data, which interestingly dealt only with the vertical accuracy (ASPRS, 2004); new ongoing efforts include the horizontal accuracy too.
88
As the LiDAR market started to grow rapidly, LiDAR vendors could invest more into development, and soon LiDAR systems showed truly outstanding performance improvements. In less than five years, the pulse rate improved by an order of magnitude, and now 100 and 150 kHz systems are available (http://www.optech.ca/prodlatm.htm; http://www.leica-geosystems.com/us/en/lgs_57608.htm). More importantly, the ranging accuracy has increased substantially and now stands close to the level of static GPS surveys, i.e., 1-2 cm, and it is almost insignificant to the navigation error budget. In parallel to these developments, users’ expectations started growing; the target vertical accuracy for demanding LiDAR products started shifting from the one foot level to the sub-dm range. The performance of the newer LiDAR systems, combined with better operational techniques, opened the door toward applications where large-scale or engineering-scale accuracy are required. At this point the georeferencing error budget and, to a lesser extent, the sensor calibration quality, are critical to achieving design level accuracy (few cm). Using ground control is an effective way to compensate for georeferencing and sensor modeling errors; also ground control can provide for independent and highly reliable QA/QC processes. Unfortunately, ground control can be costly, and may present hazard to the field crew in particular, within the transportation network.
The horizontal accuracy is especially important in steep terrain where footprint size cannot be ignored, and depending on the relationship of the laser beam and the surface normal, the vertical accuracy could significantly deteriorate. Similarly, the horizontal accuracy gets worse. Fortunately, this is rarely the situation in transportation
89 applications, as the slope in the road network is strictly controlled and limited up to no more than 10%.
4.1.1. Existing methods: review, natural surfaces and LiDAR targets
QA/QC methods used in LiDAR practice always require the use of some reference surface and can be broadly categorized whether they provide for vertical only or combined vertical and horizontal accuracy assessments. In addition, the methods could be further classified whether they use available object space features or dedicated objects for ground reference. At the beginning, flat man-made surfaces, such as runways or parking lots were used to validate the vertical accuracy of the LiDAR data, and the if a shift was identified between the measured and reference surfaces, it was simply applied as a vertical correction to the point cloud. As a practice, after takeoff, and occasionally before landing, a few over-flights above the airport used in the mission were performed, so the LiDAR system could collect reference data before and after the survey. If the mapped area had significant elevation changes, then additional over-flights were performed at various altitudes. Obviously, these tests provided no information on the horizontal component; in fact, quite significant horizontal offsets could have remained undetected.
As the LiDAR technology improved and denser data became available, the need for better validation, including the horizontal component began to grow. Since the swath width of a LiDAR survey is relatively modest as compared to airborne image a sensor, the mapped area is typically flown in parallel strips. To provide for contiguous coverage,
90
the strips should have sufficient overlap, which means that based on the two datasets a relative accuracy comparison can be performed. The improved point density allowed for
an introduction of techniques that could automatically detect the strip discrepancies and
provide for corrections to eliminate them. The process, called strip adjustment, obviously,
cannot provide for an absolute measure of accuracy but certainly removes theses artifacts
from the final LiDAR product; in other words, the LiDAR-created surface looks smooth
and homogeneous. The popular TerraMatch product has been widely used to detect
surface discrepancies and parameterize them, so strip adjustment can be performed; the
technique is described in (Burman, 2000 and 2002).
The interest to further improve the LiDAR product characterization in absolute
term further grew with the introduction of LIDAR systems with significantly improved
performance. Due to the dense point cloud, horizontal features could be extracted for the
first time. For example, buildings with flat roofs were initially used as additional
reference objects, and the building outline extracted from LiDAR was compared to
accurate survey data. This approach resulted in true 3D accuracy characterization of the
point cloud with a generally small estimation error. In parallel, there were important
developments on the algorithmic side, as techniques based on sensor calibration were
introduced; note that the previous methods simply detected the discrepancies and
attempted their elimination on the data level. In fact, this group of technique is frequently
called data-driven (Jie and Toth, 2008).
Although, the use of more complex man-made objects represented a major
milestone in product validation, applications, requiring extreme accuracies still required
91
better performance validation. For example, road subsidence or earthquake research
mapping (Toth et al., 2006) needs cm-level accuracy, preferably in both horizontal and
vertical components, which is close to the ranging error of the LiDAR system. Obviously, only dedicated targets deployed in the mapped area can provide the potential for achieving these utmost accuracies. LiDAR-specific targets were introduced by the OSU
Center for Mapping (Csanyi and Toth, 2007). Fig. 4.1 shows a target used by the Ohio
Department of Transportation.
Target
Target
Figure 4.1: LiDAR-specific ground target
Subsequent projects further confirmed the feasibility of the methods. The 2005
San Andreas Fault LiDAR mapping project, supported by NSF included the survey of
about a 1,000 km section of the Southern California fault line. Targets deployed in
clusters along the fault line could confirm the height accuracy of the processed LiDAR
data with mean absolute error at the targets of E: 0.10 m, N: 0.23 m and H: 0.06 m. This
92 clearly proved that the careful planning of the survey, the extremely tight GPS control for the geo-referencing and the use of a state-of-the-art LiDAR sensor resulted in achieving the best accuracy possible (for the given sensor suite) (Toth et al., 2007). Another mapping project in the San Obispo area (2008) used the same concept with different target implementation.
In summary, the strip adjustment and the use of limited reference surface is the widely used general approach in production for QA/QC. In applications requiring high accuracy, however, the use of signalized targets provides an effective and highly accurate way to validate LiDAR data. This exceptional performance comes with a price tag; the deployment of LiDAR-specific targets is expensive and labor-intensive. Therefore, preference should be given to the use of objects which are already in place and can be measured by simple methods that do not impose too much requirements on the surveying crew. This leads to the concept of introducing the use reflective pavement markings for
QA/QC, which is the subject of this chapter.
4.1.2. Concept of using pavement markings (intensity and technology)
The foundation behind the idea of using pavement markings as ground control to support the QA/QC of LiDAR data rests on two main conditions. First, the general availability of the LiDAR intensity signal, which is essential to extracting location information over relatively flat surfaces, is expected. Second, a technology is required to quickly measure pavement markings at good accuracy, which is safe for the crew and presents limited hazard with respect to traffic.
93
The introduction of airborne LiDAR in the late 1990s was followed by a quick
proliferation of the technology, and LiDAR quickly became the primary surface data
extraction mapping technique. Yet, the general availability of the intensity signal is a
rather recent development. The primary driving motivation for using intensity signals in
the LiDAR production is simply the fact that the intensity data provide a quite good
visualization tool, long awaited by the users, who wanted to see what was behind the
point cloud (this was the reason for using video and small format cameras with LiDAR
sensors). Fig 3.7 shows a comparison between optical imagery and LiDAR intensity and
elevation data. It must be noted that the use of the intensity signal for feature extraction
has been limited primarily to research, as the intensity signal is a relative measurement,
which is quite different from the explicitness of the LiDAR range data, and thus the
automated processing presents more challenges.
Concerning the pavement marking measurements, the rapidly broadening use of
real-time GPS correction services, based on the use of the CORS (Continuously
Operating Reference Stations) network, provides the necessary infrastructure to perform
the survey of pavement markings in a quick and accurate manner. Practically, the surveys
of10-20 m stretches of road pavement markings can be accomplished in about 5 minutes;
performance terms were discussed in Chapter 2.
This work proposes a method to use road pavement markings as ground control to
assess the quality of the LiDAR data, as well as to improve the point cloud accuracy by
post-processing. The idea behind using pavement markings is that they are widely available on the road network, albeit their quality may significantly. Their spatial
94
distribution is nearly optimal, as they are placed nearly evenly on the road surface.
Equally importantly, road pavement markings have distinct reflective characteristics
relative to the road pavement. This reflective distinction is essential from the feature
extraction perspective, as the LiDAR elevation data is identical for the road surface
regardless whether it is painted or not. Figs. 4.2 a-c show a digital orthorectified image, a
LiDAR intensity and a LiDAR elevation image, respectively, of a road intersection
acquired simultaneously. The LiDAR point density was about 4 pts/m2 with a foot print
size of 15 cm. The pavement markings in the LiDAR intensity image are quite visible and
distinct from the pavement. Note that the quality of the LiDAR intensity image is inferior to that of the optical imagery. Clearly, the affect of coarser sampling and the larger
footprint is quite noticeable. Nevertheless, the extraction of pavement markings seems to
be feasible and, thus, they can be used as ground controls, provided they are surveyed,
and consequently can support the QA/QC processes of the LiDAR data themselves. Note
that this approach can improve both horizontal and vertical accuracy of the LiDAR data,
and provide for the first time a measure of the horizontal accuracy. As a result, it enables
an evaluation of the horizontal accuracy of LiDAR data, as well as it enables to quantify
the horizontal and vertical product accuracies.
95
Straight edge
Stop bar Curved edge (a) (b)
(c)
Figure 4.2: Pavement markings at a road intersection; (a) 4k by 4k digital image
(orthorectified), (b) LiDAR intensity (gridded), and (c) LiDAR elevation (gridded)
In summary, the proposed method based on using reflective pavement markings instead of the deployable LiDAR-specific targets offers the same accuracy performance potential but it is less expensive. There is no need for deployment of sizeable targets around the road, the surveying requirements are simpler, as measurement of the road
96 surface is easier and faster compared to the elevated targets, and the injury risk for the crew is much lower. A clear advantage of the pavement markings is that they can be reused in subsequent mapping missions, as long as their quality allows for it; in time, the pavement markings wear out due to traffic and weather. In addition, the availability of the pavement marking positions can significantly improve the traffic flow extraction processes, as it accurately confines the search space. Furthermore, based on the measurement of the pavement markings at the road edges, the inner lane separator pavement markings can be extracted, as their geometry is closely known. Again, using the LiDAR intensity data they can be extracted with high success rate. This, in turn, enables better vehicles extraction, which can provide for better information for traffic flow patterns.
4.2 QA/QC method using feature-based matching of pavement markings extracted from
LiDAR with ground control
This section describes the developed technique in detail. The primary objective is to support the LiDAR strip adjustment process to provide for seamless integration of strips into the final product. In contrast to existing methods, the proposed technique provides full three-dimensional and absolute accuracy estimation.
The overall concept is introduced in Fig 4.3 and includes four main components: extraction of pavement markings supported by survey data, modeling the shape of the pavement markings, fitting the shapes of the pavement markings to the reference, and analyzing the results to determine the transformation parameters for data correction.
97
These four tasks are somewhat independent and, therefore, can be used in other context
too. Similarly, operator measurements can replace the pavement markings extraction for testing or in situation when no reliable results can be obtained.
Processed LiDAR point GPS-surveyed ground cloud and intensity data control points
Extraction of pavement markings from LiDAR, based on intensity
Piecewise weighted PiecewisePiecewise least weighted squares least squares curve least squarescurve curve
ICP-based matching, establishing 2D/3D transformation
Analyzing results, based on magnitude and distribution of residuals, creating QA/QC report; if needed, deciding on the complexity of the transformation that will be applied to the LiDAR point cloud
Figure 4.3: General procedure for using linear features for improving the topographic and infrastructure LiDAR mapping (corridor mapping)
4.2.1. Extraction of pavement markings from LiDAR intensity signal
The objective of the pavement marking extraction is to identify the LiDAR points reflected off from the pavement markings that will serve as input to subsequent processing, such as curve fitting and matching. The pavement markings shown in Fig. 4.2 are quite visible but that is not always the case, as road surface material and quality, the condition of the pavement markings and other factors may reduce the illustrated sharp
98 contrast between the road surface and markings in the LiDAR intensity data. Therefore, a simple global threshold applied to the intensity to separate pavement markings is not a directly applicable method in general case. Furthermore, even for a given situation, such as the intersection shown in Fig. 4.4, where there is a good separation, there exists no single absolute threshold value; for example, LiDAR data acquired at different flying heights over the same area would have different intensity values. Hence, an adaptive method is proposed here, which is based on the statistical evaluation of various datasets.
The basic idea is to find a locally optimal threshold that will separate the pavement markings from the pavement, and the underlying assumption is that the relative relationship of the intensity values for different materials is generally preserved. It must be emphasized again that the availability of the reference data (GPS-surveyed representation of pavement markings) provides an enormous help to accomplish this task, as it defines a rather narrow search space for finding pavement markings. Note that as a potential follow-up, pavement markings without reference data can be extracted in nearby areas based on road geometry and parameters settings adjusted to the location.
In Fig 4.4, a threshold of 180 was applied, which was determined as optimal value to separate the two slightly overlapping intensity distributions of the road surface and all the other objects, including pavement markings and the grassy area. The peak at the low intensity values reflects pavement, while the peak at the medium range reflects grassy/soil area, and the high values come from pavement markings. Note as the number of pavement marking returns is relatively low, and there is no visible peak. Applying the search window defined by the reference data, clearly, the pavement markings can be
99
easily extracted. Note that the intensity values above 220 are quantized differently, so
there are gaps in the horizontal axis of the histogram above that value.
6 x 10 4.457 4000
4.457 3500
3000 4.457
2500 4.457
2000 4.457 1500 4.457 1000
4.457 500
4.457 2.949 2.9491 2.9492 2.9493 2.9494 2.9495 2.9496 2.9497 0 0 100 200 300 400 500 600 700 800 900 5 x 10
(a) (b)
Figure 4.4: Using a locally optimal threshold in a nearly ideal situation; (a) thersholded
image is shown in, and (b) the histogram of the original LiDAR intensity data
To develop a better understanding of the intensity signal behavior, LiDAR data
acquired by different systems and under various conditions were analyzed. The histogram
of the intensity signal was evaluated for areas of interest to this investigation, such as
road surfaces, pavement markings and grassy areas/soil that are typical around roads. The
areas were manually selected in the attempt to form a good representation of the three
object categories. Fig. 4.5 shows six histograms of road pavement that were selected from
three different sub-areas. Note that relatively “clean” road surfaces were identified by the
operator to achieve a good estimate of the intensity distribution. Note that all the asphalt
100
data are from one long mission, but selected from different areas with different road
quality.
The various histograms in Fig. 4.5 as well as additional test data demonstrated that the intensity varies by missions as well as within the same mission. An additional observation is that even along a short stretch of 100 m, the intensity values can noticeably change. Also, the distributions show some variations; note the fact that fresh asphalt may
not provide any returns. The mean of the samples varies over the range of 82 and 128,
and variance is in the range of 6.00 and 11.63, see Table 4.1.
Sample Mean Std Min Max # of points (a) 82.46 6.05 58 103 243 (b) 88.44 6.00 73 105 178 (c) 120.58 11.63 97 170 604 (d) 128.92 11.22 100 159 420 (e) 87.15 7.86 69 106 256 (f) 96.11 8.10 74 126 496
Table 4.1: Road pavement sample statistics of LiDAR intensity values
101
25 15
20
10 15
10
5
5
0 50 60 70 80 90 100 110 0 70 75 80 85 90 95 100 105 (a) Area 1, sub-area 1 (b) Area 1, sub-area 2
70 35
60 30
50 25
40 20
30 15
20 10
10 5
0 0 90 100 110 120 130 140 150 160 170 100 110 120 130 140 150 160 (c) Area 2, sub-area 1 (d) Area 2, sub-area 2
15 45
40
35
10 30
25
20
5 15
10
5
0 0 65 70 75 80 85 90 95 100 105 110 70 80 90 100 110 120 130 (e) Area 2, sub-area 3 (f) Area 3, sub-area 1
Figure 4.5: Histograms of various pavement areas (asphalt)
102
Grassy and soil areas were analyzed in the next step. Fig. 4.6 shows four grassy/soil areas, and Table 4.2 summarizes the statistical parameters. The results show a clear separation from the pavement.
Sample Mean Std Min Max # of points (a) 286.88 27.85 200 380 1673 (b) 261.62 18.51 200 320 469 (c) 324.09 18.77 280 400 1474 (d) 282.89 21.68 211 360 1259
Table 4.2: Grassy and soil area sample statistics of LiDAR intensity values
In the last step, pavement markings were evaluated; the results are in Fig. 4.7. and
Table 4.3. Unfortunately, the histograms of the pavement markings are overlapping with that of both the pavement and the grassy/soil areas, with more overlap with the second group. This is a bit contradicting to the expectation, as the dark road surface and bright pavement markings should be at the opposite ends of the intensity range. The reason why the pavement markings intensity distribution has unexpectedly lower values is explained by the specifics of the spatial sampling of the LiDAR data. Due to beam divergence, the
LiDAR pulse has non-negligible footprint, ranging from a few cm to close to a meter, depending on the flying height and sensor aperture. Therefore, the LiDAR footprint is generally larger than the width of a typical pavement marking, which means that the reflection will jointly come from both areas (the pavement marking and the pavement), and thus, the final intensity value is a combination of the high intensity return from the
103
pavement markings and the low intensity return from the pavement, proportional to the
footprint overlaps of the two features. Fig. 4.8 shows the illustration based on the actual
LiDAR data.
500 250
450
400 200
350
300 150
250
200 100
150
100 50
50
0 0 200 220 240 260 280 300 320 340 360 380 200 220 240 260 280 300 320
(a) Area 1 (b) Area 2, sub-area 1 700 500 450 600 400
500 350
300 400 250
300 200
150 200 100 100 50
0 0 280 300 320 340 360 380 400 200 220 240 260 280 300 320 340 360 (c) Area 2, sub-area 2 (d) Area 3
Figure 4.6: Histograms of various grassy and soil areas
104
Sample Mean Std Min Max # of points (a) 228.19 38.33 183 320 42 (b) 295.97 95.20 181 540 47 (c) 356.51 108.28 182 560 41 (d) 221.39 29.42 184 300 33
Table 4.3: Road pavement marking sample statistics of LiDAR intensity values for points with intensity value larger than 180
50 60
45 50 40
35 40 30
25 30
20 20 15
10 10 5
0 0 50 100 150 200 250 300 350 50 100 150 200 250 300 350 400 450 500 550
(a) Area 1 (b) Area 2, sub-area 1
40 15
35
30
10 25
20
15 5
10
5
0 0 50 100 150 200 250 300 100 150 200 250 300 350 400 450 500 550 600
(c) Area 2, sub-area 2 (d) Area 3
Figure 4.7: Histograms of various pavement markings, including pavement areas; points
with intensity value larger than 180 are considered as pavement marking points
105
420 440 420 280 280 400 440 380 420 280 260 460
Figure 4.8: LiDAR points (blue) and road centerline (green); intensity values are shown numerically
The LiDAR data in the case of Fig. 4.8 were acquired at a point density of 4-5 pts/m2, with a flying height of 500 m, and the footprint size was approximately 15 cm.
Note that the varying intensity information can be used in the curve fitting process as
weight, or at least a weight factor can be associated with the intensity values. This way
the point closeness with respect to the pavement markings can be used in the adjustment
process.
An additional attribute of the intensity signal is that there is no agreement in the industry concerning the definition and the suggested range of the intensity value; note the
106 different range of the figures above. Furthermore, the formation of the intensity value could be different and is not necessarily proportional to the relative energy of the reflected pulse, as it could additionally include another value, which is an expression of the shape or change in shape of the returned pulse. Therefore, it is not a surprise that different LiDAR manufacturers use different intensity ranges, such as [0-255] or [0-
4095]. To further complicate the case, some systems come with AGC (automated gain control), so the receiver electronics adjusts the absolute intensity range according to slowly changing average signal strength, and provides intensity values relative to that value in the same intensity range.
Based on the histogram analysis and the impact of the LiDAR footprint, a method was developed that is not only adaptive but brings in the object space constraints in terms of applying the shape of the pavement markings to improve the point selection. Fig. 4.9 shows the block diagram of the proposed method. Note that some components, such as curve fitting and ICP (Iterative Closest Point) are discussed in the subsequent sections.
The process starts once the search spaces around the pavement markings have been extracted. The survey data of the control features, pavement markings, are provided as point observations along the centerline of the markings. Pavement markings can be easily surveyed using GPS VRS technology; the process is fast, typically it takes less than a minute to survey a point, and the accuracy, in general, is about 2-3 and 3-6 cm horizontally and vertically, respectively. In most cases survey data are available for the pavement markings along the edge lines, which can be very quickly and efficiently surveyed. The overall accuracy of the LiDAR system can be estimated from the
107 sensor/system and the flying height parameters. Adding a margin, the maximum error envelops can be computed, and thus, LiDAR points in the vicinity of the pavement markings can be extracted; the typical distance value to the reference curve is about 1 m or less in most situations.
Extracting the area around the curve fitted to the Control Points Results in the set of PointsWithinBoundary MaxIntensity = max (Intensity of PointsWithinBoundary) i = = 1
Threshold: PointsWithinBoundary with threshold = (MaxIntensity – i)
ICP on thresholded points
Compute residual for each point, and the mean of residuals
no Mean Residual < Predefined Value Stop yes Outlier detection: Residual of the point > Predefined Value
yes Was outlier found? Remove Outlier no
i = i + 1
Figure 4.9: Data processing block diagram of extracting LiDAR points of pavement markings
108
The method was implemented in Matlab, and a large number of tests were
performed to assess the performance of the extraction process; note that other modules
developed to support the curve fitting and the ICP processes were also used. Fig 4.10
shows how the mean residual of the selected points with respect to the reference drops at
the optimal intensity value.
Although with higher intensity threshold the mean residual continues to decrease,
this improvement is minor and more importantly this solution is less stable, as the
number of selected points will go down too, as shown in Fig. 4.11. Therefore, the final
threshold will be selected when the mean residual falls bellow a predefined value, to
assure that the good residual value is based on a reliable number of points. The threshold
value of the allowed individual residuals used in the block diagram is determined based
on the pavement width and footprint size.
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0 0 50 100 150 200 250 300 350 400 450 500
Figure 4.10: Mean residuals as a function of the intensity threshold
109
450
400
350
300
250
200
150
100
50
0 0 50 100 150 200 250 300 350 400 450 500
Figure 4.11: Number of selected points vs. the intensity threshold
The results of a typical pavement markings extraction are shown in Fig. 4.12; green points are the extracted points based on the search window, defined by the estimated maximum error envelope, red points are the reference points from the GPS survey, and blue points are the identified LiDAR points reflected from the pavement markings.
In the developed method there was no assumption on the distribution of the
LiDAR points with respect to the orientation of the pavement markings. In reality, however, LiDAR missions over roads are typically flown along the road line, so in most cases the LiDAR profiles, points collected in one swing of the oscillating (or rotating) mirror, fall across the main direction of the pavement markings. This geometrical
110
constraint can be exploited, as one-dimensional search techniques can be applied. Note
that spacing is usually denser along the profile lines. The footprint of the profile for short distances can be always modeled as a straight line, so it can be easily intersected with the pavement markings centerline. The conjugate LiDAR point of the intersection does not exist in general, but analyzing the intensity profile along the profile, such as fitting a curve and determining the maximum, can provide the matching virtual LiDAR point.
This approach leads to a slightly different processing, since a point-to-point correspondence is established right away, and thus, the transformation between the
LiDAR and reference points can be directly computed. This avenue was not further investigated in this effort.
6 x 10 4.4574
4.4574
4.4574
4.4574
4.4574 2.9619 2.962 2.9621 2.9622 2.9623 2.9624 2.9625 5 x 10
Figure 4.12: Extracted pavement marking LiDAR points
4.2.2. Curve fitting-based modeling of pavement markings
The primary purpose of the curve fitting process is to capture the shape of the
pavement markings, which can be equally applied to automatically extracted LiDAR
111
points, as well as to the reference points obtained by GPS surveying. Of course, there is a
significant difference in the two representations, as the reference points are quite
accurate, in fact, they can be considered almost error-free if compared to the horizontal
accuracy of the LiDAR points. In addition, their sampling is probably adequate to
properly describe the shape as the surveyors know quite well what point density is
required for proper representation of linear features. In contrast, the pavement marking
points extracted from LiDAR are distributed over a larger range in both directions, along
and across the pavement marking centerline. Therefore, finding a curve that represents an optimal fit in some terms is a challenge. The higher sampling rate (LiDAR point density) has a positive impact on the curve fitting process, as better error cancellation can be
expected. The section describes the curve fitting method that was developed based on the
original concept introduced in (Ichida and Kiyono, 1977). The technique (basic idea) was
adopted, modified and extended to the specifics of the LiDAR point cloud and control
points. An additional contribution is the introduction of a local coordinate system for
local curve fitting, which allows handling any curve shapes, not just the ones that can be
described by functions. The fitted curve can be described both analytically and
numerically, such as a dense polyline representation, which could offer performance
advantages in certain implementations.
The shape defined by the extracted LiDAR points of the pavement markings
should be modeled as linear features in order to be matched with their controls; the most
generic format is a 3D curve. The selected method to create best fit lines/curves through
the extracted LiDAR points is a piecewise weighted least squares curve fitting based on
112 cubic (third-order polynomial) model, which seemed to be adequate for our conditions, based on a priori experimental results. In the following, the 2D case will be discussed, although the implementation is based on the full 3D model. To handle any kind of curves, defined as the locus of points f(x, y) = 0, where f(x, y) is a polynomial, we proposed to perform the curve fitting for smaller segments in local coordinate systems, which are defined by the end points of the curve segments. The primary advantage of using a local coordinate system is to avoid problems when curves become vertical in the mapping coordinate system, i.e., when there are multiple y values for a single x value. Fig. 4.13 shows the concept of the local coordinate system used for curve fitting. The fitting results as well as the fitting constraints are always converted forth and back between the local and mapping coordinate frames. In the following, the core curve fitting for a single segment in a local coordinate system is discussed.
Local coordinate frame
Mapping frame
Figure 4.13: The curve fitting in the local coordinate systems; the local coordinate system is oriented to the main direction of the segment
113
The notation used to describe the main steps of the piecewise cubic fitting (PCF)
process is introduced in Fig. 4.14. To achieve a smooth curve, the curve fitting to any segment is constrained by enforcing identical slope of the curves’ tangents at the segment connection points; in other words, the PCF polynomials are continuous together with their first derivatives at the connection points, such as x=s, x=t, etc. Equations (4.1)-(4.6)
st nd rd describe the third-order polynomial, with the constant, 1 , 2 , and 3 -order terms, ck, dk, ak and bk, respectively, and its first derivative for curve Sk, placing the coordinate
system’s origin to the connection point s:
2 3 S k (x) cs d s (x s) as (x s) bs (x s) (4.1)
2 mk (x) slope S k (x) d s 2 as (x s) 3bs (x s) (4.2)
Since
2 3 yS yk (s) S k (x s) cS d S (s s) aS (s s) bS (s s) cS (4.3)
2 mS mk (s) Sk (x s) d S 2 aS (s s) 3bS (s s) d S (4.4)
Therefore
2 3 S k (x) ys ms (x s) as (x s) bs (x s) (4.5)
2 slope S k (x) ms 2 as (x s) 3bs (x s) (4.6)
The constant and the 1st-order term of the third-order polynomial are equal to the value of the curve, yS, at the origin, as well as the slope of the curve’s tangent at the
114 origin, respectively. As the curve’s value and slope are kept fixed at the connection points, when computing the coefficients of the third-order polynomial piece, only the 2nd and 3rd order terms are the unknowns in the least squares adjustment, and the constant and the 1st-order terms are treated as constant (non-random) variables or fixed constraints, except for the first segment, when all the coefficients are treated as unknown values (see equations in Table 4.4).
Connection points
yS m S yt mt Sk+1(x)
Figure 4.14: Piecewise weighted least squares curve fitting method
The computation of the piecewise curve fitting includes the following steps:
1) aS and bS, the coefficients of the second and third order terms of the curve Sk are
estimated; the constant term, yS, and the coefficient of the first order term, mS, are
115
considered fixed, as they are known from the curve fitting from the previous
segment. In the adjustment, the points in interval (Δi1+ i+Δi2)k (past, present, and
future data points) are used. The fitted curve is employed for interval i.
2) The value, yt, and the slope, mt, of the fitted curve are computed at x=t (the next
connection point); these values are used as fixed constraints in the curve fitting for
the next segment.
3) Step 1 is repeated to process the next segment.
x s 2 3 S k (x) ys ms (x s) as (x s) bs (x s)
Unknown parameters are : as ,bs
Step 1 Least squares adjustment for points in the interval ( i1 i i2 )k Estimated parameters : ˆ ˆ aˆ s ,bs aS aˆ s ,bs bs 2 3 S k (x) ys ms (x s) as (x s) bs (x s)
x t 2 3 Step 2 yt Sk (t) ys ms (t s) as (t s) bs (t s) 2 mt Sk (t) ms 2 as (x s) 3bs (x s)
x t 2 3 S k 1 (x) yt mt (x t) at (x t) bt (x t)
Unknown parameters are : at ,bt
Step 3 Least squares adjustment for points in the interval ( i1 i i2 )k 1 Estimated parameters : ˆ ˆ aˆt ,bt at aˆt ,bt bt 2 3 S k 1 (x) yt mt (x t) at (x t) bt (x t)
116
When the orientation of the subsequent local coordinate system is different from the previous one, the constraints (value and slope) at the connection point need to be computed in the new system. Fig. 4.15 shows the notation used in the computation, described by equation (4.7), and the transformation between local and global frames is defined by equation (4.8).
Local coordinate frame 1
01 Local coordinate frame 2
02 01 2
1 Mapping frame
Figure 4.15: Transfer of slope at connection points
Least Squares estimated parameters:
ˆ ( AT P A )1 AT P y
where A is the design matrix, P is the weight matrix, y is the vector of
observations.
Design matrix:
Unconstrained case:
117
2 3 1 y1 y1 y1 2 3 1 y2 y2 y2 A . . . . . . . . 2 3 1 yn yn yn
When value and gradient are fixed at x = 0:
2 3 y1 y1 2 3 y2 y2 A . . . . 2 3 yn yn
Table 4.4: Least squares parameter estimation of a third-order polynomial
Slope in Frame 1 Slope in Frame 2 :
m2 tan( 2 ) tan(1 01 02 ) (4.7)
where 1 atan(m1 ) *180 / pi
Where
m1 The slope of the curve’s tangent at the connection point in the local coordinate system 1
m2 The slope of the curve’s tangent at the connection point in the local coordinate system 2
1 The angle between the curve’s tangent and the x axis of the local coordinate system 1
2 The angle between the curve’s tangent and the x axis of the local coordinate system 2
01 The angle between the axises of local coordinate system 1 and the mapping frame
02 The angle between the axises of local coordinate system 2 and the mapping frame
118
Frame 1 Mapping Frame : X R 1 X T 0 01 1 01 (4.8) Mapping Frame Frame 2 : 1 X 2 R02 (X 0 T02 ) R02 (R01 X 1 T01 T02 ) where
X0 The coordinates of the fitted curve at the connection point in the mapping frame X1 The coordinates of the fitted curve at the connection point in local coordinate system 1
X2 The coordinates of the fitted curve at the connection point in local coordinate system 2
R 01 The rotation matrix between the mapping frame and local coordinate system 1
R 02 The rotation matrix between the mapping frame and local coordinate system 2 T 01 The translation in the mapping frame between the origins of the mapping frame and local coordinate system 1
T 02 The translation in the mapping frame between the origins of the mapping frame and local coordinate system 2
The discussion so far has considered all the LiDAR points with the same weight, which ignores the possible differences among the LiDAR points. While this model provides good results in most cases, improvements can be expected if the LiDAR points are weighted according to their location with respect to the pavement marking. For example, there is quite a difference in shape between a regular lane pavement marking and a stop bar, as the letter one has a larger width that is comparable to its length; consequently, the estimation of the two lines could be quite different. The answer to the question on what basis the LiDAR points can be weighted is the intensity value. As shown in Fig. 4.8, with varying overlap between the pavement marking and the LiDAR point footprint, the intensity value is somewhat proportional, as it was discussed earlier.
119
The simplest way is using the intensity value directly, or a non-linear mapping function.
Note if the pavement marking is long enough, which is not the case of a stop bar, and then there is a statistically good distribution of the intensity values as well as good spatial distribution of the LiDAR points, the intensity-weighted and equally weighted solutions will be similar.
As the characteristics of the LiDAR points of pavement markings and the surveyed control points differ from each other, we need to slightly modify the parameter settings of the piecewise curve fitting for the control points. After evaluating the results of curve fitting with different settings for control points, the following conclusions can be formulated. The control point distribution is rather even along the pavement markings and because the accuracy of the control points is high (compared to the LiDAR points), an adequate description of the shape can be achieved just by constraining the continuity at the connection points and there is no need to constrain the gradient. Furthermore, tests showed that a good solution is always achieved if the curves were fitted to shorter segments, formed from four consecutive control points. Other curve fitting techniques, including splines and higher-order polynomials, were also considered, but they seemed to be less efficient for our purpose.
The curve fitting described in this session has been implemented in Matlab. The results obtained by applying it to a variety of pavement markings samples, including both
LiDAR extracted and reference points, have indicated a good performance in terms of robustness and accuracy. Fig. 4.16 shows fitted curves, using the piecewise least squares curve fitting.
120
Figure 4.16: Curves fitted to LiDAR (blue) and GPS reference (red) points; the length of the curve is about 15 m
4.2.3. Matching pavement mark with reference
Various matching techniques were considered for matching the different representations of pavement markings, before the Iterative Closest Point (ICP) algorithm
(Besl and McKay, 1992; Madhavan et al., 2005) was selected. The primary reason for selecting ICP is its robustness, and the fact that it does not require any point correspondence. Using the polyline curve representation of the curve fitting results, the
ICP matching of free-form curves can be directly applied; the two point sets, adequately describing the curves, have no point-to-point correspondence. ICP can be applied in any dimensions, in 2D or 3D, and the correspondence between two curves is iteratively established, as well as the transformation parameters of the geometrical model are estimated. ICP is sensitive to initial approximation and outliers. A modification was proposed to the standard ICP method to deal with the various lengths of the different representations of the same pavement marking that could lead to false results when
121 searching for correspondences between point sets, if not properly treated. In the developed algorithm, this situation is properly handled, and thus, unacceptable errors are avoided.
For the sake of simplicity, the ICP technique will be discussed for the 2D case, as the generalization to 3D is straightforward. Note that in our application this is generally the case, as the road surface is almost flat, although not necessarily horizontal but for smaller areas can be almost always modeled by a plane. Also, the ICP can determine different models of the geometrical relationships between two data sets, but in our case, only the rigid body model is considered, as any deformation between the two data sets can be ruled out.
The ICP method in 2D is used to find the best correspondence between two curves (point sets) by iteratively determining the translations and rotation parameters of a
2D rigid body transformation. The ICP algorithm can be summarized as follows:
1) Establish correspondence between pairs of features based on their proximity.
2) Estimate the rigid transformation that best maps the first member of the pairs onto
the second one, based on minimizing the following expression
2 min (R,T ) M i (RDi T ) i
where R is a 2*2 rotation matrix, T is a 2*1 translation vector and subscript i refers
to the corresponding points of the sets M (model) and D (data).
3) Apply the estimated transformation to all features in the first/previous structure.
4) Repeat steps 1 – 3 until convergence is reached.
122
ICP assumes that the closest points are in correspondence, and during the repeated processing, the data sets get closer and closer, and ultimately, they converge to the correct answer. The 2D rigid body transformation used in the discussion can be described with three parameters, two translation, and one rotation, as shown in equation (4.9) and equation (4.10) where the transformation matrix includes both the translation and rotation components in homogenous format.
XC X cos sin X D (4.9) Y Y sin cos Y C D
XC cos sin X X D Y sin cos Y Y C D (4.10) 1 0 0 1 1
where XC and YC are the mapping coordinates of the control feature points; XD and YD are the mapping coordinates of the conjugate feature points; X and Y are the translation parameters between the two sets; and is the rotation angle.
The ICP method is rather robust, provided that good approximations are available to start the process. This is certainly the case for the pavement markings, as the two representations of the same linear feature are quite close in geometrical terms. However, the relatively coarse sampling and varying overlap needs additional attention, as the potential for a mismatch cannot be ruled out. To avoid these situations, two extensions were added to our ICP implementation. First, one curve was represented in a very dense polyline structure; in actual numbers, at 1 mm sampling rate. This way, the closest points
123 needed to form the pairs are guaranteed to be from the closet point on the curve. Second, in a preliminary analysis, a threshold was estimated to provide a maximum value for distances between the two point sets. Applying this threshold to all pairs, point pairs with excessive distance, which are likely to be erroneous, are eliminated from the processing.
The ICP method, customized to the matching of different pavement marking representations was implemented in Matlab and has been tested with both simulated and real data.
4.2.3.1 ICP performance test with simulated data
To assess the performance potential as well as the implementation correctness of the ICP methods, various tests were initially executed using simulated data. These tests were aimed to determine the effectiveness with respect to shape and noise content. Given the typical shape of the pavement markings, straight line, 3rd order polynomial curve, and sine wave were selected for our investigation. Points from the selected four curve types were sampled and then transformed to create the test curves. Then the ICP method was used to recover the introduced transformation based on the original and transformed curves.
Straight lines are probably the most frequently occurring pavement markings, and so they are of high importance. Furthermore, the matching between two straight lines is an ill-posed problem due to the uncertainty in one direction (along the line), resulting in infinite number of solutions for the shift parameters. Consequently, using only straight lines to determine the parameters of the 2D transformation, a group of lines with various
124 orientations is needed to counter the deficiency of matching lines. Fig. 4.17 shows an example matching two straight lines, where the yellow line is the reference, the dense polyline representation, and then the blue points are the points to be matched to the curve, red points are the results of the ICP, and the green points show the results of each iteration step. Note the difference in orientation between the two lines.
Figure 4.17: Matching straight lines; yellow: reference, blue: points matched to the curve, red: results of the ICP, and green: iterated points
125
(a)
(b)
Figure 4.18: Matching third-order curve from two different initial positions, (a) and (b); yellow: reference, blue: points matched to the curve, red: results of the ICP, and green: iterated points
126
Fig. 4.18 shows a curve modeled by a 3rd order polynomial and two different point sets that were ICP matched; the same color scheme is used to mark the point status. Clearly, the shape of the curve, varying curvature, helps to the ICP method to properly match the point sets to the curve.
More complex curves are rare in practice and can come in variety of shapes, so they have no typical shape. Therefore, a sine wave, approximating a good spatial point distribution, was simulated, as shown in Fig. 4.19.
The numerical results of these examples are listed in Tables 4.5 and 4.6.
Figure 4.19: Matching a sine wave; yellow: reference, blue: points matched to the curve, red: results of the ICP, and green: iterated points
127
Transformation parameters Number [m, m, deg] Curve type of Reference ICP results Difference iterations , , x, y, x, y, x y (Reference–ICP) 1.5 0.3655 0.0080 1.1345 Straight 3 1 -0.5392 0.0040 1.5392 line 3 2.9986 0.0002 0.0014 3rd order -1.5 -1.5020 0.0028 0.0020 polynomial 47 1 0.9937 0.0025 0.0063 (a) 2 1.9930 0.0001 0.0070 3rd order 1 0.9822 0.0063 0.0178 polynomial 54 1 0.9801 0.0057 0.0199 (b) 3 3.0052 0.0003 -0.0052 0.3 0.2986 0.0005 0.0014 Sine wave 7 0.1 0.1154 0.0006 -0.0154 2 1.9994 0.0001 0.0006
Table 4.5: Transformation parameter recovery based on ICP matching
Coordinate difference (Reference – ICP result) [m] 3rd order 3rd order Straight line Sine wave polynomial (a) polynomial (b) dx dy dx dy dx dy dx dy Mean -1.036 -1.554 0.002 0.003 -0.0025 -0.0030 -0.0001 0.015 Std 0 0 0.002 0.001 0.002 0.001 0.00005 0.00001
Table 4.6: Coordinate difference statistics at sample points
Based on the results in the Tables 4.5 and 4.6, the following can be stated. For the straight line, a correct transformation is found, but, obviously, it is not the original one, as in one direction the match is not defined due to the lack of shape in that direction. In general, good match was found with high accuracy in all the cases. As expected, the sine
128 wave provided the best match and lowest number of iterations due to its most distinctive shape. The simulation data based tests confirmed that the ICP method works well for all the typical shapes used in the pavement marking practice; obviously, it works better for curves with stronger shape characteristics, defined as good 2D spatial extent of the shape.
Fig. 4.20 shows an example when random noise was added to the straight line for the ICP testing. Noise is not expected to be an issue for the ICP method as the curve fitting has a smoothing effect and the least squares adjustment also minimizes the impact of noise. Table 4.7 shows results when various amounts of noise were added to the straight line case.
Figure 4.20: ICP result when noise was added to the data; yellow: reference, blue: points matched to the curve, red: results of the ICP, and green: iterated points
129
Coordinate difference (Reference–ICP) Noise Num of [m] Std [m] iterations Mean (dx, dy) Std (dx,dy) -1.036 0 0.00 3 -1.554 0 -1.055 0.011 0.05 3 -1.545 0.007 -1.064 0.016 0.10 3 -1.541 0.011 -1.075 0.024 0.20 3 -1.536 0.0164
Table 4.7: ICP performance with respect to random noise, straight line case
The simulation results clearly confirmed that the noise tolerance of the ICP method is remarkable; the mean coordinate differences were practically unchanged and only their accuracy terms changed accordingly to the noise amount, as expected.
4.2.3.2 ICP performance test with real data
From the logic of the ICP method, the original points describing the same pavement marking are not properly sampled. Note this should not be mixed with the sampling theory, as the point density for the usual LiDAR data and ground control certainly satisfies the Nyquist criterion. The problem for ICP is that when selecting the closest point from one set to a point in the other set, most likely the chosen point would be the closest one only from the set, but not the actual closest point in the terms of the
130 curve described by the point set. Fig. 4.21 illustrates the situation, why the curve fitting is needed to find the actual closest point. The closest point from the original point set would give a false closest point.
- control points - digitized point
Virtual matching point Closest point from the set of the surveyed control points
Figure 4.21: The effect of point sampling on ICP.
By densifying one point set, the control points in our case, the ICP will match to the nearly correct point, as instead of the distance between pairs formed from the original points, the distance to the interpolated point will be considered; the one from where the line in the perpendicular direction to the curve contains the digitized point. This is the reason why the option for dense polyline representation in the curve fitting was introduced. Another alternative could be if both curves were fitted and have dense representation. Obviously, this requires significantly more computing power, as the point pair formation will drastically increase.
To investigate the various options with respect to robustness, accuracy and execution speed, the following three combinations were considered:
1. Both, LiDAR and GPS control points are curve-fitted before running ICP; it is
invariant which one is used as reference and as data.
131
2. Only GPS control points are curve-fitted before running ICP.
3. Only LiDAR points are curve-fitted before running ICP.
[m]
[m]
Figure 4.22: ICP matched curves; magenta: curves fitted to control point; red: GPS control points, cyan: LiDAR point and curves fitted, and blue: matched points
In the test data, shown in Fig. 4.22, the LiDAR point spacing varied in the 1-3 pts/m range, and the horizontal accuracy of the GPS-surveyed points, provided by a VRS system was 1-2 cm horizontally. The transformation parameters between these two point sets (the original LiDAR-derived points and their corresponding points on the control curve) are calculated in a least squares adjustment. Table 4.8 shows the 2D transformation parameters for the three different cases, clearly indicating the robustness
132 of the ICP method with respect to noisy data, such as using the original LiDAR points.
The differences between curves and residuals after ICP matching for the three cases are shown in Table 4.9. The 2 cm horizontal precision is realistic, given the fact that the
GPS-surveyed points are known at a 1-2 cm-level accuracy, and the LiDAR-based pavement marking positioning accuracy is estimated at the few cm range. The 9-10 cm precision terms in case 2 correspond to the use of the noisy LiDAR data (no curve-fitting applied to smoothly model the pavement markings). The curve fitting process has a low- pass filtering effect on the curve representation, which is more significant for the LiDAR point data, due to its data sampling characteristics.
ICP-adjusted ICP input data transformation parameters X [m] Y [m] [] Both, LiDAR and GPS points are 0.153 -0.114 0.000 curve-fitted No fitting of LiDAR points, GPS 0.150 -0.114 0.000 points curve-fitted No fitting of GPS points, LiDAR 0.158 -0.116 0.000 points curve-fitted
Table 4.8: Transformation results (2D)
Differences/Residuals X [m] Y[m] Case Before After Before After mean Std mean std mean std mean Std 1 0.16 0.02 0.00 0.02 -0.11 0.02 0.00 0.02 2 0.16 0.10 0.00 0.10 -0.12 0.09 0.00 0.09 3 0.16 0.02 0.00 0.02 -0.12 0.01 0.00 0.01
Table 4.9: Original differences and residuals after ICP (2D) between corresponding
LiDAR and reference points
133
To carry out a performance test of the proposed method, a typical intersection was selected from a recently flown LiDAR survey, where GPS-surveyed pavement markings with 1-2 cm horizontal accuracy were available, provided by the Ohio Department of
Transportation. The results of the ICP-based curve matching for the curve lines at an intersection are shown in Fig. 4.23. Visually, the transformation shows a good fit; the blue points nicely fall on the GPS-defined curves.
Figure 4.23: Curve matching based on ICP in a 50 m by 50 intersection area; magenta: curves fitted to control points, red: GPS control points, cyan: curve points derived from
LiDAR, and blue: transformed curve points (derived from LiDAR)
134
The numerical results are listed in Table 4.10. The ICP removed the bias but the variances stayed unchanged as expected.
Table 4.10: The initial and final residuals for all the points and by features
4.2.4. Data validation and corrections
The QA/QC processes, in general, include the rigorous statistical analysis of the differences observed between the data and reference at certain areas, and then depending of the results, if needed, a corrective processing of the data can be performed. While the overall concept is well-understood, the actual implementation depends a lot on the data
135 specifics and error characterization. This section reviews the QA/QC process, as it is applied to the pavement marking-based ground control.
Ground control has been traditionally using point features for measuring differences between observed data and reference. Although, methods were developed to use linear features to form orientation of overlapped imagery in photogrammetry
(McGlone, 1986), this approach has not been used in practice. The main reasons are that earlier control features were manually measured, so it was easier to measure points, as opposed to lines and that somewhat unconventional computation was required. Recently, interest in using linear features has grown, as better feature extraction techniques can efficiently produce these features. From the pavement marking perspective, there are two strategies:
Point-based methods are used; during the matching, the virtual LiDAR points are
computed as correspondences of reference GPS-surveyed data, and thus, they can
be used in the computation.
Linear features are directly used; in which case either the analytical description or
the dense polyline representation is considered in the processing.
Factoring in all the important aspects of the implementation, the point-based technique was selected and implemented in this investigation. The main reason was that the linear feature-based techniques are directly applicable to straight line segments. Thus, either segmentation or a complex extension of the technique for allowing for free curve features would be required. Given the typical shape of the pavement markings, the point-based technique has a clear advantage in terms of straightforward computation and imposing no
136 restriction on the shape of the pavement marking (such as, that only straight lines can be used) and thus, it was selected for implementation.
The spatial distribution of ground control is important, regardless whether point or linear feature based techniques are used. As a general rule, an even distribution over the surveyed area is preferred. Furthermore, pavement markings with various orientations should be used. Points extracted from only parallel lane separator pavement markings do not satisfy this condition, as the connection between the LIDAR and reference data is undefined along the road direction. Therefore, it is important to have features with various directions in the area, such as a small group of perpendicular features in an intersection. A somewhat similar situation occurs when the area has an elongated shape, such as a road, where the extension along the road could be so long that there may not be a common transformation to model the discrepancies between the LiDAR and reference points. Therefore, larger mapping areas, such as a longer stretch of a road, must be segmented to shorter sections, which can be adequately modeled. In this case, the QA/QC process is applied independently to these local regions. Obviously, to achieve a good performance, sufficient number of pavement markings is required with good spatial distribution in every region.
In the discussion above, a 2D model was used, but the generalization to 3D is simple, as based on matching the linear features in the plane of the road, the height coordinates can be easily retrieved, and thus, the point-based transformation can be established in 3D, using all conjugate points of all the features.
137
The point-based transformation, as discussed earlier, can be data-driven or sensor model-based. Regardless of which method is used (note that the data-driven approach was selected in our investigation), a critical aspect is determination of proper weights to all observed values. Obviously, the GPS-surveyed reference point can be relatively easily characterized by the estimated variance terms from the GPS processing. Since the GPS data acquisition conditions are identical for a small area, all the points will have the same accuracy terms. In contrast, the LiDAR points determined from the intensity data have a varying error budget as they are obtained through several processing steps, including extraction of pavement marking points, then curve fitting and sampling, and finally interpolation to obtain the virtual equivalent of the control points. The estimation model of the LiDAR point accuracy in horizontal and vertical direction was developed based on:
Considering the LiDAR footprint, as the initial estimate of the horizontal error.
The relationship between the surface normal and laser beam.
Pavement marking point extraction performance.
Curve fitting results, primarily including the shape of the curve.
ICP transformation parameter estimation results.
Once all the observation is properly characterized by their error terms, the transformation parameters for the selected model can be adjusted and their accuracy terms can be estimated. Based on the model, the primary statistics is computed, including the residuals, i.e., the difference in X, Y and Z coordinates of the control feature points and the transformed feature points, and the average and the standard deviation of the
138 differences (dX, dY and dZ). Next, the variation of the residuals should be analyzed. This is of great importance, as it can shed light on the overall data quality and help identify local anomalies, including sensor performance variations, blunders in ground control, etc.
This analysis usually entails the investigation of normal distances, from the “best fit” line/curve to the control point, their spatial dependency, and the related statistical information on differences for all control points within the project area.
Once the rigorous statistical analysis has been completed, correcting steps could be applied to the data. Depending on the magnitude of the observed differences and their spatial distribution, a variety of corrections can be applied to the LiDAR point cloud to improve the point position accuracy. For example, if there is a similar vertical shift detected at the control features, a common vertical offset correction can be applied. If the amount of vertical shift detected varies by location and/or combined with non negligible horizontal differences, a more complex model, such as a 3D similarity transformation can be applied. Finally, if the differences are out of the usual range (gross errors), the process can indicate system malfunctioning.
The discrepancies identified between LiDAR data and ground control, also between overlapping strips, can be corrected in two fundamental ways. For data driven models, 2D or 3D similarity transformations are typically applied directly to the LiDAR point cloud. In most cases, the transformation parameters are different from strip to strip, so separate corrections are applied. In contrast, the data correction process is quite different for the sensor calibration-based methods, as these cases the sensor calibration parameters are updated, and the LiDAR point cloud is recreated based on the new sensor
139 model. Consequently, the corrections could be applied to several strips under favorable
(or ideal) conditions. In case, the besides sensor modeling errors, there are other anomalies, such as varying quality of the navigation solution, this approach will not provide optimal solution for the whole survey area, and therefore, data-driven or combined method could be applied. Depending on the extent of the changes in the sensor calibration parameters, the complete recreation of the LiDAR point cloud can be replaced in certain situation and a simple transformation can be applied to the point cloud data.
The pavement marking based QA/QC method described above can be combined with existing strip adjustment methods, such as with the popular TerraMatch product.
First, the strip discrepancies identified by TerraMatch can reduce the search space for the pavement marking extraction process. Then, the results of pavement marking based
QA/QC process can be passed back to TerraMatch to propagate them into the LiDAR point cloud.
140
CHAPTER 5
CONCEPT AND METHODS TO IMPROVE TRAFFIC FLOW EXTRACTION
FROM REMOTE SENSING DATA
5.1 Motivation and concept
The discussion in Chapter 2 has lead to the conclusion that vehicle extraction is becoming feasible using modern remote sensing technologies and this information could significantly improve the traffic monitoring infrastructure in two essential ways. First data obtained from topographic mapping, including city modeling, can provide vehicle distribution in a larger area at certain times, which if combined with local sensor data, can help develop better traffic models. Obviously, this approach is a no-cost opportunity for obtaining traffic data, as it requires only additional processing of the acquired data.
Second, smaller remote sensing platforms can be deployed on demand to collect data over areas of interest, where there is a need for immediate data. The economics for this solution is related to both sensor and platform performance, which, given recent trends, is already affordable, such as deploying small format cameras on small UAS platforms.
This section will address the feasibility of extracting vehicles and estimating their speed from LiDAR and image data to support the extraction of traffic flow information.
141
The primary focus is on the proper data characterization, including accuracy error terms of extracted vehicle features and derived velocity estimates.
5.2 Road extraction
Road extraction from remote sensed data is an important mapping process in current practice worldwide. Although the need for such data had existed for a long time, road extraction only began when terrestrial mobile mapping systems were introduced
(Bossler, 1992) and medium resolution satellite imagery became available. Earlier methods were primarily focused on extracting only the road centerline, while as technology improved the edge lines and, if possible, lanes were also extracted. The availability of such information is important for many purposes, besides the obvious visualization and routine use. First of all, the transportation safety aspects should be mentioned, as correlating traffic accident data with road geometry provides probably the most significant information for road design, including reconstruction, to reduce the accident rate; this topic is not discussed in this dissertation. Knowing the road geometry, including lanes, is an enormous help for the purpose of using remotely sensed data for traffic flow extraction, as it provides for an accurate search space for vehicle extraction.
Besides decreasing the processing time, it allows for better representation of traffic data, as lane-specific traffic information can be obtained, as opposed to aggregated information. Furthermore, lane changing statistics can be obtained, which could be of interest to transportation experts for better management and control.
The road extraction process started to include the road surface determination recently, as the requirements for better geometrical modeling increase. Clearly, the
142 improving sensor technology provides better observability. In addition, the use of geospatial data has been rapidly proliferating, and thus there is always some already existing geospatial information, GIS, practically available over every piece of the Earth.
In fact, multiple data coverage, including typically surface (DEM) and features (CAD) is available in several GIS layers at various scales for most industrialized nations. In the
US, the Geological Survey, USGS, provides basemap data for all the status, including a
DEM with 10 m spacing, land use classification data, vector data for natural and man- made objects, and orthoimagery with 1 m GSD. In addition, various global datasets are also provided by the USGS. These datasets include the Landsat satellite image database, which covers the globe at moderate spatial resolution and contains 30 years of data. On the surface side, the 2000 SRTM mission provides a uniform 30 m spaced grid, as discussed in Chapter 3. In addition, private databases, such as the IfSAR-derived surface models from Intermap (NextUSA, MapUK, etc.), are available continent wide. Local GIS and transportation databases typically provide large-scale data with varying currency and accuracy.
The availability of moderate resolution global and medium/high resolution local data in the developed world provides a good base for road extraction. In fact, in most cases, the task becomes updating the existing map data, which includes the detection of changes in the road network and improving the accuracy of the road geometry. In other words, the need for global search, searching for roads in the data, is negligible, and only local search methods, where the existence of the road is known within some accuracy range, should be considered in our discussion.
143
5.2.1 Road surface extraction from LiDAR
Extracting roads from LiDAR is a rather new approach (Hu and Tao, 2004). The road area extraction developed is based on using a combination of range and intensity data. The processing of the range data includes a local analysis of surface data, in particular the examination of the road cross-profiles. In parallel, the intensity data are first segmented, followed by the boundary fitting. Both processes are supported by additional information, such as centerline data available from CAD or GIS databases.
Once the road surface areas have been approximated using the elevation and intensity (if available) data, the results of both data sets are combined, and a final consistency check using object space constraints should take place to determine and delineate the road.
Since at this point the road direction and width are approximately known, the objective is to determine the edge lines of the road. As the changes in the road geometry are limited in the road direction, a similarity analysis is performed over smaller road segments with the length comparable to the road width. The overall concept is shown in Fig. 5.1
The LiDAR elevation data processing is based on two parallel approaches: (1) segmentation of LiDAR data to find flat surfaces, and (2) analyzing LiDAR scanlines to find straight line segments. In the first case, points are grouped and small surface patches are fit to them, which are described by a plane representation, see equation (5.1).
AX BY CZ D 0 (5.1)
Analyzing the normal vector, the patches can be marked as possible road segments. Depending on the quality of the GIS/CAD road data, the road slope
144
information should be used during the processing. If only horizontal data are available,
the normal vector should be split into two components, along the road and across the
road. The across the road component, which should describe a nearly horizontal surface,
should be given higher weight in the segmentation. The along the road components can
fluctuate more, but the rate of change should be consistent. Our implementation, the flat surface patches are extracted by using the Principal Component Analysis (PCA) technique; for a set of points in a surface patch, the covariance matrix of the local 3D vectors is computed and the analysis of its eigenvalues can easily reveal whether the area is flat, when the smallest eigenvalue is negligible with respect to the two others. This approach works well but is a bit computation expensive.
LiDAR Data
Elevation Intensity Centerline GIS / Cad Database ASF Analysis Segmentation
Combining Areas
Rolling Technique Filtering/Cleaning
Detection of the Left/ Right Road Edges and Medians
Figure 5.1: The road boundary extraction process
145
Since roads are usually surveyed with perpendicular scanlines in a typical corridor
mapping LiDAR mission, analyzing the road cross profiles as they are measured by
consecutive scans provides a good alternative for road detection, as changes in road
geometry are generally larger in that direction. In fact, this is the reason that roads are
conventionally modeled by cross profiles. The basic concept is finding flat segments of
the scanlines that correspond to road surfaces. There are several techniques for measuring
the roughness and roughness length of a profile such as auto-covariance, cross-
correlation, variogram, texture analysis and the fractal method (Thomas, 1999). The
correlation between points on a profile, as a random variable, is chosen and the auto-
covariance function or its modified equation called structure function is used in this
investigation. For a profile of length L, the structure function is defined in equation (5.2).
1 L S( ) z(x) z(x ) 2 dx (5.2) L 0
where, z(x) and z(x ) are pairs of height values separated by a distance . This function is often normalized as the auto-structure function, called ASF, see equation
(5.3).
S( ) ASF( ) (5.3) max(S)
Fig. 5.2 shows a few consecutive profile lines and computed ASF functions. Although the
ASF computation inherently implements some smoothing, a filtering of either the raw 146 data or the derived ASF function is recommended, as the LiDAR data usually come with noise; typically 5 cm RMS.
Figure 5.2: The road cross-profiles (top) and the computed ASF showing surface roughness (bottom)
Figure 5.3: Road estimation based on intensity segmentation
147
The LiDAR intensity data is a relative measure, thus segmentation can be only
based on the relative intensity values. Fig. 5.3 depicts a random sample data segmented
by a 40% intensity threshold value, showing a very good performance. Note the road
centerline points, marked with circles, from the GIS/CAD data. The points erroneously
segmented can be removed by either basic morphology processing or during combining
the results of elevation and intensity processes.
Fig. 5.4 shows automatically extracted road surface boundaries of a divided
highway (U.S. Route 40 in Madison County, Ohio); note that edge lines are smoothed.
Figure 5.4: Road edge delineation in LiDAR data
5.2.2 Road extraction from optical imagery
Methods for road extraction from remote sensed data are available and typically focused on linear feature extraction from satellite or airborne imagery. Earlier techniques were based on a global search (Heipke, 1996; Fua, 2001; Stephanidis, 2008). Newer 148
approaches typically work with reduced search space and primarily handle the problem as
a matching between two domains, such as between image and vector data. The topic in
general not subject of this investigation and only a minor suggestion is made here.
Since pavement markings, as discussed in the previous chapter, are surveyed by
GPS at high accuracy and also extracted from LiDAR, they can support the road extraction process at the available locations. For example, pavement marking can be projected into the image to identify search areas. Or based on co-registration of pavement
marks, a feature-based matching extracted from optical imagery and LiDAR can be
performed. Since the placement of pavement markings, including their relative location
and distribution is strictly controlled, the pavement marking used as ground references
can support the extraction of other pavement markings nearby. For example, knowing the
edge lines of a road means that the road width is known, which also suggests the number
of lanes, and thus, the likely locations of lane separator markings. As a final note, the
LiDAR intensity signal can be also valuable to register LiDAR and image data and
consequently, to help establish image scale and so forth. These topics could be valuable
directions for future research.
5.3 Vehicle extraction (static and mobile platforms)
5.3.1. Vehicle extraction from optical imagery
The task of automatically extracting vehicles from airborne imagery is a general
feature extraction and object recognition problem that is primarily addressed in computer
vision. A large variety of methods have been researched over years. Some of them have
been successfully used in the remote sensing practice (Lillesand et al., 2008). Here, a
149
short discussion is provided. A dedicated technique is also proposed to provide a simple,
yet robust solution for satellite and vertical airborne imagery.
Fig. 5.5 shows typical satellite and airborne optical imagery acquired over a busy
intersection. The usual vehicle extraction process includes edge-based feature extraction,
which is followed by grouping. Then object space constraints that are normally defined in
shape are applied, such as the rectangular footprint of vehicles, and scale, if available.
There could be several additional processing steps, not discussed here. Fig. 5.6 shows results obtained by a basic implementation.
(a) (b)
Figure 5.5: (a) 1-m satellite and (b) 15-cm airborne digital camera images, acquired over
the campus of University of Arizona, Tucson
In this example, the performance of the vehicle detection is clearly superior for
the airborne image; only one vehicle with a very dark tone is missed. The limited
150 resolution of the satellite imagery, however, results in moderate performance; there are more missed vehicles, and some objects are mistakenly identified as vehicles.
Encouraging results that demonstrate that vehicles can indeed be identified and classified accurately from imagery (McCord at al., 2002) have shown that the numbers of classified vehicles observed in satellite images can match those obtained from ground truth data.
Thus, vehicles could be counted and classified with reasonable accuracy.
(a) (b)
Figure 5.6: (a) Vehicles extracted from 1-m satellite and (b) 15-cm airborne images
The Scale Invariant Feature Transform (SIFT) matching is a sophisticated feature extraction method to detect and describe local features in images (Lowe, 1999 and 2004).
The technique is gaining popularity because of its excellent performance, although it is quite computationally intensive. The SIFT method attempts to extract features for image matching that are highly invariant to scale, rotation, affine distortion and intensity changes.
151
Therefore, in support of the research presented here a few new tests were
performed to investigate the feasibility of this technique with respect to aerial imagery.
Fig 5.7 shows two consecutive images with SIFT feature points marked in yellow and red
indicating matched features.
Figure 5.7: SIFT features extracted and matched from aerial imagery (Courtesy of N.
Markiel)
In these tests with SIFT, our interest was to investigate the likelihood that vehicles
are picked as features. A closer look at feature locations in Fig 5.7 reveals that the
vehicles will tend to be excluded from SIFT features. The possible reasons are that vehicles have a rather small footprint at the typical resolution of airborne imagery and the
fact that the SIFT algorithm is to exclude edge pixels, since they tend to be unstable. In
fact, additional tests with reduced vehicle footprint indicated stronger avoidance of
vehicles.
152
Analyzing the above vehicle extraction process from a photogrammetric point of view, a few observations can be made:
The extraction is image-based (2D) and uses only one image.
No scale information is applied.
Vehicle shape could be distorted to various extent related to camera orientation
and surface geometry.
The sensor orientation (or camera pose in computer vision terms) information is
not used.
Except for using the road location data to define the search space, no geospatial
data are used.
To support the vehicle extraction process and improve its performance, the use of orthoimage domain is proposed as a preprocessing before the actual vision-based processing is executed. Two images from the intersection image sequence with the center part of their orthorecitified versions are shown in Fig. 5.8. The orthorectification process is a well-understood and widely used technique in mapping (Krauss, 1993). The requirements include the availability of camera orientation and a surface model. Since modern remote sensing systems are predominantly based on direct georeferencing, the camera orientation data are always available. The surface data could be obtained from two sources: either existing DEM, such as USGS data, is used, or if LiDAR data are captured in the same flight, then they can be directly used. Obviously, the second solution provides a significantly better spaced and more accurate surface model. If the surface undulation is modest or small in the area, the surface model resolution is not critical, as 153 there is less impact on the orthoimagery produced. In fact, the relative surface accuracy defines the quality of the orthoimage. The benefit of moving the vehicle extraction into the orthoimage domain is quite obvious; most importantly, the vehicle shape in the horizontal footprint is preserved at true object scale. Under ideal conditions, there should be no differences in orthoimages created from images taken over the same areas, except for moving objects. Note that there is practically no difference between the orthoimages for static content in the bottom row in Fig 5.8. Consequently, subtracting orthoimages can provide an easy extraction method for moving vehicle detection, as shown in Fig. 5.9.
The proposed method is still an inherently single image-based vehicle extraction, as the orthorectified imagery is the input to the vehicle extraction. In contrast, using overlapped imagery, stereo technique and even multiple image-based vehicle extraction can be implemented, where complete 3D object extraction is possible. However, in these cases, there should be sufficient change in the camera position and attitude to allow for depth recovery, which is, for example, not the case for the sample imagery used so far. In addition, the complexity of the feature extraction process is also significantly higher, so this approach is currently probably not feasible for operational implementation.
154
Figure 5.8: Original images (top row) and their orthorectified counterparts (bottom row)
Although vehicle extraction from satellite imagery is not the subject of this investigation, it is important to note that there are some specifics to the sensor characteristics that could be exploited to obtain vehicle velocity estimates. This is important, as one of the main criticisms against satellite imagery is the limited temporal
155 resolution, the repeat pass time, which, obviously, does not allow for velocity estimates at all, as it is measured in days. The DigitalGlobe QuickBird satellite imaging sensors has independent pan and multispectral sensor components, as shown in Table 3.13. In the current implementation, there is a short time delay between the exposure of the two sensor components that allow for vehicle velocity detection, provided the vehicles can be extracted and successfully matched from the two different resolution data sets, which is a challenge. An initial investigation can be found in (Xiong and Zhang, 2008).
Figure 5.9: Differencing orthorectified images reveals moving vehicles
156
5.3.2. Vehicle extraction from LiDAR
A LiDAR point cloud offers explicit three-dimensional information of the object
space and consequently provides an excellent basis for shape-based feature extraction.
Furthermore, road surfaces and, to some extent, vehicles have simple geometry.
Therefore, vehicle extraction can conceptually be automated and a rather high performance level can be expected in open areas. Vegetation canopy over roads could pose some difficulty, although multiple returns from the LiDAR pulse can mitigate this problem to some extent.
If the road surface area is adequately determined, a simple thresholding of the vertical coordinate of the LiDAR data can extract the vehicles. In fact, the precise modeling of the road surface by a plane (for a road segment) is usually not needed. To follow the changes in road surface orientation, such as in mountainous areas, however, the thresholding scheme should be adaptive with respect to the road surface to guarantee that candidate points representing a vehicle will have true perpendicular height values with respect to the actual road surface (Grejner-Brzezinska et al., 2004b). In this way, the same vehicle description is obtained regardless of the grade of the roadway. Fig. 5.10 shows the vehicles and road outlines projected into a road surface defined plane; the data were captured in the San Andreas Fault line LiDAR mapping project in 2005 and shows a steep area of I-15 at the Canyon Pass. Besides the vehicles, there could be other extracted objects that are certainly not vehicles, such as vegetation or guide rails on the side, and thus should be removed during subsequent processing. A simple filtering based on determining the smallest envelope that includes all the laser points of a vehicle can
157
usually remove the vast majority of the non-vehicle objects. For more complex
processing, better vehicle parameterization is needed.
Figure 5.10: Vehicle footprints projected onto the road surface; white circles mark the
GIS centerline data
The 3D vehicle parameterization suggested in (Toth et al., 2003) is based on the fact that vehicle shapes change more in the vertical direction than horizontally, and thus, the vehicle footprint with a simple vertical profile along the vehicle main axis should provide a good representation. First, the 2D footprint of the vehicle must be computed.
Fig. 5.11 shows the results of thresholding the LiDAR points bounced off from a vehicle.
The task is to find an optimal fit of a rectangle to the vehicle points. The developed method is based on fitting four straight lines to the points first and then applying the rectangularity constraint.
158
Figure 5.11: Automatically extracted LiDAR points of a truck
Once the vehicle footprint has been determined, the vertical profile is computed; either an analytical description is formulated or a simpler table, containing average elevation values for equal-sized regions, is created. The height profile computation is further complicated with motion artifacts, when a large relative velocity between the sensor and object results in fewer laser points, making the vehicle appear like a blob (see the passenger cars on the right side in Fig. 5.10). Notice also the different length of the two trucks in Fig. 5.10. Consequently, there is no good basis for analytical modeling of vertical vehicle profiles, which would be a simple task for much denser terrestrial laser data. Therefore, a basic four parameter height parameterization was suggested to describe vehicle vertical profiles by dividing the vehicle envelope into four equal sized bins and averaging the height values within each bin (height values are defined with respect to the road surface) (Toth et al., 2003c). The main advantage of the four height parameter-based vertical profile description is that it is invariant to motion artifacts.
159
5.3.3. Classification of extracted vehicles
Vehicle classification is an important parameter for traffic monitoring and control.
The large variety of vehicles makes the unique classification of each vehicle type and model difficult. In fact, classifying vehicles uniquely is not required for traffic management purposes. Vehicles can be classified into as many as 19 main categories
(Chowdhury and Sadek, 2003). In practice, however, there are primarily three major classes used, which are cars, trucks and other vehicles such as pick-ups, SUVs, RVs, etc.
Statistics describing average, median and variance of vehicle length, width and height of the distribution of vehicles owned and operated are available for all vehicle categories
(Ramprakash, 2003). The average and median values differ for the three main categories, but the distributions of length, width, and height overlap in many cases, and thus make any size-based classification ambiguous. For example, using vehicle length and width data extracted from optical imagery, the classification error can be estimated. Although this error is considerable, it is usually acceptable for macro modeling of the traffic flow, as the data are derived from a larger pool of vehicles. Because of motion artifacts, this method, using vehicle width and length, does not work for laser data. Therefore, a parameter optimization was performed to identify a feature domain where good classification performance could be achieved. A training set of more than 100 vehicles was created from three datasets flown in different regions in the US at point densities ranging between 3-8 pts/m2. The vehicle parameters were automatically extracted and subsequently checked by human operators with the class information available (Toth and
Brzezinska, 2004a). A PCA analysis performed on the dataset revealed that the two largest eigenvalue components can typically represent more than 98% of the original
160
parameter content (signal energy). More importantly, the classes showed good separation
and even 100% in-sample classification result was obtained in a few cases. Two
combinations were found to be efficient: (1) 1D classification based on the largest
eigenvalue component, derived from the vehicle length, width, height and volume
parameters, and (2) 2D the classification based on the two largest eigenvalue
components, derived from the vehicle length, width and four height parameters. Typical
classification results for the first case are shown in Fig. 5.12. The second classification
was even able to differentiate between the vehicles traveling in opposite directions. Three
classifiers were tested on the training dataset, including rule-based, minimum distance and neural network classifiers. As shown in Table 5.1, the rule-based classifier performed
well while two others showed average results (Toth et al., 2003b).
Figure 5.12: The classification of test data using the class boundaries determined in a
PCA analysis on a training dataset (blue cross: car, light blue circle: trucks, green circle:
other vehicles) 161
Minimum Training Dataset Rule-based Neural network distance No direction considered 2% 10% 14% Direction considered 2% 17% 16%
Table 5.1: Misclassification rate of the training set
The LiDAR point density plays a key role in both the vehicle extraction and
classification performance. Extended experiments proved that from 2-3 points/m2 density, the vehicle extraction becomes robust and there is not much improvement beyond 5 points/m2. For vehicle classification, the situation is different, as the higher point density is essential to differentiate among vehicle categories. At the 2-5 points/m2 density range, only major vehicle classes, such as cars, trucks and the remaining other vehicles could be classified at an acceptable success rate (Toth and Grejner-Brzezinska,
2004b). The 15-20 vehicle category based classifications used by most transportation agencies requires substantially higher densities that is not routinely achieved in current airborne LiDAR practice (note that terrestrial laser scanning can easily provide that point density).
In summary, it was demonstrated that intelligent algorithms developed here are capable of fast and robust identification of specific shapes (especially the vertical profiles of the vehicles), proving LiDAR’s ability to preserve the vehicle geometry better than conventional image projection, where it can be significantly distorted. It is proven that if
LiDAR data of sufficient spatial density are available, vehicle extraction and their coarse
162
classification can be efficiently performed in parallel to the efficient and automated road
surface extraction and modeling.
5.4 Vehicle tracking
The precondition of any tracking is the availability of vehicles, extracted and
adequately described. Vehicle tracking is the process of identifying the same vehicles in a sequence of images. When tracking a vehicle, its position is computed for each epoch,
defined in image acquisition time. From this data, a variety of parameters, such as
traveled distance or average speed, can be derived. The key task in tracking is the
matching process, which pairs vehicles based on their geometrical, intensity and texture
description, obtained in the vehicle detection phase (Paska and Toth, 2005). Since the
sample images used here are geometrically corrected, it is not necessary to use least
squares matching as simpler cross-correlation should give adequate results. Cross-
correlation is based on the similarity of gray levels. The reference image, with envelope
determined in the vehicle detection phase, is moved in the search image, and at each
position a similarity value, the cross-correlation coefficient is calculated. The highest
coefficient indicates the corresponding position. Vehicle matching in our high-resolution
image case shows very good results as shown in Fig. 5.13.
163
a)
b)
Figure 5.13: (a) Reference image and reference point (red rectangle), (b) search window
(red) and the corresponding position (blue) with a maximum similarity value of 0.96
Figure 5.14: Vehicle tracking using image sequences, acquired at 6 s intervals
164
Airborne LiDAR in its standard operational mode is unable to provide for vehicle tracking, as it is based on line scanning. Note that a fast scanner, such as the Velodyne system, installed on fixed location could provide for subsequent vehicle extraction and thus allow for tracking.
5.5 Vehicle velocity estimation
Besides vehicle count, vehicle velocity is a useful variable for the traffic analysis.
The individual speed of each vehicle is usually of less interest than the average velocity of a group of vehicles. A more rigorous traffic analysis (micro modeling of the traffic flow) definitely requires a better spatial/temporal representation of the traffic flow; for example, precise modeling of individual vehicle velocity profiles is needed to fine-tune freeway entry/exit design or traffic light synchronization.
5.5.1. Vehicle velocity estimation from optical imagery
Vehicle velocity estimates can be obtained by tracking vehicles in image sequences or by exploiting motion artifacts in the scanner type of data. The main factor for achieving good tracking results is the image acquisition rate. Tracking moving objects in video rate image sequences is a proven method; for example, a helicopter platform based vehicle tracking system is described in (Mirchandani et al., 2003). Tracking
vehicles in image sequences, acquired at a slower rate, such as the intersection
monitoring from a helicopter with a medium-format digital camera, could be quite difficult; for example, during the 6 s period between two image acquisitions, vehicles can move considerably and can change lanes. In addition, vehicle velocity may vary, so
165
simple difference-based velocity estimation could have significant errors. Fig. 5.14 illustrates the tracking of a few vehicles that were first matched by vehicle size followed by image correlation (Toth et al., 2003a). Without increasing the image acquisition rate
(currently 0.2 FPS), vehicle tracking is clearly not feasible under general traffic conditions and velocity may not be reliably estimated from optical imagery. As technology continues to advance, faster medium format digital cameras will be produced, and thus, their performance in terms of good spatial resolution and faster frame rates will allow for reliable tracking and velocity estimation. Note that digital cameras are passive sensors, and thus their use depends on the light conditions in the surveyed area.
5.5.2. Vehicle velocity estimation from LiDAR
The time separation between overlapping laser datasets practically excludes any
sort of vehicle tracking, and subsequent velocity estimation. However, the motion
artifacts in laser data, the dependency of the vehicle length on to the relative velocity
between the object and sensor, forms a basis for vehicle speed estimation. The
relationship is described in equation (5.4).
ls lv vv vs (5.4) ls cos
where vv and vs are the vehicle and laser sensor velocity, ls and lv are the laser sensed and true vehicle length, and is the angle between the flight line and the vehicle trajectory.
166
The difficulty with using equation (5.4) for velocity estimates is that not all
parameters are known with sufficient accuracy (Paska and Toth, 2005). The length of the
vehicle as sensed by the laser is rather approximate, since the laser point density (point
spacing along vehicle motion) and the horizontal footprint size cause a rather substantial
error in the vehicle length parameter. Furthermore, the true size of the vehicle is not
known and must be approximated. The vehicle classification can provide a coarse length
estimate for the three main categories as well as estimation error, as discussed above. Fig.
5.15 shows an example of an 18-wheeler truck.
vs Platform direction Vehicle direction vv
Recorded length: ls 32.6 m Actual length: l 23 2 m v Scanline direction
Figure 5.15: Motion artifact: laser image of a truck
The size estimation error is more significant for shorter vehicles; Table 5.2
illustrates the case for cars and typical velocities.
Table 5.3 shows the estimated velocity error with respect to sensed vehicle length; details on the accuracy estimation are in the next subsection. It is fair to conclude that individual vehicle velocities cannot be estimated at an acceptable accuracy level.
However, if a larger number of vehicles is considered, due to error cancellation, the 167 average velocity estimated for a group of vehicles is generally acceptable, as the effect of the weak velocity data measures could be reduced (Toth et al., 2004b).
Parameters
VLiDAR = 55 m/s
VVEH MIN = 20 m/s s MIN = 4.36 m
VVEH MAX = 32 m/s s MAX = 5.23 m LiDAR-sensed length [m] Vehicle True length Vehicle and sensor velocity travel direction Same Opposite
s MIN VVEH MIN 6.85 3.19
s MIN VVEH MAX 10.42 2.75
s MAX VVEH MIN 8.21 3.83
s MAX VVEH MAX 12.50 3.30
Table 5.2: LiDAR-sensed lengths of passenger cars traveling at typical freeway speed
Measured Length Velocity Accuracy [m] [m/s] 3 9.6 7 3.0 10 2.0 15 1.3
Table 5.3: The velocity error estimates based on measured vehicle length
Although most of the airborne laser scanners are based on using a rotating mirror to obtain observation points along a scanline that is perpendicular to the flight direction, 168 there are a few systems that produce a series of overlapping elliptical scans over the ground as shown in Fig. 5.16 (Shan and Toth, 2008). Since the object space is scanned twice within a short time interval with this solution, an opportunity for vehicle velocity detection is provided. The actual time difference depends on sensor characteristics and flight conditions, but it generally falls to the several seconds’ range, which is quite adequate for velocity estimation, as vehicles at highway speed move distances that are typically many times more than the length of the vehicle. For vehicle extraction, the same techniques introduced earlier can be used, and vehicles can easily be matched by their footprints.
Figure 5.16: Elliptical ground scanning pattern and coverage of an airborne laser scanning system utilizing progressive Palmer scans (Drawn by M. Shand)
169
5.5.3. Vehicle velocity estimation from the combined dataset (optical imagery and
LiDAR)
As discussed in the previous sections, both optical and laser sensors are capable of providing vehicle counts and velocity estimates, but with varying estimation errors. Since the sensors’ limitations and strengths are complementary, their fusion could lead to better velocity estimation and, thus, better traffic flow estimation. Therefore, the next step in this investigation is to combine the LiDAR outstanding vehicle extraction performance with the excellent velocity estimation potential of the optical imagery.
To overcome the errors in the true vehicle length estimation in the LiDAR data due to generalization or possible misclassifications, the actual length of the vehicle could be determined from other sensory data, such as imagery collected simultaneously with the LiDAR data. Though a single image does not provide the absolute size information, the image may still preserve the relative object size information, such as the width/height ratio of a vehicle. Although an extra effort, such as using an adequate matching technique, is required to identify the corresponding vehicles in the two datasets, the combination of the two datasets could eventually lead to an improved velocity estimation of the moving vehicles (Paska and Toth, 2005).
Fig. 5.17 shows vehicles extracted from LiDAR data, as they are overlaid on an orthoimage formed from simultaneously acquired imagery. LiDAR vehicle points are represented in green and red, corresponding to the motion along or against the flying direction, respectively. For referencing, some static objects, such as one point on the centerline and points on the guardrail, are also marked in the figure. This figure illustrates: (1) the elongated, when vehicles are moving along the flight direction, and
170 shortened, when vehicles are moving against the flight direction, lengths (footprints) of the moving objects, as sensed by the LiDAR, and (2) the relationship between the corresponding vehicles on the imagery and in the LiDAR data. The matches of the corresponding vehicles in the two datasets are highlighted by rectangles with identical colors. Due to the different nature of the two data acquisition techniques, the continuous scanning mode of the LiDAR sensor and instantaneous capturing of the frame imagery, the locations as well as the shapes of the corresponding vehicles notably differ in the two datasets. The white triangle in Fig. 5.17 shows the approximate location of the LiDAR beam when the image was taken.
(a)
(b) (c) Figure 5.17: Vehicles extracted from the LiDAR data and overlaid on the orthoimage; (a) match of corresponding vehicles in the two datasets is marked with identical colors. Also shown are (b) vehicle elongation and (c) vehicle shortening
171
Note that the LiDAR point clouds of the vehicles fall in front of the corresponding
vehicles on the left side of the blue dotted line and behind the corresponding vehicles on
the right side of the line. This is because the LiDAR measured the vehicles either before
or after the image was taken. Based on the known relative positions of corresponding
vehicles, search areas for a matching procedure can be determined; the acquisition time
of each LiDAR point, as well as the image capture time, is precisely recorded in GPS
seconds. The possible relative distance between the image and LiDAR vehicle positions
could be calculated from the vehicle velocity and the acquisition time of the image and
the LiDAR vehicle points (coarse vehicle velocity approximations could be obtained, for
example, from vehicle velocity computation from image sequences or from the minimum and maximum speed limits of the actual road). Note in Fig. 5.17 that the relative distance
between corresponding vehicles is getting larger the farther they are from the triangle sign. Similarly, the difference between the data acquisition time of the LiDAR sensor and digital camera is also getting larger. The difficulty of matching can be substantially reduced with higher image acquisition rates that can be easily achieved with modern digital cameras; for example, at 5 FPS, the overlap between the LiDAR and image is guaranteed, assuming posted freeway vehicle velocities. Note that at this image acquisition rate, the image-based vehicle tracking is also greatly facilitated, as a very high level image overlap exists. Since the road surface, as well as the image sensor plane on the airborne platform is usually horizontal, the width/height ratio of a vehicle is fairly accurate with respect to, for example, the LiDAR point horizontal positional accuracy.
Thus, the LiDAR-sensed vehicle width can be used to determine the vehicle true length by using the width/height ratio obtained from the image.
172
5.5.4. Performance evaluation
To check the performance of the combined LiDAR and image vehicle extraction
and velocity estimation, as well as to validate the LiDAR only or image only estimates, a
dedicated test flight was organized in late 2004. The Madison County, Ohio, test range
that includes a dense network of permanently installed signalized ground controls to
support airborne surveys was temporarily extended by using LiDAR-specific targets.
These targets, shown in Fig. 4.1, could be also used for image control (Csanyi et al.,
2005). In addition, the OSU Center for Mapping GPSVan (He et al., 1994), a vehicle
equipped with high performance GPS/IMU hardware, was used as a “moving” target.
This vehicle, shown in Fig. 5.18, was constantly moving in the test area and was mapped
by both sensors several times under various LiDAR sensor settings, such as using various pulse rates and scan rates during repeated passes over the calibration range. Using different sensor settings served several purposes. Most importantly, the impact of the point density for the vehicle extraction, classification and velocity estimation was assessed. This also provided valuable data to assess the impact of the various pulse rates on the overall accuracy of the system, with and without ground controls. The airborne sensor suite included an ALTM 30/70 LiDAR system and an Applanix DSS digital camera. The LiDAR system was operated at 33, 50 and 70 kHz pulse rates, resulting in point densities ranging from 3 to 8 points/m2. The digital camera had a GSD range of 10-
15 cm.
To validate the concept of combining the optical imagery and LiDAR data to improve the velocity estimates, a comprehensive analysis was performed on the 2004 data set. Table 5.4 shows a representative set of measurements of the LiDAR sensor as it
173 mapped the GPSVan at various pulse rates. As expected, the accuracy of the vehicle size, as measured by the smallest rectangle fitted to the vehicle points, depends on the point density, which, in turn, is basically a function of the pulse rate for a given flying height.
Clearly, the vehicle width is fairly underestimated at lower point densities. The smaller size is a combined effect of the point density, laser pulse divergence and point pattern.
The image measurements for the width/length ratios, however, show a good stability.
Figure 5.18: LiDAR target and the GPSVan
The vehicle velocity estimates, shown in Table 5.5, illustrate that a larger error was introduced by the incorrect vehicle length parameter. The GPSVan has a true length of 5.5 m but falls into the other vehicle category with a class length value of 4.7 m. This length could be effectively decreased by the vehicle length estimation from the LiDAR- 174
measured width by using the image measured width/length ratio. The statistics, shown for the cases when the vehicle and the LiDAR traveled in the same direction (shaded area in
Table 5.5) clearly indicate that accuracy of the true length-based velocity estimation can be achieved for the combined LiDAR and image solution. The opposite direction case has a smaller improvement, (with statistics of estimated bias and variance of 2.39 and 1.73, respectively). Nevertheless, the availability of this less accurate data is still important as it helps to obtain a better overall error in velocity when the average velocity of a group of vehicles is computed. Further discussion of the error characteristics of the LiDAR-based length and velocity estimation is in the next subsection.
175
Image-based LiDAR sensor parameters Vehicle measurements correction ] 2 [pts/m PRF [kHz] PRF [kHz] vehicle per profiles [m] profiles [m] profiles [m] Ideal LiDAR Point density Scan angle[°] Scan angle[°] Strip number with respect to respect with (based on GPS) image ratio image ratio and Point spacing in Spacing between LiDAR platform Image-measured Vehicle direction direction Vehicle vehicle width [m] Number of points LiDAR width [m] LiDAR-measured vehicle length [m] LiDAR-measured vehicle length [m] length/width ration Vehicle length from
5 33 10 0.52 0.50 2.9 + 8.37 1.81 8.87 51 2.95 5.34 12 33 10 0.48 0.50 4.0 + 10.52 2.01 10.52 81 2.83 5.69 9 33 20 0.71 0.70 1.8 - 3.40 1.59 4.02 15 2.85 4.53
11 50 10 0.40 0.40 6.1 + 9.40 1.99 9.48 95 2.83 5.63 13 50 20 0.55 0.65 3.2 + 9.70 1.86 10.33 51 2.91 5.41 14 50 20 0.58 0.60 2.8 - 3.68 1.85 4.13 22 2.85 5.27
10 50 20 0.55 0.55 3.1 0 5.25 1.72 5.55 27 2.97 5.11
4 70 10 0.35 0.35 8.2 + 7.85 1.90 7.95 120 2.89 5.49 8 70 20 0.50 0.50 4.1 - 3.89 1.88 4.10 31 2.84 5.34
Mean 2.88 STD 0.05
Table 5.4: Vehicle length and width measurement from LiDAR and length estimation based on combined LiDAR and image data (‘+’: marks vehicle moving in opposite direction with respect to LiDAR sensor, ‘0’: vehicle is not moving, and ‘-‘: vehicle moves in the same direction as the LiDAR sensor)
176
GPS velocity Vehicle velocity computed from different vehicle length measurements [m/s] [m/s] Derived from LiDAR Using vehicle class length True vehicle length Reference width (4.7 m) (5.55 m) (used to using image ratio compute Strip number Velocity Difference Velocity Difference Velocity Difference Differences) 5 25.71 -3.90 21.22 0.59 19.75 2.06 21.81 12 27.93 -4.16 23.18 0.59 23.85 -0.08 23.77 9 20.14 -0.18 17.51 2.45 33.31 -13.35 19.96
11 26.79 -4.58 21.48 0.73 21.94 0.27 22.21 13 26.16 -2.77 22.45 0.94 21.71 1.68 23.39 14 14.64 3.37 22.83 -4.82 26.85 -8.84 18.01
10 5.22 -5.15 1.33 -1.26 2.85 -2.78 0.07
4 20.75 -5.28 15.55 -0.08 15.15 0.32 15.47 8 11.28 7.87 20.18 -1.03 23.11 -3.96 19.15
Mean 4.14 0.58 0.88 (abs) STD 0.93 0.32 0.91
Table 5.5: Velocity estimation performance for various sensor settings for LiDAR-only
and for combined LiDAR and image data. Flight lines where vehicle and LiDAR move in
the opposite directions are marked by white cells, while where vehicle and LiDAR move
in the same directions are marked by shaded cells
5.6 Accuracy assessment of the vehicle velocity estimation from the combined dataset
In order to assess the potential accuracy of vehicle velocity estimates from the
combined datasets, we have to investigate the accuracy of the various components which
are based on equation (5.4), including the velocity of the LiDAR platform, the LiDAR-
sensed vehicle length, and the true length of the vehicle. The LiDAR platform’s speed is known with high accuracy; therefore, in the calculations it is considered as fixed values.
177
The measured vehicle length from the LiDAR data, m, may not be very accurate due to
the errors in the vehicle representation, and thus, is considered with m standard error.
The accuracy of the true vehicle length derived from image measurements simply
depends on the accuracy of single ground point determination from imagery. In the following, the accuracy of size parameter and the derived velocity estimates from imagery and LiDAR data are discussed.
5.6.1. Accuracy of single ground point determination and vehicle size estimation from
optical imagery
In the proposed approach to extract vehicles from imagery, (see Toth et al.,
2003b; Paska and Toth, 2004), the detection begins with image orthorectification, which
produces good starting data for all the vehicle extraction and tracking processes. The
orthoimagery provides advantages in many respects. First, by overlaying and subtracting
images it facilitates the identification of moving objects from the images (vehicles on road surface). Second, since orthoimages are scaled to object scale, vehicle envelopes can
be compared to actual dimensions, making vehicle extraction more robust.
For orthorectification, exterior orientation parameters of images are usually
provided from the GPS/INS solution. Once the image georeferencing is known, the images are orthorectified by using available DEMs, such as existing USGS DEM, or in the case of simultaneous data acquisition of LiDAR with a digital camera, LiDAR will provide good quality, up-to-date elevation data. However, LiDAR provides a true DEM within its accuracy specifications only for stationary objects; moving objects appear
178
distorted in the data set due to the facts described in the previous section. Therefore, care
should be exercised in using current LiDAR data.
The accuracy of single ground point determination was analyzed by error
propagation assuming that all systematic errors have been removed a priori. The accuracy of the ground coordinates of any objects determined from an image depends on the accuracy of the parameters shown in Table 5.6. The errors associated with each quantity will contribute to the error in the ground coordinates.
IO parameters 3 linear parameters (c, x0, y0) (Interior orientation) 6 non-linear distortion parameters
Navigation parameters 3 positions (X0, Y0, Z0) (Platform orientation) 3 rotation angles (,,) Boresight alignment between sensors 3 rotation angles ( , , ) (IMU and camera frames) b b b DEM Elevation data of the surface (Z)
Table 5.6: The required parameters describing the mathematical relation between image
and ground coordinates by single photo resection
The well-known collinearity equation (5.5) is used to propagate the uncertainties
of the parameters to the data.
r11 x r12 y r13 c X X 0 (Z Z 0 ) r31 x r32 y r33 c (5.5)
r21 x r22 y r23 c Y Y0 (Z Z 0 ) r31 x r32 y r33 c
179 where (x, y) are image coordinates, interior orientation parameters (x0, y0 and distortion) are applied, (X, Y, Z) are ground coordinates of the object, c is the focal length, (X0, Y0,
Z0) are the ground coordinates of the camera center, rij are the elements of the rotation matrix calculated from the camera attitude angles (boresight-corrected navigation angles).
The general error propagation for a function of n variables is given below in equation (5.6), provided that the various variable errors are independent.
2 2 2 w w w i 2 i 2 ... i 2 (5.6) w v v v i v1 1 v2 2 vn n
The partial derivative wi/vn represents the change in the computed value wi with respect to the measured value vn, and Vn represents the standard error assigned to the measured value. In the more general matrix format, equation (5.7), where no assumption is made about variable correlations:
D(w) A D(v) AT
w1 w1 w1 ... v1 v2 vn w w w (5.7) 2 2 2 A ... v1 v2 vn ...... w w w m m ... m v1 v2 vn
180 where D(v) is the dispersion matrix, which contains the covariance (second moment) among all measured elements, A is the design matrix, and D(w) gives the variances/covariances of the final results. The design matrix when calculating the accuracy of the ground coordinates of any objects determined from an image by single photo resection is given by equation (5.8).
X X X X X X X X X X X X X X Y Z ω κ ω κ x y c Z A 0 0 0 b b b (5.8) Y Y Y Y Y Y Y Y Y Y Y Y Y X 0 Y0 Z 0 ω κ ωb b κb x y c Z
Table 5.7 contains typical accuracy ranges of the navigation, boresight, and interior parameters with their accuracy taken from calibration reports. In the following calculations, the accuracy parameters listed in Table 5.7 are considered.
Camera Accuracy Navigation errors Position 5 – 20 cm (Applanix POS AV Model 510) Attitude 10 – 30 arc sec Boresight misalignment (Omega, Phi, Kappa) 10 – 30 arc sec Focal Length Errors in interior orientation 9 m parameters (54.969 mm) (DSS) Principal Point 4.5 m, 4.5 m Errors in image coordinate 2 – 5 m measurement
Table 5.7: Accuracy of the orientation parameters and image coordinate measurements
181
Table 5.8 and 5.9 show the accuracy estimates of ground coordinates for two typical image point locations of the 4K by 4K image for various navigation and boresight misalignment errors, and for a common image measurement accuracy, focal length, and
DEM accuracy. The size parameter is computed as the distance between two points given by their XY ground coordinates (assuming the road is horizontal). The accuracy of the size parameter estimation calculated by error propagation for different scenarios is included in Table 5.9. In the worst case scenario, a 37-cm accuracy can be expected for the size parameter. In an average case, about 20 cm accuracy of the size estimation can be assumed.
Image Accuracy of Navigation Boresight DEM coordinate Focal errors misalignment LiDAR measureme length positioning size x = 0 nt y = 0 [mm] Attitude Om,phi,ka Z x, y c X Y Y Position [arc [arc sec] [m] [m] [m] [m] [m] [m] [cm] sec] H = 5 10 10 0.30 5 9 0.08 0.08 0.11 500 10 20 20 0.30 5 9 0.13 0.13 0.18 [m] 20 30 30 0.30 5 9 0.23 0.23 0.32
Table 5.8: The accuracy estimates of ground coordinates of a center image point, and the length parameter estimation for different scenarios
182
Image Accuracy of Navigation Boresight DEM coordinate Focal x = errors misalignment LiDAR measureme length 15.36 positioning size nt y = 0 [mm] Attitude , [arc Om,phi,ka Z x y c X Y Y Position [arc sec] [m] [m] [m] [m] [m] [m] [cm] sec] H = 5 10 10 0.30 5 9 0.12 0.08 0.17 500 10 20 20 0.30 5 9 0.17 0.13 0.23 [m] 20 30 30 0.30 5 9 0.26 0.23 0.37
Table 5.9: The accuracy estimates of ground coordinates of an image point that is at the edge of the image, and the length parameter estimation for different scenarios
5.6.2. Uncertainties in size-parameter estimation from LiDAR data
The measured vehicle length from the LiDAR data may not be very accurate due to the errors in the vehicle representation:
1. The LiDAR pulse footprint size is not negligible; the accuracy of the vehicle
envelope determination depends on the size of the LiDAR footprint and across-
track spacing.
2. LiDAR point density, along-track spacing provides for coarse spatial sampling;
the distance between the points on the ground, spatial sampling distance, limits
the accurate length estimation.
3. The shadow effect, which does not influence the length parameter estimation
directly, makes the vehicle orientation determination more ambiguous and the
width parameter estimation less accurate. Examples illustrating these problems
are shown in Fig. 5.19.
183
(a) (b)
Figure 5.19: Limitations in accurate parameterization of LiDAR-sensed vehicles: (a) data density and footprint size, (b) shadow effect
Obviously, the denser the data the less uncertainty in the vehicle length measured in the LiDAR data. When deriving the formula for vehicle velocity estimation, the assumption is made that the length of the vehicle is measured continuously by the sensor moving above the object. However, LiDAR measurement provides discrete points at some spacing, thus, resulting in an inaccuracy of the LiDAR-sensed length parameter.
5.6.3. Accuracy of the vehicle velocity estimation from the combined dataset
The accuracy of the vehicle velocity estimation based on equation (5.4) is given by (5.9).
2 v 2 LiDAR 2 s 2 v (5.9) v s LiDAR m veh m m2
where VLiDAR is the velocity of the LiDAR platform, m is the LiDAR-sensed vehicle length, and s is the true vehicle length; Vveh , m and s are the accuracy of the vehicle
184 velocity, LiDAR-sensed vehicle length, and the true vehicle length, respectively. Fig.
5.20 shows the standard deviation of vehicle velocities computed by equation (5.9) at different LiDAR-sensed lengths using the parameters listed in Table 5.10.
Fig. 5.20 s [m] s [m] m [m] Line color 5 0.30 0.30 Green (a) 5 0 0.30 Red 5 0.30 0 Blue 4.3 0.30 0.30 Green (b) 5.3 0.30 0.30 Magenta 4.3 0 0.30 Red (c) 5.3 0 0.30 Magenta 4.3 0.30 0 Blue (d) 5.3 0.30 0 Magenta 5 0.10 0.30 Green (e) 5 0.20 0.30 Red 5 0.30 0.30 Blue 5 0.10 0.10 Green (f) 5 0.20 0.20 Red 5 0.30 0.30 Blue
Table 5.10: Summary of parameters considered in the computation of the standard deviation of vehicle velocities at different LiDAR-sensed lengths
185
12 12
10 10
Alon 8 8
6 6
4 4 Standard deviation [m/s] Standard deviation [m/s] 2 2 Agains 0 0 2 4 6 8 10 12 14 16 2 4 6 8 10 12 14 16 LiDAR-sensed vehicle length [m] LiDAR-senseda) vehicle length [m] b) 12 12
10 10
8 8
6 6
4 4 Standard deviation [m/s] Standard deviation [m/s]
2 2
0 0 2 4 6 8 10 12 14 16 2 4 6 8 10 12 14 16 LiDAR-sensed vehicle length [m] LiDAR-sensed vehicle length [m] c) d) 12 12
10 10
8 8
6 6
4 4 Standard deviation [m/s] Standard deviation[m/s]
2 2
0 0 2 4 6 8 10 12 14 16 2 4 6 8 10 12 14 16 LiDAR-sensed vehicle length [m] LiDAR-sensed vehicle length [m] e) f) Figure 5.20: Accuracy of vehicle velocity estimation at different LiDAR-sensed vehicle length for scenarios summarized in Table 5.10 186
From Fig. 5.20a we can conclude that the vehicle velocity accuracy is more sensitive for the uncertainties of the size parameters at the shorter LiDAR-sensed lengths, i.e., the uncertainties in the size parameters (true vehicle length and LiDAR-measured length) has less impact when the LiDAR-sensed length is longer. The accuracy of the velocity estimation is better for vehicles traveling along the direction of the sensor motion, as their LiDAR-sensed measure is relatively long. In a similar way, if the car is traveling in the direction opposite of the LiDAR, its velocity estimate would be more accurate for lower speeds, since the shortening effect will be less severe. Moreover, uncertainties in the actual vehicle size have larger effects at the longer LiDAR-sensed lengths than uncertainties in the measured vehicle size. Figs. 5.20b and 5.20c show that longer true size vehicles have bigger standard errors. Fig. 5.20d illustrates that the true vehicle length would not affect the accuracy of velocity estimation if the LiDAR-sensed length estimation could be considered fixed. Figs. 5.20e and 5.20f illustrate how the accuracy increases with decreasing uncertainties in the size parameters.
5.7 Traffic flow computation
5.7.1 Traffic flow parameters
The full description of the vehicles that were extracted and categorized from remotely sensed imagery with velocity and optional tracking estimates provides the data that can be used to compute traffic flow data. In general, the items of interest are the following:
rates of flow (vehicles per unit time), traffic volume
velocities (distance per unit time)
187
travel time over a given length of road
occupancy (percent of time at a point on the road occupied by vehicles)
time headway between vehicles (time period per vehicle)
density (vehicles per unit distance); spacing, or space headway between vehicles
(distance per vehicle)
stop time at traffic lights and intersections
vehicle types with frequencies
An introductory discussion of these items can be found in (Gartner, 1997).
Many of the above listed parameters are generally considered as macroscopic characteristics, as they describe the traffic flow in general terms that reflect the behavior of a group of vehicles over a given road section. In contrast, the microscopic characteristics of traffic flow describe the behavior of individual vehicles, including vehicle type, velocity, driving pattern, travel time, O-D (origin – destination) data, lane changing, etc. In theory, the macroscopic parameters can be derived from the microscopic parameters of all vehicles in a given road section, but in practice, this may not be feasible, as the data for all vehicles cannot be obtained. Both types of data provide essential information for both the design and operation of streets and highways. A representative sample of the microscopic parameters is also valuable for supporting traffic flow simulations. Similarly, traffic flow models can be improved with both macro- and microscopic information.
188
The discussion so far has addressed the feasibility of traffic flow extraction from remotely sensed data from a theoretical perspective that only considered the sensor capabilities. Platform motion, however, introduces several limitations that should be understood if the overall feasibility of remote sensing for traffic flow extraction is assessed. In the following, a discussion is provided on the practicability of using various platforms to acquire traffic flow from airborne remote sensors.
5.7.2 Platforms and sensors
There are three essential airborne platforms for airborne remote sensing and surveillance:
Fixed wing aircraft; the workhorse of civilian mapping
Helicopters that are typically used in special applications
UAV systems, which are only experimentally used for feasibility testing,
including small- and medium-size platforms (both fixed-wing and helicopter)
An important platform parameter is the velocity, which can vary from near zero to typical mapping airplane speed, which is typically below 120 knots. For the zero platform velocity, mobile pole-mounted sensor can be also considered.
On the sensor front, digital cameras and LiDAR systems were considered in this investigation. These systems can be categorized by their volume and power requirements:
Airborne LiDAR systems are heavy and require significant power
High-end digital cameras are comparable to LiDAR systems and, in many cases,
are heavier then LiDAR systems 189
Medium- and small-format digital camera are relatively light and require limited
power
Large-format Medium-format Small-format LiDAR digital camera digital camera digital camera Practical (also Fixed-wing Practical (also with large-format Practical Not practical aircraft with LiDAR) camera) Helicopter Practical Not practical Practical Practical (moving) Helicopter Not practical Not practical Practical Practical (hovering) UAS (slow Not practical Not practical Practical Practical moving) UAS (hovering) Not practical Not practical Not practical Practical Fixed-pole Not practical Not practical Practical Practical
Table 5.11: Feasible platform and sensor combinations
The feasible platform and sensor configurations, considering typical systems and aircraft used in practice are listed in Table 5.11. This table is an orientation table to show the most typical configurations and should not be considered exclusive, as dedicated systems can accommodate for almost any sensor combinations. In addition, the duration of the data acquisition and the area coverage can vary over a wide range; e.g. a helicopter flight is usually less than a couple of hours, while a UAV may easily remain airborne for several hours (Gruen, 2007). Area coverage usually depends on the flying height assuming a comparable sensor’s field of view. Helicopters with optical and/or laser sensors may cover a narrow corridor, such as transmission lines. Low altitude aircraft can
190 collect imagery and LiDAR data over a wider corridor which, for example, is enough to map a road with significant coverage of both sides of the road. High altitude photography missions can cover larger areas, such as capturing small cities in a single image.
5.7.2.1 Fixed-wing aircraft with high-performance imaging sensor
The great majority of airborne surveys are currently flown by fixed-wing aircraft with one high performance imaging sensor installed; there are a similar number of state- of-the-art LiDAR and large-format digital camera systems installed worldwide, approximated as 200 each in early 2008 (ASPRS LiDAR Subcommittee meeting,
Portland OR, 2008). Since most mapping missions require both forms of sensory data, more and more aircraft are outfitted with a second camera hole, allowing for simultaneous acquisition of highly accurate LiDAR and high-resolution optical imagery.
This combined dataset provides an ideal information source for traffic data. Surveys flown along roads can provide a “moving window” type of sampling of the vehicles.
Typical aircraft speed is about two times of the federal highway speed limit, thus, the relative speed between the sensor platform and vehicles is generally 1-3 times the speed limit, which allows for velocity estimates from LiDAR data. Depending on the image acquisition rate, which depends on the camera specification and the flying parameters, an overlap between subsequent images, may not be guaranteed for larger relative velocities.
Therefore, vehicle tracking is typically feasible only for the traffic, moving in the direction of the sensor platform, where multiple image coverage is assured. Of course, vehicles can be always extracted from individual images. Since only the traffic flow extraction from LiDAR data is fully automated, an example is shown here for the LiDAR
191 sensor. Fig. 5.21a-b show the busy highway section in Toronto, Canada, surveyed by an
Optech 3070 system.
(a) (b)
Figure 5.21: LiDAR data acquired over a busy freeway in the Toronto downtown area;
(a) road surface and objects extracted, and (b) extracted and classified vehicles
As the road surface is near horizontal and there are no occlusions, the vehicles are easily extracted based on their elevation. In Fig. 5.21b, the extracted vehicles are gray- scale-coded based on the classification results. In this test there was no reference for the velocity validation (there were no “control” vehicles). Thus only statistical “worst case” assumptions were used in the evaluation process. The traffic flow parameters calculated for each lane, as well as totals for both directions, at four locations are listed in Table
5.12.
192
Velocity Flow Density Flight Vehicle ± error ± error ± error [veh/km] [km/h] [veh/h]
Left Right Left Left Left Right Right Right Right Name Name
PRF [kHz] [kHz] PRF P M T P M T Bridge 117.0 123.9 208 59 24629 7312 19 1 0 3 2 0 1 29.1 13.8 0.2 0.2 3595 600 Bridge 114.9 123.9 158 79 17729 9790 22 2 2 4 6 1 70 2 27.8 6.6 0.2 0.1 2596 375 Bridge 120.7 77.7 153 63 18315 5116 18 1 1 5 1 0 3 23.8 11.7 0.2 0.4 2124 557 Bridge 74.5 90.9 219 174 19607 14029 13 0 1 6 1 0 4 12.7 25.6 0.2 0.3 3285 1299
Table 5.12: Derived traffic flow parameters with error estimates; overall performance results for the Toronto, Ontario, tests; P- passenger car, M- multipurpose utility vehicle,
T-truck
The flow results show an about 15% estimation error in the case when the sensor platform and vehicles travel in the opposite direction. In contrast, the estimation error is less than 10% when both sensors and vehicles travel in identical direction.
5.7.2.2 Helicopter with medium format camera and LiDAR
Helicopters are typically used for specialty surveys, such as transmission line mapping, where lower platform speed is required. For a long time, only single or multiple digital camera configurations were used. Later, as LiDAR technology evolved and smaller systems could be built, dedicated LiDAR systems were introduced. Nowadays, these systems are also used for surveys where extreme point density is required, such as
100 pts/m2 to better describe buildings (http://www.flimap.com). Since helicopters are widely used for real-time traffic observations, medium-format cameras can provide 193 valuable quantitative data from these missions. To assess the potential of such imagery, image sequences were acquired by a 4K by 4K digital camera in a busy intersection in
Tucson, in cooperation with the University of Arizona. The images, acquired at 6 sec rate, were already shown in Fig. 5.8, so here only the results are illustrated. The traffic volume count per traffic light cycle is a useful parameter for traffic management and planning purposes. Examples of traffic counts from selected directions (East and West) of the analyzed intersection are shown in Fig. 5.22, indicating also the direction of motion, i.e., left, straight or right at the intersection. Fig. 5.23 illustrates the inbound and outbound traffic related to the entire intersection, including all directions. Finally, the balance flow counts, the difference between the number of incoming and outgoing vehicles for a road connecting to the intersection, from all four roads are shown in Fig.
5.24. It should be noted that newer cameras can work with much faster image capture rate, allowing for more detailed analysis of the motion of the tracked vehicles that may reveal additional information about the traffic patterns at a particular intersection.
East West
50 50
40 40 Left Left 30 30 Straight Straight 20 20 Right Right Vehicles Vehicles 10 10
0 0 1 3 5 7 9 1 3 5 7 9 11 13 15 17 19 11 13 15 17 19 Event Event
Figure 5.22: Intersection flow count by direction
194
Inbound traffic flow Outbound traffic flow
100 90 90 80 80 70 70 60 60 50 50 40 40 30 Vehicles Vehicles 20 30 10 20 0 10 1 3 5 7 9 11131517192123 0 1357911131517192123 from East from West from North from South toward West toward East toward South toward North
Figure 5.23: Intersection flow count
Figure 5.24: Flow balance at the intersection
5.7.2.3 UAV with small-format camera
There are about 200 UAV system manufacturers in the US, and a large number of applications are currently testing the technology. Applications related to mapping are still scarce. Small fixed-wing and helicopter type systems are used to survey remote, difficult to access areas, for detailed geospatial data. The deployment of similar systems, equipped with digital camera, communication link, and low-end georeferencing, in urban areas is difficult due to liability concerns. Experimental systems have confirmed the feasibility of this sensor configuration, so future progress depends on how effectively safe operations can be assured.
195
5.7.2.4 Fixed pole with medium-format camera installation
An increasing number of imaging-based traffic monitoring systems have been introduced in urban areas, and these pole- or bridge-mounted cameras have shown significant performance improvements recently. These systems could be used to calibrate loop-detectors, and the trend of switching toward imaging technologies is expected to increase. The installation can vary over a broad range, as cameras are mounted on light poles, or on tall buildings, and on temporary structures. Fig 5.25a-b show near vertical and oblique imagery taken from a 10-storey building.
(a) (b)
Figure 5.25: Medium-format imagery acquired from a tall (10-storey) building, Colombo,
Sri Lanka; (a) near vertical orientation and (b) oblique view
196
(a) (b)
Figure 5.26: Medium-format oblique imagery acquired from a building; (a) original image and (b) orthorectified image
The processing of the imagery in both cases requires camera calibration and georeferencing to achieve better vehicle extraction results. Since the camera orientation is fixed and most of the feature extraction processes are based on differential method, the camera calibration is less critical, provided that the distortion is below 5-10% at the image boundaries. Note that camera calibration can be routinely performed. The georeferencing, however, is important, as the known or approximated image-object scale is an essential parameter for the vehicle extraction processes. For the vertical imagery, where the imaging sensor plane is parallel to the road surface, a simple scale measurement could be sufficient, such as identifying two points in the image with a known object distance. The oblique imagery, however, is a different case, as the image scale varies substantially over the image. In this situation, the technique introduced in 197
Section 5.3 can be used, provided the exterior orientation of the camera and a DEM are available. Assuming a simple planar road surface model, it is sufficient if the distance from the camera to at least three (but preferably more) points on the road surface is measured; the distances can be directly measured by laser ranging or can be determined from absolute coordinate measurements, such as from GPS surveyed camera and road point coordinates. Based on these measurements, all the parameters (EO + DEM) needed for image rectification can be computed in a least squares adjustment (Csanyi et al,
2007). Fig. 5.26a-b show an example from a test performed at The Ohio State University.
Note that due to the very oblique image geometry, the vehicles are noticeably distorted and only the road has the accurate geometry.
5.7.3 Performance comparison
The discussion on the experiences gained so far from extracting traffic flow from
LiDAR and optical imagery show that the two sensors have different strengths and weaknesses for the various data processing tasks and, in most cases, they complement each other. For the relevant cases, which are considered typical in current airborne surveying practice, Table 5.13 provides a general summary of the main processes and the comparative performance level of the two sensors (Toth and Grejner-Brzezinska, 2006).
Table 5.13 clearly indicates that state-of-the-art LiDAR sensors have a comparable performance with respect to high-performance large-format digital aerial cameras, although the higher spatial resolution of the digital camera would definitely support a better image primitive extraction. In addition, the faster image acquisition rates could eventually result in higher (multiple) overlap, and thus, in better 3D extraction
198 performance. Table 5.13 also suggests that a combination of the two sensors represents an optimal solution with respect to any “one-sensor” solution. Furthermore, adding other type of sensory data, in principle, has the potential to further improve the performance of the traffic flow extraction. For example, infrared cameras and hyperspectral scanners, currently available on airborne platforms, are rarely used in mainstream mapping practice and provide for coarse spatial (and temporal) resolution presently.
The operational conditions (omitted from Table 5.13) could severely impact the data acquisition and, consequently, the vehicle extraction and traffic flow estimation processes. From this perspective, LiDAR has a clear edge over optical imagery. First,
LiDAR is less weather dependent; for example, it can cope with haze and fog to some extent. Second, being an active sensor, LiDAR can work day and night. In contrast, digital cameras need favorable illumination conditions as the processing is rather sensitive to the image intensity characteristics. Furthermore, scene complexity poses an additional difficulty for the optical imagery: dense urban areas, long and strong shadows, occlusions, etc., can severely impair the feature extraction performance.
199
Sensor LiDAR Digital Camera Platform Airplane Helicopter Airplane Helicopter Sensor Parameters: Spatial 4-8 pts/m2 5-20 pts/m2 15-20 cm GSD 10-20 cm GSD Resolution Temporal 10-30 minutes 1-10 minutes 0.2-0.5 FPS 0.2-0.5 FPS Resolution Tasks: Road Outline Moderately Moderately Relatively easy Relatively easy Extraction Difficult Difficult Vehicle Easy, robust Easy, robust Very difficult Difficult Extraction Vehicle Easy (2D) – Easy (3D) Easy (3D) Easy (2D) Parameterization Difficult (3D) Coarse Vehicle Easy Easy Difficult Difficult Classification Vehicle Not feasible Not feasible Limited Good Tracking Velocity Modest Modest Good Very Good Estimation Overall Flow Moderately Relatively Easy Relatively Easy Difficult Computation Difficult
Table 5.13: Sensor performance metrics with respect to the different processing steps of traffic flow extraction in terms of complexity of automated processing (shaded areas indicate cases when one sensor significantly outperforms the other one)
200
CHAPTER 6
SUMMARY, CONTRIBUTION, AND FUTURE RECOMMENDATIONS
6.1. Summary and contributions
The objective of this dissertation was to provide a comprehensive feasibility assessment of using state-of-the-art airborne mapping systems supported by direct georeferencing to facilitate existing and novel applications in the transportation field, including road infrastructure mapping and traffic flow extraction. The discussion was mainly concerned with using airborne LiDAR and digital camera systems (medium and large format digital cameras, in particular) installed on fixed-wing and helicopter platforms. Methodology was developed to improve the QA/QC processes of transportation corridor mapping and to derive traffic flow data from airborne LiDAR and digital camera systems to support traffic monitoring and management. The effect of the error sources was analyzed in both areas, road infrastructure mapping and traffic flow extraction, to provide a comprehensive accuracy assessment that considers all the major potential error sources to validate the quality of the end product.
The introductory part of the dissertation provided a review of recent geospatial science and technological developments. First, the transition from analog to totally digital
201
geospatial technologies, including consequences and the future potential was examined.
Then sensor developments were reviewed with the focus on performance assessment.
Imaging sensors are essential for obtaining information-rich geospatial data. The navigation sensor developments are also important, as georeferencing is both an enabling technology and key to support downstream processing, including feature extraction,
creating full 3D visualization, and information extraction. The primary aim of this
analysis was to consider the specifics of infrastructure mapping and traffic flow
application fields, and thus, to provide a comprehensive review of remote sensing
technology with respect to its potential in these applications.
Transportation infrastructure mapping is a major pillar of civilian mapping. The
road network forms probably the most widely used vector GIS/CAD database maintained by several federal and local government agencies. The update of these data requires sustained data acquisition missions and a need to cope with the ever increasing requirements for better accuracy. When the first 2D GIS/CAD systems were introduced,
the centerline-based data provided only for general road information. Current systems are
based on 3D representation and could provide for detailed road description, such as edges
lines, lanes, structures, etc. The target accuracy requirements, demanded by engineering
and environmental applications as well as vehicle navigation and safe driving, are
currently in the sub-decimeter range, although only a fraction of the existing data meets
this stringent requirement. The great majority of the road infrastructure data are acquired
by remote sensing technologies, mainly by using LiDAR and high-performance digital
camera systems. To achieve the highest accuracy for the given sensors, adequate QA/QC
202
procedures should be implemented, covering both the airborne surveying and data
processing aspects. In this dissertation, a novel highly automated method to improve
LiDAR QA/QC over the transportation network, based on pavement marking extraction and matching with GPS-based survey data, was introduced. The key element of the
technique is the extraction of the pavement markings based on LiDAR intensity data. In
an iterative approach, candidate points are selected and object space constraints are
applied to filter outlier points. Next piecewise curve fitting is performed to obtain an
analytical, as well as a numerical representation of the linear pavement marking. The
reference pavement marking data, the ground control, are provided by GPS surveying,
preferably using fast VRS technology. Based on the two representation of the pavement
marking, a transformation is established between the two datasets, using free-shape curve
matching based on the ICP method. Depending on the results, including all the pavement marking areas from a mission, different corrective steps can be taken, such as applying a
global transformation or the data should be segmented and then local corrections are applied. The actual corrective process follows conventional practice.
The developed method was tested using data acquired by state-of-the-art LiDAR sensor at several areas, and the results showed good performance that can be
characterized at three levels. First, the robustness of the pavement marking extraction was assessed, which is difficult to be described quantitatively. The relative nature of the
LiDAR intensity signal and the large variation in quality of both road surface and pavement markings make it nearly impossible to define any measure. In our experiences, the extraction worked well for most of the test datasets, and there were only a few cases
203
where the quality of the pavement marking was so bad that the extraction failed. In these
cases of poor performance the markings were either not visible or hardly visible in the
LiDAR intensity image and/or optical imagery. The second performance parameter was the positioning accuracy of the automatically extracted pavement markings that were compared to manual measurement. Ignoring the subjective aspects of operator measurements, only 1-3 cm differences were found. The third aspect was the matching accuracy between the automatically extracted and reference pavement markings, which routinely achieved better than 3-4 cm accuracy; note that the GPS-surveyed reference had the horizontal accuracy of about 2 cm. In summary, the introduced method produced excellent results, achieving well under sub-decimeter accuracy, and the whole computation process seemed quite robust, provided that the pavement marking signals were visible in the LiDAR intensity image.
The third major part of the dissertation was concerned with the innovative use of remote sensing technologies to support traffic data for traffic monitoring and management. An analysis of using state-of-the-art remote sensing technologies with respect to traffic flow extraction was provided, including detailed error performance discussion. Intelligent Transportation Systems (ITS) have evolved over the past decade into well accepted means of decreasing congestion and improving safety and security in the transport of people and goods. At present, ITS data are obtained from ground-based sensors. Similar data can be collected from remote sensing from airborne platforms or
sensing based on vehicle sensors exist. An advantage of air-based platforms, including
manned and unmanned fixed-wing aircraft and helicopters is that they can be rapidly
204 deployed and for observing traffic incidents that occur in areas where there are no sensor in the ground-based system. This disadvantage of sensing data from airborne platforms is the much shorter duration than ground-based systems. The research in this dissertation focused on LiDAR and digital camera systems and made contributions to several specific areas.
LiDAR data are the primary source for road surface extraction, and finding the edge lines is essential to the subsequent vehicle extraction. A technique to determine road edge lines based on the shape of the road was investigated. To support vehicle extraction from airborne imagery, a method was introduced that provides a true object scale data representation that can facilitate the vehicle extraction. In addition, the proposed orthoimage domain provides an easy way to detect moving vehicles (or any moving objects) from overlapping image sequences. The vehicle extraction from LiDAR data was combined with coarse classification of the extracted vehicles; basically, grouping the vehicles into three major categories based on their size. The vehicle velocity analysis revealed that optical imagery acquired at adequate capture rate is far superior to LiDAR- based speed estimation. A novel method was introduced for simultaneously acquired
LiDAR and image data, which can combine the advantages of the two sensors for obtaining better velocity estimates of LiDAR extracted vehicles. The idea is to use the scale-preserving feature of the optical imagery, and thus, transfer the scale to obtain better vehicle length estimates for LiDAR extracted vehicles. Rigorous error analysis was developed to provide performance assessment of the methods and to compare with test results, including a dataset where reference vehicle data was available.
205
Our research results and extensive testing demonstrated that airborne remote
sensing based on state-of-the-art airborne laserscanners and digital camera systems can
provide traffic flow data that can support traffic monitoring and management. In
particular, laser data proved to be a good source for vehicle extraction and coarse classification. Due to the high price of sensors, however, missions dedicated to traffic flow data acquisition are not economic at the present time. Nevertheless, the opportunity exists to integrate these techniques into regular airborne mapping surveys. In particular, this is the case for the laser data, where the vehicles can be removed from the LiDAR point cloud during regular topographic mapping, making the additional effort needed to create flow data very limited. As a substantial amount of laser data, as well as imagery, is collected for routine aerial mapping over transportation corridors and in urban areas with dense road networks, a significant volume of traffic flow data can already be obtained at practically no extra effort. Similarly, digital sensor systems can be turned on to collect data during transit between project areas if the flight path passes over transportation corridors and thus, at almost no additional cost, large amounts of data, rich in traffic flow information can be acquired.
6.2. Future recommendations
The highly-automated extraction of traffic flow, including vehicle detection, classification, and tracking, using airborne and spaceborne platforms is an emerging field of research. A wide variety of sensors, including aerial and satellite optical imagery, airborne and terrestrial laserscanning, aerial and satellite SAR systems, infrared cameras,
206
hyperspectral sensors, can provide complementary and often redundant geospatial data
that has both the spatial and temporal resolution to effectively support traffic monitoring
and management. The primary objective of using the remote sensing technology is to
complement the static data acquisition infrastructure that includes induction loops, video
cameras, radar sensors, and to provide data over larger segments of the road network.
With respect to airborne platforms, substantial technological developments are expected in the near future that will improve the performance of traffic flow extraction from remote sensory data. In particular, the following topics are likely to have an impact on future systems:
Digital cameras are still rapidly developing and consequently the current 100+
Megapixel high-end systems will be superseded by 200-500 Megapixel sensors
with similar or even better image capture rates. In the medium-format airborne
digital camera category, the current 35-50 Megapixel companion digital cameras
used with LiDAR systems are expected to be replaced by models offering a 10 K
by 10 K image size and faster image capture (1+ FPS).
The improving laser technology will allow for further increasing of the point
density of LiDAR systems. The multiple-pulse-in-the-air technique will be
ubiquitous, as this is the only way to avoid the limitations imposed by the travel
time of the pulse between the sensor and the objects. Note that the past five years
saw an increase of the factor of five in the laser pulse rate, and current systems are
at the 200 kHz pulse rate.
207
The availability of GIS data, such as the NAVTEQ road database of the USA and,
more importantly, the access to these databases are expected to further improve
mainly in two aspects. First, more data in improved resolution with better
accuracy are becoming available in GIS and CAD databases. Second, higher
communication bandwidth will allow for easier access. Note that these
developments are primarily fueled by the major Internet companies that started
introducing global services, providing the online access to images, terrain models,
vector data, and other geospatial data.
UAV aircraft technology has seen major developments recently, and offers a cost-
effective platform for geospatial data acquisition in a variety of applications.
Unfortunately, the deployment of this technology is hindered by the lack of
institutional support. In fact, from a technical perspective, this is a mature
technology, and the real question is how the institutional and liability issues could
be resolved so these systems can be deployed in production.
There are a number of research problems related to the extraction of traffic information from remotely sensed data. The most important research topics are listed below:
Algorithmic research in computer vision is expected to significantly improve the
performance of the feature extraction processes of the optical imagery, which is
essential to handle the large volume of optical imagery. Similarly, feature
extraction from the LiDAR point cloud is expected to improve. Obviously, these
developments are not specific to traffic flow extraction.
208
Another generic research topic is the fusion of multi-sensory data, which is
essential to improve the robustness of the feature extraction processes. The higher
redundancy in data, in terms multiple observation from the same sensor, as well
as from different sensors, is also essential to achieve better feature extraction
performance.
The separation of static and non-static components from both optical and laser
data is an area of great interest in traffic flow extraction. Except for stopped
traffic, moving vehicles should be detected.
Knowledge-based systems are likely to become an integral part of applications
where traffic flow is extracted from remote sensory data. For example, automated
learning processes are required for feature extraction, with or without training
data. In some cases, there may be no training data available or the system is too
dynamic and model should be continuously updated. Similarly, to support better
interaction with decision-makers, natural language interface would be preferred.
The interpretation of complex traffic patterns and motion patterns also
necessitates the use of knowledge-based techniques.
Data obtained over longer road segments could be substantially different from
data acquired by conventional detectors at fixed locations. It is likely that valuable
traffic information can be extracted from this new type of data.
Combining remotely sensed and conventionally collected data could contribute to
traffic flow modeling developments.
209
Finally, there are a few practical aspects of using airborne remote sensors in
support of traffic flow extraction:
A substantial amount of airborne data is collected over transportation corridors
and in urban areas with dense road networks. In these datasets, vehicles on the
road represent obstructions to the LiDAR pulse, as they are reflected back from
the vehicles instead of the pavement. Similarly, they cover the surface area to be
observed in optical imagery. Therefore, considerable effort must be devoted to the
“removal of the vehicles” in both data domains in normal processing. However,
rather than simply removing and discarding these signals, they can be turned into
useful traffic flow information at relatively limited effort.
In addition to routine data collection for mapping purposes over transportation
corridors, airborne sensors can be turned on to capture data during transit, which
accounts for substantial flying time. Thus, at almost no additional cost, a
significant amount of data, rich in traffic flow information, can be acquired.
Decreasing hardware cost is likely to lead to a price breakpoint where operational
systems dedicated exclusively to traffic flow extraction will be feasible at some
point in the future.
6.3. Emerging trend: terrain-referenced navigation
Terrain-referenced navigation (TRN) techniques (Runnalls et al., 2005) are of increasing interest in the research community as they can provide alternative navigation tools when GPS is not available or when the GPS signals are jammed. GPS augmentation
210 is typically required in military applications, and it provides alternative position and attitude fixes to an inertial navigation system, which drifts in time if not calibrated by
GPS. With improving imaging sensor performance, as well as growing worldwide availability of terrain data, terrain-based navigation is becoming a viable option to support navigation in GPS-denied environments. Furthermore, the feedback from the imaging sensors can be used even during GPS availability. The feedback increases the redundancy at data input to the navigation filter, enabling more reliable integrity monitoring at this stage. The relevance of TRN to transportation application is twofold:
(1) the process of obtaining real-time position and attitude fixes for the navigation filter is based on feature extraction, and, in particular, on the capability to separate the static and dynamic objects from the image data, and (2) the use of already available terrain data, including surface model (DEM), raster or vector data in CAD/GIS environment, can effectively support the vehicle search and extraction processes. These two tasks could overlap, although the separation of the static and dynamic objects should work without any terrain data. In fact, this is to a large extent the idea behind the extraction of vehicles
(moving) from remotely sensed imagery. Fig. 6.1 shows the overall TRN concept where
LiDAR and optical imagery are matched with existing terrain data; for more details see
(Toth et al., 2008a-b).
211
Figure 6.1: The concept of terrain-based navigation
212
LIST OF REFERENCES
Ackerman, F., 1994. Practical Experience with GPS supported Aerial Triangulation, The Photogrammetric Record, XIV(84), pp. 861-874.
Akca, D., and Gruen, A., 2007. Generalized Least Squares Multiple 3D Surface Matching, ISPRS Volume XXXVI, Part 3 / W52.
ASPRS LiDAR Committee, 2004. ASPRS Guidelines Vertical Accuracy Reporting for LiDAR Data,http://www.asprs.org/society/committees/lidar/Downloads/Vertical_Accuracy_Reporting_f or_Lidar_Data.pdf
Baltsavias, E.P., 1999. Airborne laser scanning: basic relations and formulas. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 54: 199-214.
Behan, A., 2000. On the matching accuracy of rasterized scanning laser altimeter data. . International Archives of Photogrammetry and Remote Sensing, 33 (Part 2B): 75-82.
Behan, A., Maas, H.-G., and Vosselman, G., 2000. Steps towards Quality Improvement of Airborne Laser Scanner Data, Proceedings of the 26th Annual Conference of the Remote Sensing Society, Leicester, September 12-14, on CD-ROM, 9 pages.
Besl, P.J., and McKay, N.D., 1992. A method for registration of 3-d shapes, IEEE Trans. Pat. Anal. and Mach. Intel. 14(2), pp 239-256.
Bossler J., 1992. GPS Van: Input to GIS, Proc. of ION GPS, pp. 427-437.
Bretar, F., Roux, M., and Pierrot-Deseilligny, M., 2004. Solving the Strip Adjustment Problem of 3D Airborne Lidar Data. Proceedings of the IEEE IGARSS’04, Anchorage, Alaska, September 2004.
Burman, H., 2000. Adjustment of laser scanner data for correction of orientation errors. International Archives of Photogrammetry and Remote Sensing, 33 (Part B3): 125-128.
Burman, H., 2002. Laser strip adjustment for data calibration and verification. International Archives of Photogrammetry and Remote Sensing, 34 (Part 3A): 67-72.
213
Burns, S. G. and Wendt, J. J., 2003. Inductive Loop Detector Vehicle Signature Analysis, ITS Institute Advanced Transportation Technologies Seminar Series
Campbell, R.J., and Flynn, P.J., 2001. A survey of free-form object representation and recognition techniques. Computer Vision and Image Understanding, 81(2), 166-210.
Chowdhury, M. A., Sadek, A., 2003. Fundamentals of Intelligent Transportation System Planning, Artech House Publishers.
Crombaghs, M.J.E., Brügelmann, R., and de Min, E.J., 2000. On the adjustment of overlapping strips of laseraltimeter height data. International Archives of Photogrammetry and Remote Sensing, 33, (Part B3/1):224-231.
Csanyi, N., Toth, C., 2007. Ground-based Aerial Photography to Support Traffic Flow Extraction, ASPRS Annual Conference, Tampa, Florida, May 7–11.
Csanyi, N., Toth, C., Grejner-Brzezinska, D., Ray, J., 2005. Improving LiDAR Data Accuracy Using LiDAR-Specific Ground Targets, ASPRS Annual Conference, Baltimore, MD, March 7- 11, CD-ROM.
Csanyi May, N., 2008. A Rigorous Approach to Comprehensive Performance Analysis of State- of-the-Art Airborne Mobile Mapping Systems, PhD dissertation, The Ohio State University.
Csanyi N., Paska E,. and Toth C., 2003. Comparison of Various Surface Modeling Methods, Terrain Data: Applications and Visualization – Making the Connection, ASPRS/MAPPS, Charleston, SC, October 27-30, 2003, CD-ROM.
Csanyi N, Toth C., Grejner-Brzezinska D., and Ray, J., 2005. Improving LiDAR data accuracy using LiDAR-specific ground targets, ASPRS Annual Conference, Baltimore, MD, March 7-11, CD-ROM.
Csanyi, N. and Toth, C., 2007. Improvement of LiDAR Data Accuracy Using LiDAR-Specific Ground Targets, Photogrammetric Engineering & Remote Sensing, Vol. 73, No. 4, pp. 385-396.
Doucet, A., N. de Freitas and N. Gordon, 2001. Sequential Monte-Carlo Methods in Practice, Springer-Verlag.
Dubayah, R. and Blair, J., 2000. Lidar Remote Sensing for Forestry Applications. Journal of Forestry 98(6), pp. 44–46.
Duda R.O., Hart P.E., 1972. Use of the Hough Transformation to detect lines and curves in pictures, Graphics and Image processing, 15: 11-15.
214
El-Sheimy, N., Valeo, C., and Habib, A., 2005. Digital Terrain Modeling: Acquisition, Manipulation, and Applications, Artech House.
Filin, S., 2001. Recovery of systematic biases in laser altimeters using natural surfaces, International Archives of Photogrammetry and Remote Sensing, 34, (3/W4), pp. 85-91.
Filin, S., 2002. Surface clustering from airborne laser scanning data, International Archives of Photogrammetry and Remote Sensing, 34, (3A), pp. 117-124.
Filin, S., 2003a. Recovery of systematic biases in laser altimetry data using natural surfaces, ISPRS Journal of Photogrammetric Engineering and Remote Sensing, 69 (11), 1235–1242.
Filin, S., 2003b. Analysis and implementation of a laser strip adjustment model. International Archives of Photogrammetry and Remote Sensing, 34 (Part 3/W13): 65-70.
Filin, S., and Vosselman, G., 2004. Adjustment of Laser Altimetry Strips. International Archives of Photogrammetry and Remote Sensing, 34 (Part 3/W13): 285-289.
Gartner, N.H, C, J. Messer, and A. K. Rathi, 1997. Revised Monograph on Traffic Flow Theory, www.tfhrc.gov/its/tft/tft.htm.
Gautier, J.D. and B.W. Parkinson, 2003. Using the GPS/INS Generalized Evaluation Tool (Comparison of loosely-coupled, tightly-coupled and ultra-tightly coupled integration navigation). Proc. of ION AM-2003, Albuquerque, NM, pp. 65-76.
Gesch, D., Oimoen, M., Greenlee, S., Nelson, C., Steuck, M., and Tyler, D., 2002. The National Elevation Dataset, Photogrammetric Engineering & Remote Sensing, Vol. 68, No. 1, pp. 5-13.
Grejner-Brzezinska, D., Li, R., Haala, N., Toth, C., 2004a. From Mobile Mapping to Telegeoinformatics: Paradigm Shift in Geospatial Data Acquisition, Processing and Management, Photogrammetric Engineering and Remote Sensing, Vol. 70, No. 2, pp. 197-210, January 2004.
Grejner-Brzezinska D., Toth, C.,Paska, E., 2004b. Airborne Remote Sensing Supporting Traffic Flow Estimates, Proc. of 3rd International Symposium on Mobile Mapping Technology, Kunming, China, March 29-31, 2004, CD-ROM.
Grejner-Brzezinska, D., Toth, C., Paska, E., 2003. Airborne Remote Sensing: Redefining a Paradigm of Traffic Flow Monitoring, ION GPS 2003, Portland, Sept 9-12, CD-ROM.
Gruber, M., F. Leberl, R. Perko, 2003. Paradigmenwechsel in der Photogrammetrie durch digitale Luftbildaufnahme? Photogrammetrie, Fernerkundung, Geoinformation (PFG), August 2003
215
Gruen, A., and Akca, D., 2005. Least squares 3D surface and curve matching. ISPRS Journal of Photogrammetry and Remote Sensing, 59 (3), 151-174.
Gruen, A, 2007. The 5th International Symposium on Mobile Mapping Technology (MMT 2007): Mobile Mapping with UAV Technology, Keynote Lecture, University of Padua, Italy. http://www.geod.ethz.ch/p02/general/persons/AG_pub/gruen_padua_keynote_MMT.pdf
Han, D., Lee, J., Kim, Y., and Yu, K., 2006. Adjustment for Disrepencies Between ALS Data Strips Using Contour Tree Algorithm, ACVIS, LNCS 4179, pp. 1026-1036.
Heywood, I., Cornelius, S., and Carver, S., 1998. An Introduction to Geographical Information Systems, New Jersey, Prentice Hall.
Hofmann-Wellenhof, B., Lichtenegger, H., Collins, J. Global Positioning System, 2001. Theory and Practice, 5th rev. ed., XXII, 382 p. SpringerWienNewYork
Hough, P.V.C., 1959. Machine analysis of bubble chamber pictures, International Conference on High Energy Accelerators and Instrumentation, CERN.
Hu, X., and Tao, V., 2004. Extraction of Streets in Dense Urban Area from Segmented LiDAR data, The 4th International Symposium of Mobile Mapping Technology, Kunming, China, March 29-31, 2004, CD-ROM.
Hu, X., Zhang, Z. and Tao, V., 2004. A Robust Method for Semi-Automatic Extraction of Road Centerlines Using a Piecewise Parabolic Model and Least Square Template Matching, Photogrammetric Engineering and Remote Sensing, Vol. 70, No. 12, pp. 1393-1398.
Jazwinski A. H., 1970. Stochastic Process and Filtering Theory, New York: Academic Press.
Julier S. J. and J. K. Uhlmann, 1997. A New Extension of the Kalman Filter to Nonlinear Systems, Proc. of the SPIE AeroSense International Symposium on Aerospace/Defense Sensing, Simulation and Controls, Orlando, Florida. Kager, H., and Kraus, K., 2001. Height discrepancies between overlapping laser scanner strips. Proceedings of Optical 3D Measurement Techniques V, October, Vienna, Austria: 103-110.
Kilian J., Haala, N., and Englich, M., 1996. Capture and evaluation of airborne laser scanner data. International Archives of Photogrammetry and Remote Sensing, 31 (Part B3), pp. 383-388.
Knoeppel, C, Schanz, A. and Michaeleis, B., 2000. Robust vehicle detection at large distance using low resolution cameras, IEEE Intelligent Vehicle Symposium, pp. 267-272.
Kocaman, S., L. Zhang, A. Gruen, 2006. Self-Calibrating Triangulation of Airborne Linear Array CCD Cameras, International Calibration and Orientation Workshop (EuroCOW 2006), Castelldefels, Catalonia, Spain, January 25-27.
216
Kolodziej, K. W. and Hjelm, J., 2006, Local Positioning Systems: LBS Applications and Services, CRC Taylor & Francis.
Kornus, W., and Ruiz, A., 2003. Strip adjustment of lidar data, Vol. XXXIV-3/W. Dresden: International Archives of Photogrammetry and Remote Sensing.
Kozo Ichida and Takeshi Kiyono, 1977. Curve Fitting by a One-Pass Method With a Piecewise Cubic Polynomial. ACM Transactions on Mathematical Software, Vol 3. No 2, pages 164-174.
Krauss, K., 1993. Photogrammetry, Volume 1: Ferd. Dümmler Verlag, Bonn
Kukko, A., and Hyyppä, J., 2007. Laser Scanner Simulator for System Analysis and Algorithm Development: A Case with Forest Measurements, International Archives of Photogrammetry and Remote Sensing, 36, Part 3/W52, pp. 234-240.
Leberl, F., M. Gruber, M. Ponticelli, S. Bernoegger, R. Perko, 2003. The Ultracam Large Format Aerial Digital Camera System, Proceedings of ASPRS 2003 Conference, May 5 –9, Anchorage, Alaska.
Leeuwen, M. and Groen, F., 2000. “Motion estimates in image sequences for traffic applications”, IMTC, Vol. 1, pp. 354-359.
Lillesand, T.M, Kiefer, R.W. and Chipman, J.W., 2003. Remote Sensing and Image Interpretation, John Wiley & Sons Inc.,
Liu, J. and R. Chen, 1998. Sequential Monte Carlo Methods for Dynamic Systems, Journal of the American Statistical Association, 93:1032–1044
Loan, V., and Charles, F., 1997. Introduction to Scientific Computing, New Jersey: Prentice Hall
Lowe, David G., 1999. "Object recognition from local scale-invariant features". Proceedings of the International Conference on Computer Vision 2: 1150–1157.
Lowe, David G., 2004. "Distinctive Image Features from Scale-Invariant Keypoints". International Journal of Computer Vision 60 (2): 91–110.
Maas, H.-G., 2000. Least squares matching with airborne laserscanning data in a TIN structure. International Archives of Photogrammetry and Remote Sensing, 33 (Part B3/1): 548-555.
Maas, H.-G., 2001. On the use of pulse reflectance data for laserscanner strip adjustment. International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 33 (Part 3/W4): 53-56.
217
Maas, H.-G., 2002. Methods for Measuring Height and Planimetry Discrepencies in Airborne Laserscanner Data. Photogrammetric Engineering & Remote Sensing, Vol. 68, No. 9. pp. 933- 940.
Maas, H.-G., 2003. Planimetric and height accuracy of airborne laserscanner data: User requirements and system performance, Proceedings 49. Photogrammetric Week (Ed. D. Fritsch), Wichmann Verlag.
Mader, G.L., 1992. Rapid Static and Kinematic Global Positioning System Solutions Using the Ambiguity Function Technique, Journal of Geophysical Research, 97, 3271-3283.
Madhavan, R., Hong, T., and Messina, E., 2005. Temporal Range Registration for Unmanned Ground and Aerial Vehicles, Journal of Intelligent and Robotic Systems, Volume 44, Number 1, pp. 47-69.
Mannering, Kilareski, and Washburn, 2009. Principles of Highway Engineering and Traffic Analysis, 4th Edition, John Wiley and Sons, Inc., 2009, ISBN 978-0-470-29075-0
Manual of Photogrammetry, 2004. Fifth Edition, Chris McGlone, Editor; Edward Mikhail & Jim Bethel Associate Editors.
Maune, D. (ed.), 2007. Digital Elevation Model Technologies and Applications: The DEM Users Manual, 2nd edition, American Society for Photogrammetry and Remote Sensing, Bethesda, MD.
McCord, M., Goel, P., Jiang, Z., Coifman, B., Yang, Y., Merry, C., 2002. Improving AADT and VDT Estimation With High-Resolution Satellite Imagery, Pecora 15/Land Satellite Information IV/ISPRS Commission I/FIEOS 2002 Conference Proceedings.
McGowan, S., 2004, Building the Nation’s First Statewide RTK GPS Infrastructure, Ohio DOT’s VRS Network.
Merry, C.J., McCord, M.R., Goel, P.K., 1999. Satellite Data Use for Traffic Monitoring, Proceedings of the 14th William T. Pecora Memorial Remote Sensing Symposium (CD-ROM), 6-10 December, Denver, Colorado, pp. 183-190.
Mikhail M. E., J. S. Bethel, J. C. McGlone, 2001. Introduction to Modern Photogrammetry, John Wiley & Sons, Inc.
Mirchandani, P., Hickman, M., Angel, A., Chandnani, D., 2003. Application of Aerial Video for Traffic Flow Monitoring and Management, EOM Vol.12, No .4, pp.10-17.
218
Moafipoor, S., Toth C., and Grejner-Brzezinska, D., 2005. Traffic Flow Estimate from LiDAR data: Operational Experiences, Proceedings of the 16th William T. Pecora Memorial Remote Sensing Symposium, 24-27 October, Sioux Falls, SD CD-ROM.
Morin, K., and El-Sheimy, N., 2002. Post-mission adjustment of airborne laser scanning data, Proceedings XXII FIG International Congress, Washington DC, USA, 19–26 April, 12 p. (on CDROM).
NCRST-F, http://www.ncrst.org/research/ncrst-f/ncrst-f_home.html
Okabe, A., Boots, B., Sugihara, K., and Chiu, S.N., 2000. Spatial Tessellations - Concepts and Applications of Voronoi Diagrams, 2nd edition, John Wiley.
Oliver, M.A., and Webster, R., 1990. Kriging: a method of interpolation for geographical information system, INT. J. Geographical Information Systems, VOL. 4, No. 3, 313-332.
Paska, E., Toth, C., 2005. Vehicle Velocity Estimation from Airborne Imagery and LiDAR, ASPRS Annual Conference, Baltimore, Maryland, CD ROM.
Perko, R., Gruber, M., 2002. Comparison of Quality and Information Content of Digital and Film-Based Images. Proc. ISPRS Commission III, Graz, Austria.
Pfeifer, N., and Briese, C., 2007. Geometrical Aspects of Airborne Laser Scanning and Terrestrial Laser Scanning, International Archives of Photogrammetry, Vol. 36 (Part 3/W52): 311-319.
Pline, J.L., 1992. Transportation and Traffic Engineering Handbook, 4th Edition, Prentice Hall.
Ramprakash, V.L., 2003. Detection and estimation of Vehicular Movement on Highways using a LiDAR sensor, MS.C. Thesis, The Ohio State University.
Renslow, M., 2005. The Status of LiDAR Today and Future Directions, 3D Mapping from InSAR and LiDAR, ISPRS WG I/2 Workshop, Banff, Canada, June 7-10, CD-ROM.
Ryerson, R. Editor-in-Chief; Volume Editors: Floyd M. Henderson and Anthony J. Lewis, 1998. The Manual of Remote Sensing, 3rd edition: Volume 2: Principles & Applications of Imaging Radar, John Wiley & Sons, Inc.
Sandau, R., Braunecker, B., Driescher, H., Eckardt, A., Hilbert, S., Hutton, J., Kirchhofer, W., Lithopoulos, E., Reulke, R., Wicki, S., 2000. Design Principle of The LH Systems ADS40 Airborne Digital Sensor. The International Archives of Photogrammetry and Remote Sensing, Amsterdam, Vol. 33, Part B1, pp. 258-265.
219
Saufel, K, 2001. Robust method to separate static and non-static: PhD dissertation, Carnegie Mellon.
Schenk, T., 2001. Modeling and Analyzing Systematic Errors in Airborne Laser Scanners, Technical Notes in Photogrammetry, Vol. 19, The Ohio State University, Columbus, USA, 46 pages.
Schowengerdt, Robert A., 1997. Remote Sensing, Academic Press.
Schuckman, K., Hoffman, G., 2004. ARIES: Technology Fusion for Emergency Response, Photogrammetric Engineering and Remote Sensing, Vol. 70, No. 4, pp. 357-360.
Shannon, C.E., 1948. A Mathematical Theory of Communication, Bell System Technical Journal, Vol. 27, pp. 379–423, 623–656.
Shannon, C.E., 1949. Communication in the presence of noise, Proc. Institute of Radio Engineers, Vol. 37, no.1, pp. 10-21.
Shastry, A., Schowengerdt, R., 2002. Airborne Video Registration For Visualization And Parameter Estimation Of Traffic Flows, Pecora 15/Land Satellite Information IV/ISPRS Commission I/FIEOS 2002 Conference Proceedings
Sithole, G., Vosselman, G., 2004. Experimental Comparison of Filter Algorithms for Bare Earth Extraction From Airborne Laser Scanning Point Clouds. ISPRS Journal of Photogrammetry and Remote Sensing 59 (1-2): 85-101.
Skaloud, J., and Lichti, D., 2006. Rigorous approach to bore-sight self-calibration in airborne laser scanning, International Journal of Photogrammetry and Remote Sensing, Vol. 61, pp. 47- 59.
Soininen, A., and Burman, H., 2005. TerraMatch for MicroStation. Terrasolid Ltd, Finland. Toth C. K., 2002. Calibrating Airborne LIDAR Systems, International Archives of Photogrammetry and Remote Sensing, Vol. XXXIV, part 2, pp.475-480.
Thomas, T. R., 1999. Rough Surfaces, Imperial College Press, pp: 100-110.
Toth, C., Grejner-Brzezinska D., Lovas, T., 2003. Traffic Flow Estimates from LiDAR Data, Proc. ASPRS Annual Conference, May 5-9, pp. 203-212, CD-ROM
Toth, C. K. and Grejner-Brzezinska, D. A., 2006. Extracting Dynamic Spatial Data from Airborne Imaging Sensors to Support Traffic Flow Estimation, ISPRS Journal of Photogrammetry & Remote Sensing, Vol. 61, pp. 137–148.
220
Toth, C., Csanyi, N. and Grejner-Brzezinska, D., 2002. Automating the calibration of airborne multisensor imaging systems, Proc. ACSM-ASPRS Annual Conference, Washington, DC, April 19-26, CD ROM.
Toth, C. 2004. Future Trends in LiDAR, Proc. ASPRS 2004 Annual Conference, Denver, CO, May 23-28, CD-ROM.
Toth , C. K. and Paska, E., 2006. Mobile Mapping and Autonomous Vehicle Navigation, ISPRS Commission I Symposium, Paris, France, July 4-6.
Toth, C. K., Paska, E., Chen, Q., Zhu, Y., Redmill, R. and Ozguner, U., 2006. Mapping Support for the OSU DARPA Grand Challenge Vehicle, Proceedings of the 2006 IEEE Intelligent Transportation Systems Conference, Toronto, Canada, September 17-20, pp. 1580-1585.
Toth C., D. Grejner-Brzezinska, 2006. Extracting Dynamic Spatial Data from Airborne Imaging Sensors to Support Traffic Flow Estimation, ISPRS Journal of Photogrammetry & Remote Sensing, 61, 137-148.
Toth, C. K., D. Grejner-Brzezinska and M. Bevis, 2006. Extreme Precision LiDAR Mapping, From Sensors to Imagery, ISPRS Commission I Symposium, Paris, France, July 4-6.
Toth, C., D. Brzezinska, N. Csanyi, E. Paska and N. Yastikly, 2007. LiDAR Mapping Supporting Earthquake Research of the San Andreas Fault, ASPRS Annual Conference, Tampa, Florida, May 7–11.
Toth, Ch, and A. Krupnik, 1996. ``Concept, Implementation, and Results of an Automatic Aerotriangulation System," Photogrammetric Engineering and Remote Sensing, Vol. 62, No. 6, pp. 711-717, June 1996.
US Bureau of Transportation Statistics, http://www.bts.gov
Vosselman, G. and Maas, H.-G., 2001. Adjustment and filtering of raw laser altimetry data. Proceedings OEEPE Workshop on Airborne Laserscanning and Interferometric SAR for Detailed Elevation Models. OEEPE Publications no. 40: 62-72.
Vosselman, G., 2002a. On the estimation of planimetric offsets in laser altimetry data. International Archives of Photogrammetry and Remote Sensing, 34 (Part 3A): 375-380.
Vosselman, G., 2002b. Strip offset estimation using linear features. 3rd International LIDAR Workshop, October 7-9, Columbus, http://www.itc.nl/personal/vosselman/papers/vosselman2002.columbus.pdf
Xiong, Z. and Zhang, Y., 2008. An initial study on vehicle information extraction from single pass QuickBird satellite imagery, PERS Vol. 74. 11, pp. 1401-1411.
221
Yi, Y, 2007. On improving the accuracy and reliability of GPS/INS-based direct sensor georeferencing, PhD dissertation, The Ohio State University.
222
APPENDIX A
A.1. Matlab code for pavement extraction
The basic program input and output definition is provided in Table A.1.
Pavement marking extraction (PavMarkExtract_main)
Input LiDAR strip points Control point
Boundary value
Threshold for residuals
Threshold for mean residuals
Intensity step
Output Selected Lidar pavement marking points
Embedded BoundaryExtract.m macros/functions
Table A.1: Program input/output specification.
223
A.2. Matlab code for for ICP-based matching, curve fitting and QA/QC computation
The basic program input and output definition is provided in Table A.2.
Main program (ICP_MAIN)
Input Control points Extracted LiDAR pavement marking points Output Transformation parameters between control points and extracted LiDAR pavement marking points: o Local shift o Global shift o Rotation angles o Rotation matrix 3x3 Statistics: o A posteriori dispersion matrix of transformation parameters o Residuals: original and final, mean and standard deviation Embedded transform2D macros/functions iterative_adjustment_transform2D curve_fit ItCPmultiplecurves_measofconv_ends_checkrange_ resolution_curve search_np eukldist nearestpoint_ends_search_checkrange pcf2 pcf2_controls_valueconstraint pcf2_linearinterpolation residuals
Table A.2: Program input/output specification.
224