Modeling and Matching of Landmarks for Automation of Mars Rover Localization

Home , Argo (crater), Beagle (crater), Endurance (crater), Erebus (crater), Fram (crater), Navcam

MODELING AND MATCHING OF LANDMARKS FOR AUTOMATION OF MARS ROVER LOCALIZATION

DISSERTATION

Presented in Partial Fulfillment of the Requirements for

the Degree Doctor of Philosophy in the Graduate

School of The Ohio State University

JUE WANG, M.S., B.S.

*****

The Ohio State University 2008

Dissertation Committee: Approved by Dr. Rongxing Li, Advisor

Dr. Tony Schenk ______Dr. Alper Yilmaz Advisor Graduate Program in Geodetic Science and Surveying

Jue Wang

2008

ABSTRACT

The Mars Exploration Rover (MER) mission, begun in January 2004, has been extremely successful. However, decision-making for many operation tasks of the current

MER mission and the 1997 Mars Pathfinder mission is performed on Earth through a predominantly manual, time-consuming process. Unmanned planetary rover navigation is ideally expected to reduce rover idle time, diminish the need for entering safe-mode, and dynamically handle opportunistic science events without required communication to

Earth. Successful automation of rover navigation and localization during the extraterrestrial exploration requires that accurate position and attitude information can be received by a rover and that the rover has the support of simultaneous localization and mapping. An integrated approach with Bundle Adjustment (BA) and Visual Odometry

(VO) can efficiently refine the rover position. However, during the MER mission, BA is done manually because of the difficulty in the automation of the cross-site tie points selection. This dissertation proposes an automatic approach to select cross-site tie points from multiple rover sites based on the methods of landmark extraction, landmark modeling, and landmark matching.

The first step in this approach is that important landmarks such as craters and rocks are defined. Methods of automatic feature extraction and landmark modeling are

ii then introduced. Complex models with orientation angles and simple models without those angles are compared. The results have shown that simple models can provide reasonably good results. Next, the sensitivity of different modeling parameters is analyzed. Based on this analysis, cross-site rocks are matched through two complementary stages: rock distribution pattern matching and rock model matching. In addition, a preliminary experiment on orbital and ground landmark matching is also briefly introduced. Finally, the reliability of the cross-site tie points selection is validated by fault detection, which considers the mapping capability of MER cameras and the reason for mismatches. Fault detection strategies are applied in each step of the cross-site tie points selection to automatically verify the accuracy. The mismatches are excluded and localization errors are minimized.

The method proposed in this dissertation is demonstrated with the datasets from the 2004 MER mission (traverse of 318 m) as well as the simulated test data at Silver

Lake (traverse of 5.5 km), California. The accuracy analysis demonstrates that the algorithm is efficient at automatically selecting a sufficient number of well-distributed high-quality tie points to link the ground images into an image network for BA. The method worked successfully along with a continuous 1.1 km stretch. With the BA performed, highly accurate maps can be created to help the rover to navigate precisely and automatically. The method also enables autonomous long-range Mars rover localization.

iii

To my families, who have supported me all the time

ACKNOWLEDGMENTS

First of all, I would like to express my sincere gratitude to my advisor, Dr.

Rongxing (Ron) Li, for his outstanding guidance, constant encouragement and patience.

The many opportunities that he gave me to participate in various projects have stimulated

my interests in different research fields and enabled me to make a contribution to the

prestigious Mars project.

I would also like to extend my sincere appreciation to Dr. Tony Schenk and Dr.

Alper Yilmaz for serving on my dissertation committee as reviewers and examiners.

Moreover, I would like to thank them for their valuable comments and suggestions.

I wish to thank the current and previous Mars team members at the OSU

Mapping and GIS Laboratory: Dr. Kaichang Di, Dr. Bo Wu, Dr. Fengliang Xu, Dr.

Xutong Niu, Shaojun He, Ju Won Hwangbo, Lin Yan, Yunhang Chen, Wei Chen,

Jeremiah Glascock, Sanchit Agarwal, Evgenia Brodyagina, Charles Serafy, and Eric

Oberg, who have encouraged me and helped me with their insights. Many of the maps

and results in this dissertation would not be possibly produced without teamwork. My

appreciation should also be extended to other colleagues who have worked in this lab: Dr.

Ruijin Ma, Dr. Tarig A. Ali, Leslie Smith, Sagar Deshpande, I-Chieh Lee, and Alok

Srivastava. We always have interesting and good-spirited discussions in the lab.

v In addition, I am grateful to the faculty, staff, and students in the Department of

Civil and Environmental Engineering and Geodetic Science for creating and promoting a unique atmosphere of academic excellence. I benefited enormously from the advanced courses offered by the faculty members in this department.

I wish to thank all my friends. No matter where I am, they always encourage me when I am in adversity. I would also like to thank Ms. Eve A. Baker, Karla Edwards,

Lisya Seloni, Dr. Di and Dr. Wu for the proofreading of my dissertation.

Finally, I would like to express my deepest thanks to my parents, who contributed the way I am. They have supported me unconditionally all the time. Special thanks also go to my husband, Feng, and my lovely son, Michael. Without their support and tremendous love, this study could never have been completed.

I conducted my dissertation at the Mapping and GIS Laboratory of The Ohio

State University. The research was supported by NASA/JPL.

VITA

November, 1977…… Born in Zhejiang Province, P.R. China

July, 1999………….. B.S., Surveying Engineering Tongji University, Shanghai, P.R. China

March, 2002 ………. M.S., GIS and Mapping Tongji University, Shanghai, P.R. China

December, 2006 …... M.S., Geodetic and Geo-information Science, The Ohio State University

1999 – 2001……...... Student Tutor, Tongji University, Shanghai, P.R. China

1999 – 2002……...... Graduate Research Assistant Tongji University, Shanghai, P.R. China

September 2004 – Graduate Teaching Assistant, The Ohio State University December 2005….....

2002 – present …….. Graduate Research Assistant, The Ohio State University

PUBLICATIONS

Research Publications 1. Di, K., F. Xu, J. Wang, S. Agarwal, E. Brodyagina, R. Li, L. Matthies. 2007. Photogrammetric Processing of Rover Imagery of the 2003 Mars Exploration Rover Mission. ISPRS Journal of Photogrammetry and Remote Sensing, doi:10.1016/j.isprsjprs.2007.07.007, available online 12 September 2007. 2. Di, K., J. Wang, R. Ma, and R. Li. 2003. Automatic Shoreline Extraction from IKONOS Satellite Imagery. EOM (Earth Observation Magazine), Vol.12, No.7, pp. 14-18. vii 3. Li, R., K. Di, J. Wang, X. Niu, S. Agarwal, E. Brodyagina, E. Oberg, and J.W. Hwangbo. 2007. A WebGIS for Spatial Data Processing, Analysis, and Distribution for the MER 2003 Mission. Journal of Photogrammetric Engineering and Remote Sensing, Vol.73, No.6, pp.671-680. 4. Li, R., K. Di, A. Howard, L. Matthies, J. Wang, and S. Agarwal. 2007. Rock Modeling and Matching for Autonomous Long-Range Mars Rover Localization. Journal of Field Robotics, Vol.24, No.3, pp.187-203. Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/rob.20182. 5. Li, R., S. W. Squyres, R. E. Arvidson, B. A. Archinal, J. Bell, Y. Cheng, L. Crumpler, D. J. Des Marais, K. Di, T. A. Ely, M. Golombek, E. Graat, J. Grant, J. Guinn, A. Johnson, R. Greeley, R. L. Kirk, M. Maimone, L. H. Matthies, M. Malin, T. Parker, M. Sims, L. A. Soderblom, S. Thompson, J. Wang, P. Whelley, and F. Xu. 2005. Initial Results of Rover Localization and Topographic Mapping for the 2003 Mars Exploration Rover Mission. Journal of Photogrammetric Engineering and Remote Sensing, Special issue on mapping Mars, Vol.71, No.10, pp.1129-1142. 6. Wang, J., K. Di, and R. Li. 2005. Evaluation and Improvement of Geopositioning Accuracy of IKONOS Stereo Imagery. ASCE Journal of Surveying Engineering, Vol.131, No.2, pp.35-42. 7. Wang, J., and Chen, Y. 2001. The production of Digital Orthophoto Map and its further application. Remote Sensing Information (in Chinese), No. 2, Sum, No.62.

FIELDS OF STUDY

Major Field: Geodetic Science

Studies in: GIS Mapping & Cartography Photogrammetry Pattern Recognition and Computer Vision

viii

TABLE OF CONTENTS

Page

Abstract …………………………………………………………………………. ii Dedication ………………………………………………………………………. iv Acknowledgments ………………………………………………………………. v Vita …………………………………………………………………...………..... vii List of Tables…………………………………………………………………….. xii List of Figures …………………………………………………………………... xiv List of Abbreviations .……………………………………..………….…………. xvii

Chapters: 1 Introduction ………………………………………………………..……… 1 1.1 Background information……………………………………..…..……. 1 1.2 Literature review……………………………………..…..…..…..……. 9 1.3 Issues and significance of this research…………………..…..…..…… 16 1.4 Overview of the dissertation…………………..…..…..………….…… 20 2 Landmark Extraction and Modeling ……………………….……….…….. 22 2.1 Landmark definition…………………………………..….……….…… 22 2.2 Landmark extraction……………..…..…..………..…..…..…………... 31 2.2.1 Crater detection…………………..…..…..…………………….. 32 2.2.2 Rock extraction……..…..…..…….………..…..…..…………... 37 2.2.3 Limitations in landmark extraction…………………..…..…….. 41 2.3 Landmark modeling ...………………..…..…..………..…..…..…….... 42 2.3.1 Crater modeling…………………..…..…..………..…..…..…... 43 2.3.2 Rock modeling…………………..…..…..………..…..…..……. 44

ix 2.4 Results analysis of rock modeling………………..…..…..…..…....….. 48 2.4.1 The complex rock models with respect to the simple models

using simulated data ……...…………………..…..…..…..…..……… 48 2.4.2 The complex rock models with respect to the simple models

using real data…………………..…..…..…..…..…....…..…....……... 52 2.4.3 Rock modeling results …………………..…..…..…..…..…...... 54 3 Landmark Matching…………………..…..…..…..…..…....…..…....…..… 58 3.1 Background for the automated landmark matching.…..….. ..…..…….. 58 3.2 Landmark matching…..………………..…..…..…..…..……………… 62 3.2.1 Analysis of model parameter sensitivity…………………...….. 63 3.2.2 Landmark model matching .…..………….…..………………... 72 3.2.3 Landmark pattern matching .…..………….…..……………….. 74 3.3 Analysis of results ……………..…..…..…..…..…....…….…………... 78 3.4 Preliminary experiments on landmark matching between ground and

orbiter…………………..…..…..……………………..…..…..…..…..…… 82 3.4.1 Ground and orbital dataset…………………..…..…..…..…..…. 83 3.4.2 Landmark matching between the ground and orbital data…….. 86 4 Localization Error Analysis and Fault Detection…..…..………………….. 96 4.1 Localization error analysis …………………..…..…..……..……..…... 96 4.1.1 Analysis of rover localization accuracy ……..…..…..………… 97 4.1.2 Rover localization error estimation based on different

configuration of tie points…………………..…..…..…..……...... 101 4.2 Fault detection (FD) for exclusion of mismatched landmarks..…..…… 122 4.2.1 Fault modeling…………………..…..…..………..…....…....…. 123 4.2.2 System-level FD strategies…………………..…..…..………… 125 4.2.3 Case study…………………..…..…..………………………….. 129 5 Implementation and Performance Analysis…………………..…..…..…... 140 5.1 Implementation…………………..…..…..……………………………. 141 5.1.1 System development and integration…………………..…..…... 141 5.1.2 Speed optimization…………………..…..…..………………… 142

x 5.2 Test and performance analysis…………………..…..…..……….…… 144 5.2.1 Verification using MER-A data…………………..…..…..……. 146 5.2.2 Verification using field test data at Silver Lake, California……. 151 5.3 Contributions…………………..…..…..……………………..…..…… 157 5.4 Discussion and direction for future research…………………..…..….. 158

Appendix…..…..…..……….………..…..…..……….………..…..…..…... 161 A. Mathematical models for rocks………………………..…..…..……...... 161

Bibliography………………………………………………………………. 163

LIST OF TABLES

Table Page 1.1 MER camera parameters.…..……….………..…..…..……….………... 5 2.1 Orbital data information.…...……….…..……….….…..……….……... 24 2.2 Craters in the Meridiani Planum and Gusev Crater landing sites……… 29 2.3 Value of ratios from the crater model.…..……….………..…..…..…… 44 2.4 Model comparison based on the simulated dataset 1……..…..…..……. 50 2.5 Model comparison based on the simulated dataset 2……..…..…..……. 51 2.6 Model comparison based on the MER-A data at Sites 9600 and 9700.... 53 2.7 Model comparison based on the MER-A data at Site 11400..………….. 53 2.8 Model comparison based on the MER-A data at Site 11493..………….. 53 2.9 Estimated model parameters and RMS errors (φ in radian, all others in cm) of rocks in Figure 2.12…..……….………..…..…..….…..….…..… 55 2.10 Comparison of estimated model parameters with ground truth for 4 rocks in Figure 2.9....….….…..….….…..….….…..….….…..….……... 56 2.11 Average difference between the estimated model parameters and ground truth …..….….…..….….…..….….…..….………..….….…..…. 56 3.1 Comparison of modeling parameters of rocks from the current and adjacent site..…..………..…..………..…..………..…..………..………. 62 3.2 Parameter sensitivity of four models..…..……...…..……...…..……….. 69 3.3 Relative error in volume and surface area with respect to that of the dimension parameters.…..……...…..……...…..……………………….. 72 3.4 Summary of rock matching results for Sites 1200 and 1300.…..………. 78 3.5 Parameters of HiRISE image covering Victoria Crater..…..…….……... 84 3.6 Parameters of Pancam image covering Victoria Crater..…..…….……... 84 3.7 Statistical results of the extracted point pairs from the matched subs….. 91

xii 4.1 Optimal traverse parameters estimated under various scenarios (Li and Di, 2005) …..…….……..…..…….……..…..…….……..…..…….…… 101 4.2 Parameters of both pairs…..…….……..…..…….……..…..…….……... 110 4.3 Results comparison of different configuration of tie points at Pair 2 (Sites 11460 and 11456) ...…..…..…….……..…..…….……..…..……. 115 4.4 Results comparison of different configurations of tie points at Sites 11444 and 11442……..…..…….……..…..…….……..…..……..……... 117 4.5 FD strategies applied in the process of cross-site tie points selection….. 127 4.6 Test results at MER-A site..…….……..…..………..…….……..…..…. 130 4.7 Telemetry information of Sites 12334 and 12338….…….……..…..….. 133 5.1 Comparison for speed test…..…….……..…..….…..…….……..…..….. 144 5.2 Test results at Husband Hill summit area of Spirit rover landing site….. 147 5.3 Statistics of cross-site tie points selection results for all 19 sites at MER-A site..……..…..….…..…….……..…..….…..…….……..……... 148 5.4 Statistics of test results for rocky outcrop area at Silver Lake 152 5.5 Statistics of test results in bush area...……………..……….…………... 155

xiii

LIST OF FIGURES

Figure Page 1.1 Integrated image network for rover localization (Li et al., 2005)……...... 6 1.2 Configuration of images….………….....….………….....….………….... 7 1.3 Cross-site images (Xu, 2004) ...... ….………….....….………….....….…. 8 1.4 Flowchart of bundle adjustment based rover localization….…………..... 17 1.5 Concept of cross-site landmark selection….………….....….………….... 18 1.6 Diagram of automatic cross-site tie points selection.…….…………….... 19 2.1 Manual registration of landmarks from orbital and ground images …….. 25 2.2 Significant landmarks (left: rover images; right: orbital images) Image Credit: NASA/JPL/University of Arizona/OSU .………...……..... 27 2.3 Perspective view of 3D DTM of Endurance Crater….…………………... 30 2.4 A rock….…………………..….…………………..….…………………...31 2.5 Crater extraction (Victoria Crater) from HiRISE image ………………… 36 2.6 Extracted rock peaks back projected to Navcam images ..……………..... 38 2.7 Plane fitting for the background of a rock ………………………………. 39 2.8 Example of iterative surface points (red dots) extraction of a rock (green dots: rock peaks) ………………………………………………...... 40 2.9 Example of rocks extracted with peaks (green) and surface points (red)... 41 2.10 Rocks shown on two types of terrains with different slopes……………... 45 2.11 Hemispheroid rock models……………………………………………..... 48 2.12 Nine rocks used for modeling with four analytical models (green dots: rock peaks) ………………………………………………………………. 55 3.1 Examples of manually matched pairs from two sites.………………...... 61 3.2 Compare the fitting plane and normal direction (Sites 1200 and 1300)..... 74 3.3 Rock matching technique using both model matching and rock

xiv distribution pattern matching.…………………...... …………………...... 77 3.4 Distribution of automatically matched rocks as cross-site tie points at Sites 1200 and 1300 of the Spirit rover site.…………...……………...…. 79 3.5 Automatically matched rocks (tie points) shown on the image mosaics of Sites 1200 and 1300 (labeled with the same identification numbers)...... 80 3.6 Example of difficult match: same features look different from two sites (25 m apart) …………………...... …………………...………………...... 81 3.7 Example of the corrected match not included in the two candidates…..... 82 3.8 Strategy of orbital and ground integration…………………...... ……….. 83 3.9 Ground and orbital integration using image texture information………... 85 3.10 Other HiRISE images with crater features……………………………...... 86 3.11 The best matching result……………………………...………………...... 88 3.12 Matched results and CC value from reference HiRISE image groups…... 88 3.13 Extracted edge maps under different thresholds……………...………...... 90 3.14 Histogram of points extracted under different distance thresholds……..... 91 3.15 Histogram of difference comparison in radius and polar angles for two sets of points extracted under different distance thresholds…….....…….. 93 4.1 Configuration with different number of landmarks...……………………. 103 4.2 Relative localization error vs. convergence angle (distance: 7.5 m).……. 105 4.3 Relative localization error vs. convergence angle (distance: 15 m)….….. 106 4.4 Relative localization error vs. convergence angle (distance: 20 m)….….. 107 4.5 Relative localization error vs. convergence angle (distance: 25 m)….….. 108 4.6 Distribution of tie points……………………………...………………….. 109 4.7 Relative error analysis for each of the 105 combinations of Pair 2….…... 111 4.8 Small REs at Sites 11460 and 11456 with different distribution of two tie points (good geometry) ……………………………...…………….…….. 113 4.9 Large REs at Sites 11460 and 11456 with different distribution of two tie points (bad geometry) ……………………...…………….…..….…..…... 114 4.10 Relative error analysis for configuration of 4 tie points of Pair 2….…..... 116 4.11 Distribution of tie points at Sites 11442 and 11444….…..….…..….……. 117

xv 4.12 Automatically selected tie points shown on the image mosaics of Sites 325 and 326 (labeled with the same identification numbers) …..…..…… 118 4.13 Distribution of tie points at Sites 325 and 326…..…..…..…..…..…..…… 119 4.14 Automatically selected tie points shown on the image mosaics of Sites 309 and 310 (labeled with the same identification numbers) …..…..…… 120 4.15 Distribution of tie points at Sites 309 and 310…..…..…..…..…..…..…… 120 4.16 Scatter plot of azimuth angle difference between the selected rover image and rover traverse direction…..…..…..…..…..…..…....…..……… 122 4.17 General fault model…..…..…..…..…..…..…..…..…..…..…..…..…..…... 123 4.18 Detailed fault model…..…..…..…..…..…..…..…..…..…..…..…..……… 124 4.19 Theory of distance ratio.…..…..…..…..…..…..…..…..…..…..…..……... 125 4.20 A system-level FD tool.…..…..…..…..…..…..…..…..…..…..…..……… 126 4.21 Relative localization error and distribution of tie points of each test pair at MER-A site.…..…..…..…..…..…..…..…..…..…..…..……………….. 131 4.22 Exclusion of pairs with long traverse distance (>30 m)…..…..…..……... 134 4.23 Exclusion of mismatched points by local terrain comparison…..……….. 135 4.24 Examples improved by fault detection..…..…..…..…….…..…………… 136 4.25 Excluded examples with bad configuration and high RE..…..…..…..…... 138 5.1 Bundle adjusted rover traverse overlaid in MOC mosaic of the MER-A background map.…..…..…..…..…..…..…..…..…..…..…..……………... 145 5.2 Traverse overlaid in satellite image at Silver Lake (base map from Google).…..…..…..…..…..…..…..…..…..…..…..……..…..…..…..……. 146 5.3 MER-A rover traverse map at the Husband Hill summit area (MATLAB screen) .…..…..…..…..…..…..…..…..…..…..…..…….…..…………….. 149 5.4 MER-A rover traverse map at Home Plate area (MATLAB screen)…….. 150 5.5 Silver Lake rover traverse map at rocky outcrop area (MATLAB screen) ...…………….…..…..…..…..…..…..…..…..…..…..…..…….…..……... 153 5.6 Integration result in rocky outcrop area (MATLAB screen) ..…..………. 154 5.7 Test results in bush area on Jan. 15 morning (MATLAB screen)..…..….. 156 A.1 Simplified rock models..…..…....…..…....…..…....…..…....…..………... 161

xvi

LIST OF ABBREVIATIONS

ASU Arizona State University BA Bundle Adjustment DIMES Descent Image Motion Estimation System DTM Digital Terrain Map CC Correlation Coefficient CMU Carnegie Mellon University FD Fault Detection FOV Field of View GESTALT Grid-based Estimation of Surface Traversability Applied to Local Terrain GLONASS GLObal NAvigation Satellite System GMM Gauss-Markov adjustment Model GNSS Global Navigation Satellite Systems GPS Global Positioning System Hazcam Hazard Camera HiRISE High Resolution Imaging Science Experiment HRSC High/Super Resolution Stereo Colour Imager IRNSS Indian Regional Navigational Satellite System IMU Inertial Measurement Unit JPL Jet Propulsion Laboratory MBF Mars Body-Fixed MDIM Mars Digital Image Mosaics MER Mars Exploration Rover MGS Mars Global Surveyor MOC/NA Mars Orbiter Camera Narrow Angle

xvii MOLA Mars Orbiter Laser Altimeter MRO Mars Reconnaissance Orbiter MSSS Malin Space Science Systems NASA National Aeronautics and Space Administration Navcam Navigation Camera NAVSTAR Navigation Satellite Timing and Ranging OSS Operational Support Services Pancam Panoramic Camera ROTO Roll-Only Targeted Observation TES Thermal Emission Spectrometer THEMIS Thermal Emission Imaging System USGS U.S. Geological Survey VO Visual Odometry

xviii

CHAPTER 1

INTRODUCTION

1.1 Background information

The Mars Exploration Rover (MER) mission, begun in January 2004, has been

successful. So far, both rovers, Spirit and Opportunity, have exceeded the initial goals in

the distance traveled, lifetime extended, the amount of scientific data acquired, and others. For example, both rovers have gone from the primary mission of traveling about one kilometer in three months to the extended mission of traveling thousands of meters in four years. Many questions about the Martian environment have been answered. Future landed missions to Mars (e.g. the 2009 Mars Science Laboratory mission) as well as to the moon and outer planets are being planned.

However, high-level rover decision making for the current MER mission and past

1997 Mars Pathfinder mission is performed on Earth through a predominantly manual, time-consuming process (Estlin et al., 2005). The command sequence was manually generated on the ground and uplinked to the rovers. The communication latency of 10 to

20 minutes between Earth and Mars, due to the round-trip light time, requires a higher degree of autonomous navigation capabilities in the rovers (Maimone et al., 2004). The rover might have to wait for the next updated command when it encounters unexpected situations that cause the rover to deviate from its uploaded sequence. Other factors, such 1 as less gravity, variable surface properties, and the unfamiliarity of the Martian environment also present challenges to the unmanned rover navigation. Therefore, autonomous rovers are better choices in the Martian environment, because they can increase the possibility of achieving as many of the science and engineering objectives as possible, by reducing rover idle time, diminishing the need for entering safe-mode, and dynamically handling opportunistic science events without required communication to

Earth (Estlin et al., 2005). Additionally, future Mars rover missions will require independent space vehicle operation and full (or minimal human-interaction) autonomy because of the huge amount of data to be collected and processed, requirement for quick decisions, and limits of communication. In these cases, it is practically impossible to have humans heavily involved in these highly synchronized real-time loops (Gor et al., 2001).

One of the key techniques to the success of autonomous mobile robot navigation depends on the accurate determination of position and attitude information of a rover.

Different methodologies, including dead-reckoning (odometry and inertial navigation) and reference-based technologies such as Global Positioning Systems (GPS), landmark navigation, and map matching have been researched for the purpose of mobile robot localization and navigation (Spero, 2004). If the rover moves on Earth, the accuracy of

GPS-based rover navigation extends to the centimeter level because NAVSTAR GPS is widely used for navigation worldwide. It is a satellite constellation developed by the

United States. Other Global Navigation Satellite Systems (GNSS), like the Russian

GLONASS (Global Navigation Satellite System) are in the process of being restored to full operation. In turn, the European Union's Galileo positioning system, China's Beidou navigation system, and the Indian Regional Navigational Satellite System (IRNSS) are 2 also in the developmental phase and are scheduled to be operational in the next five years

(Wikipedia, 2008a). However, because of the expensive cost and the limitation of

payload and power, currently, there are no immediate plans by NASA to install GPS for

other planets (Trebi-ollennu et al., 2001). This makes rover navigation and localization

more challenging on Mars. For example, the Sojourner rover in the Mars Pathfinder

mission was limited to moving only a short distance during each downlink cycle by positional uncertainty. It did not explore more than 100 meters away from the lander.

During the MER mission, lander localization and rover navigation on Mars used a combination of techniques, including radio-based, dead-reckoning-based, and vision- based techniques. However, some difficulties still existed when researchers wanted to accurately estimate the lander position before landing or to precisely monitor the rover position in real time after landing. Before the landing of the twin rovers, the possible landing location was predicted in an ellipse. The size and orientation of the landing ellipse were estimated by using the latitude of the targeted landing center, which was based on the uncertainties of navigation in the approaching process, atmospheric modeling, and vehicle aerodynamics (Golombek and Grant, 2001; Wolf et al., 2004). For example, the landing ellipse at the Gusev Crater landing site was initially estimated to have the major and minor axes of 78 km and 10 km, respectively. The lander could land at any point on this landing ellipse. Therefore, it was important to narrow the area down and find a safe location for landing. It was crucial to know what the rover was expected to see when it landed and what could be used to localize it.

3 Various Mars orbiter data for both the Meridiani Planum and Gusev Crater landing sites were collected from different sources before landing. In these images, significant surface features such as craters and hills became one of the important factors used for lander localization after landing (Li et al., 2006a). Those significant landmarks within the landing ellipse were identified by using MOLA and MOC/NA data. A visibility map was generated for each selected landmark, which represented the degree of visibility of the landmark to the rover at various distances. More details of the theory and

generation process of the visibility map can be found in Li, et al. (2007b). However, the

number of features manually selected was not sufficient because of the limitation of the

available coverage of the MOC/NA stereos around the landing sites at that time and a gap

in the resolution between the MOLA data and MOC/NA images.

The landers have an inertial measurement unit (IMU) and a radar altimeter

installed for measurement of angular and vertical velocity (Maimone et al. 2004). During

the last two kilometers of the descent phase, both landers used a vision system, the

Descent Image Motion Estimation System (DIMES), in order to reduce horizontal

velocity for safe landing. The DIMES is used to estimate horizontal velocity by tracking

features on the ground with a down-looking camera.

For MER surface operations, there are three pairs of autonomous navigation

cameras installed in the rover: one pair of navigation cameras (Navcams) installed on the

mast, one pair of forward-looking hazard cameras (Hazcams) under the solar panel in

front, and another rear-looking pair of Hazcams under the solar panel in the back

(Maimone et al., 2004). Each camera has 1024x1024 pixel CCD (charge-coupled device)

4 arrays used to create 12-bit grayscale images. The 126-degree field of view (FOV) of

Hazcams was designed to avoid obstacles and to verify the safety of turn-in-place operations. However, the useful look-ahead distance of Hazcams is limited to a maximum of three to four meters because of their wide FOV and narrow baseline. Navcams can see further because of their narrower FOV and wider baseline, but Navcam stereos can only verify the traversability of one candidate path several meters ahead of the vehicle. Two

additional scientific instruments, a stereo pair of panoramic cameras (Pancams) and the

miniature thermal emission spectrometer (mini-TES), are installed in the camera mast.

Pancams can take multispectral visible and near-infrared imaging for mineral

classification. One single pixel of the mini-TES is composed of 167 bands between 5 and

29 µm. The parameters of the Pancam, Navcam and Hazcam cameras are shown in Table

1.1 (Bell et al., 2003; Maki et al., 2003). Among them, Pancams have the highest angular

and range resolution, with a 16-degree FOV and a 30 cm baseline (Maimone et al., 2004).

Camera Field of view Baseline Focal length Angular resolution Type (FOV) (degree) (cm) (mm) (mrad/pixel) Pancam 16.8 30 43 0.28 Navcam 45 20 14.67 0.76 Hazcam 126 10 5.58 2

Table 1.1 MER camera parameters

After landing, the rover moves to different locations called sites. As shown in

Figure 1.1, rover localization can be based on an image network of orbital and rover traverse images. At each site, the rover acquires a full or partial panorama of Navcam,

5 Pancam, and/or Hazcam images. When moving toward another site, it takes some traverse images or middle point survey images.

MOC / HRSC / HiRISE Descent Images

Landmark

Rover panoramic images Rover traverse images

Rover panoramic images

Figure 1.1: Integrated image network for rover localization (Li et al., 2005)

Usually, a Navcam panorama consists of 10 pairs of Navcam images, and a

Pancam panorama has 27 pairs of Pancam images, which takes more time to be completed (Xu, 2004). Figure 1.2 (Xu, 2004) illustrates a configuration of a Navcam panorama, which is composed of inter- and intra-stereos. Intra-stereo images are taken at the same looking angle from a single rover location. The left and right stereos usually have an overlapping area of more than 90% (Figure 1.2a). Inter-stereo images are also taken at a single rover location, though their looking angles are different. The overlapping area of the left and right stereos is less than 10% (Figure 1.2b).

Left Right Angle 1 Angle 2

90% Overlap 10% Overlap

(a) Intra-stereo (b) Inter-stereo

Figure 1.2: Configuration of images

The rover utilized the onboard IMU and odometer to record its movements and to restore its position. Additionally, stereo vision was used to build up a terrain slope map in real time to avoid possible hazards (Maimone et al., 2004). For example, the rover could get stuck in loose sand or fall into a deep crater. Therefore, a local, reactive planning algorithm, called Grid-based Estimation of Surface Traversability Applied to Local

Terrain (GESTALT) was developed to choose the best direction for the next move of a rover (Goldberg et al., 2002). Because IMU-based methods accumulate errors as time goes on, the rover position needs to be refined and updated regularly. In slippery areas, stereo vision-based visual odometry (VO) is used to estimate rover motion (Olson et al.,

2003; Maimone et al., 2007). Earth-based bundle adjustment (BA) was also performed regularly, using cross-site (Figure 1.3) rover images to update the rover position. Cross- site images are taken at different locations with different-looking angles. This BA method selects distinguished landmarks from cross-site images as tie points to link all the images into an image network. The current site and adjacent site are defined according to the order in which they are to be bundle adjusted. The previous bundle adjusted site is called

7 the current site. The new site whose position is corrected with respect to the current site is called the adjacent site.

Adjacent site Current site

Overlap

Figure 1.3: Cross-site images (Xu, 2004)

Long distance localization is done on Earth using BA from manually matched tie

points in panoramic images (Li et al., 2002; 2004; 2007e). So far, the combined traverse

for both the Spirit and Opportunity rovers has totaled more than 17 km (about 6.7 km by

Sol 1570 for Spirit and 11.1 km by Sol 1550 for Opportunity, with one Sol equaling one

Martian day) across the surface of Mars on paths of scientific exploration and discovery.

However, to meet the requirements for future Mars explorations, there is a need for the

automation of safe navigation and long-range localization. The following section will

review some of the key techniques which will contribute to the automation of rover

navigation and localization.

8 1.2 Literature review

Review of techniques for rover localization and navigation

Accurate rover localization is very important both for trajectory planning and

obstacle avoidance. Various methods can be used for rover localization with direct or

indirect measurements. Classification of the different methods mainly includes the

following two types: (1) methods using trajectory and dead reckoning, (2) methods using

reference-based systems (Spero, 2004).

The first technique integrates the trajectory and dead reckoning of a rover to

roughly estimate its position and pose based on the observation of internal parameters.

For example, in the MPF mission, the rover Sojourner achieved an overall localization

error of about 10% of the distance from the lander within an area of about 10m x 10m by using the dead-reckoning method (Matthies et al., 1995). The Robotics Institute at

Carnegie Mellon University (CMU) designed and developed various robotic systems and

vehicles for industrial and military applications. Field experiments performed in recent

years gave a localization accuracy of 3-5% of the distance traveled, based on dead-

reckoning technology that integrated wheel encoders, roll and pitch inclinometers and

yaw gyro (Wettergreen et al., 2005). During surface operations of the MER mission,

onboard positions of both rovers are estimated within each sol by dead reckoning with the wheel encoders and IMU. The heading is also updated occasionally by sun-finding techniques using the Pancams. A designed accuracy of 10% is achieved (Li et al., 2004).

Since this method has no reference linked to the external world, the dead-reckoning

approach is subject to accumulative errors due to the measurement of relative pose

9 (Spero, 2004). For example, in the MER mission, the dead-reckoning error is

accumulated mainly due to the slippage between the rover’s wheels and the ground.

The second method uses reference-based systems to locate a rover. Due to the

lack of reference systems such as GPS on Mars, rover localization mainly depends on

vision techniques and visible landmarks.

Multi-sensor integration and stereo vision have been widely applied to locate a

rover. The estimates of positions can be provided to the control algorithms for mission

planning. The most frequently used sensors include sonar, laser or infrared rangefinders,

monocular vision, and stereovision. For example, 3D stereo vision can be used to determine the terrain. It uses a pair of cameras to create a disparity image, which describes the difference between features in the left and right images and can be used to infer depth. The current MER mission has applied stereo vision for automatic rover

navigation. VO (Matthies, 1989; Olson et al., 2003; Maimone et al., 2007) has also been used onboard the rover for precision instrument placement and slippage corrections in the

MER mission. This algorithm estimates the rover motion by tracking visual features between consecutive stereo pairs. The VO position estimation error achieved is less than

2.5% of the distance traveled for runs of 10 to 30 m (Angelova et al., 2006). This algorithm has been proven to be an effective tool for securing drives on difficult terrain and precision approach to scientific targets within a relatively short distance. However, it is limited in slippage adjustment for short traverses with a step of 25 cm (Nesnas et al.,

2004).

10 Methods using landmarks do not require frequent sensor measurements for position estimation. However, they depend on the reliable extraction of significant landmarks. One popular but approximate approach to estimating the lander position is through triangulation of landmarks. For example, when acquiring the initial lander position by a radio-based technique, the accuracy achieved is only about hundreds of meters within the landing ellipse. Therefore, in order to improve the accuracy, the lander position was triangulated from the measurement of the range/attitude of landmarks like mountain peaks relative to the lander in both orbital and ground images (Golombek and

Parker, 2004; Parker et al., 2004; Serafy, 2004; Li et al., 2006a). Incremental BA (Li et al., 2002; 2004; 2007a; 2008) technology is used to refine the rover positions. Unlike VO used for position adjustment for short traverses, BA is applied for rover localization with relatively long traverses. It provides accurate rover positions by building a strong image network along the traverse to maintain consistent overall traverse information. The pointing parameters (camera center position and three rotation angles) of each image in the network are adjusted to their optimal values by the least-squares method, with landmarks serving as tie points. The methods depend on the availability of distinguished landmarks detected by the rover and shown on the rover images. Other landmark-based methods utilize prior knowledge of a global model and then match the sensor data with the global model for position estimation. Much research was carried out to identify features on the horizon and to match them to an elevation map of the terrain. For instance, Talluri and Aggarwal (1992) used the shape and position of the horizon line and the known camera geometry of the perspective projection to search for a robot position in a digital terrain map (DTM). Their outdoor tests in Colorado and Texas yielded good

11 results when the research was carried out. Another study was performed by Thompson et

al. (1993). They extracted and matched features on the horizon and other visible hills and

ridges. Matches between configurations of features were then searched for in a pre-

processed map. The hypothesized locations were then refined and evaluated. Stein and

Medioni (1995) approximated the horizon line by polygonal chains and stored them in a

table which included subsections of the horizon seen from different positions on a map.

The best match was selected using geometric constraints. Cozman and Krotkov (1997)

also developed a system which detected mountain peaks on the horizon to fix the mobile

robot position roughly in a large area covering approximately 37 km2. It performed an

automatic search using a table containing all the peaks visible from every possible

position in order to maximize the posterior probability of finding the correct position. The average error in the localization of this system was about 95 m. JPL also estimated positions and headings by remote viewing of a colored cylindrical target (Volpe et al.,

1995).

All the techniques reviewed above are able to estimate the position of the robot. It is a good solution to combine different techniques in order to improve the reliability and robustness of rover localization. For example, researchers at Centre National d'Etudes

Spatiales developed Mars rover autonomous navigation technology based on IMU, odometry, and stereo vision (Mauratte, 2003). In the MER mission, the combined onboard VO and the ground (Earth) BA methods are capable of correcting position errors caused by wheel slippage, azimuthal angle drift, and other navigational errors as large as

21%, such as experienced within Eagle Crater (Meridiani Planum landing site) and 10.5%

12 in the Husband Hill area (Gusev Crater landing site) (Maimone et al., 2004; Di et al.,

2005; Li et al., 2005; 2006a).

The integrated VO and BA techniques are important for the automation of rover localization. The BA method can compensate for the shortfalls that VO presents by only working for short traverses and needing frequent sensor measurement. This integration precisely links the traverse segments, which helps to identify the rover position with respect to potential obstacles or targets of interest. The integration also adjusts the planned path with consideration for rover slips and makes reference to the global Mars body-fixed frame (Li et al., 2008). Although VO is fully automated onboard the rover.

BA is limited to Earth operation because of the difficulties in selecting cross-site tie points automatically. The following paragraphs will review the techniques of landmark detection and modeling, and fault detection, which help with the automation of cross-site tie points selection.

Review of landmark detecting and modeling techniques

There are two kinds of landmarks used for research of autonomous rover navigation, natural landmarks and artificial landmarks. Because there are no artificial landmarks, such as specially designed markers or objects, introduced in the Martian environment, this research only focuses on natural landmarks such as the most common feature of the Martian landscape, rocks. Rocks can be detected and modeled for the purposes of adaptive target selection and autonomous geological analysis. Rock detection is difficult because of diverse morphologies, visibility, and occlusions. Thompson and

Castaño (2007) compared seven different algorithms developed at CMU and JPL for rock

13 detection. Some of these algorithms included stereo-based techniques which find rocks above the average ground plane (Gor et al., 2001; Fox et al., 2002; Thompson et al.,

2005a), edge-based methods that use a Sobel and a Canny edge detector to find edges and build closed contours (Castaño et al., 2004; 2006; 2007), methods based on template, and techniques using shadows. Gor et al. (2001) developed a rock detection method which used image intensity information to detect small rocks and range information to detect large rocks from Mars rover images. CMU researchers developed a rock detection method based on segmentation, detection, and classification using texture, color, shape, shading, and stereo data from the Zoë rover (Thompson et al., 2005b). The same group also developed a multiple-view detection method. The shape of the extracted large rock was modeled by metrics such as eccentricity, ellipse error, 2D sphericity, and 2D angularity (Castaño et al., 2002). JPL’s Onboard Autonomous Science Investigation

System developed the rock detection and shape modeling methods. It has been tested for automatic rock shape analysis by the Mars rover in the MER mission. However, no perfect algorithm can detect rocks flawlessly in all kinds of situations. It is not easy to describe the shape of a rock clearly and model it precisely. In our approach, we want to seek an expedient rock detector which can find high accuracy tie points for BA. Rocks higher than 10 centimeters are detected within 25 meters’ range to the camera center.

They are modeled using an analytical model, such as a semi-ellipsoid, so that they can be matched across multiple sites. The final correct matched rocks are selected as tie points between cross sites for BA process.

14 Review of fault detection

Fault detection has been widely applied in the field of industry and technical

process such as with aircraft, trains, automobiles, power plants, and chemical plants.

Several books have covered the topic of fault detection, diagnosis, and evaluation (Pau,

1975; Patton et al., 1989; Gertler, 1998; Chen and Patton, 1999). Many literatures

concerning this topic have also been published. In an automatic or semi- automatic rover navigation system, fault detection is very important for improving the reliability, safety,

and efficiency of the system. Even in a sub-procedure like cross-site tie point selection

for onboard automatic BA process, it is necessary to develop fault detection in order to

detect a fault immediately and isolate it as quickly as possible.

The process of fault detection can be model-based or model-free. Most traditional

methods place an emphasis on a model. A straightforward model-based method of fault

detection consists of developing a fixed model that runs in parallel to the process. The

output error may be used to validate the results and decisions at each key step. In the last

20 years, mathematical models have been developed for different approaches to fault

detection. For example, Chowdhury and Aravena (1998) developed a widely applied

model, which generated residuals to serve as fault indicators. The indicators were

analyzed by standard statistical hypothesis testing or by artificial neural networks to

create intelligent decision rules. Other models using particle filters (Vandi et al., 2004)

and Bayesian belief networks (Szolovits and Pauker, 1993) have also been investigated

and have provided a valuable aid for fault diagnosis. Lerner et al., (2000) used dynamic

Bayesian networks to address temporal dependencies. However, the need for fault-free

training data to tune the model limited the application of the model-based approach in our 15 method. In a model-free case, people create fast and sensitive fault indicators by signal

processing and wavelet theories without any models (Chowdhury and Aravena, 1998), or

researchers make a decision based on rules (Schein and Bushby, 2006). A rule-based approach has the advantage of transparency, flexibility, adaptability, and lower computational cost (Tzafestas, 1989). In our research, the rule-based approach and the model-based method are both combined in the fault detection of cross-site tie point selection. More detail will be given in Chapter 4.

1.3 Issues and significance of this research

Landmark-based localization is a popular approach. However, most of the research relies on the invariance of landmarks with respect to image translations and limited scale. In order to establish a correspondence between cross sites or between the surface sensor data and the global orbital data, this method requires substantial effort to resolve issues of data format, reference systems, and cross data-set comparison (Li et al.,

2004; 2006a). Additionally, most of the methods are more focused on the orbital data at testing phase. Although orbital mapping is very important, it cannot replace the role of ground mapping when considering the resolution, speed, and precision.

In the OSU Mapping and GIS Lab, Li et al. (2002; 2004) developed a BA method for long-range Mars rover localization using descent and rover images. The flowchart of

BA-based rover localization is shown in Figure 1.4. The process starts working on the ground data, builds the correspondence between adjacent rover sites using tie points, and bundle adjusts the rover position for rover localization. In the MER mission, Spirit Rover has achieved an accuracy of 0.5% over a 6km traverse using the integrated VO and

16 ground BA method (Li et al., 2007a). One of the important factors which contributed to the success of the BA is the selection of a sufficient number of well-distributed tie points to link the ground images into an image network. Although tie points selection from the intra- and inter-stereos at one rover site was automated based on automatic image matching using the MarsMapper software developed by the OSU Mapping and GIS Lab

(Xu, 2004), many of the tie points among adjacent sites, named cross-site, are selected manually during MER mission operations.

Rover images and original image orientation parameters

Interest point extraction and matching

Cross-site tie point Intra- and inter-stereo tie selection point selection

Bundle Adjustment

Rigid Transformation

Initial image orientation parameters

Bundle-adjusted rover locations and image orientation parameters

Figure 1.4: Flowchart of bundle adjustment based rover localization

17 Therefore, this research aims to automate the whole BA process by automating

the process of cross-site tie points selection. The concept of cross-site landmark selection

is briefly presented in Figure 1.5 (Agarwal, 2006). First, an algorithm needs to be

developed in order to identify and model the significant landmarks in the object space at

the current (blue) and adjacent (red) sites (Figure 1.5a). Then, the corresponding

landmarks are matched and used as tie points for BA to refine the rover position (Figure

1.5b).

(a) Landmarks identified in object space (b) Landmark matching at cross-site

Figure 1.5: Concept of cross-site landmark selection

On Mars, rocks are the most prominent features among adjacent sites. This dissertation will pick rocks as the main study object. As shown in Figure 1.6 (Li et al.,

2007a), the concept and technology of a new approach to selecting cross-site tie points based on rock extraction, modeling, and matching are presented in this dissertation. By this approach, cross-site control points will be selected automatically. In the meantime, fault detection was developed to exclude mismatched tie points and minimize the

18 localization errors. Additionally, a preliminary experiment was carried out to match orbital and ground landmarks.

Input and output data Process steps Dense 3D ground points Rock extraction

Rock peaks and surface points

Rock modeling Rock peaks and rock model parameters

Rock matching Matched rock peaks as cross- site tie points

Figure 1.6: Diagram of automatic cross-site tie points selection

In summary, the main contribution of this research on automation of rover localization includes:

• Automatic feature identification and landmark modeling

• Registration of landmarks with different shapes and orientation among cross-

site images with a difference in perspective view

• Automatic selection of the cross-site tie points based on landmark modeling

and matching along a long traverse (e.g. one kilometer)

19 • Automatic fault detection strategy applied in the process of cross-site tie

points selection to get high quality tie points and improve the reliability of

rover position for its localization.

1.4 Overview of the dissertation

This research proposes a new, efficient approach to automatic cross-site tie point selection for the automation of bundle adjustment. The new method is being applied in the real MER mission. It is more computationally efficient than the existing manual BA work. Landmark modeling and matching by a computer provides the basis for this new automation framework, which realizes the automatic selection of cross-site tie points. In

Chapter 2, important landmarks such as craters and rocks are first defined. Next, methods of automatic feature extraction and landmark modeling are introduced. Complex models with orientation angles and simple models without those angles are compared. Finally, conclusions are made on the selection of landmark models. In Chapter 3, the sensitivity of different modeling parameters is analyzed. Based on this analysis, cross-site rocks are matched through two complementary stages: rock distribution pattern matching and rock model matching. In addition, a preliminary experiment on orbital and ground landmark matching is also briefly introduced. In Chapter 4, localization error is analyzed. Details of camera topographic mapping capability and the configuration of tie points were analyzed in order to get high quality tie points from rover images for BA. In addition, fault detection is applied in the process to exclude the mismatches and improve the localization accuracy. Finally, the proposed method is demonstrated with the datasets from 2004 MER mission as well as from the simulated test data at Silver Lake,

20 California. The performance of the proposed framework is evaluated, and a discussion of future work is included in Chapter 5.

CHAPTER 2

LANDMARK EXTRACTION AND MODELING

The Martian surface consists entirely of natural terrain features without any man-

made buildings or living plants. Significant features that can be seen on the Martian surface include large landmarks such as mountains, dunes, and craters, as well as smaller landmarks such as rocks. In order to register the same feature from different sources, it is desirable to extract, model, and match features from multiple sources such as cross-site rover images and orbital-ground images. This chapter will first define some important landmarks and introduce the method of automatic feature recognition. Next, landmark extraction and modeling will be examined. Finally, results analysis of landmark modeling and conclusions will be given. Landmark matching will be introduced in Chapter 3.

2.1 Landmark definition

Understanding the surrounding environment and identifying features are important tasks in autonomous rover navigation in an outdoor environment (Srinivasan and Kanal, 1997). Corresponding landmarks found from the cross sites on the ground and from orbit helps to build an image network. This correspondence will further improve the accuracy of maps for rover localization and aid in refining spacecraft pose. Currently, numerous orbital and ground images are available for feature extraction. Various Mars 22 orbital images, Mars Digital Image Mosaics (MDIM), and Mars Orbiter Laser Altimeter

(MOLA) DTM (Table 2.1) for both the Meridiani Planum and Gusev Crater landing sites can be downloaded at the Planetary Data System (PDS) and from the websites of U.S.

Geological Survey (USGS), Arizona State University (ASU), and Malin Space Science

Systems (MSSS) before the landing of Spirit and Opportunity rovers. Rover images and relevant navigation data were automatically transmitted from NASA Headquarters’

Operational Support Services (OSS) to the OSU MER server daily through the Internet after landing. Furthermore, newly acquired Mars Express High/Super Resolution Stereo

Color Imager (HRSC) data and Mars Reconnaissance Orbiter (MRO) High Resolution

Imaging Science Experiment (HiRISE) data are also available through the PDS web. This new orbiter data can be used to generate topographic products with more detailed information, which will make the pre-landing detection of significant landmarks more efficient and useful for future landed missions.

23 Orbital Images Resolution Reference System/ Source (m/pixel) Projection Wide Angle ~240 http://www.msss.com/ma (WA) rs_images/moc/ Narrow Angle ~1.5-12 MGS (NA) Mars Body-Fixed (MBF) http://ida.wr.usgs.gov/ MOC cPROTO 1 coordinate system, ellipsoid using planetographic http://pds- latitude and west longitude geosciences.wustl.edu/mi ROTO 1 ssions/mgs/moc.html

http://marsoweb.nas.nasa. gov/landingsites/mer2003 /mocs/ http://themis- Odyssey THEMIS IR: ~100 MBF, sphere using data.asu.edu/ images VIS: ~18 planetocentric latitude and east longitude http://pds- geosciences.wustl.edu/mi ssions/odyssey/themis.ht ml HRSC ~2 MBF http://pds- geosciences.wustl.edu/mi ssions/mars_express/hrsc. htm HiRISE ~0.3 MBF http://hiroc.lpl.arizona.ed u/HiROC/ MBF, sphere using ~230 m/pixel, planetocentric latitude and http://astrogeology.usgs.g MDIM or 256 pixels/degree east longitude, or simple ov/Projects/MDIM21/ cylindrical (equirectangular) projection in meters Along track: ~300-400 m Cross track: ~1/64th MBF, sphere using http://analyst.gsfc.nasa.go Original points degree, which planetocentric latitude and v/ryan/mola01.html MOLA corresponds to about 1 km east longitude data at the equator. http://pds- Grid-based geosciences.wustl.edu/mi global DTM ~500 MBF, simple cylindrical ssions/mgs/megdr.html derived from projection in meters MOLA points

Table 2.1 Orbital data information

As shown in Figure 2.1, during the MER mission, in order to localize the lander, features of landmarks (mountain peaks) were manually identified and extracted from both the rover Pancam image mosaics and the MOC NA images. Next, the corresponding

24 features were compared and matched to build an image network for BA operation.

Usually, smaller features were found. Therefore, the process of manual identification and matching is time-consuming. Due to the immense amount of image data that needs to be searched and the level of detail to be examined, a new algorithm needs to be developed to automatically identify those important features. This algorithm is expected to find as many landmarks as possible and to achieve a high efficiency. Before automatic identification can take place, however, it is important to define significant features of landmarks. Next, the defined landmarks are extracted and modeled using mathematical models.

Orbital image MOC/NA

Spirit Rover Pancam image mosaics taken at Lander position (Sols 1-9)

Perspective view of DTM Columbia Hills from MOC/NA images

North

Figure 2.1: Manual registration of landmarks from orbital and ground images

25 Significant features on Mars include the prominent ones like mountains, dunes,

and craters as well as the small rocks. Significant landmarks selected should satisfy the

following conditions:

• Visible in both ground and orbital imagery

• Invariant to irrelevant transformations, position, and looking angle

• Insensitive to noise.

In this section, the definition of significant landmarks will be given in order to

classify the features extracted from different images. In a traditional computer graphics

system, objects are usually defined in terms of some modeling primitives including points, lines, polygons, and parametric patches (Fournier et al., 1982). A stochastic model

of an object even extends the concept of an object by possibly including a time

parameter. The following sections will look into features of different landmarks such as

mountains, craters, and rocks on Mars. Figure 2.2 presents examples of these distinct

features shown in both the orbital and ground images. Definitions of these landmarks are

given in order to model them by computer language.

26 MOC/NA Spirit Rover pancam image mosaics taken at lander position (sols 1-9) Husband Hill Husband Hill

North (a) Mountains: Columbia Hills at Spirit Rover landing site

HiRISE (TRA_000873_1780) Duck Bay Duck Bay

North (b) Crater: Victoria Crater at Opportunity Rover landing site

Navcam HiRISE 2 image 1 Ortho photo taken at created from Site 11422 2 Navcam 1 images 2 1

Figure 2.2: Significant landmarks (left: rover images; right: orbital images) Image Credit: NASA/JPL/University of Arizona/OSU

Mountains

Mountains are very distinct. According to Webster’s online dictionary, a mountain is a landform which projects conspicuously above the surrounding terrain in a

limited area. It has regions with spatial complexity, areas of high relief, and distinct

changes in terrain slope (Ghosh et al., 2000). However, the view of a mountain observed

from 300 m or 200 m away when a person is approaching it is quite different from the

27 view while one is on the mountain itself, which, in turn, has no similarity with what the

mountain looks like from its side. Moreover, mountains are of various structural types.

For example, different mountains are composed of a different number of hills and

valleys. Some are steep with big slopes, while others are relatively flat. Considering these

features, mountains can usually be decomposed into parts such as peaks, ridges, horizons,

valleys, and watersheds. All these parts of a mountain can be described in a feature

template and represented in the form of points, poly-lines, and polygons.

Craters

There are several different types of craters, including impact, volcanic,

subsidence, and maar craters. There are more than 43,000 impact craters on the Martian

surface with diameters over five kilometers (Barlow, 2000). For example, Gusev Crater,

where Spirit Rover landed, has a diameter of 160 km (Grant, et al., 2004). Research on

the small, simple impact craters has not been carried out in detail because of the lack of high-resolution orbital images (Golombek et al., 2006). However, all of these impact

craters are important for deciphering the age and geological history of Mars. Their

morphology can provide the clues to aid in understanding the nature of the Martian

surface (Barlow and Sharpton, 2004) or even inform us of the possible existence of

significant surface and underground water (Carr, 2004). Due to the limited rover

capability of the current MER mission, the rover has only collected images of small

impact craters such as Eagle, Endurance, and Victoria Craters in the Meridiani Planum

and Bonneville, Missoula, and Lahontan Craters in the Gusev Crater area (Table 2.2).

28 Therefore, this research only focuses on the small craters with diameters less than one

kilometer.

Location Crater Approximate Approximate Ground image Diameter (m) Depth (m) acquisition time (Sols) Gusev Bonneville Crater[1] 210 10-14 68-86 Crater Missoula Crater[1] 163 3-4 105 landing site Lahontan Crater[1] 90 4.5 118 Meridiani Eagle Crater[1] 22 2-3 1 Planum Fram Crater [2] 8 N/A 84 landing site Endurance Crater[1] 150 21 95-315 Argo Crater [2] small N/A 365 Vostok Crater [2] 20 ~ 0 399 Erebus Crater [2] 350 ~ 0 550-750 Beagle Crater [3] 35 ~1-2 855 Emma Dean crater [2] small N/A 929-943 Victoria Crater [2] 750 ~70 951 [1] From Grant et al., 2005 [2] From http://www.wikipedia.com [3] From http://www.planetary.org/news/2006/0731_Mars_Exploration_Rovers_Update_Spirit.html

Table 2.2 Craters in the Meridiani Planum and Gusev Crater landing sites

The classification of craters by the degree of development includes fresh crater, degraded crater, and ghost crater (Kanefsky et al., 2001). As stated on the website of the

Mars clickworkers project (see http://clickworkers.arc.nasa.gov/basic-crater-

classification?), a fresh crater always has “a sharp rim, distinctive ejecta blanket, and

well-preserved interior features (if any).” Some even have central peaks inside the crater.

On the contrary, a degraded crater has no surrounding ejecta blanket. Its interior features

are “largely or totally obliterated”. The rim is rounded or removed. A ghost crater is

almost invisible through overlying deposits. For example, Erebus Crater in the Meridiani 29 Planum is a degraded crater. Erebus Crater is very old and eroded, and is almost invisible from the ground (Wikipedia, 2008b). In this dissertation, the definition of a crater is simplified to a bow-shaped structure with a rim, wall, and floor - three basic parts. The crater rim is usually higher than its surrounding terrain. The crater wall has a nearly constant slope, and the floor is relatively flat. Figure 2.3 (Di et al., 2006) is the perspective view of 3D DTM of Endurance Crater. Different colors represent various elevations at different parts of this crater model.

Crater wall

Crater rim

Crater floor

Figure 2.3: Perspective view of 3D DTM of Endurance Crater

Rocks

Rocks composing the Columbia Hills on the Martian surface are mainly “crustal

sections that formed by volcaniclastic processes and/or impact ejecta emplacement”

(Arvidson et al., 2006). Geologists usually divide rocks into groups according to their

30 various mineral compositions and developmental history. In this research, the study of rocks is used for tie point selection. Therefore, the shape of a rock is much more important for the matching of landmarks than the composition of the rock. In our definition, a rock is composed of a group of continuous surface points (red dots) and a peak (green dot) shown in Figure 2.4. The assumptions about a rock peak are that (1) the peak is the highest point of a rock, (2) it is visible from different views. Additionally, the maximum height difference between all those points should be more than 10 cm, that is, the rock identified should have a height greater than 10 cm.

Rock peak Rock surface points

Figure 2.4: A rock

2.2 Landmark extraction

Given the definitions of important landmarks above, the next step is to extract important features from different sources of images. Orbital and ground images have been processed separately in the MER mission and in previous missions (e.g. Mars

Pathfinder mission). MOC and HiRISE images are taken by looking straight down from the orbit, while 360-degree high-resolution Pancam panoramic images are taken on the ground by looking around the rover with a tilt angle. Therefore, the camera models of the orbital and ground images are quite different. In addition, features shown in both kinds of 31 images also have big differences in appearance. For example, craters are usually circular

looking in the MOC images, while their shape might look elliptical in the ground images.

Considering the different camera geometry, methods of feature extraction are different

for orbital and ground data.

Because of the immense amount of image data that needs to be analyzed for

presence of features, an automatic feature extraction algorithm is very important. Craters

and mountains are usually selected to strengthen the correspondence between the orbital and ground images. Rocks are picked to link the correspondence of the cross sites for the ground images, because rocks are relatively small when compared with big mountains or craters. For example, a big rock two meters in width is only shown with six to seven pixels in a HiRISE image. Ground image processing with OSU MarsMapper and BA software has been well-tested in the daily MER operation at the OSU mapping and GIS

Lab. Li et al. (2007c) have also developed a semi-automatic hierarchical stereo matching technique based on an image pyramid and a BA method for orbital images. DTM and orthophoto of Victoria Crater have been generated from the HiRISE stereo pair. These

products have differences less than two meters in any direction when compared with

those derived from the networked ground images. Victoria DTM is used for landmark

modeling in this dissertation. The following subsections will give examples of automated

crater detection (ACD) and rock extraction.

2.2.1 Crater detection

In order to detect craters of different sizes automatically, various researchers have

focused on image-based methods, which can be divided into the categories of

32 unsupervised and supervised (Bue and Stepinski, 2007). The unsupervised method is

fully autonomous, which uses pattern recognition techniques such as the Hough

transform or genetic algorithm (Honda and Azuma, 2000) to detect crater rims in an image, and approximates them with circular or elliptical shapes. For example, 2D edges were detected in images through an edge-detection algorithm and approximated with ellipse shapes (Leroy et al., 2001; Cheng et al., 2003). In a study by Barata et al. (2004), images were segmented through a Principle Component Analysis of statistical texture measures, and craters were detected using template matching. The supervised method trains crater detection algorithms using machine learning concepts. A set of continuously scaled templates provided detection of a range of crater sizes in synthetic terrain

(Vinogradova et al., 2002). Plesko et al. (2005) developed GENetic Imagery Exploitation

(GENIE) software, which applied genetic programming and support vector machines to detect craters automatically. Urbach and Stepinski (2008) used shape filters to locate candidate craters in images and classified them into craters and non-craters by supervised machine learning. These algorithms work well only for small craters or relatively simple terrain.

Due to the limitations of the unsupervised and supervised algorithms, researchers

(Kim and Muller, 2003; Magee et al., 2003) combined both methods to detect craters using edge processing and template matching based on local intensity values, intensity gradients, and geometric traits of images. However, image-based crater detection is not robust enough due to the lower quality of the images and the variety of crater structures.

In addition to the above described image-based crater detection approaches,

Michael (2003) used a constrained DTM to detect craters through Hough transform in

33 order to register the USGS MDIM 1.0 with the more precise MOLA data. Later, Bue and

Stepinski (2007) developed a novel approach to detect craters based on Martian DTM data instead of images. A curvature module and a segmentation module were conducted in parallel for this approach. A topographic profile curvature was calculated in the curvature module to reflect the change in slope angle in order to detect the crater rim. The segmentation module used the flood algorithm to divide the data into small fragments without cutting through the craters. Their algorithm was tested at a heavily cratered

Noachian site. Results proved that the DTM-based detection was on par with the manually compiled Barlow catalog (Barlow, 1988). This method even “outperformed

Barlow catalog for small craters, but failed to identify some large, degraded craters”.

However, the DTM-based method is restricted “due to the scarcity and limited resolution of planetary topography data” (Bue and Stepinski, 2007).

Both the image-based and DTM-based crater detection approaches have advantages and drawbacks. They still have room for improvement.

This research extracted craters using information from both the image and DTM.

Following is the example of Victoria Crater extracted from a HiRISE image using edge gradient operators. Since “crater rims are often associated with the steepest gradients in the Martian landscape” (Bue and Stepinski, 2007), the two sides of crater rims usually yield a large difference in the image intensity. Therefore, the continuous edge points of a crater are the potential crater rim. First, the edge feature of this crater was detected by a

Canny edge detector. As a result of this, in total, 7,375 edge points were extracted, as shown in Figure 2.5(b). Next, the shape of all the extracted edge pixels was simulated using an elliptic model by least square fitting. With this simulated elliptic model, the

34 coordinates of its center and its semi-major and semi-minor axes were calculated (Figure

2.5c). The ellipse fit quality was determined by the Ellipse-Fit-Similarity (EFS), which is defined as follows:

1 1 N EFS = × ∑ Disk (2.1) N R Ave_ellipse k=1

Where RAve_ellipse is the averaged length of the semi-major and semi-minor axes of the simulated ellipse fitting to the edge points;

Disk is the Euclidean distance from the point (xk, yk) to the center of the fitted ellipse.

EFS shows how much the fitted ellipse deviates from the original edge points. As we know, the closer the EFS is approaching one the better an ellipse fits into these edge points. The result of the EFS of Victoria Crater on the HiRISE image equaled 0.9987.

This value shows a good fit of the simulated ellipse. From the fitted ellipse, the semi- major and semi-minor axes were computed as 406.3 m and 382.5 m, respectively. The eccentricity of the ellipse is 0.34, which means the shape of the simulated ellipse is close to a circle. The radius was also estimated to be about 400 m by Squyres et al., (2008), and about 375 m by Li et al. (2007c). The difference is mainly caused by different processing and measuring methods.

North

(a) Original image

(b) Extracted crater edge points (c) Simulated edge ellipse

Figure 2.5: Crater extraction (Victoria Crater) from HiRISE image

In addition, the Victoria DTM from USGS was read in Matlab. Then, the height was measured automatically in two different ways. The first method used the maximum height in the crater rim minus the minimum height on the crater floor. The calculated height was about 78.5 m. The second method measured the difference between the averaged height in the crater rim and that on the crater floor. The height was about 55 m.

For the discussion in this dissertation, the result from the first method was used.

36 2.2.2 Rock extraction

Rocks are the primary landmarks which can be easily identified in most of the rover ground images. Usually, rocks are composed of a distinct rock peak and surface points. A peak is assumed to be the highest point always visible on a rock, no matter what the looking angle is. Therefore, rock peaks are extracted from the 3D ground points using the following assumptions: (1) they are the local maxima within a window of, for example, 50×50 cm; (2) the maximum elevation difference within the window is greater than a threshold (e.g. 10 cm); and (3) there are at least three ground points extracted within the window. These thresholds were set based on the extensive testing of Mars data in the MER 2003 mission. Based on these criteria, extraction of rock peaks can be decomposed in the following three steps: (1) interest points (e.g. rock peaks, sharp corners) extraction from stereo images using a Förstner interest operator and interest points matching using cross-correlation (Xu, 2004; Di et al., 2005); (2) dense image matching performed under the constraint of a TIN (triangulated irregular model) and dense 3D ground points calculation through space intersection of homologous image points; and (3) peak selection from the 3D ground points based on the assumptions. More details of peak extraction can be found in a study done by Li et al. (2007a). In most of the cases, the peak extraction algorithm is very successful. For example, it identified 71 peaks at Site 1200 and 95 peaks at Site 1300 of the MER-A Spirit Rover site. Figure 2.6 illustrates examples of extracted rock peaks back projected to Navcam images at

Site1300.

Figure 2.6: Extracted rock peaks back projected to Navcam images

In addition to the rock peak, it is also important to describe the shape of a rock by its surface points and to fit these points into an analytical rock model. Through the use of an automatic searching procedure, ground points that were on the same rock were searched for in the vicinity of the peak. The procedure includes the following three steps:

1) A 3D plane is estimated using the terrain points within a certain range, e.g. 0.7 m, of the peak. In principle, the candidate rock surface points (red) are those above the plane. On the other hand, the ground points (blue) are on or very close to the plane

(Figure 2.7a). The normal direction of the fitting plane is calculated and represented using a green line as shown in Figure 2.7b. A height H is calculated as the perpendicular distance from the peak to the fitted plane.

(a) Separation of the candidate rock surface points with the ground points after plane fitting

61.8

61.7

61.6

Up (m) Up 61.5

-1037 61.4

-1038 61.3 2661.5 2662 -1039 2662.5 2663 2663.5 North (m) East (m) (b): Plot of 3-D surface points (red) and background points (blue) with normal direction (green)

Figure 2.7: Plane fitting for the background of a rock

2) Surface points are searched from the points above the fitted plane using a dynamic search range. The search range is proportional to the rock height H: kH. k varies from 0.3 to 1.7 based on a ground truth experiment where manual measurements of rocks at the Spirit site were measured and the coefficient k was calculated (Li et al., 2007a).

Initially, the search from the rock peak is made in a range with the low limit radius of

0.3H. Each time a point is found, the distance from this point to the closest neighboring point is calculated. The maximum of these distances (denoted as MaxD_Neighbor) is determined.

3) An extended search is performed from each point found in the last step and within the range of MaxD_Neighbor. Therefore, the overall search radius from the peak is increased by MaxD_Neighbor. This step repeats for all points found in the last step 39 until the overall search radius reaches the upper limit of 1.7H or until no new points in the neighborhood can be added.

Figure 2.8 (Li et al., 2007a) shows the rock peak and rock surface points extracted in the iterative process. The green dot is the rock peak, while the red dots are the extracted surface points.

Figure 2.8: Example of iterative surface points (red dots) extraction of a rock (green dots: rock peaks)

The above method is successfully applied to extract peaks and surface points of various types of rocks that are within 15 m from the rover, including large rocks (e.g. 0.5 m high). It is adaptive to rocks of various sizes. Examples of extracted rock peaks and surface points of other rocks are shown in Figure 2.9. The algorithm meets difficulties when dealing with a rock complex where a number of rocks stand closely together and severe occlusions block some rock peaks and surface points. However, as long as a sufficient number of rocks can be extracted and matched between two sites, the incremental BA can be achieved.

40 1 2 3 4

5 6 7 8

9 10 11 12

Figure 2.9: Example of rocks extracted with peaks (green) and surface points (red)

2.2.3 Limitations in landmark extraction

Most of the time, features such as rocks and craters are extracted successfully from the images. However, some of the limitations in the process of feature extraction are listed below:

• The quality of the image will be decreased by the image noise, which will further

affect the results of feature extraction.

• It is also difficult to extract those small and medium-sized rocks which are more

than 15 meters away from the camera center. Because they only occupy several

pixels in the image space, the algorithm can not extract a sufficient number of

surface points for the rock to be modeled.

• Occlusion is another problem. Some features are visible to the camera at one site

but not visible in the adjacent site, a factor which is due to the scene and camera

geometries. 41 2.3 Landmark modeling

With only 2D features extracted from the image space, it is not easy to match the landmarks successfully from different sources of data. In addition, the 3D information from landmark modeling needs to be combined when comparing the landmarks from cross sites or between the orbital and ground. Modeling a specific landmark is relative easy, whereas when it comes to a class of generic landmarks, there are significant difficulties in defining an ideal model. The key problems are (1) lack of specific definition of the landmark, (2) inconsistencies in resolution and viewing angle, (3) different background information, contrast, and shadow effects, and (4) occlusions. For example, before high-resolution (0.3m) MRO HiRISE was available, there was resolution discrepancy between the one to five meters MOC NA and the ground images with resolution at the centimeter level. Additionally, the shapes of the landmarks in the images may be quite different depending on the looking angle, the location of the sun, and weather. Further, even if the landmarks are similar, sometimes, it is sometimes possible that other features are occluding part of the landmark. Even when the landmarks are clearly visible, their complexity makes representation with simple descriptors difficult.

Therefore, it is not an easy task to give a general model which will fit the same type of landmarks from multiple sources. Description of a landmark with single edge grouping or texture feature is not enough.

In order to overcome these difficulties associated with landmark recognition, the landmark will be modeled by parts. The following gives examples of crater modeling and rock modeling.

42 2.3.1 Crater modeling

First of all, a crater has one of two basic shapes; it is either bowl-shaped or flat- floored (Jones et al., 1997). The points extracted from the edge of a crater can be used to simulate an elliptic equation, which is a shape factor. Then, the crater can be described in more detail with some dimension measure. The size of a crater can be measured with the parameters of the semi-major and semi-minor axes of the crater rim, height, and floor radius. The height information can be acquired from either MOLA DTM or stereo vision.

Finally, several ratios are given as follows (Equations 2.2-2.4) to form some constraints of a crater model.

Ratio1 = Length of the semi-minor axis /Length of the semi-major axis (2.2)

Ratio2 = Floor radius /Averaged radius of the crater rim (2.3)

Height Ratio3 = (2.4) Averaged radius of the crater rim

In the previous study, the height of Victoria Crater derived from its DTM is about

78.5 m. The semi-major and semi-minor axes of the crater rim are 406.3 m and 382.5 m.

Therefore, the averaged radius of the rim calculated from the simulated ellipse is around

394.4 m. The floor radius is about 139 m. With these numbers, three ratios are calculated as shown in Table 2.3. These ratios, together with the crater center position, height, and radius, combine to form a feature template to describe Victoria Crater. When matching

Victoria Crater from different sources, all the parameters in the feature template need to be compared. Currently, since the rover images are not sufficient to generate a 3D model of Victoria Crater, this feature template of the orbital data cannot be compared with the ground data.

43 Ratios Value Ratio1 0.94 Ratio2 0.35 Ratio3 0.2

Table 2.3 Value of ratios from the crater model

2.3.2 Rock modeling

Rock surface points extracted from images taken at certain view points are usually distributed in the front face pointing to an observer. In order to use a symmetric model, such as a cone to model the rock, the ideal assumption is made that the peak and surface points can be used to infer the part of the rock surface which is not visible to the camera.

This analytical model provides more information when compared to the above incomplete depiction with the rock peak and surface points. Therefore, a rock model with its parameters is more efficient for comparison with other rocks modeled in the same way.

Based on the above-mentioned extracted rock peak and surface points, a rock can be modeled by using a 3D analytical surface (Figure A.1 in Appendix A), such as hemispheroid, semi-ellipsoid, cone, or tetrahedron (Equations A.1-A.3). More detail of the models can be found in Appendix A. The parameters of each individual rock model are estimated by a least-squares fitting using the surface points on the rock. More details on model equations, linearization, least-square solution, and fitting accuracy are given in the study by Li et al. (2007a).

In the above implementation of rock modeling, the geometric shapes selected for representing the rocks are simple and idealized. An assumption is made that all of the 44 rocks stand upright on the Martian surface, i.e., the major axis is perpendicular to the XY plane. As illustrated in Figure 2.10b, the axes of some of the rocks may not be perpendicular to the XY plane. This may happen particularly in areas with big slopes. An complex rock model, considering the rock orientation angles, may be more suitable for modeling the rock, in order to reduce mismatches and produce more accurate tie points.

For example, for the big rock in the middle of Figure 2.10a, using the simple models (a special case of the improved models with three orientation angles being zeros) is sufficient. For the rock shown in Figure 2.10b, in order to get more accurate rock modeling parameters, using the complex models might be better.

(a) Rocks standing in a flat horizontal plane (b) Rocks located in steep terrain

Figure 2.10: Rocks shown on two types of terrains with different slopes

The rock models are enhanced by adding orientation angles into all the modeling equations. A single model can be transformed to a complicated model by performing three successive rotations, which can change the orientation of the reference frame from the x-y-z system of a single model to the x'-y'-z' system of a complicated model.

Equations 2.5 to 2.7 describe the complex model of rocks using a hemispheroid, semi- ellipsoid, and cone. No analytical equation exists for the tetrahedron model. Therefore,

45 three parameters are used to represent a tetrahedron: height h, radius of the enclosing circle of the bottom triangle, and orientation angle φ of the bottom triangle.

x'2 y'2 z'2 + + =1 (2.5) r 2 r 2 h2

x'2 y'2 z'2 + + = 1 (2.6) a 2 b2 h 2

r 2 x'2 + y'2 = ()z'−h 2 (2.7) h2

As shown in Equation 2.8, [x'y' z']T is acquired by multiplying []x y z T with the transformation matrix T, which is composed of three rotation matrixes (Equation

2.9, Schenk, 1999).

T T [][]x'y' z' = T ⋅ x y z = Rz (γ )⋅ Ry (β )⋅ Rx (α )⋅ [x y z] (2.8)

⎡1 0 0 ⎤ R α = ⎢0 cosα sinα ⎥ x () ⎢ ⎥ ⎣⎢0 − sinα cosα⎦⎥

⎡cos β 0 − sin β ⎤ R β = ⎢ 0 1 0 ⎥ y () ⎢ ⎥ (2.9) ⎣⎢sin β 0 cos β ⎦⎥

⎡ cosγ sinγ 0⎤ R γ = ⎢− sinγ cosγ 0⎥ z () ⎢ ⎥ ⎣⎢ 0 0 1⎦⎥

Since the semi-ellipsoid, hemispheroid, and cone models are symmetric around Z axis, only the angles α and β around the X and Y axes are actually necessary to represent the orientation of a rock. Therefore, Equation 2.10 is used to transform []x y z T to

46 []x'y' z' T for these three symmetric models. It is simpler than the transformation of a tetrahedron model, which needs three rotation angles (α, β, and γ) as illustrated by

Equation 2.11. Actually, the original simple models are special cases of the new models with the three orientation angles being zeros.

⎡x'⎤ ⎡cos β sin α ⋅sin β − cosα ⋅sin β ⎤⎡x⎤ ⎢ y'⎥ = ⎢ 0 cosα sin α ⎥⎢ y⎥ ⎢ ⎥ ⎢ ⎥⎢ ⎥ (2.10) ⎣⎢ z'⎦⎥ ⎣⎢sin β − sin α ⋅cos β cosα ⋅cos β ⎦⎥⎣⎢ z⎦⎥

⎡x'⎤ ⎡ cos β ⋅cos γ sin α ⋅sin β ⋅cosγ + cosα ⋅sin γ ⎢ y'⎥ = ⎢− cos β ⋅sin γ − sin α ⋅sin β ⋅sin γ + cosα ⋅cosγ ⎢ ⎥ ⎢ ⎣⎢ z'⎦⎥ ⎣⎢ sin β − sin α ⋅cos β (2.11) − cosα ⋅ sin β ⋅ cosγ + sinα ⋅ sin γ ⎤ ⎡x⎤ cosα ⋅ sin β ⋅ sin γ + sinα ⋅ cosγ ⎥ ⎢ y⎥ ⎥ ⎢ ⎥ cosα ⋅ cos β ⎦⎥ ⎣⎢ z⎦⎥

Given a hemispheroid rock model as an example (Figure 2.11), according to the equation of a hemispheroid model, the new model is similarly presented by Equation 2.5, where the parameter r is radius of the hemispheroid, and h is height. The transformation from the x-y-z coordinate system to the x'-y'-z' coordinate system is reached by rotating α and β angles along x and y axes using Equation 2.12, where the rotation matrices Rx and

Ry are calculated from Equation 2.9. Putting x', y', and z' calculated from Equations 2.12 or 2.10 into Equation 2.5 and linearizing the equation, the unknown rock model parameters (h, r, α, β) are estimated using the method of Least Squares Adjustment.

47 z z' y' y

h h x' x

r r x 2 y 2 z 2 x'2 y'2 z'2 + + = 1 + + =1 r 2 r 2 h2 r 2 r 2 h2 (a) Simple rock models (b) Rotated rock models

Figure 2.11: Hemispheroid rock models

T T []x'y' z' = Ry ()β ⋅ Rx (α )⋅ [x y z] (2.12)

2.4 Results analysis of rock modeling

The initial results from crater modeling can be found in Section 2.3.1. This dissertation will mainly focus on rock modeling due to the data available. During the rock modeling process, rock surface points are first extracted in the vicinity of the peak based on a fitted plane. Next, four models (semi-ellipsoid, hemispheroid, cone, and tetrahedron) with orientation angles and without those angles are used to model specific rocks. The parameters are calculated using a Least Squares fitting. Both the simulated data and real test data are applied in both types of models. Following is a comparison of results.

2.4.1 The complex rock models with respect to the simple models using simulated data

The simple models and the complex models mentioned in the last section are tested and compared using the simulated data of a hemispheroid. In all, 121 evenly distributed points are simulated in a hemispheroid with a height equal to 1 m and the 48 radius equal to 60 cm. Also, the hemispheroid is rotated along the X and Y axes from 1 to

20 degrees (20 degrees was chosen because with a slope more than 20 degrees, the terrain is too difficult to traverse). Table 2.4 shows the parameters of both the simple and the complex models. The IDs in the table refer to different rotation angles. The H column represents height, and the R column represents radius. Also, a root mean square (RMS) error is used to evaluate the fitting accuracy of a model. RMS is calculated from the differences of height z of the surface points and the fitted model height, which is shown in Equation 2.13.

n ()z − z 2 ∑ i i,model (2.13) RMS = i=1 n

Where zi is the height value of a surface point, zi, model is the height value calculated from the fitted model using the model parameters and the known horizontal position (xi, yi) of the point.

49 ID Simple model Complex model H (cm) R (cm) RMS(cm) H (cm) R (cm) α(deg) β(deg) RMS(cm) 1 100.9 59.7 3.2 100 60 1 1 0.0 2 101.7 58.8 4.3 100 60 2 2 0.0 3 101.7 58.5 4.5 100 60 3 3 0.0 4 101.3 59.3 4.6 100 60 4 4 0.0 5 100.8 60.1 5.3 100 60 5 5 0.0 6 99.5 60.3 5.5 100 60 6 6 0.0 7 98.1 61.0 5.9 100 60 7 7 0.0 8 95.8 62.3 6.0 100 60 8 8 0.0 9 96.7 62.6 5.6 100 60 9 9 0.0 10 96.7 63.0 5.3 100 60 10 10 0.0 11 97.0 64.0 5.4 100 60 11 11 0.0 12 97.0 65.3 5.5 100 60 12 12 0.0 13 95.8 66.2 5.6 100 60 13 13 0.0 14 97.8 65.1 5.1 100 60 14 14 0.0 15 98.7 65.6 5.0 100 60 15 15 0.0 16 94.8 68.7 5.7 100 60 16 16 0.0 17 97.6 66.4 5.0 100 60 17 17 0.0 18 98.6 65.2 5.6 100 60 18 18 0.0 19 98.5 66.0 5.7 100 60 19 19 0.0 20 98.5 66.9 5.9 100 60 20 20 0.0

Table 2.4 Model comparison based on the simulated dataset 1

In real cases, the extracted rock surface points are distributed only on one face of a rock. Therefore, the simple models and the complex models have also been tested using the simulated data distributed over only half of the model surface. In all, 61 evenly distributed points are simulated in half of a hemispheroid with the height equal to 1 m and the radius equal to 60 cm. Similarly to the first test, the hemispheroid is rotated along the X and Y axes from 1 to 20 degrees. The following table represents the parameters of both the simple and the complex models.

50 ID Simple model Complex model H (cm) R (cm) RMS(cm) H (cm) R (cm) α(deg) β(deg) RMS(cm) 1 100.7 60.9 3.0 100 60 1 1 0.0 2 101.7 61.1 3.0 100 60 2 2 0.0 3 101.5 61.7 3.5 100 60 3 3 0.0 4 102.5 61.4 4.0 100 60 4 4 0.0 5 102.8 62.0 4.3 100 60 5 5 0.0 6 102.7 63.5 4.7 100 60 6 6 0.0 7 102.4 65.3 4.8 100 60 7 7 0.0 8 106.3 63.4 4.2 100 60 8 8 0.0 9 104.1 63.6 5.1 100 60 9 9 0.0 10 103.6 64.3 5.3 100 60 10 10 0.0 11 102.9 65.0 4.8 100 60 11 11 0.0 12 104.4 65.5 4.6 100 60 12 12 0.0 13 103.6 66.1 4.5 100 60 13 13 0.0 14 103.8 66.8 4.8 100 60 14 14 0.0 15 102.6 67.6 4.4 100 60 15 15 0.0 16 102.7 68.3 4.6 100 60 16 16 0.0 17 102.9 69.1 4.8 100 60 17 17 0.0 18 102.5 70.8 5.0 100 60 18 18 0.0 19 102.7 71.7 5.2 100 60 19 19 0.0 20 103.5 71.5 5.3 100 60 20 20 0.0

Table 2.5 Model comparison based on the simulated dataset 2

Simulated data are used in both tests for the simple models and the complex models. The conclusions drawn are as follows:

(1) The complex model is much more accurate than the simple model, especially

when the rock has large rotation angles.

(2) For the ideally simulated data, the distribution of the simulated points does not

affect the results of the complex models, because the parameters (height, radius,

and rotation angles) calculated are exactly the same as the real data. In addition,

the RMS errors are all equal to zero.

51 (3) As for the simple models, in order to get more accurate modeling parameters,

it is better to have all the surface points evenly distributed around the model.

Furthermore, as the values of the rotation angles increase, the rock modeling

parameters become less accurate, and the RMS errors also increase.

2.4.2 Comparison of the complex rock models with the simple models using real data

Although the complex model with the rotation angles seems more accurate in describing a rock, for the real data, based on extensive tests at Sites 9600 and 9700, as well as Sites 11400 and 11493 from the MER mission data at the Spirit landing site, it was found that the parameters (e.g. height and radius) estimated from the simple models and the complex models are close. This is because the shape of a real rock is not actually a true hemisphere, ellipsoid, cone, or tetrahedron. With more unknown parameters added into the observation equation, over-fitting happens, and it is actually more difficult to get more accurate results. The test results (Tables 2.6-2.8) also show that most rocks stand upright in real world. Rocks with large rotation angles are rare. Furthermore, the simple models make the rock modeling process computationally more efficient and more robust with a limited number of surface points. Therefore, at the Spirit landing site, the simple models are sufficient for rock modeling.

52 Simple model Complex model Difference ID H R RMS H R α β RMS dH dR dRMS Model Model (cm) (cm) (cm) (cm) (cm) (deg) (deg) (cm) (cm) (cm) (cm) 1 2 15.6 11.1 3.92 3 16.7 17.9 -2 -10 3.58 1.1 6.8 -0.34 2 4 28.5 28.6 4.32 3 28.5 25.3 -10 -4 4.09 0 -3.3 -0.23 4 4 13.8 8.9 4.06 4 13.8 9.5 -10 -10 3.98 0 0.6 -0.08 5 4 9.4 22 1.51 2 9.5 10.6 5 2 1.4 0.1 -11.4 -0.11 6 4 40.5 41.2 4.03 1 36.4 32.6 -10 -10 4.31 -4.1 -8.6 0.28 7 4 22.3 25.2 4.21 2 21.4 15.1 10 -4 3.58 -0.9 -10.1 -0.63 8 3 25.4 35.9 4.11 1 24.4 24.8 9 7 3.88 -1 -11.1 -0.23 9 4 14.6 19.2 3.06 3 14.6 15.5 -1 1 3.17 0 -3.7 0.11

Table 2.6 Model comparison based on the MER-A data at Sites 9600 and 9700

Simple model Complex model Difference ID H R RMS H R α β RMS dH dR dRMS Model Model (cm) (cm) (cm) (cm) (cm) (deg) (deg) (cm) (cm) (cm) (cm) 2 3 18.8 25.5 1.91 3 18.8 22.9 0 -1 0.86 0 -2.6 -1.05 3 1 22 31 3.3 4 25.9 37.2 -10 -10 3.37 3.9 6.2 0.07 4 4 28.7 31.5 3.52 4 28.7 30.4 -10 -10 3.36 0 -1.1 -0.16 5 4 32.4 43.2 3.56 4 32.4 47.9 -10 -10 3.65 0 4.7 0.09 9 1 13.7 18.5 2.63 1 13.9 20.7 5 10 2.13 0.2 2.2 -0.5 10 3 10.3 16.7 0.93 3 10.3 15.6 -5 -4 1.03 0 -1.1 0.1 12 2 11.2 7.2 1.8 2 11.2 5.7 7 -10 1.73 0 -1.5 -0.07 14 3 9.7 24 2.34 3 9.7 14.1 -2 -10 2.38 0 -9.9 0.04 16 4 8.2 8.8 1.97 4 8.2 10.3 -10 -10 1.93 0 1.5 -0.04 17 3 15.3 20.5 3.33 3 15.3 17.3 2 -2 3.69 0 -3.2 0.36 18 2 12 21.4 1.13 2 12 20.2 -10 6 0.77 0 -1.2 -0.36 19 3 9 9.3 2.08 3 9 10.3 -10 -2 2.09 0 1 0.01

Tables 2.7 Model comparison based on the MER-A data at Site 11400

Simple model Complex model Difference ID H R RMS H R α β RMS dH dR dRMS Model Model (cm) (cm) (cm) (cm) (cm) (deg) (deg) (cm) (cm) (cm) (cm) 1 4 14.3 19.3 0.9 4 14.3 21.5 -10 -10 2.62 0 2.2 1.72 2 4 14.7 17.7 4.08 1 12.5 18.8 10 10 3.81 -2.2 1.1 -0.27 4 4 19.6 22.8 2.06 4 19.6 22.3 -10 -10 1.93 0 -0.5 -0.13 7 1 15.4 12.8 1.34 1 14.4 17.5 10 4 2.81 -1 4.7 1.47 8 4 16.4 43.8 2.81 3 16.4 40.9 1 2 2.75 0 -2.9 -0.06 9 4 19.1 30 2.69 3 19.1 25.9 -2 -1 3.71 0 -4.1 1.02 11 4 16.9 30.6 3.2 4 16.9 38.1 -10 -10 2.14 0 7.5 -1.06 13 4 16.5 23.6 0.85 3 16.5 13.8 0 1 1.23 0 -9.8 0.38 20 3 33.8 69.4 4.24 3 33.8 85.1 -3 -7 3.97 0 15.7 -0.27 23 1 33.5 30.6 3.64 1 34.7 42.5 -9 -1 3.73 1.2 11.9 0.09 26 1 37.2 26.2 4.13 2 45.5 30.9 6 -6 4.33 8.3 4.7 0.2 27 4 16.5 34 2.84 2 18.9 20.6 -10 10 2.61 2.4 -13.4 -0.23 28 4 28 45.1 2.81 1 29.5 27.9 5 -10 2.84 1.5 -17.2 0.03 29 4 37.9 48.2 3.63 1 34.4 46.1 -10 -9 3.73 -3.5 -2.1 0.1

Tables 2.8 Model comparison based on the MER-A data at Site 11493

53 From the above comparison, it is concluded that the calculated orientation angle for the same rock of both sites may be quite different, although theoretically, the orientation angles should be identical. This is caused by the different looking angles of the cameras at these two sites. Sometimes, the extracted rock peaks are not the same points. They might be located on the same rock but deviate by some distance. In addition, the extracted surface points can also be different for the same rock pictured at two adjacent sites. These surface points may not have much overlapping.

Even if the orientation angles of a rock were calculated at both sites, the comparison would not help much for the rock matching between the two sites. It also takes more time to do the computation in the rock modeling process. Therefore, the simple models without any orientations work better in terms of efficiency and speed than the complex models. However, it is important to add the normal direction of a fitting plane around a rock as one of the rock modeling parameters, which can be further used for rock model matching.

2.4.3 Rock modeling results

Figure 2.12 (Li et al., 2007a) shows nine examples of rocks of different sizes and shapes. Each rock is modeled by using the four analytical models. Table 2.9 (Li et al.,

2007a) gives the estimated model parameters of the four models for each rock, as well as the associated RMS errors of each model. The highlighted (red) RMS error of each rock is the minimum out of the four models. The corresponding model is considered the best model for that rock. For example, Rock 2 is best represented by a tetrahedron, while

Rock 1 is best represented by a semi-ellipsoid.

Figure 2.12: Nine rocks used for modeling with four analytical models (green dots: rock peaks)

Hemispheroid model Cone model Semi-Ellipsoid model Tetrahedron model

S.No a h rms r h rms a b h rms a h φ rms 1 12.2 15.5 4.5 17.0 16.7 3.93 11.1 14.6 15.6 3.92 14.0 16.7 1.2 4.1 2 20.8 26.8 5.1 23.5 28.5 4.9 20.1 23.2 27.4 5.2 22.5 28.5 1.0 4.4 3 26.3 23.1 5.0 30.6 24.6 4.8 21.0 32.0 24.0 5.4 32.6 24.6 1.2 3.8 4 9.3 13.4 5.4 5.7 13.8 5.4 17.0 5.2 12.4 4.8 8.7 13.8 1.3 4.1 5 9.1 9.6 2.0 19.0 9.4 1.7 10.6 6.8 9.7 1.52 22.0 9.4 1.2 1.51 6 40.9 35.3 5.1 31.6 40.5 5.8 38.5 59.4 34.1 5.1 34.6 40.5 0.1 3.4 7 22.6 20.0 4.7 25.8 22.3 5.2 14.7 27.3 20.3 4.6 22.8 22.3 0.8 3.7 8 28.7 24.2 5.5 30.9 25.4 4.1 25.4 31.3 24.4 5.2 34.9 25.4 0.8 4.5 9 13.5 13.2 4.0 16.2 14.6 3.4 15.1 13.0 13.2 3.9 22.2 14.6 0.8 2.9

Table 2.9 Estimated model parameters and RMS errors (φ in radian, all others in cm) of rocks in Figure 2.12

The models of landmarks usually deviate considerably from the natural appearance. To verify the rock modeling results, Li et al. (2007a) compared the modeled parameters with the ground truth (manual measurements) of 79 rocks in the area between two adjacent sites that were 26 m apart at the Spirit landing site. Table 2.10 is an example of comparison results from four rocks shown in the top row of Figure 2.9. The big 55 difference in the height and radius is caused by the following reasons: (1) real rocks are not symmetric. A simple model may not be the best representation of the rock shape. (2)

Surface points are only locate on one face of the rock. The other face is not visable and is assumed to be symmetric. On average, for the 79 rocks, the relative error between a modeled parameter and the corresponding ground truth measurement was 25.1% in height, 43.7% in radius, 57.1% in surface area, and 103.4% in volume, respectively

(Table 2.11). It is obvious that height is the most reliable and thus the most comparable parameter among the four metrics. The very high difference in volume suggests that it should not be used for comparison. This verification result is important for designing the algorithm for the matching of rocks with various models.

Model Results Ground Truth ID Dif_H Dif_R h a b 2r/hrms Model h r 2r/h 1 40.5 34.6 1.71 3.4 Tetrahedron 61 62.5 2.05 -33.6% -44.6% 2 30.7 34.2 41.7 2.47 8.2 Semi-Ellipsoid 22 31.0 2.82 39.5% 22.4% 3 28.5 22.5 1.58 4.4 Tetrahedron 19 28.1 2.96 50.0% -19.9% 4 21.6 22.9 2.12 2.3 Tetrahedron 23 30.8 2.68 -6.1% -25.6%

Table 2.10 Comparison of estimated model parameters with ground truth for 4 rocks in Figure 2.9

Parameters Relative error Standard (%) deviation (%) Height 25.1 24.1 Radius 43.7 54.6 Surface area 57.1 60.6 Volume 103.4 145.5

Table 2.11 Average difference between the estimated model parameters and ground truth

56 Landmarks are one of the important factors that can be used for lander localization and rover navigation after landing (Li et al., 2006a). After landmark extraction and modeling, the next step is to match landmarks from different sources.

However, the inconsistency of the viewing angle is a major challenge for achieving a correspondence between cross-site landmarks. The next chapter will present the techniques used to match the landmarks from cross rover sites.

CHAPTER 3

LANDMARK MATCHING

In Chapter 2, the methods for landmark extraction and modeling were presented.

This chapter will further describe the matching method to select cross-site tie points by

comparing those landmark models automatically. This chapter is composed of three main

sections. The first section will introduce the difficulties in the current work on cross-site

tie points selection, where automatic matching is one of the key techniques. Next,

different matching techniques will be described and compared. They include landmark

modeling matching and global pattern matching. Both methods are used complementarily with each other to eliminate the potential mismatches. In the third section, the application results of the matching methods will be discussed. The results of this study show that the proposed approach functions successfully to select good cross-site tie points for medium- range (up to 26 meters) traverse segments. In addition, a preliminary study of landmark

matching between the orbital and ground images has been conducted. Some initial results

are included in this chapter.

3.1 Background for the automated landmark matching

Landmark matching is one of the key steps in building an image network used in

bundle adjustment for rover localization. A great deal of effort has been spent on

58 matching landmarks from different sources of images including the stereo

Pamcam/Navcam images from a single site and cross sites, as well as orbital images.

People have been responsible for overlaying and comparing various data products including orthophotos, DTMs, footprints, anaglyph maps, and back-projection maps to interpret the image and make a decision based on their own human intelligence. In most cases, human eyes can reliably perceive landmarks and match feature patterns. This manual process is also labor-intensive for the human operator. For example, in the MER mission, the locations of the landers in the Gusev Crater and on the Meridiani Planum were identified manually by cartographic triangulation of landmarks visible in both orbital and ground images (Li et al., 2005). Additionally, in order to get a global control for the local coordinate system, ground imagery is manually referenced to MDIM and

MOLA grid data through a four-stage process: 1) Pancam to DIMES and MOC using an incremental BA technique, 2) MOC to THEMIS, 3) THEMIS to MDIM, and finally, 4)

MDIM to MOLA. The manual cross-site tie points selection for BA in the MER mission operation is one of the most time-consuming and difficult tasks.

Landmark matching is classified into two sub-categories: cross-site and orbital to ground. Since orbital and ground integration is a complex and, as of yet, unsolved topic, this dissertation will only include some of the preliminary results at the end of this chapter. The technique for cross-site landmark matching is what will mainly be covered in this chapter. This process consists of three levels of image registration: intra-stereo, inter-stereo, and cross-site (Xu, 2004). Definitions of intra-stereo, inter-stereo, and cross- site are given in Chapter 1 (Figures 1.1 and 1.2).

59 Tie points selection from the intra- and inter-stereos at one rover site was

automated based on automatic image matching using the MarsMapper software

developed by the OSU Mapping and GIS Lab (Xu, 2004). However, the image matching

strategy in MarsMapper cannot be used to link rocks from different sites because of the

shape deformation introduced by large perspective transformations and different

picturing time. For example, different parts of the rock might be visible or occluded from

different sites because of different looking angles. Additionally, rock size might be

measured differently due to difference in camera-object distance. Sometimes the distinct

landmarks shown at one site might be absent from the adjacent site. A short traverse (e.g.

<30 m) is usually desired in order to get more overlapping at cross-site when considering the camera capabilities. Therefore, many of the tie points among the pairs of adjacent sites, named cross-site tie points, are selected manually during MER mission operations.

This chapter aims to automate the process of cross-site tie points selection through automatic matching of cross-site landmark models. The concept of cross-site landmark selection is briefly presented in Figure 1.4 (Agarwal, 2006). As described in the last chapter, rocks are the most common feature present at both landing sites. After the significant rock features are identified and modeled in the object space at the current and adjacent sites, corresponding rocks are matched and used as tie points for bundle adjustment. The rover position at the adjacent site is refined.

Figure 3.1 lists six pairs of rocks identified manually from two sites (current and adjacent). The matched pairs are labeled with the same identification numbers in green and red. The shapes of some of the pairs look quite different at the two sites (e.g. Rock

60 4). It is even hard for human eyes to judge that they are correct matches at the first glance without any other auxiliary tools.

1 2 3 4 5 6

Figure 3.1: Examples of manually matched pairs from two sites

However, by the modeling method proposed in Chapter 2, these rocks are extracted and modeled automatically. The modeling parameters of rocks extracted at both sites are compared. Results are shown in Table 3.1. In the table, parameters h (height), r

(radius), a (semi-major axis), b (semi-minor axis), and RMS are in the unit of cm. RMS errors are calculated to judge the model accuracy of a rock model. Parameter φ (in radians) of the tetrahedron model is not used for comparison because this angle depends on the looking direction. The parameters of corresponding rocks from different sites are not exactly the same but are comparable. Some rules concerning these rock parameters can be defined to build the correspondence between the rocks from two sites.

61 Master Site Adjacent Site ID h r/a b 2r/h RMS Model h r/a 2r/h RMS Model dif_H dif_R 1 40.5 34.6 1.71 3.4 Tetrahedron 40.1 36.2 1.80 7.0 Cone 0.9% -4.6% 2 30.7 34.2 41.7 2.47 8.2 Semi-Ellipsoid 21.1 22.8 2.16 5.2 Cone 31.2% 39.8% 3 28.5 22.5 1.58 4.4 Tetrahedron 22.9 21.8 1.90 3.1 Cone 19.6% 3.1% 4 21.6 22.9 2.12 2.3 Tetrahedron 25.4 17.5 1.38 2.5 Cone -17.4% 23.7% 5 12.7 14.3 2.26 1.4 Cone 11.6 10.3 1.77 0.8 Cone 8.6% 28.4% Hemi- 6 24.2 16.5 17.1 1.36 4.2 Semi-Ellipsoid 27.6 19.3 1.40 4.8 -13.9% -17.4% spheroid

Table 3.1 Comparison of modeling parameters of rocks from the current and adjacent site

Most of the rocks shown in a terrain surface are irregular, while the hemispheroid, semi-ellipsoid, and cone models defined in Chapter 2 are symmetric except the tetrahedron model. Sometimes, the same rocks are not modeled by an identical model. In addition, some rocks may not have their corresponding match extracted at the adjacent site. In order to match the cross-site landmarks, assumptions are made as follows.

• The rock modeling parameters represent the shape of a rock well.

• Although two rocks are approximated with different models, they are

comparable by the parameters of height, radius (or averaged radius for a semi-

ellipsoid), and surface area.

• Each rock has no more than one correct match at the adjacent site.

3.2 Landmark matching

Landmark matching is one of the key steps in the cross-site tie points selection. In order to build the correspondence between cross-site landmarks, a great deal of effort has been made on setting matching rules for landmark comparison. In this section, the methods of the landmark modeling matching and global distribution pattern matching are

62 studied and applied to our dataset. Landmark model matching uses an objective function to compare a single landmark’s individual similarity with potential corresponding landmarks from the other source. Global distribution pattern matching considers the global offset between the distributions of two sets of landmarks from two sources. It is also a kind of point pattern matching. In order to select the most important parameters when designing a cost function, parameter sensitivity is analyzed in the following section.

In addition, match criteria are given based on this analysis.

3.2.1 Analysis of model parameter sensitivity

After rock modeling, the sensitivity of volume and surface area of the hemispheroid, semi-ellipsoid, cone, and tetrahedron models with respect to different parameters are analyzed. Li et al. (2006b) gave a brief introduction of parameter sensitivity analysis in their project report. More details of the sensitivity analysis will be given in this section. Matching criteria will be created based on this analysis. In the following equations, the parameter r is the radius of the hemispheroid or the radius of the bottom circle of a cone; a and b are semi-major and semi-minor axes of the semi- ellipsoid; and h is rock height. Parameters V and S are the volume and surface area of each model. An assumption is made that the r and h are measured with the same uncertainty.

Hemispheroid Model

The volume of a hemispheroid model is represented in Equation 3.1.

2 V = πr2h (3.1) 3

63 The first derivative of the volume function is taken with respect to the parameters

of height and radius. Next, the difference between the two derivatives is shown in

Equation 3.2.

⎧> 0, when r > 2h ∂V ∂V 2 2 4 2 ⎪ − = πr − πrh = πr(r − 2h) = ⎨= 0, when r = 2h (3.2) ∂h ∂r 3 3 3 ⎪ ⎩< 0, when r < 2h

As seen in the equation above, if the hemispheroid radius r is larger than 2h, the

volume of a hemispheroid is more sensitive to h parameter. Otherwise, the volume is

more sensitive to r when r is less than 2h.

The calculation of the surface area of a hemispheroid model is different when the

shape of a hemispheroid is in oblate or prolate condition. Equation 3.3 illustrates two

representations of these conditions.

2 ⎧ 2 πh ⎛1+ e ⎞ πr + ln⎜ ⎟, when h < r (oblate) ⎪ 2e 1− e S = ⎨ ⎝ ⎠ (3.3) πrh ⎪ πr 2 + sin −1 e, when h > r (prolate) ⎩⎪ e

h 2 Where e = 1− r 2

Usually, the surface area of a hemispheroid model is approximated by Equation

3.4.

1 ⎛ 2 p p p ⎞ p r + 2r h (3.4) S ≈ 2π ⎜ ⎟ ⎝ 3 ⎠

Where p = 1.6075 yields a relative error of at most 1.061%.

Similarly, the first derivative of the surface area function is taken with respect to parameters of radius and height. The difference between the two derivatives is shown in 64 Equation 3.5. The derivation in Equations 3.5 and 3.6 shows ∂S /∂r is larger than∂S /∂h .

Therefore, the surface area of a hemispheroid is more sensitive to r.

⎛ 1 ⎞ ⎜ ⎟ 2 p p p ⎜ −1⎟ 2 p−1 p−1 p p p−1 ∂S ∂S ⎛ r + 2r h ⎞⎝ p ⎠ ⎛ 2r + 2r h − 2r h ⎞ − = 2π ⎜ ⎟ ⎜ ⎟ ∂r ∂h ⎝ 3 ⎠ ⎝ 3 ⎠ p−1 (3.5) ⎛ 1 ⎞ r ⎜ ⎟ p−1 p−1 2 p p p ⎜ −1⎟ 2r h (r + h − r) ⎛ r + 2r h ⎞⎝ p ⎠ h p−1 = 2π ⎜ ⎟ > 0 ⎝ 3 ⎠ 3

r p−1 r r + h − r = r( ) p−1 + h − r > r *1 + h − r = h > 0, when h < r h p−1 h r p−1 r + h − r > h − r > 0, when h > r (3.6) h p−1 r p−1 r + h − r = r = h > 0, when h = r h p−1

Semi-ellipsoid Model

The volume of a semi-ellipsoid is represented in Equation 3.7.

2 V = πabh (3.7) 3

Equation 3.8 stems from the first derivatives of the volume function with respect to parameters of height, semi-major axis, and semi-minor axis. This equation demonstrates that the volume of a semi-ellipsoid is more sensitive to the smallest parameter. For example, if h is less than both a and b, the volume is more sensitive to h.

⎧∂V 2 = πab ⎪ ∂h 3 ⎪∂V 2 ⎨ = πbh (3.8) ⎪ ∂a 3 ∂V 2 ⎪ = πah ⎩⎪ ∂b 3

The surface area of a semi-ellipsoid model is shown in Equation 3.9.

65 2 ⎛ 2 bh 2 2 ⎞ S = π ⎜h + F()θ,m + b a − h E()θ,m ⎟ (3.9) ⎝ a2 − h2 ⎠

a 2 ()b2 − h2 Where m = , b2 ()a 2 − h2

θ = arcsin()e ,

h2 e = 1− , a2

and F(θ, m) and E(θ, m) are the incomplete elliptic integrals of the

first and second kind.

The surface area is approximated by Equation 3.10.

1 p p p p p p p ⎛ a b + a h + b h ⎞ (3.10) S ≈ 2π ⎜ ⎟ ⎝ 3 ⎠

Where p = 1.6075 yields a relative error of at most 1.061%.

In the same way, the first derivative of the surface area function is taken with respect to parameters of height, semi-major axis, and semi-minor axis. From Equation

3.11, the conclusion is that the surface area of a semi-ellipsoid is more sensitive to the smallest parameter. For example, if h is less than both a and b, the surface area is more sensitive to h.

⎛ 1 ⎞ ⎧ p p p p p p ⎜ −1⎟ p−1 p p−1 p ∂S ⎛ a b + a h + b h ⎞⎝ p ⎠ ⎛ h a + h b ⎞ ⎪ = 2π ⎜ ⎟ ⎜ ⎟ ⎪ ⎜ ⎟ ⎜ ⎟ ∂h ⎝ 3 ⎠ ⎝ 3 ⎠ ⎪ ⎛ 1 ⎞ ⎪ p p p p p p ⎜ −1⎟ p−1 p p−1 p ⎪∂S ⎛ a b + a h + b h ⎞⎝ p ⎠ ⎛ a b + a h ⎞ ⎨ = 2π ⎜ ⎟ ⎜ ⎟ (3.11) ⎪∂a ⎝ 3 ⎠ ⎝ 3 ⎠ ⎛ 1 ⎞ ⎪ p p p p p p ⎜ −1⎟ p−1 p p−1 p ⎪∂S ⎛ a b + a h + b h ⎞⎝ p ⎠ ⎛ b a + b h ⎞ = 2π ⎜ ⎟ ⎜ ⎟ ⎪ ∂b ⎝ 3 ⎠ ⎝ 3 ⎠ ⎩⎪

66 Cone Model

The volume of a cone model is shown in Equation 3.12.

1 V = πr2h (3.12) 3

Similarly, the first derivative of the volume function is taken with respect to parameters of radius and height, respectively. The difference between the two derivatives is shown in Equation 3.13. It concludes that the volume of a cone is more sensitive to h parameter when r is larger than 2h; while the volume is more sensitive to r as r is less than 2h.

⎧> 0, if r < 2h ∂V ∂V 2 1 2 1 ⎪ − = πrh − πr = πr(2h − r) = ⎨= 0, if r = 2h (3.13) ∂r ∂h 3 3 3 ⎩⎪< 0, if r > 2h

The calculation of the surface area of a cone model is shown in Equation 3.14.

S = πr r2 + h2 (3.14)

The first derivative of the surface area function is taken with respect to the parameters of radius and height. The difference between the two derivatives shown in

Equation 3.15 demonstrates that∂S / ∂r is larger than∂S / ∂h . Therefore, the surface area of a cone is more sensitive to r.

∂S ∂S πr 2 πrh − = π r 2 + h 2 + − ∂r ∂h r 2 + h 2 r 2 + h2 πr(r − h) = π r 2 + h2 + r 2 + h 2 (3.15) π (r 2 + h 2 + r 2 − rh) = r 2 + h2 π (r 2 + (h − r)2 + rh) = > 0 r 2 + h 2

67 Tetrahedron Model

No analytical equation exists for the tetrahedron model. Therefore, three parameters are used to represent a tetrahedron: height (h), radius (r) of the enclosing circle of the bottom triangle, and orientation angle (φ) of the bottom triangle. The volume and surface area of a tetrahedron model are shown in Equation 3.16 and Equation 3.17.

1 1 V = Ah = r2h 3 4 (3.16) 3 Where A = r 2 4

3 3 S = r r2 + 4h2 (3.17) 4

By taking the first derivatives of the volume and surface area functions with respect to the parameters of height and radius, respectively, Equation 3.18 and Equation

3.19 are deduced.

⎧> 0, if r > 2h ∂V ∂V 1 2 1 1 ⎪ − = r − rh = r * (r − 2h) = ⎨= 0, if r = 2h (3.18) ∂h ∂r 4 2 4 ⎪ ⎩< 0, if r < 2h

∂S ∂S 3 3 3 3 r 2 3 3rh − = r 2 + 4h2 + − ∂r ∂h 4 4 r 2 + 4h2 r 2 + 4h 2 3 3 3 3r 2 −12 3rh = r 2 + 4h2 + 4 4 r 2 + 4h 2 (3.19) 3 3(r 2 + 4h 2 + r 2 − 4rh) = 4 r 2 + 4h2 3 3(r 2 + (2h − r)2 ) = > 0 4 r 2 + 4h 2

Further, it is concluded that if r is larger than 2h, the volume of a tetrahedron is more sensitive to h. Otherwise, if r is less than 2h the volume is more sensitive to r 68 (Equation 3.18). On the contrary, the surface area of a tetrahedron is always more sensitive to r because∂S / ∂r is larger than ∂S /∂h (Equation 3.19).

Two conclusions are drawn from the analysis of parameter sensitivity for these four models (Table 3.2). The first is that hemispheroid, cone, and tetrahedron models have some similarities in parameter sensitivity. The volumes of these models are more sensitive to h when r is larger than 2h; otherwise, the volume is more sensitive to r when r is less than 2h. In addition, the surface area of these three models is more sensitive to r.

Unlike the first conclusion, the second finding is that both the volume and surface area of a semi-ellipsoid model are more sensitive to the smallest parameter. In the test at the

MER-A sites, most of the rock models calculated have r parameter less than twice of h parameter. Therefore, both volume and surface area of the hemispheroid, cone, and tetrahedron are more sensitive to the r parameter.

Models / Sensitivity Volume Surface area Hemispheroid, Cone, more sensitive to h, when r>2h more sensitive to r Tetrahedron more sensitive to r, when r<2h Semi-ellipsoid more sensitive to the smallest more sensitive to the parameter smallest parameter

Table 3.2 Parameter sensitivity of four models

Before further experiments were performed on rock model matching, the comparability of relative errors was analyzed. The relative errors in volume and surface area were deduced generally from Equations 3.20 to 3.23 for the four models.

69 Hemispheroid, Cone, Tetrahedron Semi-ellipsoid ∂V ∂V ∂V ∂V ∂V δV = δh + δr δV = δh + δa + δb (3.20) ∂h ∂r ∂h ∂a ∂b

∂S ∂S ∂S ∂S ∂S δS = δh + δr δS = δh + δa + δb (3.21) ∂h ∂r ∂h ∂a ∂b

δV 1 ⎛ ∂V ∂V ⎞ δV 1 ⎛ ∂V ∂V ∂V ⎞ = ⎜ δh + δr⎟ = ⎜ δh + δa + δb⎟ (3.22) V V ⎝ ∂h ∂r ⎠ V V ⎝ ∂h ∂a ∂b ⎠

δS 1 ⎛ ∂S ∂S ⎞ δS 1 ⎛ ∂S ∂S ∂S ⎞ = ⎜ δh + δr⎟ = ⎜ δh + δa + δb⎟ (3.23) S S ⎝ ∂h ∂r ⎠ S S ⎝ ∂h ∂a ∂b ⎠

The analysis of a cone model was described in a report by Li et al. (2006b).

Equations 3.24 to 3.27 are also given as an example of the hemispheroid model.

2 4 δV = πr 2δh + πrhδr (3.24) 3 3

⎛ 1 ⎞ ⎜ ⎟ 2 p p p ⎜ −1⎟ p p−1 2 p−1 p−1 p ⎛ r + 2r h ⎞⎝ p ⎠ ⎛ 2r h ⎛ 2r + 2r h ⎞ ⎞ ⎜ ⎟ ⎜ ⎜ ⎟ ⎟ (3.25) δS = 2π ⎜ ⎟ ⎜ δh + ⎜ ⎟δr ⎟ ⎝ 3 ⎠ ⎝ 3 ⎝ 3 ⎠ ⎠

δV δh 2δr = + (3.26) V h r

()−1 δS ⎛ r 2 p + 2r p h p ⎞ ⎛ 2r p h p−1 ⎛ 2r 2 p−1 + 2r p−1h p ⎞ ⎞ ⎜ ⎟ ⎜ ⎜ ⎟ ⎟ = ⎜ ⎟ ⎜ δh + ⎜ ⎟δr ⎟ S ⎝ 3 ⎠ ⎝ 3 ⎝ 3 ⎠ ⎠ 2 ⎛ δh δr ⎞ = ⎜r p h p + ()r 2 p + r p h p ⎟ r 2 p + 2r p h p ⎝ h r ⎠ (3.27) ⎛ ⎛ p ⎞ ⎞ 2 ⎛ p δh p p δr ⎞ 2 ⎜ δh ⎜⎛ r ⎞ ⎟ δr ⎟ = p p ⎜h + ()r + h ⎟ = p + ⎜ ⎟ +1 r + 2h ⎝ h r ⎠ ⎛ r ⎞ ⎜ h ⎜⎝ h ⎠ ⎟ r ⎟ ⎜ ⎟ + 2 ⎝ ⎝ ⎠ ⎠ ⎝ h ⎠

Table 3.3 presents the deduced equations for the relative error in volume or surface area for the four models with respect to the relative error of the other dimension

70 parameters (h, r, or a, and b). The relative error in volume (δ V V ) for the hemispheroid, cone, and tetrahedron models is always the sum of the relative error in height (δ h h ) and twice the relative error in radius (δ r r ). The relative error in volume for the semi- ellipsoid is also approximated by this relationship, if an averaged radius ( r ) is used to represent a and b. This relationship also explains why the average relative error in volume reaches 103.4% from the ground experiment results (Table 2.11). From the theoretical calculation, δ h h + 2δ r r is 112.5% (25.1% for δ h h and 43.7% for

δ r r ), which is very close to the experimental value of 103.4%. The relative error in surface area for the three models (hemispheroid, cone, and tetrahedron) is always more than the maximum of δ h h and δ r r , while it is hard to draw a conclusion from the equation for that of a semi-ellipsoid. It was proven that the relative errors of volume,

δV δS δh δr area, height, and radius ( , , , ) have the same order and are directly V S h r comparable. However, their contributions in the objective function vary.

71 δV δS Models V S ⎛ ⎛ p ⎞ ⎞ 2 ⎜ δh ⎜⎛ r ⎞ ⎟ δr ⎟ Hemispheroid δh 2δr p 1 + ⎜ ⎟ +1 + ⎛ r ⎞ ⎜ h ⎜⎝ h ⎠ ⎟ r ⎟ h r ⎜ ⎟ + 2 ⎝ ⎝ ⎠ ⎠ ⎝ h ⎠ p−1 p−1 p p δh δa δb ⎛ h ⎞ ⎛ h ⎞ ⎛ h ⎞ ⎛ h ⎞ Semi- + + (a⎜ ⎟ + b⎜ ⎟ )δh + b(1 + ⎜ ⎟ )δa + a(1 + ⎜ ⎟ )δb h a b ⎝ b ⎠ ⎝ a ⎠ ⎝ b ⎠ ⎝ a ⎠ Ellipsoid p p δh 2δ r ⎛ h ⎞ ⎛ h ⎞ ≈ + ab(1 + ⎜ ⎟ + ⎜ ⎟ ) h r ⎝ a ⎠ ⎝ b ⎠ ⎛ ⎞ δh 2δr ⎜ ⎟ + 1 δh ⎜ 1 ⎟ δr Cone h r + 1 + 2 h ⎜ 2 ⎟ r ⎛ r ⎞ ⎛ h ⎞ ⎜ ⎟ + 1 ⎜ 1 + ⎜ ⎟ ⎟ ⎝ h ⎠ ⎝ ⎝ r ⎠ ⎠ ⎛ ⎞ ⎜ ⎟ 1 δh ⎜ 1 ⎟ δr Tetrahedron δh δr + 1 + + 2 2 h ⎜ 2 ⎟ r h r ⎛ r ⎞ ⎛ 2h ⎞ ⎜ ⎟ + 1 ⎜ 1 + ⎜ ⎟ ⎟ ⎝ 2h ⎠ ⎝ ⎝ r ⎠ ⎠

Table 3.3 Relative error in volume and surface area with respect to that of the dimension parameters

3.2.2 Landmark model matching

Landmark model matching is processed in two steps: (1) shape comparison and

(2) local terrain comparison around two landmarks.

The shape of a model can be decided by its surface area and volume. From the above sensitivity analysis, we know both radius (or the semi-major and semi-minor axis) and height are important shape parameters of a model. They decide the volume and surface area of each model. In order to match different models of a landmark from various sources, it is important to select important parameters and design the cost function based on pairwise parameter difference between various models. 72 In the experiment on rock model verification (Li et al., 2007a), the high difference

(103.4%) in volume suggests that it should not be used for comparison of different models. Therefore, parameters such as rock model height, radius, and surface area are considered important factors in the cost function for rock model matching (Equation

3.28).

Cost = c1f1 + c2f2 + c3f3 (3.28)

The factors f1, f2, f3 are the potential relative error (in percentage) in modeling parameters like model height, radius, and surface area between the two rocks calculated from the two rock models. The coefficients c1, c2, and c3 are the relevant weights, which are set as 1/2, 1/3, and 1/6 based on the experimental results of rock model verification at the Spirit site (Li et al., 2007a). The landmark at the adjacent site with the minimum cost is chosen as the best match.

In addition to the rock shape comparison, terrain comparison around two rocks can give more validation for rock matching. The terrain information around a rock is approximated by a fitting plane. Although different sets of surface points and background points may be extracted from the same rock at the current site and adjacent site, the fitting planes simulated using the background points are supposed to be similar at both sites. The normal direction of the fitting plane is calculated. By comparing the difference in the normal directions of the fitting planes, those pairs with a large difference in the normal directions are excluded. Figure 3.2 presents a comparison of the fitting planes and normal directions of two matched rock pairs. The matched rocks are marked with the same identification number shown in the top part of the figure. In the bottom plots, the red dots and green dots represent the background points of a rock extracted from the

73 current and adjacent sites, respectively. Fom the plots, we see that although the extracted terrain background points (red and green dots) from the current and adjacent sites are different in their numbers and positions for both rocks, each rock has similar fitting planes at both sites, because the fitting planes have the normal direction (blue line) almost in parallel.

2 1 2 1

current site adjacent site

-1.2 -1.1 -1.25 -1.15

-1.3 -1.2

-1.35 -1.25

-1.4 -1.3 Up (m) Up Up (m) Up -1.35 -1.45 -1.4 -1.5 140 -1.45 128 -1.55 -1.5 130 135 129 -1.6 130 North (m) 131 132 126 132 127 128 130 129 130 131 133 134 132 134 North (m) East (m) East (m) (a) Comparison for Rock 1 (b) Comparison for Rock 2

Figure 3.2: Compare the fitting plane and normal direction (Sites 1200 and 1300)

3.2.3 Landmark pattern matching

On the other hand, the point pattern matching method can be used to match sets of landmarks represented in the form of points. Given two sets of points in 3-D space, we need to find out if the first set can be mapped onto (or satisfactorily close to) the second set of points by some space transformation such as affine transformation. The assumption

74 is made that there is distance invariance between points and angle invariance between lines joining the points. In principle, a rigid transformation (Casselman, 2004) with three rotations and a 3D translation should be used to depict the relationship between two sets of points.

However, for rock pattern matching, based on extensive tests using Spirit rover data, it was found out that the rotational errors are significantly smaller than the translation errors. Therefore, a 3D translation is sufficient for rock pattern matching. This technique compares rocks from two sites by matching their geometric distributions. First, two sets of rock peaks Pi (i = 1, 2, 3, …, I) and Qj (j = 1, 2, 3 …, J) are extracted from the current site and adjacent site, respectively. Next, a subset of significant peaks Pk (k=1,2, .

. . ,K) is selected from the points Pi (i=1,2, . . . , I) and is used to find matches with Qj

(j=1,2, . . . , J). The point pattern matching process is simplified by the following steps, which include a local pattern matching (steps 1-3) and a global pattern matching (step 4).

This process is computationally more efficient and more robust.

(1) A translation vector (Equation 3.29) is calculated for each significant peak

Pk at the current site and each candidate peak Qm (within a certain distance

from Pk) at the adjacent site.

Vk, m = Pk − Qm (3.29)

(2) This translation vector is applied to all the rock peaks in Q (Equation

3.30). The total number of rocks that can be matched after the translation

is counted by Equation 3.31, where ε is a predefined tolerance.

Q′j = Q j + Vk, m (3.30)

75 Countk, m = Card( Pi − Q′j ≤ ε ) (3.31)

denotes the norm of a vector, which is the distance between two

points. Card represents the cardinal number of the sets of rock peaks that

have possible matches, i.e., the number of rock peaks at the current site

that have rock peaks from the adjacent within the range of ε.

(3) For each significant peak Pk, two candidate matches Qm1 and Qm2 that

have the maximum and second maximum of count are selected. Their

corresponding translation vectors are stored as Vk, m1 and Vk, m2. The above

process is referred to as local pattern matching. Up to this point, two best

candidates are found for each significant rock.

(4) A median vector Vmedian is calculated from all the translation vectors of the

candidates (two for each significant rock). Then the final match for each

significant rock is found by checking the global consistency of candidate

translation vectors and Vmedian. That is, Pk matches Qm1 if

Vk, m1 − Vmedian ≤ Vk, m2 − Vmedian , otherwise Pk matches Qm2. This step is

referred to as global pattern matching.

Rock matching is used to find corresponding rocks in the two sets of extracted rocks from cross sites. Through testing of many data sets from Spirit rover, it was found that most of the correct matches have either the maximum or the second maximum count.

That is why, with local pattern matching two candidate rocks are selected from the adjacent site for each significant rock at the current site. It should also be noted that local pattern matching improves the efficiency and robustness of the rock model matching

76 technique by limiting the number to two rock candidates and reducing the number of possible mismatches in model matching. Therefore, the rock matching technique we developed considers both global rock distribution patterns by rock pattern matching and individual rock similarities by rock model matching. Figure 3.3 (Li et al., 2007a) shows a detailed diagram of the entire rock matching process.

Current site Adjacent Rock peaks and rock Significant rock selection Rock peaks and rock model parameters in each grid cell model parameters

Significant rock peaks and Local rock pattern rock model parameters matching

Rock matching candidates

Rock model Global pattern

Rock model matching Rock pattern matching results

Combining matching results

Final matched rocks

Figure 3.3: Rock matching technique using both model matching and rock distribution pattern matching

In both rock model matching and rock pattern matching, there are cases of multiple matches, i.e., a rock from one site matches more than one rock at the other site.

To eliminate multiple matches, only the “best” one match (the match with minimum 77 objective function value in model matching, or the match that generates maximum count in rock pattern matching) is kept. The results of rock model matching and rock pattern matching may not be very different. The final matching results are the combination of the outputs of the two methods. Only the rocks that pass both matching methods are considered to be matched rock pairs.

3.3 Analysis of results

Landmark matching is tested using the MER Navcam data at the Husband Hill area taken by the Spirit rover. The following is an example of automatic cross-site tie point selection supported by rock matching at Sites 1200 and 1300. The two sites are 23 m apart. There are 71 peaks extracted from Site 1200 and 95 peaks from Site 1300.

Twelve significant rocks were automatically selected from the rocks of Site 1200 in a grid defined between the two sites. Seven rocks were automatically matched by combining the results of rock pattern matching and model matching. Table 3.4 gives a summary of the experimental results of rock matching.

Result \ Rock ID 1 2 3 4 5 6 7 8 9 10 11 12 Rock model × matching (11) Pattern matching (8) × × × ×

Combined result (7) O O O O O

: Correct match; ×: Mismatch; O: No match; −: Eliminated because of multiple matches The numbers in the parentheses are the numbers of correct matches.

Table 3.4 Summary of rock matching results for Sites 1200 and 1300

78 In Figure 3.4, the matched rock pairs are labeled with the same identification numbers and represented as triangle symbols for both sites. Sites positions are marked as circles. The green color represents the current site and the red one represents the adjacent site. The pattern of the distribution of matched rocks looks fairly consistent in both the x and y directions.

Figure 3.4: Distribution of automatically matched rocks as cross-site tie points at Sites 1200 and 1300 of the Spirit rover site

Figure 3.5 also shows the final seven matched rocks marked as crosses on the image mosaics of Sites 1200 and 1300.

7 12 6 10 2 3 5

5 3 2 12 7 10 6

Figure 3.5: Automatically matched rocks (tie points) shown on the image mosaics of Sites 1200 and 1300 (labeled with the same identification numbers)

Although the landmark matching strategy was tested using various data set, difficulties with cross-site landmark matching still exist under certain conditions. One such condition is when we were unable to find the corresponding match of a landmark at another site because of the invisibility or difficult identification caused by occlusion, luminance, or significant distance from the camera center at the other site. The navigation camera only gives desired 3D measurement accuracy up to about 20 m from the camera center (Di and Li, 2007). A landmark such as a rock which is visible at one site might be out of the reliable measurement limit at another site. Therefore, it might be eliminated in the data processing. For example, Figure 3.6 shows two sites, 5702 and 5557, which are

25 m apart. The dune in the red circle is closer to the camera center at Site 5702 than it is

80 at Site 5557, where the dune is more difficult to identify. Another reason it can be difficult to match is because of the different appearance of the same landmark when extracted from different views. For example, a rock from forward and backward looking views may appear quite different, because only the part facing the camera is visible from one site, and the same rock may not have the same surface area imaged from the two separate viewpoints. A third problem with matching occurs when unreliable information is used for matching small landmarks because of the limitation of the stereo range capability.

Site 5702 Site 5557

Figure 3.6: Example of difficult match: same features look different from two sites (25 m apart)

Figure 3.7 gives another example of a corrected match not included in the two candidates. A rock at the current site is mismatched with the two candidates below. The actual correct rock seems to be blocked by other rocks. Better algorithms are expected in the future to improve the accuracy of selected candidates.

Current Site Adjacent Site Correct Match Candidate 1 Candidate 2

Figure 3.7: Example of the corrected match not included in the two candidates

3.4 Preliminary experiments on landmark matching between ground and orbiter

Orbital imagers have provided global topographic information for defining the global coordinate system, for landing site selection, and for landing site mapping. The first Mars control network was created as a global image mosaic using the data obtained from the Mariner 9 and Viking orbiters. The next generation of Mars topographic surveying came from the MGS. Its MOLA data was applied as a control network to reference the MDIM 2.1, as well as the MOC topographic mapping products (Archinal et al., 2003; Archinal et al., 2004; Shan et al., 2005). If the ground and orbital data could be registered precisely, the accuracy of the rover position in the local coordinate system would be highly improved and controlled by the Mars global control network.

However, up until now, orbital and ground images have mostly been processed separately for topographic mapping in Mars missions such as the MER mission and previous Mars Pathfinder missions. The manual matching work between orbital and ground data, as described in Section 3.1, requires a lot of effort from the human operator.

In this section, a brief introduction will be given to some of the preliminary experiments conducted to automatically match the landmarks between the ground and orbiter. Figure

82 3.8 gives the strategy of the data processing plan for automatic landmark matching between the ground and orbital images.

Orbital image Ground images (MOC/NA, HiRISE) (Pancam/Navcam)

Image Image Image preprocessing Space preprocessing

DTM, contour DTM, contour map, and slope Edge extraction in 2D Edge extraction in 2D map, and slope line generation line generation

Landmark (craters, Landmark (craters, mountain peaks, ridges & mountain peaks, ridges , and valleys) model building Object valleys) model building Space

Integration of orbital and ground image by landmark matching

Landmarks serving as control points and tie points in BA Helping rover localization Refining spacecraft pose

Figure 3.8: Strategy of orbital and ground integration

3.4.1 Ground and orbital dataset

The resolution of orbital imagery varies from the decimeter level (HiRISE) to a few meters (MOC/NA), and even to hundreds of meters (MDIM). HiRISE imagery with a resolution of 30 cm links the gap between ground and MOC/NA, while THEMIS visible imagery, with a resolution of 18 m, serves as a bridge to link MOC/NA and

MDIM imagery. The registration results are further grouped into the Mars global coordinate system through MOLA data. In this experiment, HiRISE stereo images were used as the orbital data while ground data included MER-B Pancam stereo pairs and their pointing information (Figure 3.9). Both datasets cover Victoria Crater. More information 83 on the HiRISE images can be found in Table 3.5 (Li et al., 2007c). In addition to the

HiRISE images from Victoria Crater (Figure 3.9c), a group of HiRISE images with other crater features was also chosen (Figure 3.10). Those images worked as the test data for results comparison.

The ground data used in this experiment is the orthophoto mapped at the Duck

Bay area, which is a feature of Victoria Crater. The orthophoto was generated using multi-site images, including wide baseline (5 m) Pancam images from sols 1204 and

1210 and hard baseline Pancam images from sols 953 and 1204. (Li et al., 2007d). An integrated DTM and orthophoto were generated in MarsMapper using 3D points from all these positions. Figure 3.9a and 3.9b shows the Pancam image mosaics and derived orthophoto (resolution: 0.2 m/pixel). More information on the Pancam images can be found in Table 3.6.

Parameters Left Right Orbiter number TRA_000873 PSP_001414 Resolution 0.267 (m/pixel) 0.265 (m/pixel) Location centered at (2.1ºS, 5.5ºW) in (lat, lon) 90 km offset to the east Date October 3, 2006 November 14, 2006

Table 3.5 Parameters of HiRISE image covering Victoria Crater

Parameters Hard baseline Hard baseline Wide baseline (5m) Position 76EV 85HE 85HE and 85JW Sol (Mars time) Sol 953 Sols 1204 Sols 1204 and 1210 Date (Earth time) September 29, 2006 June 14, 2007 June 20, 2007 Number of images 26 stereo pairs 12 stereo pairs 12 left images (at each position)

Table 3.6 Parameters of Pancam image covering Victoria Crater 84

Sol 953 (Hard Baseline)

5 m Wide Baseline

Sol 1204 (Hard Baseline)

Sol 1210

(a) Ground image mosaic at Duck Bay

Duck Bay

North North

(b) Duck Bay orthophoto from ground (c) Orbital image image

Figure 3.9: Ground and orbital integration using image texture information

Sub PSP_005658_1760 Sub PSP_005680_1525 Sub PSP_006250_2200

Sub PSP_006271_2210 Sub1 TRA_000882_1595 Sub2 TRA_000882_1595

Figure 3.10: Other HiRISE images with crater features

3.4.2 Landmark matching between the ground and orbital data

The HiRISE images had a close-to-nadir look, while the ground Pancam camera captured images with near-horizontal looking angles. The shapes of features represented in both types of images were quite different because of the difference in their looking angles, time to be pictured, the location of the sun, and weather conditions. The image resolutions were also different. Therefore, instead of using the original Pancam images, the derived orthophoto was used as a template to be matched with the HiRISE images.

Since the 3D measurement error of Pancam stereo pairs was less than 2 m (about 2.5%) within a range of 80 m and less than 1 m (about 1.82%) within a range of 55 m (Li and

Di, 2005; Di and Li, 2007), the area beyond the 80-meter range to the camera center was

86 considered inaccurate. Therefore, only a small subset (~56 m*32 m) of the orthophoto was selected in the image processing to be matched with the HiRISE images shown in

Figure 3.9c and Figure 3.10. In addition, from these sub-Pancam images, because it was only a small part of Victoria Crater, the information about the height and radius of the crater cannot be derived. Only one channel of each HiRISE image was matched with the sub-orthophoto template in order to save some processing time.

Following the strategy of orbital and ground integration shown in Figure 3.8, both the Pancam othophoto and the HiRISE images were preprocessed to select the common interest area. Each HiRISE image was matched with the sub-orthophoto template through the following steps. First, the original sub-orthophoto was resampled to have the same resolution as the HiRISE image. Next, the matching algorithm went through the HiRISE image with a block size identical to the size of the sub-orthophoto template. The correlation coefficient (CC) (Schenk, 1999) was computed between each block and the sub-orthophoto template. Finally, the block with the maximum value for the correlation coefficient was chosen as the best match of the sub-orthophoto template for each of the

HiRISE image with crater features. The matching results and maximum CC values (range from 0.37 to 0.627) of each matching test were represented in Figures 3.11 and 3.12.

Among all seven matches from different HiRISE images, the one with the largest CC value (0.627) was considered as the best match of the sub-orthophoto template at the

Duck Bay area.

(a) Original sub-image from ground (b) Best matched sub-image from HiRISE orthophoto CC: 0.627

Figure 3.11: The best matching result

Sub PSP_005658_1760 Sub PSP_005680_1525 Sub PSP_006250_2200

CC: 0.385 CC: 0.434 CC: 0.413 Sub PSP_006271_2210 Sub1 TRA_000882_1595 Sub2 TRA_000882_1595

CC: 0.37 CC:0.398 CC: 0.438

Figure 3.12: Matched results and CC value from reference HiRISE image groups

From the visual checking, Figure 3.11b is the correct match of Figure 3.11a. In order to further prove the chosen sub with the maximum CC value is the best match of the sub-orthophoto template. Both image subs in Figure 3.11 were further processed to extract more identical features. A Canny edge detector was applied to both images to extract some edge features as shown in Figure 3.13. In total, there were 1536 edge points

88 extracted from the ground sub-orthophoto and 1312 edge points from the matched

HiRISE sub-image.

In order to compare the similarity between these two groups of points, their patterns are compared using a concept of the world view vector of a point proposed by

Murtagh (1992). Given two sets of points, A and B, in order to map A and B, a world view vector is first built for each point i of the two sets. This vector includes the polar coordinates (r, tetha) of the other n-1 points of the set in respect to point i. Next, each point i in A is compared with each in B on the world vector. The matching pair has the smallest difference in both x and y coordinates, polar radius, and angles. A translation vector is calculated based on the coordinate difference in both the x and y directions between the matching points in A and B. This method was first developed to match star patterns with a limited number of points. It considers both the magnitudinal similarity and the directional similarity. The approach is “unaffected by translation, rotation, rescaling, random perturbations, and some random additions and deletions of coordinate couples in one list relative to another” (Murtagh, 1992).

In my application, the processing was simplified by setting the comparison in two steps. First, the corresponding point pairs were extracted according to the difference in their image coordinates. Next, the extracted sub-data sets were further compared by their world view vectors. These two steps will be explained in more detail in the following section.

The similarity was first compared between the points set A, extracted from the ground sub-orthophoto, and the points set B, extracted from the HiRISE sub-image. Each point in A was compared with each point in B. The points pair far away from each other

89 was filtered out by a distance threshold, which was set to four different levels from three pixels to zero distance. The results of paired points are plotted in different colors with respect to the four levels of distance thresholds as shown in Figure 3.13. The red points are those pairs of points which have no difference in their image coordinates. The green points are those pairs of points which have no more than one pixel’s difference in their image coordinates. The blue pairs have a difference more than one but less than two pixels. The white pairs have a difference larger than two but less than three pixels.

(a) from ground sub (b) from matched HiRISE sub image

Extracted edge map using Canny edge detector

Matched edge points extracted by different distance thresholds (red: 0, green: 1, blue: 2, white: 3 pixels)

Figure 3.13: Extracted edge maps under different thresholds

Statistical results of the extracted point pairs are also listed in Table 3.7 and shown in Figure 3.14. The table shows about 64.5% of the points from set A located

90 within three pixels’ range to the corresponding points, which takes about 75.5% of the number of points from set B.

Level Threshold Number of Rate1 (ground sub- Rate2 (matched (Pixels) grouped pairs orthophoto) HiRISE sub-image) 0 0 116 7.6% 8.8% 1 1 310 20.2% 23.6% 2 2 437 28.5% 33.3% 3 3 128 8.3% 9.8% Total 991 64.5% 75.5%

Table 3.7 Statistical results of the extracted point pairs from the matched subs

Histogram of points extracted under different distance Histogram of points extracted under different distance thresholds thresholds

1200 80.0% Ground 70.0% HiRISE 1000

60.0%

800 50.0%

600 40.0%

Numbers 30.0% 400 Percentage

20.0%

200 10.0%

0 0.0% 0123 0123 Distance threshold (Pixel) Distance threshold (Pixel)

Figure 3.14: Histogram of points extracted under different distance thresholds

The above extracted subsets (991 feature points) from A and B have similarity in their image coordinates. Both sub-sets were divided into four sub-groups according to the different distance threshold. In addition to the similar positions, the world view vectors in each pair of groups were further compared in order to match the pattern. Since each group linked the pairs of points based on the smallest difference in the image coordinates, the world view vector only includes the polar coordinates (r, tetha) of the other n-1 points of the set in respect to the first point. A histogram of the difference in the radius and polar 91 angle are shown in Figure 3.15 for these four groups. From the comparison, we see most of the pairs have a difference of less than half of a pixel in the radius and half a degree in the polar angle. These pairs have matched patterns. The translation from A to B is calculated based on the matched pattern.

92 Histogram of Difference in Radius Histogram of Difference in Polar Angle 120 120

100 100

80 80

60 60 Numbers Numbers

40 40

20 20

0 0 0 2 4 6 8 10 0 2 4 6 8 10 Difference in Radius (pixel) Difference in Polar Angle (deg) Group1: Edge points with no difference in the image coordinates

Histogram of Difference in Radius Histogram of Difference in Polar Angle 160 250

140 200 120

100 150

80 Numbers Numbers 100 60

40 50

0 0 0 2 4 6 8 10 0 2 4 6 8 10 Difference in Radius (pixel) Difference in Polar Angle (deg) Group2: Edge points with a difference of (0,1] pixels in the image coordinates Histogram of Difference in Radius Histogram of Difference in Polar Angle 300 250

250 200

200 150

150 Numbers Numbers 100 100

50 50

0 0 0 2 4 6 8 10 0 2 4 6 8 10 Difference in Radius (pixel) Difference in Polar Angle (deg) Group3: Edge points with a difference of (1,2] pixels in the image coordinates

Histogram of Difference in Radius Histogram of Difference in Polar Angle 90 60

80 50 70

60 40

50 30 40 Numbers Numbers

30 20

20 10 10

0 0 0 2 4 6 8 10 0 2 4 6 8 10 Difference in Radius (pixel) Difference in Polar Angle (deg) Group4: Edge points with a difference of (2,3] pixels in the image coordinates

Figure 3.15: Histogram of difference comparison in radius and polar angles for two sets of points extracted under different distance thresholds

93 All this information proves that Figure 3.11b is the correct match of Figure 3.11a.

From both images, feature points can be matched by comparing world view vectors. The matched orbital and ground feature points can serve as tie points for BA. The coordinated orbital and ground observations will also be used for ground truth which can be employed to calibrate and verify, for instance, the accuracy of the Mars global reference system, orbital sensor pointing accuracy, and scales of large topographic features observed from orbit and on the ground. This investigation will contribute significantly to the design and implementation of future orbital and landed Mars missions.

In the above example, only part of a crater is matched. It is possible to have some craters which match with part of the crater template but do not match with the other part.

This kind of situation is not considered in this dissertation. Additionally, the algorithm developed for the registration of orbital and ground in this dissertation is not fully automatic. There are difficulties with setting the adaptive thresholds during the matching strategies. However, better algorithms are expected in the future to improve the process.

This chapter develops a new technology to match landmarks from cross sites based on pattern matching and modeling matching. A preliminary experiment was also carried out to integrate the orbital and ground data. Although ground and orbital data integration is very important, orbital mapping cannot replace the role of ground mapping when considering the resolution, speed, and precision.

By landmark matching, tie points are selected to create a geometric configuration of the images that strengthens the geometry of the image network of surface-based rover images. Landmark registration helps to solve the conflicts and inconsistencies in sensor locations, image scales, and object correspondences inherent in various data sources. The

94 precision of those tie points affects the localization accuracy of the rover. In order to remove the mismatches in cross-site rock matching, fault detection is very important.

This will be introduced and discussed in the next chapter.

CHAPTER 4

LOCALIZATION ERROR ANALYSIS AND FAULT DETECTION

The previous two chapters introduced the methods of landmark modeling and matching in detail. Both techniques are important for the automatic selection of cross-site

tie points for bundle adjustment, which contributes to the automation of rover

localization. This chapter will focus on the analysis of localization error and fault

detection in order to get high quality cross-site tie points. First, details of camera

topographic mapping capability and the configuration of tie points will be analyzed.

Following that, it will describe how to select good cross-site tie points for rover

localization. Also included is a discussion of fault detection strategies for removing the

mismatches and pairs with bad configurations of tie points in the process of the cross-site

tie point selection.

4.1 Localization error analysis

After automatic landmark modeling and matching, the next step is to select the

high quality cross-site tie points automatically for BA operation. The final localization

accuracy greatly depends on two important factors: topographic mapping capabilities of different cameras and the configuration of tie points, which will be analyzed in detail in the following two subsections.

96 4.1.1 Analysis of rover localization accuracy

Mapping and localization error accumulates as the rover moves along its traverse.

Di and Li (2007) performed a detailed quantitative analysis of the topographic mapping capabilities of the MER Pancam and Navcam images. In their comprehensive study, they present their analysis in a systematic way: from stereo pairs to a single panorama, then to multiple panoramas along the rover traverse, and finally to wide-baseline images. The mapping capability of multi-site panoramas is determined by two factors: the mapping capability of single-site panoramas and the rover localization accuracy at each site (Di and Li, 2007). The mapping capability of a single-site panorama depends on both the measurement accuracy of stereo pairs and the accuracy and consistency of the pointing information of the adjacent stereos. The rover localization error depends on the traverse length, number of tie points and their distribution, and the parameters of the camera system, such as the focal length and stereo base (Li and Di, 2005; Di and Li, 2007).

Analysis of position error and rover localization across two sites will be briefly introduced in the following sub-sections.

(1) Position error

In a local coordinate system, X and Y are the axes pointing to the north and east directions respectively. The (X, Y) position can be calculated from the range (from an object point to the camera) r and the azimuth angle θ through Equation 4.1.

X = r cosθ

Y = r sinθ (4.1)

97 Position error is depicted through a co-variance matrix, as shown in Equation 4.2.

It shows position error is mainly affected by the range measurement error (σ r ) and

azimuth measurement error (σ θ ).

⎡ 2 ⎤ ⎡ 2 2 2 2 2 2 2 2 2 ⎤ σ X σ XY (cosθ ) σ r + r (sinθ ) σ θ (sinθ cosθ )σ r − r (sinθ cosθ ) σ θ ⎢ 2 ⎥ = ⎢ 2 2 2 2 2 2 2 2 2 ⎥ ⎣σ XY σ Y ⎦ ⎣(sinθ cosθ )σ r − r (sinθ cosθ ) σ θ (sinθ ) σ r + r (cosθ ) σ θ ⎦ (4.2)

Range measurement error is calculated through Equation 4.3. In this equation, b is

the stereo base, while f is the focal length. In addition,σ p is the measurement error of

parallax (i.e. correlation / matching error). σ p is assumed to be 0.5 pixel. In the research

that Li and Di conducted, the expected range measurement error is reasonable, at less

than 2 m (about 5.26%) within a range of 38 m and less than 1 m (about 3.7%) within a

range of 27 m for Navcam.

r 2 σ = σ r bf p (4.3)

Azimuth measurement error is supposed to be one pixel based on the MER

observation in which the azimuthal inconsistency between adjacent stereo pairs (caused

by telemetry error and other errors such as camera calibration error) is about one pixel

(Di et al., 2004; Li and Di, 2005; Di and Li, 2007).

(2) Rover localization cross two sites

In the MER mission operations, the rover position is refined by using bundle adjustment with the coordinates of tie points (landmarks) as observations. One landmark

is usually not sufficient for rover localization. Two or more landmarks between two sites

are preferred in order to improve localization accuracy. A conformal transformation

98 (scale, rotation and transformation, Equation 4.4) is used to adjust the position of adjacent

T T site, which is adjusted in object space directly. In this equation, (x1, y1) and (x2, y2) are the ground coordinates of the tie points measured from the current site and the adjacent site, respectively. Elevation z is ignored, for the consideration of flat terrain during our experiment. Parameter s is the scale factor, β is the rotation angle, and (a, b, c, d)T are the

unknown transformation parameters to be determined.

⎡a⎤ x cos β − sin β x c ax − by + c x − y 1 0 ⎢b⎥ ⎡ 1 ⎤ ⎡ ⎤ ⎡ 2 ⎤ ⎡ ⎤ ⎡ 2 2 ⎤ ⎡ 2 2 ⎤ ⎢ ⎥ ⎢ ⎥ = s⎢ ⎥ ⎢ ⎥ + ⎢ ⎥ = ⎢ ⎥ = ⎢ ⎥ ⎣y1 ⎦ ⎣sin β cos β ⎦ ⎣y2 ⎦ ⎣d⎦ ⎣bx2 + ay2 + d⎦ ⎣y2 x2 0 1 ⎦ ⎢c⎥ (4.4) ⎢ ⎥ ⎣d⎦

T T Because all the observations (x1, y1) and (x2, y2) have errors, a general

linearized model as shown in Equation 4.5 is used to solve the transformation parameters.

AV+B Δ = L (4.5)

⎡1 0 −1 0⎤ Where A = ⎢ ⎥ ⎣0 1 0 −1⎦

⎡x2 − y2 1 0⎤ B = ⎢ ⎥ ⎣y2 x2 0 1⎦

V is the vector of coordinates’ corrections of the observations

T Δ = (a b c d) , it is the unknown vector

The covariance matrix of the transformation parameters is deducted using the

error propagation principle for least squares adjustment (Equation 4.6). More detail can

be found in Li and Di (2005).

−1 = BT AΣ −1 AT B (4.6) ∑ΔΔ []()

Where Σ is the covariance matrix calculated from Equation 4.2.

99 Assuming the original position of the adjacent site is (xadjacent, yadjacent), after least squares adjustment of the transformation parameters, the new position of the adjacent site is calculated as (Xadjacent, Yadjacent) using Equation 4.7. Therefore, the covariance matrix at the adjacent site is represented in Equation 4.8. With this equation, the rover localization accuracy can be calculated at each site after the least squares adjustment.

⎡a⎤ ⎡a⎤ x − y 1 0 ⎢ ⎥ ⎢ ⎥ ⎡X adjacent ⎤ ⎡ adjacent adjacent ⎤ b b (4.7) = ⎢ ⎥ ⎢ ⎥ = E ⎢ ⎥ ⎢ Y ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ adjacent ⎦ ⎣⎢yadjacent xadjacent 0 1 ⎦⎥ c c ⎢ ⎥ ⎢ ⎥ ⎣d⎦ ⎣d⎦

= E E T ∑∑adjacent ΔΔ (4.8)

After the rover position error is derived, relative error (RE) is defined in Equation

4.9 as a ratio of the position error at the end of the traverse (adjacent site) to the traverse length (distance between the current site and adjacent site). RE is an important parameter for excluding those pairs which have a bad distribution of tie points and a high RE value.

Position error at the end of the traverse (adjacent site) (4.9) RE = *100% Traverse lengthcurrent site - adjacent site

Li and Di (2005) investigated the rover localization accuracy with a different number of tie points and various configurations of traverses. Some of the simulation results are shown in Table 4.1, which lists the optimal traverse parameters (convergence angle and leg length) that meet the 1% error limit with different image network configurations. They concluded that the rover can have a localization error less than 1% at a traverse of 22 to 26 m for Navcam, when six to nine tie points are well distributed in the middle of the two sites. In addition, Li and Di concluded that if the relative

100 localization accuracy of each traverse leg is better than one percent, an overall relative localization accuracy of 0.1 to 0.2 percent would be achievable for a 5 km traverse.

Image network configuration MER MSL MER No. of tie Convergence angle Navcam Navcam Pancam points and leg length Angle (°) 71 71 71 2 Length (m) 7.5 11.2 33 Angle (°) 88 to 86 90 to 86 86 4 Length (m) 19 28.6 85 Angle (°) 86 to 84 88 to 84 84 to 83 6 Length (m) 22 32.4 100 Angle (°) 79 to 64 86 to 66 64 to 62 9 Length (m) 26 38.8 118 Note: In Table 4.1, convergence angles are defined as an angle at either site between the beginning and end sights covering the set of tie points in the middle.

Table 4.1 Optimal traverse parameters estimated under various scenarios (Li and Di, 2005)

Some of the fault detection (FD) strategies applied in the selection of tie points will be based on this study. However, in their study, the design of the traverse and landmarks are assumed to be in an ideal case. Landmarks are symmetrically distributed along the traverse. The current site and adjacent site also have the same size of convergence angles. The rover localization accuracy will be decreased when the ideal conditions cannot be satisfied during the MER mission.

4.1.2 Rover localization error estimation based on different configuration of tie points

In most case, localization accuracy depends largely on the configuration of the tie points selected and the traverse length. The quality of the configuration is decided by the geometry of these tie points with respect to the camera centers. Based on the study of Li 101 and Di (2005), this research will further examine the relationship between rover localization accuracy with ideal configurations of the tie points and various traverse lengths. Next, the rover localization errors are estimated based on different configurations of tie points using Spirit Rover and field test data.

Figure 4.1 presents different numbers (2, 3, 4, 6, and 9) of tie points selected in an ideal situation. Green and red crosses in the plots are the positions of the current and adjacent sites. Blue crosses are the landmark positions selected ideally. For example, in

Figure 4.1(a), the two landmarks are at the middle of the two sites. As the convergence angle increases, the two landmarks move further away from the camera center. The distance between both landmarks also increases. In Figure 4.1(b), the three nodes of an equilateral triangle are chosen as the three landmarks. The side length of the triangle increases as the convergence angle grows. Similar, nodes of a square and a hexagon are selected as landmarks in Figure 4.1(c) and Figure 4.1(d), respectively. The side length of the square and hexagon also increases as the convergence angle grows. In Figure 4.1(e), nine landmarks are selected along the edge of a square.

102 Localization Error with Two Landmarks (Navcam) Localization Error with Three Landmarks (Navcam) 50 12 Current Site Current Site 40 Adjacent Site 10 Adjacent Site Landmark Landmark 30 8 20 6 10 4 0 Y (m) Y (m) 2 -10 0 -20 -2 -30

-40 -4

-50 -6 0 5 10 15 20 25 0 5 10 15 20 25 X (m) X (m) (a) Two landmarks (Li and Di, 2005) (b) Three landmarks Localization Error with Four Landmarks (Navcam) Localization Error with Six Landmarks (Navcam) 15 10

8 10 6

4 5 2

0 0 Y (m) Y (m) -2 -5 -4

-6 Current Site -10 Current Site -8 Adjacent Site Adjacent Site Landmark -10 Landmark -15 0 5 10 15 20 25 0 5 10 15 20 25 X (m) X (m) (c) Four landmarks (Li and Di, 2005) (d) Six landmarks

Localization Error with Nine Landmarks (Navcam) 15 Current Site Adjacent Site Landmark 10

0 Y (m)

-5

-10

-15 0 5 10 15 20 25 X (m) (e) Nine landmarks (Li and Di, 2005)

Figure 4.1: Configuration with different number of landmarks

103 Figures 4.2 to 4.5 show the relative localization error versus convergence angle for different number of tie points selected from Navcam images at various configurations of traverse length (7.5m, 15m, 20m, and 25m). The conclusions are as follows.

• Theoretically, when the traverse distance is 7.5 m, the 1% relative error limit

is reachable even when there are only two landmarks. When the traverse

distance increases to 15 m, at least three landmarks are needed to reach a

relative error around the 1% level. When the traverse distance is 20 m, the 1%

relative error limit is reachable when four or more landmarks are selected.

When the traverse distance goes to 25 m, the 1.2% relative error limit is

reached only when six or more landmarks are selected. Therefore, the more

the good quality landmarks, the better the rover localization accuracy.

• When two landmarks are selected, the 1% error limit is reachable if the

convergence angle is about 71° and the traverse length is less than 7.5 m. The

relative localization error increases as the traverse length and the convergence

angle (>100°) increase.

• When six or nine well-distributed landmarks are selected, the 1% error limit is

reachable if the traverse length is less than 25 m. The relative localization

error increases as the traverse length and the convergence angle (>80°)

increase.

• The relative localization error is proportional to the traverse length under the

condition of the same number of landmarks selected. The relative localization

error also increases if the convergence angle is too small or too big.

104 Relative Localization Error with Three Landmarks (Navcam) 0.74

0.72

0.7

0.68

0.66 Relative Error (%) Error Relative 0.64

0.62

0.6 20 40 60 80 100 120 140 160 Convergence Angle (degree) (a) Two landmarks (b) Three landmarks Relative Localization Error with Four Landmarks (Navcam) Relative Localization Error with Six Landmarks (Navcam) 0.54 0.46

0.52 0.45

0.5 0.44

0.48 0.43

0.46 0.42 Relative Error (%) Error Relative Relative Error (%) Error Relative 0.44 0.41

0.42 0.4

0.4 0.39 20 40 60 80 100 120 140 160 20 40 60 80 100 120 140 160 Convergence Angle (degree) Convergence Angle (degree) (c) Four landmarks (d) Six landmarks

Relative Localization Error with Nine Landmarks (Navcam) 0.4

0.39

0.38

0.37

0.36

0.35

0.34 Relative Error (%) Error Relative 0.33

0.32

0.31

0.3 20 40 60 80 100 120 140 160 Convergence Angle (degree) (e) Nine landmarks

Figure 4.2: Relative localization error vs. convergence angle (distance: 7.5 m)

105 Relative Localization Error with Three Landmarks (Navcam) 1.5

1.45

1.4

1.35

1.3

1.25 Relative Error (%) Error Relative

1.2

1.15

1.1 20 40 60 80 100 120 140 160 Convergence Angle (degree) (a) Two landmarks (Li and Di, 2005) (b) Three landmarks Relative Localization Error with Four Landmarks (Navcam) Relative Localization Error with Six Landmarks (Navcam) 0.96 0.9

0.94 0.88

0.92 0.86

0.9 0.84

0.88 0.82

0.86 0.8 Relative Error (%) Error Relative 0.84 (%) Error Relative 0.78

0.82 0.76

0.8 0.74

0.78 0.72 20 40 60 80 100 120 140 160 20 40 60 80 100 120 140 160 Convergence Angle (degree) Convergence Angle (degree) (c) Four landmarks (d) Six landmarks Relative Localization Error with Nine Landmarks (Navcam) 0.7

0.68

0.66

0.64

0.62 Relative Error (%) Error Relative 0.6

0.58

20 40 60 80 100 120 140 160 Convergence Angle (degree) (e) Nine landmarks

Figure 4.3: Relative localization error vs. convergence angle (distance: 15 m)

106 Relative Localization Error with Three Landmarks (Navcam) 2

1.95

1.9

1.85

1.8

1.75

1.7 Relative Error (%) Error Relative 1.65

1.6

1.55

1.5 20 40 60 80 100 120 140 160 Convergence Angle (degree) (a) Two landmarks (b) Three landmarks Relative Localization Error with Four Landmarks (Navcam) Relative Localization Error with Six Landmarks (Navcam) 1.35 1.25

1.3 1.2

1.25 1.15

1.2 1.1 1.15 Relative Error (%) Error Relative Relative Error (%) Error Relative 1.05 1.1

1.05 1

1 0.95 20 40 60 80 100 120 140 160 20 40 60 80 100 120 140 160 Convergence Angle (degree) Convergence Angle (degree) (c) Four landmarks (d) Six landmarks

Relative Localization Error with Nine Landmarks (Navcam) 0.94

0.92

0.9

0.88

0.86

0.84

Relative Error (%) Error Relative 0.82

0.8

0.78

0.76 20 40 60 80 100 120 140 160 Convergence Angle (degree) (e) Nine landmarks

Figure 4.4: Relative localization error vs. convergence angle (distance: 20 m)

107 Relative Localization Error with Three Landmarks (Navcam) 2.5

2.4

2.3

2.2

2.1 Relative Error (%) Error Relative 2

1.9

1.8 20 40 60 80 100 120 140 160 Convergence Angle (degree) (a) Two landmarks (b) Three landmarks Relative Localization Error with Four Landmarks (Navcam) Relative Localization Error with Six Landmarks (Navcam) 1.65 1.55

1.6 1.5

1.55 1.45

1.5 1.4 1.45 1.35 1.4 Relative Error (%) Error Relative Relative Error (%) Error Relative 1.3 1.35

1.3 1.25

1.25 20 40 60 80 100 120 140 160 20 40 60 80 100 120 140 160 Convergence Angle (degree) Convergence Angle (degree) (c) Four landmarks (d) Six landmarks

Relative Localization Error with Nine Landmarks (Navcam) 1.15

1.1

1.05

1 Relative Error (%) Error Relative

0.95

0.9 20 40 60 80 100 120 140 160 Convergenc e Angle (degree) (e) Nine landmarks

Figure 4.5: Relative localization error vs. convergence angle (distance: 25 m)

108 The above analysis uses the simulated landmarks to study the relationship between rover localization accuracy with different configurations of the tie points and various traverse lengths. The 1% relative error limit is theoretically reachable at the design level with two landmarks. However, in real application, it is not easy to reach because of the real distribution of tie points. The following examples will give test results using Spirit Rover and field test data based on different traverse distance and various configurations of tie points.

(1) Analysis of a traverse less than 5 m

Figure 4.6 gives the distribution of tie points for two pairs of test data at the

MER-A Spirit site. Table 4.2 also provides some parameters for both pairs. The relative error is about 1.05% and 0.49% for Pair 1 and Pair 2, respectively, because the distance between the current and adjacent sites is very close (less than 5 m) for both pairs. The number of tie points is also sufficient for the BA process.

(a) Pair 1: Sites 11460 and 11489 (b) Pair 2: Sites 11460 and 11456

Figure 4.6: Distribution of tie points

109 Parameters Pair 1 Pair 2 Sites 11460 and 11489 11460 and 11456 Traverse length (m) 4.6 3.5 Number of tie points 5 15 Relative error 1.05% 0.49%

Table 4.2 Parameters of both pairs

Assuming that there were some redundant tie points which did not contribute to the position improvement for these two pairs, two points from the original data pool were randomly chosen. Pair 1 has 10 combinations, while Pair 2 has 105 combinations. For each of the configurations with two tie points, their convergence angle and the relative localization error were calculated. The results of Pair 2 are shown in Figure 4.7. There are

40 combinations with an RE less than 5% (~0.18 m), and 65 combinations with an RE larger than 5%.

110

(a) RE < 5% (b) RE > 5%

Figure 4.7: Relative error analysis for each of the 105 combinations of Pair 2

As we have analyzed above, theoretically, two tie points should be sufficient for traverse segments which are less than 5 m, if they have a good configuration with two cameras. However, all the REs are greater than 1% for all the 105 combinations of Pair 2, although there are eight combinations which have an RE less than 2% as shown in Figure

4.8. The common factor for these eight distributions is that they have two points located at both sides of the traverse. The convergence angle is more than 70°. This experiment concluded that small convergence angles (e.g. <50°) always lead to big relative errors.

Another eight examples of bad distributions with big REs (e.g. >20%) are shown in

Figure 4.9. All these distributions have two landmarks closely chosen at one direction of

111 the traverse. The convergence angles are less than 50°. The conclusion regarding the results from Pair 1 is similar to that from Pair 2.

112

(a) RE: 1.3% (b) RE: 1.9%

(e) RE: 1.4% (f) RE: 1.8%

(g) RE: 1.4% (h) RE: 1.9%

Figure 4.8: Small REs at Sites 11460 and 11456 with different distribution of two tie points (good geometry)

113

(a) RE: 49.1% (b) RE: 42.8%

(e) RE: 26.8% (f) RE: 23.6%

(g) RE: 22.6% (h) RE: 20.5%

Figure 4.9: Large REs at Sites 11460 and 11456 with different distribution of two tie points (bad geometry) 114 Similarly, a configuration of three tie points and four tie points can be designed.

Pair 2 has 455 different combinations for configurations of three tie points and 1365 different combinations for configurations of four tie points. Table 4.3 and Figure 4.10 give the results comparison of both configurations. From the results, we see 2% RE (7 cm) is achievable for both three tie points and four tie points configurations. The RE is more stable when four tie points are selected which have RE less than 5% (17.5 cm) in

97.9% of the combinations. In order to have less RE, it is better to select tie points which have a convergence angle of more than 100°.

Configuration of 3 tie points Configuration of 4 tie points Category Number percentage Number percentage [0, 1%] 3 0.7% 47 3.4% (1%, 2%] 140 30.8% 818 59.9% (2%, 3%] 112 24.6% 299 21.9% (3%, 4%] 71 15.6% 114 8.4% (4%, 5%] 50 11.0% 59 4.3% RE<5% <5% total 376 82.6% 1337 97.9% RE>5% 79 17.4% 29 2.1% Total 455 100% 1365 100%

Table 4.3 Results comparison of different configuration of tie points at Pair 2 (Sites 11460 and 11456)

115

(a) Configuration of three tie points (b) Configuration of four tie points

Figure 4.10: Relative error analysis for configuration of 4 tie points of Pair 2

Based on the above analysis performed for the case of a traverse (<5 m) with two, three, and four landmarks, we conclude that two tie points are not enough to achieve an

RE less than 2%. Three tie points are needed in most of the cases.

(2) Analysis of a traverse about 10 m

The relative error at Sites 11442 and 11444 is about 0.8% when seven tie points are selected. The distance between these two sites is about 11 m. In total, there are 35 combinations when selecting either three or four tie points from the original seven tie points. RE values are calculated for different scenarios, and the histograms of RE values are compared for both configurations. The results are shown in Table 4.4 and Figure 4.11.

The conclusion is made that three tie points are sufficient to achieve an RE less than 2%

116 (22 cm). In the following plots of RE versus convergence angle, no specific relationship is found between these two parameters.

(a) Configuration of three tie points (b) Configuration of four tie points

Figure 4.11: Distribution of tie points at Sites 11442 and 11444

Configuration of 3 tie points Configuration of 4 tie points RE Category Number percentage Number percentage [0, 1%] 0 0% 5 14.3% (1%, 2%] 19 54.3% 27 77.1% (2%, 3%] 12 34.3% 3 8.6% (3%, 4%] 3 8.6% 0 0% (4%, 5%] 1 2.9% 0 0% RE>5% 0 0% 0 0% Total 35 100% 35 100%

Table 4.4 Results comparison of different configurations of tie points at Sites 11444 and 11442

117 (3) Analysis of a traverse about 20 m

The relative error at Sites 325 and 326 is about 0.75% when seven tie points are selected (Figure 4.12). The distance between these two sites is about 22.4 m. Like in the previous subsection, there are 35 combinations when selecting either three or four tie points from the original seven tie points. Again, RE values are calculated for different scenarios, and the histograms of RE values are compared for both configurations. The results are shown in Figure 4.13, which illustrates that the configuration of four tie points can reach an RE lower than 1%, and the configuration of three tie points can reach an RE around 1.4%. It is concluded that three tie points are sufficient to achieve an RE less than

2% (45 cm). Additionally, the following plots of RE versus convergence angle indicate that the interval of 60° to 120° for the convergence angle is a good region to select tie points.

6 2 3 5 7 1 4

Sub Pancam mosaic at Site 326

2 1 5 4 6 7 3

Sub Pancam mosaic at Site 325

Figure 4.12: Automatically selected tie points shown on the image mosaics of Sites 325 and 326 (labeled with the same identification numbers)

118

(a) Configuration of three tie points (b) Configuration of four tie points

Figure 4.13: Distribution of tie points at Sites 325 and 326

(4) Analysis of a traverse about 25 m

The relative error at Sites 309 and 310 is about 0.95% when six tie points are selected (Figure 4.14). The distance between these two sites is about 24.7 m. There are 20 or 15 different combinations when selecting three or four tie points from the original six tie points, respectively. Similarly, RE values are calculated for different scenarios, and the histograms of RE values are compared for the two configurations. The results are shown in Figure 4.15. It is conclude that three tie points are sufficient to achieve an RE less than 2% (0.5 m) when the traverse distance is about 25 m. The plots of RE versus convergence angle do not represent any specific relationship between these two parameters.

119

31 4 2 6 5

Sub Pancam mosaic at Site 309

5 6 4 3 1 2

Sub Pancam mosaic at Site 310

Figure 4.14: Automatically selected tie points shown on the image mosaics of Sites 309 and 310 (labeled with the same identification numbers)

(a) Configuration of three tie points (b) Configuration of four tie points

Figure 4.15: Distribution of tie points at Sites 309 and 310

120 Based on the above analysis of the selection of the number of tie points according to the different traverse length, it is concluded that at least three tie points are needed to achieve RE less than 2% (< 0.5 m). Therefore, an important rule to be included in the fault detection is to exclude those pairs which can not find at least three tie points.

Another important rule for fault detection is the RE values. Pairs with good geometry are required to have a small RE value (<4%). This threshold is decided with respect to different traverse distances and the accuracy required to be achieved.

Furthermore, the relationship between the image coverage and the traverse distance is studied. Initially, all of the Navcam panoramas are processed to select tie points for BA. In order to improve the possibility of successful matching, it is important to choose the right coverage of images. Therefore, 100 correct tie points from 13 pairs of sites at the MER-A site are picked. The position information of the corresponding image where each tie point lay is first collected. Next, the azimuth angle difference between these images and the rover traverse direction is calculated. A scatter plot of the azimuth angle difference versus traverse distance is shown in Figure 4.16. In both plots, the x-axis is the traverse distance between the current and adjacent sites. The y-axis is the azimuth angle difference. As traverse distance increases to 15 m, the difference in the azimuth angle decreases, which means as the rover moves farther away from the current site, it is important to select tie points in the middle of the traverse to link both sites.

121

200 150

100 100 50

0 0 -50

-100 -100 Azimuth angle difference Azimuth angle difference -150

-200 -200 5 10 15 20 25 5 10 15 20 25 Distance (m) Distance (m) (a) Current site (b) Adjacent site

Figure 4.16: Scatter plot of azimuth angle difference between the selected rover image and rover traverse direction

4.2 Fault detection (FD) for exclusion of mismatched landmarks

In the process of cross-site tie points selection, the limited available processor and memory capacity require the software to be efficient, stable, and reliable. Therefore, a FD algorithm is needed so that the software system can mitigate failures and meet the needs for long-range rover localization. Detecting the faults in this process is very difficult because the space over which possible faults can occur is very large. The occurrence of a fault might be caused in different steps by various parameters such as camera sensors

(e.g. stereo base, focal length), terrain property of the environment being pictured, point extraction algorithms, matching strategies, and the number and distribution of the tie points. There are also interactions between different parameters. Therefore, a general fault model will be analyzed in the following section. The strategy of the FD system and a case study will also be presented in detail.

122 4.2.1 Fault modeling

Figure 4.17 gives a fault modeling framework, which considers a fault and its causes and conditions, as well as its consequences. Figure 4.18 explains the fault model in more detail. In this study, all the faults occurring in the process of cross-site tie points selection can possibly lead to localization errors. These localization errors will further cause the rover to make a wrong decision. For example, the rover might not notice that there is a crater right in front of it if a 10 m error occurs. Most localization errors are caused by mismatched tie points and the limitations of the topographic mapping capabilities of the MER Pancam and Navcam images.

Consequence Cause Fault (Failures)

Symptom Condition (State Variable (State Variable Deviation) Limits)

Figure 4.17: General fault model

123

Cause: Traverse length Cause: Visibility of landmarks Traverse length Modeling error

Camera system (focal length, stereo base) Matching error Number of tie points Terrain slope Distribution of tie points (convergence Distribution of landmarks angle)

Fault: Camera topographic mapping capabilities Fault: Mismatches

Consequence: Localization Error

Figure 4.18: Detailed fault model

An analysis of the topographic mapping capabilities of the MER Pancam and

Navcam images was given in section 4.2. A suggestion as to the configuration of tie points was also provided by conducting experiments under different scenarios and setting different numbers of tie points with various traverse lengths. Additionally, if we want to improve the position accuracy of rover localization, the FD algorithm needs to exclude the mismatched tie points. Mismatched tie points are caused by longer traverse length, the visibility of landmarks, modeling error of landmarks, and the distribution of landmarks, as shown in Figure 4.18. Traverse length is defined as the distance between two sites. The visibility of landmarks can be judged by an important parameter - distance ratio. It is defined as the ratio between the distance from the overlapping area to the current site and that to the adjacent site (as shown in Figure 4.19). This ratio reflects the scale difference of a feature observed from two sites. It is sensitive to the success of tie

124 point selection and the fault detection strategy. We found most of the successful tie points have a distance ratio greater than 0.5. Other factors such as the modeling error of landmarks and distribution of landmarks should also be considered.

Rock feature If Distance 1 <= Distance 2 Distance ratio = Distance 1 / Distance2 Distance 1 Distance 2 Else Distance ratio = Distance 2 / Distance 1 Current site Adjacent site

Figure 4.19: Theory of distance ratio

4.2.2 System-level FD strategies

After fault modeling, system-level FD strategies will be introduced in this section.

Figure 4.20 gives a schematic illustration of the proposed system-level FD tool, which aims to prevent the simple faults that might lead to serious failure or unexpected mistakes in the BA process. This FD tool is developed through a hierarchical decision-making framework. The framework collects measured parameters and intermediate result reports from each module of the software, and sends them to the rule set for evaluation. The rules are used to determine whether there is a fault and export a final fault report. Table 4.5 gives more details of the FD strategies which are applied in the process of cross-site tie point selection and bundle adjustment.

125 I. Input of rover II. Intra- images and and original inter- III. Cross image stereo tie site tie orientation point point IV. Bundle parameters selection selection Adjustment

Pre-screening Measurement of Measurement of Measurement of results parameters and parameters and parameters and intermediate intermediate intermediate result reports result reports result reports Hierarchical FD Algorithm

Integrated System Fault Report

Final Decision

Figure 4.20: A system-level FD tool

126 Module FD rules Based on Program stops if the initial parameters (e.g., Routine check camera information) are not set I. Rover images and Program stops if the data (e.g. image, VO original image Routine check data) is not prepared orientation Estimation of parameters Pairs are excluded if their traverse length is optimal traverse more than 30 m parameters In stereo matching, the left to right correlation is verified by doing an inverse right to left check Vertical disparity must be less than 4 pixels II. Interest point XYZ intersection must be computable extraction and Pairs are excluded if the number of peaks Statistical analysis matching extracted at one site is less than 10 of the test data Mismatched feature points are excluded if the maximum correlation coefficient is less Experiment result than 0.85 Classification of False peak exclusion 2-D points Topographic mapping Peak is excluded if the distance to the capabilities of the camera center is more than 30 m MER Navcam Peak images selection Peak is excluded if it does not have enough Experiment result surrounding points III. Tie Peak is excluded if the distance to the fitting Experiment result point plane is less than 15 cm selection Pairs are excluded if the averaged distance Experiment result ratio is less than 0.5 for all the points Rock is excluded if distance ratio is less Statistical analysis than 0.3 of the test data Rock is excluded if it has less than 5 surface 3 points can make Rock points in the vicinity and cannot be a surface in the modeling modeled. space Rock is excluded if the modeling parameter is not reasonable (e.g. more than 1 m in Experiment result height or radius?)

continued

Table 4.5 FD strategies applied in the process of cross-site tie points selection

127 Table 4.5 continued

Multiple-match cases (two or more rocks from the current site match the III. Tie Rock same rock from the adjacent site) are point matching excluded selection Mismatches are excluded if their Experiment result local terrain does not match Estimation of optimal traverse parameters (most Pairs are excluded when the number of the traverse is more of selected tie points is less than 3 than 7 m. Therefore, only 2 tie points are not IV. Bundle enough for BA. adjustment Topographic mapping capabilities of the MER Pairs are excluded when the RE value Navcam images and is larger than the threshold. statistical analysis of the test data

Optimal traverse length and configuration of tie points (e.g. convergence angle and numbers) have been analyzed in the previous sections. This analysis is the foundation for FD. As shown in Figure 4.20, the total process includes four modules. In the first step of FD, the algorithm pre-screens the data based on the property of each site. FD will eliminate those pairs of sites having bad conditions to get the correct corresponding match of the same rock from different views. The pairs in bad conditions may have a traverse distance over 30 m for Navcam images in the MER mission, or the image may not be good enough to extract a sufficient number of rock peaks for tie point selection. In addition, the program also stops if the initial parameters or data are not prepared in

Module I.

After pre-screening, FD strategies are continued in the next three modules. Five filters are applied in the module of interest point extraction and matching (Table 4.5). For

128 example, the matched feature points should have a correlation coefficient larger than

0.85. The left to right correlation should also be verified by doing an inverse right to left check. For those filter banks, finding a proper value of different threshold is very important because values too small may result in many fault matching, while values too large may lead to a lower matching rate. However, it is difficult to set a suitable value from the first beginning because it depends on the characteristics of the datasets. Most of the thresholds chosen in the FD strategy are based on the statistic analysis of test data.

After the interest points are extracted from Module II, the next step of FD in

Module III of the cross-site tie points selection is conducted through three sub-steps: peak selection, rock modeling and rock matching. After rock peaks are extracted, a fault should be determined if the number of peaks extracted in the overlapping area is not sufficient or the number of extracted significant rocks is not sufficient. In the third module, the FD strategy will filter out the least suitable matches by modeling parameters, pattern of the landmarks and other constraints. Thus the best possible match will be the one with the minimum error, and most of the mismatches will be eliminated. Finally, if there is not a sufficient number of rocks in the final matching result, or the relative error is larger than the threshold, Module IV, bundle adjustment, should not be continued.

4.2.3 Case study

Some experiments are currently being conducted to test the FD strategy using the

MER-A Spirit data on Mars and the Silver Lake field test data in California. After applying the above FD strategies, the relative error of each bundle adjusted pair is calculated as shown in the following group of figures and Table 4.6. In the “Result” column of Table 4.6, the numbers 1, 2, and 3 represent successfully matched pairs, pairs 129 excluded by pre-screening, and pairs excluded by fault detection, respectively. From the table, we see that most of the successful pairs have relative position errors smaller than

2%, with the exception of two pairs (3.5% for Sites 11308 and 11312, and 2.2% for Sites

11304 and 11308). There are eight pairs with relative errors that are less than 1.0%. The test indicates valid results.

Number Conv1 Conv2 Distance RE RE Site1 Site2 of tie Results* (deg) (deg) (m) (%) (m) points 11304 122.3 11308 138.6 16.4 5 2.2 0.37 1 11308 90.4 11312 100.6 21.84 3 3.5 0.76 1 11316 0.0 11312 0.0 42.68 0 2 11316 95.7 11400 93.0 15.16 4 1.8 0.28 1 11400 80.5 11493 91.0 20.52 7 1.5 0.31 1 11495 284.0 11493 133.2 11.01 7 0.8 0.09 1 11495 213.2 11422 242.5 9.83 14 0.5 0.05 1 11434 37.3 11422 16.0 21.38 2 9.6 2.06 3 11434 143.1 11442 122.3 11.04 6 0.7 0.07 1 11444 268.5 11442 248.1 10.97 7 0.8 0.09 1 11444 199.7 11448 145.9 9.16 5 0.9 0.09 1 11448 79.8 11456 41.7 10.07 4 0.7 0.07 1 11460 296.5 11456 174.7 3.51 15 0.6 0.02 1 11460 129.8 11489 278.4 4.58 5 1.2 0.06 1 11489 0.0 11500 0.0 39.73 0 2 11500 0.0 11600 0.0 19.91 0 3 11700 0.0 11600 0.0 25.97 0 3 11800 0.0 11700 0.0 17.76 0 3 11800 331.9 11838 175.6 5.23 14 0.5 0.03 1 *: 1: successfully matched pairs; 2: pairs excluded by pre-screening; 3: pairs excluded by fault detection

Table 4.6 Test results at MER-A site

130

(a) Sites 11304 and 11308 (RE: 2.2%) (b) Sites 11308 and 11312 (RE: 3.5%)

(e) Sites 11493 and 11495 (RE: 0.8%) (f) Sites 11495 and 11422 (RE: 0.5%)

Conitnued

Figure 4.21: Relative localization error and distribution of tie points of each test pair at MER-A site

131 Figure 4.21 continued

(g) Sites 11434 and 11422 (RE: 9.6%) (h) Sites 11434 and 11442 (RE: 0.7%)

(i) Sites 11444 and 11442 (RE: 0.8%) (j) Sites 11444 and 11448 (RE: 0.9%)

(k) Sites 11448 and 11456 (RE: 0.7%) (l) Sites 11460 and 11456 (RE: 0.6%)

(m) Sites 11460 and 11489 (RE: 1.2%) (n) Sites 11800 and 11838 (RE: 0.5%)

132 Some examples of pairs excluded by different fault detection strategies are reported as follows. One assumption made for the cross-site tie points selection was made that the panorama taken from two sites should have sufficient overlap and that the overlapping area is not too far from the camera center. Therefore, pairs will be excluded if their traverse is more than 30 m. Figure 4.22 gives an example excluded by traverse distance. This pair (Sites 12334 and 12338) is at the MER-A Spirit site. The distance between the two sites calculated from telemetry (Table 4.7) is about 49 m. In the following plots, a green square and a red square represent the camera position at the current site (Site 12334) and adjacent site (Site 12338), respectively. Green crosses represent peaks extracted from Site 12334 and red crosses are peaks from Site 12338. As we see from the left figure, there are not many points located in the overlapping area between these two sites. We cannot extract a sufficient number of rock peaks (22 for Site

12334 and 20 for Site 12338). In order to make a quick decision and save some processing time, it is better to exclude this pair at the beginning.

Site Position Sol X_East (m) Y_North (m) Z (m) 12334 ANRY 741 3079.15 -1862.42 70.98 12338 ANW7 742 3093.42 -1909.48 70.97

Table 4.7 Telemetry information of Sites 12334 and 12338

133

(a) All the peaks extracted from the raw (b) Peaks extracted in the overlapping images at Sites 12334 and 12338 area at Sites 12334 and 12338

Figure 4.22: Exclusion of pairs with long traverse distance (>30 m)

Figure 4.23 gives a mismatched example excluded by local terrain comparison at

Sites 331 and 332 of the Silver Lake test site. The green dots in Figure 4.23a are the peaks of a matched feature automatically selected by the algorithm before fault detection.

It is easy for human eyes to identify this mismatch. However, the algorithm identifies the mismatch as the same feature because they pass both the modeling matching and pattern matching. In order to exclude this mismatch, local planes around the features are fitted and compared. The parameters of the fitting plane are acquired in the rock modeling step.

After applying the strategy of local terrain comparison, we see from Figure 4.23b that there is a big difference in the normal direction (blue lines) between the surrounding terrain of both features. That is about 21, 20, and 6 degrees of difference along with the x, y, and z axis, repectively. Therefore, the mismatches are removed. The accuracy of rock matching is thus improved.

134

Site 331 Site 332

2 2

(a) Extracted peak points shown in the images

-4.5 Site 331

-5

-5.5

-6 Site 332 Up (m) Up -6.5

-7 -454 -7.5 -380 -381 -452 -382 -383 North (m) -384 -450 East (m) (b) Terrain comparison of matched features

Figure 4.23: Exclusion of mismatched points by local terrain comparison

Figure 4.24 gives an example improved by fault detection. This pair (Sites 11456 and 11460) is at the MER-A Spirit site. The distance between the two sites is about 3.5 m. It should be easy to find correponding features. However, the old code got a mismatched rock (No. 2) at the adjacent Site 11460, although rock 1 and other features were matched correctly. The mismatched rock is located in an area of slope. This

135 mismatch is about 50 cm from the correct peak. The mismatched result is excluded and improved by a combination of different FD strategies.

Site 11460 Site 11456 2 1 Distance 0.5 m 2 1

(a) Mismatches

Site 11460 Site 11456 2 1 2 1

(b) Correct matches

Figure 4.24: Examples improved by fault detection

Figure 4.25 shows six examples excluded by RE value. The tie points are assumed to be covered by images from both sites. We assume that good tie points which connect forward looking and backward looking images of the two sites are evenly

136 distributed in the middle of the two sites. Although these pairs selected correct tie points, the distribution of these tie points is crowded. The convergence angles at both sites of each pair are all less than 40°. In addition, the traverse length of each segment is about 25 m. According to the large RE value of each pair (>4%), these pairs with bad configurations are excluded.

137

(a) Sites 421 and 422 (RE: 6.2%) (b) Sites 418 and 419 (RE: 7.5%)

(e) Sites 405 and 406 (RE: 7.8%) (f) Sites 402 and 403 (RE: 5.4%)

Figure 4.25: Excluded examples with bad configuration and high RE

138 Currently, RE value considering the configuration of the tie points is used in FD to remove the mismatched pair, or pair with high relative error. Furthermore, the quality of a single tie point (landmark, such as a rock peak) should be considered. For example, using a statistical method, the probability of getting a successful match at the adjacent site can be estimated when considering some of the factors.

The theory of fault detection was discussed in detail in this chapter. The strategy of FD was applied and verified using MER-A data and the Silver Lake field test data.

More situations can be considered and added into the strategy later.

139

CHAPTER 5

IMPLEMENTATION AND PERFORMANCE ANALYSIS

The techniques of landmark modeling, landmark matching and fault detection have been described in detail in the previous chapters. All these techniques contribute to the BA-based long-range rover localization system. This system was developed by the

Mars Technology Program (MTP) team at The Ohio State University (OSU). Its integration and optimization will be briefly covered in this chapter. Results of experiments performed using this software are given, as are conclusions and areas of future research. The software has been extensively tested using MER data acquired by the

Spirit rover and test data from Silver Lake in California. Along with the Spirit rover traverse, 19 pairs of test sites (total traverse length 318m, Figure 5.1) were processed from sols 574 to 648 at the Husband Hill summit area. Additionally, Silver Lake data was also tested with a 5.5 km traverse (186 sites of panoramic images, Figure 5.2). Test results demonstrate that the software can select cross-site tie points automatically along a long traverse up to 1.1 km. The good performance of this system shows that the software can be used in the ongoing MER mission for daily operations of the BA process.

140 5.1 Implementation

In the MER mission, starting with Sol 1 after landing, the accumulated Pancam,

Navcam, and Hazcam images were used to progressively build an image network as the

rover traverse extended itself beyond the landing site. BA was manually performed at

most of the sites to refine the rover position. In order to save time and effort spent on the

BA process, an algorithm used for the automatic selection of cross-site tie points was proposed in this research. Additionally, a software system was developed by the team for the BA-based long-range rover localization. Different modules of the software system were integrated and tested for the first delivery to JPL in August 2007. The processing speed of the system was further optimized for the second delivery. BA and VO integration were also included in the second delivery in February 2008. The system integration and optimization will be briefly introduced in the following subsections.

5.1.1 System development and integration

The BA-based long-range rover localization was developed by the MTP team at

OSU. The flowchart of this software can be found in Figure 1.1. The process includes four basic modules: (1) preparation of rover images and original image orientation parameters, (2) interest point extraction and matching, (3) tie point selection, and (4) bundle adjustment. The software modules for interest point extraction and matching, as well as BA, were taken from MarsMapper, a software system developed for JPL by the

OSU team (Li et al., 2002; 2004; Xu, 2004). All the other software programs for image dense point matching, automatic rock modeling, rock model and pattern matching, and fault detection were developed and have been continuously improved by the MTP team since 2004. All source codes are converted from MATLAB to optimized C++ programs, 141 which can be compiled and run in the Linux machine environment (2.8GHz, 2Gb RAM).

In the end, the components of interest point extraction and matching, cross-site tie point

selection, and the BA code were integrated in July 2007. The total software package has about 35,000 lines of C++ code. The first version of the integrated BA-based onboard rover localization system was delivered to JPL in August 2007. It took about 10 minutes of processing time for one traverse segment of MER-A data and about 25 minutes for

Silver Lake data.

5.1.2 Speed optimization

In order to create an efficient BA-based rover localization system onboard the rover, the speed is very important, because the limitation of the payload and power will affect the available processor and memory capacity. After the first delivery, by examining different modules of the software, we found that there were two major factors which took up most of the processing time. One was the slow performance in the process of interest point extraction and matching. The other was the inefficient time use caused by the repetitive processing of the same panorama at two consecutive segments. Efforts were made to optimize the software in order to improve the speed for the second version.

In order to improve the software, as a part of this doctoral research, I first looked into the inefficient use of time in regards to the repetitive processing of the same panorama at two consecutive segments. For a stretch of traverse including multiple sites, all sites except the first and last have both backward-looking images of the previous site and forward-looking images to the succeeding site. In the program code of version 1, each segment was processed as an independent unit, so processing was duplicated for those sites located in the middle of the rover traverse. The new code of the second 142 version eliminated the duplications by re-organizing the folders and data structure and by saving the intermediate results. The process of interest point extraction and matching was performed only once at each site. All the intermediate results were saved in files for later access when processing the next segment. As a result, the new code in the second version saved about half of the processing time for most of the segments, except the first one.

The second time consuming factor, interest point extraction and matching, took about 90% of the overall processing time. The slowness of process was caused by the convolution with a 2D Gaussian filter (17*17) applied to each image. By using the separability property of the convolution operation (Smith, 1999), I applied two 1D filters instead. For each image, I first convolved each row in the image with a separable filter

(17*1), resulting in an intermediate image. Next, I convolved each column of this intermediate image with the transposed filter (1*17). This process is identical to the direct convolution of the original image with the 2D filter kernel. The processing is much faster using the new method with two 1D filters. The time spent to extract interest points from one image (i.e., 1024*1024) is only about nine seconds when the filtering method uses two 1D convolutions, whereas the old method in the first version using the 2D convolution took about 20 seconds for each image.

Another aspect of optimization involves the improvement of the cache performance, which will also contribute to the improvement of processing speed. In the study by Lam et al. (1991), they found that blocking is an important optimization technique for effectively improving the memory hierarchies. Therefore, in the image processing module, instead of operating on entire rows or columns of an array, I used the

143 block algorithms to operate on the sub-matrices by methods of row-block partition and

column-block partition.

Both the original version of the code and the improved second version were tested

using the same data sets and the same Linux machine environment (2.8GHz, 2Gb RAM).

The data included a stretch of Spirit rover traverse of 318 m and another stretch of 205 m

at the Silver Lake test field. The computational results of both versions of the codes were

exactly the same. The speed comparison of the old version and the second version of the

code is presented in the following table. It shows that the new code is 4.5 times (2.2 times

for the first segment) faster than the old code for the MER-A data and 3.7 times (2 times

for the first segment) faster than the old code for the Silver lake data.

Time Cost The First Version The Second Version Test Data MER-A Data About 10 minutes for About 4.5 minutes for the first segment (BA only) each segment and 2.2 minutes for other segments Silver Lake Data About 25 minutes for About 12 minutes for the first segment (BA only) each segment and 6.7 minutes for other segments Silver Lake Data About 13 minutes for the first segment (VO+BA) and 7.6 minutes for other segments

Table 5.1 Comparison for speed test

5.2 Test and performance analysis

After system integration and optimization, the BA-based long-range rover

localization system was first extensively tested using the Spirit rover images. Along with the Spirit rover traverse, 19 pairs of test sites were processed from sols 574 to 648 at the

Husband Hill summit area. The total traverse length was about 318m (Figure 5.1).

144 Furthermore, a field test was conducted at Silver Lake in California from January 14 to

16, 2007. Along this 5.5 km traverse, VO images were taken continuously at a rate of 0.5 frames per second and BA panoramic images were taken at the end-of-traverse segments

(typically 20 to 30 m). The terrain features captured in the images were classified into three categories: a 205 m traverse over rocky outcrops, a 2.2 km traverse through a bushy area, and a 3.1 km traverse across the lakebed (Figure 5.2). Differential GPS (DGPS) was employed to measure the rover positions at a data rate of 2Hz, which matched the acquisition rate of VO images. We downloaded the GPS data of a continuously operating reference station in Bastow, California from California Spatial Reference Center, which served as the standard data to adjust the positions provided by receivers on the rover and base station. According to the DGPS data processing result, the accuracy (RMSE) of the reference station was 15 mm. The DGPS-determined rover positions were used as ground truth to evaluate the localization accuracy of VO and BA as well as their integration. Test results of MER-A and Silver Lake data will be presented in the following subsections.

Based on our teamwork, Dr. Di, Shaojun He, and I generated most of the results shown below.

Sol 648

Sol 574

Figure 5.1: Bundle adjusted rover traverse overlaid in MOC mosaic of the MER-A background map 145

Bushes Start/end point

Rocky outcrops

Dry lake

Figure 5.2: Traverse overlaid in satellite image at Silver Lake (base map from Google)

5.2.1 Verification using MER-A data

The test results of MER-A data sets are shown in Figure 5.1 and Table 5.2. In the figure, black dots are the rover site positions where panoramic images were taken. Green segments are the traverses out of the selected test area. On the other hand, red segments are those pairs of sites passing fault detection and finishing BA successfully. Both yellow and orange segments are those pairs of sites failing to do BA. Yellow segments are those that are excluded by pre-screening, while orange segments are those that are excluded by fault detection. The MER-A traverse was not designed for autonomous BA and resulted in the oversized yellow and orange segments of Figure 5.1 for the BA. Table 5.2 illustrates that there were 760 peaks extracted in total at site 1 and 786 at site 2, respectively. Among all the peaks, 209 (about 27% of the peaks extracted at a single site) were picked as significant peaks, and 98 of those (about 47% of the number of significant

146 peaks) were selected as cross-site tie points. In the “Result” column of this table, the numbers 1, 2, and 3 represent successfully matched pairs, pairs excluded by pre- screening, and pairs excluded by fault detection, respectively.

Number of Number of Number of Number Distance selected selected Site1 Site2 significant of tie Result* (m) peaks at peaks at peaks points Site 1 Site 2 11304 11308 16.96 57 60 17 5 1 11308 11312 22.13 23 21 12 3 1 11316 11312 42.68 0 0 0 0 2 11316 11400 14.61 33 33 13 4 1 11400 11493 21.29 48 60 21 7 1 11495 11493 11.48 86 101 11 7 1 11495 11422 9.51 85 82 27 14 1 11434 11422 21.38 14 11 4 2 3 11434 11442 11.07 22 23 11 6 1 11444 11442 11.44 65 68 20 7 1 11444 11448 9.32 46 53 18 5 1 11448 11456 10.13 25 29 10 4 1 11460 11456 3.49 58 49 17 15 1 11460 11489 4.48 83 81 9 5 1 11489 11500 39.73 0 0 0 0 2 11500 11600 19.91 15 17 0 0 3 11700 11600 25.96 12 11 0 0 3 11800 11700 17.76 44 45 0 0 3 11800 11838 5.25 44 42 19 14 1 Total 318.58 760 786 209 98 *: 1: successfully matched pairs; 2: pairs excluded by pre-screening; 3: pairs excluded by fault detection

Table 5.2 Test results at Husband Hill summit area of Spirit rover landing site

The statistics of cross-site tie points selection results for all 19 pairs of sites at the

MER-A landing sites are shown in Table 5.3. In this final analysis, 13 pairs out of 19

(68.4% of the sites, 47.4% of the total traverse distance) were successful in finding sufficient cross-site tie points for BA. Two pairs of sites (10.5% of the pairs, 25.9% of the total traverse distance) were excluded by pre-screening and another four pairs of sites

147 (21.1% of the pairs, 26.6% of the total traverse distance) were excluded by fault

detection. These results are reasonable because the MER-A traverse was not designed for

autonomous BA.

Classification Number Percentage Traverse Percentage of pairs (pairs) (m) (traverse) Excluded by pre-screening 2 10.5% 82.64 25.9% Excluded by fault detection 4 21.1% 84.78 26.6% Successful BA 13 68.4% 151.16 47.4% Total 19 100% 318.58 100%

Table 5.3 Statistics of cross-site tie points selection results for all 19 sites at MER-A site

In order to lend further insight into the BA process, I also developed an additional

MATLAB program, which shows the telemetry and bundle-adjusted rover traverse segments for monitoring the progress. Figure 5.3 illustrates the performance monitoring

results and the rover traverse map of the MER-A site. Note that the differences are based

on the comparison of BA and telemetry positions. In this plot, blue solid segments are

rover traverses based on telemetry data; red solid segments are those pairs of sites passing

fault detection and finishing BA successfully; red dash segments are those pairs failing to

perform BA (due to being excluded either by pre-screening or fault detection). Test

results using MER data demonstrate that the proposed automatic method is effective for

the selection of cross-site tie points for medium-range traverse segments up to 26 m.

148

Figure 5.3: MER-A rover traverse map at the Husband Hill summit area (MATLAB screen)

Additionally, at the Home Plate area of the MER-A landing site, the MER operation team at OSU has continued (since August 2007) to use the software to perform autonomous BA for the Spirit rover localization to support MER operations. Results are shown in the following Figure 5.4. The software can select cross-site tie points automatically at 71% of the total 38 segments and can correct the position by 11.03 m out of a traverse of 270.92 m. For the other 29% of the segments, although the software did not pick enough cross-site tie points due to a lack of significant rocks or due to long traverse (lengths in excess of 30 m), the software still helped the operator to rapidly select cross-site tie points of a segment within 30 m. Whereas the software-assisted selection was possible within a few minutes, in the past it took from tens of minutes to hours to

149 manually select only one cross-site tie point. This demonstrates that the software is being effectively used in the ongoing MER mission for daily operations, saving time and manual effort spent on the BA process.

Figure 5.4: MER-A rover traverse map at Home Plate area (MATLAB screen)

150 5.2.2 Verification using filed test data at Silver Lake, California

As shown in Figure 5.2, the traverse at Silver Lake, CA is about 5.5 km. The features of the terrain captured in the images fall into three categories: rocky outcrops, bushes, and lakebeds. The stretch of rocky outcrops is about 205 m and is covered with

14 sites of panoramic images. This part of the data is highly suitable for testing our software. The bushy area has 80 sites of panoramic images along a traverse of 2.2 km.

Although the shapes of bushes are different from rocks, which are the main features on the Martian surface, our software successfully selected correct cross-site tie points 56% of the time in the bushy area. The remaining images were obtained mainly on the dry lakebed. The lakebed cracks can be used for image matching in VO. However, it is impossible to pick cross-site tie points from these cracks for BA.

In order to evaluate the performances of BA and the integration of BA and VO, we tested our software in two different ways: (1) without VO data and (2) with the integration of BA and VO data. Before the VO processing result was provided, the software was tested with only the panoramic data. The positions of the panoramic images were initially obtained by DGPS and added with noise that was 10% of the distance between two consecutive panorama sites. The percentage was set to be 10% in order to make the positioning error similar to the maximum positioning error caused by the wheel odometry. After the cross-site tie points were selected, the image position and attitude, both with error, were refined by BA.

This test was conducted in the area of rocky outcrops with 13 segments. The test results for the 205 m path through the rocky outcrop area are shown in Figure 5.5.

Among these 13 segments, the software was capable of automatically achieving correct

151 cross-site tie points within 11 segments. One segment of 31 m was excluded by pre- screening and another was excluded by fault detection. The success rate is 84.6% (91.7% after pre-screening) of the pairs and 81.1% (94.5% after pre-screening) of the total traverse distance. The test data from the rocky outcrop area of Silver Lake shows a more successful rate than the MER-A data in terms of both the number of pairs and total traverse distance covered (Table 5.4), as the Silver Lake traverse was designed for this field test.

Classification Number of Percentage Traverse Percentage pairs (pairs) (m) (traverse) Excluded by pre-screening 1 7.7% 30.49 14.8% Excluded by fault detection 1 7.7% 9.66 4.7% 84.6% (91.7% 81.1% (94.5% Successful BA 11 after pre- 166.69 after pre- screening) screening) Total 13 100% 205.41 100%

Table 5.4 Statistics of test results for rocky outcrop area at Silver Lake

152

Figure 5.5: Silver Lake rover traverse map at rocky outcrop area (MATLAB screen)

Furthermore, BA and VO were integrated to improve the localization accuracy.

When the VO test results were received from JPL, we integrated into BA both the result of the cross-site tie points selection and the VO result, to refine the position and attitude for sequential images used in VO and panoramic images used in cross-site tie points selection. In this test, the positions of the panoramic images were generated from the VO data. The VO positions were used as the initial positions in the test. The integrated software with both VO and BA was tested in the rocky outcrop area and the bushy area.

Figure 5.6 shows the test results at the rocky outcrop area. The integrated BA and VO methods significantly improved the results when compared with the initial VO result. The relative accuracy reduced the error rate from 27.1% to 3.9%. This indicates that the BA and VO integration improved the image network geometry and provided better

153 localization accuracy than the VO-only method. For the rocky outcrop area, we achieved

the same successful rate as that of the result without VO data. Note that the last four

segments in Figure 5.5 are not shown in Figure 5.6, because the VO data for these four

segments are separated from the other nine segments.

Figure 5.6: Integration result in rocky outcrop area (MATLAB screen)

The software was also tested using the integrated VO and BA method in the bushy area with a traverse of 2.2 km. There were 81 pairs in total. Among the 81 pairs, 10 consecutive pairs (about 0.7 km) each had a traverse of about 50 m, which was far beyond the capability of our software to extract reliable features. The fault detection module excluded these 10 pairs automatically. There were 71 pairs left for the rest of the

1.6km traverse. These were obtained during two different time slots in the morning and afternoon of January 15, 2008. We did not include the 10 excluded pairs in our statistical

154 analysis shown in Table 5.5. These results were not as good as the results in the rocky

outcrop area, because our software was designed for rocky, not bushy, environments.

Although the success rate was lower than that of the rocky outcrop area, the software still achieved a much better localization result in the bushy area than did the method of exclusive VO use. Figure 5.7 gives the results for the traverse on the morning of January

15, 2008. The relative error of positioning was reduced from 19.7% to 4.1%.

Classification Number Percentage Traverse Percentage of pairs (pairs) (m) (traverse) Excluded by pre- 3 4.2% 90.69 5.5% screening Excluded by fault 12 16.9% 281.19 17.1% detection (FD) Matched Successful 40 56.3% (58.9% 894 54.3% (57.5% (56 pairs, (judged from after pre- after pre- judged ground truth) screening) screening) by Mismatched 16 22.5% (23.5% 380.65 23.1% (24.5% Software) (judged from after pre- after pre- ground truth) screening) screening) Total 71 100% 1646.53 100%

Table 5.5 Statistics of test results in bush area

155 Rover Traverse Map

0 S301 Traverse based on VO S302 OSU Bundle Adjusted Traverse S303 S304 Segments without BA Performed -100 S305 Ground Truth from GPS S306 S307 S308 S309 S310 -200 S311 S312 S313 S314 S315 -300 S316 S317 S318 S319 S320 -400 S321 S322 S323

North S331S330S327S324 S332 S328S326S325 S333 S334 -500 S335 S336 S337 S338 S339 -600 S340 S341 S342 S343 S344 S345 -700 S346 S348S347 S349 S350 S351 -800 S352

-700 -600 -500 -400 -300 -200 -100 0 East The difference of the rover traverse in the above plot is exaggerated with a factor of 1.0!

Rover Location (x,y,z) in Meter Site Traverse from Ground Truth Traverse from VO OSU (Bundle Adjusted) Difference Accumulated Distance Relative Difference 301 (-0.01, 0.01, 0.03) (0.80, -0.30, -0.10) (-0.00, 0.01, 0.03) 0.86 0.00 0.00 0.0% 0.0% 302 (-16.55, -16.72, -0.23) (-16.16, -16.11, 0.13) (-16.08, -16.26, -0.77) 0.73 0.66 23.54 3.1% 2.8% 303 (-17.41, -41.87, -0.89) (-18.32, -40.73, -1.87) (-16.94, -41.40, -1.42) 1.46 0.67 48.70 3.0% 1.4% 304 (-33.78, -60.97, -1.05) (-35.64, -58.79, -2.09) (-32.86, -60.22, -1.95) 2.86 1.18 73.85 3.9% 1.6% 305 (-53.85, -86.44, -0.99) (-57.04, -82.54, -1.91) (-52.09, -85.71, -1.52) 5.04 1.91 106.28 4.7% 1.8% 306 (-63.18, -100.27, -1.00) (-67.26, -95.49, -1.94) (-60.23, -101.68, -3.05) 6.28 3.27 122.96 5.1% 2.7% 307 (-75.05, -114.93, -0.92) (-80.02, -109.14, -1.74) (-69.83, -116.75, -3.81) 7.63 5.52 141.82 5.4% 3.9% 308 (-77.65, -140.24, -1.26) (-83.97, -133.66, -2.58) (-70.32, -139.76, -4.14) 9.12 7.35 167.27 5.5% 4.4% 309 (-93.05, -160.68, -0.82) (-100.11, -153.28, -2.94) (-84.36, -160.06, -4.77) 10.23 8.71 192.86 5.3% 4.5% 310 (-113.49, -174.52, -1.15) (-121.23, -165.54, -2.99) (-103.88, -174.20, -5.25) 11.86 9.61 217.55 5.5% 4.4% 311 (-129.58, -195.82, -0.65) (-138.52, -185.20, -3.28) (-119.27, -195.49, -4.88) 13.89 10.31 244.23 5.7% 4.2% 312 (-138.69, -219.72, -0.65) (-149.42, -208.01, -3.84) (-128.03, -218.88, -6.30) 15.88 10.69 269.81 5.9% 4.0% 313 (-148.94, -242.29, -0.54) (-161.61, -229.37, -4.03) (-137.16, -241.39, -7.03) 18.09 11.82 294.60 6.1% 4.0% 314 (-171.42, -253.85, -0.17) (-184.94, -238.34, -3.16) (-159.63, -252.37, -7.41) 20.58 11.89 319.88 6.4% 3.7% 315 (-187.68, -273.22, 0.11) (-202.96, -255.57, -2.41) (-175.31, -271.82, -8.16) 23.34 12.45 345.17 6.8% 3.6% 316 (-193.40, -286.94, 0.30) (-210.18, -268.40, -2.21) (-180.45, -285.11, -6.40) 25.00 13.08 360.03 6.9% 3.6% 317 (-212.34, -303.63, 0.60) (-230.75, -282.45, -1.11) (-198.08, -302.49, -8.92) 28.06 14.30 385.27 7.3% 3.7% 318 (-225.97, -320.34, 0.88) (-246.22, -297.20, -0.25) (-211.97, -318.82, -9.20) 30.75 14.08 406.83 7.6% 3.5% 319 (-236.54, -342.86, 0.93) (-259.55, -317.67, 0.32) (-219.25, -342.21, -9.13) 34.11 17.31 431.71 7.9% 4.0% 320 (-241.74, -366.71, 1.38) (-268.02, -340.30, 0.75) (-224.48, -365.78, -9.88) 37.25 17.29 456.12 8.2% 3.8% 321 (-257.00, -387.06, 1.88) (-286.06, -357.91, 1.82) (-237.93, -387.14, -10.26) 41.16 19.08 481.56 8.5% 4.0% 322 (-271.70, -406.96, 2.43) (-303.36, -375.04, 3.11) (-252.58, -406.66, -9.66) 44.96 19.12 506.30 8.9% 3.8% 323 (-293.07, -420.91, 3.10) (-326.52, -385.07, 4.49) (-272.46, -420.84, -10.49) 49.02 20.62 531.82 9.2% 3.9% 324 (-307.03, -435.98, 3.69) (-342.59, -397.25, 5.78) (-285.57, -436.10, -11.41) 52.58 21.46 552.36 9.5% 3.9% 325 (-307.23, -446.63, 4.01) (-344.44, -407.65, 6.21) (-285.63, -446.57, -12.11) 53.89 21.60 563.02 9.6% 3.8% 326 (-330.19, -446.72, 4.71) (-366.49, -404.38, 9.10) (-307.86, -447.33, -11.30) 55.77 22.34 585.98 9.5% 3.8% 327 (-339.99, -442.29, 4.53) (-374.51, -397.04, 9.80) (-316.92, -442.68, -12.08) 56.92 23.07 596.73 9.5% 3.9% 328 (-357.17, -448.32, 5.87) (-392.53, -398.15, 13.23) (-330.81, -450.69, -12.32) 61.39 26.46 614.94 10.0% 4.3% 330 (-378.93, -438.81, 6.61) (-409.37, -381.60, 15.29) (-352.13, -441.94, -13.79) 64.80 26.98 638.69 10.1% 4.2% 331 (-401.05, -437.38, 7.71) (-428.69, -372.30, 18.48) (-373.65, -440.83, -11.66) 70.71 27.61 660.85 10.7% 4.2% 332 (-421.85, -450.94, 8.53) (-451.73, -377.71, 22.47) (-392.85, -453.55, -12.27) 79.09 29.12 685.68 11.5% 4.2% 333 (-431.89, -471.48, 9.55) (-465.78, -395.02, 25.80) (-404.21, -469.52, -13.65) 83.63 27.75 708.54 11.8% 3.9% 334 (-423.47, -489.22, 9.12) (-463.20, -414.47, 26.51) (-400.70, -485.76, -14.70) 84.65 23.03 728.18 11.6% 3.2% 335 (-409.35, -504.71, 8.92) (-454.66, -433.47, 26.99) (-385.42, -499.97, -13.67) 84.43 24.39 749.14 11.3% 3.3% 336 (-405.35, -526.25, 8.57) (-459.06, -454.90, 27.79) (-380.05, -519.63, -14.48) 89.30 26.15 771.04 11.6% 3.4% 337 (-389.41, -547.05, 7.58) (-452.73, -479.67, 27.55) (-364.44, -540.03, -14.37) 92.47 25.94 797.25 11.6% 3.3% 338 (-390.88, -571.66, 7.29) (-463.48, -502.29, 28.50) (-366.32, -563.82, -15.50) 100.41 25.78 821.91 12.2% 3.1% 339 (-404.87, -589.87, 7.73) (-484.03, -512.28, 28.82) (-378.55, -580.70, -16.76) 110.85 27.88 844.88 13.1% 3.3% 340 (-426.23, -600.07, 8.61) (-507.37, -511.78, 29.54) (-399.17, -590.79, -17.44) 119.92 28.60 868.54 13.8% 3.3% 341 (-439.50, -619.77, 9.25) (-527.42, -523.90, 30.53) (-412.35, -609.24, -18.51) 130.09 29.12 892.29 14.6% 3.3% 342 (-452.07, -641.16, 9.50) (-547.21, -538.37, 30.38) (-421.91, -625.89, -19.20) 140.06 33.81 917.10 15.3% 3.7% 343 (-468.46, -660.16, 9.94) (-569.60, -548.92, 30.21) (-439.40, -642.51, -20.47) 150.35 34.00 942.19 16.0% 3.6% 344 (-482.07, -681.15, 10.25) (-589.96, -562.77, 29.31) (-455.01, -660.80, -20.86) 160.17 33.86 967.21 16.6% 3.5% 345 (-490.47, -701.98, 10.04) (-605.79, -578.19, 28.24) (-458.79, -681.72, -19.27) 169.19 37.61 989.67 17.1% 3.8% 346 (-509.73, -716.34, 10.54) (-628.95, -583.33, 27.27) (-475.37, -696.04, -22.39) 178.61 39.90 1013.69 17.6% 3.9% 347 (-529.83, -730.69, 10.94) (-652.58, -589.43, 26.07) (-494.63, -711.06, -22.72) 187.14 40.30 1038.39 18.0% 3.9% 348 (-546.65, -749.86, 11.36) (-674.92, -601.03, 25.15) (-518.33, -718.66, -21.07) 196.47 42.14 1063.89 18.5% 4.0% 349 (-554.48, -763.25, 11.47) (-686.97, -610.45, 24.50) (-525.49, -731.33, -22.62) 202.24 43.12 1079.41 18.7% 4.0% 350 (-562.98, -774.94, 11.31) (-699.08, -617.90, 23.38) (-533.67, -742.76, -23.39) 207.80 43.52 1093.86 19.0% 4.0% 351 (-574.23, -797.31, 11.27) (-717.71, -633.92, 21.86) (-542.74, -765.66, -23.29) 217.45 44.64 1118.90 19.4% 4.0% 352 (-575.88, -819.07, 10.56) (-727.43, -653.41, 20.53) (-543.09, -785.61, -24.05) 224.52 46.85 1140.72 19.7% 4.1%

Figure 5.7: Test results in bush area on Jan. 15 morning (MATLAB screen) 156 5.3 Contributions

In this dissertation, a method of automatic selection of cross-site tie points based on landmark modeling and matching has been proposed. It is one of the key steps in automating the process of BA-based long-range rover localization. Promising results were achieved both from the Silver Lake test data and the MER-A data. The automation of BA-based rover localization saves time and effort spent on the manual process.

Additionally, this dissertation included the preliminary study of integration of orbital and ground data by landmark matching and presented some of the initial results of this integration. In addition, fault detection was shown to guarantee the removal of mismatches, elimination of outliers, and selection of well-distributed, reliable landmarks as cross-site tie points for BA, which will improve the position accuracy for rover localization.

After BA, the refined EO data and Pancam and Navcam images can be employed for the generation of mapping products of the landing site. These include panoramic image maps, DTMs, orthophotos, and a spatial GIS database of measured objects such as the traverse route, rocks, ditches, and spots where scientific samples were taken. This database can be used for a number of landing site science and engineering investigations.

All the mapping products are published through the OSU Mars WebGIS and updated daily. This website has greatly supported the tactical and strategic MER mission planning

(Li et al., 2007b).

In summation, the research conducted in this dissertation makes the following contributions:

• Automatic rock identification and rock modeling;

157 • Registration of landmarks with different shapes and orientation among cross-

site images with difference in perspective view;

• Preliminary study of the orbital and ground landmark matching;

• Automatic selection of the cross-site tie points based on landmark modeling

and matching;

• Automatic fault detection strategy applied in the process of cross-site tie

points selection to get high quality tie points and improve the reliability of

rover localization;

• Autonomous BA performed successfully over single segments that were up to

a medium range in length (up to 26 m for the MER-A data and up to 27 m for

the Silver Lake test data);

• Autonomous BA successfully carried out along a continuous traverse of 1.1

km;

• Successfully obtained a high number of site pairs through automous BA (the

success rate for the number of site pairs at the Silver Lake test site is 84.6%,

which is higher than that of the MER-A data (65%), because the MER-A

traverse was not designed for autonomous BA).

5.4 Discussion and direction for future research

Knowing the accurate current position of a rover is very important for Mars rover exploration. Although the above proposed methods were successfully applied in the real mission and test data to refine the rover position, several issues that need to be improved or explored are identified as follows: 158 • Rock models defined in this dissertation are only limited to hemispheroids,

semi-ellipsoids, cones, and tetrahedrons. These models simulate rocks

approximately. More accurate models are needed to better represent a rock.

• Rocks are the major study object in this dissertation. This research only

presents a preliminary study of crater modeling and crater matching. Future

work needs to deal with many more features like mountain ranges, dunes, and

rover tracks.

• The algorithm currently works at MER-A and Silver Lake test sites where

there is a large number of rock or bush features on the terrain. New algorithms

need to be developed to handle terrain with a lack of rock features, such as

that of the MER-B site and the lakebed area at Silver Lake.

• Currently, the image network created in the BA operation does not include the

orbital data. Since the orbital data is based on a global control network,

integrated use of the ground and orbital imagery can also improve the position

accuracy of the rover when it moves. Therefore, future work of automatic tie

point selection is expected to integrate both the orbital and ground data to

strengthen the links in the image network built. The global accuracy of the

orbital data can be used to control the precision of the local BA process.

• In the future, we will use the overlapping periods of operations of the MRO,

Mars Odyssey, and MER missions to design experiments and acquire data for

calibration and validation of topographic features on the Martian surface.

• Software processing speed is an important aspect of rapid rover localization.

Currently, the time to finish BA at the MER-A site is about 4.5 minutes for 159 the first segment (with 10 pairs of stereo images for each site) and 2.2 minutes

for other continuous segments (with a similar number of image

configurations). Further efforts are needed to develop a better algorithm to

acquire high accuracy cross-site tie points more quickly. If the speed is

improved, it will help the rover to make real- or near-real time decisions.

Furthermore, this algorithm could even be applied on Earth to autonomous

driving for highway lane-keeping in the near future.

• Finally, in areas where large amounts of slippage can occur (e.g. along steep

slopes of loose sand), precise positioning is not enough for safe navigation.

For future research, automatic analysis of terrain slope, prediction of rover

slip on Earth and on Mars, and measurement of landmark location

uncertainties could be an important research topic.

160

APPENDIX A

MATHEMATICAL MODELS FOR ROCKS

A rock can be modeled using a 3D analytical surface, such as hemispheroid, semi- ellipsoid, cone, or tetrahedron (Figure A.1).

z z

y y

h h x b x

r a (a) Hemispheroid (b) Semi-ellipsoid z h y N Site h position C x φ a = 3r r r A B

base triangle (c) Cone (d) Tetrahedron Figure A.1: Simplified rock models

161 The equations for a hemispheroid, a semi-ellipsoid, and a cone are listed as

follows. Where the parameter r is radius of the hemispheroid or radius of the bottom

circle of a cone; a and b are semi-major and semi-minor axes of the semi-ellipsoid; and h

is height in all three equations.

x 2 y 2 z 2 + + = 1 (A.1) r 2 r 2 h2

x 2 y 2 z 2 + + = 1 (A.2) a 2 b2 h2

r 2 x 2 + y 2 = ()z − h 2 (A.3) h2

No analytical equation exists for the tetrahedron model. Therefore, three parameters are used to represent a tetrahedron: height h, radius of the enclosing circle of the bottom triangle, and orientation angle φ of the bottom triangle.

The parameters of each individual rock model are estimated by a least-squares

fitting using the surface points on the rock. More details on model equations,

linearization, and least-square solution are given in the study of Li et al. (2007b).

162

BIBLIOGRAPHY

Archinal, B.A., R.L. Kirk, T.C. Duxbury, E.M. Lee, R. Sucharski, and D. Cook. 2003. Mars Digital Image Model (MDIM) 2.1 control network. ISPRS Working Group IV/9 Workshop, Advances in Planetary Mapping 2003, Houston, March 2003.

Archinal, B.A., E.M. Lee, R.L. Kirk, T.C. Duxbury, R.M. Sucharski, D.A. Cook, and J.M. Barrett. 2004. A new Mars Digital Image Model (MDIM 2.1) control network. XXth ISPRS Congress, Istanbul, Turkey, July 2004.

Agarwal, S.. 2006. Rock extraction, modeling and matching for the automation of Mars rover localization. Master Thesis, The Ohio State University, Columbus, OH.

Angelova, A., L. Matthies, D. Helmick, and P. Perona. 2006. Slip prediction using visual information. Proceedings of Robotics Science and Systems, Philadelphia, PA, August 16- 19, 2006.

Arvidson, R.E., S.W. Squyres, R.C. Anderson, J.F. Bell, D. Blaney, J. Brückner, N.A. Cabrol, W.M. Calvin, M.H. Carr, P.R. Christensen, B.C. Clark, L. Crumpler, D.J. Des Marais, P.A. de Souza, C. d'Uston, T. Economou, J. Farmer, W.H. Farrand, W. Folkner, M. Golombek, S. Gorevan, J.A. Grant, R. Greeley, J. Grotzinger, E. Guinness, B.C. Hahn, L. Haskin, K.E. Herkenhoff, J.A. Hurowitz, S. Hviid, J.R. Johnson, G. Klingelhofer, A.H. Knoll, G. Landis, C. Leff, M. Lemmon, R. Li, M.B. Madsen, M.C. Malin, S.M. McLennan, H.Y. McSween, D.W. Ming, J. Moersch, R.V. Morris, T. Parker, J.W. Rice, L. Richter, R. Rieder, D.S. Rodionov, C. Schroder, M. Sims, M. Smith, P. Smith, L.A. Soderblom, R. Sullivan, S.D. Thompson, N.J. Tosca, A. Wang, H. Wanke, J.Ward, T. Wdowiak, M. Wolff, and A. Yen. 2006. Overview of the Spirit Mars Exploration Rover mission to Gusev Crater: Landing Site to Backstay Rock in the Columbia Hills. Journal of Geophysical Research, Vol. 111, E02S01, doi: 10.1029/2005JE002499.

Barata, T., E. Ivo Alves, J. Saraiva, and P. Pina. 2004. Automatic recognition of impact craters on the surface of mars. Proceeding ICIAR, Porto, Portugal, 2004, pp. 489-496.

Barlow, N.G.. 1988. Crater size-distributions and a revised Martian relative chronology. Icarus, Vol. 75(2), pp. 285-305.

Barlow, N.G.. 2000. Updates to the “catalog of large Martian impact craters”. Lunar and Planetary Science Conference XXXI, #1475, 2000.

163 Barlow, N.G. and V.L. Sharpton. 2004. Stone, wind, and ice: a guide to Martian impact craters. Lunar and Planetary Institute. August 15, 2004. On-line. Available from Internet, http://www.lpi.usra.edu/publications/slidesets/stones/ accessed March 15, 2008.

Bell, J.F. III, S.W. Squyres, K.E. Herkenhoff, J.N. Maki, H.M. Arneson, D. Brown, S.A. Collins, A. Dingizian, S.T. Elliot, E.C. Hagerott, A.G. Hayes, M.J. Johnson, J.R. Johnson, J. Joseph, K. Kinch, M.T. Lemmon, R.V. Morris, L Scherr, M. Schwochert, M.K. Shepard, G.H. Smith, J.N. Sohl-Dickstein, R.J. Sullivan, W.T. Sullivan, M. Wadsworth. 2003. Mars Exploration Rover Athena Panoramic camera (Pancam) investigation. Journal of Geophysical Research, Planets, Vol. 108, No. E12, 8063, [doi: 10.1029/2003JE002070].

Bue B.D., and T.F. Stepinski. 2007. Machine detection of martian impact craters from digital topography data. IEEE Transactions on Geoscience and Remote Sensing. Vol. 45, pp. 265-274.

Carr, M.H. 2004. The proof is in: ancient water on Mars.” The Planetary Society. Vol. XXIV num. 3. May/June 2004. p. 6-11.

Casselman, B.. 2004. Mathematical Illustrations: A Manual of Geometry and PostScript. Cambridge University Press.

Castaño, R., R.C. Anderson, J. Fox, J. Dohm, A.F.C. Haldemann, and W. Fink. 2002. Automating shape analysis of rocks on Mars. 33rd Lunar and Planetary Science Conference, Abstract #2000, Lunar and Planetary Institute, Houston, TX, March 11-15, 2002.

Castaño, A., R.C. Anderson, R. Castano, T. Estlin, and M. Judd. 2004. Intensity-based rock detection for acquiring onboard rover science. 35th Lunar and Planetary Science Conference, Abstract #2015, League City, TX, March 15-19, 2004.

Castaño, R., M. Judd, T. Estlin, R.C. Anderson, D. Gaines, A. Castaño, B. Bornstein, T. Stough, and K. Wagstaff. 2006. Current results from a Rover Science Data Analysis System. Proceedings of the IEEE Aerospace Conference, Big Sky, Montana, 2006.

Castaño, R., T. Estlin, D. Gaines, C. Chouinard, B. Bornstein, R. C. Anderson, M. Burl, D. Thompson, and M. Judd. 2007. Onboard autonomous rover science. Proceedings of the IEEE Aerospace Conference, Big Sky, Montana, 2007.

Chen, J. and R. Patton. 1999. Robust model-based fault diagnosis for dynamic systems. Kluwer Academic Publishers, Boston.

164 Cheng, Y., A.E. Johnson, L.H. Matthies, and C.F. Olsen. 2003. Optical landmark detection for spacecraft navigation. Proceeding of the 13th AAS/AIAA Space Flight Mechanics Meeting, Ponce, Puerto Rico, 2003, pp. 1785-1803, AAS 03-224.

Chowdhury, F.N. and J.L. Aravena. 1998. A modular methodology for fast fault detection and classification in power systems. IEEE Transactions on Control Systems Technology, Vol. 6, NO. 5, September 1998, pp. 623-634.

Cozman F. and E. Krotkov. 1997. Automatic mountain detection and pose estimation for teleoperation of lunar rovers. Proceedings of IEEE Conference on Robotics and Automation, Vol. 3, 1997, pp. 2452–2457.

Di, K., F. Xu, and R. Li. 2004. Constrained bundle adjustment of panoramic stereo images for Mars landing site mapping, The 4th International Symposium on Mobile Mapping Technology (MMT2004), Kunming, China, March 29-31, 2004 (CD-ROM).

Di, K., F. Xu, J. Wang, X. Niu, C. Serafy, F. Zhou, R. Li, and L. Matthies. 2005. Surface imagery based mapping and rover localization for the 2003 Mars Exploration Rover mission. Proceedings of ASPRS 2005 Annual Conference, Baltimore, MD, March 2005 (CD-ROM).

Di, K., J. Wang, S. Agarwal, E. Brodyagina, L. Yan, R. Li, and L. H. Matthies. 2006. New photogrammetric techniques used in the 2003 Mars Exploration Rover mission. ASPRS 2006 Annual Conference, Reno, Nevada, May 1-5, 2006. (CD-ROM).

Di, K. and R. Li. 2007. Topographic mapping capability analysis of Mars Exploration Rover 2003 mission imagery. 5th International Symposium on Mobile Mapping Technology (MMT 2007), Padua, Italy, May 28-31. (CD-ROM).

Estlin, T., D. Gaines, C. Chouinard, F. Fisher, R. Castano, M. Judd, R. Anderson, and I. Nesnas. 2005. Enabling autonomous rover science through dynamic planning and scheduling. Proceedings of the IEEE Aerospace Conference, Big Sky, Montanna, March 2005.

Fox, J., R. Castaño, and R.C. Anderson. 2002. Onboard autonomous rock shape analysis for Mars rovers. Proceedings of the IEEE Aerospace Conferernce, Big Sky, Montana, 2002.

Fournier, A., D. Fussell, and L. Carpenter. 1982. Computer rendering of stochastic models. Communications of the ACM, 25(6):164–172, June 1982.

Gertler, J., 1998. Fault detection and diagnosis in engineering systems. Marcel Dekker, New York.

165 Ghosh, S., U. Rana, K.S., Rao, and K.K. Sen. 2000. GIS application for mountainous terrains: some considerations and options. ENVIS Bulletin - Himalayan Environment and Development, 8 (1): 1-8.

Goldberg, S., M. Maimone, and L. Matthies. 2002. Stereo vision and rover navigation software for planetary exploration. IEEE Aerospace Conference, Big Sky, Montana, March 2002.

Golombek, M, and J. Grant. 2001. Mars exploration rover landing site memorandum. On- line. Available from Internet, http://marsoweb.nas.nasa.gov/landingsites/mer2003/doc/pasadena_01/site_ellipse_dimen sions_10_05_01.html, accessed March 03, 2007.

Golombek, M., and T. Parker. 2004. Lander localization. MER Project Memorandum, JPL/NASA, 2004.

Golombek, M.P., L.S. Crumpler, J.A. Grant, R. Greeley, N.A. Cabrol, T.J. Parker, J.W. Rice Jr., J.G. Ward, R.E. Arvidson, J.E. Moersch, R.L. Fergason, P.R. Christensen, A. Castaño, R. Castaño, A.F.C. Haldemann, R. Li, J.F. Bell III and S.W. Squyres. 2006. Geology of the Gusev Cratered plains from the Spirit Rover traverse. Journal of Geophysical Research-Planets, Vol. 111, No. E02S07 doi:10.1029/2005JE002503.

Gor, V., R. Castano, R. Manduchi, R.C. Anderson, and E. Mjolsness. 2001. Autonomous rock detection for Mars terrain. AIAA Space 2001-Conference and Exposition, Albuquerque, NM, August 2001.

Grant, J.A., R.E. Arvidson, J.F. Bell, III, N.A. Cabrol, M.H. Carr, P. Christensen, L. Crumpler, D.J. Des Marais, B.L. Ehlmann, J. Farmer, M. Golombek, F.D. Grant, R. Greeley, K. Herkenhoff, R. Li, H.Y. McSween, D.W. Ming, J. Moersch, J. W. Rice, Jr., S. Ruff, L. Richter, S. Squyres, R. Sullivan, and C. Weitz. 2004. Surficial deposits at Gusev Crater along Spirit Rover traverses. Science, Special Issue on MER 2003 Mission, Vol. 305, No. 5685, pp. 807-810 [doi: 10.1126/science.1099849].

Grant, J.A., M.P. Golombek, A.F.C. Haldemann, L. Crumpler, R. Li, W.A. Watters. 2005. Field studies of crater gradation in Gusev Crater and Meridiani Planum using the Mars Exploration Rovers. Workshop on the role of volatile and atmospheres on martian impact craters, pp. 42-43, DocumentID: 20050201825.

Honda R., and R. Azuma. 2000. Crater extraction and classification system for lunar images. Memoirs of the Faculty of Science Kochi University, Vol. 21, pp. 13-22, 2000.

166 Jones GHS, D.J. Roddy, R.W. Henny, and J.E. Slater. 1997. Defence research establishment suffield explosion craters and planetary impact craters: morphological and structural deformation analogues. 15th International Symposium on Military Aspect of Blast and Shock, Alberta, Canada, 1997.

Kanefsky, B., N.G. Barlow, and V.C. Gulick. 2001. Can distributed volunteers accomplish massive data analysis task?. Lunar and Planetary Science XXXII, #1272, 2001.

Kim, J.R., and J-P. Muller. 2003. Impact crater detection on optical image and DEM. ISPRS WG IV/9: Extraterrestrial Mapping Workshop "Advances in Planetary Mapping 2003",http://astrogeology.usgs.gov/Projects/ISPRS/MEETINGS/Houston2003/index_hou ston.html.

Lam, M.S., E.E. Rothberg and M.E. Wolf. 1991. The cache performance and optimization of blocked algorithms. Proceedings of the Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, April, 1991.

Lerner, U., R. Parr, D. Koller, and G. Biswas. 2000. Bayesian fault detection and diagnosis in dynamic systems. Proceedings of the 17th National Conference on Artificial Intelligence, pp. 531-537, 2000

Li, R., F. Ma, F. Xu, L.H. Matthies, C.F. Olson, and R.E. Arvidson. 2002. Localization of Mars rovers using descent and surface-based image data. Journal of Geophysical Research-Planets, Vol.107, No.E11, 10.1029/2000JE001443, pp.4.1-4.8.

Li, R., K. Di, L.H. Matthies, R.E. Arvidson, W.M. Folkner, and B.A. Archinal. 2004. Rover localization and landing site mapping technology for 2003 Mars Exploration Rover mission. Photogrammetric Engineering & Remote Sensing, 70(1), pp. 77-90.

Li, R., and K. Di. 2005. Rover traverse design and error analysis. Technical report submitted to JPL Mars Technology Program, February 2005, Columbus, OH: Mapping and GIS Laboratory, CEEGS, The Ohio State University.

Li, R., S.W. Squyres, R.E. Arvidson, B.A. Archinal, J. Bell, Y. Cheng, L. Crumpler, D. J. Des Marais, K. Di, T.A. Ely, M. Golombek, E. Graat, J. Grant, J. Guinn, A. Johnson, R. Greeley, R.L. Kirk, M. Maimone, L.H. Matthies, M. Malin, T. Parker, M. Sims, L.A. Soderblom, S. Thompson, J. Wang, P. Whelley, and F. Xu. 2005. Initial results of rover localization and topographic mapping for the 2003 Mars Exploration Rover mission. Photogrammetric Engineering & Remote Sensing, 71(10):1129–1142.

167 Li, R., B.A. Archinal, R.E. Arvidson, J. Bell, P. Christensen, L. Crumpler, D.J. Des Marais, K. Di, T. Duxbury, M. Golombek, J. Grant, R. Greeley, J. Guinn, A. Johnson, R.L. Kirk, M. Maimone, L.H. Matthies, M. Malin, T. Parker, M. Sims, S. Thompson, S.W. Squyres, L.A. Soderblom. 2006a. Spirit rover localization and topographic mapping at the landing site of Guesv Crater, Mars. Journal of Geophysical Research, Planets, Vol. 111:E02S06 [doi:10.1029/2005JE002483].

Li, R., A.B. Howard, L. Matthies, and K. Di. 2006b. Incremental Bundle Adjustment Techniques for Long-range Autonomous Rover Localization. Report of monthly management review submitted to JPL Mars Technology Program, February and March, 2006, Columbus, OH: Mapping and GIS Laboratory, CEEGS, The Ohio State University.

Li, R., K. Di, A.B. Howard, L. Matthies, J. Wang, and S. Agarwal. 2007a. Rock modeling and matching for autonomous long-range Mars rover localization. Journal of Field Robotics: Special Issue on Space Robotics, Vol. 24, Issue 3, pp. 187-203, March 2007.

Li, R., K. Di, J. Wang, X. Niu, S. Agarwal, E. Brodyagina, E. Oberg, J.W. Hwangbo. 2007b. A webGIS for spatial data processing, analysis, and distribution for the MER 2003 mission. Photogrammetric Engineering & Remote Sensing, 73(6), June 2007, pp. 671-680.

Li, R., K. Di, J. Hwangbo, and Y. Chen. 2007c. Integration of orbital and ground images for enhanced topographic mapping in Mars landed missions. Annual NASA Science Technology Conference, College Park, Maryland, June 19-21. (CD-ROM).

Li, R., K. Di, B. Wu, W. Chen, M.W. Maimone, L.H. Matthies, L. Richter, R. Kirk, R. Sullivan, L. Crumpler, T. Parker, D. Des Marais, R. Arvidson, M.H. Sims, S. Squyres, and the Athena Science Team. 2007d. Over three years of rover localization and topographic mapping for MER 2003 mission. Abstract #3277 and poster, the 7th International Conference on Mars. Pasadena, CA, July 9-13, 2007, 4 p. (CD-ROM).

Li, R., R.E. Arvidson, K. Di, M. Golombek, J. Guinn, A. Johnson, M. Maimone, L.H. Matthies, M. Malin, T. Parker, S.W. Squyres, and W.A. Watters. 2007e. Opportunity rover localization and topographic mapping at the landing site of Meridiani Planum, Mars. Journal of Geophysical Research, Vol. 112: E02S90 [doi:10.1029/2006JE002776].

Li R., B. Wu, K. Di, Anelia Angelova, R.E. Arvidson, I-Chieh Lee, M. Maimone, L.H. Matthies, L. Richer, R. Sullivan, M.H. Sims, R. Greenberger, S.W. Squyres, and the Athena Science Team. 2008. Characterization of traverse slippage experienced by Spirit rover on Husband Hill at Gusev Crater. Submitted to Journal of Geophysical Research.

168 Leroy B., G. Medioni, A.E. Johnson, and L.H. Matthies. 2001. Crater detection for autonomous landing on asteroids. Image Vision Computing, Vol. 19, No. 11, pp. 787-792, Sep. 2001.

Magee, M., C.R. Chapman, S.W. Dellenback, B. Enke, W.J. Merline, M.P. Rigney. 2003. Automated identification of Martian craters using image processing. Lunar and Planetary Science Conference, Abstract #1756, Houston, TX, 17-21 March, 2003.

Maki, J.N., J.F. Bell III, K.E. Herkenhoff, S.W. Squyres, A. Kiely, M. Klimesh, M. Schwochert, T. Litwin, R. Willson, A. Johnson, M. Maimone, E. Baumgartner, A. Collins, M. Wadsworth, S.T. Elliot, A. Dingizian, D. Brown, E.C. Hagerott, L. Scherr, R. Deen, D. Alexander, and J. Lorre. 2003. Mars Exploration Rover engineering cameras. Journal of Geophysical Research, Planets, Vol.108, No.E12, 8071 [doi:10.1029/2003JE002077].

Matthies, L.H.. 1989. Dynamic stereo vision. Ph.D. dissertation, Carnegie Mellon University, Pittsburgh, PA, CMU-CS-89-195.

Matthies, L., E. Gat, R. Harrison, B. Wilcox, R. Volpe, and T. Litwin. 1995. Mars microrover navigation: performance evaluation and enhancement. Autonomous Robots, 2(4), 291-311.

Mauratte, E. 2003. Mars rover autonomous navigation. Autonomous Robots, 14(2-3), 199-208.

Maimone, M., A. Johnson, Y. Cheng, R. Willson, and L. Matthies. 2004. Autonomous navigation results from the Mars Exploration Rover (MER) mission. Proceedings of the 9th International Symposium on Experimental Robotics (ISER), Singapore, June 2004.

Maimone, M., Y. Cheng, and L. Matthies. 2007. Two years of visual odometry on the Mars Exploration Rovers. Journal of Field Robotics, 24(3), 169-186.

Michael, G.. 2003. Coordinate registration by automated crater recognition. Planetary and Space Science, Vol. 51, No. 9/10, pp. 563-569, Aug./Sep. 2003.

Murtagh, F.. 1992. A New Approach to Point Pattern Matching. Publications of the Astronomical Society of the Pacific, April, 1992, pp. 301-307.

Nesnas, I.A.D., M. Bajaracharya, R. Madison, E. Bandiari, C. Kunz, M. Deans, and M. Bualat. 2004. Visual target tracking for rover-based planetary exploration. Proceedings of the 2004 IEEE Aerospace Conference, Vol. 2, pp. 747- 761, Big Sky, Montana, 6-13 March 2004.

Olson, C.F., L.H. Matthies, M. Shoppers, and M. Maimone. 2003. Rover navigation using stereo ego-motion. Robotics and Autonomous Systems, 43(4), 215-229. 169 Parker, T., M. Malin, M. Golombek, T. Duxbury A. Johnson, J. Guinn, T. McElrath, R. Kirk, B. Archinal, L. Soderblom, R. Li, the MER Navigation Team, and the Athena Science Team. 2004. Localization, localization, localization. Paper presented at 35th Lunar and Planetary Science Conference, League City, TX, March 15–19, 2004.

Patton, R., P. Frank, and R. Clark (Editors). 1989. Fault diagnosis in dynamic systems: theory and applications. Prentice-Hall, New York.

Pau, L.. 1975. Failure diagnosis and performance monitoring. Marcel Dekker, New York.

Plesko, C.S., S.P. Brumby, E. Asphaug. 2005. A comparison of automated and manual surveys of small craters in Elysium Planitia. 36th Lunar and Planetary Science Conference, Abstract #1971, March 14-18, 2005.

Schein, J., and S. T. Bushby. 2006. A Hierarchical Rule-Based Fault Detection and Diagnostic Method for HVAC Systems. International Journal of Heating Ventilation Air Conditioning and Refrigerating Research. Vol. 12(1), pp. 111-126, 2006.

Schenk, T. 1999. Digital Photogrammetry. TerraScience, Laurelville, Ohio, 428 p.

Serafy, C.. 2004. Mars exploration rover lander localization. Master Thesis, The Ohio State University, Columbus, OH.

Shan, J., J. Yoon, D.S. Lee, R.L. Kirk, G.A. Neumann, and C.H. Acton. 2005. Photogrammetric analysis of the Mars Global Surveyor mapping data. Photogrammetric Engineering & Remote Sensing, 71(1), pp. 97-108.

Smith, S.W.. 1999. The scientist and engineer's guide to digital signal processing, the second edition. California Technical Publishing, San Diego, CA.

Spero, D.J.. 2004. A review of outdoor robotics research. Technical report MECSE-17- 2004. Department of Electrical and Computer Systems Engineering, Monash University, Melbourne, Australia, November 2004.

Stein F. and G. Medioni. 1995. Map-based localization using the panoramic horizon. IEEE Transactions on Robotics and Automation, Vol. 11, pp. 892–896, Dec. 1995.

Squyres S.W., R.E. Arvidson, and Athena Science Team. 2008. Overview of Recent Results from the Opportunity Rover at Victoria Crater. Lunar and Planetary Science Conference XXXIX, #2192, 2008.

Srinivasan, S. and L. Kanal. 1997. Qualitative landmark recognition using visual cues. Pattern Recognition Letters (18), No. 11-13, November 1997, pp. 1405-1414.

170 Szolovits, P. and S.G. Pauker. 1993. Categorical and probabilistic reasoning in medical diagnosis revisited. Artificial Intelligence, 1993.

Talluri R. and J. K. Aggarwal. 1992. Position estimation for an autonomous mobile robot in an outdoor environment. IEEE Transactions on Robotics and Automation, Vol. 8, pp. 573-584, Oct. 1992.

Thompson W.B., T.C. Henderson, T.L. Colvin, L.B. Dick, and C.M. Valiquette. 1993. Vision-based localization. Proceedings of the DARPA Image Understanding Workshop, 1993, pp. 491-498.

Thompson, D.R., T. Smith and D.Wettergreen. 2005a. Data Mining during rover traverse: from images to geologic signatures. ISAIRAS, 2005.

Thompson, D.R., S. Niekum, T. Smith, and D. Wettergreen. 2005b. Automatic detection and classification of geological features of interest. Proceedings of the IEEE Aerospace Conference, Big Sky, MT, March 2005.

Thompson, D.R. and R. Castaño, 2007. A Performance Comparison of Rock Detection Algorithms for Autonomous Planetary Geology. Proceedings of the IEEE Aerospace Conference, March, 2007.

Trebi-Ollennu, A., T. Huntsberger, Y. Cheng, E.T. Baumgartner, B. Kennedy, and P. Schenker. 2001. Design and analysis of a sun sensor for planetary rover absolute heading detection. IEEE Transactions on Robotics and Automation, Vol. 17, No. 6, December 2001, pp. 939-947.

Tzafestas, S.G. 1989. System fault diagnosis using the knowledge-based methodology. Fault diagnosis in dynamic systems theory and applications, pp. 514–543. New York: Prentice Hall.

Urbach, E.R., and T.F. Stepinski.2008. Automatic detection of subkilometer craters in high resolution images of MARS. Lunar and Planetary Science XXXIX, #2184, 2008.

Vandi V., G.Gordon, R. Simmons, and S. Thrun. 2004. Particle filters for rover fault diagnosis. IEEE Robotics & Automation Magazine special issue on Human Centered Robotics and Dependability, June 2004.

Volpe, R., T. Litwin, and L.H., Matthies. 1995. Mobile robot localization by remote viewing of a colored cylinder. Proceedings of the International Conference on Robots and Systems (IROS), Pittsburgh, PA, August 5-9, 1995.

171 Vinogradova, T., M. Burl, and E. Mjosness. 2002. Training of a crater detection algorithm for Mars crater imagery. Proceeding IEEE Aerospace Conference, 2002, Vol. 7, pp. 7-3201–7-3211.

Wettergreen, D., Cabrol, N., Teza, J., Tompkins, P., Urmson, C., Verma, V., Wagner, M.D., & Whittaker, W.L. 2005. First experiments in the robotic investigation of life in the Atacama Desert of Chile. IEEE International Conference on Robotics and Automation, Barcelona, Spain, April 2005.

Wikipedia. On-line. Available from Internet, http://en.wikipedia.org/wiki/Satellite_navigation_system, accessed March 20, 2008a.

Wikipedia. On-line. Available from Internet, http://en.wikipedia.org/wiki/Erebus_%28crater%29, accessed May 30, 2008b.

Xu, F.. 2004. Mapping and localization for extraterrestrial robotic explorations. Ph.D. Dissertation, The Ohio State University, Columbus, OH.

172