Wide Baseline Mapping for Mars Rovers

609-618_10-056.qxd 5/21/11 12:48 PM Page 609

Kaichang Di and Man Peng

Abstract investigations of distant geological features, a mapping Wide-baseline mapping technology has been applied in capability of up to hundreds of meters is desirable, making NASA’s Mars Exploration Rover (MER) mission to improve the wide-baseline mapping technology necessary. A wide- accuracy of mapping of far-range terrain from Rover stereo baseline stereo pair is formed by two or more images taken images. As a basic research topic in photogrammetry and at two separate rover positions, resulting in a “soft” baseline computer vision, it is desirable to perform a comprehensive that is much wider than the “hard” baseline designated on investigation on the accuracy and automation of wide- the camera bar (Figure 1). During Mars Exploration Rover baseline mapping. This paper presents the results of a mission operations, wide-baseline stereo images have been systematic accuracy analysis of wide-baseline mapping used to map a number of far-range major features including through theoretical derivation and Monte Carlo simulation. Husband Hill, McCool Hill, and Von Braun at the Spirit Automated bundle adjustment and 3D wide-baseline mapping landing site as well as Endurance and Victoria Craters at the techniques are developed and tested using wide-baseline Opportunity landing site (Li et al., 2005; Di and Li, 2007; images acquired by the Spirit Rover. Experimental results Chen, 2008). These wide-baselines were designed empiri- demonstrate the effectiveness of the proposed techniques and cally using the parallax equation for “normal case” stereo as verify that the mapping capability of rover stereo images can a reference and the wide-baseline mapping process involved be extended from tens of meters (hard-baseline) to hundreds quite a few manual interactions, for example, manual of meters using wide-baseline mapping techniques. selection of cross-site (wide-baseline) tie points for bundle adjustment and interactive image matching to obtain terrain details (Di and Li, 2007; Chen, 2008). Introduction Photogrammetric bundle adjustment is a key technique In an unmanned planetary rover mission, 3D mapping of the for achieving high-precision topographic products and has surrounding terrain is of fundamental importance for safe been widely used in MER mission for 3D mapping as well as rover navigation and for scientific investigation of geological rover localization (Li et al., 2004 and 2005; Di and Li, 2007; features. During the 2003 Mars Exploration Rover (MER) Di et al., 2008). Bundle adjustment of an image network mission, Navcam (navigation camera), and Pancam (panoramic formed by relevant Pancam and Navcam stereo images camera) rover images have been extensively used for topo- generates high-precision exterior orientation parameters of graphic mapping of both the Spirit and Opportunity landing the images as well as ground positions of tie points. In sites; these high-accuracy maps have greatly supported rover particular, bundle adjustment of multi-site images ensures traverse planning and scientific investigation in mission that the mapping products generated from bundle-adjusted operations (Li et al., 2005; Di et al., 2008). Both Navcam and images are of high accuracy and geometrically seamless. Pancam stereo cameras are mounted on the same camera bar Image matching is one of the core problems in computer atop a rover mast. This bar can be rotated 360° in azimuth and vision and digital photogrammetry. Traditionally, interest Ϯ90° in elevation, enabling the acquisition of full or partial point detectors such as the Harris operator (Harris and panoramas by Navcam and Pancam camera systems. Navcam Stephens, 1988) and the Förstner operator (Förstner and is an engineering stereo camera used for navigation purposes Gulch, 1987) have been used widely in stereo-image (Maki et al., 2003) while Pancam is a high-resolution, multi- matching. In the last decade, wide-baseline matching has spectral stereo panoramic imager used for scientific investiga- been an active area of research in the computer vision tion of the morphology, topography, and geology of the two community. A variety of invariant detectors and descriptors MER landing sites (Bell et al., 2003). Pancam has a longer have been proposed for wide-baseline matching, object baseline and a longer focal length than Navcam, making it recognition, and image retrieval purposes including affine more effective in mapping medium to far objects. invariant regions (Tuytelaars and Van Gool, 2000), MSER Due to the limited length of a hard-baseline, the (maximally stable extremal regions) (Matas et al., 2002), expected measurement error from hard-baseline stereo Harris-Laplace and Harris-Affine interest-point detectors images is less than 1 m within a range of 27 m for Navcam (Mikolajczyk and Schmid, 2004), SIFT (Scale Invariant Feature and 55 m for Pancam (Di and Li, 2007). This baseline Transform) (Lowe 1999 and 2004), and SURF (Speeded-Up length usually satisfies the requirements of day-to-day, Robust Features) (Bay et al., 2008). These detectors and their short-term planning. However, for long-term planning and

Photogrammetric Engineering & Remote Sensing Vol. 77, No. 6, June 2011, pp. 609–618. State Key Laboratory of Remote Sensing Science, Institute of Remote Sensing Applications, Chinese Academy Sciences, 0099-1112/11/7706–0609/$3.00/0 P. O. Box 9718, Datun Road, Chaoyang District, Beijing © 2011 American Society for Photogrammetry 100101, P.R. China ([email protected]). and Remote Sensing

PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING June 2011 609 609-618_10-056.qxd 5/21/11 12:48 PM Page 610

TABLE 1. CAMERA PARAMETERS OF MER PANCAM AND NAVCAM

Pancam Navcam

Stereo base 30 cm 20 cm Focal length 43 mm 14.67 mm Image dimension 1024 ϫ 1024 pixels 1024 ϫ 1024 pixels Pixel size 12 mm 12 mm Field of view16.8° * 16.8° 45° * 45°

geometric parameters of the MER Pancam and Navcam are listed in Table 1 (Bell et al., 2003; Maki et al., 2003). To simplify the derivation, we assumed that the wide-baseline stereo pair was acquired in a “normal case” configuration, i.e., where the two camera axes are parallel to each other Figure 1. Conceptual illustration of wide-baseline mapping. and perpendicular to the baseline (Di and Li, 2007). Since measurement error in the range direction is always larger than that in other two directions (Di and Li, 2007), in the following discussion we have used range descriptors were tested and applied mainly in structured error to represent mapping accuracy. In the “normal case” scenes of indoor or outdoor environments on Earth. Olson et stereo image configuration, the range Y can be calculated al. (2003) addressing the issues of wide-baseline stereo vision using the parallax equation: for Mars rovers, developed an algorithm consisting of the following four steps: (a) motion refinement through Förstner B Y ϭ f (1) interest-point extraction, hierarchical matching, and Leven- p berg-Marquardt optimization for motion estimation, (b) image rectification (epipolar resampling), (c) disparity calculation where f is the focal length, B is the baseline, and p is the with maximum-likelihood image matching, and (d) triangula- stereo parallax of a specific point. It is easy to understand tion for determination of the 3D positions of matched image from this equation that the mapping accuracy of hard-baseline pixels. This algorithm has been tested for wide-baseline stereo depends only on the measured parallax p (because f mapping using images of Mars-like terrain on Earth (Olson et and B are fixed for Navcam and Pancam, respectively). For al., 2007) and Mars rover images (Olson and Abi-Rached, wide-baseline stereo, the baseline is not fixed (as it is with a 2009). Invariant detectors could be applied to Mars rover hard-baseline) and therefore the baseline error will affect the images to address the challenges of wide-baseline matching. mapping accuracy. Through error-propagation derivation With the current availability of wide-baseline stereo images (Equation 1), range error can be calculated as: from the Spirit and Opportunity rovers, it would be more interesting to test the wide-baseline mapping algorithms using Y 2 Y 2 2 s2 ϭ s2 ϩ s2 (2) these actual Mars surface images. Y a B b B a Bfb p Overall, it is highly desirable to perform a systematic

study of wide-baseline mapping accuracy and optimal where sY is the standard error of range Y, sB is the standard configuration; it is also desirable to develop fully automated error of the wide-baseline, and sp is the parallax measure- techniques for wide-baseline bundle adjustment and dense ment error. In digital photogrammetry, sp is determined by matching for Mars rovers. This paper presents accuracy the accuracy of image matching. According to theoretical analysis results of wide-baseline mapping through theoreti- analysis, pixel-level image matching (correlation) can reach cal analysis and Monte Carlo simulation. An automated an accuracy of about one-third pixel (Zhang and Zhang, bundle-adjustment technique is developed, and a TIN 1997). In practice, the pixel measurement error can be controlled matching method is applied for wide-baseline superior to one-third pixel for image pairs that belong to the mapping. The techniques are tested using wide-baseline same site (hard-baseline) if a sub-pixel matching algorithm is images acquired by the Spirit Rover. applied. If the image pairs belong to different sites (wide- It should be noted that the research team and its baseline), the pixel measurement errors may be larger than affiliated institute is not part of the MER operations, and the one-third pixel due to a large difference in viewing angles. developed method has not been used in MER operations as Since there are image pairs at both one site (hard-baseline) well. The purpose of this work is to solve the basic problem and two sites in wide-baseline mapping, generally we used of wide-baseline mapping in the research field of pho- one-third pixel as the pixel measurement error in our togrammetry and computer vision and test the developed analysis to represent the overall error. methods using publicly available rover data sets. Baseline error can be depicted by rover localization with the traverse length of baseline B. During MER mission operations, Navcam images were used for localization Geometric Configuration and Accuracy Analysis for purpose at most rover locations. Therefore, in this research Wide-Baseline Mapping we have used the Navcam parameters to estimate the baseline error. For simplicity, we assumed there is only one Theoretical Accuracy Analysis landmark in the center of two sites (the two ends of the In order to investigate mapping capability under different wide-baseline). In this ideal case, rover localization can be wide-baseline configurations, we first performed a theoreti- performed simply by a translation of Site 1 with respect to cal analysis of the possible attainable accuracy. For this Site 0 based on the difference between the locations of the research, we used the Pancam parameters to analyze the landmark measured from the two sites. Thus, the wide- attainable accuracy of wide-baseline mapping; Navcam baseline can be calculated as: parameters were used only to estimate the baseline error, ϭ ϩ which is equivalent to the rover localization error. The B y0 y1 (3)

610 June 2011 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING 609-618_10-056.qxd 5/21/11 12:48 PM Page 611

ABLE PTIMAL ASELINES FOR ARGETS AT IFFERENT ISTANCES BASED where y0 and y1 are coordinates (ranges) of the landmark T 2. O B T D D with respect to Site 0 and Site 1, respectively. Through ON THEORETICAL ANALYSIS simple error propagation, the error of the baseline is the sum of the two range errors: Target 100 200 300 400 500 600 700 distance (m) s2 ϭ s2 ϩ s2 . (4) Optimal 4.39 6.21 7.60 8.78 9.82 10.76 11.62 B y0 y1 baseline (m) Range 0.299 0.847 1.556 2.395 3.348 4.401 5.546 Assuming that the landmark is in the center of these error (m) ϭ ϭ two sites (i.e., y0 y1 B/2), the range error for the Navcam hard-baseline is calculated from Equation 2 as:

2 B sp 2 s2 ϭ s2 ϭ (5) As the width of the baseline increases, the area of y0 y1 a b 4bNavcam fNavcam overlap between the wide-baseline stereo pair becomes smaller. In photogrammetric engineering, the general rule is where bNavcam is the Navcam hard-baseline and fNavcam is the Navcam focal length. Note that the first term of Equation 2 that end lap should be at least 60 percent. Through simple was not included here because the fixed Navcam hard- calculations, we can check to see if the optimal baselines baseline (with no baseline error) was used for localization. satisfy the needs of end lap. Figure 3 is an illustration of the Then, Equation 4 can be represented as: end lap of a wide-baseline stereo pair. Using similar triangles in Figure 3, we can easily obtain: B4 s2 ϭ s2. (6) Y ¢X ϩ B B 8b 2 f 2 p ϭ (8) Navcam Navcam f a 2 Substituting sB into Equation 2, the wide-baseline range error is finally represented as: and f Y2B2 Y2 2 ¢ ϭ ¢ s2 ϭ s2 ϩ s2. (7) x X (9) Y 2 2 p p Y 8bNavcam fNavcam a BfPancam b ⌬ ⌬ It should be noted that in this research, we use Pancam where a is the image size and X and x are the projections parameters to evaluate mapping accuracy, and use Navcam of the area of overlap in object space and image space, parameters to estimate the baseline error through rover respectively. Thus, the end lap is: localization. Therefore, in Equation 7, Navcam parameters ¢x Bf bNavcam and fNavcam are used in the baseline error term (the ϭ 1 Ϫ . (10) first term), while Pancam parameter fPancam is used in the a aY parallax measurement error term (the second term). Based on Equation 7, Figure 2 was generated to show For Pancam, the value of f/a is 3.499. To insure an end the range measurement errors of targets at different distances lap of 60 percent or better for Pancam, we should have: from the camera (100 to 700 m) under different wide- … Ϫ 6 baselines. It can be seen that mapping errors do not change 0.6 1 3.499B/Y 1 (11) monotonically with respect to the baselines; there exists an therefore, the valid range of B is: optimal baseline for a given distance to a target. Table 2 shows the optimal baselines for targets at different distances. 0 6 B … 0.114Y (12)

Figure 2. Range measurement error of different Pancam wide-baselines. Figure 3. End lap for wide-baseline stereo.

PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING June 2011 611 609-618_10-056.qxd 5/21/11 12:48 PM Page 612

TABLE 3. MAXIMUM BASELINE OF DIFFERENT RANGE DISTANCES Pancam hard-baseline and focal length along with the fixed relationship between the left and right Pancam cameras were Target distance (m) 100 200 300 400 500 600 700 used. As in the theoretical analysis, the pixel measurement Maximum baseline (m) 11.4 22.8 34.2 45.6 57.0 68.4 79.8 error was set to one-third pixel, and the baseline error was estimated using Equation 6. Based on experimental findings, after bundle adjustment the azimuthal inconsistency The maximum baselines for targets at different distances between the adjacent sites is usually within one pixel, the Ϫ are shown in Table 3. When comparing Tables 2 and 3, it angular error was set to tan 1(1 pixel/f) for azimuth using can be seen that the optimal baselines are always less than the known Pancam pixel size and focal length. the corresponding maximum baselines. This verifies the The steps of the Monte Carlo simulation for wide- assumption that theoretically derived optimal baselines can baseline mapping accuracy analysis are: be used in practical implementations of wide-baseline mapping. 1. Generate the exterior orientation parameters (EOPs) for a In the above theoretical analysis, pixel measurement Pancam hard-baseline stereo of Site 0 of the wide-baseline. (parallax) error and baseline error were considered for wide- 2. Suppose the wide-baseline and the hard-baseline are parallel baseline mapping accuracy analysis using the “normal case” to a site frame, whose X-axis points to local east, Z-axis stereo configuration. Results from this analysis can be used points up in the local normal direction, and Y-axis is as a general guideline for wide-baseline design. A more defined to form a right-handed system. Generate the EOPs for comprehensive analysis, one based on simulation that a Pancam hard-baseline stereo of Site 1 by adding the given wide-baseline to the X positions of the Pancam and keeping incorporates additional factors such as angular errors and all other EOPs (Y, Z, azimuth, swing, tilt) the same as those multi-baseline intersection, is described below. of Site 0. 3. Given the 3D coordinates of a target that is with a specific Monte Carlo Simulation distance to the center of the wide-baseline, calculate the In the above theoretical derivation, only a single wide- image coordinates in the wide-baseline images with baseline stereo pair was considered. The single stereo pair is collinearity equations using the EOPs generated in Steps 1 formed by linking a single image taken at the first site and 2 above (so far, all the EOPs and 3D and 2D coordinates (Site 0) with a single image taken at the second site (Site 1). are “true” values without any error involved). In practice, a wide-baseline stereo pair can be acquired at 4. Generate 10,000 Gaussian random numbers with 0 mean and a standard deviation that is equal to the standard error of both of the wide-baseline sites (Site 0 and Site 1). Thus, baseline sB as calculated in Equation 6, and add these there are three possible stereo modes in wide-baseline random errors to the X positions of the EOPs at Site 1; mapping: one single image at Site 0 and another single generate 10,000 Gaussian random numbers with 0 mean and image at Site 1 (single ϩ single), one hard-baseline stereo a standard deviation of one-third pixel, and add them to the pair at Site 0 and one single image at Site 1 (stereo ϩ x image coordinates at Site 1; generate 10,000 Gaussian single), or one hard-baseline stereo pair at Site 0 and random numbers with 0 mean and standard deviation of Ϫ another hard-baseline stereo pair at Site 1 (stereo ϩ stereo). tan 1(1 pixel/f), and add these angular errors to the azimuth These three configurations are illustrated in Figure 4. In angle of the EOPs at Site 1; now the EOPs of Site 1 have addition, in the theoretical derivation the baseline error was included baseline and angular errors, and the 2D coordinates in Site 1 have included pixel measurement errors. estimated without taking into consideration the orientation 5. Calculate the 3D coordinates (10,000 times) of the target by (azimuth, tilt, and swing) error between Site 0 and Site 1. space intersection using the EOPs and image coordinates. To incorporate these factors for a comprehensive wide- 6. Calculate the difference between the calculated and true baseline accuracy analysis, we employ the Monte Carlo 3D coordinates of the target, and obtain the RMS error from simulation method. In this simulation process, the known the 10,000 runs.

Figure 4. Stereo modes of wide-baselines: (a) “single ϩ single,” (b) “stereo ϩ single,” and (c) “stereo ϩ stereo.”

612 June 2011 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING 609-618_10-056.qxd 5/21/11 12:48 PM Page 613

The above simulation is performed for each combination are initially determined by onboard dead-reckoning, may be of baseline and target distance. A complete accuracy analysis of poor accuracy due to wheel slippage and IMU drift. To is fulfilled by using all the combinations of baseline (1 m to reach the highest theoretical accuracy in wide-baseline 25 m) and target distance (100 m to 700 m) for all the three mapping, usually a bundle adjustment is necessary to fix the stereo modes using Pancam parameters (along with Navcam relative position and orientation of the two end sites. The parameters for baseline error estimation). For stereo modes key to the success of wide-baseline bundle adjustment is to “stereo ϩ single” and “stereo ϩ stereo”, random pixel select a sufficient number of tie points to link the wide- measurement errors are also added to the x coordinates of baseline images to form an image network. The wide- the right images at Site 0. In addition, in order to verify the baseline (cross-site) tie points used for this bundle adjust- error analysis results from theoretical derivation, we simu- ment were chosen manually during MER mission operations. lated the “single ϩ single” stereo mode with pixel measure- For this paper, we have developed an algorithm for auto- ment error and baseline error without considering angular matic wide-baseline tie-point selection so that the bundle error, which is the same setting as used in the theoretical adjustment process is also automated. analysis. The simulation results were found to be the same as the results of the theoretical analysis. Automatic Wide-baseline Tie Point Selection Figures 5a, 5b, and 5c show mapping error versus the baseline under the “single ϩ single,” “stereo ϩ single,” and Tie point selection between wide-baseline images is more “stereo ϩ stereo” modes, respectively. Table 4 summarizes the challenging than tie point selection between hard-baseline optimal baselines and range errors for targets of different images. As can be seen in Figure 6, the challenge is that distances from the cameras. We can observe from the figures close-range and medium-range objects look significantly and Table 4 that: (a) there is always an optimal baseline at different in the separate wide-baseline images due to a large which the mapping error achieves the lowest value, (b) the difference in viewing angles; in addition, far-range objects farther the distance to the target, the longer the optimal baseline, lack visible texture. To tackle these challenges, we use a (c) the mapping accuracy can be improved by multiple-baseline combination of Förstner operator and SURF operator. intersection with “stereo ϩ single” and “stereo ϩ stereo” modes, Förstner operator (Förstner and Gulch, 1987) tends to and (d) as multiple-baseline intersection can improve mapping provide more unique interest points in natural outdoor accuracy with redundant observations, the optimal baseline scenes with high accuracy and reliability under conditions tends to become shorter when employing multiple-baseline of similar viewing angle and illumination. Matching of the intersection. The above results can be used in detailed designs extracted Förstner interest points is performed using cross for wide-baseline mapping. Whenever possible, multiple- correlation followed by least-squares matching. From this baseline intersection using “stereo ϩ single” or “stereo ϩ stereo” process, a sufficient number of interest points can be modes are recommended. extracted and matched for medium- and far-range terrain, with only a few interest points being matched for close- range terrain. SURF (Bay et al., 2008) is a scale- and rotation- Automated Bundle Adjustment for Wide-baseline Mapping invariant detector and descriptor, which provides similar The relative position and orientation of the wide-baseline results to SIFT though at a faster speed (http://www. images acquired at the two ends of the wide-baseline, which chrisevansdev.com/computer-vision-opensurf.html). We used

Figure 5. Range errors with different wide-baselines under the three different stereo modes: (a) “single ϩ single,” (b) “stereo ϩ single,” and (c) “stereo ϩ stereo.”

TABLE 4. OPTIMAL BASELINES AND RANGE ERRORS FOR TARGETS AT DIFFERENT DISTANCES FROM THE CAMERAS

Target distance(m) 100 200 300 400 500 600 700

SingleϩSingle Optimal baseline(m) 4 6 8 8 10 10 12 Range error(m) 0.304 0.842 1.561 2.425 3.383 4.441 5.577 StereoϩSingle Optimal baseline(m) 3 5 6 6 7 8 9 Range error(m) 0.217 0.608 1.104 1.719 2.361 3.182 3.953 StereoϩStereo Optimal baseline(m) 3 5 6 6 7 8 9 Range error(m) 0.201 0.586 1.065 1.646 2.307 3.023 3.802

PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING June 2011 613 609-618_10-056.qxd 5/21/11 12:48 PM Page 614

the SURF-64 descriptor and applied a distance-ratio measure Bolles, 1981) procedure (Kovesi, 2000). Under this iterative (the ratio of Euclidean distance of the closest neighbor to procedure, any mismatches are automatically eliminated as that of the second-closest neighbor) for matching. As a outliers. Figure 6b shows the results of feature-point result, SURF provided a sufficient number of matched points matching after outlier elimination. In this particular pair, for close- and medium-range terrain, but only a few for far- 93 mismatches were eliminated from the 222 matched range terrain. By combining the matched points from both points (Figure 6a), resulting in 129 correctly matched points the Förstner and the SURF matching processes, we were able (Figure 6b). Consequently, wide-baseline tie points were to obtain a sufficient number of matched points for close-, selected from these corrected matched points. By drawing a medium- and far-range terrain scenarios. grid (e.g., 3 ϫ 3) on the image and selecting the one tie Figure 6a shows the results of feature-point matching of point having the highest correlation coefficient (or distance a wide-baseline stereo pair taken by the Spirit rover. ratio) in each patch, we can obtain evenly distributed wide- Matched points are marked as black crosses and linked by baseline tie points that are good for the subsequent bundle white lines. Those lines that have significant differences in adjustment. orientation and length in comparison with neighboring lines We tested the automatic wide-baseline tie point selec- indicate mismatches. In order to eliminate these tion process using eight pairs of Pancam wide-baseline mismatches, we computed the fundamental matrix between images taken by the Spirit rover at sites APFI (on Sols 774 the wide-baseline image pair using a RANSAC (Fischler and and 775) and APGB (on Sols 776 to 778) for mapping McCool

Figure 6. Matched feature points (a) before, and (b) after outlier elimination.

614 June 2011 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING 609-618_10-056.qxd 5/21/11 12:48 PM Page 615

Figure 7. Automatically selected tie points overlaid on image mosaics at sites: (a) APFI, and (b) APGB.

Hill at a range of up to 400m. The length of this wide- There are six unknown EOPs for each wide-baseline baseline was about 8 m. The “stereo ϩ stereo” mode was image in the bundle adjustment. In the “stereo ϩ single” used, thus we had eight hard-baseline stereo pairs at each and “stereo ϩ stereo” modes, hard-baseline images are also site. Image data for epipolar-resampled images (FFL files) involved. In order to reduce the correlation between the was downloaded from the MER Analyst’s Notebook website EOPs of the hard-baseline images, the fixed relationship (http://an.rsl.wustl.edu/mer/mera/mera.htm) and used in this between the left and right images of the hard-baseline stereo experiment. Figure 7 shows the automatically selected tie pairs was applied as constraint equations in the bundle points (white and black crosses) overlaid on the two image adjustment to improve the accuracy and reliability of the mosaics. Overall there were 218 wide-baseline tie points, out adjustment computation (Di et al., 2004). Since there is no of which 15 (black crosses) were used as checkpoints in the absolute ground control on the Martian surface, the bundle subsequent bundle adjustment. adjustment is basically a free net adjustment in which the normal matrix is rank deficient. We used the Singular Value Automated Bundle Adjustment of Wide-baseline Images Decomposition technique to solve the normal equation in Using the collinearity equation as the basic model, bundle which the minimum norm principle was applied along with adjustment was used to adjust the EOPs of the images and the principle of least squares. the 3D coordinates of the tie points so that their optimal The mapping accuracy of wide-baseline images is least-squares values were achieved. The image orientation reflected in the accuracy of the bundle adjustment of the parameters were extracted in the headers of the FFL files and two sites. Since no absolute ground control is available, it converted from the CAHV model to EOPs of the conventional is impossible to evaluate the accuracy using ground truth. photogrammetry model (Di and Li, 2004); they were defined Instead, the bundle adjustment accuracy was estimated by in the Landing Site Local (LSL) frame (Li et al., 2004; checking consistencies between positions in the 3D ground Di et al., 2008) and used as the initial values of the space and corresponding positions in 2D image space. From unknowns in the bundle adjustment. They were refined sites APFI and APGB, we selected 15 cross-site tie points through the iterative solutions of the bundle adjustment. (appearing in multiple wide-baseline stereo pairs) as

PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING June 2011 615 609-618_10-056.qxd 5/21/11 12:48 PM Page 616

checkpoints, while another 203 tie points were used in the Dense Matching and Digital Elevation Model (DEM) Generation bundle adjustment. First, the 3D position of each check The bundle adjustment provides improved EOPs of the point was calculated from one wide-baseline stereo pair images as well as high-precision 3D ground positions for all and then from another wide-baseline stereo pair; any the tie points. The 3D positions of all matched feature points difference was obtained by calculating the 3D distance. The (out of which the tie points were selected) were calculated 3D accuracy was then depicted by the average of this by space intersection. However, their number was still difference for all the check points in object space. Second, sparse for reconstruction of the details of the terrain. Based the 3D position of a check point, as calculated from one on these matched feature points and the refined EOPs, we wide-baseline stereo pair, was projected onto another applied a dense matching of the wide-baseline images. A wide-baseline stereo pair. The projected image point was triangulated irregular network (TIN)-controlled dense image compared to the actual measured point and the matching method was developed by Li et al. (2007) for the difference was obtained by calculating the 2D distance. dense matching of hard-baseline images. We adopted this The 2D accuracy was then depicted by the average of the strategy and imposed the epipolar geometry of wide-baseline differences for all the checkpoints in image space. The images. First, a TIN was constructed using the matched maximal distance from the wide-baseline sites to the feature points of the wide-baseline images at Site 0. Second, checkpoints is up to 400 m. Results were 3D accuracy of triangles and the epipolar geometry were used to predict the 0.563 m and 2D accuracy of 0.714 pixel after bundle approximate location of the homologous point in the wide- adjustment. With a 3D accuracy of 1.118 m and a 2D accu- baseline images at Site 1. Third, image matching was racy of 2.952 pixels before bundle adjustment, it is clear performed in a small search range around the predicted that bundle adjustment has improved accuracy in location along the epipolar line using correlation coeffi- both image and object space. This bundle adjustment cients. Fourth, least-squares matching was applied to all the accuracy represents the mapping accuracy of the wide- matched dense points to refine the matched locations to a baseline images. sub-pixel level. Finally, 3D coordinates of the matched We also tested the automated bundle adjustment using points were calculated using the refined EOPs of the wide- the Pancam wide-baseline images taken by the Spirit rover baseline images. In our mapping of McCool Hill, we used a at sites AVLF (on Sol 1348) and AVMA (on Sol 1350) for grid of 3 ϫ 3 pixels for dense matching. Figure 8 shows the mapping Von Braun Hill at a range up to 120 m. The wide- results of dense matching of a wide-baseline stereo pair. baseline was about 8 m. The “stereo ϩ single” mode was Overall, we obtained 7,185 matched points (including the employed; we selected ten hard-baseline stereo pairs at site previously matched feature points) in this pair. In order to AVLF and ten single images at site AVMA. A total of 238 ensure the quality of the DEM, an outlier detection procedure wide-baseline tie points were selected automatically, out of was performed on the dense 3D points using a local-surface- which 18 were used as checkpoints. The maximal distance fitting technique. The outliers are caused by mismatches in from the sites to the checkpoints is up to 110 m. Before the dense matching process. Any 3D points that are larger bundle adjustment, 3D accuracy was 1.681 m, and 2D than three times the standard deviation of the distances to accuracy was 2.515 pixels; after adjustment, 3D accuracy was the local fitted plane were eliminated as outliers; 331 0.623 m, and 2D accuracy was 0.842 pixel. Again, results outliers were eliminated in this experiment. The resultant show that the bundle adjustment improves mapping 44,533 3D points were used to generate a 1 m resolution DEM accuracy, and the method for automatic wide-baseline tie based on Kriging interpolation. Figure 9 shows the DEM of point selection is effective. McCool Hill generated from wide-baseline images.

Figure 8. Dense image matching result of a wide-baseline stereo pair.

616 June 2011 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING 609-618_10-056.qxd 5/21/11 12:48 PM Page 617

References Bay, H., A. Ess, T. Tuytelaars, and L. Van Gool, 2008. Speeded-up robust features, Computer Vision and Image Understanding, 110(3):346–359. Bell, J.F. III, S.W. Squyres, K.E. Herkenhoff, J.N. Maki, H.M. Arneson, D. Brown, S.A. Collins, A. Dingizian, S.T. Elliot, E.C. Hagerott, A.G. Hayes, M.J. Johnson, J.R. Johnson, J. Joseph, K. Kinch, M.T. Lemmon, R.V. Morris, L. Scherr, M. Schwochert, M.K. Shepard, G.H. Smith, J.N. Sohl-Dickstein, R.J. Sullivan, W.T. Sullivan, and M. Wadsworth, 2003. Mars exploration rover Athena panoramic camera (Pancam) investigation, Journal of Geophysical Research - Planets, 108(E12):8063 (doi:10.1029/2003JE002070). Chen, W., 2008. Wide Baseline Mapping in Mars Exploration, M.Sc. thesis, The Ohio State University, Columbus, Ohio, 45 p. Di, K., and R. Li, 2004. CAHVOR camera model and its photogrammetric conversion for planetary applications, Journal of Geophysical Research - Planets, 109(E4):E04004, (doi:10.1029/2003JE002199). Di, K., F. Xu, and R. Li, 2004. Constrained bundle adjustment of Figure 9. 3D view of the McCool Hill DEM. panoramic stereo images for Mars landing-site mapping, Proceedings of the 4th International Symposium on Mobile Mapping Technology, 29–31 March, Kunming, China, unpaginated CD-ROM. Di, K., and R. Li, 2007. Topographic mapping capability analysis of Conclusions Mars exploration rover 2003 mission imagery, Proceedings of This paper has described a systematic accuracy analysis of the 5th International Symposium on Mobile Mapping wide-baseline mapping through theoretical derivation and Technology (MMT 2007), 28–31 May, Padua, Italy, unpaginated Monte Carlo simulation. Considering pixel measurement CD-ROM. error and baseline error, an analytical equation was derived Di, K., F. Xu, J. Wang, S. Agarwal, E. Brodyagina, R. Li, L.H. using the “normal case” configuration. In addition, a more Matthies, 2008. Photogrammetric processing of rover imagery of comprehensive analysis was performed using Monte Carlo the 2003 Mars exploration rover mission, ISPRS Journal of simulation taking into consideration pixel measurement Photogrammetry and Remote Sensing, 63(2):181–201. error, baseline error and angular error along with three Fischler, M.A., and R.C. Bolles, 1981. Random sample consensus: separate stereo modes. The results can be used as guidelines A paradigm for model fitting with applications to image in the design of wide-baseline mapping for such variables as analysis and automated cartography, Communication of the optimal baseline and stereo mode. ACM, 24(6):381–395. An automatic method for wide-baseline tie point Förstner, W., and E. Gulch, 1987. A fast operator for detection and selection was developed in this research based on a precise location of distinct points, corners and centers of circular features, Proceedings of the ISPRS Intercommission combination of Förstner operator and SURF operator Conference on Fast Processing of Photogrammetric Data, 02–04 followed by a RANSAC-based outlier elimination procedure. June, Interlaken, Switzerland, pp. 281–305. The sufficient number of evenly distributed tie points Harris, C., and M. Stephens, 1988. A combined corner and edge enabled an automated bundle adjustment. Experiments detector, Proceedings of the 4th Alvey Vision Conference, 31 using wide-baseline images taken by the Spirit rover for August - 02 September, Manchester, UK, pp. 147–151. mapping McCool Hill and Von Braun Hill demonstrated Kovesi, P.D., 2000. MATLAB and Octave functions for computer that the developed automated bundle adjustment is vision and image processing, School of Computer Science & effective. After bundle adjustment, the mapping capability Software Engineering, The University of Western Australia, of the rover stereo images can be extended from tens of URL: http://www.csse.uwa.edu.au/ϳpk/research/matlabfns/ meters (hard-baseline) to hundreds of meters using wide- (last date accessed: 17 March 2011). baseline mapping techniques. Finally, a TIN-controlled Li, R., K. Di, L.H. Matthies, R.E. Arvidson, W.M. Folkner, and dense image matching method was used to generate B.A. Archinal, 2004. Rover localization and landing-site dense matched points from wide-baseline images. mapping technology for 2003 Mars exploration rover mission, Consequently, a high-resolution DEM was generated by Photogrammetric Engineering & Remote Sensing, 70(1):77–90. Kriging interpolation. Li, R., S.W. Squyres, R.E. Arvidson, B.A. Archinal, J. Bell, Y. Cheng, The reliability of wide-baseline image matching is an L. Crumpler, D.J. Des Marais, K. Di, T.A. Ely, M. Golombek, E. Graat, J. Grant, J. Guinn, A. Johnson, R. Greeley, R.L. Kirk, important issue for wide-baseline mapping. The TIN- M. Maimone, L.H. Matthies, M. Malin, T. Parker, M. Sims, controlled dense matching may fail in some areas due to L.A. Soderblom, S. Thompson, J. Wang, P. Whelley, and F. Xu, large perspective changes and/or poor textures. In the 2005. Initial results of rover localization and topographic future, we will improve the speed and reliability of dense mapping for the 2003 Mars exploration rover mission, wide-baseline matching and conduct field tests on Earth, Photogrammetric Engineering & Remote Sensing, Special Issue using ground truth to validate the attainable mapping on Mapping Mars, 71(10):1129–1142. accuracy. Li, R., K. Di, A.B. Howard, L.H. Matthies, J. Wang, and S. Agarwal, 2007. Rock modeling and matching for autonomous long-range Mars rover localization, Journal of Field Robotics, Acknowledgments 24(3):187–203. Funding of this research by the National High Technology Lowe, D.G., 1999. Object recognition from local scale-invariant features, Proceedings of the International Conference on Research and Development Program of China Computer Vision, 20–25 September, Corfu, Greece, (2009AA12Z310) and National Natural Science Foundation pp. 1150–1157. of China (40871202) is acknowledged. The authors Maki, J.N., J.F. Bell, III, K.E. Herkenhoff, S.W. Squyres, A. Kiely, appreciate the constructive comments from the M. Klimesh, M. Schwochert, T. Litwin, R. Willson, A. Johnson, reviewers. M. Maimone, E. Baumgartner, A. Collins, M. Wadsworth,

PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING June 2011 617 609-618_10-056.qxd 5/21/11 12:48 PM Page 618

S.T. Elliot, A. Dingizian, D. Brown, E.C. Hagerott, L. Scherr, Olson, C.F., and H. Abi-Rached, 2010. Wide-baseline stereo vision R. Deen, D. Alexander, and J. Lorre, 2003. Mars exploration for terrain mapping, Machine Vision and Applications, rover engineering cameras, Journal of Geophysical Research - 21(5):713–725. Planets, 108(E12):8071, (doi:10.1029/2003JE002077). Strecha, C., T. Tuytelaars, and L. Van Gool, 2003. Dense Matas, J., O. Chum, M. Urban, and T. Pajdla, 2002. Robust wide- matching of multiple wide baseline, Proceedings of the 9th baseline stereo from maximally stable extremal regions, IEEE International Conference on Computer Vision, 14–17 Proceedings of the 13th British Machine Vision Conference October, Nice, France, pp. 1194–1201. (BMVC2002), 02–05 September, Cardiff, Wales, UK, Schmid, C., R. Mohr, and C. Bauckhage, 2000. Evaluation of interest pp. 384–394. point detectors, International Journal of Computer Vision, Mikolajczyk, K., and C. Schmid, 2004. Scale and affine invariant 37(1):151–172. interest point detectors, International Journal of Computer Tuytelaars, T., and L. Van Gool, 2000. Wide-baseline stereo Vision, 60(1):63–86. matching based on local, affinely invariant regions, Proceedings Olson, C.F., H. Abi-Rached, M. Ye, and J.P. Hendrich, 2003.Wide- of the 11th British Machine Vision Conference (BMVC), 11–14 baseline stereo vision for Mars rovers, Proceedings of the 2003 September, Bristol, UK, pp. 412–422. IEEE/RSJ International Conference on Intelligent Robots and Zhang Z., and J. Zhang, 1997. Digital Photogrammetry, Press of the Systems, 27–31 October, Las Vegas, Nevada, pp. 1302–1307. Wuhan Technical University of Surveying and Mapping, Olson, C.F., L.H. Matthies, J.R. Wright, R. Li, and K. Di, 2007. Wuhan, China, 296 p. Visual terrain mapping for Mars exploration, Computer Vision (Received 13 July 2010; accepted 24 September 2010; final version and Image Understanding, 105(1):73–85. 07 February 2011)

Certification Seals & Stamps  Now that you are certified as a remote sensor, photo grammetrist Send completed form with your payment to: or GiS/LiS mapping scientist and ASPRS Certification Seals & Stamps, 5410 Grosvenor Lane, Suite 210, Bethesda, MD 20814-2144 you have that certificate on the wall, make sure everyone knows! NAMe: PhoNe:  An embossing seal or rubber stamp adds a certified finishing touch to CeRtifiCAtioN #: exPiRAtioN DAte: your professional product.  you can’t carry around your certifi- ADDReSS: cate, but your seal or stamp fits in your pocket or briefcase. City: StAte: PoStAL CoDe: CouNtRy:  to place your order, fill out the neces- pleaSe Send me: embossing Seal ...... $45  Rubber Stamp ...... $35 sary mailing and certification informa- tion. Cost is just $35 for a stamp and method of payment:  Check  Visa  MasterCard  American express  Discover $45 for a seal; these prices include domestic uS shipping. international CReDit CARD ACCouNt NuMBeR exPiReS shipping will be billed at cost. Please allow 3-4 weeks for delivery. SiGNAtuRe DAte

Note: ASPRS certification seals and stamps can only be authorized by ASPRS and manufactured by ASPRS-approved vendors. unauthorized seal or stamp production is a violation of copyright law and the ASPRS Code of ethics.

618 June 2011 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

June 2011 layout.indd 618 5/23/2011 11:12:44 AM