Chapter 4 Photometric Stereo Shape-From-Shading (PS-Sfs) for 3D Modelling of the Lunar Surface

By reading and using the thesis, the reader understands and agrees to the following terms:

1. The reader will abide by the rules and legal ordinances governing copyright regarding the use of the thesis.

2. The reader will use the thesis for the purpose of research or private study only and not for distribution or further reproduction or any other purpose.

3. The reader agrees to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.

IMPORTANT If you have reasons to believe that any materials in this thesis are deemed not suitable to be distributed in this form, or a copyright owner having difficulty with the material being included in our database, please contact [email protected] providing details. The Library will look into your claim and consider taking remedial action upon receipt of the written requests.

Pao Yue-kong Library, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong

http://www.lib.polyu.edu.hk

CONSTRUCTION OF HIGH-RESOLUTION LUNAR DEMS BY FUSING PHOTOGRAMMETRY AND SHAPE-FROM-SHADING

Wai Chung LIU

PhD The Hong Kong Polytechnic University 2020

The Hong Kong Polytechnic University Department of Land Surveying and Geo-Informatics

Construction of High-Resolution Lunar DEMs by Fusing Photogrammetry and Shape-from-Shading

Wai Chung LIU

A Thesis Submitted in Partial Fulfilment of the Requirements for the Degree of Doctor of Philosophy

December 2019

CERTIFICATE OF ORIGINALITY

I hereby declare that this thesis is my own work and that, to the best of my knowledge and belief, it reproduces no material previously published or written, nor material that has been accepted for the award of any other degree or diploma, except where due acknowledgment has been made in the text.

(Signed)

Wai Chung LIU (Name of student) Abstract

3D topographic modelling of the lunar surface is fundamental to lunar and planetary science and exploration. Topographic models such as digital elevation models (DEMs) of the lunar surface have seen extensive use in the study of the lunar surface and geologic processes and for planning of lunar exploration missions. Lunar DEMs are usually generated from remote sensing data acquired by instruments such as cameras and laser altimeters onboard lunar orbiters. The currently available lunar DEMs have the following limitations: (1) lunar DEMs with global or near-global coverage usually have a comparatively low spatial resolution (i.e., 60 m/pixel or lower); (2) regional high-resolution DEMs generated with photogrammetry have limited spatial coverage; and (3) lunar DEMs cannot reach the effective spatial resolution of optical images acquired by lunar orbiters. The performance of photogrammetry, in particular, depends heavily on the robustness of image matching, which directly affects the spatial resolution and accuracy of the resulting lunar DEMs. Photogrammetry may even fail due to the limitations of the image matching approach when the stereo images are poorly textured and/or acquired with large variations in illumination, which are common problems for lunar images. To overcome these limitations, lunar topographic modelling was performed with Shape- from-Shading (SfS; also known as photoclinometry), a technique that generates 3D models based on the image’s photometric content (i.e., intensity value). It can reconstruct lunar DEMs with optimal spatial resolution using single or multiple images, and it is robust to fine-grained details, subtle surface textures and variations in illumination. However, SfS lacks accuracy for large-scale topographic modelling, at which photogrammetry excels. As a result, SfS and photogrammetry are natural complements for the generation of accurate and optimal 3D representations of the lunar surface, which can better support exploration missions (e.g., landing site selection) and scientific research (e.g., lunar geology). This research aims to integrate SfS and photogrammetry to construct accurate lunar DEMs with optimal spatial resolution. This study includes three major developments. First, a novel hierarchical SfS approach was developed. Given a single monocular image, the single-image SfS (SI-SfS) approach refines an initial DEM of lower resolution to an optimal resolution of the image. The corresponding pixel-wise albedo map of the surface is estimated simultaneously in the process and

I is used by SI-SfS to regularise the 3D refinement of the initial DEM. A hierarchical architecture was used for effective SfS at multiple resolutions, and an explicit shadow constraint was developed and incorporated into the SI-SfS approach to overcome image regions with shadows. Experiments were carried out using Narrow Angle Camera (NAC) images from the Lunar Reconnaissance Orbiter Camera (LROC), with a spatial resolution of 0.5 to 1.5 m/pixel. The integrated Selenological and Engineering Explorer and LRO Elevation Model (SLDEM), with a spatial resolution of 60 m/pixel, was chosen as the initial DEM for constraint. The results indicate that local topographic details were well recovered by the SI-SfS approach with plausible albedo estimation. The refined DEM achieved geometric accuracies of 2 to 7 m relative to the reference NAC DEMs generated from photogrammetry. The low-frequency topographic consistency depends upon the quality of the low- resolution DEM and the difference in the spatial resolution between the image and the initial DEM. Second, a photometric stereo SfS (PS-SfS) approach was developed. The PS- SfS approach uses two co-registered images acquired under different illumination conditions to generate high-resolution lunar DEMs. In contrast to the previous SI- SfS approach, PS-SfS does not require an initial DEM. A novel formulation was used to effectively factor out the effects of albedo variations, leading to more robust performance. Moreover, a quantitative investigation of the effects of illumination differences on the performance of 3D reconstruction was carried out for the PS-SfS. First, the fundamental process of PS-SfS was mathematically modelled to correlate the numeric solution of PS-SfS and the images’ differences in illumination. Based upon the mathematical model, an error model was then derived to analyse the relationships between the azimuthal and zenith angles of the images’ illumination and the reconstruction qualities. The developed PS-SfS approach and the error model were verified with LROC NAC images. Our experimental analyses reveal that the resulting error in PS-SfS depends upon both the azimuthal and the zenith angles of illumination and upon the general intensity of the images. The predictions from the proposed error model are consistent with the actual slope errors obtained by PS-SfS using LROC NAC images. The proposed error model enriches the theory of PS-SfS and is significant for optimised lunar surface reconstruction based on SfS techniques. Third, an integrated photogrammetric and photoclinometric approach was developed. The integrated approach can generate lunar DEMs with optimal II resolution and is invariant to variations in illumination. The fusion of photoclinometry and photogrammetry involves two main steps. First, a photoclinometry-assisted image matching (PAM) approach is developed by integrating PS-SfS into the image matching stage to create pixel-wise matches, even for images with large differences in illumination. Second, the DEM derived from photogrammetry using the matching results is refined to optimal resolution using the SI-SfS approach. The proposed approach has been validated with high-resolution LROC NAC images acquired in various conditions of illumination at the Chang’E-4 and Chang’E-5 landing sites. The results indicate that the integrated approach is robust to severe inconsistencies in illumination and to subtle textures in cases in which the conventional approaches fail. The integrated approach can achieve geometric accuracies comparable to photogrammetry, while giving more small-scale topographic detail. The presented research and development allows for the generation of high- precision and high-resolution lunar DEMs from a single image or multiple images, which effectively extends the ability of photogrammetry in lunar mapping. The developed approaches have been widely used in China’s Chang’E-4 and Chang’E-5 lunar missions to generate high-resolution DEMs of the landing regions to support optimised landing site selection and surface operation. The developed approaches can also be used for high-resolution topographic mapping of other planetary bodies, such as Mercury and asteroids, and to provide a useful reference for topographic mapping on Earth.

III

Publications Arising from the Thesis

Journal Papers: [1] Liu, W. C. and Wu, B., 2020. An Integrated Photogrammetric and Photoclinometric Approach for Illumination-Invariant Pixel-Resolution 3D Mapping of the Lunar Surface. ISPRS Journal of Photogrammetry and Remote Sensing, 159: 153-168. [2] Liu, W. C., Wu, B., and Wöhler, C., 2018. Effects of Illumination Differences on Photometric Stereo Shape-and-Albedo-from-Shading for Precision Lunar Surface Reconstruction. ISPRS Journal of Photogrammetry and Remote Sensing, 136: 58-72. [3] Wu, B., Liu, W. C., Grumpe, A., and Wöhler, C., 2017. Construction of Pixel- Level Resolution DEMs from Monocular Images by Shape and Albedo from Shading Constrained with Low-Resolution DEM. ISPRS Journal of Photogrammetry and Remote Sensing, 140: 3-19. [4] Wu, B. and Liu, W. C., 2017. Calibration of Boresight Offset of LROC NAC Imagery for Precision Lunar Topographic Mapping. ISPRS Journal of Photogrammetry and Remote Sensing, 128: 372-387.

Conference Papers: [1]. Wohlfarth, K. S., Liu, W. C., Wu, B., Grumpe, A., Wöhler, C., 2018. High Resolution Digital Terrain Models of the Martian Surface: Compensation of the Atmosphere on CTX imagery. 49th Lunar and Planetary Science Conference, March 19-23, 2018, The Woodlands, Texas, USA. [2]. Liu, W.C., Wu, B., 2017. Photometric Stereo Shape-And-Albedo-From- Shading for Pixel-Level Resolution Lunar Surface Reconstruction. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII-3/W1: 91-97. [3] Wu, B., Liu, W. C., Grumpe, A., Wöhler, C., 2016. Shape and Albedo from Shading (SAfS) for Pixel-Level DEM Generation from Monocular Images Constrained by Low-Resolution DEM. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLI-B4: 521-527.

Acknowledgements

First and foremost, I would like to express my gratitude and thanks to Dr. Bo Wu, my chief supervisor, for his ongoing support, rigorous attitude, and thoughtful advice. Dr. Wu shares a lot of personal experience and insights regarding new ideas, the science community, career paths, personal well-being and way of life; under his supervision, I am gradually trained as an independent researcher throughout the years of study. Furthermore, I would like to thank Prof. Dr. Christian Wöhler, for his experience, insights and willingness to share. Prof. Wöhler provided a lot of assistance to me in learning and understanding the theories; he shared a lot of stories and advice not only related to academia but also related to daily life. Special thanks are given to Prof. Randy Kirk, who pinpointed the possible problems and provided detailed explanations for me to push forward my research. I would also like to acknowledge all of the academic and administrative staff members of LSGI for their support. Thank Dr. Bruce King for providing suggestions on my research, and to Dr. Man-sing Wong for his comments during the confirmation of my study. Moreover, I would like to thank the group members of the Photogrammetry Lab in LSGI, PolyU. Thanks to Dr. Lei Ye and Dr. Shengjun Tang for providing advice and support during the early and middle stages of the study. Thanks to Dr. Han Hu, Dr. Xuming Ge, Mr. Linfu Xie and Mr. Long Chen for providing technical advice and mathematical insights. Thanks to Ms. Yuan Li and Ms. Yiran Wang for providing advice on this dissertation. Thanks to Mr. Manish Sharma and Mr. Zeyu Chen for providing comments from different perspectives and generating interesting new ideas together. Thanks are given to the Image Analysis Group, TU Dortmund, led by Prof. Dr. Wöhler, for their warm welcome and assistance during my visit to Germany and the Netherlands. Thanks to Mr. Kay Wohlfarth and Mr. Marcel Hess for sharing ideas and generating interesting topics together. Thanks to Dr. Arne Grumpe for providing advice regarding the publications. Last but not least, I would like to express my gratitude and thanks to my mother, who provided unlimited support and love to me throughout all the ups and downs in life.

Table of Contents

Abstract ...... I

Publications Arising from the Thesis ...... IV

Acknowledgements ...... V

List of Figures ...... IX

List of Tables ...... XII

Chapter 1 Introduction ...... 1

1.1 Topographic Modelling of the Lunar Surface ...... 1

1.2 Motivation of the Research ...... 4

1.3 Contributions and Innovations of the Research ...... 6

1.4 Structure of the Dissertation ...... 7

Chapter 2 Literature Review...... 9

2.1 Photogrammetric 3D Modelling of the Lunar Surface ...... 9 2.1.1 Photogrammetry ...... 9 2.1.2 Image Matching ...... 11

2.2 Photoclinometric 3D Modelling of the Lunar Surface ...... 15 2.2.1 Reflectance Models ...... 15 2.2.2 Shape-from-Shading (Photoclinometry) ...... 20

2.3 Fusion of Photogrammetry and Shape-from-Shading for 3D Modelling of the Lunar Surface ...... 26

2.4 Summary ...... 29

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface ...... 33

3.1 Overview of the SI-SfS Approach ...... 33

3.2 Derivation of Surface Gradients ...... 35 3.2.1 Surface Gradients for Illuminated Regions ...... 35 3.2.2 Initial DEM Used as a Constraint ...... 37 3.2.3 Image Shadow Used as a Constraint ...... 39

3.3 Reconstruction of a DEM from Surface Gradients ...... 40

3.4 Experimental Analysis of the SI-SfS Approach ...... 43 3.4.1 Description of Datasets ...... 43 3.4.2 Buisson V Dataset ...... 45 3.4.3 Reiner Gamma Dataset ...... 49 3.4.4 Rima Sharp Dataset ...... 53

3.5 Summary ...... 62

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface ...... 63

4.1 Overview of the PS-SfS Approach ...... 63

4.2 Derivation of Surface Gradients from Multiple Images ...... 64 4.2.1 Albedo-Invariant Computation of Surface Gradients ...... 64 4.2.2 Determination of the Principal Direction ...... 66

4.3 Reconstruction of a DEM from Surface Gradients ...... 69

4.4 Effects of Illumination Inconsistencies on the 3D Modelling Performance...... 71 4.4.1 Modelling the Process of Photometric Stereo Shape-from-Shading...... 71 4.4.2 Error Propagation with Respect to Angular Illumination Differences... 74

4.5 Experimental Analysis of the PS-SfS Approach ...... 78 4.5.1 Description of Verification Routine ...... 78 4.5.2 Description of Datasets ...... 81 4.5.3 Evaluation and Analysis of the Antoniadi Dataset ...... 85 4.5.4 Evaluation and Analysis of the Bogslwsky dataset ...... 92

4.6 Summary ...... 98

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface ...... 99

5.1 Overview of the Integration Approach ...... 99

5.2 Pixel-Wise Image Matching Invariant to Illumination Inconsistencies ...... 100 5.2.1 Photoclinometry-Assisted Image Matching ...... 100

VII

5.2.2 Computation of Matching Score ...... 102 5.2.3 Filtering of Mismatches ...... 104

5.3 Refinement of a Photogrammetric DEM based on Shape-from- Shading ...... 105

5.4 Experimental Analysis of the Integration Approach ...... 107 5.4.1 Experimental Analysis at the Chang’E-4 Landing Site ...... 107 5.4.2 Experimental Analysis at the Chang’E-5 Candidate Landing Site ...... 115

5.5 Summary ...... 122

Chapter 6 Conclusions and Discussion ...... 123

6.1 Summary of the Research Work ...... 123

6.2 Discussions ...... 125

6.3 Conclusions ...... 128

6.4 Considerations for Future Works...... 129

References ...... 133

VIII

List of Figures

Figure 1.1 Structure of the dissertation research...... 8 Figure 2.1 Conceptual illustration of the photometric angles during image acquisition...... 17 Figure 2.2 Conceptual illustration of SfS...... 22 Figure 2.3 Merits of fusing of SfS and photogrammetry...... 31 Figure 2.4 Summary of common image matching approaches in planetary mapping...... 31 Figure 2.5 Summary of existing SfS algorithms and fusion approaches with photogrammetry for 3D modelling of planetary surfaces...... 32 Figure 3.1 Overview of the SI-SfS approach...... 35 Figure 3.2 The layout of a DEM node during height optimisation. The centre node is optimised according to the surrounding surface gradients...... 41 Figure 3.3 Optimisation steps of the Modified-Jacobi relaxation strategy...... 42 Figure 3.4 The LROC NAC images used for the three experimental analyses...... 45 Figure 3.5 The (a) input image; (b) high sun angle image; and 3D views of (c) initial SLDEM and (d) the output SfS DEM of the Buisson V dataset...... 47 Figure 3.6 Profile comparisons of the Buisson V dataset...... 48 Figure 3.7 Albedo comparison of the Buisson V dataset...... 49 Figure 3.8 The (a) input image; and 3D views of (b) initial SLDEM; (c) the output SfS DEM; and (d) independent reference DEM of the Reiner Gamma Dataset...... 50 Figure 3.9 Albedo comparison of the Reiner Gamma dataset...... 51 Figure 3.10 Profile comparisons of the Reiner Gamma dataset...... 52 Figure 3.11 The (a) input image; and 3D views of (b) initial SLDEM; (c) the output SfS DEM; and (d) independent reference DEM of the Rima Sharp Dataset...... 54 Figure 3.12 Profile comparisons for the Rima Sharp dataset...... 55 Figure 3.13 Intensity profile of the Rima Sharp dataset...... 56 Figure 3.14 Albedo comparison of the Rima Sharp dataset...... 57 Figure 3.15 The 3D views of (a) Down-sampled initial NAC DEM; (b) output SfS DEM; and; (c) the reference NAC DEM of the Rima Sharp Dataset...... 58 Figure 3.16 Profile comparisons for the Rima Sharp dataset...... 59 Figure 3.17 Comparison of absolute difference DEM between the proposed result and Grumpe et al. (2014)’s result...... 61

Figure 4.1 Workflow for the PS-SfS approach...... 64 Figure 4.2 Determination of the principal direction in photometric stereo. Vector u denotes the principal direction and v denotes the perpendicular of the principal direction...... 68 Figure 4.3 Comparison of the range of reflectance ratio evaluated along the principal direction u (blue solid line) and its perpendicular v (red dashed line), respectively. 68 Figure 4.4 Determination of surface slope at point O through PS-SfS...... 73 Figure 4.5 Verification routine with real data for the proposed error model...... 80 Figure 4.6 Pixel-synchronous LROC NAC images and the reference NAC DEM for the Antoniadi dataset...... 81 Figure 4.7 Pixel-synchronous LROC NAC images and reference NAC DEM of the Bogslwsky dataset ...... 83 Figure 4.8 Comparison between the estimated theoretical error from the proposed error model and actual error from PS-SfS of the Antoniadi dataset...... 87 Figure 4.9 3D views of PS-SfS DEMs at different α of the Antoniadi dataset...... 88 Figure 4.10 Angular error maps from photometric stereo SAfS at different αof the Antoniadi dataset...... 89 Figure 4.11 Vertical RMSE and the corresponding absolute difference DEMs of the Antoniadi dataset...... 91 Figure 4.12 3D views of photometric stereo SAfS DEMs at different α of the Bogslwsky dataset...... 93 Figure 4.13 Angular error maps from photometric stereo SAfS at different αof the Bogslwsky dataset...... 94 Figure 4.14 Comparison between the estimated theoretical error from the proposed error model and actual error from PS-SfS of the Bogslwsky dataset...... 95 Figure 4.15 Vertical RMSE and the corresponding absolute difference DEMs of the Bogslwsky dataset...... 97 Figure 5.1 The overview of the integrated photogrammetric and photoclinometric approach...... 100 Figure 5.2 The workflow of photoclinometry assisted image matching (PAM). .... 101 Figure 5.3 The LROC NAC image pair covering the Chang’E-4 landing site. Blue boxes show the enlarged views of the landing area, and green crosses indicate the landing site on each of the images. Yellow arrows indicate the illumination direction of each image...... 108 X

Figure 5.4 The epipolar images used for matching and the disparity maps generated by PAM for the Chang’E-4 landing site. Poor matches are outlined in blue...... 110 Figure 5.5 3D views of the DEMs and the corresponding ortho-image covering the Chang’E-4 landing site. The blue boxes show close up views of the landing site (the red cross)...... 111 Figure 5.6 The shaded relief of the DEMs covering the Chang’E-4 landing site. The blue boxes show close up views of the landing site (the red cross)...... 112 Figure 5.7 Profile comparison of the DEMs of the Chang’E-4 landing site. The blue boxes indicate the topographic details revealed by the integrated DEM...... 114 Figure 5.8 The LROC NAC image pair within the Chang’E-5 candidate landing region. Blue boxes show the enlarged views of the lunar dome region, and yellow arrows indicate the illumination direction of each image...... 116 Figure 5.9 The epipolar images used for matching and the disparity maps generated by PAM within the Chang’E-5 candidate landing region...... 118 Figure 5.10 3D views of the DEMs and the corresponding ortho-image within the Chang’E-5 candidate landing region...... 119 Figure 5.11 The shaded relief of the DEMs within the Chang’E-5 candidate landing region...... 120 Figure 5.12 Profile comparison of the DEMs within the Chang’E-5 candidate landing site. The blue boxes indicate the small topographic details revealed by the integrated DEM...... 121

List of Tables

Table 3.1 Details of LRO NAC imagery used in the experiments...... 45 Table 3.2 Statistical analysis of profiles of the Reiner Gamma dataset...... 51 Table 3.3 Statistical analysis of profiles for the Rima Sharp dataset ...... 56 Table 3.4 Statistical analysis of profiles for the Rima Sharp dataset constrained by 20m NAC DEM...... 60 Table 3.5 Statistical comparison of absolute difference DEMs for the Rima Sharp dataset...... 61 Table 4.1Imaging and illumination conditions of the images in the Antoniadi dataset...... 82 Table 4.2 Azimuthal difference (α)and zenith difference (embedded in r) of the illuminations for photometric stereo pairs used in the Antoniadi dataset...... 82 Table 4.3 Imaging and illumination conditions of the images in the Bogslwsky dataset...... 84 Table 4.4 Azimuthal difference (α)and zenith difference (embedded in r) of the illuminations for photometric stereo pairs in the Bogslwsky dataset...... 84 Table 4.5 Estimated error from the proposed error model and actual PS-SfS error for the Antoniadi dataset...... 86 Table 4.6 Estimated error from the proposed error model and actual PS-SfS error of the Bogslwsky dataset...... 96 Table 5.1 Comparative statistics of the DEMs of the Chang’E-4 landing site...... 113 Table 5.2 Comparative statistics of the DEMs within the Chang’E-5 candidate landing region...... 120

XII

Chapter 1 Introduction

The moon, as our closest neighbour in the solar system, has been a prime target of planetary science and extraterrestrial explorations and expeditions. The moon remains an attractive target because our understanding of it increases with the vast amount of data returned from an increasing number of orbital and landing missions. The topography of the lunar surface is the most fundamental data for understanding the moon. It is also of utmost importance for planning exploration missions. This chapter gives a brief introduction of the current status of topographic modelling of the lunar surface, followed by the motivations and objectives of this study.

1.1 Topographic Modelling of the Lunar Surface

3D topographic modelling of the lunar surface is critical to many aspects of fundamental lunar science and planetary explorations. 3D models of the lunar surface, usually presented in the form of digital elevation models (DEMs), describe the moon’s geodesic and geometric properties. They are used to extract and characterise various surface features such as craters (Wöhler et al., 2006; Di et al., 2014; Salamunicćar et al., 2014; Wu et al., 2018b; Peng et al., 2019) to facilitate our understanding of the lunar surface’s evolution by techniques such as surface dating (Neukum, 2001). Lunar DEMs are also required for accurate interpretation of lunar spectral signatures and for the subsequent analysis of lunar surface composition and geologic properties (Grumpe et al., 2014; Wöhler et al, 2014; 2017). In addition to geologic and other scientific studies of extraterrestrial bodies, DEMs are also required to analyse potential landing sites for future missions (Beyer et al., 2003; Mazarico et al., 2011; Beyer and Kirk, 2012; De Rosa et al., 2012; Beyer, 2017) and to identify optimal paths for robotic rovers (Speyerer et al., 2016). High-resolution and high-precision topographic modelling of the lunar surface generally uses photogrammetry or laser altimetry data (Smith et al., 2010; Wu et al., 2014). Photogrammetry acquires images of the same area from multiple viewpoints (i.e., stereo images). Image matching is then performed to obtain

Chapter 1 Introduction

matched points on the stereo images that represent the same terrain features. The 3D coordinates of the matched points are determined by space intersection, which is based on the collinearity equation. This approach uses both the exterior orientation (i.e., position and pointing) and the interior orientation (i.e., internal configuration) of the images at the time they are captured and the image coordinates of the matched points. Photogrammetry is known to produce accurate 3D point clouds, and the number of 3D points is determined by the number of matching points identified in the images. A high-resolution photogrammetric lunar DEM is facilitated by a dense 3D point cloud, which is achieved by dense image matching. Usually, the resolutions of the lunar DEMs are about three times the resolution of the underlying images. For poorly textured surfaces or images with inconsistent illumination, the accuracy and resolution of the resulting DEMs degrade significantly due to the limitations of dense image matching (Heipke, 1992; Grumpe et al., 2014). Laser altimetry uses laser altimeters onboard lunar orbiters, such as the Lunar Orbiter Laser Altimeter (LOLA) of the Lunar Reconnaissance Orbiter (LRO), to determine the elevation of surveyed points on the lunar surface. The laser altimeter emits laser pulses to the lunar surface and receives the returning signals. By analysing the laser response, and the position and pointing of the laser altimeter, the elevation of the lunar surface can be accurately determined (Smith et al., 2010). Because of its very high accuracy, laser altimetry data is an important tool in determining the spacecraft orbit and the subsequent global geodetic framework of the Moon (Henriksen et al., 2017; Smith et al., 2017; Mazarico et al., 2018). Laser altimetry and photogrammetry are often combined to produce geodetically accurate 3D models (Henriksen et al., 2017; Haase, et al., 2019). The resolution of laser- altimetric 3D models depends on the density of laser-surveyed points. The cross- track and along-track resolutions of laser point clouds vary with respect to the mission orbit design. For longitudinal orbits like the LRO, the along-track point density is generally stable, whereas the cross-track point density in the lunar equatorial regions is significantly lower than in the polar regions. Laser altimetry is particularly useful for 3D topographic modelling of the lunar polar regions, which are potential targets of future expeditions in search of water ice (Mazarico et al., 2011; Gläser et al., 2014). This is because a significant portion of the lunar surface in these regions is either severely or permanently shadowed, which hinders accurate 3D modelling based on optical image measurements. In contrast, laser altimetry is an 2

Chapter 1 Introduction

active sensing system whereby laser pulses are emitted by the altimeter. Hence, it is invariant to the illumination and shadowing conditions of the lunar surface and allows for robust 3D measurements of the lunar polar regions. Currently, many lunar DEMs are widely used, including the LDEM, SLDEM, GLD100, and NAC DEMs. The LDEM is a global lunar DEM constructed using data from LOLA (Smith et al., 2010; Neumann et al., 2011). The spatial resolution of the LDEM near the lunar equator is approximately 60 m/pixel, which increases poleward because of the longitudinal orbit of the LRO. The LDEM for the lunar polar regions is by far the most comprehensive high-resolution DEM of the lunar poles (Barker et al., 2016). The SLDEM is a DEM derived by integrating and co- registering the profile tracks from LOLA and the images from the Terrain Camera onboard the Japanese lunar orbiter SELENE (Barker et al., 2016). The SLDEM has near-global coverage with a spatial resolution of approximately 60 m/pixel at the equator. The SLDEM exhibits a higher effective spatial resolution and better geometric accuracy, which leads to fewer artefacts than the LDEM. The GLD100 is a near-global lunar DEM derived from the Lunar Reconnaissance Orbiter Wide Angle Camera (LROC WAC) images (Scholten et al., 2012). The LROC WAC images have a spatial resolution of 75 m/pixel at nadir pointing, and the derived GLD100 DEM has an effective spatial resolution of approximately 1 km (Barker et al., 2016). The NAC DEMs are regional DEMs derived from the Lunar Reconnaissance Orbiter Narrow Angle Camera (LROC NAC) images. Because the LROC NAC images have the highest spatial resolution (0.5 to 1.5 m/pixel) to date (Robinson et al., 2010), the NAC DEMs have a very high spatial resolution of 2 to 5 m/pixel, which is currently the most detailed DEM of the lunar surface. However, the coverage of NAC DEMs is comparatively limited because the LROC NAC images are mostly acquired in nadir viewing. To produce a stereo pair of images to generate the NAC DEM, the LRO must steer off-nadir, which will intervene with the measurements of other onboard instruments; therefore such actions are limited (Tran et al., 2010). The NAC DEM is usually available for local regions of interest for lunar science and explorations. The NAC DEM is also available for the lunar south pole. However, because of the permanently shadowed regions in the lunar poles, the NAC DEM of the lunar south pole is less complete than the LDEM.

Chapter 1 Introduction

1.2 Motivation of the Research

In general, the lunar DEMs currently available have the following limitations. Global/near-global DEMs, such as LDEM, SLDEM, and GLD100, have very large spatial coverage but coarse spatial resolution. In particular, LDEM is known to have a comparatively lower effective cross-track resolution (East-West direction), and longitudinal artefacts are common in most of the lunar surface except for the polar regions (Barker et al., 2016). Regional DEMs such as the NAC DEM have higher spatial resolutions, but a comparatively limited coverage because of the restrictions in mission design. Moreover, the LROC NAC provided optical data of the lunar surface with the highest spatial resolution to date (0.5-1.5 m/pixel) at global coverage (mostly nadir), which is a few times higher than the spatial resolution provided by the NAC DEM. Although the LROC NAC images have been widely used to characterise small topographic features such as lunar rocks and small craters (Bandfield et al., 2011; Li and Wu, 2018), the NAC DEMs have a very limited ability for such tasks. This implies that lunar DEMs currently available cannot fully exploit the spatial resolution potential of the imagery data provided. To summarise, the lunar DEMs currently available have the following limitations: (1) Global/near-global photogrammetric/laser-altimetric DEMs have favourable spatial coverage and geometric accuracy, but coarse spatial resolution; (2) Regional photogrammetric DEMs (e.g., those from LROC NAC) have high spatial resolution and geometric accuracy, but are limited in spatial coverage; (3) The effective spatial resolutions of the DEMs are far below the spatial resolutions of the images. Lunar DEMs generated by photogrammetry are also limited by the availability of stereo images and the performance of image matching because dense and reliable image matching is a prerequisite to generate DEMs with favourable geometric accuracy and spatial resolution (Wu et al., 2011; 2012). However, this requires sufficient image texture and adequate similarities between the images. In the case of lunar and other planetary surfaces, images are often acquired under different illumination conditions. Significant illumination variations in the remote sensing images can result in dissimilar image patterns of the same terrain surface, causing failures in image matching (Heipke et al., 2007; Grumpe et al., 2014). As a result,

Chapter 1 Introduction

the limitations of image matching pose challenges to the creation of high-quality DEMs (Wu et al., 2012; Preusker et al., 2017; 2018). Shape-from-Shading (SfS), or photoclinometry, is an emerging technique for high-resolution 3D modelling of the lunar surface. Based on the fact that the energy captured by a sensor (i.e., the image intensity) is dependent on the slope of the surface with respect to the incoming solar irradiance and viewing direction of the sensor onboard the orbiter, SfS inversely determines the slope of the surface by the intensity values of the image and eventually converts the surface slopes into a 3D model. The technique has been widely applied in the community of computer vision for 3D modelling of indoor scenes and close-range objects. SfS has also been investigated and applied for topographic mapping of planetary surfaces (Barron and Milak, 2011; 2012). The use of SfS on the moon is perceived as a rigorous mapping problem more than a computer vision problem because the photometric behaviour of the lunar surface, which is the core component of lunar SfS, is well-defined by physically motivated models that have been rigorously and extensively studied for decades (Hapke, 1981; Shkuratov et al., 2011; Soderblom et al., 2006). Because the intensity of an image pixel provides information about the 3D topography of the lunar surface, SfS can produce 3D models up to the spatial resolution of the image. Moreover, because only the photometric content of the image is analysed, SfS does not require stereo images. As a result, the technique performs well with a single image or multiple monocular images, with the latter method known as photometric stereo, which performs best when the images are acquired under different illuminations. However, the major challenge of SfS is that it can only ensure geometric accuracy at the local scale. For large scales, SfS tends to produce inaccurate results because SfS determines local surface gradients from the photometric content of the image pixels. The resulting surface gradients are then affected by the surface reflecting properties, or albedo. Inaccurate determination of location-dependent surface albedo may lead to misinterpretation of surface gradients. For example, a bright flat surface (with higher albedo but lower geometric reflectance) can be misinterpreted as a sunward-facing dark surface (with lower albedo but higher geometric reflectance). Such misinterpretation is common for regions with significant albedo variations, such as lunar swirls and ejecta blankets around fresh craters. The inaccuracy of each pixel then accumulates when a surface is generated from the surface gradients, leading to inaccurate geometry at large 5

Chapter 1 Introduction

scales. In general, SfS has the following characteristics: (1) Surface reconstruction with a spatial resolution comparable to the image; (2) Ability to model the lunar surface from only one image or multiple monocular images; (3) Problems of geometric accuracy at large scales. SfS and photogrammetry naturally complement each other. Taking advantage of the high-resolution reconstruction ability of SfS and the geometric accuracy of photogrammetry/laser altimetry, the existing limitations in 3D modelling of the lunar surface can be largely minimised or solved. With the above situation, this study aims to fuse photogrammetry and SfS for optimised 3D modelling of the lunar surface.

1.3 Contributions and Innovations of the Research

Following these motivations, this study presents integrated photogrammetric and SfS approaches for high-resolution and high-precision 3D modelling of the lunar surface. The major contributions of this research are as follows: (1) Development of a novel single-image SfS (SI-SfS) approach. Given a single monocular image, the SI-SfS approach refines an initial DEM of lower resolution to the optimal resolution of the image. The SI-SfS approach simultaneously retrieves the 3D shape of the lunar surface and its corresponding location-dependent albedo, while the albedo serves as an additional constraint for the shape reconstruction. Image shadows are explicitly considered and introduced to the SI-SfS approach to reduce the defects caused by shadows. (2) Development of a novel photometric stereo SfS (PS-SfS) approach. The PS-SfS approach uses two co-registered images acquired under different illumination conditions to generate high-resolution lunar DEMs. The approach does not require an initial DEM. A novel formulation was developed to remove the effects of albedo variations, leading to more robust performance. Based on the PS-SfS approach, an error propagation model was derived to quantitatively analyse the relationships between the illumination conditions of the images and the reconstruction qualities. (3) Development of an integrated photogrammetric and photoclinometric approach for illumination-invariant 3D modelling of the lunar surface. The integrated

Chapter 1 Introduction

approach can generate lunar DEMs with optimal resolution and is robust to illumination variations. The integration consists of two components. First, PS- SfS is integrated with the image-matching stage of a photogrammetric pipeline, and a novel photoclinometry-assisted image-matching (PAM) approach was developed to create pixel-wise matches, even for images with large illumination differences. Second, the DEM derived from photogrammetry using the matching results is refined to the optimal resolution using the SI-SfS approach. These contributions of this study allow for the generation of accurate lunar DEMs with optimal spatial resolution from single or multiple images. The approaches developed here extend the robustness of photogrammetry to illumination variations and subtle image textures, by using the integrated photogrammetric and photoclinometric approach. They facilitate more sophisticated technology fusion toward the optimal 3D representation of the lunar and other planetary surfaces. The developed approaches offer opportunities to study the lunar surface at a high level of detail, which allow applications to research areas such as morphologic characterisation of very small lunar landforms (Li and Wu, 2018; Yue et al.,2019; Bickel et al., 2020) and its implications to the regolith (Yue et al.,2019; Bickel et al., 2020). The scientific applications of this research can improve the understanding of the Moon’s geologic processes, and provide insights into the evolution of planetary surfaces. Moreover, these approaches were practically applied to the detailed mapping and geomorphological analysis of the landing sites of China’s Chang’ E-4 (Wu et al., 2020) and Chang’ E-5 missions.

1.4 Structure of the Dissertation

The dissertation is structured as shown in Figure 1.1. After the introduction in Chapter 1, Chapter 2 presents a literature review of related research works. In Chapter 3, the SI-SfS approach using a single image and an initial DEM to generate a high-resolution DEM is described and verified with experimental analysis using real data. In Chapter 4, the PS-SfS approach of generating high-resolution DEMs from two co-registered images acquired under different illumination conditions is presented, followed by the mathematical derivation of the error propagation model for PS-SfS. The PS-SfS approach and the subsequent error model were validated by

Chapter 1 Introduction

experimental analysis. In Chapter 5, the integrated photogrammetric and photoclinometric approach is presented and explained. Using the approach, detailed DEMs of the landing sites of China’s Chang’ E-4 and Chang’ E-5 missions were generated. Finally, in Chapter 6 the research works are concluded and discussed, and considerations for future works are addressed.

Figure 1.1 Structure of the dissertation research.

Chapter 2 Literature Review

2.1 Photogrammetric 3D Modelling of the Lunar Surface

2.1.1 Photogrammetry

Photogrammetry is a technology that allows 3D measurements of objects from photographs (Konecny, 1985; Heipke et al., 2007); it generates 3D representations of objects, such as point clouds and DEMs, from a pair or set of overlapping images (Wu, 2017). Modern photogrammetry combines a variety of techniques in order to achieve accurate 3D modelling of the lunar surface. In general, camera calibration is needed to determine the interior orientation (IO) parameters (i.e., the internal configurations) of the sensor, while bundle adjustment is needed to accurately determine the exterior orientation (EO) parameters (i.e., the position and pointing) of the sensor when the images were acquired. Photogrammetry requires a minimum of two images observing the same target object (i.e., the lunar surface) from different viewing angles. By identifying matching points on each of the images that correspond to the same terrain feature, the 3D coordinates of the points can be determined by space intersection. That is, for each image, a ray is constructed according to the IO and EO parameters of the sensor as well as the location of the image point, and then the two rays are intersected in object space to determine the 3D coordinates of the target point. The LROC NAC images have been widely used for 3D modelling of the lunar surface due to their spatial resolution of 0.5 to 1.5 m/pixel, by far the highest for this type of data. The LROC NAC has a special twin-camera configuration that generates a pair of LROC NAC images (i.e., a right image [NAC-R] and a left image [NAC-L]). A 3D model is usually generated from overlapping LROC NAC stereo pairs (i.e., four images). Because of the high popularity of the data, the LROC NAC has been extensively investigated to achieve the best-quality lunar DEMs. Tran et al. (2010) presented the method of generating lunar DEMs from LROC NAC stereo images and compared the accuracy of the NAC DEMs with respect to LOLA. Oberst

Chapter 2 Literature Review et al. (2010) investigated the positional accuracy of NAC DEM using the reference coordinate of the Apollo 17 Lunar Module (LM). They proposed a two-step approach to adjust the camera pointing parameters of the stereo image set, such that best accuracy with respect to the LM reference coordinate can be achieved. Burns et al. (2012) improved the accuracy of NAC DEM by registering it to LOLA elevation profiles. Speyerer et al. (2012; 2016) investigated both pre-flight and in-flight distortions of LROC NAC and LROC WAC, calibrating the focal length, lens distortions, and pointing of both cameras, and the boresight offsets between NAC-R and NAC-L. Wu and Liu (2017) specifically investigated the boresight offsets between NAC-R and NAC-L and proposed an approach to calibrate these offsets. Henriksen et al. (2017) presented a method to create accurate NAC DEMs from LROC NAC images and a method to evaluate their precision and accuracy. Haase et al. (2019) proposed a rigorous bundle adjustment approach, based on a Gauss- Markov model, to improve the geometric accuracy of EO parameters of LROC NAC images and the subsequent NAC DEM. Hu and Wu (2018) proposed a bundle block adjustment approach for the special twin-camera configuration of LROC NAC, in which the NAC-L and NAC-R images are first bundle-adjusted to remove the in- between inconsistency, and then merged by the coupled epipolar rectification algorithm to allow seamless generation of a NAC DEM. Hu and Wu (2019) later presented a software toolkit, Planetary3D, to generate 3D topographic models of planetary surfaces such as the moon and Mars. The LROC NAC images can be processed by Planetary3D, which uses rational polynomial coefficients (RPCs) to model the EO parameters of the images. Planetary3D is equipped with a number of photogrammetric processing pipelines such as epipolar rectification, bundle adjustment, and dense matching for the generation of DEMs from LROC NAC images. To summerise, photogrammetric 3D modelling of the lunar surface using LROC NAC images has been intensively investigated and refined. Nevertheless, the requirement of at least two overlapping images, acquired under desirable stereo geometry and consistent illumination conditions, cannot be compromised in any case because the critical and fundamental theories and concepts of photogrammetry are built upon the concepts of stereo viewing. Such a configuration also facilitates dense image matching for DEM generation.

Chapter 2 Literature Review

2.1.2 Image Matching

Image matching is a critical step in photogrammetry. Successful photogrammetric processing of remote sensing images requires reliable and sufficient matching points. Sufficient, reliable and well-distributed matching points are required for robust bundle adjustment of the images. Dense and reliable matching points are needed for high-resolution 3D modelling of the lunar surface. In general, image matching can be divided into two main categories: feature point matching and pixel-wise dense matching. The first category produces a sparse set of matching points and usually follows a three-step approach: feature detection, feature description and feature matching (Speyerer, 2019). Feature detection is the step that identifies distinctive points (feature points or interest points) on the image, usually at locations of distinctive features on the lunar surface such as craters, rocks or edges of fresh ejecta where apparent albedo changes exist. The feature points are often determined by feature detectors based on local intensity contrast or patterns of the image. For example, the Harris detector (Harris and Stephens, 1988) identifies the locations of image corners where the local image contrast exceeds a certain threshold. Because the evaluation of local image contrast is sensitive to scale (Linderberg, 1998), the Harris-Laplace detector (Mikolajczyk and Schmid, 2002), which is robust to scale, was developed on the basis of the Harris detector. The difference of Gaussian (DoG) detector is also widely used for feature detection (Lowe, 2004; Speyerer, 2019). It identifies blobs on images – dark patches surrounded by bright background or vice versa – based on the Laplacian of Gaussian response of the image (Linderberg, 1998). Because detection is performed on scale- spaces, the DoG detector is robust to scale and can determine the best scale for each feature point (Linderberg, 1998; Lowe, 2004). Feature description is the step that encodes the characteristics and unique information about the feature points (i.e., feature descriptors) that will be used for matching (Lowe, 2004; Speyerer, 2019). Feature descriptors are usually generated based on image patterns within a local neighbourhood of the feature point (Lowe, 2004; Banz et al., 2012). For example, the simplest form of feature descriptor is the image intensity of the pixel itself (Birchfield and Tomasi, 1998; Hirschmüller, 2008; Wu et al., 2018a), which can be used for dense and pixel-wise matching. The scale-

Chapter 2 Literature Review invariant feature transform (SIFT) (Lowe, 2004) and the histogram of oriented gradients (HOG) (Dalal and Triggs, 2005) describe features by the statistics of image gradients and orientations. The normalised cross-correlation (NCC) describes a feature by the normalised intensity difference with respect to the mean intensity of the local neighbourhood or template and is widely used in template matching and image matching (Lewis, 1995; Zhao et al., 2006; Wan et al., 2014). Order-based descriptors such as rank-transform and census-transform encode features by their relative intensity order within the local neighbourhood (Banz et al., 2012). Rank- transform converts the intensity values of a local neighbourhood of each pixel into a sorted rank, whilst census-transform compares the intensity values of the local neighbourhood with that of the centre pixel; it assigns zeros for values less than or equal to that of the centre pixel and assigns ones for values above that of the centre pixel. Order-based descriptors are computationally less costly and are hence popular for dense and pixel-wise matching (Banz et al., 2012). Feature matching is the step in which the feature descriptors of the feature points, generated by the aforementioned step, are compared for each image. The matching step produces matching costs (the level of dissimilarity) or matching scores (the level of similarity). Birchfield and Tomasi (1998) proposed an approach to compute the matching cost by the image intensity differences between a pair of candidate pixels, based on Euclidean distance. SIFT (Lowe, 2004) evaluates matching cost by the Euclidean distance of the feature descriptors and proposes a fast nearest-neighbour searching algorithm to efficiently obtain the best match. NCC produces matching scores based on the correlation between feature descriptors, which can be explained intuitively as the cosine of the angle between the feature descriptor vectors. For order-based descriptors such as census-transform, the Hamming distance is usually used (Hirschmüller, 2008; Banz et al., 2012). The Hamming distance evaluates the distance between feature descriptors by the sum of bit difference per entry of the descriptor vectors. The second category, pixel-wise dense matching, produces a match for each pixel on the image pair, and it outperforms feature point matching in terms of the matching point density. Due to the challenges and complexity of pixel-wise matching, the images are usually processed; for example, by epipolar rectification (Hirschmüller, 2008), such that scale and rotation inconsistencies are minimised, retaining only the translational parameter (a.k.a. disparity) to be solved by matching. 12

Chapter 2 Literature Review

Because each pixel will be evaluated and matched, pixel-wise matching approaches do not have the feature detection step, but the remaining feature description and feature matching steps are required. For feature description, the approach must assign a feature descriptor for each pixel of the image, which can be challenging and computationally costly. As a result, less-costly descriptors such as the image pixel intensity (Birchfield and Tomasi, 1998) and order-based descriptors (Hirschmüller, 2008; Banz et al., 2012) are preferred. Direct comparison of image intensity values has another advantage of being robust to discontinuities because the local neighbourhood is not considered (Birchfield and Tomasi, 1998). For the feature- matching step, pixel-wise matching approaches usually attempt to solve for a global transformation model (Hirschmüller, 2008; Grumpe et al., 2014), such that the total matching cost for the whole image achieves a minimum. The transformation model can be parametric (e.g., affine model, perspective model) (Grumpe et al., 2014) or non-parametric (e.g., semi-global matching, SGM) (Hirschmüller, 2008). Because pixel-wise matching is more sensitive to image noise and subtle textures, a certain level of smoothness constraint is required to avoid severe mismatches. Such a constraint is reasonable for planetary images because the planetary surface is usually natural and smooth. For pixel-wise matching based on parametric models such as in Grumpe et al. (2014), the smoothness constraint is enforced by the transformation model itself. In contrast, the global transformation model of SGM is non-parametric (Hirschmüller, 2008); hence, to achieve an approximation of the global model, SGM aggregates the matching costs from multiple paths, hence the name “semi-global.” The smoothness constraint is imposed by adding penalty costs to the final matching cost. Both categories of image-matching approaches excel in different aspects. Feature point matching approaches are less computationally costly because only sparse points are evaluated. They are also more robust to scale, rotation and perspective variations, which are common in stereo images (Speyerer, 2019). As a result, feature point matching approaches are often used in bundle adjustment to determine the EO parameters for photogrammetric processing. In contrast, pixel- wise matching approaches are critical for generating high-resolution 3D models because matching points can be obtained for each pixel. However, these algorithms require pre-processing of images to minimise factors such as scale and rotation. Both approaches can be integrated for high-resolution photogrammetric 3D modelling, by 13

Chapter 2 Literature Review first using feature point matching to obtain a sparse set of reliable matching points for bundle adjustment, and then, based on the resulting EO parameters, pixel-wise dense matching points can be obtained from rectified images and used to generate 3D models. The performance of image-matching approaches is affected by a variety of factors, such as variations in scale, rotation, texture and illumination (Lowe, 2004; Wu et al. 2011; Speyerer, 2019). In particular, Speyerer (2019) investigated the robustness of a number of feature point–matching algorithms commonly used in lunar and planetary mapping, such as Speeded-Up Robust Features (SURF) and KAZE, against the aforementioned factors. They concluded that most feature point– matching algorithms are usually able to handle different levels of scale and rotation variations and limited illumination variations (Speyerer, 2019). However, these methods require sufficient image texture to identify dense and robust points. In regions in which the images are poorly textured or contain repeated similar patterns, these algorithms are prone to failure (Heipke, 2007; Grumpe et al., 2014). For images with subtle textures, Wu et al. (2011; 2012) proposed a self-adaptive triangulation-based strategy to obtain robust matches from poorly textured images. Given a set of initial matching points, a triangulated irregular network (TIN) is constructed, whose corresponding facets will be used to constrain and refine the search area of candidate matches. The TIN will be gradually densified by the newly obtained matching points. Because most of the image-matching algorithms determine the best match based on image intensity or patterns created by image intensities, they are likely to fail when the images have obvious differences in illumination (Grumpe et al., 2014; Preusker, 2017; 2018) because the changes in illumination lead to complex changes in the intensities of the image pixels and the subsequent image patterns (Wu et al., 2018a). To overcome this problem, Wu et al. (2018a) proposed an illumination-invariant SIFT by suppressing the feature descriptor based on the distribution of the dominant orientation of the feature points. However, this approach can only be used for feature matching. Foroosh et al. (2002) investigated image matching based on the phase component in the frequency space instead of the image space. They showed that the frequency components of images are much less sensitive to illumination changes. The developed approaches are known as phase-correlation or phase-congruency matching. Wan et al. (2014; 2015) investigated the robustness of phase-correlation against illumination variations and 14

Chapter 2 Literature Review proposed a phase-correlation–based matching approach invariant to illumination variations. Wan et al. (2018) developed the illumination invariant change detection (IICD) on the basis of Wan et al. (2014; 2015) and applied the IICD approach for change detection of the Martian surface. Xie et al. (2016) improved the accuracy of phase-correlation matching to the sub-pixel level. Phase-based approaches perform better when the viewing conditions remain the same, which may not be feasible for stereo images. This problem has also been addressed by using SfS (Grumpe and Wöhler, 2011; Grumpe et al., 2014), in which the underlying surface gradients of each image are estimated individually and matched accordingly. Because surface gradients are illumination-invariant representations of the image content, image matching based on surface gradients is invariant to illumination changes. The SfS- based method proposed by Grumpe et al. (2014) achieved pixel-wise matching by solving for the best parameters of a predefined transformation model, such as a polynomial model, between the images. The parameters are determined based on matched interest points or mutual information. However, such transformation is not appropriate for co-registering high-resolution stereo images because stereo images are connected by complex epipolar geometry and a simple transformation model will likely create erroneous co-registration of certain regions. To summarise, the creation of high-resolution lunar DEMs using photogrammetry requires dense and accurate matching points that are robust to variations in illumination. The current approaches are either not invariant to illumination variations or are unable to create pixel-wise matches suitable for high- resolution images. There is thus a need to develop effective pixel-wise image- matching approaches that are invariant to illumination variations.

2.2 Photoclinometric 3D Modelling of the Lunar Surface

2.2.1 Reflectance Models

A reflectance model describes the relationship between the inclination of a surface, with respect to the illumination and viewing geometry, and the brightness perceived by the sensor. The photometric relationship for image acquisition on a

Chapter 2 Literature Review surface is illustrated in Figure 2.1. Assuming that a single light source from a certain direction L illuminates a point O on the surface, the incidence angle (i, ∠N-O-L) is formed between the vector pointing to the light source and the normal vector of the surface (N). The surface absorbs some of the light energy and reflects/scatters the remaining light in different directions. When an observer (e.g., a camera) is viewing from a direction C, the emission angle (e, ∠N-O-C) is formed between the vector pointing to the observer and the normal vector of the surface. The phase angle (g, ∠ L-O-C) is formed between the incident light ray and the emission ray. The incidence angle, emission angle, and phase angle are not necessarily coplanar; hence in most cases 𝑖 + 푒 ≠ 𝑔. The amount of light received by the observer depends on the incidence angle and the emission angle; and the surface reflecting properties, or albedo. In general, the photometric relationships can be modelled as:

퐼 = 퐴퐺(푝, 푞), (2.1) where the image intensity I is the product of the surface reflecting properties (i.e., albedo A) and the reflectance G(p,q) generated from the surface Z(x,y), and p and q denote the normal vector of the surface along the x-direction and y-direction, respectively. The normal vector of the surface is the negative form of the surface 휕푍 휕푍 gradients (i.e., 푝 = − ; 푞 = − ). The function G(p,q) is known as the reflectance 휕푥 휕푦 model, which describes how a surface reflects incoming energy to an observer. It is important to note that the albedo A may refer to slightly different concepts in different reflectance models depending upon their formulations and definitions. For example, the Lambert model (Lambert, 1760) referred to albedo as the proportion of incident solar irradiance reflected by the surface into solid angle π (Soderblom et al., 2006), while the Lunar-Lambert model (McEwen, 1986; 1991) referred to albedo as intrinsic albedo, which is a combination of factors such as surface reflectivity, porosity, and macroscopic roughness.

Chapter 2 Literature Review

Figure 2.1 Conceptual illustration of the photometric angles during image acquisition.

Reflectance models for modelling the photometric properties of the lunar and planetary surface have been extensively studied. The Lambert model (Lambert, 1760) is one of the early models used to describe the relationship between surface geometry and reflectance (Soberblom et al., 2006). The Lambert model defines surface reflectance as the cosine of the incidence angle:

퐿푇푁 퐺 (푝, 푞) = 휇 (푝, 푞) = , (2.2) 퐿푎푚 0 ‖퐿‖‖푁‖

푝퐿 where 퐿 = [푞퐿] is the vector pointing to the direction of solar irradiance, and 1 푝 푁 = [푞] is the normal vector of the surface. The Lambert reflectance is hence the 1 normalised dot product of the two vectors. The Minnaert model (Minnaert, 1941) is developed on the basis of the Lambert model (Soberblom et al., 2006) and is defined as follows:

푘 푘−1 퐺푚푖푛푛(푝, 푞) = 휋 휇0(푝, 푞) 휇(푝, 푞) 푉푇푁 휇(푝, 푞) = , (2.3) ‖푉‖‖푁‖

Chapter 2 Literature Review

푝푉 where 푉 = [푞푉] is the vector pointing to the direction of the sensor, hence 휇(푝, 푞) is 1 the cosine of the emission angle, and k is a constant controlling the contribution of the incidence term and emission term. Compared to the Lambert model, the Minnaert model has taken viewing conditions into account (Minnaert, 1941; Soberblom et al., 2006). Later, the Lommel-Seelinger model (Chandrasekhar, 1960) was developed and is defined as follows:

휇0(푝,푞) 퐺퐿푆(푝, 푞) = (2.4) 휇0(푝,푞)+휇(푝,푞)

This model was developed to describe the moon’s reflectance behaviour (Chandrasekhar, 1960; Soberblom et al., 2006) and facilitates a number of widely used reflectance models in planetary photometry. For example, the Lunar-Lambert model (Meador and Weaver, 1975; McEwen 1986; 1991) is a linear combination of the Lambert model (equation 2.2) and the Lommel-Seelinger model (equation 2.4), as follows:

퐺퐿퐿(푝, 푞) = 푐1퐺퐿푎푚(푝, 푞) + 푐2퐺퐿푆(푝, 푞), (2.5)

where c1 and c2 are empirical coefficients that control the relative contributions of the Lambert component and Lommel-Seelinger component, respectively. The coefficients can depend on the phase angle (McEwen, 1986; 1991). McEwen (1986) presented a reformulated version of the Lunar-Lambert model for the use of SfS as follows:

퐺퐿퐿(푝, 푞) = (1 − 휆)퐺퐿푎푚(푝, 푞) + 2휆퐺퐿푆(푝, 푞), (2.6) where λ is a phase-dependent coefficient generated by an empirical phase function, which can be modelled by a polynomial function (Lohse et al., 2006) or an exponential function (Gaskell, 2008). The Hapke model (Hapke, 1981; 1986; 2002; 2012) has been widely adopted for photometric and surface analysis of the moon (Labarre et al., 2017) and other planetary bodies such as Mars (Soderblom et al., 2006; Hess et al., 2019) and Mercury (Warell, 2004). The Hapke model is defined as

Chapter 2 Literature Review follows:

1 퐺 (푝, 푞) = 퐺 (푝, 푞)휑퐵 + 푀(푝, 푞), (2.7) 퐻푎푝푘푒 4 퐿푆 푠ℎ where φ is the output of a single particle phase function, such as the Henyey- Greenstein function (Henyey and Greenstein, 1941), which determines how a particle reflects incident energy depending on the phase angle. Bsh is a phase- dependent coefficient used to handle shadow-hiding opposition effects (SHOE), and M(p,q) is a function that accounts for multiple scattering (i.e., incident energy may bounce multiple times within the surface layer). The Hapke model has been critically tested and evaluated (Zhang and Voss, 2011; Shkuratov et al., 2012; 2013; Hapke, 2013; Ciarniello et al., 2014) and was proven to accurately fit the photometric behaviour of a wide spectrum of planetary surfaces (Soderblom et al., 2006). Because the Hapke model contains a relatively large number of interdependent parameters (Gehrke, 2008; Shkuratov et al., 2012; 2013), it is difficult and computationally complex to determine the correct photometric parameters in a robust manner (McEwen, 1991; Gehrke, 2008; Shkuratov et al., 2012; 2013). As a result, simpler models, such as the Minnaert model (equation 2.3) and the Lunar-Lambert model (equation 2.6), are often used as an approximation of the Hapke model (McEwen, 1991; Soderblom et al., 2006; Gehrke, 2008). McEwen (1991) showed that the Lunar-Lambert model can adequately approximate the Hapke model for the lunar surface within a reasonable range of macroscopic roughness. Soderblom et al. (2006) also showed that the Lunar-Lambert model can approximate the Hapke model for the Martian surface. The Lunar-Lambert model is generally preferred for applications such as SfS for its lower complexity and hence its computational efficiency. The Shkuratov model (Shkuratov et al., 1999) is another physically derived reflectance model used for lunar photometry (Shkuratov et al., 2011) and differs from the Hapke model by the modelling of the phase function (Grumpe et al., 2014). The reflectance models used for planetary photometry have undergone a series of critical assessments and improvements. For example, the Hapke model (Hapke, 1981) is modified by taking into account the effects of macroscopic roughness (Hapke, 1984) and later the SHOE and anisotropic scattering effects

Chapter 2 Literature Review

(Hapke, 2002). Based on the Hapke (1984) model, Labarre et al. (2017) further investigated the macroscopic roughness and proposed a simplified version of the model. Regarding macroscopic roughness, Oren and Nayar (1994) improved the Lambert model and proposed macroscopic roughness corrections based on statistical estimation of the moon’s surface roughness. The modified reflectance models have also been used in later studies in planetary photometry such as O’Hara and Barnes (2012). The photometry of planetary bodies other than the moon has also been studied. For example, Soderblom et al. (2006) studied the photometric behaviour of the Martian surface and suggested reflectance models, such as the Lunar-Lambert model, appropriate for Martian applications. Fernando et al. (2015) used the Hapke model and the data obtained by Compact Reconnaissance Imaging Spectrometer for Mars (CRISM) to study the photometric and geologic properties of particular regions on Mars. Belgaccem et al. (2019) studied the regional photometry of Europa and suggested that the photometric behaviour of the surface of Europa is very diverse. Longobardo et al. (2016; 2019) studied the photometry of asteroids such as Vesta and Lutetia, as well as Ceres.

2.2.2 Shape-from-Shading (Photoclinometry)

Shape-from-Shading, or photoclinometry, has been extensively studied in the past (van Diggelen, 1951; Rindfleisch, 1966), but mostly for close-range 3D modelling applications. The basic idea of SfS is illustrated in Figure 2.2. When the surface is illuminated by an energy source, such as the sun, the incoming ray interacts with the surface, a portion of which is reflected/scattered to the sensor and is then recorded as an image. The amount of reflected/scattered energy received by the sensor is determined by the geometry and the photometric properties (e.g., albedo) of the surface, and the position of the sun and the sensor relative to the surface. SfS is the inverse problem of the above process, which determines the gradients of the surface by the intensity values of the image and eventually converting the surface gradients into a 3D model. Mathematically the problem can be formulated as follows:

2 arg 푚𝑖푛푝,푞 (퐴퐺(푝, 푞) − 퐼) , (2.8)

Chapter 2 Literature Review where I is the image intensity measured on the image, for radiometrically calibrated planetary images such as the LROC NAC images, the image values are usually referring to radiance factor (I/F) or radiance. In the field of computer vision, SfS has been intensively studied for close-range scene reconstruction applications (Chandraker et al., 2007; Barron and Malik, 2012; Wang et al., 2016), in which SfS is considered as a specific intrinsic image decomposition problem (Barron and Malik, 2012). The technology is known for recovering pixel-wise surface shapes from a single image (Grumpe et al., 2014) and it is of interest to applications such as one- shot planetary imagery acquired by fly-by missions and orbital imagery without suitable photogrammetric stereo settings (Kirk, 1987). Early works on photoclinometry focused on the reconstruction of height profiles from images (van Diggelen, 1951; Rindfleisch, 1966) because the surface slope along the illumination direction has a dominating effect on the resulting image reflectance (Horn, 1977; 1990). Profiles along lines parallel to the illumination direction are called characteristic stripes (Horn, 1990). A two-dimensional surface, such as a DEM, is photoclinometrically reconstructed by combining adjacent reconstructed characteristic stripes. Profile-based reconstruction would present visible inconsistencies between adjacent stripes because of albedo variations (Kirk, 1987; Kirk et al., 2003) and the lack of connections between each characteristic stripe. These artefacts can be addressed by using different filters to consecutively convolve the resulting DEM (Kirk et al., 2003). Another alternative is to consider photoclinometry as a two-dimensional reconstruction problem that simultaneously solves for surface gradients along both x- and y- directions, which are then converted to a DEM (Kirk, 1987; Horn, 1990). Kirk (1987) addressed the problem by rotating the image such that the phase plane (i.e., the plane that contains the phase angle) aligns with the horizontal axis of the coordinate system. When the image is nadir viewing, it is equivalent to aligning the illumination direction to the horizontal axis. Horn (1990) did not rotate the image and developed a sequence of computations in estimating the surface gradients and reconstructing the subsequent DEM. Since then the difference between photoclinometry and SfS is less distinct (Grumpe et al., 2014). The results of these approaches usually have errors associated with the direction of illumination because the image cannot supply sufficient information to constrain the estimation of gradients along lines orthogonal to the direction of illumination.

Chapter 2 Literature Review

Figure 2.2 Conceptual illustration of SfS.

The performance of SfS is affected by uncertainties from various sources. These error sources were critically reviewed by Jankowski and Squyres (1991), which include errors from imaging devices (e.g., noise) and the photometric properties of the planetary body (e.g., albedo and reflectance models). Greenberg et al. (2011) investigated the relationship between macroscopic roughness and the quality of SfS and highlighted its importance for high-quality SfS. Kirk (1987) suggested that the major contributing errors in SfS are the accuracy of the reflectance model and local variations in albedo. The former issue does not pose a severe challenge because most of the planetary bodies being studied can be well described by the Hapke model or equivalents with appropriate parameters. The latter issue cannot be avoided without extra information or regularisation because of the fact that simultaneous estimation of surface shape and location-dependent albedo from a single image is a mathematically ill-posed problem. This issue is addressed in a number of approaches by assuming a constant albedo, they are usually restricted to small local regions where variations in albedo are negligible (Horn, 1990; Lohse and Heipke, 2004) or designed for studying specific features such as volcanic plumes (Glaze, 1999), and lunar domes (Wöhler et al., 2006). Attempts to incorporate location-dependent albedo involve additional assumptions to regularise albedo variations. For example, Lee and Rosenfeld (1983) proposed an approach of

Chapter 2 Literature Review location-dependent albedo estimation by assuming that the variations in albedo are negligible in a small locality. Tsai and Shah (1998) incorporated this algorithm to their SfS method (Tsai and Shah, 1994) for 3D reconstruction of close-range objects. Barron and Malik (2011; 2012) incorporated a statistical assumption of albedo variations, which is derived from studying images of natural scenes, to an SfS framework and applied the SfS approach to both close-range and planetary images. Danzl and Scherer (2002) and Grumpe et al. (2014) incorporated DEMs of lower resolution to regularise the SfS problem. The initial DEM constrains the resulting 3D model and its corresponding location-dependent albedo, which is gradually refined by a smoothing algorithm with decreasing kernel width (Grumpe et al., 2014). These approaches may require manual inspection of the resulting albedo to ensure that it does not contain apparent topographic shading, and terminate the optimisation of albedo whenever necessary. SfS using multiple images, with each of the images acquired under a different illumination condition, is referred to as photometric stereo (Woodham, 1980). Photometric stereo SfS (PS-SfS) significantly relaxes the problem of mathematical insufficiency by providing multiple observations for each pixel. The surface gradients and location-dependent albedo of each co-registered pixel can be uniquely recovered by a minimal of three images (Woodham, 1980; Chandraker et al., 2007). Piechullek and Heipke (1996) developed a PS-SfS approach for generating high- resolution DEMs from photometric stereo images, which was extended and applied for topographic mapping using real aerial images by Piechullek et al. (1998). The minimal number of images used by PS-SfS can be reduced to two when the albedo is known or when only the surface gradients along a particular direction are required to be recovered. For example, Wöhler (2004) investigated a special case of photometric stereo which is referred to as co-planar light source. This situation usually occurs in equatorial regions of the moon where the surface is illuminated from either the East or the West. In this case, only the surface gradients along the light source plane (the longitudinal surface gradients in the case of the moon) can be accurately determined by using the ratio of image intensities which cancels out the effect of surface albedo. Liu and Wu (2017) and Liu et al. (2018) extended the approach described by Wöhler (2004) to photometric stereo images of non-coplanar light source and proposed a PS- SfS approach using two images, they also investigated the performance of PS-SfS with respect to variations in illumination. The robustness and ability to recover 23

Chapter 2 Literature Review surface details make PS-SfS well-suited for applications in modelling and analysis of fine features such as bones (Wu et al., 2010) and leaves (Zhang et al., 2018), where the imaging and illumination geometry is well-controlled. The major challenges of deploying PS-SfS into larger real-world scenes are the availability of multiple illumination sources and the accuracy of pixel-wise image registration under inconsistent illumination. For Earth-applications, these challenges are overcome by acquiring daily or seasonal image sequences from static cameras such as webcams or surveillance cameras (Ackermann et al., 2012; Jung et al., 2015). For planetary applications, Lohse et al. (2006) investigated the use of PS-SfS, which is referred to as multi-image SfS by the authors, in generating high-resolution DEMs of planetary surfaces. The algorithm uses a camera model to geometrically connect each node of the DEM and the corresponding image pixel on each of the images, and then iteratively refines the DEM such that the reflectance produced by the DEM best matches the intensity value of the pseudo-co-registered image pixels. This method requires accurate EO parameters of the camera model and an accurate DEM, while the former can be achieved using bundle adjustment, the latter is usually not feasible at high-resolution. The convergence radius in DEM height of the algorithm is approximately 4 times the horizontal spatial resolution (Lohse et al., 2006), which is not favourable for high-resolution images such as LROC NAC (0.5-1.5 m/pixel). Alexandrov et al. (2018) overcome the problem by adopting a coarse-to-fine strategy, in which the images are appropriately down-sampled with respect to the resolution of the DEM. Their approach successfully reconstructed high-resolution lunar DEMs (1 m/pixel) from an initial lunar DEM derived from laser altimetry (approx. 60 m/pixel at the lunar equator). To generate a DEM from the surface gradients recovered using SfS or PS-SfS, the surface gradients must be integrated (Horn 1990; Grumpe et al., 2014). For the one-dimensional case in which height profiles are created, the integration is straightforward by summing up the surface slopes between any two points. For a two-dimensional case, however, the integrability condition must be considered. The integrability condition is formulated as follows (Horn, 1990):

휕2푍(푥,푦) 휕2푍(푥,푦) = , (2.9) 휕푥휕푦 휕푦휕푥

Chapter 2 Literature Review where Z(x,y) refers to the DEM. The integrability condition suggests that the height difference between any two points on a continuous surface (e.g., the DEM) is independent of the path chosen. This is particularly important in SfS because the surface gradients at each pixel are computed independently and are very likely to be non-integrable (Frankot and Chellappa, 1988; Horn, 1990). The integrability problem is generally overcome by creating a DEM, which is perfectly integrable by nature, that best represents the SfS-computed surface gradients and can be formulated as follows:

휕푍(푥,푦) 2 휕푍(푥,푦) 2 arg 푚𝑖푛 ∑ [(− − 푝 ) + (− − 푞 ) ] , (2.10) 푍 푖 휕푥 푆푓푆 휕푦 푆푓푆 푖

푝푆푓푆 where i refers to each of the height nodes of the DEM, and [ ] refers to the 푞푆푓푆 surface gradients estimated from SfS. The solution for equation 2.10 can be approximated by using relaxation algorithms (Kirk, 1987; Horn, 1990; Tsai and Shah, 1994), for which only a finite set of height nodes are adjusted at a time while other nodes remain constant. Relaxation algorithms are particularly suitable for SfS because each surface gradient affects only a local neighbourhood, leading to a sparse global system (Kirk, 1987). Horn (1990) proposed the Euler equation as a solution to equation 2.10, as follows:

휕2푍(푥,푦) 휕2푍(푥,푦) 휕푝 휕푞 Δ푍(푥, 푦) = + = − ( + ), (2.11) 휕푥2 휕푦2 휕푥 휕푦 where Δ푍(푥, 푦) is defined as the Laplacian of the DEM at node (x,y). Intuitively, because the Laplacian is the sum of second-order derivatives of the DEM, the reconstruction emphasises the local variations of surface gradients. Horn (1990) proposed an iterative method to reconstruct a DEM using equation 2.11 and a discrete approximation of the Laplacian, assuming that the DEM has a square grid. Grumpe et al. (2014) extended the method to account for non-square DEM grids. The methods by Horn (1990) and Grumpe et al. (2014) can be applied to the data by a series of kernel filtering operations. Frankot and Chellappa (1988) approached the problem by projecting the non-integrable surface gradients onto a set of integrable base functions. They used the Fourier series as the base function set because doing 25

Chapter 2 Literature Review so allows for efficient computation. The algorithm fits a DEM, generated by a series of Fourier functions, to the surface gradients estimated from SfS. SfS approaches using the above DEM reconstruction algorithms generally follow a two-step strategy that includes (1) estimation of surface gradients by SfS and (2) reconstruction of the DEM based on the surface gradients (Shape-from-Gradients) (Frankot and Chellappa, 1988). Another alternative to the problem exists, which is to directly connect the DEM height to the reflectance model and optimise the DEM height nodes directly based on the image intensity (Tsai and Shah, 1994; 1998; Alexandrov et al., 2018). Mathematically, such an approach can be formulated as follows:

휕푍 휕푍 2 arg 푚𝑖푛 (퐴퐺 (− , − ) − 퐼) (2.12) 푍 휕푥 휕푦

According to Alexandrov et al. (2018), such formulation achieves better results than the conventional two-step strategy, likely because the integrability condition is implicitly but strictly enforced. In contrast, such a formulation also increases the complexity of the problem, because the connection between DEM height and reflectance is less direct, which in turn requires better initial DEMs or multi- resolution strategies to increase the robustness of the approaches.

2.3 Fusion of Photogrammetry and Shape-from-Shading for 3D Modelling of the Lunar Surface

The Shape-from-Shading method is known to have the remarkable ability of local detail recovery, but it lacks stable geometric accuracy at large scales, especially when compared to photogrammetry and laser altimetry. For accurate high-resolution 3D modelling of the lunar surface, the technologies are naturally complementary. The fusion of photogrammetry and SfS, in general, can be divided into two categories: fusion based on the topographic products of photogrammetry such as DEMs, and fusion of SfS into a photogrammetric pipeline to improve and extend the power of photogrammetry. For the first type, which is the more common application, the integration is focused on using the products of photogrammetry such as DEMs and ortho-images; hence, this type of integration is also known as integration with

Chapter 2 Literature Review depth/height data (Grumpe et al., 2014). The incorporation of the initial DEMs derived from photogrammetry or laser altimetry can provide accurate initial shape estimates for SfS, plausible estimates for location-dependent albedo and geometric constraints during the SfS process. Kirk (1987; 2003) highlighted the advantages of using an initial DEM as a constraint to the SfS problem. Frankot and Chellappa (1988) suggested using the low-frequency components of the initial DEM to compensate for the low geometric accuracy at large scales of the SfS and Shape- from-Gradients methods. Soderblom et al. (2002) proposed a method that combines SfS with laser tracks from the Mars Orbiter Laser Altimeter (MOLA), which are height profiles of lower spatial resolution. The photometric parameters are adjusted such that the high-resolution height profile, reconstructed by SfS from high- resolution Mars imagery, corresponds with the MOLA profile. Soderblom and Kirk (2003) extended the method from processing profiles to processing DEMs, for which the MOLA DEM was used as constraints, and generated a high-resolution DEM. These approaches provide regularisation over photometric parameters during SfS, but they do not possess explicit constraints over the geometric correspondence between the initial 3D model and the reconstructed high-resolution 3D model. Kirk et al. (2003a) introduced the method into USGS’s Integrated Software for Imagers and Spectrometers (ISIS) pipeline. Horovitz and Kiryati (2004) used sparse height posts to constrain the reconstruction, in which a number of weighting functions are considered to control the contributions of the sparse control points. Barron and Malik (2011) imposed constraints based on the initial DEM, and other geometric constraints such as smoothness, to reconstruct a high-resolution DEM based on SfS. The photometric parameters were estimated based on the statistics of photometric behaviour in natural scenes. Grumpe et al. (2014; 2015) introduced explicit constraints such that the low-resolution components of the reconstructed DEM approached that of the initial DEM. Their method used the initial DEM to constrain both the surface gradients and the absolute height of the solution. Wu et al. (2017) proposed explicit surface gradient constraints from the initial DEM and an interactive multiple-resolution reconstruction strategy to strengthen the performance of SfS. Heipke (2004) and Lohse et al. (2006) introduced the camera model into SfS. First, a pseudo-orthoimage was created based on an initial DEM and the camera model, and the DEM was optimised according to SfS. This method is particularly effective when oblique images are used because for oblique images, changes in 27

Chapter 2 Literature Review surface height will have more significant changes in image coordinates and its subsequent intensity value, whilst this effect is much less significant for nadir images. The incorporation of camera models into SfS requires sufficiently accurate camera EO parameters, which define the transformation between a point in 3D object space and its corresponding point on the image and hence ensuring the correct extraction of the image intensity values. The convergence radius of the method is approximately 4 times that of the spatial resolution (Lohse et al., 2006), which means that for a DEM with a spatial resolution of 1 m/pixel, the method can obtain a solution if the initial height is within 4 m of the true height. As a result, a hierarchical architecture is introduced to allow for optimisation at a lower spatial resolution when the initial DEM does not meet the criteria. This architecture has been followed by topographic and atmospheric modelling of the Martian surface with an algorithm known as stereo facet vision (Gehrke, 2008; Gehrke and Haase, 2019). Alexandrov et al. (2018) extended the concept and included SfS of planetary surfaces using single or multiple images integrated with photogrammetric processing routines into NASA’s Ames Stereo Pipeline. Gaskell et al. (2008) developed a method known as stereophotoclinometry for modelling asteroids. Stereophotoclinometry requires an initial 3D model of the asteroid, a camera model, and the EO information of each of the images. Similarly, pseudo-orthoimages are created and SfS is performed based on the pseudo-orthoimages and the initial model. Moreover, stereophotoclinometry introduced an explicit geometric constraint where the reconstructed surface model is encouraged towards the initial surface model. Stereophotoclinometry has recently been applied to study Ceres (Park et al., 2019). The second type of integration aims to improve and extend the power of photogrammetry, which focuses on illumination-invariant image matching. In general, modern photogrammetry relies heavily on dense and robust image matching for determining accurate EO parameters and producing high-resolution 3D models. For images with inconsistent illumination, such as one image with the sunlight coming from the East and another with sunlight coming from the West, or when images are poorly textured, most image-matching algorithms are likely to fail (Heipke, 2004; Grumpe et al., 2014). As a result, the use of photogrammetry is hindered by the limitations of image matching. Performing SfS on the basis of photogrammetrically derived 3D models does not solve this problem because photogrammetric DEMs cannot be derived using inconsistently illuminated images. 28

Chapter 2 Literature Review

Grumpe et al. (2014) addressed the problem by proposing an image-matching algorithm based on the high-resolution surface gradient maps derived from SfS, which has been discussed in section 2.1.2. This contribution can be seen as an integration of photogrammetry and SfS, in the way that it allows photogrammetric processing invariant to illumination inconsistency. To summarise, most approaches regarding the fusion of photogrammetry and SfS focus on the use of photogrammetric 3D products (e.g., DEMs or 3D points) to constrain SfS. Although such fusion allows significant improvements in the spatial resolution of the topographic products, the robustness of photogrammetry to illumination variations is largely ignored. With the knowledge that SfS can provide solutions to the problem of illumination variations in photogrammetry, there is a need to develop an illumination-invariant pixel-wise image-matching approach for photogrammetric pipelines.

2.4 Summary

From the reviewed existing works, a few key points can be summarised. First, photogrammetry requires at least two stereo images, and this cannot be compromised in any case. However, factors that decrease the robustness of photogrammetry due to the limitations of image matching, such as subtle image textures and illumination variations, can be overcome by fusing photogrammetry with SfS. A summary of the current situation of image matching for lunar and planetary mapping is presented in Figure 2.4a. Second, most feature point–matching approaches have limited robustness against illumination variations (Speyerer, 2019). Focusing on pixel-wise matching, which is the key technique for high-resolution 3D modelling, illumination-invariant approaches are generally given by phase correlation (Foroosh et al., 2002; Wan et al., 2014; 2015; 2018) and SfS-based slope matching (Grumpe and Wöhler, 2011; Grumpe et al., 2014). Whilst the phase correlation approach is not suitable for stereo images, the SfS-based approach requires a parametric transformation model, such as a polynomial function, to achieve pixel-wise matching. Such transformations may produce suboptimal matching results, especially when matching high-resolution stereo images such as the LROC NAC images. In contrast, better pixel-wise approaches such as SGM do not require a parametric

Chapter 2 Literature Review transformation model and allow pixel-wise evaluation but are not invariant to illumination due to the formulation of matching cost. As a result, the current situation necessitates the development of an image matching algorithm that is invariant to illumination variations and is favourable for photogrammetric pipelines. SfS and photogrammetry are natural complements, as shown in Figure 2.3, and Figure 2.5a summarises the current situation of fusion of photogrammetry and SfS for 3D reconstruction of lunar and planetary surfaces. It is noted that most methods emphasise fusion with 3D data, whilst fusion with the photogrammetric pipeline, as a way to empower the performance and applicability of photogrammetry, is less common. This situation implies the potential to exploit various forms of fusing photogrammetry and SfS. Moreover, recent methods for multiple-image SfS that require accurate image EO were presented by Alexandrov et al. (2018), and Gaskell (2008) introduced a method designed for asteroids. Consequently, multiple- image SfS for the lunar surface is less developed than single-image SfS approaches, implying that there is room for development. This study fills the gap in existing methods of 3D modelling of the lunar surface. First, photogrammetric methods usually produce 3D models of the lunar surface at suboptimal spatial resolutions, compared to the underlying images. This limitation is addressed in this research by developing approaches that generate 3D models of the lunar surface far more detailed than any photogrammetric methods while ensuring plausible geometric accuracy. Second, as shown in Figure 2.4b, the existing pixel-wise image matching approaches are not favourable to illumination variations and high-resolution stereo images. This limitation is addressed in this study by developing a pixel-wise image matching algorithm invariant to illumination variations and specifically designed for photogrammetric pipelines. Third, as illustrated in Figure 2.5b, most existing approaches that fuse SfS and photogrammetric pipelines cannot extend the robustness of photogrammetric 3D modelling. This limitation is addressed in this research by developing a fusion approach that allows reliable photogrammetric 3D modelling invariant to significant differences in illumination. By addressing the aforementioned limitations, it is possible to generate lunar 3D models with global/near-global spatial coverage and optimal spatial resolution.

Chapter 2 Literature Review

Figure 2.3 Merits of fusing of SfS and photogrammetry.

Robustness to Matching illumination References Remarks type inconsistency Robust to illumination Illumination-Invariant

Robust variations but not dense SIFT (Wu et al., 2018a) matching Not dense and not

Feature point Feature Not robust SIFT (Lowe, 2004); robust to illumination variations Phase Correlation Dense and robust to (Foroosh et al., 2002; Wan illumination variations.

Robust et al., 2018); But not favourable to wise

- SfS-based (Grumpe et al., high-resolution stereo

2014) images Pixel Dense but not robust to Not robust SGM (Hirschmüller, 2008) illumination variations a) Summary of different image matching methods.

b) Limitations of existing approaches that are addressed by this research. Figure 2.4 Summary of common image matching approaches in planetary mapping.

Chapter 2 Literature Review

Reconstruction Input Fusion type Representative works type van Diggelen (1951); Without fusion Rindfleisch (1966) Profile-based Fusion with Kirk et al. (2003a) reconstruction photogrammetric 3D data

Fusion with

photogrammetric pipeline Without fusion Kirk (1987); Horn (1990)

Singleimage Soderblom and Kirk (2003); Fusion with Surface-based Grumpe et al. (2014); photogrammetric 3D data reconstruction Alexandrov et al. (2018) Fusion with Gaskell et al. (2008); photogrammetric pipeline Grumpe et al. (2014) Without fusion Wöhler (2004) Fusion with Profile-based Grumpe et al. (2014) photogrammetric 3D data reconstruction

Fusion with

photogrammetric pipeline Without fusion Woodham (1980) Lohse et al. (2006); Fusion with Multipleimages Surface-based Gaskell et al. (2008); photogrammetric 3D data reconstruction Alexandrov et al. (2018) Fusion with

photogrammetric pipeline a) Summary of different SfS algorithms for 3D modelling of planetary surfaces.

b) Limitations of existing approaches that are addressed by this research. Figure 2.5 Summary of existing SfS algorithms and fusion approaches with photogrammetry for 3D modelling of planetary surfaces.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface Chapter 3 Single Image Shape-from-Shading (SI- SfS) for 3D Modelling of the Lunar Surface

High-resolution lunar remote sensing images such as the LROC NAC images (0.5 - 1.5 m/pixel spatial resolution) are usually acquired under near-nadir viewing, resulting in limited stereo coverage for generating high-resolution 3D models using photogrammetric methods. Lunar DEMs of global/near-global spatial coverage such as the LDEM and SLDEM, on the other hand, usually derived from laser altimetry and stereo images with much lower resolutions, hence they have reliable accuracy but are limited in spatial resolution. This chapter describes an algorithm that integrates SfS with DEMs of lower resolution. Taking advantage of the high- resolution reconstruction ability of SfS and the accuracy of photogrammetric/laser- altimetric lunar DEMs, the algorithm produces a high-resolution DEM (i.e., the same resolution of the image) with the accuracy comparable to the photogrammetric/laser- altimetric DEM. Another merit of the algorithm is that it only requires one image and hence it is not limited by the availability of stereo images.

3.1 Overview of the SI-SfS Approach

The Single Image SfS (SI-SfS) algorithm takes an initial DEM and a corresponding high-resolution image as input, and then it generates a DEM with the resolution of the image and overall accuracy comparable to the initial DEM. It is assumed that the illumination and viewing conditions of the image are known. Figure 3.1 outlines the framework of the SI-SfS algorithm. First, the initial DEM and its corresponding high-resolution image are imported to the system; then, a shadow map is derived from the image through shadow detection methods or adopted from an initial shadow map if available. The initial DEM, high-resolution image and the shadow map are then forwarded to a hierarchical multi-grid SfS system. The system consists of two hierarchies. In the current hierarchy (dashed box in blue in Figure 3.1), the image, shadow map and the DEM are resampled to the same pre-defined spatial resolution. Then the surface gradients (in the form of normal vectors of the surface) of each pixel are derived based on its image intensity and the constraints

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface from the DEM. For pixels determined as shadows on the shadow map, a shadow constraint, which is constructed based on the 3D geometric properties of shadows under the illumination condition during image acquisition, is imposed for estimation of surface gradients. The resulted surface gradients are then forwarded to the initial hierarchy (dashed box in magenta in Figure 3.1). In the initial hierarchy, the estimated surface gradients from the current hierarchy, the image, and the shadow map are first resampled to the spatial resolution of the initial DEM (i.e., initial resolution). Then the surface gradients of each cell (i.e., a pixel at initial resolution) are refined according to the image intensity and the constraints imposed by the initial DEM and shadows. The surface gradients refined at the initial resolution, in the initial hierarchy, provide constraints for the current hierarchy to generate the final refined surface gradients. Based on the resulted final refined surface gradients, the DEM will be refined accordingly. The resulted SfS DEM will be up-sampled by a pre-defined factor (e.g., two) and the process is repeated until it reaches the spatial resolution of the image. Since there is a large difference in spatial resolution between the initial DEM and the image, and that SfS is a sparse system where each pixel only affects a close neighbourhood (Kirk, 1987). As a result, processing at image resolution usually results in slow convergence and propagation of large scale errors. A hierarchical architecture effectively increases the performance of SfS by evaluating the result at multiple scales. Moreover, the multi-grid architecture allows the SfS results at a higher resolution (i.e., current hierarchy) to be transferred to the initial resolution (i.e., initial hierarchy), which is then propagated back to the current hierarchy after SfS at the initial hierarchy. Such architecture possesses better convergence properties (Horn, 1990) and allows frequent interactions between multiple resolutions. The transferring and propagation between multiple hierarchies can be achieved by down- sampling and up-sampling operations such as averaging and bilinear interpolation. Since the initial DEM is involved in the initial hierarchy, the multi-grid architecture also ensures a high degree of geometric correspondence between the initial DEM and the SfS DEM.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

Figure 3.1 Overview of the SI-SfS approach.

3.2 Derivation of Surface Gradients

3.2.1 Surface Gradients for Illuminated Regions

Retrieval of surface gradients from image intensities requires inverse solving 푝 the reflectance model G(p,q) for the corresponding surface normal vector [푞]. Because the Lunar-Lambert model (McEwen, 1991) is well suited for lunar photometry and is computationally less complex, it will be employed throughout this dissertation. The Lunar-Lambert model (McEwen, 1991) is expressed as follows:

휇0(푝,푞) 퐺(푝, 푞) = (1 − λ)휇0(푝, 푞) + 2λ , (3.1) 휇0(푝,푞)+휇(푝,푞)

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

where µ0 is the cosine of the incidence angle (i), µ is the cosine of the emission angle (e), and λ is the phase function. The relationship between surface shape (as represented by surface gradients) and the acquired image intensity is also affected by the albedo of the surface. It is mathematically insufficient to solve for both albedo and reflectance in a single image without additional constraints. To account for local albedo variations it is assumed that the albedo of a point on the lunar surface is consistent with its close neighbourhood (Lee and Rosenfeld, 1983; Tsai and Shah, 1998; Grumpe et al., 2014). This assumption is practical for natural surfaces where surface albedo changes smoothly. The reflectance of a location (x,y) can hence be estimated from its neighbouring pixels, yielding an albedo-free equation:

퐴 퐺 퐼 푥,푦 푠푓푠 = 푥,푦| 퐴푗퐺푗 퐼푗 푗∈푁푥,푦

퐼푥,푦 퐺푗 퐺푠푓푠 = ∑푗∈푁푥,푦 , (3.2) 푛 퐼푗

where each Ij is an image pixel within a predefined local neighbourhood Nx,y of the centre pixel Ix,y. It is assumed that the albedo at the centre Ax,y equals the Aj of its vicinity, and hence the ratio of the reflectance value equals that of the image intensity. The estimated reflectance Gsfs is then computed as the mean of all of the

퐼푥,푦 valid candidate reflectance values 퐺푗 within Nx,y. The shadowed regions are not 퐼푗 considered as valid pixels in this process. Equation 3.2 largely minimises the effects of albedo variations by factoring out albedo using the ratio of image intensities. The surface gradients p and q are then connected to the estimated reflectance by:

∂G 휕퐺 퐸 = 퐺(푝 , 푞 )| + ∆푝 + ∆푞 − 퐺 , (3.3) 푠푓푠 0 0 푥,푦 ∂p 휕푞 푠푓푠

푝 ∆푝 where [ 0] is the surface normal vector of the DEM at current iteration and [ ] is 푞0 ∆푞 the corresponding update values. In order to solve for two parameters with only one observation, the integrability constraint (Horn, 1990) is introduced:

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

2 2 퐹(푝, 푞) = (푝 − 푝0) + (푞 − 푞0) (3.4)

It encourages the surface normal vector to approach the closest integrable solutions. 푝 In this case [ 0], the initial surface normal vector of the DEM at the current iteration 푞0 will be the most suitable choice. Note that the integrability constraint introduced by Horn (1990) is only approximate enforcement of the integrability condition. The resulted surface gradients are not necessarily integrable. Because the numerical unit 2 of Esfs and F(p,q) differ significantly, a joint minimisation of [퐸푠푓푠 + 퐹(푝, 푞)] usually requires assigning appropriate weight to each individual component. Another alternative is to treat the more important Esfs as an absolute constraint and minimise F(p,q) accordingly:

푝푠푓푠 [ ] = arg 푚𝑖푛푝,푞 퐹(푝, 푞) 푠. 푡. 퐸푠푓푠 = 0 (3.5) 푞푠푓푠

Doing so ensures the topographic details represented by Esfs can be effectively enforced and F(p,q) will not suppress necessary details of the surface. Esfs can be scaled down by a factor in order to control the magnitude of changes per iteration. However, this strategy is only necessary when the iterative SfS process becomes unstable or starts to diverge. In most cases, it is not required since the initial DEM has already provided sufficient regularisation over the estimation of Gsfs.

3.2.2 Initial DEM Used as a Constraint

The initial DEM is further incorporated into the SfS process by providing the initial conditions as well as constraints explicitly. Due to the fact that the effective resolution of the initial DEM is lower than that of the image, the resulted surface normal vectors are expected to approach that of the initial DEM at lower resolution (Grumpe et al., 2014). This constraint can be incorporated into the existing cost function F(p,q):

T 푝 − 푝0 푝 − 푝0 2 2 퐹(푝, 푞) = [ ] [ ] + 푤푖푛푖(∆퐿푖푛푖 + ∆푇푖푛푖 ) 푞 − 푞0 푞 − 푞0

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

푇 푝 푇 푝푖푛푖 ∆퐿푖푛푖 = 퐿 𝑔([ ] , 휎) − 퐿 [ ] 푞 푞푖푛푖 푘

푇 푝 푇 푝푖푛푖 ∆푇푖푛푖 = 푇 𝑔([ ] , 휎) − 푇 [ ] , (3.6) 푞 푞푖푛푖 푘

푞 where 푇 = [ 퐿 ] is the unit vector pointing toward the direction perpendicular to −푝퐿 푝 the illumination vector L. [ 푖푛푖] refers to the surface normal vector of the initial 푞푖푛푖 푘 DEM, at kth iteration, at the corresponding location and 𝑔(∗, 휎) refers to a Gaussian 푝 smoothing operator with a standard deviation σ. 𝑔([푞] , 휎) evaluates the surface normal vector at the resolution of the initial DEM where they are expected to 푝 approach [ 푖푛푖] . Because SfS is generally most effective along illumination 푞푖푛푖 푘 direction and least along the perpendicular direction, the last two components in F(p,q) are designed to evaluate the constraints along these two directions in order to increase the robustness of optimisation. Since the SI-SfS algorithm implements a multi-grid system as described in section 3.1, the surface normal vectors from the 푝 initial DEM are refined iteratively by SfS at the initial hierarchy. [ 푖푛푖] is obtained 푞푖푛푖 푘 by minimizing the cost function Fini:

T 2 2 퐹푖푛푖(푝푖푛푖|푘, 푞푖푛푖|푘) = ∆푁푖푛푖 ∆푁푖푛푖 + 푤0(∆퐿0 + ∆푇0 )

푝푖푛푖 푝푖푛푖 ∆푁푖푛푖 = [ ] − [ ] 푞푖푛푖 푘 푞푖푛푖 푘−1

푇 푝푖푛푖 푇 푝푖푛푖 푇 푝푖푛푖 푇 푝푖푛푖 ∆퐿0 = 퐿 [ ] − 퐿 [ ] ; ∆푇0 = 푇 [ ] − 푇 [ ] , (3.7) 푞푖푛푖 푘 푞푖푛푖 0 푞푖푛푖 푘 푞푖푛푖 0

푝 where [ 푖푛푖] refers to the surface normal vector of the previous iteration, and 푞푖푛푖 푘−1 푝푖푛푖 [ ] refers to that of the initial DEM. 푤0 controls the weight of the initial DEM, it 푞푖푛푖 0 푝푖푛푖 is usually set higher than 푤푖푛푖 in equation 3.6 such that [ ] remains high 푞푖푛푖 푘 correspondence with the initial DEM.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface 3.2.3 Image Shadow Used as a Constraint

In shadowed regions, it is not appropriate to use SfS because there is no shading information. However, shadows also provide information about the surface normal vectors under the shadows. The magnitude limit of the normal vectors within the shadows can be obtained by:

cos 푖 1 푠 = − = − , (3.8) 푠ℎ푤 sin 푖 tan 푖

where i is the incidence angle and sshw is the signed magnitude of the vector of a shadow along the illumination direction, which provides an upper limit of the normal 푝 vectors of a shadowed surface. When the normal vector [푞] is projected to the direction of the illumination, the magnitude cannot be higher than sshw because, in that case, the surface would be illuminated and no longer shadowed. A straightforward way to achieve this condition is to force the normal vector of each shadowed pixel not to exceed sshw when they are evaluated along the direction of the illumination. This constraint can be formulated as:

푇 푝 퐸푠ℎ푤 = max (퐿 [푞] − 푠푠ℎ푤 , 0), (3.9)

푝 푝 where 퐿 = [ 퐿] is the unit vector pointing toward the illumination source, and 퐿푇 [ ] 푞퐿 푞 projects the normal vector (p,q) to L. Since for a shadowed pixel, the projected magnitude is considered to be equal to or less than sshw, Eshw is valid when Eshw ≥ 0.

Similar to Esfs, Eshw is treated as an absolute constraint for the shadowed pixels. Since

Esfs is switched off for the shadowed pixels, the optimisation becomes:

푝푠ℎ푤 [ ] = arg 푚𝑖푛푝,푞 퐹(푝, 푞) 푠. 푡. 퐸푠ℎ푤 = 0 (3.10) 푞푠ℎ푤

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface 3.3 Reconstruction of a DEM from Surface Gradients

푝푠푓푠 To summarise, each pixel contains two solutions of normal vectors, [ ] for 푞푠푓푠

푝푠ℎ푤 the illuminated pixels derived based on the reflectance constraint Esfs, and [ ] for 푞푠ℎ푤 the shadowed pixels derived based on the shadow constraint Eshw. We assume that a shadow map of the image is available that contains the weight of the shadow for each pixel, which ranges from 0 to 1. The weights represent the likelihood of a pixel being shadowed based on a shadow detection algorithm, or the fraction of a pixel that is shadowed. The final estimated normal vector is then a linear combination of

푝푠푓푠 푝 [ ] and [ 푠ℎ푤] based on the weight values of the shadow map, which can be 푞푠푓푠 푞푠ℎ푤 defined as follows:

푝푓푖푛푎푙 푝푠푓푠 푝푠ℎ푤 [ ] = (1 − 푤푠ℎ푤) [ ] + 푤푠ℎ푤 [ ], (3.10) 푞푓푖푛푎푙 푞푠푓푠 푞푠ℎ푤

푝푓푖푛푎푙 where wshw is the weight value of each pixel on the shadow map, and [ ] is the 푞푓푖푛푎푙 final estimated surface normal vector. The DEM is then adjusted such that its derived surface normal vectors optimally

푝푓푖푛푎푙 reproduce [ ] at each cell. The relationship between the DEM and the 푞푓푖푛푎푙 corresponding surface normal vectors are first modelled by finite differences:

1 1 −1 퐷퐸푀푥,푦+1 퐷퐸푀푥+1,푦+1 푝 = ∑ [ ] ∘ [ ] 2ℎ푥 1 −1 퐷퐸푀푥,푦 퐷퐸푀푥+1,푦

1 −1 −1 퐷퐸푀푥,푦+1 퐷퐸푀푥+1,푦+1 푞 = ∑ [ ] ∘ [ ], (3.12) 2ℎ푦 1 1 퐷퐸푀푥,푦 퐷퐸푀푥+1,푦

where hx and hy are the cell sizes of the DEM in the x- and y-directions, respectively, and [∗] ∘ [∗] is an element-wise multiplication operator. As depicted in Figure 3.2, 푝 the surface normal vectors of each cell [ 퐷퐸푀] are computed by the four 푞퐷퐸푀 푖,푗 neighbouring height nodes. The centre node DEMx,y (the middle square in Figure 3.2) is adjusted in a least-squares solution according to the adjacent surface normal

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface vectors estimated by SfS. The updated equation of the centre height node is as follows:

−1 푍 푘+1 = 푍 푘 + (∑ [휕 푁푇∆푁 ] ) (∑ [휕 푁푇휕 푁] ) 푥,푦 푥,푦 푖,푗 푍 푓푖푛푎푙 푖,푗 푖,푗 푍 푍 푖,푗

푝 − 푝 푇 휕푝 휕푞 푓푖푛푎푙 퐷퐸푀 휕푍푁 = [ ] ; ∆푁푓푖푛푎푙 = [ ], (3.13) 휕푍 휕푍 푞푓푖푛푎푙 − 푞퐷퐸푀

푘 th 푘+1 where 푍푥,푦 is the height value at node (x,y) at the k iteration, and 푍푥,푦 is the updated height value. Because the whole system is large (each DEM node is an unknown to be solved) and sparse (each DEM node is only affected by its neighbour), it is not appropriate to model and update the DEM height nodes all at once. Therefore, a Modified-Jacobi relaxation strategy (Horn, 1990) is adopted. Each iteration consists of four steps, and for each step, the height nodes of the DEM are evaluated and updated, one every other row and column at a time (Figure 3.3). This approach allows efficient reconstruction of the DEM while minimising possible sequence-dependent systematic biases, such as strip and chessboard artefacts, being introduced to the DEM during the process.

Figure 3.2 The layout of a DEM node during height optimisation. The centre node is optimised according to the surrounding surface gradients.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

Figure 3.3 Optimisation steps of the Modified-Jacobi relaxation strategy.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface 3.4 Experimental Analysis of the SI-SfS Approach

In order to analyze the performance of the SI-SfS approach presented in this chapter, a number of real lunar datasets were selected for experimental validations. This section presents experimental analyses of high-resolution 3D modelling of the lunar surface using a high-resolution image and an initial DEM of much lower resolution, using the proposed SI-SfS approach. Moreover, in order to test the ability of the algorithm in recovering the location-dependent surface albedo of the lunar surface, the surface albedo of the datasets is retrieved and analyzed.

3.4.1 Description of Datasets

Three experimental datasets using the LROC NAC images were presented and analyzed in this section. The specifications of the images are listed in Table 3.1. The first dataset (Buisson V dataset) is selected based on a post published on the LROC website (https://www.lroc.asu.edu/posts/790). According to the post, an unnamed crater, near the Buisson V crater, was presented with a pair of LROC NAC images, one acquired at high sun angle (i.e., overhead sun) while the other at lower sun angle (Figure 3.4a). The high sun angle image clearly shows the location- dependent surface albedo, which allows a comparison with the surface albedo recovered by SfS. The second dataset (Reiner Gamma dataset) shows a part of the Reiner Gamma area, which is well known for the existence of lunar swirls (Denevi et al., 2016). Lunar swirls are patches of higher albedo on the lunar surface. As a result, this dataset undergoes significant albedo changes even on a relatively flat area, which can be revealed by the rotated ‘‘Y” shape of the dark patch in the middle of Figure 3.4b. Choosing this area helps to better understand the performance of SfS in areas with significant albedo variations. The third dataset (Rima Sharp dataset) covers a crater (about 300 m in diameter) located right on the edge of the Rima Sharp (Figure 3.4c), it exhibits significant height differences between the two sides of the crater. Around the crater there are groups of rocks, leading to higher surface albedo due to increased surface roughness. This dataset allows better analysis of SfS performance under more complex terrain and with albedo variations due to surface roughness. For the experimental analysis, the SLDEM is selected as the initial DEM

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface to provide initial conditions and constraints for the SfS algorithm. The SLDEM has a spatial resolution of 60m/pixel and is available from the PDS Geoscience Node (http://pds-geosciences.wustl.edu/missions/lro/lola.htm). The LROC NAC images are first ortho-rectified by the software program ISIS3 (https://isis.astrogeology.usgs.gov/) and co-registered to the SLDEM. As an independent performance check, the high-resolution NAC DEMs (2 - 5 m/pixel), which are also available in the LROC Archive, are used for comparison analysis. As introduced in Chapter 1 and Chapter 2, the NAC DEMs are derived from rigorous photogrammetric processing of LROC NAC stereo images and have a mean vertical RMSE of about 2.5 m (Henriksen et al., 2017). They have resolutions much higher than that of the SLDEM but lower than that of the LROC NAC image (about 3 times the image resolution). As a result, although the NAC DEMs cannot provide perfect ground truth, they can provide a direct comparison analysis for the resulted DEMs from SfS. However, as for the Buisson V dataset, there is no NAC DEM available from the LROC Archive.

a) Buisson V Dataset

b) Reiner Gamma Dataset

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface c) Rima Sharp Dataset Figure 3.4 The LROC NAC images used for the three experimental analyses. Table 3.1 Details of LRO NAC imagery used in the experiments.

Buisson V Reiner Gamma Rima Sharp

Dataset Dataset Dataset

Image ID M146255155 M104905963 M173246166

Resolution 1.5 m/pixel 1.5 m/pixel 0.5 m/pixel

Sun azimuth 90.31o 263.31o 151.95o

Approximated 67.96o 49.16o 49.01o incidence angle

Sensor azimuth 33.69o 271.16o 88.54o

Approximated 1.15o 11.76o 17.73o emission angle

Image dimension 480 x 560 pixel 1520 x 1880 pixel 1320 x 1200 pixel (height x width)

3.4.2 Buisson V Dataset

This dataset covers a relatively smooth area, as revealed by Figure 3.5a. A larger crater that spans approximately 3-by-3 cells on the SLDEM is clearly visible on the image whereas its geometry is not preserved in the SLDEM. The high sun elevation angle image (Figure 3.5b) shows that the area is covered by traces of the ejecta blanket of a larger impact nearby, which leads to significant albedo changes. These albedo changes are almost absent in the input image, which implies that albedo recovery from the area would be challenging. The three-dimensional view of the initial SLDEM and the resulted high-resolution DEM after SfS are illustrated in Figure 3.5c and d, respectively. It is apparent that small details on the image are now visible on the SfS DEM, and the distinctive small craters at the upper right and lower 45

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface left corners of the area and the larger crater mentioned above are clearly recovered. Figure 3.6 compares two selected profiles with their counterparts on the SLDEM. Profile 1 (Figure 3.6a) shows the recovery of the larger crater as compared to the profile of the SLDEM, which is almost linear. The image of the feature enclosed in the dotted box was extracted (Figure 3.6b) and compared with the shaded relief of the same area (Figure 3.6c) to indicate how much the details on the image are preserved in the SfS DEM, the comparison clearly shows a high degree of correspondence with each other. Another profile (Figure 3.6e) shows the recovery of the distinctive small crater by SfS, which is not on the SLDEM. The crater has a sharp intensity contrast on its edge, which allows the rim to be reconstructed with the SfS algorithm. The albedo map derived from the SfS results is presented in Figure 3.7 and compared to the high sun elevation angle imagery of the same area because images with high sun elevation angles minimise topographic shading effects and highlight the local albedo variations. The derived albedo map has plausible correspondence with the high sun elevation angle image, especially the dark stripes at the lower-left portion of the area. The albedo changes are largely covered by topographic shading in the lower sun angle images and only very few clues remain, which makes it difficult to recover a reasonable albedo from a single image. On the contrary, SfS requires sufficient topographic shading on the image, which can only be provided by images with lower sun elevation angle, to produce reliable 3D models.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

a) Input LROC NAC image b) High sun elevation angle image

c) Initial SLDEM (60m/pixel) d) SfS DEM Figure 3.5 The (a) input image; (b) high sun angle image; and 3D views of (c) initial SLDEM and (d) the output SfS DEM of the Buisson V dataset.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

(b) Image

(a) Profile 1 (c) Shaded Relief

(d) Location of profiles

(f) Image

(e) Profile 2 (g) Shaded Relief Figure 3.6 Profile comparisons of the Buisson V dataset.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

(a) Albedo map from SfS (b) High sun elevation angle image

Figure 3.7 Albedo comparison of the Buisson V dataset.

3.4.3 Reiner Gamma Dataset

For the Reiner Gamma dataset, Figure 3.8 shows the input LROC NAC image (Figure 3.8a), 3D views of the initial SLDEM (Figure 3.8b), the reference NAC DEM (Figure 3.8c) and the SfS result (Figure 3.8d). The key geometry of the shape represented by the NAC DEM is well preserved in the SLDEM, which allows the SfS algorithm to converge to a result with high correspondence with the reference NAC DEM. The profile comparisons in Figure 3.10 demonstrate the consistency between the SfS DEM and the reference NAC DEM, whereas very small details are recovered in the SfS DEM, as shown by the shaded relief. The profiles have an absolute vertical root-mean-square error (RMSE) of 5-6 m with a maximum error of about 10 m as shown in Table 3.2, the larger residuals stem from the depth of the craters. The SfS DEM, constrained by the input SLDEM, has shallower craters than the reference NAC DEM and thus contributing to the error. The albedo of the Reiner Gamma dataset is presented in Figure 3.9a, and compared with the corresponding high sun elevation angle image (Figure 3.9b). The comparison suggests promising albedo recovery where shading components are visually minimised in the albedo image, and the albedo changes caused by, for example, impact ejecta and lunar swirls are preserved in the albedo image. Stripes are visible

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface on the recovered albedo aligned along the illumination direction, which is similar to the results obtained by Barron and Malik (2011) and Kirk et al. (2003). This may be explained by the physical nature of the SfS, in which slopes along the illumination direction can be better estimated than slopes in the perpendicular direction.

a) Input LROC NAC image b) Initial SLDEM

c) SfS DEM d) Reference NAC DEM Figure 3.8 The (a) input image; and 3D views of (b) initial SLDEM; (c) the output SfS DEM; and (d) independent reference DEM of the Reiner Gamma Dataset.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

(a) Albedo map from SAfS (b) High sun elevation angle image

Figure 3.9 Albedo comparison of the Reiner Gamma dataset.

Table 3.2 Statistical analysis of profiles of the Reiner Gamma dataset.

Maximum Mean RMSE absolute deviation

Profile 1 1.04m 6.34m 11.19m

Profile 2 3.42m 4.94m 10.08m

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

(b) Image

(a) Profile 1 (c) Shaded Relief of SAfS DEM

(d) Shaded Relief of NAC DEM

(e) Location of profiles (g) Image

(h) Shaded Relief of SAfS DEM

(f) Profile 2 (i) Shaded Relief of NAC DEM Figure 3.10 Profile comparisons of the Reiner Gamma dataset.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface 3.4.4 Rima Sharp Dataset

For the Rima Sharp dataset, it can be seen from Figure 3.11b that the SLDEM does not preserve the overall geometry of the crater as compared to the NAC DEM (1.5m/pixel) showed in Figure 3.11d. As a result, part of the crater is not well recovered from the SfS routine, as indicated in the SfS DEM in Figure 3.11c, even though it reveals much more terrain detail than the NAC DEM cannot capture. Profile comparisons and the shaded relief in Figure 3.12 suggest that small boulders and craters are recovered. When comparing to the image, the shaded relief of the SfS DEM may present slightly smoother. It is reasonable because albedo variations and image noise are absent in the shaded relief. Table 3.3 shows the absolute vertical RMSE of the profiles. The residuals stem mainly from the unsatisfactory reconstruction of the crater (Figure 3.12f). The general consistency between the SfS DEM and the reference NAC DEM is visibly lower than for the previous two datasets, and two reasons for such inconsistency are identified. The profiles in Figure 3.12 and 3D views of the input SLDEM and NAC DEM suggest that similar to previous two datasets, the input SLDEM cannot preserve the key geometry of the crater and the lower-left portion of the area. The crater is represented by a flat slope in the SLDEM, whereas the lower-left portion of the area has a higher elevation in the SLDEM than in the reference NAC DEM. However, due to the more complex geometry of the crater vicinity, reconstruction of the rim becomes difficult. A profile along the direction perpendicular to illumination is extracted, and the intensity values of the image and the shaded NAC DEM are compared in Figure 3.13. The shaded NAC DEM profile, as shown by the upper green line in Figure 3.13b, shows a significant decrease in intensity at the edge of the crater, whereas the image intensity (blue line) shows no visible change. This suggests an albedo increase in the crater area and that insufficient information content has been captured in the image or in the input SLDEM. Therefore, the monocular imagery SfS algorithm does not have sufficient information to estimate an accurate reflectance and its subsequent shape. The second reason is an albedo issue. Because it cannot be uniquely and accurately solved with a single image due to mathematical insufficiency, the geometric control of the algorithm can be increased by using a better input DEM.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

a) Input LROC NAC image b) Initial SLDEM

c) SfS DEM d) Reference NAC DEM Figure 3.11 The (a) input image; and 3D views of (b) initial SLDEM; (c) the output SfS DEM; and (d) independent reference DEM of the Rima Sharp Dataset.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

(b) Image

(d) Shaded Relief of NAC DEM

(g) Image (e) Location of profiles

(h) Shaded Relief of SfS DEM

(i) Shaded Relief (f) Profile 2 of NAC DEM Figure 3.12 Profile comparisons for the Rima Sharp dataset.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface Table 3.3 Statistical analysis of profiles for the Rima Sharp dataset

Maximum Mean RMSE absolute deviation

Profile 1 0.99m 3.42m 5.21m

Profile 2 0.98m 4.43m 8.03m

(a) Location of profile (b) Intensity profile

Figure 3.13 Intensity profile of the Rima Sharp dataset.

In order to verify the above explanation, the NAC DEM was down-sampled to 20m per pixel and used as the input DEM (Figure 3.15a), such that the new input DEM possesses better properties in terms of resolution and preservation of geometry than the previous SLDEM in Figure 3.11b. The SfS result (Figure 3.15b) produces a much better DEM with preservation of the details on the image and a high degree of consistency with the reference NAC DEM. This shows a good example of solution uniqueness mentioned by Horn (1990) that the algorithm may converge to a different solution (Figure 3.11 and 3.15) depending on factors such as initial conditions, whereas the image details are almost equally preserved in the resulting DEM. Profile comparisons and the shaded reliefs in Figure 3.16 show that the SfS DEM constrained by a better input DEM also reconstructs very small boulders and craters.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface Comparing profile 1 of the new result (Figure 3.16a) with the old result (Figure 3.12a) illustrates an example of the influence of the quality of the initial DEM. In the old profile in Figure 3.12a, the SfS DEM is following the SLDEM with its profile tilted slightly to the right, whereas it becomes generally horizontal in the new profile in Figure 3.16a because its initial DEM possesses a horizontal profile. The absolute vertical RMSE of the profiles as concluded in Table 3.4 also improves to less than 2 m, with a maximum of less than 4 m. The shaded reliefs showed in Figure 3.12 and 3.16 also indicate that the shaded SfS DEM with SLDEM as the initial is brighter than that with the 20m NAC DEM as the initial DEM, which implies that the former DEM is more sunward-facing than the latter. The albedo image constrained by the higher-resolution input DEM (Figure 3.14b) is apparently brighter in the crater area than that constrained by the SLDEM, which the intensity variations on the image are unable to effectively represent such changes. Although the albedo image constrained by the SLDEM is also brighter in the crater area, its boundary is less distinctive than its counterpart, which implies a smoother shape in those boundaries. Both albedo images possess stripe artefacts along the illumination direction, which are similar to those discussed in the Reiner Gamma dataset.

(a) SfS constrained by SLDEM (b) SfS constrained by 20 m NAC DEM

Figure 3.14 Albedo comparison of the Rima Sharp dataset.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

(a) Down-sampled NAC DEM (20m/pixel) as initial input

(b) SfS DEM derived from 20m NAC (c) Reference NAC DEM DEM

Figure 3.15 The 3D views of (a) Down-sampled initial NAC DEM; (b) output SfS DEM; and; (c) the reference NAC DEM of the Rima Sharp Dataset.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface

(b) Image

(d) Shaded Relief of NAC DEM

(e) Location of profiles (g) Image

(h) Shaded Relief of SAfS DEM

(i) Shaded Relief (f) Profile 2 of NAC DEM Figure 3.16 Profile comparisons for the Rima Sharp dataset.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface Table 3.4 Statistical analysis of profiles for the Rima Sharp dataset constrained by 20m NAC DEM.

Maximum Mean RMSE absolute deviation

Profile 1 0.41m 1.06m 2.63m

Profile 2 0.85m 1.85m 3.80m

To further examine the performance of the proposed SfS approach, a DEM generated by the SfS approach presented by Grumpe et al. (2014) using the LROC NAC image of the Rima Sharp dataset (hereafter referred to as Grumpe et al. (2014)’s DEM) was involved for a detailed comparative analysis. Figure 3.17 shows a side-by-side comparison of the two DEMs, both show satisfactory details. The SfS DEM generated by our approach (Figure 3.17a) preserves better geometry especially at the crater rim while Grumpe et al. (2014)’s DEM (Figure 3.17b) shows clear defects for the south-west part of the crater rim. To quantitatively evaluate the DEM accuracies, our SfS DEM and Grumpe et al. (2014)’s DEM were down-sampled to the same resolution of the reference NAC DEM, and difference DEMs were obtained by directly comparing them with the reference NAC DEM as shown in Figure 3.17c and d, respectively. It is apparent that our SfS DEM shows higher consistency with the independent reference NAC DEM. The difference DEM between Grumpe et al. (2014)’s DEM and the reference NAC DEM shows overall darker (larger inconsistency) values, especially at the crater and its nearby region. Table 3.5 summarises the statistics of the two difference DEMs. Our SfS DEM has a mean absolute error of 0.73 m and a maximum error of 4.44 m, while Grumpe et al. (2014)’s DEM has a mean absolute error of 1.23 m with a maximum error of 10.19 m. This indicates the superior performance of the proposed SfS approach.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface Table 3.5 Statistical comparison of absolute difference DEMs for the Rima Sharp dataset.

Proposed result Grumpe et al.’s result

Mean absolute deviation 0.73m 1.23m

Maximum absolute deviation 4.44m 10.19m

(a) SfS DEM of the proposed method (b) Grumpe et al. (2014)’s DEM

Figure 3.17 Comparison of absolute difference DEM between the proposed result and Grumpe et al. (2014)’s result.

Chapter 3 Single Image Shape-from-Shading (SI-SfS) for 3D Modelling of the Lunar Surface 3.5 Summary

This chapter presented the SI-SfS approach for modelling the lunar surface using monocular imagery and an initial DEM, based on SfS. Using SfS, the surface gradients, in the form of surface normal vectors, are derived and the DEM is refined according to the derived surface gradients. The initial DEM provides initial conditions and constraints for the SfS algorithm to result in a precise and detailed lunar DEM, with optimal spatial resolution comparable to the image. A hierarchical and multi-grid architecture is employed to ensure robust and efficient SfS performance and high geometric correspondence with the initial DEM. A shadow constraint is added for the handling of image shadows. The performance of the SI- SfS approach was quantitatively evaluated by experimental analysis, the experimental results showed that the resulted DEMs generated by the approach successfully generate DEMs with an effective resolution of the image (0.5- 1.5m/pixel) and effectively recover the topographic details visible in the image. The SfS DEMs achieved an absolute vertical RMSE of better than 7 m and an absolute maximum vertical error of approximately 10 m. The surface albedo map, simultaneously generated along with the SfS DEMs, also showed a high degree of correspondence with high sun elevation angle images, which serve as references of albedo patterns. The Rima Sharp dataset showed that the performance of the SI-SfS approach depends on the quality of the initial DEM, especially on the preservation of key geometry by the initial DEM, for its effective resolution. The SI-SfS approach tends to smooth out the boundaries of actual drastic albedo changes due to equation 3.2. Therefore, if the resolution difference between the image and the initial DEM is too large, it will create greater ambiguity for the solution, which may lead to inaccuracies in both shape and albedo. In this case, an initial DEM with better properties such as geometry preservation and effective resolution would be preferred.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

SfS using multiple images (i.e., photometric stereo SfS, PS-SfS) provides another alternative for high-resolution 3D modelling of the lunar surface. Compared to that of a single image, PS-SfS allows more robust estimation of surface gradients due to additional images. The first part of this chapter presents a PS-SfS algorithm for high-resolution 3D modelling of the lunar surface, it is designed for a pair of co- registered images acquired under different illumination and it does not need an initial DEM in order to work. Although a pair of images is required for the algorithm, it is not limited to stereo images. The second part of this chapter explores the performance of PS-SfS with respect to illumination differences. This is achieved by first modelling the PS-SfS process and then analysing the errors propagated by the model. Theoretically evaluating the performance of the algorithm allows early anticipation of SfS performance and hence can assist in choosing image pairs best suited for high-resolution 3D modelling.

4.1 Overview of the PS-SfS Approach

As illustrated in Figure 4.1, the PS-SfS algorithm takes a pair of co-registered images as input, each image of the pair is acquired under different illumination condition and hence it is also known as photometric stereo pair (Woodham, 1980). Because the Moon’s rotation axis is almost perpendicular to the ecliptic, the lunar sub-solar point is always along the lunar equatorial belt. As a result, the lunar equatorial region is illuminated from either the East or the West (Wöhler, 2004), while higher latitudes have more variations in illumination azimuth, as well as shadows. The PS-SfS approach is designed to process a pair of images to account for the aforementioned lunar illumination properties, it is also flexible to the availability of data. The images are first jointly analysed in the PS-SfS system to create a principal gradient map. The principal gradient map represents the surface gradients along the principal direction determined according to Section 4.2.2, it is mathematically robust to location-dependent albedo and is used to estimate the initial

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface albedo of each pixel. Then, the pixel-wise initial albedo map will be used as a constraint in the PS-SfS algorithm to generate robust surface normal vector field iteratively. During the iterative reconstruction process, the DEM and the pixel-wise albedo map will be refined accordingly. Finally, a high-resolution DEM will be generated from the surface normal vector field.

Figure 4.1 Workflow for the PS-SfS approach.

4.2 Derivation of Surface Gradients from Multiple Images

4.2.1 Albedo-Invariant Computation of Surface Gradients

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface Given a photometric stereo image pair under known imaging and 푝 illumination conditions, at any point (x,y), the surface normal vector [푞] is expected to satisfy the following reflectance constraints:

퐼 퐺 (푝, 푞) = 푖, (4.1) 푖 퐴

where for two images i = (1,2), Gi(p,q) is the corresponding reflectance values for 푝 the ith image estimated using the surface normal vector [푞] at point (x,y), Gi(p,q) is

퐼푖 expected to comply with the ratio between the image intensity Ii and the albedo A. 퐴 푝 It is insufficient to solve for [푞] and A simultaneously on the basis of two reflectance constraints. However, the albedo A can be mathematically removed by evaluating the ratio of intensities, yielding an albedo-free observation as follows:

퐼1 퐴퐺1(푝,푞) 퐺1(푝,푞) = = , (4.2) 퐼2 퐴퐺2(푝,푞) 퐺2(푝,푞)

This is referred to as the quotient condition. Because the two pixels correspond to the same physical location, the intrinsic albedo A of the pixels from the different images in this location must be the same. Hence, the differences are cancelled out by division in Eq. (3). This method has been used for images with coplanar light sources (Wöhler, 2004) where the gradient along only one direction is estimated:

퐼1 퐺1(푠) = , (4.3) 퐼2 퐺2(푠) where s refers to the signed magnitude of the surface normal vector (normal magnitude) along a predefined direction; 퐺푖(푠) = 퐺푖(푠푝푢, 푠푞푢) is the corresponding 푝 reflectance value of s at imaging condition of the ith image and [ 푢] is a unit vector 푞푢 denoting the predefined direction. The normal magnitude s is estimated by the PS- SfS algorithm using equation 4.3, with the resulted s, the albedo A can be computed

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface as follows:

퐼푖 퐴 = ∑푖=1,2 푤푖 , (4.4) 퐺푖(푠)

퐼푖 where is the albedo derived from the ith image and wi refers to the 퐺푖(푠) corresponding weight. The weight wi can be adjusted depending on factors such as the image quality and the imaging conditions of each individual image. The surface 푝 normal vector [푞] can then be obtained by minimizing:

퐼 2 퐸 = ∑ (퐺 (푝, 푞) − 푖) , (4.5) 푃푆 푖=1,2 푖 퐴

The PS-SfS algorithm computes iteratively the surface normal vectors and their corresponding DEM. After each iteration, the albedo A is refined by the following equation:

퐴푘 = (푤푖푡푒푟)퐴푘−1 + (1 − 푤푖푡푒푟)퐴, (4.6)

where Ak is the albedo at kth iteration; witer is the weight introduced to constrain the change in albedo between successive iteration and A is the albedo of the current iteration derived from equation 4.4. The albedo Ak will be used as the albedo for the next iteration.

4.2.2 Determination of the Principal Direction

푝 The principal direction 푢 = [ 푢] is defined such that the normal magnitude s 푞푢 퐺 (푠) can be robustly estimated from the ratio of reflectance 1 . It is defined according to 퐺2(푠)

푝퐿1 푝퐿2 Figure 4.2. Let 퐿1 = [ ] and 퐿2 = [ ] be the unit vectors pointing toward the 푞퐿1 푞퐿2 illumination sources of each image, respectively. A unit vector v is constructed which bisects the angle ∠L1-O-L2 such that ∠a1=∠a2. This can be achieved by

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface normalizing the mean vector of L1 and L2:

푝푣 1 1 −1 푣 = [ ] = (퐿1 + 퐿2) ‖ (퐿1 + 퐿2)‖ (4.7) 푞푣 2 2

The principal direction u is then determined by rotating vector v 90o clockwise:

푝 푞 푢 = [ 푢] = [ 푣 ] (4.8) 푞푢 −푝푣

In general, that the ratio of reflectance equation 4.3 can be best represented along the principal direction u because G1(s) and G2(s) behave in reverse manner as the normal magnitude s along this direction changes. Specifically, when the surface is inclined toward L1 (i.e., brighter G1), its reflectance with respect to L2 will be lower (i.e., darker G2), and vice versa, resulting in a wider spectrum of the ratio of reflectance as the s changes (blue solid line in Figure 4.3). However, when evaluating along the perpendicular of the principal direction (i.e., v), G1 and G2 change similarly as s along v changes (i.e., both brighter or both darker). Hence, the ratio of reflectance will be almost constant as the gradient changes (red dashed line in Figure 4.3), leading to ambiguities and inaccurate estimation. The principal direction u will then be used in equation 4.3 for the robust determination of surface normal vectors.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

Figure 4.2 Determination of the principal direction in photometric stereo. Vector u denotes the principal direction and v denotes the perpendicular of the principal direction.

Figure 4.3 Comparison of the range of reflectance ratio evaluated along the principal direction u (blue solid line) and its perpendicular v (red dashed line), respectively.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface 4.3 Reconstruction of a DEM from Surface Gradients

Since the PS-SfS algorithm described in this dissertation does not involve any initial DEM, it is not appropriate to generate a DEM from surface normal vectors by directly using the iterative algorithm explained in section 3.4. This is because the aforementioned iterative algorithm requires an initial DEM for the best performance. If a flat plane is assumed as the initial condition, extra iterations will be needed to slowly refine the flat plane to a reasonable DEM and unnecessary uncertainties such as gradual systematic error may be introduced. An alternative approach is to first initialise a DEM as closely to the surface normal vectors as possible and then refine it by the iterative method described in section 3.4. This strategy can be achieved by first generating a DEM using one of the state-of-the-art algorithms in the shape- from-gradients problem: the Frankot and Chellappa algorithm (Frankot and Chellappa, 1988). Given a set of surface gradient maps, the Frankot and Chellappa algorithm finds the best possible surface by projecting the surface gradients onto a set of predefined surface functions in Fourier space. In other words, the algorithm uses the Fourier series as a set of surface functions and best fit them to the given surface gradients map. The algorithm has the advantages of being efficient and able to create a reasonable DEM without any iteration. It is particularly useful in PS-SfS where the retrieved surface normal vectors are less biased compared to the case of monocular imagery SfS explained in Chapter 3. On the other hand, the iterative algorithm has the advantage of being robust to abrupt errors in the surface normal vectors, which will be easier to be propagated in the Frankot and Chellappa algorithm. An implementation of the Frankot and Chellappa algorithm has been developed by Agrawal (2006), which is available at (http://www.cs.cmu.edu/~ILIM/projects/IM/aagrawal/software.html). Based on our experience, the Frankot and Chellappa algorithm will lose reconstruction accuracy in low-frequency components (i.e., very gradual slope changes) and is unable to recover the absolute height of the DEM without extra information. Both problems can be solved if an initial DEM, or the mean normal vector and the mean height of the region of interest, is available:

퐷퐸푀푓푖푛푎푙 = 퐷퐸푀퐹퐶 − 푝푙푎푛푒(퐷퐸푀퐹퐶) + 푝푙푎푛푒푖푛푖 + 푍푖푛푖, (4.9)

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

where 퐷퐸푀퐹퐶 corresponds to the DEM constructed using the Frankot and Chellappa algorithm and 푝푙푎푛푒(퐷퐸푀퐹퐶) is the best-fit plane of 퐷퐸푀퐹퐶 . 푝푙푎푛푒푖푛푖 refers to a plane constructed by the mean normal vector, the mean height of 푝푙푎푛푒푖푛푖 is zero.

Zini refers to the mean height of the region of interest. In other words, 퐷퐸푀푓푖푛푎푙 is computed by extracting the high-pass component of the reconstructed DEM with respect to a best-fit plane and then adding it back to the plane constructed using the mean normal vector and the mean height of the region of interest.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface 4.4 Effects of Illumination Inconsistencies on the 3D Modelling Performance

4.4.1 Modelling the Process of Photometric Stereo Shape-from-Shading

Because the described PS-SfS algorithm generates a DEM based solely on two images, it will be interesting to understand the nature of the problem and how uncertainties will be propagated. With this understanding, one can have a general estimation about the quality of the reconstruction given two images, or to decide which pair (out of a set) provides the best performance. In this section, we will first derive the mathematical relationship between PS-SfS and the azimuthal difference between the illuminations of the image pair. The mathematical analysis described here focuses on slope estimation, which is a direct and strictly forward outcome of PS-SfS. As shown in Figure 4.4, the derivation focuses on a single point O on a surface and on a case wherein only one photometric stereo pair is available (i.e., double-image SfS). It also assumes that the zenith angles of both the illumination sources are non-zero (i.e., non-overhead sun) to produce the non-zero 2D illumination vectors. Using the notations described in section 4.2.2, the unit vector

푝퐿1 퐿1 = [ ] points towards illumination direction 1, it is also aligned with the 푞퐿1 1 horizontal axis p of the Cartesian coordinate frame. Therefore, 퐿 = [ ]. The unit 1 0 푝퐿2 vector 퐿2 = [ ] points towards illumination direction 2, which is rotated by an 푞퐿2 angle of α counter-clockwise from illumination direction 1. The unit vectors L1 and

L2 can be modelled as follows:

cos 훼 − sin 훼 1 cos 훼 퐿 = 푅(훼)퐿 = [ ] [ ] = [ ], (4.10) 2 1 sin 훼 cos 훼 0 sin 훼

where α is the azimuthal difference between the L1 and L2. Note that R(α) denotes a counter-clockwise rotation, which implies that α will be negative when a clockwise rotation is needed. The PS-SfS is then decomposed into two SI-SfS processes, one 푝 푝 along each illumination direction, resulting in slope vectors [ 1] and [ 2] where: 푞1 푞2

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

푝1 푠1 [ ] = 푠1퐿1 = [ ] 푞1 0 푝2 푠2cos 훼 [ ] = 푠2퐿2 = [ ], (4.11) 푞2 푠2 sin 훼

푠1 and 푠2 are scalars denoting the normal computed by SI-SfS along each illumination direction, the unit vectors will be scaled accordingly to obtain the 푝 푝 desired surface normal vector suggested by SfS. The two slope vectors [ 1] and [ 2] 푞1 푞2 provide conditions for the PS-SfS solution, which implies that the final solution

푝푃푆 푝1 푝2 [ ] has to satisfy [ ] and [ ] simultaneously when projected to L1 and L2, 푞푃푆 푞1 푞2 respectively. This can be derived by computing the intersection point of the

푝1 푝2 perpendiculars of L1 and L2, originating at [ ] and[ ]: 푞1 푞2

푝푃푆 푝1 푝2 [ ] = 푚1푇1 + [ ] = 푚2푇2 + [ ], (4.12) 푞푃푆 푞1 푞2

0 −1 0 0 −1 −sin 훼 where 푇 = [ ] 퐿 = [ ] ; 푇 = [ ] 퐿 = [ ]. 푚 and 푚 are two 1 1 0 1 1 2 1 0 2 cos 훼 1 2 푝1 푝2 scalars such that (푚1푇1 + [ ]) and (푚2푇2 + [ ]) intersect. Combining equation 푞1 푞2 푝푃푆 푠1 4.11 and 4.12 yields ([ ] = [ ]) , where 푠1 is solved by SI-SfS along 푞푃푆 푚1 illumination direction 1 and 푚1 can be solved by searching 푚1 such that projecting

푠1 [ ] to 퐿2 yield 푠2: 푚1

[푠1 푚1]퐿2 = 푠2

푠2−푠1푝퐿2 푠2−푠1 cos 훼 푚1 = = , (4.13) 푞퐿2 sin 훼 Therefore,

푠 푠 푝푃푆 푠1 1 1 푠 −푠 cos 훼 푠 푠 [푞 ] = [푚 ] = [ 2 1 ] = [ 2 − 1 ], (4.14) 푃푆 1 sin 훼 sin 훼 tan 훼

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

From equation 4.14, we inferred that 푝푃푆 is solely dependent on 푠1, the SI- SfS slope along illumination direction 1, which can be attributed to the fact that it lies along the horizontal axis p of the coordinate frame. 푞푃푆 is affected by both

푠1 and 푠2 . Their relative contributions depend on the azimuthal illumination difference α. As a result, equation 4.14 contains the relationship between the PS-SfS solution and the azimuthal illumination difference, allowing for further analysis and propagation.

Figure 4.4 Determination of surface slope at point O through PS-SfS.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface 4.4.2 Error Propagation with Respect to Angular Illumination Differences

The sources of error of the solution as in equation 4.14 stem from 푠1, 푠2and α.

Among the three sources, 푠1 and 푠2 are directly connected to SfS since they are partial solutions estimated from the algorithm. To be more specific, we assumed that the error of α is negligible. To propagate the error caused by SI-SfS along each 푝 illumination direction, we first assume that [ 푃푆] is the true solution for PS-SfS and 푞푃푆

푝푃푆′ that [ ] is the solution where 푠1and 푠2 contain a certain level of errors: 푞푃푆′

푠1 + ∆푠1 푠 + ∆푠 푝푃푆′ 1 1 [ ] = [(푠 +∆푠 )−(푠 +∆푠 ) cos 훼] = [푠 +∆푠 푠 +∆푠 ], (4.15) 푞 ′ 2 2 1 1 2 2 − 1 1 푃푆 sin 훼 sin 훼 tan 훼

where ∆푠1 and ∆푠2 are positive or negative errors contained in 푠1and 푠2, leading to a distorted solution. Note that ∆푠1 and ∆푠2 are a combination of error introduced by multiple factors such as the error in reflectance modelling and in albedo, these errors are implicitly represented by equation 4.15. The squared error ε2 will then be the sum of the squared difference between the true solution and the distorted solution:

2 2 2 2 2 휀 = ∆푝푃푆 + ∆푞푃푆 = (푝푃푆′ − 푝푃푆) + (푞푃푆′ − 푞푃푆) , (4.16)

Rearranging and substituting yields:

∆푠 2 ∆푠 ∆푠 ∆푠 2 휀2 = ∆푝 2 + ∆푞 2 = ∆푠 2 + 2 − 2 1 2 + 1 푃푆 푃푆 1 sin2 훼 sin 훼 tan 훼 tan2 훼 1 1 ∆푠 ∆푠 = (1 + ) ∆푠 2 + ( ) ∆푠 2 − 2 1 2 (4.17) tan2 훼 1 sin2 훼 2 sin 훼 tan 훼

1 1 The first two components (1 + ) and ( ) contributes to a regular tan2 훼 sin2 훼 flat-bottomed U-shape function and the minimal will be at (α = 90o) in all cases. For these two components, ∆푠1 and ∆푠2 will affect the steepness of the flat-bottomed U- ∆푠 ∆푠 shape function. The last component (−2 1 2 ) alters the shape of the error sin 훼 tan 훼 function from a regular flat-bottomed U-shape to an inclined U-shape function. The

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface minima of the whole function will then depend on ∆푠1 and ∆푠2. If ∆푠1 and ∆푠2 are connected such that (∆푠2 = 푟∆푠1), equation 4.17 can be re-written as follows:

1 1 ∆푠 ∆푠 휀2 = (1 + ) ∆푠 2 + ( ) ∆푠 2 − 2 1 2 tan2 훼 1 sin2 훼 2 sin 훼 tan 훼 1 1 ∆푠 푟∆푠 = (1 + ) ∆푠 2 + ( ) 푟2∆푠 2 − 2 1 1 tan2 훼 1 sin2 훼 1 sin 훼 tan 훼 1 푟2 2푟 = (1 + + − ) ∆푠 2 tan2 훼 sin2 훼 sin 훼 tan 훼 1 2 2 = 푐 ∆푠1

1 푟2 2푟 휀 = √푐2∆푠 2 = 푐휎 = √1 + + − 휎 (4.18) 1 푠1 tan2 훼 sin2 훼 sin 훼 tan 훼 푠1

The coefficient c implies a root mean square error (RMSE) of gradients of c units per gradient error along the first illumination direction (i.e., 1휎푠1) derived from 1 푟2 SfS. The component ( + ) is U-shaped with respect to α for any real r, and tan2 훼 sin2 훼 2푟 its minimum is at 90°. However, the term (− ) changes the shape of c sin 훼 tan 훼 1 푟2 depending on α and r. When(0 < 푟 ≪ 1), ( ) will dominate because ( ) tan2 훼 sin2 훼 2푟 and (− ) will be scaled down by r and will be very close to zero. As a sin 훼 tan 훼 푟2 result, the curve of c will be U-shaped. When (푟 ≫ 1), ( ) will dominate and sin2 훼 2푟 overcome the effect of (− ), thus making the curve of c U-shaped. A special sin 훼 tan 훼 case exists when r is equal to or very close to 1. At small α, sin 훼 ≈ tan 훼, and therefore,

1 12 1 ≈ ≈ (4.19) tan2 훼 sin2 훼 sin 훼 tan 훼

This implies that

1 12 1 12 2 + ≈ + = (4.20) tan2 훼 sin2 훼 sin 훼 tan 훼 sin 훼 tan 훼 sin 훼 tan 훼

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

Hence, when (푟 = 1),

1 푟2 2푟 + − ≈ 0 tan2 훼 sin2 훼 sin 훼 tan 훼 1 푟2 2푟 푐2 = 1 + + − ≈ 1 (4.21) tan2 훼 sin2 훼 sin 훼 tan 훼

This implies that the error is very small at small α when (푟 ≅ 1) and increases with an increase in α, creating an exponential-shape curve. In general, the curve of the error coefficient c with respect to the azimuthal illumination difference α has a flat U-shape for most cases except when (푟 ≅ 1), where smaller c can be found at smaller α. While the azimuthal illumination difference α can be retrieved directly ∆푠 from the metadata of images in most cases, determining 푟 = 2, the ratio between ∆푠1 the SfS errors along the two illumination directions is a more difficult task. Hence, modelling r is important to understand the relationship between the azimuthal illumination difference and the performance of PS-SfS. Assuming a Lambertian reflectance model, PS-SfS solves for a solution that simultaneously satisfies:

푠푖∗sin 훽푖+cos 훽푖 퐿푎푚푖(푠푖) = = 퐿푎푚̅̅̅̅̅̅̅푖, (4.22) 2 √푠푖 +1

where 퐿푎푚푖(푠푖) represents the Lambertian reflectance for illumination direction i and βi is the sun-zenith angle (i.e., 0° at the overhead sun) of illumination direction i. 퐼 퐿푎푚̅̅̅̅̅̅̅ = 푖 is the expected reflectance value for the ith image, computed using image 푖 퐴 intensity Ii and intrinsic albedo A. A is the same for all images i as it corresponds to the same point on the surface, in this case, (𝑖 = 1,2) as only two images are

2 considered. 퐿푎푚푖(푠푖) can be simplified by assuming (√푠푖 + 1 = 1) to produce a unique solution without iterations. Hence, equation 4.22 can be simplified as follows:

퐿푎푚푖(푠푖) = 푠푖 ∗ sin 훽푖 + cos 훽푖 = 퐿푎푚̅̅̅̅̅̅̅푖 (4.23)

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

Therefore,

퐿푎푚̅̅̅̅̅̅̅̅푖−cos 훽푖 푠푖 = (4.24) sin 훽푖

If we assume that the error in sun-zenith angle (i.e., βi) is negligible, the SfS solution 푠푖 will solely be affected by 퐿푎푚̅̅̅̅̅̅̅푖. If the errors in Ii are assumed to be negligible, which is possible for radiometrically well-calibrated images, 퐿푎푚̅̅̅̅̅̅̅푖 will be largely affected by the errors in albedo A. Hence, the errors in 푠푖 (i.e., ∆푠푖) can be modelled as follows:

휕푠푖 퐼푖 ∆푠푖 = ∆퐴 = − 2 ∆퐴, (4.25) 휕퐴 퐴 sin 훽푖 where ∆퐴 corresponds to an error in A. Substituting equation 4.25 to r yields the following:

2 ∆푠2 −퐼2∆퐴 퐴 sin 훽1 퐼2 sin 훽1 푟 = = 2 = (4.26) ∆푠1 퐴 sin 훽2 −퐼1∆퐴 퐼1 sin 훽2

Equation 4.26 implies that the error model presented in equation 4.18 relies on not only the azimuthal illumination difference (horizontal angular difference) but also the zenith illumination difference (vertical angular difference). The image intensity ratio of the photometric stereo pair also contributes to the error model as described in equation 4.26. Note that this estimation does not require external information as the illumination zenith angles (i.e., βi) and the image intensities (i.e.,

Ii) are given at the very beginning. In this case, r is estimated on the basis of a simplified Lambert reflectance model, the viewing conditions (e.g., viewing direction and zenith) are currently not modelled. The viewing conditions may also contribute to r if other reflectance models such as the Lommel-Seeliger law, Lunar- Lambert model, or the Hapke model are utilized, in which case the complexity of the propagation is expected to increase significantly.

In general, I1 and I2 are positive as long as the images are not shadowed, the illumination zenith angles (i.e., βi) are positive because (1) βi are assumed to be non- 77

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface zero as stated in section 4.4.1 (2) negative βi implies a 180° change in α and hence, is compensated by α. Therefore, r will always be positive according to equation 4.26, which implies that the errors in SI-SfS along each of the directions (i.e., ∆푠1 and ∆푠2) are positively correlated and will increase or decrease simultaneously. This is reasonable in the sense that when the albedo A is lower than its true value, both 푠1 and 푠2will tend to incline towards the sun, leading to a positive error for along each the illumination directions and vice versa. Therefore, the ratio r will always be positive.

4.5 Experimental Analysis of the PS-SfS Approach

The PS-SfS algorithm and its subsequent error propagation model described in this chapter are verified using LROC NAC images of the lunar surface. LROC NAC images of the same location, acquired under multiple illumination conditions, are grouped into multiple pairs to produce high-resolution 3D models. Then, they are compared against a reference NAC DEM and the errors are analyzed.

4.5.1 Description of Verification Routine

The verification routine for the PS-SfS and the proposed error model is illustrated in Figure 4.5, multiple images covering the same area under different illumination conditions, known as photometric stereo image set, were obtained. They were co-registered and resampled to align and synchronise the pixels. This process was performed using the software ArcGIS 10.0 with manually selected tie points. The pixel-synchronous images were then geo-referenced to a reference DEM of the same area created independently by using photogrammetric methods (Burns et al., 2012). Both the images and the DEM are available in the LROC archive (http://lroc.sese.asu.edu/archive). The photometric stereo image set was decomposed into multiple photometric stereo pairs (i.e., two images per pair) with each pair containing a specific illumination azimuthal difference α. For each photometric stereo pair, the error coefficient c was estimated using the error model in equation 4.18 given α, and the parameter r was estimated using equation 4.26. The parameter

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface r was first estimated pixel-wise and then summarised for the entire image by using the mean, median, or other appropriate statistical methods. In this section, the mode values are used in the histogram of r to avoid outliers and to retain representativeness. Then, the photometric stereo pair is forwarded to PS-SfS reconstruction to produce a gradient map, the PS-SfS gradient map was compared with the slope map generated from the reference DEM, which will be treated as ground truth. As a result, the actual slope error was analyzed. The verification results were obtained by comparing the estimated slope error obtained using the proposed error model and the actual slope error. The slope error was computed by first resampling the slope map resulted from PS-SfS to the dimension of the ground truth and then applying the following equation:

1 1 푅푀푆퐸 = ( ∑ ∆푉 푇∆푉 )2 , 푠푙표푝푒 푛 푖∈푛 푖 푖

∆푉 = 푉푃푆 − 푉푟푒푓 (4.27)

푝푃푆 푝푟푒푓 where 푉푃푆 = [푞푃푆] and 푉푟푒푓 = [푞푟푒푓] are the normal vector of the surface generated 1 1 from PS-SfS and the reference DEM, respectively; n is the number of samples (i.e., pixels). The angular error was also analyzed for a more intuitive presentation, which describes the inclination angle in three dimensions between the normal vector of the surface computed from PS-SfS and that of the ground truth, as follows:

1 2 1 푉 푇푉 푅푀푆퐸 = ( ∑ cos−1 푃푆 푟푒푓 | ) (4.28) 푎푛푔푢푙푎푟 푛 푖∈푛 ‖푉 ‖‖푉 ‖ 푃푆 푟푒푓 푖

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

Figure 4.5 Verification routine with real data for the proposed error model.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface 4.5.2 Description of Datasets

Two LROC NAC datasets are selected for experimental evaluation of the PS- SfS algorithm. Due to the moon’s rotation characteristics, most image sets that covered a wide range of illumination differences are found in higher latitudes. The first dataset (Antoniadi dataset) set contains five co-registered LROC NAC images covering the same region. Figure 4.6 shows the pixel-synchronous images (2431 × 372 pixels) processed according to Section 4.5.1, of which the imaging conditions are summarised in Table 4.1. The resolution for both the pixel-synchronous images and the reference DEM is 2m/pixel. The region is located in the southern hemisphere (i.e., 70°S latitude), near the Antoniadi crater, the illumination is therefore mainly coming from the northern region and the sun-zenith angle is relatively high. The five images produce 10 photometric stereo pairs with their azimuthal and zenith illumination differences summarised in Table 4.2.

a) M1410 b) M1363 c) M1646 d) M1122 e) M1168 f) Reference DEM 45774 03002 22650 344699 337480 Figure 4.6 Pixel-synchronous LROC NAC images and the reference NAC DEM for the Antoniadi dataset.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface Table 4.1Imaging and illumination conditions of the images in the Antoniadi dataset. Subsolar Sun-zenith Sub-spacecraft Emission Image ID azimuth angle azimuth angle

M141045774 5.53° 69.13° 292.84° 19.62°

M136303002 313.10° 75.52° 61.98° 11.67°

M164622650 279.50° 86.23° 161.92° 1.15°

M1122344699 59.22° 79.64° 4.18° 1.69°

M1168337480 46.46° 76.77° 269.97° 1.14°

Table 4.2 Azimuthal difference (α)and zenith difference (embedded in r) of the illuminations for photometric stereo pairs used in the Antoniadi dataset.

r M1410457 M1363030 M1646226 M1122344 M1168337 α 74 02 50 699 480

M141045774 1.373 3.128 0.489 0.641

M136303002 52.4° 2.613 0.674 0.872

M164622650 86.0° 33.6° 1.384 1.535

M1122344699 53.7° 106.1° 139.7° 0.769

M1168337480 40.9° 93.4° 127.0° 12.8°

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface The second dataset (Bogslwsky dataset) contains four LROC NAC images covering a region located at about 70°S latitude, near the Bogslwsky crater. Figure 4.7 shows the pixel-synchronous images (1536 × 512 pixels) used for the verification, and the imaging conditions are summarised in Table 4.3. The resolution for both the pixel-synchronous images and the reference DEM is 2m/pixel. Similar to the case of the Antoniadi dataset, the illumination of the images mainly comes from the northern region and the solar zenith angle is relatively high. Two images out of four have a solar zenith angle of more than 80°, which may contain significant shadows in craters. The four images produce six photometric stereo pairs and their azimuthal and zenith illumination differences are summarised in Table 4.4.

a) M1225 b) M1153 c) M1443 d) M1188 e) Reference DEM 787254 886196 49709 124163 Figure 4.7 Pixel-synchronous LROC NAC images and reference NAC DEM of the Bogslwsky dataset

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface Table 4.3 Imaging and illumination conditions of the images in the Bogslwsky dataset. Sub- Subsolar Image ID Sun-zenith angle spacecraft Emission angle azimuth azimuth

M1225787254 3.64° 69.01° 282.17° 2.86°

M1153886196 72.43° 83.11° 27.21° 1.69°

M144349709 45.46° 74.16° 165.26° 1.69°

M1188124163 278.71° 85.37° 357.80° 1.13°

Table 4.4 Azimuthal difference (α)and zenith difference (embedded in r) of the illuminations for photometric stereo pairs in the Bogslwsky dataset. r M1225787254 M1153886196 M144349709 M1188124163 α

M1225787254 0.304 0.681 3.499

M1153886196 68.8° 0.438 1.055

M144349709 41.8° 27.0° 2.217

M1188124163 84.9° 153.7° 126.8°

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface 4.5.3 Evaluation and Analysis of the Antoniadi Dataset

Table 4.5 summarises the essential parameters for the proposed error model, the error estimated using the proposed error model in the case of real data and the actual error from PS-SfS. Figure 4.8a shows the plot of the estimated error coefficient c with respect to α. It is apparent that when α is 33.6°, 86.0°, 127.0°, and 139.7°, c jumps significantly because of the comparatively high r. From equation 4.26, we inferred that a large r implies that the error in SI-SfS along illumination direction 2 (i.e., ∆s2) is a few times larger than the SI-SfS error in illumination direction 1 (i.e., ∆s1), contributing an even larger error to the final result. These photometric stereo pairs with a high error coefficient c shared image M164622650, as shown in Table 4.2. This image has a sun-zenith angle of 86.23°, which implies that the sunlight is coming from almost the horizon. As a result, it makes the image darker than the others, possibly creating a considerable amount of cast shadow, hence leading to relatively large errors. Figure 4.9 reveals that the PS-SfS DEM derived from pairs with the image M164622650 (i.e., Figures 4.9b, f, i, and j) have significantly fewer topographic details than the other pairs. The angular error maps in Figure 4.10 also suggest similar findings, when a DEM contains fewer topographic details than the ground truth, small local topographic features will be represented by smooth surfaces, yielding a larger angular error in these regions and hence, increasing the global angular error. It is noted that there are apparent horizontal stripes on the angular error maps in Figure 4.10 because there are small but dense horizontal systematic artefacts on the reference DEM which is treated as ground truth, hence these artefacts are then inherited to the angular error maps during analysis. However, as all photometric stereo pairs within this dataset share the same ground truth, the horizontal artefacts do not affect the internal consistency between pairs and hence the verification of the error model is not affected. The shape of the error coefficient curve c with respect to α (Figure 4.8a) exhibits a very high correspondence with the actual slope error shown in Figure 4.8b (blue solid line). According to equation 4.18, the error estimated using the proposed error model (i.e., ε) is the product of the error coefficient c and an estimate of the SI-

SfS slope error along illumination direction 1 (i.e., σs1). Assuming that the values of

σs1 of all photometric stereo pairs within the image set 1 are identical, the estimated

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface error curve yields the least-squares best fit at σs1 = 0.070, approximately 4°, as shown in Figure 4.8b (magenta dotted line). The result shows a high consistency with the actual error curve with a fitting RMSE of 0.017, which implies that the proposed error model can estimate the relative error of PS-SfS between different photometric stereo pairs and estimate the PS-SfS slope error when the estimated SI-

SfS slope error along illumination direction 1 (i.e., σs1) is given.

Table 4.5 Estimated error from the proposed error model and actual PS-SfS error for the Antoniadi dataset.

α° 12.8 33.6 40.9 52.4 53.7 86.0 93.4 106.1 127.0 139.7

r 0.769 2.613 0.641 1.373 0.489 3.128 0.872 0.674 1.535 1.384

c 1.369 3.368 1.015 1.388 1.008 3.225 1.367 1.407 2.854 3.468

(흈풔ퟏ 0.096 0.235 0.071 0.097 0.070 0.226 0.096 0.098 0.200 0.243 = ퟎ. ퟎퟕퟎ) Actual slope 0.105 0.223 0.092 0.066 0.106 0.234 0.074 0.095 0.209 0.241 error Actual angular 4.64 11.30 4.02 3.12 4.95 11.63 3.40 4.45 10.38 12.30 error (°)

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

a) Estimated theoretical error coefficient

b) Best-fit slope error estimate and actual slope error

Figure 4.8 Comparison between the estimated theoretical error from the proposed error model and actual error from PS-SfS of the Antoniadi dataset.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

a) α = b) α = c) α = d) α = e) α = k) 12.8° 33.6° 40.9° 52.4° 53.7° Reference DEM

f) α = g) α = h) α = i) α = j) α = Colour 86.0° 93.4° 106.1° 127.0° 139.7° scale (m) Figure 4.9 3D views of PS-SfS DEMs at different α of the Antoniadi dataset.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

a) α = 12.8° b) α = 33.6° c) α = 40.9° d) α = 52.4° e) α = 53.7°

f) α = 86.0° g) α = 93.4° h) α = 106.1° i) α = 127.0° j) α = 139.7°

Figure 4.10 Angular error maps from photometric stereo SAfS at different αof the Antoniadi dataset.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface A vertical error analysis usually gives a more strictly forward and intuitive impression of the quality of the DEMs. With reference to the vertical RMSE plots with respect to α (Figure 4.11a), larger fluctuations are observed for α ≤ 90°. The shape of the vertical error plot with respect to α is visually different from that of the slope error curves shown in Figure 4.8, this is reasonable because the conversion of the reconstructed slopes to DEMs involves various processes such as offsetting and integrability enforcement. As a result, the DEM is, in fact, the best-fit solution of the slope map instead of a perfect fit, and hence the error patterns in the SfS slope maps are unlikely to be inherited completely after the process. However, we might notice a higher RMSE at the lowest available α, this may imply that photometric stereo pairs with α ≤ 90° have a higher possibility of resulting in larger vertical errors. An in- depth investigation is required to confirm this observation. Figure 4.11b reveals that the greyish patches (i.e., areas with a larger absolute vertical difference) are in large clusters instead of scattered small patches and exhibit gradual changes from bright to dark. This probably implies that there are slope deviations in the low-resolution components (i.e., low-frequency components) of the resulting SfS DEMs. Low- frequency inconsistency is a common issue in photoclinometric surface reconstruction algorithms and DEM estimation from slope algorithms (Frankot and Chellapa, 1988; Grumpe et al., 2014). Figure 4.11b also reveals significant spatial patterns in absolute vertical differences of the SfS DEMs, this implies that all the resulted SfS DEMs deviated from the reference DEM in similar spatial distribution. Hence implying the performance of the proposed algorithm is generally stable under a wide spectrum of α. In general, although there are fluctuations depending on different values of α, the variations are not large, the vertical RMSE is always around 15 m, which is considered relatively stable.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

a) Vertical RMSE plot

α = 12.8° α = 33.6° α = 40.9° α = 52.4° α = 53.7°

α = 86.0° α = 93.4° α = 106.1° α = 127.0° α = 139.7°

b) Absolute difference DEMs

Figure 4.11 Vertical RMSE and the corresponding absolute difference DEMs of the Antoniadi dataset.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface 4.5.4 Evaluation and Analysis of the Bogslwsky dataset

Table 4.6 summarises the parameter r and estimated coefficient c values of the proposed error model, the error estimated by the proposed error model according to the real data situation, and the actual error from PS-SfS, Figure 4.14a shows the corresponding plot of c with respect to α. When α is 84.9°, 126.8°, and 153.7°, the estimated error coefficient c increases significantly because of high r. This increase may be attributed to the image M1188124163 shared by these three pairs. Image M1188124163 (Figure 4.7d) has a large solar zenith angle according to Table 4.3, and although its solar zenith angle is not significantly larger than that of M1153886196 (Figure 4.7b), it is apparently darker than the other images, as shown in Figure 4.7. This contributes to a larger value of r and possibly larger errors because of the presence of shadows. The 3D views of the PS-SfS DEMs in Figure 4.12 reveal that the aforementioned pairs (Figures 4.12d, e, and f) contain fewer local scale topographic details, the reconstructed shape of the larger crater at the bottom-left of the image almost vanishes in Figures 4.12d and e because of strong shadows. In Figure 4.12f, although the larger crater is reconstructed slightly better, most of the small topographic features such as craters and boulders could not be reconstructed very well in terms of the crater depth or the boulder height. The angular error maps in Figure 4.13 present a visual correspondence between the dark patches in Figures 4.13d, e, and f and the shadow areas in image M1188124163 (Figure 4.13d), this correspondence suggests that the errors are likely to be associated with shadows. The error coefficient plot c with respect to α (Figure 4.13a) has a significant correspondence with the actual slope error in Figure 4.13b (blue solid line). Although the similarity is not as high as in the Antoniadi dataset because of a considerable increase in α ≥ 84.9°, the shape correspondence of the curves is generally maintained. Assuming that σs1 for all photometric stereo pairs are the same, we found that the estimated error plot of ε with respect to α obtained the least- squares best fit at σs1 = 0.062, approximately 3.5°, with a fitting RMSE of 0.057. The deviation between the estimated error curve ε from the proposed error model and the actual error curve is believed to stem from the fact that the iterative PS-SfS algorithm described in this chapter has implicitly enforced the integrability of the

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface resulting gradient map, suppressing its errors to a certain extent. Integrability enforcement is an essential component of most shape from gradients applications and hence, is not within the scope of our PS-SfS error analysis with respect to the azimuthal illumination difference. This also implies that the resulting slope maps for α = 84.9°, 126.8°, and 153.7° in this image set are likely to be more strongly non- integrable. Another possible reason for the inconsistency points to the possibility that the parameter σs1 for the first three pairs (i.e., α = 27.0°, 41.8°, and 68.8°) is considerably lower than that of the last three pairs, likely due to shadows.

a) α = 27.0° b) α = 41.8° c) α = 68.8° g) Reference DEM

d) α = 84.9° e) α = 126.8° f) α = 153.7° h) Colour scale (m) Figure 4.12 3D views of photometric stereo SAfS DEMs at different α of the

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface Bogslwsky dataset.

a) α = 27.0° b) α = 41.8° c) α = 68.8°

d) α = 84.9° e) α = 126.8° f) α = 153.7°

Figure 4.13 Angular error maps from photometric stereo SAfS at different αof the Bogslwsky dataset.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

a) Estimated theoretical error coefficient

b) Best-fit slope error estimate and actual slope error Figure 4.14 Comparison between the estimated theoretical error from the proposed error model and actual error from PS-SfS of the Bogslwsky dataset.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface Table 4.6 Estimated error from the proposed error model and actual PS-SfS error of the Bogslwsky dataset.

α° 27.0 41.8 68.8 84.9 126.8 153.7

r 0.438 0.681 0.304 3.499 2.217 1.055

c 1.413 1.005 1.002 3.567 3.653 4.520

ε 0.088 0.062 0.062 0.221 0.226 0.280 (흈풔ퟏ = ퟎ. ퟎퟔퟐ)

Actual slope error 0.175 0.134 0.131 0.199 0.192 0.267

Actual angular error 7.943 5.921 5.990 9.908 9.213 13.630 (°)

With reference to the vertical RMSE plots with respect to α of the Bogslwsky dataset (Figure 4.15a), the RMSE curves of this dataset exhibit a very slight U- shape. Similar to the case of the previous dataset, the vertical error plot presents a visible deviation from its slope error plot, for which similar reasons explained in the previous experimental datasets are expected for this deviation. The Bogslwsky dataset also presents its highest RMSE at the lowest available α, this follows the findings and implications of the experimental analysis of the Antoniadi dataset. Figure 4.15b reveals large greyish clusters with gradual grey level changes. They are likely to be slope errors remaining in the low-resolution components of the resulting SfS DEMs. Similar to the case of section 4.5.3, they also present strong similarities in spatial patterns, which implies the proposed algorithm is producing stable results under various α. In general, the vertical RMSE of the Bogslwsky dataset corresponds to 12-13 m. In conclusion, the general performance of PS-SfS is relatively stable for different values of α in terms of the vertical analysis.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface

a) Vertical RMSE plot

α = 27.0° α = 41.8° α = 68.8°

α = 84.9° α = 126.8° α = 153.7°

b) Absolute difference DEMs

Figure 4.15 Vertical RMSE and the corresponding absolute difference DEMs of the Bogslwsky dataset.

Chapter 4 Photometric Stereo Shape-from-Shading (PS-SfS) for 3D Modelling of the Lunar Surface 4.6 Summary

This chapter presents a PS-SfS approach for constructing high-resolution lunar DEM based on a pair of co-registered images acquired under different illumination conditions, the approach does not require any initial DEM. First, the effects of location-dependent albedo are mathematically eliminated and the surface gradient along the principal direction is derived. A DEM is then generated and refined according to the resulted surface gradients. Moreover, an error model is developed, based on the configuration of the proposed PS-SfS algorithm, to relate illumination inconsistency of the image pair and its subsequent PS-SfS performance. The error model suggests that the slope error in PS-SfS is affected by not only the illumination azimuthal difference but also the illumination zenith angles and the general intensity (e.g., mean intensity) of the images. The error model was verified by the developed PS-SfS approach, with experimental analyses using real datasets. The experimental results showed that the slope errors estimated from the proposed error model correspond well with the actual slope errors obtained from PS-SfS. Slight inconsistencies were attributed to the integrability constraint enforced by the PS-SfS algorithm, which was out of the scope of the developed error model. The estimated coefficients from the error model were scaled in order to best fit the actual error from the experimental analysis, the fitted result suggested that the PS-SfS approach has to a slope error along the first illumination direction of around 0.06- 0.07, approximately 3o-4o. The vertical error analysis did not show a very apparent pattern that corresponded to the proposed error model, which is reasonable because the point-to-point vertical error was unable to directly represent slope errors and the process of converting surface slopes into heights includes other constraints such as enforcement of integrability and/or vertical offsetting, which might alter the error values. The vertical error analysis revealed the possible low-resolution slope deviations in PS-SfS, which are common in photoclinometry. In general, the vertical error fluctuations with respect to the directional illumination difference are small for both experimental datasets.

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

The methods described in previous Chapters focus on using SfS, with or without the assistance of initial DEM derived from photogrammetry and/or laser- altimetry, to generate lunar DEMs of optimal spatial resolution, comparable to the image(s). This chapter focuses on how SfS concepts and techniques can be integrated to the photogrammetric pipeline, and to extend the ability of photogrammetry. As will be described in the following sections, SfS will be integrated with photogrammetry in two ways to assist. On one hand, PS-SfS is employed to produce pixel-wise illumination-invariant matching points for the photogrammetric pipeline to generate a 3D model of the lunar surface. On the other hand, the resulted lunar 3D model will be refined to optimal resolution based on SI- SfS. The approach was applied to the detailed 3D mapping of the Chang’E-4 and Chang’E-5 landing sites, which are presented at the end of this chapter.

5.1 Overview of the Integration Approach

The workflow of the integrated photogrammetric and photoclinometric approach is illustrated in Figure 5.1. The approach starts with a pair of lunar surface images with their corresponding EO parameters. The images are expected to contain visible illumination differences. Epipolar rectification of the images, based on their EO parameters, is required for generating 3D models using photogrammetric techniques. The epipolar images are then forwarded to the photoclinometry assisted image matching (PAM) algorithm to obtain the disparity maps. PAM is illumination- invariant and is able to match images with large illumination differences. A photogrammetric DEM is then generated using the disparity maps and the EO parameters of the images. Finally, the photogrammetric DEM is refined to optimal resolution, comparable to the image, based on the SI-SfS approach described in Chapter 3.

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

Figure 5.1 The overview of the integrated photogrammetric and photoclinometric approach.

5.2 Pixel-Wise Image Matching Invariant to Illumination Inconsistencies

5.2.1 Photoclinometry-Assisted Image Matching

Photoclinometry-Assisted Image Matching (PAM) is developed based on the concepts and developments of PS-SfS described in Chapter 4. By exploiting the nature of PS-SfS, which its robustness generally increases with illumination inconsistency, the PAM algorithm finds matches by evaluating the underlying terrain, in the form of surface gradients, instead of the direct image intensities. As a result, the differences in image intensity caused by the illumination differences do not significantly influence the matching.

100

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

The workflow of the PAM algorithm is illustrated in the left diagram of Figure 5.2, the design of PAM follows the general architecture of SGM

(Hirschmüller, 2008), in which the base image (Ib) and matching image (Im) are epipolarly rectified according to their EO parameters. The objective is to find the horizontal disparity dx and vertical disparity dy such that a pixel on the matching image Im(x+dx, y+dy) correctly corresponds to the pixel on the base image Ib(x,y). Using disparities in two directions (x and y) ensures the matching is robust when the epipolar images are not perfectly aligned. For each pixel within a predefined disparity range (dx,dy), PAM extracts a local patch centered at Ib(x,y) and Im(x+dx, y+dy). Using PS-SfS, PAM then analyzes the underlying terrain produced by the pair of image patches and generates a matching score representing the likelihood of the resulting terrain (right diagram of Figure 5.2, see section 5.2.2). The algorithm chooses the one with the highest matching score as the best match. PAM also considers the local smoothness and conducts multiple-scale evaluations when computing the matching score. Matches with significant changes in disparity are treated as mismatches and are filtered out, and the remaining matches are forwarded to the sub-pixel refinement. As the PAM algorithm incorporates the principles of PS- SfS, the illumination direction with respect to each of the images has to be known. This information can be obtained from the image metadata or derived by manual observation.

Base Image Matching Image Local Patch on Local Patch on Base Image Matching Image

Matching Score for Each Ratio of the Local Patches Candidate Disparity

Photometric Stereo Reconstruction Pixel-level Disparity Maps

Mismatch Filtering Gradient (s) in Principal Direction

Sub-pixel Refinement

Sub-pixel Disparity Maps Matching Score Function

Figure 5.2 The workflow of photoclinometry assisted image matching (PAM).

101

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

5.2.2 Computation of Matching Score

The workflow for computing the matching score is also presented in Figure 5.2 (right diagram). For any local patch on the base image centred at pixel (x,y), a patch of the same dimension is extracted from the matching image with its centre pixel translated by the disparities (dx,dy) under evaluation. A ratio of the two local patches is computed by dividing the patch on the base image by the one on the matching image. Using the ratio minimises the effects of varying albedo in the photometric stereo analysis, as can be seen in equation 4.3. The PS-SfS technique described in Chapter 4 is then used to determine the gradient map of the pair along the principal direction (i.e., determined according to section 4.2.2). The rationale behind PAM is that when the two image patches are matched correctly, they produce a highly reasonable gradient map due to the removal of the albedo variations by equation 4.2 and the accurate alignment of the corresponding pixels. However, when the two patches are matched incorrectly, they produce a biased gradient map due to misalignment. Because the map presents the gradient along the principal direction (i.e., u in Figure 4.2), it represents more topographic information along this direction and less information along its perpendicular (i.e., v in Figure 4.2). Hence, the gradient changes more frequently along direction u and less frequently along direction v. This is similar to the situation in which an image shows more changes in intensity along the direction of illumination and less along its perpendicular. Based on this intuition, biased gradient maps usually show similar roughness along multiple directions and have similar patterns of input images (influenced by the illumination), while reasonable gradient maps usually show high roughness only along the principal direction and low roughness along its perpendicular. Hence, the roughness of the gradient map can be evaluated along two orthogonal directions. The two directions are then combined using a mathematical function representing the matching score. When a patch on the base image is matched with a candidate patch on the matching image, a gradient map can be generated using the aforementioned PS-SfS technique. If the two patches are correctly matched, the gradient map is likely to have more frequent changes along the principal direction and less along its perpendicular. Otherwise, the map will have frequent changes along different

102

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface directions due to “leakage” of the topographic and albedo information as a result of misalignment. PAM uses the gradient roughness to measure the changes. A reasonable gradient map is evaluated as having high gradient roughness along the principal direction and low gradient roughness orthogonal to the principal direction. The roughness of the gradient (s) map along direction t is evaluated using the following function:

2 푆(푠, 푡, 휎)푥,푦,푑푥,푑푦 = 𝑔(푠푡푡 , 휎)푥,푦,푑푥,푑푦, (5.1)

2 where 푆(푠, 푡, 휎)푥,푦,푑푥,푑푦 is the mean squared deviation of 푠푡푡 , the squared second derivative of s along the direction t, within a local region centred at pixel (x,y) with disparity (dx, dy). 𝑔(∗, 휎) is a Gaussian function with the parameter 휎 controlling the size of the local region in the multiple-scale analysis and the relative contribution of 2 each component within the region. 푠푡푡 can be approximated by finite differences:

2 2 푠푡푡 = (푠푡−1 − 2푠푡 − 푠푡+1) (5.2)

A correct match is then defined as a high 푆(푠, 푢, 휎) and a low (푠, 푣, 휎), of which u is the principal direction and v is its perpendicular. To achieve this, 푆(푠, 푢, 휎) and 푆(푠, 푣, 휎) are connected using a score function:

푀(푥, 푦, 푑 , 푑 , 휎) = 1 − 푒−푤푥푆(푠,푢,휎) + 푒−푤푦푆(푠,푣,휎)| (5.3) 푥 푦 푥,푦,푑푥,푑푦

In the above function, the first component 1 − 푒−푤푥푆(푠,푢,휎) increases when 푆(푠, 푢, 휎)

−푤푦푆(푠,푣,휎) increases, while the second component 푒 penalises a high 푆(푠, 푣, 휎). wx and wy are weighting factors used to control the relative contribution of each component. A larger wx and wy mean the score is more easily affected by a small change in 푆(푠, 푢, 휎) and 푆(푠, 푣, 휎), respectively, and vice versa. M ranges from 0 to 2, with the best match approaching 2. To ensure the robustness of the matching score, PAM takes the multiple-scale evaluation into consideration by aggregating the matching scores computed from multiple image scales:

103

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

푀푓푖푛푎푙(푥, 푦, 푑푥, 푑푦) = ∑푘휖퐾 푀푘(푥, 푦, 푑푥, 푑푦, 휎푘), (5.4)

where 푀푘(푥, 푦, 푑푥, 푑푦, 휎푘) is the matching score at a scale denoted by 휎푘 within a predefined number K of scales. For computational consideration, this can be approximated by down-sampling the image by a factor (2 in this research) and computing the matching score for each down-sampled image. For each pixel (x,y), the disparities (dx,dy) with the highest matching score 푀푓푖푛푎푙(푥, 푦, 푑푥, 푑푦) are chosen as the best match.

5.2.3 Filtering of Mismatches

Mismatches are unavoidable in most cases. In this section, mismatches are defined as pixels with a disparity significantly different from their adjacent neighbours:

퐷푥(푥, 푦) = |푑푥(푥, 푦) − 푑푥∈ 푛| ≥ 휀

퐷푦(푥, 푦) = |푑푦(푥, 푦) − 푑푦∈ 푛| ≥ 휀, (5.5) where n is the vicinity of any pixel (x,y), and is defined as 1 pixel adjacent to the centre pixel. 휀 is a predefined threshold default at 1. Equation 5.5 is implemented by using morphological operators on the disparity maps:

퐷푥(푥, 푦) = |푚표푟푝ℎ푑푖푙(푑푥(푥, 푦), 푛) − 푚표푟푝ℎ푒푟표(푑푥(푥, 푦), 푛)|

퐷푦(푥, 푦) = |푚표푟푝ℎ푑푖푙(푑푦(푥, 푦), 푛) − 푚표푟푝ℎ푒푟표(푑푦(푥, 푦), 푛)|, (5.6)

where 푚표푟푝ℎ푑푖푙(∗, 푛) is a dilation operator whereby the value of a pixel is replaced by the highest value within a neighbourhood defined by n, and 푚표푟푝ℎ푒푟표(∗, 푛) is an erosion operator whereby the value of a pixel is replaced by the smallest value within the neighbourhood. The neighbourhood defaults as a 3x3 kernel, which means that only the disparities of the adjacent pixels are compared. A pixel is labelled as a mismatch when 퐷푥(푥, 푦) ≥ 휀 or 퐷푦(푥, 푦) ≥ 휀. Pixels enclosed by mismatches are also labelled as mismatches, which can be achieved by using an algorithm such as flood fill or watershed analysis. After filtering the mismatches, the sub-pixel values

104

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface of the matches are estimated by fitting a quadratic curve over the matching score of the neighbouring disparities and finding the minima (Hirschmüller, 2008).

5.3 Refinement of a Photogrammetric DEM based on Shape-from-Shading

Using the above matching results from PAM, a photogrammetric DEM can be generated based on the EO parameters. Afterwards, the SI-SfS reconstruction algorithm described in Chapter 3 is applied to recover details on the image to the photogrammetric DEM. In the integrated framework, the photogrammetric DEM is automatically used as the initial condition and as a constraint for the SfS algorithm. Because only one image of the stereo pair is needed for the process, the image with better illumination conditions, such as fewer shadows and adequate topographic shading, is chosen. The chosen image from the stereo pair is first ortho-rectified using the associated EO parameters and the photogrammetric DEM. Because the ortho-image and the photogrammetric DEM are generated from the same routine, they are automatically co-registered. The ortho-image and the photogrammetric DEM are then imported to the SI-SfS reconstruction algorithm to produce a lunar DEM with spatial resolution and topographic details of the image. Based on the reflectance (topographic shading) map generated from the ortho-image and constrained by the initial DEM, the photogrammetric DEM is refined based on the image and a local constant albedo assumption, using the following equation:

퐼푥,푦 퐺푗 퐺푠푓푠 = ∑푗∈푁푥,푦 , (5.7) 푛 퐼푗

th where each Ij is the image intensity value of the ortho-image at the j pixel, within a predefined local neighbourhood Nx,y of the centre pixel Ix,y. The surface normal vectors of the photogrammetric DEM are then refined according to the estimated reflectance and the constraints imposed by the photogrammetric DEM, whilst the image pixels under shadows are refined according to the shadow

푝푓푖푛푎푙 constraint. The final refined surface normal vectors [ ] are a weighted 푞푓푖푛푎푙 combination of the intermediate results:

105

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

푝푓푖푛푎푙 푝푠푓푠 푝푠ℎ푤 [ ] = (1 − 푤푠ℎ푤) [ ] + 푤푠ℎ푤 [ ], (5.8) 푞푓푖푛푎푙 푞푠푓푠 푞푠ℎ푤

푝푠푓푠 where [ ] is the estimated surface normal vectors for the non-shadowed pixels, 푞푠푓푠

푝푠ℎ푤 and [ ] is that for the shadowed pixels. wshw is the weight that determines the 푞푠ℎ푤 likelihood of an image pixel being shadowed, which can be computed by a shadow detection algorithm. The derivation of the surface normal vectors was described and discussed in detail in Chapter 3. Next, each height post of the photogrammetric DEM is refined iteratively according to the resulting surface normal vectors, using the following equation:

−1 푍 푘+1 = 푍 푘 + (∑ [휕 푁푇∆푁 ] ) (∑ [휕 푁푇휕 푁] ) 푥,푦 푥,푦 푖,푗 푍 푓푖푛푎푙 푖,푗 푖,푗 푍 푍 푖,푗

푝 − 푝 푇 휕푝 휕푞 푓푖푛푎푙 퐷퐸푀 휕푍푁 = [ ] ; ∆푁푓푖푛푎푙 = [ ], (5.9) 휕푍 휕푍 푞푓푖푛푎푙 − 푞퐷퐸푀

푘 th 푘+1 where 푍푥,푦 is the height value at post (x,y) at k iteration, and 푍푥,푦 is the

푝푓푖푛푎푙 updated height value. [ ] is the surface normal vectors refined using the SI-SfS 푞푓푖푛푎푙 푝 approach, and [ 퐷퐸푀] is the surface normal vectors of the DEM at the kth iteration. 푞퐷퐸푀 As described in Chapter 3, the Modified-Jacobi relaxation strategy (Figure 3.2 and 3.3) was applied to solve the large but sparse system, whilst a hierarchical multi-grid architecture was used to effectively reconstruct the DEM.

106

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

5.4 Experimental Analysis of the Integration Approach

5.4.1 Experimental Analysis at the Chang’E-4 Landing Site

The LROC NAC images feature a high resolution, 0.5-2 m/pixel, which is favourable for the detailed 3D mapping and analysis of the landing sites for lunar missions. Figure 5.3 shows a pair of LROC NAC images covering the landing site of the Chinese Chang’E-4 mission, which successfully landed on the far side of the moon on January 3, 2019. To study the topographic conditions of the landing site and the nearby environment, it is critical to have high-resolution DEMs of the region to support the topographic analysis. The stereo pair of LROC NAC images shown in Figure 5.3 are those only available before and immediately after the landing. As can be seen in Figure 5.3, the difference in the slew angles (convergence angles) between the images is approximately 22o, which is ideal for photogrammetric processing. However, the illumination directions of the two images are almost opposite to each other, differing by about 164o, which implies that the conventional matching algorithms are likely to fail on this dataset. The images are associated with their EO parameter obtained from the SPICE kernels (Acton, 1996; Speyerer et al., 2016). To facilitate comparison, a reference DEM was generated independently using photogrammetry based on another stereo pair of LROC NAC images (image IDs: M1303619844LR and M1303640934LR) collected one month after the landing of Chang’E-4, which has consistent illumination conditions. The reference DEM has a resolution of 5 m/pixel, and it is available from the LROC Archive (http://www.lroc.asu.edu/posts/1100).

107

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

Landing region on Landing region on M134022629L M178833263L M134022629L M178833263L

Image ID M134022629L M1145135367L

Slew angle (o) 0.0 22.1

Illumination azimuth (o) 282.4 86.2

Incidence angle (o) 77.6 86.7

Emission angle (o) 1.7 23.0

Resolution (m/pixel) 1.3 1.4

Figure 5.3 The LROC NAC image pair covering the Chang’E-4 landing site. Blue boxes show the enlarged views of the landing area, and green crosses indicate the landing site on each of the images. Yellow arrows indicate the illumination direction of each image.

108

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

We focus our experimental analysis on the region around the Chang’E-4 landing site. Figure 5.4a shows the image subset after epipolar rectification. The resolution of the epipolar image is resampled to 4 m/pixel, which is considered optimal for photogrammetric processing, given the resolution of the original images. The resolution of 4 m/pixel is also suitable for comparison with the reference DEM. The dimension of the image is 2000 x 1320 pixels. Due to the illumination differences, the conventional matching algorithms fail to produce any reasonable DEMs. The disparity maps are generated by matching the images using PAM and are shown in Figure 5.4b. Although the disparity maps correspond to the general topography, the detailed topographic features (e.g., small craters) are missing because they are severely shadowed in both images. These regions are outlined in blue in Figure 5.4b and mostly correspond to crater and depressions, of which the western portions are shadowed in the left image while the eastern portions are shadowed in the right image. Hence, when the images are analyzed during PAM matching, most of the craters are either shadowed by the left image or by the right image and thus there is insufficient information inside the craters for matching. In general, the image pixels are required to be illuminated in both images for the best performance of PAM. In situations like this, PAM matches the portions of the craters by their illuminated surroundings such as the crater rim, and hence the disparities of the inner parts are replaced by those of the surroundings, leading to a smooth surface. If the craters are shaded instead of shadowed, there will be information within the craters and their disparities can be more accurately determined by PAM. For the regions not shadowed, the disparity maps successfully retrieve their shapes. The 3D view of the photogrammetric DEM from PAM (Figure 5.5b) closely corresponds with the reference DEM (Figure 5.5a) in most of the regions except for the shadowed regions. This problem can be overcome by choosing the image with the least shadows from the image pair and performing SI-SfS. Figure 5.5d shows the ortho-image generated using the photogrammetric DEM with the least shadows. The ortho-image (1.5 m/pixel) and the photogrammetric DEM (4 m/pixel) are used as inputs in the SI-SfS system, and an integrated DEM (1.5 m/pixel) is generated using the method described in Chapter 3, as shown in Figure 5.5. Since the reference DEM (Figure 5.5a) and the photogrammetric DEM (Figure 5.5b) are produced independently using different sets of stereo images, small low-frequency deviations (about 1-3 m) in elevation, with respect to the reference DEM, can be noted on the 109

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface photogrammetric DEM (Figure 5.5b) and the integrated DEM (Figure 5.5c). The 3D views of the DEMs (Figure 5.5) and their corresponding shaded relief (Figure 5.6) indicate that the integrated DEM (Figure 5.5c and 5.6b) recovers the missing details in the photogrammetric DEM and the reference DEM.

a) The base image (left) and the matching image (right)

b) The x-disparity (left) and y-disparity (right) map. Figure 5.4 The epipolar images used for matching and the disparity maps generated by PAM for the Chang’E-4 landing site. Poor matches are outlined in blue.

110

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

a) Reference DEM (5 m/pixel) b) Photogrammetric DEM from PAM (4 m/pixel)

c) The integrated DEM (1.5 m/pixel) d) The ortho-image (1.5 m/pixel)

Figure 5.5 3D views of the DEMs and the corresponding ortho-image covering the Chang’E-4 landing site. The blue boxes show close up views of the landing site (the red cross).

111

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

a) Photogrammetric DEM b) Integrated DEM c) Reference DEM Figure 5.6 The shaded relief of the DEMs covering the Chang’E-4 landing site. The blue boxes show close up views of the landing site (the red cross).

The topography of the Chang’E-4 landing site derived from different DEMs is also compared. The integrated DEM provides the necessary information about the topography around the landing site, while no useful information can be derived from the other DEMs. Two profiles are derived from the DEMs and presented in Figure 5.7. The first profile reveals two of the crater triplets surrounding the Chang’E-4 landing site. It is apparent that only the integrated DEM is able to retrieve the shape of these small craters. The relatively larger craters in Figure 5.7 show that 112

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface incorporating the shadow geometry allows precise reconstruction of the shadowed terrain where photogrammetric DEM fails to do so. There are small horizontal offsets in the shadowed regions, which likely stem from possible inaccuracies in the shadow detection. The performance of the DEMs generated using the proposed approach is quantitatively analyzed using the root mean square error (RMSE) and the maximum absolute differences in elevations with respect to the reference DEM. The absolute deviation at the 99.5 percentile is also provided to enable a more comprehensive analysis of the performance of the approach. The indicators are summarised in Table 5.1. The photogrammetric DEM has an RMSE of approximately 4.3 m, an absolute maximum deviation of 38.5 m, and an absolute deviation at the 99.5 percentile of 21.7 m. The major contributor to the aforementioned error is that the image pair shadowed a significant amount of the topographic features, especially those corresponding to steeper slopes such as the crater walls. This problem is significantly improved in the integrated DEM, which has a slightly lower RMSE of 3.7 m and a significantly reduced absolute maximum difference of 19.8 m, an approximately 19 m reduction compared to the photogrammetric DEM. The 99.5 percentile also drops to approximately 11.2 m, a 10 m reduction with respect to the photogrammetric DEM. It should be noted that it is impossible to obtain a reference DEM with a better resolution of 1.5 m/pixel for comparison. However, the 5-m/pixel reference DEM can be used to provide an overall evaluation of the geometric accuracy of the DEM generated by the proposed approach.

Table 5.1 Comparative statistics of the DEMs of the Chang’E-4 landing site.

Photogrammetric DEM Integrated DEM

RMSE 4.32m 3.47m

Maximum absolute 38.49m 19.84m deviation

Absolute deviation 21.66m 11.17m at 99.5 percentile

113

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

a) Profile 1

b) Profile 2 Figure 5.7 Profile comparison of the DEMs of the Chang’E-4 landing site. The blue boxes indicate the topographic details revealed by the integrated DEM.

114

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

5.4.2 Experimental Analysis at the Chang’E-5 Candidate Landing Site

The Chang’E-5 mission is planned for launch around late 2019. Figure 5.8 shows a stereo pair of LROC NAC images of areas within the Chang’E-5 candidate landing region (Wu et al., 2018b). There is a dome located in the middle area of the image. The difference in the slew angles between the images is about 15o, which is suitable for photogrammetric processing. However, the illumination directions of the two images differ by about 80o, indicating that conventional matching algorithms will likely fail on this stereo pair. The images are associated with their EO parameters obtained from the SPICE kernels. Similarly, a reference DEM generated from photogrammetry using other NAC images with consistent illumination conditions (image IDs: M1119207667LR and M1119228976LR) is used for comparison. The reference DEM has a resolution of 5 m/pixel. The comparison focuses on the dome area in the middle of the image. Figure 5.9a shows the image subsets after epipolar-rectification. The resolution of the epipolar images is resampled to 4 m/pixel, and the dimension of the image is 1330 x 1800 pixels. After PAM, we obtain the disparity maps shown in Figure 5.9b, which largely correspond to the topography of the region. The high horizontal disparity in the middle of the image corresponds to the dome, while the dark circles above correspond to the craters. Because of the illumination configuration, most of the inner parts of the craters are illuminated in both images, and hence the effects of shadows are much less apparent than in the previous dataset. Subtle stripe patterns can be observed on the y-disparity map. A possible reason for their occurrence is that the proposed photometric stereo method, which is described in Chapter 4, performs best when the horizontal illumination differences are close to 180o, whereas in cases with illumination horizontal differences significantly less than 180o (such as about 80o in this dataset), the performance declines and becomes less sensitive to subtle changes. Nevertheless, the x-disparity is the dominating parameter for 3D reconstruction in this dataset, and most of the topographic features are revealed by the x-disparities. Hence, with this dataset, the y-disparity has a subtle effect and may contain very slight discrepancies.

115

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

Lunar dome region on Lunar dome region on M1119207667R M1145135367L M1119207667R M1145135367L

Image ID M1119207667R M1145135367L

Slew angle (o) 14.8 0.0

Illumination azimuth (o) 154.4 233.0

Incidence angle (o) 45.7 56.0

Emission angle (o) 17.2 1.8

Resolution (m/pixel) 1.5 1.5

Figure 5.8 The LROC NAC image pair within the Chang’E-5 candidate landing region. Blue boxes show the enlarged views of the lunar dome region, and yellow arrows indicate the illumination direction of each image.

Figure 5.10 shows the 3D views of the reference DEM (Figure 5.10a), the photogrammetric DEM (4 m/pixel) from PAM (Figure 5.10b), the DEM (1.5 m/pixel) from the integrated photogrammetric and photoclinometric approach (Figure 5.10c), and the ortho-image based on the photogrammetric DEM and the 116

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface image EO parameters (Figure 5.10d). The conventional photogrammetric routines based on SGM fail to generate any reasonable DEMs for this dataset. In contrast, although the effective resolution is not as high as the reference DEM, photogrammetric processing with PAM successfully generates a plausible DEM. Moreover, the integrated DEM successfully recovers the details better than the other DEMs, as compared with the details shown in the ortho-image, which can also be observed from the shaded relief of the DEMs as shown in Figure 5.11. Two profiles are extracted from the DEMs and presented in Figure 5.12. The first profile (Figure 5.12a), which shows the larger crater near to the dome, demonstrates that the photogrammetric DEM is able to retrieve a precise topography while the integrated DEM is able to reconstruct the fine details. The profile of the dome (Figure 5.12b) shows that the steep edges and artefacts from the photogrammetric DEM are corrected by photoclinometry. The profiles also show small craters and structures that are preserved on the integrated DEM but not on the photogrammetric DEM or the reference DEM. The quantitative indicators of the performance of the proposed approach are summarised in Table 5.2. The RMSE of the photogrammetric DEM is about 5.8 m with respect to the reference DEM, while the absolute maximum is about 54.5 m and the absolute deviation at the 99.5 percentile is about 18.7 m. The pixels with large errors are usually located in the shadows and can be filtered out using operators such as a median filter. The integrated DEM has an RMSE of approximately 6.6 m, with a maximum of 45 m and an absolute deviation at the 99.5 percentile of 24.8 m. In general, the indicators are slightly higher than those of the photogrammetric DEM, except for the absolute maximum. In this case, there are slight gradual deviations in terms of very low-resolution surface gradients on the integrated DEM. This is possibly due to the spatially varying albedo, especially across the lunar dome. This can be improved by tuning the parameters or incorporating photometric stereo reconstruction for a more robust estimation of the surface.

117

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

a) The base image (left) and the matching image (right)

b) The x-disparity (left) and y-disparity (right) maps

Figure 5.9 The epipolar images used for matching and the disparity maps generated by PAM within the Chang’E-5 candidate landing region.

118

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

b) Photogrammetric DEM from PAM a) Reference NAC DEM (5 m/pixel) (4 m/pixel)

c) The integrated DEM (1.5 m/pixel) d) The ortho-image (1.5 m/pixel)

Figure 5.10 3D views of the DEMs and the corresponding ortho-image within the Chang’E-5 candidate landing region.

119

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

a) Photogrammetric b) Integrated DEM c) Reference DEM DEM Figure 5.11 The shaded relief of the DEMs within the Chang’E-5 candidate landing region.

Table 5.2 Comparative statistics of the DEMs within the Chang’E-5 candidate landing region.

Photogrammetric DEM Integrated DEM

RMSE 5.76m 6.55m

Maximum absolute 54.52m 45.13m deviation

Absolute deviation 18.70m 24.80m at 99.5 percentile

120

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

a) Profile 1

b) Profile 2

Figure 5.12 Profile comparison of the DEMs within the Chang’E-5 candidate landing site. The blue boxes indicate the small topographic details revealed by the integrated DEM.

121

Chapter 5 Integration of Photoclinometry and Photogrammetry for Illumination-Invariant 3D Modelling of the Lunar Surface

5.5 Summary

This chapter describes an integrated framework of photogrammetry and photoclinometry. In particular, photogrammetry is used as the primary approach to generate accurate 3D models of the lunar surface, while photoclinometry allows robust photogrammetric performance when processing stereo images with severe illumination differences, which is not uncommon for planetary images (Preusker et al., 2017). The photogrammetric DEM is then refined to optimal resolution, comparable to the image, by the SI-SfS algorithm described in Chapter 3. In the explained framework, PS-SfS plays a vital role in handling illumination-inconsistent stereo images by the developed PAM algorithm. This extends the use of SfS in applications other than 3D reconstruction. The approach was applied to the high- resolution 3D mapping of Chang’E-4 and Chang’E-5 landing sites. The experimental results showed that PAM successfully generated correct pixel-wise matches for images acquired under significant illumination variations (nearly 180o in azimuth for the Chang’E-4 dataset and about 90o in azimuth for Chang’E-5). The RMSE of the generated DEMs ranges from 3 to 7 m compared to the reference DEMs (i.e., DEMs generated from photogrammetry using images with consistent illumination conditions). There are occasions where the absolute error exceeds 40 m, which is mostly due to shadows or overly bright regions. These discrepancies arise because the images of these regions contain no useful information for the approach to work on, and most of the discrepancies can be corrected by photoclinometry in the subsequent refinement step. The experimental results also show that photoclinometric refinement sometimes produces small accumulative errors when the surface gradients are continuously over or under-estimated, which can be improved by adjusting weights and/or introducing different constraints to the approach.

122

Chapter 6 Conclusion and Discussion

Chapter 6 Conclusions and Discussion

This dissertation presents a variety of approaches fusing SfS and state-of-the- art technologies, especially photogrammetry, for accurate and high-resolution 3D modelling of the lunar surface. Each of the presented approaches can be used independently, while they are also integrated into one photogrammetric- photoclinometric framework. The presented algorithms are verified and evaluated by experimental analysis using LROC NAC images as input, and NAC DEMs obtained independently as reference. This chapter summarises a number of concluding remarks of this research, followed by a few recommendations and possible future works.

6.1 Summary of the Research Work

The existing technologies for 3D modelling of the lunar surface, such as photogrammetry, and their subsequent 3D topographic products, lunar DEMs, are limited in spatial resolution and spatial coverage. The limitations are discussed in previous chapters. This study aims to address these limitations by integrating photogrammetry and SfS. In this research, three major contributions are proposed and presented with experimental evaluations using LROC NAC data of the lunar surface. First, the fusion strategy of combining SI-SfS and initial DEMs derived from photogrammetry/laser-altimetry was presented and explained in Chapter 3. In this approach, the initial DEM provides initial conditions and constraints for SfS to construct precise high-resolution 3D models of the lunar surface based on a single co-registered image. The method allows lunar 3D modelling at the optimal spatial resolution while maintaining the geometric accuracy provided by the initial DEM. Because of the photometric nature of SfS, the approach can retrieve the surface albedo of the region according to the pre-defined reflectance model. Moreover, image shadows were explicitly considered. This approach was evaluated using LROC NAC images and SLDEM in and near the Buisson V crater and Reiner Gamma, where albedo variations are significant, and near the edge of Rima Sharp,

123

Chapter 6 Conclusion and Discussion where surface roughness limits the interpretation of albedo variations. Second, a PS-SfS approach, based on two co-registered images acquired under different illumination conditions, was presented and described in Chapter 4. This method constructs a precise high-resolution lunar DEM without the need for any initial DEM as a constraint. The approach is designed based on SfS of two images to account for the illumination conditions of the Moon. This approach is flexible because it does not require an initial DEM or stereo viewing geometry of the images. An error propagation model was developed to evaluate the performance of PS-SfS with respect to the illumination conditions, which provides insight in choosing the best images for 3D modelling. The presented algorithm and its subsequent error propagation model were verified using LROC NAC datasets near the Antoniadi crater and near the Bogslwsky crater, where images with a wider spectrum of illumination conditions are available. Third, the SI-SfS algorithm and PS-SfS were integrated with a photogrammetric pipeline, as described in Chapter 5. PS-SfS concepts are integrated with image matching in a photogrammetric pipeline, namely photoclinometry- assisted image matching (PAM), which was particularly designed for pixel-wise image matching of stereo images with significantly inconsistent illumination conditions. Using the matching results, or disparity maps, from PAM, a photogrammetric DEM can be generated by space intersection. The SI-SfS algorithm is then performed on the basis of the photogrammetric DEM, which generates a more accurate high-resolution lunar DEM. The presented framework allows for 3D modelling of the lunar surface based on photogrammetry, even under severe illumination variations. As a result, the integrated approach extends the applicability and flexibility of both techniques. Moreover, this proves that SfS can be applied to image matching and other applications. Furthermore, the SI-SfS step optimally increases the effective spatial resolution of the 3D models. The integrated approach was evaluated with applications to landing site choice and candidate landing site topographic modelling for China’s lunar mission Chang’E-4 and Chang’E-5. For the Chang’E-4 mission, the only available stereo image pair by the time of landing (3 January 2019) has almost opposite illumination, with one image illuminated from the East and the other from the West. The developed approach successfully generated DEMs of the landing sites using images acquired under significant difference in illumination. The resulting DEMs are rich in topographic details while maintaining 124

Chapter 6 Conclusion and Discussion the accuracy achieved by photogrammetric methods. In summary, this study successfully accomplished the stated objectives and led to the following merits. (1) The newly developed methods allow for the generation of accurate lunar DEMs, with spatial resolutions comparable to the highest spatial resolution of the underlying image, from a single image or multiple images acquired under different illumination. (2) The methods allow for optimal 3D modelling of the lunar surface based on an integrated photogrammetric- photoclinometric approach, which extends the applicability and flexibility of photogrammetry to datasets with inconsistent illumination. (3) These approaches facilitate a more sophisticated technological fusion toward optimal 3D representations of lunar and planetary surfaces. (4) The approaches can be applied to study the lunar surface at a high level of detail, which can improve the understanding of the Moon’s geologic processes, and provide insights into the evolution of planetary surfaces.

6.2 Discussions

For the SI-SfS approach (Chapter 3), the experimental results showed that the resulted pixel-resolution SfS DEMs successfully recovered the topographic details visible in the image. The absolute vertical RMSE of the results, compared to the reference DEMs, is less than 7 m and the absolute maximum error is approximately 10 m. The average RMSE of the reference DEMs is about 2.5 m (Henriksen et al., 2017). The surface albedo maps generated by the approach showed apparent correspondence with high sun elevation angle images, which provide references for albedo patterns. The reconstructed albedo maps tend to be much smoother than the high sun elevation angle images, which is reasonable because of the assumptions and formulations used by the approach. One critical merit of the proposed approach lies in the multi-gridding optimisation architecture, where the SfS result per iteration is propagated to the initial spatial resolution, followed by a constrained SfS at initial spatial resolution and subsequent adjustment propagated back to the current iteration. Such a system allows frequent interactions between multiple resolutions and hence preventing errors accumulating as the spatial resolution increases. The importance of multi-gridding architecture in SfS is also discussed by other researchers (Kirk, 1987;

125

Chapter 6 Conclusion and Discussion

Horn, 1990). The experimental results showed that the quality of the initial DEM has direct effects on the performance of the approach. It is therefore important for the initial DEM to preserve key geometry at its resolution. Because the approach tends to smooth out the boundaries of actual drastic albedo changes, greater ambiguity and hence larger inaccuracies may exist if the resolution difference between the image and the initial DEM is too large. In this case, an initial DEM with better properties such as geometry preservation and effective resolution would be preferred. For the PS-SfS approach (Chapter 4), the experimental results showed a successful recovery of topographic details. The vertical error of the resulted DEMs is generally stable (RMSE within the range of 12 m - 17 m), showing that the overall accuracy is partially independent to the illumination azimuthal difference. However, PS-SfS using images with more apparent illumination differences (e.g., at least 30o in azimuth) tend to produce results with minimal systematic errors. Because the SfS DEM is registered to an initial DEM by using only the mean plane and height, subtle systematic errors may be induced when the terrain is largely non-planar (e.g., the rim of a very large crater where surface slopes change significantly). As a result, subtle errors accumulate and contribute to the final vertical RMSE. This problem can be improved by increasing the influence of the initial DEM (as in SI-SfS) or to include additional images to the PS-SfS approach. Apart from the SfS DEMs, the developed error model demonstrated that the slope error in PS-SfS is affected by multiple factors, such as illumination azimuthal and zenith angles, as well as the general intensity (e.g., mean intensity) of the images. The error model also showed a good correspondence with the actual slope errors obtained from PS-SfS. Slight inconsistencies were attributed to the integrability constraint enforced by the PS-SfS algorithm, which was not considered by the error model. Scalar fitting between the error model estimates and the actual error suggested that the PS-SfS results have a slope error along the first illumination direction of around 0.06-0.07, approximately 3o-4o. The vertical error analysis did not show apparent correspondence to the error model, this is reasonable because the point-to-point vertical error was unable to directly represent slope errors and the process of converting surface slopes into heights includes other constraints such as the enforcement of integrability and/or vertical offsetting, which might alter the error values. This also highlights the importance of integrability enforcement in PS-SfS, which complies with the suggestions by other researchers (Kirk, 1987; Frankot and Chellappa, 1988; Horn, 126

Chapter 6 Conclusion and Discussion

1990). The proposed error model enriches the theory of PS-SfS. It can serve as a component for the quantitative quality control of SfS-based approaches, it can also be combined with the SI-SfS error models proposed by Jankowski and Squyres (1991) to provide a more comprehensive overview of uncertainties in photometric stereo photoclinometry. For the integrated approach (Chapter 5), the experimental results showed that the PAM, an image matching approach developed based on SfS, successfully generates robust pixel-wise matches from images with severely inconsistent illuminations (e.g., 80o - 200o differences in azimuth). PAM is the key to handling images with complex illumination inconsistencies. Intuitively, PAM evaluates image matching by 3D information since PS-SfS is used to recover the surface gradients of the local image patches, which are then analysed for image matching. Because the relationship between the performance of SfS and illumination difference of images is complex (as shown in Chapter 4), specifically designed formulations and experimental analyses will be necessary to rigorously determine the threshold of illumination difference that is favourable for PAM. In general, PAM will be applicable for images with visually apparent illumination differences (e.g., 30o or above in incidence azimuth). The integrated DEMs have RMSE of 3 m - 7 m when compared to the reference DEMs (i.e., DEMs generated from photogrammetry using images with consistent illumination conditions). There are occasions where the absolute error exceeds 40 m, which is mostly due to shadows or overly bright regions, where no image information is available for PAM and the integrated approach. Most of the discrepancies can be corrected by SI-SfS in the subsequent refinement step. The experimental results also show that photoclinometric refinement sometimes produces small accumulative errors when the surface gradients are continuously over or under-estimated, which can be improved by adjusting weights and/or introducing different constraints to the approach. For the experimental analysis, SI-SfS is used because one of the images in the Chang’E-4 stereo pair is severely shadowed and is not suitable for PS-SfS. However, it is possible to use PS-SfS as a final step by using PAM to produce pixel-synchronous images. Whether to use SI-SfS or PS-SfS as the final refinement stage depends on the illumination conditions of the images and the user requirements.

127

Chapter 6 Conclusion and Discussion

6.3 Conclusions

The SI-SfS approach creates pixel-resolution DEMs (0.5-1.5m/pixel) that preserves the details visible in the image while keeping the overall geometry consistent with the initial DEM used as constraints. The approach is also able to estimate plausible but smooth location-dependent albedo map. The performance of SI-SfS depends on the quality of the initial DEM, and especially on whether the key geometry is preserved in the initial DEM for its effective resolution. Larger resolution difference between the image and the initial DEM may lead to increased reconstruction inaccuracies, particularly at regions with significant albedo variations. In this case, an initial DEM with better geometry preservation and/or resolution would be preferred to increase the reliability for detailed shape recovery. The PS-SfS approach can generate pixel-resolution DEMs with optimal topographic details, from a pair of images acquired under significant illumination conditions. The approach requires the statistical information of an initial DEM (e.g., mean slope and mean height) to constrain the overall geometry. By analysing the PS- SfS error model, we found that the slope error in PS-SfS is affected by not only the illumination azimuthal difference but also the illumination zenith angles and the general intensity (e.g., mean intensity) of the images. The experimental analysis showed a good correspondence between the slope errors estimated from the error model and the actual slope errors from PS-SfS. The model is useful in decision making, quality control, and performance improvements when utilising PS-SfS. The vertical error of PS-SfS is stable and did not show apparent patterns corresponding to the proposed error model, which is reasonable and highlighted the importance of integrability enforcement in SfS. Image pair with larger illumination difference (e.g., at least 30o). The integrated approach fuses photoclinometry and photometric stereo techniques in the photogrammetric processing in a synergistic manner. The approach outperforms the conventional approaches and allows generating accurate pixel- resolution DEM from photogrammetric stereo images with severe illumination differences (e.g., about 80o and 200o azimuthal difference). The PAM algorithm showed that PS-SfS can be used for robust pixel-wise illumination-invariant image matching, which is of high importance for accurate photogrammetric processing. In

128

Chapter 6 Conclusion and Discussion general, PAM will be applicable for images with visually apparent illumination differences (e.g., 30o or above in incidence azimuth). Shadows still pose a challenge in PAM and other image matching approaches but can be utilised by SI-SfS in the refinement step. The integrated approach has the advantage of being robust to illumination inconsistencies and subtle textures and can produce DEMs with an optimal spatial resolution of the lunar surface. This research shows that the integration of SfS and existing state-of-the-art 3D modelling technology (e.g., photogrammetry) can produce 3D models of the lunar surface at pixel-resolution and wide spatial coverage. The developed approaches are of significance in mapping the lunar surface at very high detail and wide coverage, which allows in-depth study on the Moon’s surface evolution, and greatly supports future exploration missions.

6.4 Considerations for Future Works

Based on the experimental evaluations and current state of the research, the following aspects are summarised for the future development of this research.

(1) Automatic determination of effective spatial resolution and self-adaptive hierarchical architecture

Currently, the frequency components of the initial DEM to be imposed as constraints to SfS are controlled by the Gaussian operator with a pre-defined standard deviation σ. A suboptimal choice of σ may lead to over- or under- constraining power, thus affecting the quality of the resulting DEM. Intuitively, the choice of σ is affected by the effective spatial resolution of the initial DEM, which is usually lower than the spatial resolution defined by the map size of the grid cells. As a result, an automatic approach to determine the effective spatial resolution of the initial DEM, and hence the subsequent σ, could help build a more comprehensive 3D reconstruction system based on SfS. This would allow the SfS system to adaptively construct the reconstruction hierarchy based on the determined σ. The determination of σ can be realised by iteratively resampling the image at a sequence

129

Chapter 6 Conclusion and Discussion of spatial resolutions and by comparing the images with the DEM until the best correspondence is achieved.

(2) Confidence intervals and uncertainty principles as constraints

Global/near-global DEMs are often associated with a vertical accuracy that applies to the whole DEM, implying constant confidence for the whole dataset. However, this obscures local confidence variations due to factors such as the abundance and accuracy of sample points. For example, the confidence of an LDEM grid is higher when its vicinity contains a LOLA survey point. For regional DEMs created by photogrammetry, such as NAC DEMs, the confidence of each DEM grid differs according to factors such as the point density and quality of space intersection, and lower confidence should also be given to grids in which the corresponding images are shadowed. The approaches presented in this dissertation do not explicitly consider the confidence level of the initial DEM. Intuitively, it is partially involved when controlling the weights, which is a manual interaction. Considering confidence levels can determine a location-dependent solution range, which helps the SfS system refine the optimal solution and effectively separate the surface albedo and topographic shading. Because the constraints imposed by the initial DEM are in the form of surface gradients, the confidence level of each surface patch must be determined by uncertainty propagation from each DEM grid.

(3) Reflectance models

The Lunar-Lambert reflectance model (McEwen, 1991) is used throughout this dissertation, However, because the Lunar-Lambert model approximates the Hapke model in a global sense, local fitness variations may occur and lead to suboptimal results (Grumpe et al., 2014). The actual effects of changing a reflectance model to the performance of the SfS 3D reconstruction system are yet to be evaluated. This could be achieved by comparing the matching and reconstruction results presented in previous chapters to the results that use the same datasets and approaches presented in this dissertation but switching to other reflectance models such as the Hapke model. It is also important to evaluate the brightness variations of the model with respect to parameters such as phase angles and topography, which 130

Chapter 6 Conclusion and Discussion can be useful for choosing the appropriate model for the given data. Reflectance models such as the Hapke model have explicit parameters to describe the macroscopic roughness of the surface, which is combined into surface albedo in the Lunar-Lambert model (McEwen 1986; 1991), as shown in the case of the Rima Sharp dataset in Section 3.4.4. As a result, with the use of an appropriate model, it is possible to infer the macroscopic roughness of the surface during the SfS reconstruction process and increase the scientific value of such algorithms. Nevertheless, the algorithms presented in this dissertation do not require a specific reflectance model, as the emphasis is placed on how initial DEMs contribute to accurate and high-resolution 3D lunar modelling based on SfS and how SfS can be integrated with photogrammetry.

(4) Developing packaged deliverables

The presented algorithms are implemented in MATLAB to facilitate experimental analysis. Currently, the prototypes are implemented separately and require expertise to control the parameters. In the future, packaging the algorithms presented in this dissertation into a user-friendly deliverable would be a solid contribution to the planetary science and mapping community. Although some research groups have their own pipeline for 3D modelling of planetary surface based on SfS (Gaskell, 2008; Grumpe, 2014), publicly available pipelines are lacking. Commonly known SfS pipelines for planetary remote sensing include the photoclinometry tools integrated into the ISIS system (Kirk, 2003) and the SfS tools integrated into NASA’s ASP system (Alexandrov et al., 2018). The first does not account for PS-SfS, the second requires accurate EO parameters, and neither supports the illumination-invariant image matching that facilitates automatic PS-SfS. The packaged SfS deliverable is expected to contain the following features: (1) automatic determination of effective spatial resolution; (2) automatic co-registration of the image and the DEM; (3) automatic co-registration of illumination-inconsistent images (PAM, Chapter 5); (4) Single monocular imagery SfS with an initial DEM as a constraint (Chapter 3); (5) Photometric stereo SfS with automatic image co- registration (Chapter 4); and (6) an integrated photogrammetric-photoclinometric framework (Chapter 5), on the basis of existing planetary mapping pipelines such as Planetary3D and ASP. 131

Chapter 6 Conclusion and Discussion

(5) Applications to other planetary bodies and Earth applications

SfS has also been applied to other extraterrestrial bodies such as Mars (Wohlfarth et al., 2018; Hess et al., 2019) and asteroids (Gaskell, 2008). Extending the research work of this dissertation to other bodies allows for a more comprehensive and robust 3D reconstruction system on the basis of SfS and other state-of-the-art modelling technologies. For example, the image-matching algorithm developed here (PAM) can be useful for mapping planetary bodies with significant illumination variations such as Mercury (Fassett, 2016; Preusker et al., 2017; 2018). Such extensions require extra considerations and new innovations to handle various factors. For example, planets such as Mars contain an atmosphere, which would need to be addressed when evaluating the photometry and subsequent SfS reconstruction of the Martian surface (Gehrke, 2008; Wohlfarth et al., 2018; Hess et al., 2019). The photometric properties and models of different planetary bodies have also been studied, such as Mars (Soderblom et al., 2006; Fernando et al., 2015), Europa (Belgacem et al., 2019), and asteroids (Longobardo et al., 2016; 2019). These models facilitate high-precision and high-resolution topographic modelling of corresponding planetary bodies using SfS. Planetary bodies such as Mars have dynamic surfaces (Fanara, et al., 2019), which must be considered when applying PS-SfS. For the Earth, applying SfS in terms of regional-scale 3D modelling is a challenge due to the complexity of surface properties and reflecting behaviour. More emphasis is hence placed on indoor and close-range scenes for which the fitness of a reflectance model is not considered. As a rigorous mapping technique, SfS can be first adapted for surface modelling of ice-sheets and glaciers in icy regions such as Antarctica, sandy regions such as deserts, and other areas where the reflecting behaviour is relatively homogeneous. Because these regions have almost no texture and are dynamic (i.e., the surface changes from time to time), the performance of photogrammetry, which requires image matching and static objects between multiple image acquisitions, is therefore limited. As a result, applying SfS for 3D modelling of these regions can be of higher significance in facilitating long-term monitoring and related scientific studies. Moreover, complex urban applications can also be developed, such as 3D modelling of roadside slopes, building walls, and road surfaces, and their subsequent inspections and safety analyses. All of these applications require additional 132

Chapter 6 Conclusion and Discussion innovation for this research to solve real-world problems in the future. References

Ackermann, J., Langguth, F., Fuhrmann, S., and Goesele, M., 2012. Photometric stereo for outdoor webcams. 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR): 262-269. Acton, C., 1996. Ancillary data services of NASA’s navigation and ancillary information facility. Planetary and Space Science, 44(1): 65-70. Alexandrov, O. and Beyer, R. A., 2018. Multiview Shape-From-Shading for Planetary Images. Earth and Space Science, 5: 652-666. Agrawal, A., Raskar, R., and Chellappa, R., 2006. What is the range of surface reconstructions from a gradient field?. Computer Vision - ECCV 2006: 578-591. Bandfield, J. L., Ghent, R. R., Vasavada, A. R., Paige, D. A., Lawence, S. J., and Robinson, M. S., 2011. Lunar surface rock abundance and regolith fines temperatures derived from LRO Diviner Radiometer data. Journal of Geophysical Research: Planets, 116(E12). Banz, C., Pirsch, P., and Blune, H., 2012. Evaluation of penalty functions for semi- global matching cost aggregation. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XXXIX-B3: 1-6. Barker, M. K., Mazarico, E., Neumann, G. A., Zuber, M. T., Haruyama, J., and Smith, D. E., 2015. A new lunar digital elevation model from the Lunar Orbiter Laser Altimeter and SELENE Terrain Camera. Icarus, 73: 346-355. Barron, J. T., and Malik, J., 2011. High-frequency shape and albedo from shading using natural image statistics. 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR): 2521-2528. Barron, J. T., and Malik, J., 2012. Shape, albedo, and illumination from a single image of an unknown object. IEEE Conference on Computer Vision and Pattern Recognition (CVPR): 334-341. Belgacem, I., Schmidt, F., and Jonniaux, G., 2019. Regional study of Europa’s photometry. Icarus, 338, doi: https://doi.org/10.1016/j.icarus.2019.113525. Beyer, R. A., McEwen, A. S., and Kirk, R. L., 2003. Meter-scale slopes of candidate MER landing sites from point photoclinometry. Journal of Geophysical Research: Planets, 108(E12).

133

References

Beyer, R. A., and Kirk, R. L., 2012. Meter-scale slopes of candidate MSL landing sites from point photoclinometry. Space Science Reviews, 170(1-4): 775-791. Beyer, R. A., 2017. Meter-scale slopes of candidate InSight landing sites from point photoclinometry. Space Science Reviews, 211(1-4): 97-107. Bickel, V. T., Aaron, J., Manconi, A., Loew, S., and Mall, U., 2020. Impacts drive lunar rockfalls over billions of years. Nature Communications, 11, 2862(2020). doi: 10.1038/s41467-020-16653-3. Birchfield, S., and Tomasi, C., 1998. A pixel dissimilarity measure that is insensitive to image sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence. IEEE, 20(4): 401-406. Burns, K. N., Speyerer, E. J., Robinson, M. S., Tran, T., Rosiek, M. R., Archinal, B. A., and Howington-Kraus, E., 2012. Digital elevation models and derived products from LROC NAC stereo observations. International Archives of the Photogrammetry, Remote Sensing, Spatial Information Sciences, XXXIX-B4: 483-488. Ciarniello, M., Capaccioni, F., and Filacchione, G., 2014. A test of Hapke’s model by means of Monte Carlo ray-tracing. Icarus, 237(2014): 293-305. Chandrasekhar, S., 1960. Radiative Transfer. Dover, New York. Chandraker, M., Agarwal, S., and Kriegman, D., 2007. ShadowCuts: photometric stereo with shadows. 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR): 1-8. Dalal, N., and Triggs, B., 2005. Histograms of oriented gradients for human detection. 2005 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1: 886-893. Danzl, R., and Scherer, S., 2002. Integrating shape from shading and shape from stereo for variable reflectance surface reconstruction from SEM images. Proceedings of 26th Workshop of the Austrian Association for Pattern Recognition: 281-288. Denevi, B.W., Robinson, M. S., Boyd, A. K., Blewett, D. T., and Klima, R. L., 2016. The distribution and extent of lunar swirls. Icarus, 273(2016): 53-67. De Rosa, D., Bussey, B., Cahill, J. T., Lutz, T., Crawford, I. A., Hackwill, T., van Gasselt, S., Neukum, G., Witte, L., McGovern, A., Grindrod, P. M., and Carpenter, J. D., 2012. Characterisation of potential landing sites for the European Space Agency's Lunar Lander project. Planetary and Space Science, 134

References

74(1): 224-246. Di, K., Li, W., Yue, Z., Sun, Y., and Liu, Y., 2014. A machine learning approach to crater detection from topographic data. Advances in Space Research, 54(2014): 2419-2429. Fanara, L., Gwinner, K., Hauber, E., and Oberst, J., Automated detection of block falls in the north polar region of Mars, Planetary and Space Science (2019), doi: https://doi.org/10.1016/j.pss.2019.104733. Fassett, C. I., 2016. Ames stereo pipeline-derived digital terrain models of Mercury from MESSENGER stereo imaging. Planetary and Space Science, 134: 19-28. Fernando, J., Schmidt, F., Pilorget, C., Pinet, P., Ceamanos, X., Douté, S., Daydou, Y., and Costard, F., 2015. Characterization and mapping of surface physical properties of Mars from CRISM multi-angular data: Application to Gusev Crater and Meridiani Planum. Icarus, 253(2015): 271-295. Foroosh, H., Zerubia, J. B., and Berthod, M., 2002. Extension of phase correlation to subpixel registration. IEEE Transactions on Image Processing, 11(3): 188-200. Gaskell, R. W., Barnouin-Jha, O. S., Scheeres, D. J., Konopliv, A. S., Mukai, T., Abe, S., Saito, J., Ishiguro, M., Kubota, T., Hashimoto, T., Kawaguchi, J., Yoshikawa, M., Shirakawa, K., Kominato, T., Hirata, N., and Demura, H., 2008. Characterizing and navigating small bodies with imaging data. Meteoritics and Planetary Science, 43(6): 1049-1061. Gehrke, S., Haase, I., 2007. HRSC data processing by matching in object space: joint derivation of DTM and orthoimage for an impact crater on Mars. 38th Lunar and Planetary Science Conference, March 12-16, 2007, League City, Texas, USA. Gehrke, S., 2008. Geometric and Radiometric Modeling of the Martian Surface Based on Object Space Matching and Photoclinometry. International Archives of the Photogrammetry, Remote Sensing, Spatial Information Sciences, XXXVII- B4: 1031-1036. Gläser, P., Scholten, F., De Rosa, D., Marco Figuera, R., Oberst, J., Mazarico, E., Neumann, G. A., and Robinson, M. S., 2014. Illumination conditions at the lunar south pole using high resolution Digital Terrain Models from LOLA. Icarus, 243 (2014): 78-90. Glaze, L. S., 1999. Volcanic eruption plume top topography and heights as determined from photoclinometric analysis of satellite data. Journal of Geophysical Research, 104(B2): 2989-3001. 135

References

Greenberg, R., Morris, M., and Foley, M. A., 2011. The effect of sub-pixel slope variations on photoclinometry. Icarus, 214(1): 348-350. Grumpe, A., and Wöhler, C., 2011. DEM construction and calibration of hyperspectral image data using pairs of radiance images. 2011 7th International Symposium on Image and Signal Processing and Analysis (ISPA): 609-614. Grumpe, A., Belkhir, F., and Wöhler, C., 2014. Construction of lunar DEMs based on reflectance modelling. Advances in Space Research, 53: 1735-1767. Grumpe, A., Wöhler, C., Liu, W. C., and Wu, B., 2015. Construction of very high resolution DTMs based on NAC images using stereo analysis and shape from shading: A first glance. 46th Lunar and Planetary Science Conference, March 16- 20, 2015, The Woodlands, Texas, USA. Haase, I., Gläser, P., Oberst, J., 2019. Bundle adjustment of spaceborne double- camera push-broom imagers and its application to LROC NAC imagery. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-2/W13: 1397-1404. Hapke, B., 1981. Bidirectional reflectance spectroscopy: 1. Theory. Journal of Geophysical Research, 86(B4): 3039-3054. Hapke, B., 1986. Bidirectional reflectance spectroscopy: 4. The extinction coefficient and the opposition effect. Icarus, 67(2): 264-280. Hapke, B., 2002. Bidirectional reflectance spectroscopy: 5. The coherent backscatter opposition effect and anisotropic scattering. Icarus, 157(2): 523-534. Hapke, B., 2012. Theory of Reflectance and Emittance Spectroscopy. Cambridge, UK: Cambridge University Press. Hapke B., 2013. Comment on "A critical assessment of the Hapke photometric model" by Y. Shkuratov et al. Journal of Quantitative Spectroscopy and Radiative Transfer, 116: 184-190. Harris, C. G., and Stephens, M., 1988. A Combined Corner and Edge Detector. Alvey Vision Conference. doi: 10.5244/C.2.23. Heipke, C., 1992. Integration of digital image matching and multiple image shape from shading. International Archives of the Photogrammetry and Remote Sensing, XXIX-B3: 832-841. Heipke, C., Oberst, J., Albertz, J., and Attwenger, M. 2007. Evaluating planetary digital terrain models: The HRSC DTM test. Planetary and Space Science, 55(14): 2173-2191. 136

References

Henriksen, M. R., Manheim, M. R., Burns, K. N., Seymour, P., Speyerer, E. J., Deran, A., Boyd, A. K., Howington-Kraus, E., Rosiek, M. R., Archinal, A. A., and Robinson, M. S., 2017. Extracting accurate and precise topography from LROC narrow angle camera stereo observations. Icarus, 283(2017): 122-137. Hess, M., Wohlfarth, K., Grumpe, A., Wöhler, C., Ruesch, O., and Wu, B., 2019. Atmospherically compensated shape from shading on the Martian surface. International Archives of the Photogrammetry, Remote Sensing, Spatial Information Sciences, XLII-2: 1405-1411. Hirschmüller, H., 2008. Stereo processing by semi-global matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2): 328-341. Horn, B.P., 1977. Understanding image intensities. Artificial Intelligence, 8(2): 201- 231. Horn, B.K.P., 1990. Height and gradient from shading. International Journal of Computer Vision, 5(1): 37-75. Hu, H., and Wu, B., 2018. Block adjustment and coupled epipolar rectification of LROC NAC images for precision lunar topographic mapping. Planetary and Space Science, 160(1): 26-38. Jung, J., Lee, J.-Y., and Kweon, I. S., 2015. One-day outdoor photometric stereo via skylight estimation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR): 4521-4529. Kirk, R. L., 1987. III. A Fast Finite-element Algorithm for Two-dimensional Photoclinometry. Ph.D. Thesis, Caltech: 165-258. Kirk, R. L., Barrett, J. M., and Soderblom, L. A., 2003. Photoclinometry made simple...?. ISPRS-ET Working Group IV/9, “Advances in Planetary Mapping” Workshop, Houston, TX. Konecny, G., 1985. The International Society for Photogrammetry and Remote Sensing: 75 years old, or 75 years young?. Photogrammetric Engineering and Remote Sensing, 51(7): 919-933. Labarre, S., Ferrari, C., and Jacquemoud, S., 2017. Surface roughness retrieval by inversion of the Hapke model: A multiscale approach. Icarus, 290: 63-80. Lambert, J.H., 1760. Photometria Sive de Mensura et Gradibus Luminis, Colorum et Umbrae. Detleffsen, Augsburg. Lee, C.-H. and Rosenfeld, A., 1983. Albedo estimation for scene segmentation. 137

References

Pattern Recognition Letters, 1(3): 155-160. Lewis, J. P., 1995. Fast Template Matching. Vision Interface: 120-123. Li, Y and Wu, B., 2018. Analysis of rock abundance on lunar surface from orbital and descent images using automatic rock detection. Journal of Geophysical Research-Planets, 123(5): 1061-1088. Lindeberg, T., 1998. Feature detection with automatic scale selection. International Journal of Computer Vision, 30(2): 77-116. Liu, W. C., and Wu, B., 2017. Photometric Stereo Shape-And-Albedo-From-Shading for Pixel-Level Resolution Lunar Surface Reconstruction, International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII- 3/W1: 91-97. Liu, W. C., Wu, B., and Wöhler, C., 2018. Effects of Illumination Differences on Photometric Stereo Shape-and-Albedo-from-Shading for Precision Lunar Surface Reconstruction. ISPRS Journal of Photogrammetry and Remote Sensing, 136: 58-72. Liu, W. C., and Wu, B., 2018. An Integrated Photogrammetric and Photoclinometric Approach for Pixel-Resolution 3D Modelling of Lunar Surface, International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII-3: 1117-1121. Liu, W. C., and Wu, B., 2019. An Integrated Photogrammetric and Photoclinometric Approach for Illumination-Invariant Pixel-Resolution 3D Mapping of the Lunar Surface. ISPRS Journal of Photogrammetry and Remote Sensing, 159: 153-168. Lohse, V., and Heipke, C., 2004. Multi-image shape-from-shading: Derivation of planetary digital terrain models using Clementine images. The International Archives of the Photogrammetry and Remote Sensing, XXXV-B4: 828-833. Lohse, V., Heipke, C., and Kirk, R. L., 2006. Derivation of planetary topography using multi-image shape-from-shading. Planetary and Space Science, 54(7): 661- 674. Longobardo, A., Palomba, E., Ciarniello, M., Tosi, F., De Sanctis, M. C., Zambon, F., Ammannito, E., Filacchione, G., and Raymond, C. A., 2016. Disk-resolved photometry of Vesta and Lutetia and comparison with other asteroids. Icarus, 267: 204-216. Longobardo, A., Palomba, E., Galiano, A., De Sanctis, M. C., Ciarniello, M., Raponi, A., Tosi, F., Schröder, S. E., Carrozzo, F. G., Ammannito, E., Zambon, F., 138

References

Stephan, K., Capria, M. T., Rognini, E., Raymond, C. A., and Russell, C. T., 2019. Photometry of Ceres and Occator faculae as inferred from VIR/Dawn data. Icarus, 320: 97-109. Lowe, D., 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2): 91-110. Mazarico, E., Neumann, G. A., Smith, D. E., Zuber, M. T., and Torrence, M. H., 2011. Illumination conditions of the lunar polar regions using LOLA topography. Icarus, 211(2): 1066-1081. Mazarico, E., Neumann, G. A., Barker, M. K., Goossens, S., Smith, D. E., and Zuber, M. T., 2018. Orbit determination of the Lunar Reconnaissance Orbiter: Status after seven years. Planetary and Space Science, 162 (2018): 2-19. McEwen, A. S., 1986. Exogenic and endogenic albedo and color patterns on Europa. Journal of Geophysical Research, 91(B8): 8077-8097. McEwen, A. S., 1991. Photometric functions for photoclinometry and other applications. Icarus, 92(2): 298-311. Mikolajczyk, K., and Schmid, C., 2002. An affine invariant interest point detector. Computer Vision - ECCV 2002: 128-142. Minnaert, M., 1941. The reciprocity principle in lunar photometry. Astrophysical Journal, 93: 403-410. Neukum, G., Ivanov, B. A., and Hartmann, W. K., 2001. Cratering records in the inner solar system in relation to the lunar reference system, Chronology and evolution of Mars. Space Science Reviews, 96(1-4): 55-86. O’Hara, R., and Barnes, D., 2012. A new shape from shading technique with application to Mars Express HRSC images. ISPRS Journal of Photogrammetry and Remote Sensing, 67: 27-34. Oren, M., and Nayar, S. K., 1994. Generalization of Lambert’s reflectance model. 21st Conference on Computer Graphics and Interactive Techniques, Orlando, FL, USA: 239-246. Peng, M., Wang, Y., Yue, Z., and Di, K., 2019. Automated Detection of Lunar Ridges Based on DEM Data. International Archives of the Photogrammetry, Remote Sensing, Spatial Information Sciences, XLII-2/W13: 1431-1435. Piechullek, C., and Heipke, C., 1996. DTM refinement using multi image shape from shading. International Archives of the Photogrammetry and Remote Sensing. XXXI-B3: 644-651. 139

References

Piechullek, C., Heipke, C., and Ebner, H., 1998. Multi image shape from shading: results using real aerial imagery. International Archives of the Photogrammetry and Remote Sensing. XXXII-3/W1: 160-167. Preusker, F., Stark, A., Oberst, J., Matz, K.-D., Gwinner, K., Roatsch, T., and Watters, T. R., 2017. Toward high-resolution global topography of Mercury from MESSENGER orbital stereo imaging: A prototype model for the H6 (Kuiper) quadrangle. Planetary and Space Science, 142: 26-37. Preusker, F., Oberst, J., Stark, A., and Burmeister, S., 2018. High-resolution topography of Mercury from Messenger Orbital Stereo Imaging: The Southern Hemisphere quadrangles. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-3: 1389-1394. Robinson, M. S., Brylow, S. M., Tschimmel, M., Humm, D., Lawrence, S. J., Thomas, P. C., Denevi, B. W., Bowman-Cisneros, E., Zerr, J., Ravine, M. A., Caplinger, M. A., Ghaemi, F. T., Schaffner, J. A., Malin, M. C., Mahanti, P., Bartels, A., Anderson, J., Tran, T. N., Eliason, E. M., McEwen, A. S., Turtle, E., Jolliff, B. L., and Hiesinger, H., 2010. Lunar reconnaissance orbiter camera (LROC) instrument overview. Space and Science Reviews. 150 (1-4): 81-124. Salamunicćar, G., Lončarić, S., Grumpe, A., and Wöhler, C., 2014. Hybrid method for crater detection based on topography reconstruction from optical images and the new LU78287GT catalogue of lunar impact craters. Advances in Space Research, 53(12): 1783-1797. Scholten, F., Oberst, L., Matz, K. -D., Roatsch, T., Wählisch, M., Speyerer, E. J., and Robinson, M. S., 2012. GLD100: The near-global lunar 100 m raster DTM from LROC WAC stereo image data. Journal of Geophysical Research: Planets, 117(E12). Shkuratov, Y., Starukhina, L., Hoffmann, H., and Arnold, G., 1999. A Model of Spectral Albedo of Particulate Surfaces: Implications for Optical Properties of the Moon. Icarus, 137: 235-246. Shkuratov, Y., Kaydash, V., Korokhin, V., Velikodsky, Y., Opanasenko, N., and Videen, G., 2011. Optical measurements of the Moon as a tool to study its surface. Planetary and Space Science, 59: 1326-1371. Shkuratov, Y., Kaydash, V., Korokhin, V., Velikodsky, Y., Petrov, D., Zubko, E., Stankevich, D., and Videen, G., 2012. A critical assessment of the Hapke photometric model. Journal of Quantitative Spectroscopy and Radiative Transfer, 140

References

113: 2431-2456. Shkuratov, Y., Kaydash, V., Korokhin, V., Velikodsky, Y., Petrov, D., Zubko, E., Stankevich, D., and Videen, G., 2013. Response to the comment by B. Hapke on "A critical assessment of the Hapke photometric model". Journal of Quantitative Spectroscopy and Radiative Transfer, 116: 191-195. Smith, D. E., Zuber, M. T., Jackson, G. B., Cavanaugh, J. F., Neumann, G. A., Riris, H., and Sun, X., 2010. The Lunar Orbiter Laser Altimeter Investigation on the Lunar Reconnaissance Orbiter Mission. Space Science Reviews, 150(1-4): 209- 241. Smith, D. E., Zuber, M, T., Neumann, G. A., Mazarico, E., Lemoine, F. G., Head, J. W., Lucey, P. G., Aharonson, O., Robinson, M. S., Sun, X., Torrence, M. H., Barker, M. K., Oberst, J., Duxbury, T. C., Mao, D., Barnouin, O. S., Jha, Kopal, Rowlands, D. D., Goossens, S., Baker, D., Bauer, S., Gläser, P., Lemelin, M., Rosenburg, M., Sori, M. M., Whitten, J., and Mcclanahan, T., 2017. Summary of the results from the lunar orbiter laser altimeter after seven years in lunar orbit. Icarus, 283(2017): 70-91. Soderblom, J. M., Bell III, J. F., Hubbard, M. Y. H., and Wolff, M. J., 2006. Martian phase function: Modeling the visible to near-infrared surfacephotometric function using HST-WFPC2 data. Icarus, 184(2006): 401-423. Speyerer, E. J., Lawrence, S. J., Stopar, J. D., Gläser, P., Robinsona, M. S., and Jolliff, B. L., 2016. Optimized traverse planning for future polar prospectors based on lunar topography. Icarus, 273(2016): 337-345. Speyerer, E. J., Wagner, R. V., Robinson, M. S., Licht A., Thomas P. C., Becker, K., Anderson, J., Brylow, S. M., Humm, D. C., and Tschimmel, M., 2016a. Pre- flight and on-orbit geometric calibration of the Lunar Reconnaissance Orbiter Camera. Space Science Reviews, 200(1-4): 357-392. Tsai, P.-S., and Shah, M., 1998. Shape from shading with variable albedo. Optical Engineering, 37(4): 1212-1220. Tran, T., Rosiek, M. R., Beyer, R. A., Mattson, S., Howington-Kraus, E., Robinson, M. S., Archinal, B. A., Edmundson, K., Harbour, D., Anderson, E., and the LROC Science Team, 2010. Generating digital terrain models using LROC NAC images. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XXXVIII-4. Wan, X., Liu, J., and Yan, H., 2014. Phase Correlation Based Local Illumination- 141

References

Invariant Method for Multi-Tempro Remote Sensing Image Matching. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XL-3: 365-372. Wan, X., Liu, J. G., and Yan, H., 2015. The Illumination Robustness of Phase Correlation for Image Alignment. IEEE Transactions on Geoscience and Remote Sensing, 53(10): 5746-5759. Wan, X., Liu, J., Qin, M., and Li, S. Y., 2018. Illumination Invariant Change Detection (IICD): From Earth to Mars. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-3: 1673-1680. Wang, T. C., Chandraker, M., Efros, A. A., and Ramamoorthi, R., 2016. SVBRDF- invariant shape and reflectance estimation from light-field cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA. Warell, J., 2004. Properties of the Hermean regolith: IV. Photometric parameters of Mercury and the Moon contrasted with Hapke modelling. Icarus, 167(2): 271- 286. Wöhler, C., 2004. Shape from shading under coplanar light sources. Pattern Recognition. Lecture Notes in Computer Science, 3175: 278-285. Wöhler, C., Lena, R., Lazzarotti, P., Phillips, J., Wirths, M., and Pujic, Z., 2006. A combined spectrophotometric and morphometric study of the lunar mare dome fields near Cauchy, Arago, Hortensius, and Milichius. Icarus, 183(2): 237-264. Wöhler, C., Grumpe, A., Berezhnoy, A., Bhatt, M. U., and Mall, U., 2014. Integrated topographic, photometric and spectral analysis of the lunar surface: Application to impact melt flows and ponds. Icarus, 235: 86-122. Wöhler, C., Grumpe, A., Rommel, D., Bhatt, M., and Mall, U., 2017. Elemental and topographic mapping of lava flow structures in Mare Serenitatis on the Moon. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLII-3/W1: 163-170. Wohlfarth, K. S., Liu, W. C., Wu, B., Grumpe, A., and Wöhler, C., 2018. High Resolution Digital Terrain Models of the Martian Surface: Compensation of the Atmosphere on CTX imagery, 49th Lunar and Planetary Science Conference, March 19-23, The Woodlands, Texas, USA. Woodham, R., 1980. Photometric method for determining surface orientation from 142

References

multiple images. Optical Engineering, 19(1): 139-144. Wu, B., Zhang, Y., and Zhu, Q., 2011. A triangulation-based hierarchical image matching method for wide-baseline images. Photogrammetric Engineering and Remote Sensing, 77(7): 695-708. Wu, B., Zhang, Y., and Zhu, Q., 2012. Integrated point and edge matching on poor textural images constrained by self-adaptive triangulations. ISPRS Journal of Photogrammetry and Remote Sensing, 68: 40-55. Wu, B., Hu, H., and Guo, J., 2014. Integration of Chang’E-2 imagery and LRO laser altimeter data with a combined block adjustment for precision lunar topographic modeling. Earth and Planetary Science Letters, 391(2014): 1-15. Wu, B., Liu, W. C., Grumpe, A., and Wöhler, C., 2016. Shape and Albedo from Shading (SAfS) for Pixel-Level DEM Generation from Monocular Images Constrained by Low-Resolution DEM. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLI-B4: 521-527. Wu, B., and Liu, W. C., 2017. Calibration of Boresight Offset of LROC NAC Imagery for Precision Lunar Topographic Mapping. ISPRS Journal of Photogrammetry and Remote Sensing, 128: 372-387. Wu, B., Liu, W. C., Grumpe, A., and Wöhler, C., 2017. Construction of Pixel-Level Resolution DEMs from Monocular Images by Shape and Albedo from Shading Constrained with Low-Resolution DEM. ISPRS Journal of Photogrammetry and Remote Sensing, 140: 3-19. Wu, B., 2017. Photogrammetry: 3D from imagery. In: D. Richardson, N. Castree, M. F. Goodchild, A. Kobayashi, W. Liu, and R. A. Marston (Eds.): The International Encyclopedia of Geography. New York, NY: John Wiley & Sons, Ltd.: 1-13. Wu, B., Zeng, H., and Hu, H., 2018a. Illumination invariant feature point matching for high-resolution planetary remote sensing images. Planetary and Space Science, 152: 45-54. Wu, B., Huang, J., Li, Y., Wang, Y., and Peng, J., 2018b. Rock abundance and crater density in the candidate Chang’E-5 landing region on the Moon. Journal of Geophysical Research: Planets, 123(12): 3256-3272. Wu, B., Li, F., Hu, H., Zhao, Y., Wang, Y., Xiao, P., Li, Y., Liu, W. C., Chen, L., Ge, X., Yang, M., Xu, Y., Ye, Q., Wu, X., and Zhang, H., 2020. Topographic and geomorphological analysis of the Chang’E-4 landing site on the far side of the 143

References

Moon. Photogrammetric Engineering & Remote Sensing, 86: 247-258, doi: 10.14358/PERS.86.4.247. Yue, Z., Di, K., Liu, Z., Michael, G., Jia, M., Xin, X., Liu, B., Peng, M., and Liu, J., 2019. Lunar regolith thickness deduced from concentric craters in the CE-5 landing area. Icarus, 329(2019): 46-54. Zhang, H., and Voss, K., 2011. On Hapke photometric model predictions on reflectance of closely packed particulate surfaces. Icarus, 215(1): 27-33. Zhao, F., Huang, Q., and Gao, W., 2006. Image Matching by Normalized Cross- Correlation. 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings: II-II.

144