Chromatic Aberration Correction and Spectral Reconstruction from Colour Images
Total Page：16
File Type：pdf, Size：1020Kb
Chromatic Aberration Correction and Spectral Reconstruction from Colour Images by Bernard Llanos A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science Department of Computing Science University of Alberta c Bernard Llanos, 2019 Abstract We present an algorithm for simultaneously demosaicing digital images, and correcting chromatic aberration, that operates in a latent space of spectral bands. Light refraction by a camera lens system depends on the wavelength of the light, causing relative shifting, and blurring, between intensity patterns in different wavelengths on the image sensor. The effect on the image is called chromatic aberration, and appears as colour fringes around edges in the image, and blur. Chromatic aberration depends not only on the camera's optical system, but also on the spectral characteristics of the light entering the camera. Pre- vious works on calibrating chromatic aberration produce models of chromatic aberration that assume fixed discrepancies between image channels, an as- sumption that is only valid when the image channels capture narrow regions of the electromagnetic spectrum. When the camera has wideband channels, as is the case for conventional trichromatic (RGB) cameras, the aberration observed both within and between channels can only be accurately predicted given the spectral irradiance of the theoretical, aberration-free image. We develop a physically-correct chromatic aberration calibration procedure for RGB cameras. Using bandpass-filtered light, we calibrate a model of chro- matic aberration as an image distortion that is parameterized by both image position, and light wavelength. To correct chromatic aberration, we estimate a spectral image that corresponds to the RGB image by solving a global nu- merical optimization problem. We include our model of chromatic aberration ii in the data-fitting term of the optimization problem that models the transfor- mation from the spectral image to the captured RGB image. We also include regularization terms in our optimization problem to enforce smoothness in the output image. Whereas the captured RGB image is mosaiced, meaning that each pixel senses only one colour channel, our algorithm does not require a demosaicing preprocessing step to recover all colour channel intensities at each pixel. Therefore, we avoid introducing bias from demosaicing algorithms, which is important because chromatic aberration and demosaicing are known to interact. Since we model within-channel chromatic aberration, our reconstructed im- ages are sharper than those obtained by previous works on calibrated warping of colour channels. In contrast to explicit deblurring algorithms, our algo- rithm leaves defocus blur intact, separating it from chromatic aberration. We also avoid introducing artifacts, such as ringing, that are commonly produced by deblurring algorithms. Nevertheless, recovering spectral images from RGB images is an ill-posed problem, and this ill-posedness is the major limitation of our approach. We determined that our spectral images have higher accu- racy than measurements made using a consumer-grade spectrometer. Still, we recommend further research on RGB-to-spectral reconstruction, especially in relation to chromatic aberration, which may serve as useful constraint. iii Preface This thesis is an original work by Bernard Llanos. Research for this thesis was supervised by Dr. Herb Yang at the University of Alberta. No part of this thesis has been previously published. iv Acknowledgements The research described in this thesis involved many people, and I am grateful for their assistance. I would like to thank my supervisor, Dr. Herb Yang, who patiently advised me even when I did not have a clear research direction, and who gave me freedom to find a project I was interested in. They helped me find the joy of discovery. I also thank Dr. Martin Jagersand, for providing me with similar freedom as their teaching assistant, and for teaching me many of the concepts and techniques used in my research. Dr. Pierre Boulanger has also benefitted my work by serving on my examining committee with Dr. Yang and Dr. Jagersand. I am especially grateful to Steve Sutphen, from the Department of Com- puting Science, who built the physical apparatus for my experiments. They translated my requirements into a design that exceeded my expectations. Ad- ditionally, I am grateful to them for sharing their experience with me through interesting conversations. Similarly, I have benefitted from the experience of numerous members of the Department of Computing Science. Dr. Yiming Qian introduced me to computer vision research and gave me practical advice for success. The other members of the research group were a continual source of ideas through their presentations of literature. They include: G M Mashrur E Elahi, Juehui Fan, Dong Huo, Dr. Zhao Pei, H M Ata-E-Rabbi, Katie Tran, and Dr. Shibai Yin. Outside the Department, I am grateful to Dr. Jilu Feng and Dr. Benoit Rivard for providing reference spectrometry data for my experiments. I am grateful to the Faculty of Science, the Faculty of Graduate Studies and Re- search, and the Graduate Students' Association of the University of Alberta for financial support in the form of scholarships and travel grants. v Alberta Innovates, and the Natural Sciences and Engineering Research Council of Canada granted significant funding for my studies. vi Contents 1 Introduction 1 1.1 Image Formation . 3 1.1.1 The Point spread function . 3 1.1.2 Chromatic Aberration . 4 1.2 Contributions . 7 2 Related Work 9 2.1 Demosaicing . 9 2.2 Chromatic aberration correction . 12 2.3 Spectral Reconstruction . 16 2.3.1 Physically-Based Approaches . 16 2.3.2 Data-Driven . 18 3 Image Reconstruction 19 3.1 Overview . 19 3.1.1 Terminology . 21 3.2 Modelling Dispersion . 21 3.2.1 Keypoint Localization . 21 3.2.2 Vignetting correction . 23 3.2.3 Image registration . 25 3.2.4 Dispersion Interpolation . 26 3.2.5 Image Warping . 32 3.3 Image Representation . 34 3.3.1 Converting Spectra to Colour . 34 3.3.2 Spectral Image Representation . 36 3.3.3 Spectral irradiance computation . 40 3.4 Mosaicing . 43 3.5 Regularization Penalties . 43 3.6 Weights on Regularization Penalties . 46 3.7 Optimization Algorithm . 51 3.8 Implementation . 55 3.8.1 Patch-Wise Image Estimation . 55 3.8.2 Running Time Comparison with Other Systems . 59 3.9 Algorithm Variants . 60 3.9.1 Direct Colour Image Estimation . 60 3.9.2 Image Estimation without Dispersion . 61 4 Experimental Design 63 4.1 Approaches to Image Reconstruction Evaluation . 63 4.1.1 Spectral Reconstruction Evaluation . 63 4.1.2 Chromatic Aberration Correction Evaluation . 64 4.2 Real Experiments . 67 4.2.1 Apparatus . 67 vii 4.2.2 Image Collection . 69 4.2.3 High-Dynamic Range Image Synthesis . 71 4.2.4 Multispectral Image Synthesis . 77 4.3 Synthetic Experiments . 84 4.3.1 Datasets . 84 4.3.2 Limitations . 88 4.4 Quantitative Evaluation . 88 4.4.1 Spectral Error . 89 4.4.2 Colour Error . 91 4.4.3 Residual Chromatic Aberration . 92 4.5 Visual Evaluation . 97 4.6 Comparison Methods . 100 4.6.1 Spectrometry . 100 4.6.2 RGB-to-Spectral Algorithms . 101 4.6.3 Demosaicing Algorithms . 103 4.6.4 Chromatic Aberration Correction Algorithms . 104 5 Results 109 5.1 Overview . 109 5.2 Reconstruction of Images from the KAIST Dataset . 111 5.2.1 Comprehensive Evaluation . 111 5.2.2 Non-Negativity Constraint . 119 5.2.3 Selection of Image Priors . 120 5.2.4 Weights on Regularization Penalties . 123 5.3 Models of Dispersion . 124 5.4 Image Reconstruction from Raw RGB Images . 128 5.4.1 Experiment Details . 128 5.4.2 Image Estimation Evaluation . 129 5.4.3 Spectral Image Representation . 142 5.4.4 Weights on Regularization Penalties . 151 5.5 Image Reconstruction from Synthetic RGB Images . 151 5.5.1 Ideal Chromatic Aberration Correction . 153 5.5.2 Image Estimation Evaluation . 154 6 Discussion 160 6.1 Chromatic Aberration and Image Quality . 160 6.2 Alternative Models of Dispersion . 161 6.3 Spectral Response Calibration . 162 6.4 Comparison with \Compact single-shot hyperspectral imaging using a prism" . 164 7 Future Work 167 7.1 Calibration . 167 7.1.1 Camera Spectral Response Calibration . 167 7.1.2 Dispersion . 168 7.2 Spectral Reconstruction . 169 7.2.1 Regularization . 169 7.2.2 Data-Driven Spectral Reconstruction . 171 7.3 Multispectral Imaging . 172 8 Conclusion 174 References 178 Appendix A Additional Results on the KAIST Dataset 191 viii Appendix B Supplemental Experiments on Dispersion Model Calibration 200 B.1 Vignetting Correction . 200 B.2 Image Registration vs. Disk Keypoints . 201 B.3 Polynomial vs. Thin-Plate Spline Interpolation . 201 B.4 Comparison of Dispersion between Lenses . 205 B.5 Summary . 207 ix List of Tables 3.1 Combinations of image prior terms evaluated . 44 4.1 Scaling factors relating different shutter times . 73 5.1 KAIST dataset spectral error . 112 5.2 KAIST dataset colour error . 115 5.3 Non-negativity constraint evaluation using the KAIST dataset (spectral error) . 120 5.4 Non-negativity constraint evaluation using the KAIST dataset (colour error) . 121 5.5 Spectral residual chromatic aberration evaluation . 135 5.6 Colour residual chromatic aberration evaluation . 136 5.7 Spectral reconstruction evaluation . 141 5.8 Spectral reconstruction evaluation for different spectral control bands . 145 5.9 Spectral residual chromatic aberration evaluation (multispec- tral images) . 154 5.10 Colour residual chromatic aberration evaluation (multispectral images) . 155 A.1 KAIST dataset first image spectral error . 191 A.2 KAIST dataset first image colour error . 194 x List of Figures 1.1 A diagrammatic explanation of chromatic aberration . 6 1.2 Refractive index of SCHOTT N-BK7 glass . 6 2.1 A Bayer colour-filter array pattern . 9 3.1 Graphical overview of image reconstruction .