RRG-GAN Restoring Network for Simple Lens Imaging System
Total Page:16
File Type:pdf, Size:1020Kb
sensors Article RRG-GAN Restoring Network for Simple Lens Imaging System Xiaotian Wu 1,2 , Jiongcheng Li 3, Guanxing Zhou 3, Bo Lü 2, Qingqing Li 2 and Hang Yang 2,* 1 College of Opto-Electronic Engineering, Changchun University of Science and Technology, Changchun 130022, China; [email protected] 2 Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China; [email protected] (B.L.); [email protected] (Q.L.) 3 School of Informatics, Xiamen University, Xiamen 361005, China; [email protected] (J.L.); [email protected] (G.Z.) * Correspondence: [email protected] Abstract: The simple lens computational imaging method provides an alternative way to achieve high-quality photography. It simplifies the design of the optical-front-end to a single-convex-lens and delivers the correction of optical aberration to a dedicated computational restoring algorithm. Traditional single-convex-lens image restoration is based on optimization theory, which has some shortcomings in efficiency and efficacy. In this paper, we propose a novel Recursive Residual Groups network under Generative Adversarial Network framework (RRG-GAN) to generate a clear image from the aberrations-degraded blurry image. The RRG-GAN network includes dual attention module, selective kernel network module, and residual resizing module to make it more suitable for the non-uniform deblurring task. To validate the evaluation algorithm, we collect sharp/aberration- degraded datasets by CODE V simulation. To test the practical application performance, we built a display-capture lab setup and reconstruct a manual registering dataset. Relevant experimental comparisons and actual tests verify the effectiveness of our proposed method. Citation: Wu, X.; Li, J.; Zhou, G.; Keywords: computational imaging; deep learning; image restoring; non-uniform deblurring Lü, B.; Li, Q.; Yang, H. RRG-GAN Restoring Network for Simple Lens Imaging System. Sensors 2021, 21, 3317. https://doi.org/10.3390/ 1. Introduction s21103317 Computational imaging is a new interdisciplinary subject in recent years, which offers Academic Editor: Gwanggil Jeon imaging functionalities and convenient design beyond traditional imaging design [1]. It emphasizes the task-oriented global optimization design in the full imaging chain, and Received: 13 March 2021 systematically balances the system resource dependence in the physical and computing Accepted: 28 April 2021 domain. Computational imaging has many sub-topics according to different imaging Published: 11 May 2021 backgrounds [2,3]. This paper will focus on the simple lens imaging system and its restoring method. Publisher’s Note: MDPI stays neutral Aberration is not only the main consideration in the optical design stage, but also with regard to jurisdictional claims in a factor limiting the imaging quality in the actual use due to the change of aperture, published maps and institutional affil- object distance, and other factors. To minimize optical aberrations, the manufacturing of iations. photographic lenses has become increasingly complex. Optical designers systematically balance optical aberrations and design constraints (such as focal length, field of view, and distortion). They utilize a combination of several lens elements with various materials and shapes to achieve a close-to-perfect optical design, which will result in a significant impact Copyright: © 2021 by the authors. on the cost, size, and weight. The simple lens computational imaging method provides Licensee MDPI, Basel, Switzerland. an alternative way to achieve high-quality photography. It simplifies the design of the This article is an open access article optical-front-end to a single-convex-lens, and delivers the correction of optical aberration distributed under the terms and to a dedicated computational restoring algorithm, as shown in Figure1. As aberration is a conditions of the Creative Commons common problem in many optical imaging systems, aberration correction algorithm will Attribution (CC BY) license (https:// have great significance to improve the quality of other optical imaging systems, and has creativecommons.org/licenses/by/ broad application prospects. 4.0/). Sensors 2021, 21, 3317. https://doi.org/10.3390/s21103317 https://www.mdpi.com/journal/sensors Sensors 2021, 21, x FOR PEER REVIEW 2 of 17 Sensors 2021, 21, 3317 2 of 16 have great significance to improve the quality of other optical imaging systems, and has broad application prospects. Conventional SLR lens Single convex lens Direct Capture VS Restoring Comparable FigureFigure 1. 1.The The schematic schematic diagram diagram of of single-convex-lens single-convex-lens computational computational imaging imaging method. method. InIn this this paper, paper, we we will will build build a a single-convex-lens single-convex-lens imaging imaging system, system, and and further further study study thethe restoring restoring methodmethod ofof aberrationaberration degraded image. image. The The contributions contributions of of this this paper paper are are as asfollows: follows: 1.1. WeWe collectcollect sharp/aberration-degraded sharp/aberration-degraded datasets datasets by by CODE CODE V ( https://www.synopsys.V (https://www.synop- com/optical-solutions/codev.htmlsys.com/optical-solutions/codev.html), accessed simulation on 11 Mayand 2021)manual simulation registering, and which man- ualwill registering, be publicly which willreleased be publicly on released Github on Githubfor forfurther further researchesresearches ((https://github.com/wuzeping1893/RRGhttps://github.com/wuzeping1893/RRG-GAN-single-convex-lens-GAN-single-convex-lens). To, accessedthe best on of 11our May knowledge, 2021). To this the is best the offirst our dataset knowledge, for the thissingle is- theconvex first-lens dataset computational for the single- im- convex-lensaging field. computational imaging field. 2.2. WeWe proposepropose the the application application of deep-learning-only-basedof deep-learning-only-based methods methods for image for denoisingimage de- andnoising deblurring and deblurring to the special to the case special of single-lens case of single camera-lens images camera restoring, images restoring, in contrast in withcontrast optimization-based with optimization methods-based methods with great with improvement great improvement in efficiency in efficiency and efficacy. and 3. Byefficacy. redesigning the generator network, the proposed RRG-GAN network includes 3. theBy dualredesigning attention the module, generator selective network, kernel the networkproposed module, RRG-GAN and residualnetwork resizingincludes module.the dual Itattention has better module, multi-scale selective feature kernel extraction network and module, fusion ability,and residual which resizing makes themodule. network It has have better better mu recoverylti-scale effect.feature extraction and fusion ability, which makes Thethe followingnetwork have sections better are recovery arranged effect. as follows: Section2 briefly describes the related work andThe thefollowing chosen sections direction are of arranged this paper. as Section follows:3 introduces Section 2 briefly hardware describes implementation the related of single-convex-lens imaging system and our proposed RRG-GAN restoring network. work and the chosen direction of this paper. Section 3 introduces hardware implementa- Section4 provides the experimental preparation, results, and analysis. Section5 discusses tion of single-convex-lens imaging system and our proposed RRG-GAN restoring net- our work and future research. Section6 presents the conclusions of our methods. work. Section 4 provides the experimental preparation, results, and analysis. Section 5 2.discusses Related our Work work and future research. Section 6 presents the conclusions of our meth- 2.1.ods. Single-Lens Imaging System 2. RelatedThe idea Work of single-convex-lens computational imaging was first proposed by Schuler et al. [4]. They utilized a single-convex-lens as the only optical element, measured the non-uniform point2.1. Single spread-Lens function Imaging (PSF) System grid covering via an automated calibration procedure, and elim- inatedThe the idea effect of of single aberration-convex through-lens computational a non-stationary imaging convolution was first [4]. proposed Following by Schuler’s Schuler work,et al. [4]. Felix They improved utilized the a single restoring-convex method-lens byas the only proposed optical cross-channel element, measured prior [5 the]. non- uniformIn order point to spread expand function the field (PSF) of view grid (FOV), covering Peng via [6an] proposedautomated a calibration thin-plate opticsproce- computationaldure, and eliminated imaging the method effect using of aberration a single-Fresnel-lens, through a non another-stationary construction convolution method [4]. of single lens. Similar to the single-convex-lens computational imaging method, some scholars proposed dedicated image restoring methods for computational imaging systems using diffractive optical elements (DOEs) [7,8]. Sensors 2021, 21, 3317 3 of 16 2.2. Motion Deblurring Algorithms Most computational imaging methods are all indirect imaging, which means the source image data obtained from the sensor is seriously affected by the aberration. Therefore, the design of robust computational restoring algorithms is the key issue of these computational imaging systems. Usually, the aberration correcting and restoring methods refer to some ideas of