RRG-GAN Restoring Network for Simple Lens Imaging System

sensors

Article RRG-GAN Restoring Network for Simple Lens Imaging System

Xiaotian Wu 1,2 , Jiongcheng Li 3, Guanxing Zhou 3, Bo Lü 2, Qingqing Li 2 and Hang Yang 2,*

1 College of Opto-Electronic Engineering, Changchun University of Science and Technology, Changchun 130022, China; [email protected] 2 Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China; [email protected] (B.L.); [email protected] (Q.L.) 3 School of Informatics, Xiamen University, Xiamen 361005, China; [email protected] (J.L.); [email protected] (G.Z.) * Correspondence: [email protected]

Abstract: The simple lens computational imaging method provides an alternative way to achieve high-quality photography. It simplifies the design of the optical-front-end to a single-convex-lens and delivers the correction of optical aberration to a dedicated computational restoring algorithm. Traditional single-convex-lens image restoration is based on optimization theory, which has some shortcomings in efficiency and efficacy. In this paper, we propose a novel Recursive Residual Groups network under Generative Adversarial Network framework (RRG-GAN) to generate a clear image from the aberrations-degraded blurry image. The RRG-GAN network includes dual attention module, selective kernel network module, and residual resizing module to make it more suitable for the non-uniform deblurring task. To validate the evaluation algorithm, we collect sharp/aberration- degraded datasets by CODE V simulation. To test the practical application performance, we built a display-capture lab setup and reconstruct a manual registering dataset. Relevant experimental comparisons and actual tests verify the effectiveness of our proposed method.

Citation: Wu, X.; Li, J.; Zhou, G.; Keywords: computational imaging; deep learning; image restoring; non-uniform deblurring Lü, B.; Li, Q.; Yang, H. RRG-GAN Restoring Network for Simple Lens Imaging System. Sensors 2021, 21, 3317. https://doi.org/10.3390/ 1. Introduction s21103317 Computational imaging is a new interdisciplinary subject in recent years, which offers

Academic Editor: Gwanggil Jeon imaging functionalities and convenient design beyond traditional imaging design [1]. It emphasizes the task-oriented global optimization design in the full imaging chain, and Received: 13 March 2021 systematically balances the system resource dependence in the physical and computing Accepted: 28 April 2021 domain. Computational imaging has many sub-topics according to different imaging Published: 11 May 2021 backgrounds [2,3]. This paper will focus on the simple lens imaging system and its restoring method. Publisher’s Note: MDPI stays neutral Aberration is not only the main consideration in the optical design stage, but also with regard to jurisdictional claims in a factor limiting the imaging quality in the actual use due to the change of aperture, published maps and institutional affil- object distance, and other factors. To minimize optical aberrations, the manufacturing of iations. photographic lenses has become increasingly complex. Optical designers systematically balance optical aberrations and design constraints (such as focal length, field of view, and distortion). They utilize a combination of several lens elements with various materials and shapes to achieve a close-to-perfect optical design, which will result in a significant impact Copyright: © 2021 by the authors. on the cost, size, and weight. The simple lens computational imaging method provides Licensee MDPI, Basel, Switzerland. an alternative way to achieve high-quality photography. It simplifies the design of the This article is an open access article optical-front-end to a single-convex-lens, and delivers the correction of optical aberration distributed under the terms and to a dedicated computational restoring algorithm, as shown in Figure1. As aberration is a conditions of the Creative Commons common problem in many optical imaging systems, aberration correction algorithm will Attribution (CC BY) license (https:// have great significance to improve the quality of other optical imaging systems, and has creativecommons.org/licenses/by/ broad application prospects. 4.0/).

Sensors 2021, 21, 3317. https://doi.org/10.3390/s21103317 https://www.mdpi.com/journal/sensors Sensors 2021, 21, x FOR PEER REVIEW 2 of 17

Sensors 2021, 21, 3317 2 of 16 have great significance to improve the quality of other optical imaging systems, and has broad application prospects.

Conventional SLR lens Single convex lens

Direct Capture

VS Restoring Comparable FigureFigure 1. 1.The The schematic schematic diagram diagram of of single-convex-lens single-convex-lens computational computational imaging imaging method. method.

InIn this this paper, paper, we we will will build build a a single-convex-lens single-convex-lens imaging imaging system, system, and and further further study study thethe restoring restoring methodmethod ofof aberrationaberration degraded image. image. The The contributions contributions of of this this paper paper are are as asfollows: follows: 1.1. WeWe collect collect sharp/aberration-degraded sharp/aberration-degraded datasets datasets by by CODE CODE V ( https://www.synopsys.V (https://www.synop- com/optical-solutions/codev.htmlsys.com/optical-solutions/codev.html), accessed simulation on 11 Mayand 2021)manual simulation registering, and which man- ualwill registering, be publicly which will released be publicly on released Github on Githubfor for further further researches researches ((https://github.com/wuzeping1893/RRGhttps://github.com/wuzeping1893/RRG-GAN-single-convex-lens-GAN-single-convex-lens). To, accessedthe best on of 11our May knowledge, 2021). To this the is best the offirst our dataset knowledge, for the thissingle is- theconvex first-lens dataset computational for the single- im- convex-lensaging field. computational imaging field. 2.2. WeWe propose propose the the application application of deep-learning-only-based of deep-learning-only-based methods methods for image for denoising image de- andnoising deblurring and deblurring to the special to the case special of single-lens case of single camera-lens images camera restoring, images restoring, in contrast in withcontrast optimization-based with optimization methods-based methods with great with improvement great improvement in efficiency in efficiency and efficacy. and 3. Byefficacy. redesigning the generator network, the proposed RRG-GAN network includes 3. theBy dualredesigning attention the module, generator selective network, kernel the networkproposed module, RRG-GAN and residualnetwork resizingincludes module.the dual Itattention has better module, multi-scale selective feature kernel extraction network and module, fusion ability,and residual which resizing makes themodule. network It has have better better mu recoverylti-scale effect.feature extraction and fusion ability, which makes Thethe followingnetwork have sections better are recovery arranged effect. as follows: Section2 briefly describes the related work andThe thefollowing chosen sections direction are of arranged this paper. as Section follows:3 introduces Section 2 briefly hardware describes implementation the related of single-convex-lens imaging system and our proposed RRG-GAN restoring network. work and the chosen direction of this paper. Section 3 introduces hardware implementa- Section4 provides the experimental preparation, results, and analysis. Section5 discusses tion of single-convex-lens imaging system and our proposed RRG-GAN restoring net- our work and future research. Section6 presents the conclusions of our methods. work. Section 4 provides the experimental preparation, results, and analysis. Section 5 2.discusses Related our Work work and future research. Section 6 presents the conclusions of our meth- 2.1.ods. Single-Lens Imaging System

2. RelatedThe idea Work of single-convex-lens computational imaging was first proposed by Schuler et al. [4]. They utilized a single-convex-lens as the only optical element, measured the non-uniform point2.1. Single spread-Lens function Imaging (PSF) System grid covering via an automated calibration procedure, and elim- inatedThe the idea effect of of single aberration-convex through-lens computational a non-stationary imaging convolution was first [4]. proposed Following by Schuler’s Schuler work,et al. [4]. Felix They improved utilized the a single restoring-convex method-lens byas the only proposed optical cross-channel element, measured prior [5 the]. non- uniformIn order point to spread expand function the ﬁeld (PSF) of view grid (FOV), covering Peng via [6an] proposedautomated a calibration thin-plate opticsproce- computationaldure, and eliminated imaging the method effect using of aberration a single-Fresnel-lens, through a non another-stationary construction convolution method [4]. of single lens. Similar to the single-convex-lens computational imaging method, some scholars proposed dedicated image restoring methods for computational imaging systems using diffractive optical elements (DOEs) [7,8]. Sensors 2021, 21, 3317 3 of 16

2.2. Motion Deblurring Algorithms Most computational imaging methods are all indirect imaging, which means the source image data obtained from the sensor is seriously affected by the aberration. Therefore, the design of robust computational restoring algorithms is the key issue of these computational imaging systems. Usually, the aberration correcting and restoring methods refer to some ideas of motion deblurring algorithms. The motion deblurring algorithms are mainly based on two kinds of theories: one is traditional optimization, the other is deep learning. The motion deblurring methods based on the optimization framework belong to a theoretical method driven by the physical model. Image deblurring is a typical ill-posed problem, the restoration methods usually need to add prior constraints and integrate them into optimization framework to make ill-conditioned problems solvable. Representative image priors mainly include L0 gradient [9], gradient sparsity [10,11], dark channel [12], color-line [13] etc. The optimization framework adopted mainly includes maximum a posteriori approach (MAP) and variational expectation maximization (VEM). The motion deblurring methods based on deep learning belong to a data-driven theoretical implementation method, but the pointcuts and data-driven implementation methods are different. Chakrabarti [14] used a deep learning network to estimate the blur kernel Fourier coefficient. Wang trained a discriminative classifier to distinguish blurry and sharp images as the regularization term in the maximum a posteriori (MAP) framework [15]. Similarly, based on the MAP framework, Ren et al. [16] further obtained the image priors and blur kernel priors all by deep learning network, and thus improved the image restoration effect in the case of complex or large-scale blur kernel. The above methods are mainly based on the combination of deep learning and traditional optimization: the traditional optimization method is responsible for the main process, while the deep learning algorithm is used to improve the robustness of various priors. Due to the joint application of deep learning and optimization theory, these methods improve the recovery effect, but there is no improvement in efficiency. The end-to-end deep network is another way to implement computational recovery. Nah et al. [17] proposed an end-to-end multi-scale convolutional neural network to realize the deblurring algorithm. Zhang et al. [18] make use of the feature extraction advantage of RNN and the weight learning advantage of CNN to realize a non-uniform deblurring neural network. Zhou et al. [19] proposed a multi-stream bottom-top-bottom attention network, which can effectively facilitate feature extracting and reduce computational complexity. With the development of generative adversarial networks (GAN), the implementation method of directly generating end-to-end restoration images and ignoring the physical model process is also applied in the field of image deblurring, which is represented as Deblur-GAN, proposed by Kupyn [20].

2.3. Motivation of This Paper The motivation of this paper is to utilize the deep learning method instead of the previous optimization-based methods [4,5] to improve the efficacy and efficiency. From the perspective of implementation effect, previous methods will fail when the size of blurry kernel is large. In contrast, our experiments in Section4 will prove that the deep learning restoration method has better robustness and better subjective restoration effect in this aspect. In terms of operational efficiency, the optimization methods usually require several iterations, which means they are difficult to apply in practical engineering. In contrast, the deep learning end-to-end restoring methods are more efficient, although they require additional hardware resources such as GPU. We make appropriate improvements on the Deblur-GAN [20] restoring method, which is originally used in motion blur recovery. The improvements mainly reflect the restructuring of the generator network, which includes the dual attention module, selective kernel network module, and residual resizing module. These improvements mean the network restoration has better multi-scale feature extraction and fusion ability, as shown in Section4. Sensors 2021, 21, x FOR PEER REVIEW 4 of 17

deep learning restoration method has better robustness and better subjective restoration effect in this aspect. In terms of operational efficiency, the optimization methods usually require several iterations, which means they are difficult to apply in practical engineering. In contrast, the deep learning end-to-end restoring methods are more efficient, although they require additional hardware resources such as GPU. We make appropriate improvements on the Deblur-GAN [20] restoring method, which is originally used in motion blur recovery. The improvements mainly reflect the restructuring of the generator network, which includes the dual attention module, selective kernel network module, and residual resizing module. These improvements mean the network restoration has better multi-scale feature extraction and fusion ability, as shown in Section 4.

3. Methodology

Sensors 2021, 21, 3317 3.1. Hardware Implementation of Single-Convex-Lens Imaging System 4 of 16 3.1.1. Self-Made Optical Lens The self-made optical lens consists of a flat-convex lens, a gland ring, and a lens barrel. The flat-convex lens is the only optical element to converge light. The focal length of 3. Methodology the lens is 50 mm with 15° field-of-view (FOV), which is larger than the previous work 3.1. Hardware Implementation of Single-Convex-Lens Imaging System [4,5]. The gland ring is used to fix the lens and act as a simple aperture. Our self-made 3.1.1. Self-Made Optical Lens optical lens has a corresponding standard single-convex-lens reflex (SLR) lens with the sameThe focal self-made length and optical field lens angle, consists as shown of a flat-convex in Figure 2. lens, a gland ring, and a lens barrel. The flat-convexThe parameters lens isof theour only self-made optical optical element lens to can converge be found light. in CODE The focal V source length project of the ◦ lensdocuments, is 50 mm which with is 15 availablefield-of-view on Github (FOV), project which page. is larger The thanlight thepath previous analysis work in Figure [4,5]. The2a and gland the ringsubsequent is used simulations to fix the lens are and all actbased as aon simple the premise aperture. of infinite Our self-made object distance. optical lensFor the has case a corresponding of small object standard distance, single-convex-lens the change of PSF reflex is varied (SLR) very lens withsharply. the sameWe do focal not lengthdiscuss and these field situations angle, as in shown this paper. in Figure 2.

(a) (b) (c)

Figure 2.2. The procedureprocedure ofof self-madeself-made optical optical lens: lens: ( a(a)) The The light light path path of of lens lens using using CODE CODE V V software; software; (b )(b The) The components components of self-madeof self-made optical optical lens; lens; (c) ( Thec) The assemble assemble of self-madeof self-made optical optical lens lens and and its correspondingits corresponding standard standard SLR SLR lens. lens. The parameters of our self-made optical lens can be found in CODE V source project 3.1.2. Aberration Analysis and Imaging Model documents, which is available on Github project page. The light path analysis in Figure2a and theThe subsequentaberrations simulationsproduced by are the all single-conve based on thex-lens premise will lead of infinite to the objectdegradation distance. of Forimage the quality. case of These small objectaberrations distance, can thebe divide changed ofinto PSF on-axis is varied aberrations very sharply. and off-axis We do notab- discusserrations. these The situations on-axis aberrations in this paper. mainly include the longitudinal chromatic aberration and the axial spherical aberration. The off-axis aberrations mainly include lateral chro- 3.1.2.matic Aberrationaberration, Analysiscoma, curvature and Imaging of field, Model astigmatism, and distortion. The overall influence Theof the aberrations above aberrations produced on by imaging the single-convex-lens is reflected in the willcomplexity lead to theof the degradation point spread of imagefunction quality. (PSF). The These PSFs aberrations are different can in be the divided RGB channels, into on-axis global aberrations non-uniform and with off-axis large aberrations.kernel size. The on-axis aberrations mainly include the longitudinal chromatic aberration and theAffected axial spherical by optical aberration. aberrations, The the off-axis non-un aberrationsiform blurry mainly imaging include model lateral in RGB chromatic chan- aberration,nel is described coma, in curvatureEquation (1): of field, astigmatism, and distortion. The overall influence of the above aberrationsOIKN(,i j ) on=+∈ imaging ( mn , is ) reflected ( mni , ,, j in ) the (,complexity i j ), c { RGB , of , the } point spread cccc (1) function (PSF). The PSFs are(,) differentmn in the RGB channels, global non-uniform with large kernel size. Affected by optical aberrations, the non-uniform blurry imaging model in RGB channel is described in Equation (1):

Oc(i, j) = ∑ Ic(m, n)Kc(m, n, i, j) + Nc(i, j), c ∈ {R, G, B} (1) (m,n)

In Equation (1), Oc(i, j) denotes the source image collected by the imaging system, (i, j) depicts the pixel position of the observed image space; Ic(m, n) represents the ground truth image, (m, n) is the pixel position of the ground truth image space; Kc(m, n, i, j) describes the non-uniform point spread function (PSF) of the motion blur; Nc(i, j) is the CMOS noise. The observed blurry image Oc(i, j) is known, the ground truth image Ic(m, n), the point spread function Kc(m, n, i, j), and the CMOS noise Nc(i, j) are all unknown, which means it is a typical ill-posed problem in mathematics. Mathematically, the ill-posed problems can be solved by adding prior constraints under the optimization framework. However, the optimization-based restoring methods usually require multiple iterations. The computing time of a single image with 1920 × 1080 resolution is usually more than 5 min, which is unacceptable in industrial applications. With the continuous development of deep learning theory and GPU acceleration hardware Sensors 2021, 21, x FOR PEER REVIEW 5 of 17

OIKN(,)ij= (,) mn (,,,) mnij + (,), ijcRGB  {,,} c c c c (1) (,)mn

In Equation (1), Oc (,)ij denotes the source image collected by the imaging system,

(,)ij depicts the pixel position of the observed image space; Ic (,)mn represents the ground truth image, (,)mn is the pixel position of the ground truth image space;

Kc (,,,)m n i j describes the non-uniform point spread function (PSF) of the motion blur;

Nc (,)ij is the CMOS noise. The observed blurry image Oc (,)ij is known, the ground

truth image Ic (,)mn , the point spread function Kc (,,,)m n i j , and the CMOS noise

Nc (,)ij are all unknown, which means it is a typical ill-posed problem in mathematics. Mathematically, the ill-posed problems can be solved by adding prior constraints under the optimization framework. However, the optimization-based restoring methods usually require multiple iterations. The computing time of a single image with 1920 × 1080 Sensors 2021, 21, 3317 resolution is usually more than 5 min, which is unacceptable in industrial applications.5 of 16 With the continuous development of deep learning theory and GPU acceleration hardware technology, the restoring methods based on deep learning show great advantages technology,in effect and the performance restoring methods [21,22]. basedThe restoring on deep method learning proposed show great in advantages this paper is in based effect andon the performance deep learning [21, 22theory]. The and restoring realizes method the image proposed restoration in this procedure paper is based in an onend the-to deep-end learningway. theory and realizes the image restoration procedure in an end-to-end way.

3.2. Proposed RestoringRestoring Method 3.2.1. Network Architecture We proposepropose aa novelnovel generativegenerative adversarialadversarial networknetwork (GAN)(GAN) toto generategenerate aa clear image from the aberrations-degradedaberrations-degraded blurry image. The generative adversarial network (GAN), proposed byby Goodfellow [[23],23], became one of the mostmost attractive schemes in deep learning in recent years. years. GAN GAN utilize utilize a adiscriminator discriminator network network as asa special a special robust robust loss loss function function in- insteadstead of ofthe the traditional traditional hand hand-crafted-crafted loss loss function, function, which which improves improves the performance the performance and androbustness robustness of deep of deepnetworks networks [24,25]. [24 Sp,25ecifically,]. Speciﬁcally, we adopt we the adopt recursive the recursive residual residual groups groups(RRG) network (RRG)network [26] as the [26 generative] as the generative model G, thus model our G, proposed thus our generative proposed adversarial generative adversarialnetwork is named network as is RRG named-GAN. as RRG-GAN. The discriminator The discriminator of RRG-GAN of RRG-GAN is a double is scale a double dis- scalecriminator discriminator network network [20]. The [ 20proposed]. The proposed network network architecture architecture is shown is in shown Figure in 3. Figure 3.

Generator

RRG RRG RRG Discriminator

CONV CONV

Blur Patch Generate Patch LLLLG=0.5* p + 0.006* x + 0.01* adv Global RRG Detail Local

LRaLSGAN = E[(() D x − E D (())1)] G z − 2 D x~ pdata ( x ) z ~ P z ( z ) +E[((()) D G z − E D ()1)] x + 2

z~Pz ( z ) x ~ P data ( x )

CONV

CONV CONV RRG Sharp Patch Sub module

Upsampling Downsampling Dual attention Selective kernel MRB Detail MRB module module unit feature fusion

FigureFigure 3.3. TheThe RRG-GANRRG-GAN networknetwork architecture.architecture. (*(* denotesdenotes thethe multiplicationmultiplication operation).operation)

3.2.2. Discriminator Network: Double Double-Scale-Scale Discriminator Network The discriminator of RRG-GANRRG-GAN is a double-scaledouble-scale discriminator network.network. The original GAN network’s discriminator maps the entire input to a probability of judgingjudging whetherwhether the input sample is a real imageimage [[20]20].. However, thisthis methodmethod doesdoes notnot workwork veryvery wellwell forfor high-resolution and high-deﬁnition detailed images. PatchGAN [27] maps the entire input to an n*n patch matrix to classify, and then averages them to obtain the ﬁnal output of the discriminator. Therefore, PatchGAN can make the model pay more attention to the details of the local image to achieve better results. The double scale discriminator, proposed by Kupyn [20], further improves PatchGAN by adding a global image view. The combine use of local and global view was proved more suitable for deblurring task [20]. The loss function of discriminant network is RaGAN-LS loss, also proposed by Kupyn [20], as shown in Equation (2), which can be seen as an upgrade version of LSGAN loss function [28].

LRaLSGAN = E [(D(x) − E D(G(z)) − 1)2] D x∼pdata(x) z∼Pz(z) (2) +E [(D(G(z)) − E D(x) + 1)2] z∼Pz(z) x∼Pdata(x)

where x and z respectively represent the real image and generator G’s latent variables input. Pz(z) is the probability distribution of z, Pdata(x) is the probability distribution of the dataset. G(z) denotes the generator of z. D(x) represents real image x’s double-scale discriminator. D(G(z)) represents G(z)’s double-scale discriminator. Sensors 2021, 21, 3317 6 of 16

3.2.3. Generator Network: RRG-Net The generator network is implemented by cascading three recursive residual groups (RRGs) as shown in Figure3. RRG is proposed in the structure of Cycle ISP network by Google Research Institute [26]. Cycle ISP network is an end-to-end network, which can simulate the real CMOS noise degradation, and then achieve image restoration, it can realize bidirectional conversion and circulation in sRGB domain and RAW domain. In this work, there is no cycle in the construction of generation network, we only use the one-way generation function in Cycle ISP network. The cascading recursive residual groups can be described as:

T0 = Ms(Iin), Td = RRGN(... RRG1(T0)), Iout = ME(Td) (3)

H×W×3 where Ms is the starting convolution operation on the input color image Iin ∈ R to H×W×C H×W×C get the low-level feature parameters T0 ∈ R ; the high-level features of Td ∈ R are obtained by the iteration of recursive residual groups (RRGs); then Td is performed the ﬁnal convolution operation ME on the high-level features Td is to obtain the reconstructed H×W×3 image Iout ∈ R . RRG’s framework is basically similar to the earlier proposed recursive residual network [29], but the construction of “group network” is more complex. In this paper, we choose multi-scale residual block (MRB) as the implementation of “group network”. The multi-scale residual block network contains several popular sub-modules to improve the performance of feature extraction [30]. These sub-modules include dual attention blocks [31], selective kernel networks [32], and residual resizing modules [33]. The dual attention blocks are used in each scale, which can simulate the process of human eyes searching for important information in the visual scene for centralized analysis while ignoring irrelevant information in the scene. The selective kernel networks are used in multi-scale. They can effectively learn to fuse important features extracted by the dual attention blocks. The residual resizing networks provide additional network parameters for learning in the inter-scale up-down-sampling operation, so as to further improve the performance of the whole network. We use a hybrid three term loss function as the overall loss function for generator network, which is proposed in DeblurGAN-v2 [20]. The loss function is shown in Equation (4).

LG = 0.5 ∗ Lp + 0.006 ∗ Lx + 0.01 ∗ Ladv (4)

where Lp is the similarity between two images, we use the SSIM (Structural Similarity) [34] measurement; Lx denotes the perceptual loss, which computes the Euclidean loss on the VGG19 [35] feature maps to make the network pay more attention to learning coding perception and high-level semantic information; Ladv represents the adversarial loss, which contains both global and local discriminator losses (for more details in Equation (4), please refer to Kupyn’s paper [20]).

LLLLG=0.5* p + 0.006* x + 0.01* adv (4)

where Lp is the similarity between two images, we use the SSIM (Structural Similarity)

[34] measurement; Lx denotes the perceptual loss, which computes the Euclidean loss on the VGG19 [35] feature maps to make the network pay more attention to learning cod-

ing perception and high-level semantic information; Ladv represents the adversarial loss, which contains both global and local discriminator losses (for more details in Equation (4), please refer to Kupyn’s paper [20]).

4. Experiments and Results 4.1. Preparation of Dataset Based on CODE V Simulation Datasets The optical aberrations are not the only factor of actual image degradation, which we will discuss later in the paper. The construction of simulation dataset based on CODE V software (Synopsys Corporate Headquarters, 690 East Middlefield Road Mountain View, CA, USA) is of great significance, which can provide reference for the evaluation of recovery algorithm, especially for the supervised learning method. CODE V software, a famous optical design and analysis tool, has the excellent simulation function for optical aberration. We build a dataset based on the simulation function Sensors 2021, 21, 3317 of CODE V. All the data sets used in this paper are from MIT Adobe FiveK dataset, which7 of 16 is mainly used in the research of tone adjustment of deep learning method [36]. The dataset consists of raw camera data and five groups of processed data tuned by professionals using Adobe professional image processing software. The dataset basically coverscovers the the natural natural scene, scene, artificial artificial scene, scene, character character photography, photography, and andother other areas. areas. This work This workselects selects a group a group of optimized of optimized data asdata the as original the original verification verification data of data the of system. the system. WeWe manually filter filter out the experimental pictures which show obvious defocusing images,images, and and finally finally select select 200 200 experimental experimental pictures pictures including including characters, characters, text text and and natural natural scenes.scenes. U Usingsing the the two two-dimensional-dimensional image image simulation simulation function function of of CODE CODE V, V, we take the clearclear images images as as inputs, inputs, and and obtain obtain the the simulation simulation degraded images. Part Part of of the the simulation simulation datasetdataset is is shown shown in in the the Figure 44..

Figure 4. Part of data set based on CODE V simulation. In this way, we obtain 200 groups of supervised samples with true value and optical degradation simulation data. Overall, 128 groups are selected as training dataset, 36 groups as validation dataset and 36 groups as test dataset. In the network training, we use the online random block strategy for all the comparison methods (except the optimization-based method). When we read the training samples, we randomly crop 256 × 256 blocks in the original image during the training. The epoch is set to 1000 rounds, that is, 128,000 training blocks are used in the network training process. The batch size is set to 8.

4.2. Algorithm Evaluation Based on CODE V Simulation Dataset Based on the CODE V simulation data set, we train our network, and obtain 34 M parameters after 1000 epochs of training, and ﬁnally get blurry removal results in test- dataset, part of them are shown in Figure5. Furthermore, we validate our proposed method against several existing supervised methods which are applicable in the current ﬁeld. The comparison methods include: the non-uniform aberration deblurring method [37], Unet-based restoring method [38], FOV- GAN restoring method [6], DeblurGAN-v2 restoring method [20]. We do not compare with previous simple lens restoring methods [4,5] owing to the complex calibration process. We choose a blind non-uniform aberration deblurring method as the representative of the Sensors 2021, 21, x FOR PEER REVIEW 8 of 17

Figure 4. Part of data set based on CODE V simulation.

In this way, we obtain 200 groups of supervised samples with true value and optical degradation simulation data. Overall, 128 groups are selected as training dataset, 36 groups as validation dataset and 36 groups as test dataset. In the network training, we use the online random block strategy for all the comparison methods (except the optimization-based method). When we read the training samples, we randomly crop 256 × 256 blocks in the original image during the training. The epoch is set to 1000 rounds, that is, 128,000 training blocks are used in the network training Sensors 2021, 21, 3317 process. The batch size is set to 8. 8 of 16

4.2. Algorithm Evaluation Based on CODE V Simulation Dataset Based on the CODE V simulation data set, we train our network, and obtain 34 M parametersoptimization-based after 1000 methods epochs [of37 ].training, Other restoringand finally methods get blurry are basedremoval on deepresults network, in test- which can achieve end-to-end image restoration. dataset, part of them are shown in Figure 5.

(a) (b) (c)

Figure 5. RRGRRG-GAN-GAN restoring restoring results results based based on on CODE CODE V Vsimulation simulation dataset: dataset: (a)( aBlurry) Blurry images images simulated simulated by CODE by CODE V; (b V;) (Theb) The restoring restoring results; results; (c) (Thec) The ground ground-truth.-truth.

Furthermore,The running environment we validate ofour this proposed work is: method Intel (R) against core (TM) several i7-6700 existing CPU @supervised 3.40GHz, methods32GB system which memory, are applicable NVIDIA in GeForce the current RTX field. 2080 The Ti with comparison 11,264 Mb methods GPU memory. include: Thethe nontraining-uniform parameters aberration are setdeblurring according method to the [37], original Unet paper,-based and restoring the training method epoch [38], are FOV all- set to 1000. The recovery results obtained by the above methods are shown in Figure6. GAN restoring method [6], DeblurGAN-v2 restoring method [20]. We do not compare Figure6b is the restoring results of optimization-based theory. Although the estimation with previous simple lens restoring methods [4,5] owing to the complex calibration pro- of blur kernel has been iterated for many times, it is still not accurate, and thus cannot achieve cess. We choose a blind non-uniform aberration deblurring method as the representative good restoration of degraded images. In addition, the running time is more than 20 min of the optimization-based methods [37]. Other restoring methods are based on deep net- on one test image with 1920 × 1080 resolution. Other optimization-based methods that work, which can achieve end-to-end image restoration. can be used in this computational imaging restoration, such as [39,40], the restoring results The running environment of this work is: Intel (R) core (TM) i7-6700 CPU @ 3.40GHz, and running time are similar to this method. Compared with the traditional methods, the 32GB system memory, NVIDIA GeForce RTX 2080 Ti with 11,264 Mb GPU memory. The restoring methods based on deep learning have a greater improvement in the restoration effect and efficiency. Figure6c shows the restoration results of Unet, and Figure6e shows the restoration results of Deblur-GAN-v2. These methods are not as good as our proposed method in the restoration of weeds, text and other complex details. Figure6d shows the restoration results implemented by FOV-GAN network, which produce some ringing effect in the complex detail texture. The proposed RRG-GAN network achieves better results than other deep-learning-based methods in the restoration of complex detail texture. The restored image edge is clear and natural, and there is no obvious ringing effect, which is more in line with human subjective visual effect. This is the role played by the fusion of attention module and feature module in a generative network. Sensors 2021, 21, x FOR PEER REVIEW 9 of 17

Sensors 2021, 21, 3317 9 of 16 training parameters are set according to the original paper, and the training epoch are all set to 1000. The recovery results obtained by the above methods are shown in Figure 6.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Figure 6. The results comparison using different restoring methods based on simulation dataset: (a) Original blurry images; (b) Non-uniform aberration deblurring results; (c) Unet restoring results; (d) FOV-GAN restoring results; (e) DeblurGAN-v2 restoring results; (f) Our proposed RRG-GAN restoring results; (g) The ground truth.

Since the images in the test dataset are obtained by CODE V simulation, there are true values that can be used for reference. Several objective evaluation indexes can be used to Sensors 2021, 21, 3317 10 of 16

evaluate the above algorithms: Structural SIMilarity (SSIM) [34], Peak Signal to Noise Ratio (PSNR), Root Mean Square Error (RMSE) [41], and Spectral Angle Mapper (SAM) [42]. The quantitative comparisons of restoring performance are shown in Table1. As can be seen from Table1, the objective evaluation index obtained by the RRG-GAN network method described in this section is superior to other recovery methods.

Table 1. Quantitative comparisons of restoring performance.

SSIM PSNR RMSE SAM Original 0.6278 24.4386 16.3857 4.5950 Multiscale 0.6859 25.0048 15.1276 4.7078 Unet 0.7922 28.3427 10.6450 4.1911 Fov-GAN 0.7486 25.2653 14.4563 5.6238 Deblur-GAN 0.7843 28.0252 10.9077 4.5637 RRG-GAN 0.8650 30.4102 8.3224 3.7855

The running time and the parameter-size of the above methods are summarized in Table2. Through the comparison, we ﬁnd that the running time of deep learning methods is greatly better than the traditional optimization-based method, but worse than other deep-learning-based methods. Compared with other deep-learning-based methods, our network structure is more complex, but the recovery effect is better, as shown in Figure6. Furthermore, the parameter size of our method is much smaller than other deep learning methods, which is determined by the recursive structure of the network.

Table 2. Statistics of running time and network parameter size.

MultiScale Unet FovGAN DeblurGAN-V2 RRG-GAN Runtime 25 min 0.489 s 0.714 s 2.677 s 2.738 s Sensors 2021, 21, x FOR PEER REVIEW 11 of 17 Para size / 363.701 MB 83.852 MB 238.982 MB 31.494 MB

4.3. Apllication in Real Scene 4.3.1.4.3. The Apllication Reason in ofReal Constructing Scene Manual Registering Dataset 4.3.1.Simulation The Reason evaluation of Constructing is an Manual ideal test Registering scenario, Dataset and its signiﬁcance is to provide a validationSimulation benchmark evaluation for algorithm is an ideal analysis.test scenario, However, and its significance when the single-convex-lensis to provide a is actuallyvalidation assembled benchmark for for imaging, algorithm the analysis. actual However, imaging when effect the is differentsingle-convex from-lens CODE is V’s simulation,actually assembled as shown for in imaging, Figure7 the. actual imaging effect is different from CODE V`s simulation, as shown in Figure 7.

(a) (b) (c) (d)

FigureFigure 7. Network 7. Network restoring restoring results results based based on on simulation simulation datasetdataset fails fails in in real real scene scene test: test: (a) Original (a) Original sharp sharp images; images; (b) Blurry (b) Blurry images simulated by CODE V; (c) The real blurry images directly captured by simple lens imaging system; (d) The restor- imagesing simulated results from by CODE (c) using V; the (c) parameters The real blurry trained images by CODE directly V simulation captured data by set simple. lens imaging system; (d) The restoring results from (c) using the parameters trained by CODE V simulation data set. From Figure 7, the results show that the network parameters trained by the simulation test set are not effective when deployed in the actual scene. The main reason for this problem is that the data distribution of the simulation dataset is inconsistent with the real scene. In addition to the conventional image blur caused by optical aberration, there are two additional degradation factors, color shift and bright background interference, which are not simulated in CODE V software. The color shift is caused by the white balance problem of CMOS camera. The bright background interference is caused by straylight. The single-convex-lens imaging system has no special aperture, and the gland ring plays the role of aperture, which makes the straylight easier to produce and has a direct impact on the image formation. Straylight will produce bright background, and further reduces the modulation transfer function (MTF) of the optical system. These two kinds of additional interferences cause the data distribution difference between the simulation dataset and the actual scene, we further need to reconstruct a dataset for the actual scene.

4.3.2. The Constructure of Manual Registering Datasets Similar to Peng [6], we built a display-capture lab setup as shown in Figure 8a. We display the 200 images in the dataset on an LCD device (52-Inch) and collect them through our single-convex-lens imaging system. The LCD device is placed 8 m away from the camera, and the intrinsic discretization of the LCD screen can be ignored at this distant. There is a large position error due to the mismatch between the original image and the captured image. Therefore, we specially write a MATLAB script program for interactive selection of artificial feature points to register two images, as shown in Figure 8b.

Sensors 2021, 21, 3317 11 of 16

From Figure7, the results show that the network parameters trained by the simulation test set are not effective when deployed in the actual scene. The main reason for this problem is that the data distribution of the simulation dataset is inconsistent with the real scene. In addition to the conventional image blur caused by optical aberration, there are two additional degradation factors, color shift and bright background interference, which are not simulated in CODE V software. The color shift is caused by the white balance problem of CMOS camera. The bright background interference is caused by straylight. The single-convex-lens imaging system has no special aperture, and the gland ring plays the role of aperture, which makes the straylight easier to produce and has a direct impact on the image formation. Straylight will produce bright background, and further reduces the modulation transfer function (MTF) of the optical system. These two kinds of additional interferences cause the data distribution difference between the simulation dataset and the actual scene, we further need to reconstruct a dataset for the actual scene.

4.3.2. The Constructure of Manual Registering Datasets Similar to Peng [6], we built a display-capture lab setup as shown in Figure8a. We display the 200 images in the dataset on an LCD device (52-Inch) and collect them through our single-convex-lens imaging system. The LCD device is placed 8 m away from the camera, and the intrinsic discretization of the LCD screen can be ignored at this distant. There is a large position error due to the mismatch between the original image and the Sensors 2021, 21, x FOR PEER REVIEW captured image. Therefore, we specially write a MATLAB script program for interactive12 of 17

selection of artiﬁcial feature points to register two images, as shown in Figure8b.

(a) (b)

FigureFigure 8. 8.TheThe key key procedure procedure constructing constructing manual manual registering registering database; database; (a) ( aThe) The display display-capture-capture lab lab setup;setup; (b) ( bThe) The interactive interactive selection selection of ofartificial artiﬁcial feature feature points points in inMATLAB MATLAB software. software.

InIn the the manual manual calibration calibration shown shown in inthe the Figure Figure 8b,8b, after after 10 10 feature feature point point pairs pairs are are se- se- lectedlected manually manually for for each each image image pair, pair, the affineaffine transformationtransformation parameters parameters can can be be calculated calcu- latedby usingby using the leastthe least square square method. method. It is foundIt is found that athat set ofa set fine of calibrated fine calibrated affine aff parametersine pa- rameterscan be appliedcan be applied to all images to all images due to thedue relatively to the relatively static acquisition static acquisition environment, environment, and there andis nothere need is no to calibrateneed to calibrate each image. each Therefore,image. Therefore, we obtained we obtained a dataset a containsdataset contains 200 “true 200value-picking “true value-picking up” registered up” registered samples. samples.

4.3.3.4.3.3. Algorithm Algorithm Evaluation Evaluation Based Based on on Manual Manual Registering Registering Datasets Datasets Again,Again, we we select select 128 128 groups groups as training as training datasets, datasets, 36 groups 36 groups as validation as validation datasets datasets and 36and groups 36 groups as test asdatasets. test datasets. After 1000 After epochs 1000 epochs of training of training process, process, the restoring the restoring images images are shownare shown in Figure in Figure 9. From9. Fromthe effect the effect of image of image restoration, restoration, we can we see can that see our that method our method can notcan only not eliminate only eliminate the optical the opticalblur, but blur, also but correct also correctthe color the deviation color deviation and bright and back- bright groundbackground interference. interference. Under the same experimental conditions, we test the effect of our proposed method and the other two deep learning methods on the manual registering dataset, as shown in the Figure 10. From the ﬁgure, we can see that our method is superior to the other two methods.

Figure 9. RRG-GAN restoring results based on manual dataset.

(a) (b)

Sensors 2021, 21, x FOR PEER REVIEW 12 of 17

(a) (b) Figure 8. The key procedure constructing manual registering database; (a) The display-capture lab setup; (b) The interactive selection of artificial feature points in MATLAB software. (a) (b) In the manual calibration shown in the Figure 8b, after 10 feature point pairs are se- Figure 8. The key procedure constructing manual registering database; (a) The display-capture lab setup;lected ( bmanually) The interactive for each selection image of pair, artificial the affinefeature transformation points in MATLAB parameters software. can be calculated by using the least square method. It is found that a set of fine calibrated affine pa- rametersIn the can manual be applied calibration to all images shown due in the to theFigure relatively 8b, after static 10 acquisitionfeature point environment, pairs are se- lectedand there manually is no needfor each to calibrate image pair, each the image. affine Therefore, transformation we obtained parameters a dataset can contains be calcu- lated200 “ bytrue using value the-picking least squareup” registered method. samples. It is found that a set of fine calibrated affine parameters can be applied to all images due to the relatively static acquisition environment, 4.3.3. Algorithm Evaluation Based on Manual Registering Datasets and there is no need to calibrate each image. Therefore, we obtained a dataset contains 200 “trueAgain, value we -selectpicking 128 up groups” registered as training samples. datasets, 36 groups as validation datasets and 36 groups as test datasets. After 1000 epochs of training process, the restoring images are 4.3.3.shown Algorithm in Figure Evaluation9. From the Based effect oofn imageManual restoration, Registering we Datasets can see that our method can not only eliminate the optical blur, but also correct the color deviation and bright back- Again, we select 128 groups as training datasets, 36 groups as validation datasets and ground interference. Sensors 2021, 21, 3317 36 groups as test datasets. After 1000 epochs of training process, the restoring images12 of are 16 shown in Figure 9. From the effect of image restoration, we can see that our method can not only eliminate the optical blur, but also correct the color deviation and bright background interference.

Figure 9. RRG-GAN restoring results based on manual dataset.

Under the same experimental conditions, we test the effect of our proposed method and the other two deep learning methods on the manual registering dataset, as shown in theFigure Figure 9. RRG 10.- GANFrom restoring the figure, results we basedcan see on that manual our dataset.method is superior to the other two Figure 9. RRG-GAN restoringmethods. results based on manual dataset. Under the same experimental conditions, we test the effect of our proposed method and the other two deep learning methods on the manual registering dataset, as shown in the Figure 10. From the figure, we can see that our method is superior to the other two methods.

Sensors 2021, 21, x FOR PEER REVIEW 13 of 17

(a) (b)

Sensors 2021, 21, x FOR PEER REVIEW 13 of 17

(a) (b)

(c) (d) Figure 10. The results comparison using different deep -learning methods based on manual dataset: (a) Original blurry Figureimages; 10. (b)The FOV results-GAN restoring comparison results; (c) using DeblurGAN different-v2 restoring deep-learning results; (d) Our methods proposed RRG based-GAN on restoring manual results dataset:. (a) Original blurry images; (b) FOV-GAN restoring results; (c) DeblurGAN-v2 restoring results;

(d) Our proposed RRG-GAN(c) Furthermore, restoring results.we apply our restoring method to real( dscenes,) the results are as shown in the Figure 11. Through the actual imaging experiments, it can be seen that after adopt- Figure 10. The results comparisoning usingthe manual different registering deep-learning dataset methods training, based ou ron proposed manual dataset:storing method(a) Original can blurrywell elimi- images; Furthermore,(b) FOV-GAN restoring wenate results; apply three (c) our DeblurGANtypical restoring image-v2 degradation restoring method results; effects to ( reald) Ourof single scenes, proposed-convex RRG the--lens,GAN results including restoring are results asthe showntexture. in the Figure 11. Throughblurry the caused actual by optical imaging aberrations, experiments, the color shift it can caused be by seen the thatCMOS after camera, adopting and the the manual registeringcontrastFurthermore, dataset reduction training, we causedapply our ourby the restoring proposed straylight. method storing to real method scenes, the can results well are eliminate as shown in the Figure 11. Through the actual imaging experiments, it can be seen that after adopt- three typical imageing degradation the manual registering effects ofdataset single-convex-lens, training, our proposed including storing method the texture can well blurry elimi- caused by optical aberrations,nate three typical the image color degradation shift caused effects by of the single CMOS-convex camera,-lens, including and the the contrast texture reduction caused byblurry the caused straylight. by optical aberrations, the color shift caused by the CMOS camera, and the contrast reduction caused by the straylight.

(a)

(b) Figure 11. The restoring results in real scenes: (a) Images directly captured by single-convex-lens imaging system; (b) The restoring images processed by our restoring method.

4.4. Imaging Effect Comparison with SLR Camera Lens

In this section, we compare(b) the restoring effect obtained by single-convex-lens imaging system with the direct imaging effect using conventional SLR camera lens. We stabi- FigureFigure 11. 11.The restoringThe restoring results liinze resultsreal the scenes: camera in ( reala )and Images scenes: capture directly (thea) captured scene Images by by these directly single two-convex lenses captured-lens separately. imaging by single-convex-lens system; The imaging (b) The effect restoring images processed by ourare restoringshown in method. Figure 12. imaging system; (b) The restoring images processed by our restoring method. 4.4. Imaging Effect Comparison with SLR Camera Lens In this section, we compare the restoring effect obtained by single-convex-lens imaging system with the direct imaging effect using conventional SLR camera lens. We stabilize the camera and capture the scene by these two lenses separately. The imaging effect are shown in Figure 12.

Sensors 2021, 21, 3317 13 of 16

4.4. Imaging Effect Comparison with SLR Camera Lens Sensors 2021, 21, x FOR PEER REVIEW 14 of 18 In this section, we compare the restoring effect obtained by single-convex-lens imaging system with the direct imaging effect using conventional SLR camera lens. We stabilize the camera and capture the scene by these two lenses separately. The imaging effect are shown in Figure 12.

(a) (b) (c)

FigureFigure 12. ImagingImaging effect effect comparison comparison with with SLR SLR camera camera lens: lens: ( (aa)) Images Images directly directly captured captured by by single-convex-lens imaging imaging system;system; ( (bb)) The The restoring restoring images images processe processedd by by our restoring method; (cc)) Images Images directly directly captured captured by by conventional SLR SLR cameracamera lens. lens. From Figure 12, we can see that the experimental images processed by our restoring From Figure 12, we can see that the experimental images processed by our restoring method have a good restoration effect for the edge information of the natural scene, and the method have a good restoration effect for the edge information of the natural scene, and restoring effect is close to the direct imaging effect using conventional SLR Lens. However, the restoring effect is close to the direct imaging effect using conventional SLR Lens. How- for the special texture such as text, fringe pattern, due to the lack of relevant sample set in ever,the database, for the special it cannot texture have such a good as text, detail fringe recovery pattern, effect, due which to the willlack beof improvedrelevant sample in the setfollowing in the database, work. it cannot have a good detail recovery effect, which will be improved in the following work. 5. Discussion 5.5.1. Discussion Improvement Analysis 5.1.5.1.1. Improvement Efﬁciency ImprovementAnalysis over Traditional Methods 5.1.1. DueEfficiency to the Improvemen indirect imagingt over mechanism, Traditional computationalMethods imaging is also faced with the problemsDue to the of indirect “accurate imaging description mechanism, of forward computational modulation transformationimaging is also model”faced with and the“robustness problems of of reverse “accurate reconstruction description algorithm”. of forward Withmodulation such problems, transformation especially model” the partic- and “robustnessularity of single-convex-lens’s of reverse reconstr illuction conditioned algorithm”. problem, With traditionalsuch problems, methods especially need tothe utilize par- ticularitymany iterations of single-convex-lens’s to get the accurate ill conditioned description problem, of forward traditional modulation methods transformation need to uti- lizemodel, many while iterations requiring to get other the iterationsaccurate desc to ensureription the of forward robustness modulation of reverse transformation reconstruction model,algorithm. while However, requiring the other deep-learning-based iterations to ensure network the robustness can avoid of the reverse above reconstruction problems in an algorithm.end-to-end However, way, and efﬁcientlythe deep-learning-base restore the latentd network images can from avoid the blurrythe above source problems data. in an end-to-end way, and efficiently restore the latent images from the blurry source data. 5.1.2. Effect Improvement over Deep Learning Methods 5.1.2. ComparedEffect Improvement with other Over restoring Deep Learning networks Methods like U-net [38], our proposed RRG-GAN network can achieve better recovery effect. This is determined by the adversarial training Compared with other restoring networks like U-net [38], our proposed RRG-GAN process of the generator and discriminator. During training, the generator is trained to networkproduce can latent achieve images better which recovery can “fool” effect. the This discriminator is determined network, by the and adversarial the discriminator training process of the generator and discriminator. During training, the generator is trained to produce latent images which can “fool” the discriminator network, and the discriminator is trained to better distinguish the images captured by the single-convex-lens and the latent images generated from the generator. Our RRG-GAN network can achieve better detail recovery effect than other GAN networks like Deblur-GAN [20]. This is due to the use of multi-scale dual attention blocks

Sensors 2021, 21, 3317 14 of 16

is trained to better distinguish the images captured by the single-convex-lens and the latent images generated from the generator. Our RRG-GAN network can achieve better detail recovery effect than other GAN networks like Deblur-GAN [20]. This is due to the use of multi-scale dual attention blocks module, which can simulate the process of human eyes searching for important information in the visual scene for centralized analysis while ignoring irrelevant information in the scene. The selective kernel networks are used combined with the dual attention blocks. They can effectively learn to fuse important features in multi-scale. The residual resizing networks provide additional network parameters for learning in the inter-scale up-downsampling operation, so as to further improve the performance of the whole network.

5.2. Limitations and Deﬁciencies Like most deep learning methods, the data-driven-based methods require the consistency of the distribution of training data and actual data. However, the mechanism of computational imaging system determines that it is difficult to obtain the ideal training data which is consistent with the actual data. Through the experimental results in Section 2.3, we ﬁnd that although the color restoration is greatly improved compared with the original image, it still retains a certain color offset, which is caused by the background light of LCD display. Furthermore, small errors are inevitable when we manually register the dataset. Although our RRG-GAN network can overcome this small error, it will still affect the performance of network. Therefore, how to design the training dataset more reasonably is also one of our further works.

5.3. Further Work Aberration is the core problem of optical design. The aberration correction method based on computational imaging has an important significance in other optical systems. Especially in some special optical imaging scenes, under the premise of limited cost and limited optical lens size, the traditional optical design method is difficult to meet the design needs. In this case, it is a very feasible scheme to allow optical designers to reduce optical index and keep proper aberrations, and deliver aberration correction work to calculation and restoration algorithm. Our following work will try to apply our restoring method to complex optical systems, such as reflective optical systems, to achieve the joint optimization of optical design and restoration algorithm design, so as to achieve a higher index of computational imaging system design.

6. Conclusions In this paper, we propose a novel RRG-GAN network to restore the latent images from blurry source data captured by a single-convex-lens imaging system in an end-to-end way. We restructure the generator network, which includes a dual attention module, selective kernel network module, and a residual resizing module. These improvements make RRG-GAN more suitable for the non-uniform deblurring task. We collect sharp/aberration- degraded datasets by CODE V simulation and manual registering. Relevant experimental comparisons and actual tests verify the effectiveness of our proposed method.

Author Contributions: Conceptualization, X.W.; validation, X.W., J.L. and G.Z.; methodology, X.W., J.L. and G.Z.; hardware and software, B.L.; review and editing, X.W., Q.L. and H.Y.; funding acquisition, H.Y. All authors have read and agreed to the published version of the manuscript. Funding: This research was funded by the Chinese Academy of Sciences-Youth Innovation Promotion Association, grant number 2020220. Institutional Review Board Statement: Not applicable. Informed Consent Statement: All subjects gave their informed consent for inclusion before they participated in the study. Participants were allowed to use their personal portrait rights in the paper. Sensors 2021, 21, 3317 15 of 16

Data Availability Statement: The data presented in this study are openly available in Github at (https://github.com/wuzeping1893/RRG-GAN-single-convex-lens). Acknowledgments: The authors would like to thank the anonymous reviewers for their construc- tive comments. Conﬂicts of Interest: The authors declare no conﬂict of interest.

Abbreviations

RRG Recursive Residual Groups GAN Generative Adversarial Networks CNN Convolutional Neural Networks RNN Recurrent Neural Networks CMOS Complementary Metal Oxide Semiconductor PSF Point Spread Function MRB Multi-scale Residual Block MAP Maximum A Posteriori Approach VEM Variational Expectation Maximization MTF Modulation Transfer Function LCD Liquid Crystal Display SLR Single-convex-lens Reﬂex

References 1. Koppal, S.J. A Survey of Computational Photography in the Small: Creating intelligent cameras for the next wave of miniature devices. IEEE Signal Process. Mag. 2016, 33, 16–22. [CrossRef] 2. Courtier, G.; Lapray, P.-J.; Thomas, J.-B.; Farup, I. Correlations in Joint Spectral and Polarization Imaging. Sensors 2021, 21, 6. [CrossRef][PubMed] 3. Cao, J.; Zhou, D.; Zhang, F.; Cui, H.; Zhang, Y.; Hao, Q. A Novel Approach of Parallel Retina-Like Computational Ghost Imaging. Sensors 2020, 20, 7093. [CrossRef][PubMed] 4. Schuler, C.J.; Hirsch, M.; Harmeling, S. Non-stationary correction of optical aberrations. In Proceedings of the IEEE International Conference on Computer Vision, Barcelona, Spain, 6–13 November 2011; pp. 659–666. 5. Heide, F.; Rouf, M.; Hullin, M.B.; Labitzke, B.; Heidrich, W.; Kolb, A. High quality computational imaging through simple lenses. ACM Trans. Graph. 2013, 32, 149. [CrossRef] 6. Peng, Y.; Sun, Q.; Dun, X.; Wetzstein, G.; Heidrich, W.; Heide, F. Learned large field-of-view imaging with thin-plate optics. ACM Trans. Graph. 2019, 38, 1–14. [CrossRef] 7. Dowski, E.R.; Cathey, W.T. Extended depth of field through wave-front coding. Appl. Opt. 1995, 34, 1859–1866. [CrossRef] [PubMed] 8. Peng, Y.; Fu, Q.; Heide, F.; Heidrich, W. The diffractive achromat full spectrum computational imaging with diffractive optics. ACM Trans. Graph. 2016, 35, 31. [CrossRef] 9. Xu, L.; Zheng, S.; Jia, J. Unnatural l0 sparse representation for natural image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, 25–27 June 2013; pp. 1107–1114. 10. Fergus, R.; Singh, B.; Hertzmann, A.; Roweis, S.T.; Freeman, W.T. Removing camera shake from a single photograph. ACM Trans. Graph. 2006, 25, 787–794. [CrossRef] 11. Levin, A.; Weiss, Y.; Durand, F.; Freeman, W.T. Understanding and evaluating blind deconvolution algorithms. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009; pp. 1964–1971. 12. Pan, J.; Sun, D.; Pfister, H.; Yang, M.H. Blind Image Deblurring Using Dark Channel Prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 26 June–1 July 2016; pp. 1628–1636. 13. Lai, W.S.; Ding, J.J.; Lin, Y.Y.; Chuang, Y.Y. Blur kernel estimation using normalized color-line prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 8–10 June 2015; pp. 64–72. 14. Chakrabarti, A. A neural approach to blind motion deblurring. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 8–16 October 2016; pp. 221–235. 15. Wang, R.; Ma, G.; Qin, Q.; Shi, Q.; Huang, J. Blind UAV Images Deblurring Based on Discriminative Networks. Sensors 2018, 18, 2874. [CrossRef][PubMed] 16. Ren, D.; Zhang, K.; Wang, Q.; Hu, Q.; Zuo, W. Neural Blind Deconvolution Using Deep Priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 3338–3347. 17. Nah, S.; Hyun Kim, T.; Mu Lee, K. Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, HI, USA, 22–25 July 2017; pp. 257–265. Sensors 2021, 21, 3317 16 of 16

18. Zhang, J.; Pan, J.; Ren, J.; Song, Y.; Bao, L.; Lau, R.W.; Yang, M.H. Dynamic scene deblurring using spatially variant recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 2521–2529. 19. Zhou, Q.; Ding, M.; Zhang, X. Image Deblurring Using Multi-Stream Bottom-Top-Bottom Attention Network and Global Information-Based Fusion and Reconstruction Network. Sensors 2020, 20, 3724. [CrossRef][PubMed] 20. Kupyn, O.; Martyniuk, T.; Wu, J.; Wang, Z. Deblurgan-v2: Deblurring (orders-of-magnitude) faster and better. In Proceedings of the IEEE International Conference on Computer Vision, Seoul, Korea, 27 October–2 November 2019; pp. 8878–8887. 21. Vantaggiato, E.; Paladini, E.; Bougourzi, F.; Distante, C.; Hadid, A.; Taleb-Ahmed, A. COVID-19 Recognition Using Ensemble- CNNs in Two New Chest X-ray Databases. Sensors 2021, 21, 1742. [CrossRef][PubMed] 22. Garillos-Manliguez, C.A.; Chiang, J.Y. Multimodal Deep Learning and Visible-Light and Hyperspectral Imaging for Fruit Maturity Estimation. Sensors 2021, 21, 1288. [CrossRef][PubMed] 23. Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Bengio, Y. Generative Adversarial Nets. In Proceedings of the Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 2672–2680. 24. Chen, J.; Wu, C.; Chen, H.; Cheng, P. Unsupervised Dark-Channel Attention-Guided CycleGAN for Single-Image Dehazing. Sensors 2020, 20, 6000. [CrossRef][PubMed] 25. Li, C.; He, K.; Liu, K.; Ma, X. Image Inpainting Using Two-Stage Loss Function and Global and Local Markovian Discriminators. Sensors 2020, 20, 6193. [CrossRef][PubMed] 26. Zamir, S.W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F.S.; Yang, M.H.; Shao, L. CycleISP: Real Image Restoration via Improved Data Synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 14–19 June 2020; pp. 2693–2702. 27. Isola, P.; Zhu, J.Y.; Zhou, T.; Efros, A.A. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, HI, USA, 21–26 July 2017; pp. 5967–5976. 28. Mao, X.; Li, Q.; Xie, H.; Lau, R.Y.; Wang, Z.; Paul Smolley, S. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2813–2821. 29. Tai, Y.; Yang, J.; Liu, X. Image Super-Resolution via Deep Recursive Residual Network. In Proceedings of the IEEE Conference on Computer Vision & Pattern Recognition, Hawaii, HI, USA, 21–26 July 2017; pp. 2790–2798. 30. Zamir, S.W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F.S.; Yang, M.H.; Shao, L. Learning Enriched Features for Real Image Restoration and Enhancement. In Proceedings of the Europeon Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; pp. 492–511. 31. Woo, S.; Park, J.; Lee, J.Y.; Kweon, I.S. CBAM: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision, Munich, Germany, 8–14 September 2018; pp. 3–19. 32. Li, X.; Wang, W.; Hu, X.; Yang, J. Selective Kernel Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 510–519. 33. Sim, H.; Kim, M. A Deep Motion Deblurring Network Based on Per-Pixel Adaptive Kernels with Residual Down-Up and Up-Down Modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 16–20 June 2019; pp. 2140–2149. 34. Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [CrossRef][PubMed] 35. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. 36. Bychkovsky, V.; Paris, S.; Chan, E.; Durand, F. Learning Photographic Global Tonal Adjustment with a Database of Input/Output Image Pairs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, USA, 20–25 June 2011; pp. 97–104. 37. Wu, X.; Yang, H.; Liu, B.; Liu, X. Non-Uniform Deblurring for Simple Lenses Imaging System. In Proceedings of the 2020 3rd International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE), Shenzhen, China, 24–26 April 2020; pp. 274–278. 38. Chen, C.; Chen, Q.; Xu, J.; Koltun, V. Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3291–3300. 39. Xu, L.; Jia, J. Two-phase kernel estimation for robust motion deblurring. In Proceedings of the European Conference on Computer Vision, Heraklion, Greece, 5–11 September 2010; pp. 157–170. 40. Pan, J.; Sun, D.; Pfister, H.; Yang, M.H. Deblurring Images via Dark Channel Prior. IEEE Trans. Pattern Anal. Mach. Intell. 2018, 40, 2315–2328. [CrossRef][PubMed] 41. Willmott, C.J.; Matsuura, K. On the use of dimensioned measures of error to evaluate the performance of spatial interpolators. Int. J. Geogr. Inf. Sci. 2006, 20, 89–102. [CrossRef] 42. Conel, J.E.; Hoover, G.; Nolin, A.; Alley, R.; Margolis, J. Emperical Relationships Among SemiArid Landscape Endmembers Using the Spectral Angle Mapper (SAM) Algorithm. In Summaries of the 4nd Annual JPL Airborne Geoscience Workshop, JPL Pub-92-14, AVIRIS Workshop; Jet Propulsion Laboratory: Pasadena, CA, USA, 1992; pp. 150–151.