CODED APERTURE STEREO For Extension of Depth of Field and Refocusing Yuichi Takeda1, Shnisaku Hiura2 and Kosuke Sato1 1Graduate School of Engineering Science, Osaka University, 1-3 Machikaneyama-cho, Toyonaka, 560-8531 Osaka, Japan 2Graduate School of Information Sciences, Hiroshima City University, Ozuka-Higashi, Asa-Minami-Ku, 731-3194 Hiroshima, Japan Keywords: Computational Photography, Coded Aperture, Stereo Depth Estimation, Deblur, Depth of Field, Refocusing. Abstract: Image acquisition techniques using coded apertures have been intensively investigated to improve the perfor- mance of image deblurring and depth estimation. Generally, estimation of the scene depth is a key issue in the recovery of optical blur because the size of the blur kernel varies according to the depth. However, since it is hard to estimate the depth of a scene with a single image, most successful methods use several images with different optical parameters captured by a specially developed camera with expensive internal optics. On the other hand, a stereo camera configuration is widely used to obtain the depth map of a scene. Therefore, in this paper, we propose a method for deblurring and depth estimation using a stereo camera with coded apertures. Our system configuration offers several advantages. First, coded apertures make not only deconvolution but also stereo matching very robust, because the loss of high spatial frequency domain information in the blurred image is well suppressed. Second, the size of the blur kernel is linear with the disparity of the stereo images, making calibration of the system very easy. The proof of this linearity is given in this paper together with several experimental results showing the advantages of our method. 1 INTRODUCTION ced(Levin et al., 2007). However, if the prior is not suitable for the scene, simultaneous estimation of the Optical blur (defocusing) and motion blur are typical PSF and latent image will fail. Therefore, multiple errors in photographs taken in dark environments. If images with different optical parameters such as fo- the aperture of the lens is stopped down to extend the cus distance or aperture shape are commonly used depth of field, moving objects are likely to be blurred to make the problem solvable(Hiura and Matsuyama, with a slow shutter speed to keep the exposure value 1998; Zhou et al., 2009). In this paper, we explore the constant. In other words, insufficient light from the use of disparity introduced by a stereo camera pair to scene causes a trade-off between depth of field and determine the size of the PSF. Contrary to sequential exposure time, and thus it is difficult to take a clear capture with different optical settings, stereo cameras sharp picture of a moving object in a dark scene using can deal with moving scenes. To determine the size a hand-held camera. Consequently, post processing of of the PSF, we need to calibrate the relationship be- captured images including deblurring and denoising, tween disparity and focus. In this paper, we present a have been intensively studied in prior years. In this proof showing that the diameter of the blur kernel is paper, we propose a method that extends the depth directly proportional to the relative disparity from the of field computationally using a stereo camera with focus distance. coded apertures. The other important factor in the preciseness of Typically, the performance of deblurring depends deblurring is the characteristics of the PSF. If a nor- on the knowledge about the point spread function mal circular aperture is used, information in the high (PSF). If the shape of the PSF is not known, it should spatial frequency domain is almost lost (Veeraragha- be estimated during deconvolution. This technique, van et al., 2007). Therefore, we introduce coded aper- called blind deconvolution, is generally an ill-posed tures to two lenses of the stereo camera to retain the problem when dealing with only a single image, and information of the scene as far as possible. The spe- some kind of priors about the scene must be introdu- cial aperture also makes the stereo matching robust, Takeda Y., Hiura S. and Sato K.. 103 CODED APERTURE STEREO - For Extension of Depth of Field and Refocusing. DOI: 10.5220/0003818801030111 In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP-2012), pages 103-111 ISBN: 978-989-8565-03-7 Copyright c 2012 SCITEPRESS (Science and Technology Publications, Lda.) VISAPP 2012 - International Conference on Computer Vision Theory and Applications kernel size [pixel] 100 120 140 20 40 60 80 0 0 Object Conversion of Disparity to Blur Kernel Size Disparity! 50 Stereo Matching disparity [pixel] 100 Aperture Pattern! Right input image! Disparity Map! 150 Lens Scaling of Blur Kernel 200 Coded Wiener Apertures Left input image! Deconvolution On the other hand, the circular (conventional) aperture is Image Sensors Integration! close to optimal when the noise level is very high. While (weighted sum) All-in-focus Image! there are different optimal apertures for different levels of Window Function! image noise, we may want a single aperture to use in a va- Circular Annular Multi-Annular Random Figure 1: Process flow of proposed method. riety of imaging conditions. In this case, we could pick the optimized pattern for σ =0.001 as it performed well over a and the clear scene is stably recovered using the pre- designed coded apertures. Desirable aperture shapes wide range of noise levels (from σ =0.0001 to 0.01). cisely determined PSF. have been explored by Zhou et al. (Zhou and Nayar, Related work is briefly summarized in Section 2. 2009) and Levin et al. (Levin, 2010). It is interesting to note that the image pattern (Lena) An overview of the proposed system and the algo- MURA Image Pattern Levin Veeraraghavan also produces deblurring results of fairly high quality. We rithms is presented in Section 3, together with a proof believe this is because the power spectrum of the image of the linear relationship between blur size and dis- pattern follows the 1/f law–it successfully avoids zero- parity. Experiments are discussed in Section 4, while crossings and, at the same time, has a heavy tail covering the Section 5 concludes the paper. high frequencies. Unfortunately, the image pattern consists ed =0.0001 =0.001 =0.002 =0.005 of a lot of small features, which introduce strong diffraction pos effects. We believe that it is for this reason that the image 2 RELATED WORK Pro pattern did not achieve as high quality results in our experi- ments as predicted by our simulations. Optical blur (defocus) appearing in images captured with conventional cameras is modeled as the convo- =0.008 =0.01 =0.02 =0.03 lution of the sharp original scene and a blur kernel 7. Experiments with Real Apertures (i.e., the PSF). Thus, the latent image can be recon-Figure 3.FigureAll the 2: Shapes aperture of Zhou’s patterns codes we for used various in our noise simulations. levels, As shown in Figure 4(a), we printed our optimized aper- structed by applying an inverse filter or deconvolutionTop twos rows:(Zhou Eight and Nayar, patterns, 2009). including circular, annular, multi- ture patterns as well as several other patterns as a single high techniques to the captured image. Several methodsannular, random, MURA, image pattern, Levin et al.’s pattern [13], resolution (1 micron) photomask sheet. To experiment with including Richardson-Lucy deconvolution (Richard-and VeeraraghavanSince the et al.’s size pattern of the blur [3]. kernelBottom varies two rows: withthe Eight of a specific aperture pattern, we cut it out of the photomask son, 1972) and MAP estimation (Lam and Goodman,our patternsdistance optimized between for thenoise camera levels and from object,σ =0 it.0001 is neces-to 0.03. 1 2000) have been proposed; however, the performance sary to estimate the depth of the captured scene accu- sheet and inserted it into a Canon EF 50mm f/1.8 lens . of the reconstruction depends greatly on the correct-have usedrately. it in Depth all of estimation our comparisons through the and optical experiments. defocus It In Figure 4(b), we show 4 lenses with different apertures ness of the blur kernel. In particular, a circular blurmust beeffect noted is that called similar Depth algorithmsfrom Defocus, have and beena number advocated of (image pattern, Levin et al.’s pattern, Veeraraghavan et al’s kernel in a conventional aperture contains many zeroin the paststudies (see on [19 this] for aspect example). have been carried out (Schech- pattern, and one of our optimized patterns) inserted in them, crossings in the spatial frequency, while reconstruc- ner and Kiryati, 2000). In general, it is not easy to and one unmodified (circular aperture) lens. Images of real tion at a frequency of low gain is unstable with much estimate the depth of a scene using a single image, be- noise influence. Therefore, the idea of designing6. an Performancecause simultaneous Comparison estimation of of the Apertures blur kernel and scenes were captured by attaching these lenses to a Canon aperture shape with the desirable spatial frequencyBeforelatent conducting image is under real constrained. experiments, Therefore, we first Levin performed et EOS 20D camera. As previously mentioned, we choose the characteristics was proposed. Hiura et al. introducedextensiveal. simulations (Levin et al., to 2007) verify assumed our aperture a Gaussian evaluation prior to cri- pattern which is optimized for σ =0.001, as it performs a coded aperture to improve the preciseness of depthterion andthe optimization edge power histogram algorithm. to make For the this, problem we used well- the 16 well over a wide range of noise levels in the simulation. estimation using the defocus phenomenon (Hiura and conditioned.