1 Learning Deep Convolutional Networks for Demosaicing Nai-Sheng Syu∗, Yu-Sheng Chen∗, Yung-Yu Chuang Abstract—This paper presents a comprehensive study of ap- for reducing visual artifacts as much as possible. However, plying the convolutional neural network (CNN) to solving the most researches only focus on one of them. demosaicing problem. The paper presents two CNN models that The Bayer filter is the most popular CFA [5] and has been learn end-to-end mappings between the mosaic samples and the original image patches with full information. In the case the widely used in both academic researches and real camera Bayer color filter array (CFA) is used, an evaluation on popular manufacturing. It samples the green channel with a quincunx benchmarks confirms that the data-driven, automatically learned grid while sampling red and blue channels by a rectangular features by the CNN models are very effective and our best grid. The higher sampling rate for the green component is con- proposed CNN model outperforms the current state-of-the-art sidered consistent with the human visual system. Most demo- algorithms. Experiments show that the proposed CNN models can perform equally well in both the sRGB space and the saicing algorithms are designed specifically for the Bayer CFA. linear space. It is also demonstrated that the CNN model can They can be roughly divided into two groups, interpolation- perform joint denoising and demosaicing. The CNN model is based methods [6], [7], [8], [9], [10], [11], [12], [13], [14] very flexible and can be easily adopted for demosaicing with any and dictionary-based methods [15], [16]. The interpolation- CFA design. We train CNN models for demosaicing with three based methods usually adopt observations of local properties different CFAs and obtain better results than existing methods. With the great flexibility to be coupled with any CFA, we present and exploit the correlation among wavelengths. However, the the first data-driven joint optimization of the CFA design and handcrafted features extracted by observations have limitations the demosaicing method using CNN. Experiments show that the and often fail to reconstruct complicated structures. Although combination of the automatically discovered CFA pattern and iterative and adaptive schemes could improve demosaicing re- the automatically devised demosaicing method outperforms other sults, they have limitations and introduce more computational patterns and demosaicing methods. Visual comparisons confirm that the proposed methods reduce more visual artifacts. Finally, overhead. Dictionary-based approaches treat demosaicing as a we show that the CNN model is also effective for the more general problem of reconstructing patches from a dictionary of learned demosaicing problem with spatially varying exposure and color base patches. Since the dictionary is learned, it can represent and can be used for taking images of higher dynamic ranges with the distribution of local image patches more faithfully and a single shot. The proposed models and the thorough experiments provide better color fidelity of the reconstructed images. together demonstrate that CNN is an effective and versatile tool for solving the demosaicing problem. However, the online optimization for reconstruction often takes much longer time, making such methods less practical. Index Terms—Convolutional neural network, demosaicing, Despite its practical use for decades, researches showed color filter array (CFA). the Bayer CFA has poor properties in the frequency-domain analysis [17]. Thus, some efforts have been put in proposing I. INTRODUCTION better CFA designs for improving color fidelity of the demo- OST digital cameras contain sensor arrays covered saiced images [18], [19], [20]. Earlier work mainly focused by color filter arrays (CFAs), mosaics of tiny color on altering the arrangement of RGB elements to get better de- M mosaicing results in terms of some handcrafted criteria. Some arXiv:1802.03769v1 [cs.CV] 11 Feb 2018 filters. Each pixel sensor therefore only records partial spectral information about the corresponding pixel. Demosaicing, a also explored color filters other than primary colors. Inspired process of inferring the missing information for each pixel, by the frequency representation of mosaiced images [17], plays an important role to reconstruct high-quality full-color several theoretically grounded CFA designs have been pro- images [2], [3], [4]. Since demosaicing involves prediction posed [18], [19]. They however involve considerable human of missing information, there are inevitably errors, leading to effort. Recently, automatic methods for generating CFAs have visual artifacts in the reconstructed image. Common artifacts been proposed by exploiting the frequency structure, a matrix include the zipper effects and the false color artifacts. The recording all the luminance and chrominance components of former refers to abrupt or unnatural changes of intensities over given mosaiced images [20]. However, although theoretically neighboring pixels while the later is for the spurious colors better, most of these CFAs can only reach similar perfor- that are not present in original image. In principle, the CFA mances as the state-of-the-art demosaicing methods with the design and the demosaicing method should be devised jointly Bayer CFA. The main reason is that the more complicated CFA designs require effective demosaicing methods to fully release ∗Nai-Sheng Syu and Yu-Sheng Chen contributed equally to this work. This their potential. Unfortunately, due to the complex designs, work is based on Nai-Sheng Syu’s master thesis [1]. All authors are with such demosaicing methods are more difficult to devise and, the Department of Computer Science and Information Engineering, National Taiwan University, Taipei, Taiwan, 106. E-mail: vm3465s939873 j nothinglo compared with the Bayer CFA, less efforts have been put into j [email protected]. developing demosaicing methods for these CFAs. 2 We address these issues by exploring the convolutional Bai et al. [20] each introduced a pattern design algorithm neural network (CNN). Because of breakthroughs in theory based on the frequency structure proposed by Li et al. [24]. and improvements on hardware, recently CNNs have shown Hao et al. [19] formulated the CFA design problem as a con- promises for solving many problems, such as visual recog- strained optimization problem and solved it with a geometric nition, image enhancement and game playing. By learning method. Later, Bai et al. [20] introduced an automatic pattern through data, the network automatically learns appropriate fea- design process by utilizing a multi-objective optimization tures for the target applications. We first address the demosaic- approach which first proposes frequency structure candidates ing problem with the popular Bayer CFA (Section III). Inspired and then optimizes parameters for each candidate. by CNN models for super-resolution [21], [22], we present two General demosaicing. In addition to colors, other properties CNN models, DeMosaicing Convolutional Neural Network of light, such as exposures (spatially varying exposure, SVE) (DMCNN, Section III-A) and Very Deep DMCNN (DMCNN- and polarization, could also be embedded into the filter array VD, Section III-B), for demosaicing. In contrast with hand- and more general demosaicing algorithms can be used for crafted features/rules by many interpolation-based methods, recovering the missing information. Nayar et al. [25] pro- the CNN models automatically extract useful features and posed a general demosaicing framework, Assorted Pixel, by captures high-level relationships among samples. Experiments assuming the demosaiced result can be obtained from an n- show that the CNN-based methods outperforms the state-of- degree polynomial function of neighboring mosaiced pixels. the-art methods in both the sRGB space (Section III-C) and the The whole process can therefore be thought as a regression linear space (Section III-D). In addition, they could perform problem by solving a linear system. Yasuma et al. [26] denoising and demosaicing simultaneously if providing proper later proposed a more general pattern, Generalized Assorted training data. We next show that the CNN-based methods can Pixel, with the capability to recover monochrome, RGB, high be easily adopted for demosaicing with CFA designs other dynamic range (HDR), multi-spectral images while sacrific- than the Bayer one (Section IV). The data-driven optimization ing spatial resolutions. We adopt a similar spatially varying approach makes it easy to train the CNN-based demosaicing exposure and color (SVEC) setting as Nayar et al. [25] to methods with different CFAs and outperform existing methods demonstrate the potential of the CNN-based methods for (Section IV-A). With its flexibility to be used with any CFA, generalized demosaicing. we present the first data-driven method for joint optimization Convolution neural networks. To date, deep learning based of the CFA design and the demosaicing method (Section IV-B). approaches have dominated many high-level and low-level vi- Finally, we demonstrate that the CNN-based method can also sion problems. Krizhevsky et al. [27] showed the deep CNN is be applied to solving a more challenging demosaicing problem very effective for the object classification problem. In addition where the filter array has spatially varying exposure and to high-level vision problems, CNN is also found effective in color (Section IV-C). It enables taking images