Video and Image Bayesian Demosaicing with a Two Color Image Prior Eric P. Bennett1, Matthew Uyttendaele2,C.LawrenceZitnick2, Richard Szeliski2, and Sing Bing Kang2 1 The University of North Carolina at Chapel Hill, Chapel Hill, NC 2 Microsoft Research, Redmond, WA Abstract. The demosaicing process converts single-CCD color repre- sentations of one color channel per pixel into full per-pixel RGB. We introduce a Bayesian technique for demosaicing Bayer color filter array patterns that is based on a statistically-obtained two color per-pixel im- age prior. By modeling all local color behavior as a linear combination of two fully specified RGB triples, we avoid color fringing artifacts while preserving sharp edges. Our grid-less, floating-point pixel location archi- tecture can process both single images and multiple images from video within the same framework, with multiple images providing denser color samples and therefore better color reproduction with reduced aliasing. An initial clustering is performed to determine the underlying local two color model surrounding each pixel. Using a product of Gaussians statis- tical model, the underlying linear blending ratio of the two representative colors at each pixel is estimated, while simultaneously providing noise re- duction. Finally, we show that by sampling the image model at a finer resolution than the source images during reconstruction, our continuous demosaicing technique can super-resolve in a single step. 1 Introduction Most digital cameras use a single sensor to record images and video. They use color filter arrays (CFAs) to capture one color band per pixel, and interpolate colors to produce full RGB per pixel. This interpolation process is known as demosaicing. The Bayer filter is the most popular type of CFA used today. Demosaicing a raw Bayer image requires an underlying image model to guide decisions for reconstructing the missing color channels. At every pixel only one color chan- nel is sampled, so we must use that information, combined with that of nearby samples, to reconstruct plausible RGB triples. An image model provides a prior for reconstructing the missing colors based on patterns of the surrounding sam- ples. Demosaicing algorithms differ in how local spatial changes in a single color channel are used to propagate information to the other channels. Demosaicing is inherently underspecified because there are no complete RGB triples anywhere in the image to learn an image specific prior from. Even worse, Research performed while an intern at Microsoft Research, Redmond. A. Leonardis, H. Bischof, and A. Pinz (Eds.): ECCV 2006, Part I, LNCS 3951, pp. 508–521, 2006. c Springer-Verlag Berlin Heidelberg 2006 Video and Image Bayesian Demosaicing with a Two Color Image Prior 509 if every pixel in an image has a random ratio of red, green, and blue, there is no hope of reconstructing the image. Only by assuming some local coherence between channels can we reasonably reconstruct the original image. In this paper, we reduce the problem’s complexity by developing an under- lying statistical image model that treats all colors in a local area as a linear combination of no more than two representative colors. To use this model, we estimate the two representative colors for the local area centered at each pixel and find the linear blending for each pixel that determines its color. Our two color model is motivated by the need to reconstruct accurate colors at edges. Current demosaicing algorithms can accurately reconstruct colors in image areas where only low frequencies are present. However, yellow or purple color fringing can appear at high frequency edges due to the edges in multi- ple color channels not being aligned in the reconstruction. By constraining the system to interpolate between fully specified RGB colors, there is less risk of misalignment. This constraint also provides noise reduction in smooth areas. The underlying two colors at each pixel are estimated using K-Means cluster- ing. The RGB colors used for clustering can come from any existing demosaicing algorithm. The final color at each pixel results from discovering the proper linear blending coefficient between the two representative colors. Based on knowledge of a small set of CFA samples around each pixel, our prob- lem is posed using Bayesian probabilities. Stating the problem statistically allows the model to include non-grid-aligned samples from multiple images or tempo- rally adjacent video frames to increase color accuracy. Also, by sampling the de- mosaicer’s output at an increased resolution, information from these additional samples exposes details between pixels, providing super-resolution in a single step. 2 Previous Work There are many approaches to demosaicing. A simple technique for demosaicing a Bayer color filter array [1] (shown in Figure 1) is bilinear interpolation, which is able to reconstruct smooth and smoothly varying image areas. At the edges, bilinear interpolation risks creating aliasing or “zippering” artifacts where every other pixel along an edge alternates between being considered on or off the edge. Color fringing is the other significant artifact, where yellows, purples, and cyans appear along or on sharp edges. These artifacts result from bilinear interpolation incorrectly placing an edge in a color channel one pixel offset from the same edge in a different channel. R G R G R G G B G B G B R G R G R G G B G B G B Fig. 1. The Bayer color filter array pattern 510 E.P. Bennett et al. Solving the color fringing and zippering issues was the focus of much sub- sequent research. One approach is to bilinearly interpolate the green chan- nel and then interpolate the red:green and blue:green ratios for the remaining samples [2]. This assumes there exists no green detail smaller than two pixels and that red and green locally vary in a fixed ratio with green. Median in- terpolation [3] assumes that bilinear interpolation can be repaired by median filtering the red-green and blue-green spaces. Both of these methods target fringing artifacts but can result in over-smoothing. Comparisons can be found in [4]. The approach for Vector Color Filter Array Demosaicing [5] uses filtering in a different way. It first generates pseudo-colors using the local combinations of R, G, and B CFA samples. The chosen color is the “median color” whose total distance to the other pseudo-colors is minimized. Techniques sensitive to gradients were introduced to reduce over-smoothing by performing color interpolation only along sharp edges and not across them. Laroche and Prescott [6], Hamilton and Adams [7], and Chang et al. [8] presented algorithms with a chronologically increasing number of gradient directions evalu- ated and interpolated. Kimmel [9] modeled images as smooth surfaces separated by edge discontinuities which were enhanced with inverse diffusion. There are also grid-based techniques that learn statistical image models. Mal- var et al. [10] presents a fast linear interpolation scheme with color-specific ker- nels which were learned using a Wiener approach from the popular Kodak data set [11]. Another approach, independent of color filter array pattern, is Assorted Pixels [12] which constructs a kernel based on any multi-spectral array. Hel-Or [13] modeled correlation between channels using Canonical Correlation Analysis. Bayesian statistical methods were applied to the demosaicing problem by Brainard [14] who modeled the base image priors as a set of sinusoids. Closer to our two color approach is Bayesian Matting [15], which finds the foreground and background colors from the surrounding areas. Using a Bayesian system, it finds a linear blending ratio between these two colors. To perform super-resolution enhancement, additional resolution information must be acquired from somewhere. Zomet and Peleg [16] use information in multiple images taken from different sensors while Freeman et al. [17] inferred resolution from different resolution scalings of the same image. Demosaicing while providing super-resolution from multiple images was investigated by Fung and Mann [18]. Their approach places samples into a regularly spaced grid and each output pixel component is found using a nearest neighbor search of the registered inputs. Gotoh and Okutomi [19] generalized earlier super-resolution approaches to directly process Bayer samples from many frames. Their results used primarily synthetic frames (>20), and did not give quantitative results or consider single-image demosaicing. To evaluate our results, we are interested in using a more perceptually valid measure than merely SNR or MSE. The S-CIELAB [20, 21] model provides an extension to the Lαβ color space that is aware of local contrast and can hint if the human visual system (HVS) cannot detect errors due to masking. Video and Image Bayesian Demosaicing with a Two Color Image Prior 511 Also of interest is the iCAM [22] model which predicts the color the HVS per- ceives in the presence of nearby colors. Although we do not use such a com- plex model, there are methods that measure perceptual error considering more aspects of the HVS, such as the Multiscale Adaptation Model [23] and Visi- ble Differences Predictor [24]. These and other models have proved useful in detecting HDR compression error and in allocating rendering tasks based on contrast [25, 26]. 3 The Two Color Model Central to our processing is the assumption that at most two representative colors exist within a local neighborhood. Every pixel within that neighborhood is either one of the representative colors or is a linear combination of both. This assumption is violated in areas where more than two different colors meet, but such occurrences are relatively rare. Assuming a Gaussian noise model, this distribution represents a cylindrical volume in color space which spans the two representative colors, as shown in Figure 2. The two color model serves multiple purposes. Primarily, it serves as a con- straint to the ill-conditioned demosaicing problem.