A Colorization Algorithm Based on Local MAP Estimation
Total Page:16
File Type:pdf, Size:1020Kb
A Colorization Algorithm Based on Local MAP Estimation ∗ Hideki Noda a, , Jin Korekuni b, Michiharu Niimi a aDepartment of Systems Innovation and Informatics, Kyushu Institute of Technology, 680-4 Kawazu, Iizuka, 820-8502 Japan bDepartment of Electrical, Electronic and Computer Engineering, Kyushu Institute of Technology, 1-1 Sensui-cho, Tobata-ku, Kitakyushu, 804-8550 Japan Abstract This paper presents a colorization algorithm which adds color to monochrome im- ages. In this paper, the colorization problem is formulated as the maximum a poste- riori (MAP) estimation of a color image given a monochrome image. Markov random field (MRF) is used for modeling a color image which is utilized as a prior for the MAP estimation. The MAP estimation problem for a whole image is decomposed into local MAP estimation problems for each pixel. Using 0.6% of whole pixels as references, the proposed method produced pretty high quality color images with 25.7 dB to 32.6 dB PSNR values for eight images. Key words: Colorization, Monochrome image, MAP estimation, MRF 1 Introduction Colorization is a process, usually a computer-aided process of adding color to monochrome images or movies. There should be considerable demands for colorization of monochrome images or movies. Colorization is now generally carried out manually using some drawing software tools. A user typically car- ries out segmentation of a monochrome image by giving region boundaries by hand and then assigns a color to each region. Obviously such manual work is very expensive and time-consuming. ∗ Corresponding author. Tel.: +81-948-29-7714; fax: +81-948-29-7709 Email address: [email protected] (Hideki Noda). Preprint submitted to Elsevier Science 28 March 2006 Recently several colorization methods [1–3] have been proposed which do not require intensive manual effort. Welsh et al. proposed a semi-automatic method to colorize a monochrome image by transferring color from a reference color image [1]. The entire color ”mood” of the reference image is transferred to the target monochrome image by matching luminance and texture information between the two images. This method requires an appropriate reference color image which should be prepared by a user and works well only for images where differently colored regions have distinct luminance values or distinct textures. Levin et al. have proposed an interactive method which does not require precise manual segmentation [2]. In their method, instead of manual segmentation, a user needs to give some color scribbles, and the colors are automatically propagated to produce a fully colorized image. Horiuchi [3] has proposed a method where a user gives colors for some pixels and colors for all other pixels are determined automatically by using the probabilistic relax- ation. One of serious problems in his method is that it is computationally very expensive; it takes almost one day to colorize one image. Unlike previously proposed colorization methods, this paper formulates the colorization problem as Bayesian inference, i.e., the maximum a posteriori (MAP) estimation of a color image given a monochrome image. Markov ran- dom field (MRF) [4] is used for modeling a color image which is utilized as a prior for the MAP estimation. In this paper, the global MAP estimation problem for a whole image is approximately decomposed into local MAP es- timation problems for each pixel, and the local MAP estimation is reduced to a simple quadratic programming problem with constraints. 2 Color Image Modeling By Markov Random Field 2.1 Markov Random Field Let L = {(i, j); 1 ≤ i ≤ N1, 1 ≤ j ≤ N2} denote a finite set of sites of an N1 × X N2 rectangular lattice. Let ηij ⊂ L denote the (i, j) pixel’s neighborhood of a 1 X random field XL defined on L.LetCij denote the set of cliques C associated X X with ηij which contains the (i, j) pixel, i.e., (i, j) ∈ Cij . For example, in the X first-order neighborhood, ηij = {(i, j +1), (i, j − 1), (i +1,j), (i − 1,j)} and X Cij = {{(i, j)}, {(i, j), (i, j+1)}, {(i, j), (i, j−1)}, {(i, j), (i+1,j)}, {(i, j), (i− 1,j)}} which consists of one singleton and four doubleton cliques. Let the random field XL = {Xij;(i, j) ∈ L} be a Markov random field (MRF) defined on L with Xijs taking values from a common local state space QX .Itiswell 1 x f x {x ,...,x } In this paper, A and ( A)denotetheset a1 al and the multivariable f x ,...,x A {a ,...,a} function ( a1 al ) respectively, where = 1 l . 2 known that an MRF is completely described by a Gibbs distribution 1 p(xL)= exp{−U(xL)}, (1) ZX N1×N2 where xL is a realization of XL from the configuration space ΩX = QX and U(xL)= U(xC )(2) (i,j)∈L X C∈Cij is the global energy function whereas U(xC ) is the clique energy function and ZX = exp{−U(xL)} (3) xL∈ΩX is the partition function. For details on MRFs and related concepts such as the neighborhoods and cliques, see Ref. [4]. 2.2 A Color Image Model Using Gaussian MRF A color image can be considered as a realization xL = {xij;(i, j) ∈ L} of T a random field XL = {Xij;(i, j) ∈ L},wherexij =(rij,gij,bij) is a color vector at (i, j) pixel composed of red rij, green gij and blue bij components. Color images are modeled by a Gaussian MRF (GMRF) characterized by the following local conditional probability density function (pdf) 2 : 1 1 T −1 | X {− − − } p(xij xη )= 3/2 1/2 exp (xij mij) (ΣX ) (xij mij) ,(4) ij (2π) |ΣX | 2 1 mij = xij+τ . (5) |N| τ∈N Here mij is the mean of neighboring pixels’ color vectors x X = {xij+τ ,τ ∈ ηij N},whereN denotes the neighborhood of (0, 0)-pixel. For example, N = {(0, 1), (0, −1), (1, 0), (−1, 0)} for the first-order neighborhood, and if τ = (0, 1), xij+τ = xi,j+1. ΣX is the covariance matrix of xij − mij. 2 The used GMRF is one of the simplest GMRFs, which can model only nontextured smooth images. We here used this GMRF as a first step, though there are more complicated GMRFs applicable to textured images. 3 3 Color Image Estimation 3.1 Derivation of Estimation Algorithm We assume that a monochrome image yL = {yij;(i, j) ∈ L} is associated with a color image xL = {xij;(i, j) ∈ L} under the following relation: T yij = a xij =0.299rij +0.587gij +0.114bij, 0 ≤ yij,rij,gij,bij ≤ 255.(6) Given yL, xL can be estimated by maximizing the a posteriori probability p(xL | yL), i.e., by MAP estimation. The MAP estimate xˆL is written as xˆL =argmax p(xL | yL), (7) xL∈ΩX where the a posteriori probability p(xL | yL) is described as p(yL | xL)p(xL) | p(xL yL)= | . (8) xL∈ΩX p(yL xL)p(xL) Note that it is practically impossible to find the MAP estimate xˆL since the 3|L| search space over all possible configurations of xL is huge, i.e., |ΩX | = 256 . To overcome this problem, hereinafter we consider mean-field-based decom- position of the a posteriori probability. Considering (6), p(yL | xL) is described as T p(yL | xL)=1({yij = a xij, (i, j) ∈ L}) T = 1(yij = a xij), (9) (i,j)∈L where ⎧ ⎨⎪ T T 1ifyij = a xij 1(yij = a xij)=⎪ (10) ⎩ 0otherwise. Using the mean field approximation, p(xL) can be decomposed as [5] p(xL) p(xij |x X ), (11) ηij (i,j)∈L 4 where xηX denotes the mean fields for xηX . Substituting (9) and (11) into ij ij (8) and replacing xL∈ΩX (i,j)∈L by (i,j)∈L xij ∈QX , we obtain the following decomposition for p(xL | yL): p(xL | yL) p(xij | yij, x X ), (12) ηij (i,j)∈L where T 1(yij = a xij)p(xij |x X ) ηij p(xij | yij, x X )= . (13) ηij T ∈Q 1(yij = a xij)p(xij |x X ) xij X ηij In the following, x X is simply used for x X .Thenp(xij | yij, x X )=p(xij | ηij ηij ηij yij, x X ) is considered as a local a posteriori probability (LAP). Using these ηij LAPs, the global optimization problem shown by Eq. (7) is approximately decomposed into the local optimization problems xˆij =arg max p(xij | yij, xηX ). (14) xij ∈QX ij In order to solve (14) for all (i, j) pixels, their neighboring color vectors x X ηij should be given. Since such a problem as shown in (14) can be solved iteratively as is popular in numerical analysis, we rewrite Eq. (14) as (p+1) | (p) xij =arg max p(xij yij, xηX ), (15) xij ∈QX ij where p represents the pth iteration. Considering (4), (5), (6) and (13), the local MAP estimation (15) is rewritten as the following constrained quadratic programming problem: T −1 1 (p) minimize (xij − mij) (ΣX ) (xij − mij)withmij = xij+τ (16) |N| τ∈N T subject to a xij = yij, 0 ≤ rij,gij,bij ≤ 255 (17) 3.2 Initial Color Estimation Since the color estimation shown by Eq. (15) is carried out iteratively, an initial color image is needed to start the iterative procedure. Initial color image estimation using some reference colors is here described. Assuming that color 5 vectors for K pixels, cikjk ,k =1,...,K are given, consider how to derive an initial color image. We consider an initial color estimation procedure which consists of two steps.