Structure-Preserving Color Transformations Using Laplacian Commutativity

Structure-preserving color transformations using Laplacian commutativity

D. Eynard, A. Kovnatsky and M. M. Bronstein

Institute of Computational Science, Faculty of Informatics, University of Lugano, Switzerland

Abstract Mappings between color spaces are ubiquitous in image processing problems such as gamut mapping, decolorization, and image optimization for color-blind people. Simple color transformations often result in information loss and ambiguities (for example, when mapping from RGB to grayscale), and one wishes to find an image-specific transformation that would preserve as much as possible the structure of the original image in the target color space. In this paper, we propose Laplacian colormaps, a generic framework for structure-preserving color transformations between images. We use the image Laplacian to capture the structural information, and show that if the color transformation between two images preserves the structure, the respective Laplacians have similar eigenvectors, or in other words, are approximately jointly diagonalizable. Employing the relation between joint diagonalizability and commutativity of matrices, we use Laplacians commutativity as a criterion of color mapping quality and minimize it w.r.t. the parameters of a color transformation to achieve optimal structure preservation. We show numerous applications of our approach, including color-to-gray conversion, gamut mapping, multispectral image fusion, and image optimization for color deficient viewers.

Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Computer Graphics]: Picture/Image Generation–Display algorithms—I.4.3 [Image processing and computer vision]: Enhancement–Grayscale manipulations—I.4.10 [Image processing and computer vision]: Image Representations–Multidimensional—

1. Introduction and chrominance [GOTG05], color distances [GD07], image gradients [ZF12] and Laplacians [BD13]. A wide class of image processing problems relies on transformations between color spaces. Some notable exam- Color-to-gray maps can be classified into global (using ples include gamut mapping, image optimization for color- the same map for each pixel) and local (or spatial, allow- deficient viewers, and multispectral image fusion. Often ing different pixels with the same color to be mapped to arXiv:1311.0119v1 [cs.CV] 1 Nov 2013 these transformations imply a reduction in the dimensional- different gray values, at the advantage of a better percep- ity of the original color space, resulting in information loss tion of color contrasts). Members of the first group in- and ambiguities. clude the pixel-based approaches by Gooch et al. [GOTG05] and Grundland et al. [GD07], and the color-based ones by Decolorization or color-to-gray conversion is a classical Rasche et al. [RGW05b], Kuhn et al. [KOF08b], Kim et example one frequently encounters when printing a color im- al. [KJDL09], Lu et al. [LXJ12]. Between local methods age on a black-and-white printer. The ambiguity of such a [NvN07, KAC10, ZF12], several try to preserve information conversion (called metamerism, when many different RGB in the gradient domain. Smith et al. [SLTM08] present a hy- colors are mapped to the same gray level) may result in brid (local+global) approach that relies on both an image- a loss of important structure in the image (see Figure1). independent global mapping and a multiscale local contrast Preserving salient characteristics of the original image is enhancement. Lau et al. [LHM11] propose an approach de- thus crucial for a quality color transformation process. These fined as ‘semi-local’, as it clusters pixels based on both their characteristics can be represented in different ways, e.g. as spatial and chromatic similarities. The color mapping prob- contrasts between color pixels in terms of their luminance lem is solved with an optimization aimed at finding optimal 2 Eynard et al. / Structure-preserving color transformations using Laplacian commutativity

Original imageLuma Luma0.0012 Lau11 [LHM11] 0.0008 OursLaplacian 0.0006

4x10-3

0.5 0.98 1.23 0.50

Figure 1: Decolorization experiment results. From left: original RGB image, grayscale conversion results using the Luma channel, the method of [LHM11], and our Laplacian colormap, with their respective RWMS error images and mean RWMS score values. Luma conversion results in loss of image structure due to metamerism (the green island disappears). The proposed Laplacian colormap better preserves the original image structure. cluster colors such that the contrast between clusters is pre- contrast between color clusters and the reduced image col- served. ors.

Gamut mapping is the process of adjusting the colors of Multispectral image fusion aims to combine a collection an input image into the constrained color gamut of a given of images captured at different wavelengths into a single one, device. Gamut mapping algorithms can be mainly divided containing details from several spectra. Zhang et al. [ZSM] into clipping and compression approaches [Mor08]. The for- and Lau et al. [LHM11] present a method that adaptively mer ones change the source colors that fall outside of the adjusts the contrast of photographs by using the contrast and destination gamut (e.g. HPMINDE [CIE04, BSBB06]); the texture information from near-infrared (NIR) image. Kim et latter also modify the in-gamut colors. Similarly to color- al. [JKDB11] show how to use different bands of the invisi- to-gray conversion, gamut mapping methods can also be ble spectrum to improve the visual quality of old documents. categorized as global and local. To address metamerism Süsstrunk and Fredembach [SF10] provide a good introduc- in gamut mapping, local approaches [BdQEW00, NHU99, tion to the topic and present, as examples of image enhance- KSES05] allow two spatially-distant pixels of equal color to ments, haze removal and realistic skin smoothing. be mapped to different in-gamut colors. Global approaches, General approaches. We should stress that despite a sig- conversely, will always apply the same map to two pixels nificant corpus of research on color transformations, most of the same color, regardless of their location. Many gamut of the methods are targeted to specific applications and mapping algorithms optimize some image difference crite- lack the generality of a framework that could be applied rion [NHU99, KSES05, AF09, LHM11]. to different classes of problems. At the same time, there Color-blind viewers cannot perceive differences between is an obvious common denominator between the aforemen- some given colors, due to the lack of one or more types of tioned problems: for example, both color-blind transforma- cone cells in their eyes [dal, MG88]. Image perception by tions [RGW05b] and color-to-gray conversions [CHRW10, a color-deficient observer is typically simulated by first ap- ZT10, ZF12] can be regarded as mappings to a gamuts of plying a linear transformation from a standard color space lower dimension [GOTG05]. To the best of our knowledge, such as RGB [KJY∗12, BVM07, VBM99], XYZ [MG88, only the recent work of [LHM11] introduces a comprehen- RGW05a], or CIE Lab* [KOF08a, HTWW07] to a spe- sive approach that works with generic color transformation cial LMS space, which specifies colors in terms of the and easily adapts to different applications. relative excitations of the cones. Then, the color domain Main contribution. In this paper, we present Lapla- is reduced in accordance with the color deficiency (typ- cian colormaps, a new generic framework for computing ically, by means of a linear transformation in the LMS structure-preserving color transformations that can be ap- ∗ space [VBM99, KJY 12, HTWW07]). Finally, the reduced plied to different problems. Our main motivation comes LMS space is mapped back to RGB. from recent works on Laplacians as structure descriptors [BD13] and joint diagonalization of Laplacians [EGBB12, When trying to adapt an image for a color-blind viewer, KBB∗13, GB13, BGL13] (to the best of our knowledge, our one has to ensure that the structure of the original image is paper is the first application of these methods in the domain not lost due to color ambiguities. Kuhn et al. [KOF08a] fo- of image analysis). cus on obtaining natural images by preserving, as much as possible, the original image colors. Rasche et al. [RGW05a], Using Laplacians as image structure descriptors, we ob- instead, try to maintain distance ratios during the reduction serve that an ideal color transformation should preserve the process. Lau et al. [LHM11] is aimed at preserving both the Laplacian eigenstructure, implying that the Laplacians of the Eynard et al. / Structure-preserving color transformations using Laplacian commutativity 3 original and color-converted image should be jointly diago- weights (adjacencies) as a combination of spatial and ‘ra- nalizable. Employing the relation between joint diagonaliz- diometric’ distances, ability and commutativity of matrices [GB13, BGL13], we 2 2 δ kxk −xk k2 − i j − i j use Laplacians commutativity as a criterion of image struc- 2σ2 2σ2 wi j = e s e r , (2) ture preservation. We try to find such a colormap that would produce a converted image whose Laplacian commutes as where δi j is the spatial distance between pixels ki and k j, much as possible with the Laplacian of the original image. and σs,σr ≥ 0 are parameters (more generally, the ‘radio- Since Laplacians can be defined in any colorspace, our ap- metric’ part of the adjacency wi j does not have to work on proach is generic and applicable to any kind of color con- pixel-wise colors, and one can consider some local features, versions (in particular, color-to-gray, gamut mapping, color- the simples of which are patches [WK12]). For practical blind optimization, etc.). Furthermore, we can work with computations, it is usually assumed that wi j ≈ 0 between both global and local colormaps. spatially-distant pixels, so they are disconnected. We define the (unnormalized) Laplacian† of this graph as a symmetric The rest of the paper is organized as follows: in Section 2, positive semi-definite L × L matrix LX = DX −WX, where we review the main results related to joint diagonalization WX is the adjacency matrix with elements as in (2), and and commutativity of matrices. In Section 3 we formulate DX = diag(∑ j6=i wi j). In the following, we refer to LX as our optimization problem and discuss its numerical solution. the Laplacian of image X. Section 4 shows examples of applications of our framework to different problems involving color transformations. Fi- Since LX is symmetric, it admits an orthonormal eigen- T nally, Section 5 concludes the paper. Technical derivations decomposition by means of a matrix U, such that U LXU = are given in the Appendix. ΛX, where the columns of U are orthonormal eigenvectors, X X and ΛX = diag(λ1 ,...,λL ) are the corresponding eigenval- ues, sorted in ascending order 0 = λ1 ≤ λ2 ≤ ... ≤ λL. For 2. Background simplicity, we assume that there are no repeating eigenval- ues, and thus the eigenvectors are defined up to sign. We say Notation and definitions. We denote by A a matrix, by a a that two Laplacians LX and LY are jointly diagonalizable if T (column) vector, and by a a scalar. We denote by they have the same eigenvectors U, i.e., U LXU = ΛX and T 1/2 1/2 U LYU = ΛY. LX and LY are said to commute if their com- 2 2 kAkF = ∑i j ai j ; kak2 = ∑i ai (1) mutator is [LX,LY] = LXLY − LYLX = 00. the Frobenius norm of the matrix A and the Euclidean norm Image Laplacians as structure descriptors. Lapla- cians have been successfully used in image processing to of a vector aa, respectively. diag(a1,...,an) is a diagonal ma- guide anisotropic diffusion [SKM98]. Shi and Malik [SM00] trix with diagonal elements a1,...,an, diag(A) are the diagonal elements of A arranged as a column vector, Diag(A) is a showed that a spectral relaxation of the normalized cut cri- diagonal matrix obtained by setting to zero the off-diagonal terion for image segmentation boils down to finding the first elements of A, and vec(A) is a column vector obtained by eigenvectors of an image Laplacian and performing segmen- column-stacking of A. tation in the low-dimensional eigensubspace. This approach inspired the popular spectral clustering algorithm [NJW02]. Let us be given an N × M image with d color channels, More recently, Bansal and Daniilidis [BD13] used the eigen- T column-stacked into an NM ×d matrix X = (x1,...,xNM) . vectors of image Laplacians to perform matching of images The problem of color conversion is creating a new image taken in different illumination conditions, arguing that the Y = Φ(X) with d0 color channels, by means of a colormap Laplacian acts as a self-similarity descriptor [SI07] of the NM×d NM×d0 Φ : R → R . In particular, we are interested in image. n parametric colormaps Φθ, parametrized by an -dimensional Applying this idea to color transformations, we can use θ vector of parameters θ. In the simplest case, Φθ is a global the similarity of Laplacian eigenspaces as a criterion of color transformation applied pixel-wise, i.e., each pixel xi ∈ d structural similarity of two images. Figure2 shows three R of the original image is mapped by means of the same images (original RGB image and two decolorized versions d d0 T φθ : R → R such that Φθ(X) = (φθ(x1),...,φθ(xNM)) (a thereof, a ‘bad’ and a ‘good’ one) and the first eigenvectors simple example is linear RGB to gray mapping, where d = 3, of the corresponding Laplacians. One can see that a good 0 d d = 1, n = 3 and φθ(xi) = ∑ j=1 θixi j, where in addition we colormap preserves the image structure, which is manifested d require θ ≥ 0 and ∑i=1 θi = 1). in the two Laplacians having similar eigenvectors (first and

Let {k1,...,kL} ⊆ {1,...,NM} denote a subset of the image pixel indices (this subset can be the whole set of NM † There exist numerous ways of deﬁning Laplacian matrices; we pixels, a regularly subsampled M/s × N/s image, ‘represen- consider the unnormalized one merely for the sake of simplicity. The tative’ pixels obtained by clustering the image, etc.). Con- ability to work with practically any operator capturing the image sidering these pixels as vertices of a graph, we deﬁne edge structure is one of the strengths of our method. 4 Eynard et al. / Structure-preserving color transformations using Laplacian commutativity 1 2 3 4 SC

Figure 2: Image structure similarity is conveyed by the eigenstructure of their Laplacians. Top: original RGB image; middle: grayscale conversion by our method; bottom: luma only conversion. Left-to-right: original image, first four eigenvectors of the corresponding Laplacian, result of spectral clustering. third rows). In particular, if one applies spectral clustering Since A˜ ,B˜ commute, they are jointly diagonalizable. Fur- to such images, the resulting segmentation will be similar. thermore, if A,B approximately commute, C(A,B) is guar- Thus, an ideal color transformation Φ(X) should make the anteed to be small, i.e., almost-commuting matrices are close corresponding Laplacians LX and LΦ(X) jointly diagonaliz- to commuting ones [Lin97]. able. Somewhat surprisingly, it turns out that the JAD and CCO Joint approximate diagonalization (JAD) is a way to problems are equivalent, in the following sense: enforce two matrices to have the same eigenstructure. Given Theorem 2.2 (Bronstein et al. [BGL13]) Let A,B be sym- two matrices A and B, one seeks a joint approximate eigen- metric matrices, Uˆ be the minimizer of the JAD problem (3), basis Uˆ such that Uˆ TAUˆ and Uˆ TBUˆ are approximately diag- and A˜ ,B˜ be the minimizers of the CCO problem (4), jointly onal, diagonalized by U˜ . Then: J(A,B) = min off(Uˆ TAUˆ ) + off(Uˆ TBUˆ ) s.t. Uˆ TUˆ = I, (3) Uˆ 1. C(A,B) = J(A,B); ˜ ˆ 2 2. U = U; where off(A) = ∑i6= j ai j. JAD has been recently applied 3. A˜ = Uˆ Diag(Uˆ TAUˆ )Uˆ T and B˜ = Uˆ Diag(Uˆ TBUˆ )Uˆ T. to jointly diagonalize Laplacian matrices in order to find compatible Fourier bases on graphs [EGBB12] and surfaces Despite being equivalent, JAD and CCO problem have [KBB∗13]. The drawback of this formulation is that both a key difference: in the former, optimization is performed matrices are assumed to be given, while in our problem w.r.t. the joint eigenbasis, while in the latter, optimization is only one matrix (the original image Laplacian, LX) is given, performed w.r.t. closest commuting matrices. In our prob- while the other (the transformed image Laplacian, LY) has lem, the CCO formulation allows to optimize w.r.t. Lapla- to be found. cians, which, in turn, can be parametrized through the colormap. Closest commuting operators (CCO). Joint diagonalizability is intimately related to matrix commutativity. It is Let us summarize the main results of this section, which well-known that A and B are jointly diagonalizable iff they will motivate our approach described in the following. First, commute, i.e., [A,B] = 0 [HJ90]. It appears that this relation Laplacians can be used as structural descriptors of images. also holds for almost-commuting matrices, in the following Second, two images having similar structures translates into sense: having the corresponding Laplacians jointly diagonalizable. Third, joint diagonalizability is equivalent to commutativity. Theorem 2.1 (Glashoff-Bronstein [GB13]) Let A,B be two N × N symmetric matrices normalized such that kAkF = The key idea of this paper is to find such a colormap Φ(X) kBkF = 1. Then, that the Laplacian LX of the input image and the Lapla- cian L of the output image commute as much as possi- δ (k[A,B]k ) ≤ J(A,B) ≤ δ (k[A,B]k ) Φ(X) 1 F 2 F ble. Due to the relation between approximate commutativity where δ1(x),δ2(x) are functions satisfying limx→0 δi(x) = 0; and joint diagonalizability, it will imply that LX and LΦ(X) have similar eigenvectors, and thus the underlying images or in other words, almost commuting matrices are almost are structurally similar. jointly diagonalizable. Bronstein et al. [BGL13] studied an alternative problem 3. Laplacian colormaps of finding the closest commuting matrices A˜ ,B˜ to the given A and B, Problem formulation. Let X be a given NM × d original 0 ˜ 2 2 ˜ ˜ image and Φθ(X) be the desired color-converted NM × d C(A,B) = minkA − AkF + kB˜ − BkF s.t. AB˜ = B˜ A (4) A˜ ,B˜ image. Our goal is to find a set of parameters θ such that Eynard et al. / Structure-preserving color transformations using Laplacian commutativity 5 the structures of the images X and Φθ(X) are as similar as blind people, gamut mapping, and multispectral image fu- possible, where the similarity is judged by the commutativity sion, providing extensive comparison to previous works. of the corresponding Laplacians. This leads us to a class of As a quantitative criterion of the colormap quality, we use optimization problems of the form the root weighted mean square (RWMS) error proposed by [KOF08b], measuring the distortion of relative color dis- min µ k[L ,L ]k2 + µ kL − L k2 n 0 X Φθ (X) F 1 X Φθ (X) F θ∈R tances in two images, 2 2 / +µ2kθ −θ0k2 + µ3kΦθ(Xc) − YckF (5) NM 2 !1 2 1 (R kxi − x jk − R kyi − y jk) ε = Y X , (7) s.t. constraints on θ. i ∑ 2 2 NM j=1 RYkxi − x jk One can easily recognize in problem (5) a parametric ver- d d0 sion of the CCO problem (4) with one of the Laplacians where N × M is the image size, xi ∈ R and yi ∈ R denote

fixed. Note that the Laplacian LΦθ(X) is parametrized by a the ith pixel of the input and output images, respectively, small number of degrees of freedom n L, and thus it would and RX = maxi j kxi − x jk is the color range of image X. be usually impossible to make it exactly commute with the Plotting the pixel-wise RWMS error εi as an image allows given LX - hence, unlike the CCO problem, the commutator to see which pixels are most affected by the color transfor- norm appears as a penalty rather than a constraint. 1 NM mation. The average NM ∑i=1 εi is used as a single number Additional regularization (third and fourth terms in (5)) representing the quality of the colormap. is used if we have some ‘nominal’ parameters θ0 repre- All experiments share a common setup: first of all, RGB senting a standard color transformation, or if some col- values are scaled by 255. Then we calculate a weighted ad- T ors Xc = (x1,...,xp) should be mapped into some Yc = jacency matrix according to (2) using all pixels (L = MN) if T (y1,...,yp) known in advance (for example, in some cases the images are small enough, and resizing the image to have it is important to preserve black and white colors). Finally, long side of 300 pixels otherwise. We used fixed 4-neighbors depending on the type of the colormap , one may impose Φθ connectivity and parameters σr = 1, σs = 0. Default weights some constraints on the parameters θ (e.g., in linear RGB- for the cost function are µ1 = 1,µ2 = 1,µ3 = 0, and regular- to-gray conversion, θ ≥ 0 and θT1 = 1). ization term θ0 = 00. Parameters are initialized randomly and Local maps. Our approach imposes no limitations on the normalized to satisfy the condition θT1 = 1. complexity of the colormap Φ ; in particular, this map does θ As a last step, since mapping might produce color val- not have to be global. Let us assume that the source image ues out of the [0,1] range, output channels are normalized. is partitioned into q (soft) regions, represented by weight Optimization was implemented in MATLAB, using interior- vectors w ,...,w of size NM × 1, such that q w = 1 1 q ∑i=1 i point method from the Optimization Toolbox. and wi ≥ 00. In each region i, we allow for a different colormap Φθ . Then, the overall colormap is given as Φθ(X) = Decolorization. For RGB-to-gray mapping, we used a q i ∑i=1 Φθi (X), parametrized by θ = (θ1,...,θq). Optimiza- global colormap, applying in each pixel xi the following 3 γi tion w.r.t. to the parameters of the local colormap is per- transformation: yi = α + ∑ j=1 βixi j, where xi j is the jth formed in exactly the same manner as described above. RGB channel of the ith pixel, yi is the grayscale output, and θ = ( , , ,..., , ) Multiple Laplacians. In some applications like multi- θ α β1 γ1 β3 γ3 are the colormap parameters w.r.t. spectral image fusion, one may wish to impose structural which the optimization is performed. similarity between the output image and multiple images, Images used for this experiment were taken from [Cad08ˇ ]. X1,...,XK with colorspaces of dimensionality d1,...,dK. Figure8 shows the results of our transformations, com- The input image X may be one of the K images or a merged pared to previous works [GOTG05,RGW05b,GD07,NvN07, K image with ∑k=1 dk-dimensional colorspace. In this case, SLTM08]. Results were evaluated using two different met- our optimization problem (5) assumes the form rics: quantitative (RWMS) and qualitative perceptual evalu- K ation following [Cad08ˇ ]. In the perceptual evaluation con- 2 2 min µ0kk[LXk ,LΦ (X)]kF + µ1kkLXk − LΦ (X)kF ducted through a Web survey, 107 volunteers were shown θ∈ n ∑ θ θ R k=1 the original RGB image together with a pair of its gray con- 2 2 +µ2kθ −θ0k2 + µ3kΦθ(Xc) − YckF (6) versions, and were asked which of the two results better pre- s.t. constraints on θ, served the original image. Then, we used Thurstone’s law of comparative judgments to convert the 1857 pairwise eval- where µ01,...,µ0K,µ11,...,µ1K,µ2,µ3 ≥ 0 are constants de- uations into interval z-score scales [Thu27, TG11]. Table1 termining the tradeoff between different penalties. provides average RWMS values and z-scores calculated on an 8-images subset of Cadik’s.ˇ Our approach performs the 4. Results and Applications best w.r.t. both criteria. In this section, we show several applications of our ap- Color-blind viewers. We model the color distortion of proach for decolorization, image optimization for color- an RGB image X as perceived by a color-blind person by 6 Eynard et al. / Structure-preserving color transformations using Laplacian commutativity

CIE Y [GOTG05][RGW05b][GD07][NvN07][SLTM08] Laplacian RWMS 2.86 2.31 2.49 2.21 4.91 2.94 1.42 z-score -0.14 -0.24 -0.55 0.63 -0.45 -0.06 0.81

Table 1: Comparison of color-to-gray conversions in terms of mean RWMS value and z-score, averaged on all images.

Original image Luma [LHM11] Gamut mapping is a problem similar to the previous one, and has a setting similar to the one in the previous experiment. A transformation Ψ which maps colors from RGB to the XY chromaticity space and a color gamut G (a con- vex polytope, and in this particular experiment a triangle) are given. Our goal is to find θ minimizing the cost (8) subject to (Φθ ◦Ψ)(X) ⊆ G, which is imposed as a set of linear con- Laplacian (global) Laplacian (local) Clusters straints. We used the parameters µ01 = 1,µ11 = 0.25,µ2 = 0.1, and µ02 = µ12 = 0. Figure5 compares our results with the outputs of HPMINDE [CIE04] and by the method of Lau et al. [LHM11]. Qualitatively, the output of Laplacian colormaps preserves more details of the original picture (see e.g. the plumage on the red parrot’s head). Quantitatively, our algorithm outperforms the other methods in term of per- centage of out-of-gamut. Figure 3: Global vs Local maps results. Top row, left-to- Multispectral image fusion can be seen as an extension right: original image, Luma, result by Lau et al. [LHM11]. of the color-to-gray problem, where the number of input Bottom row: Laplacian colormaps using a global (left) and channels d > 3 and the output image has d0 = 3 channels. local (middle) map; the spatial weights used in the latter. We use the cost function (6), with µ01 = µ02 = µ11 = µ12 = NM×d NM×d µ2 = 1 and µ3 > 0; the latter does not only act as regulariza- means of a map Ψ : R → R . Since Ψ is given tion, but also provides us a way to automatically order the and beyond our control, we try to ‘pre-transform’ the orig- NM×d NM×d three output channels. inal image by means of Φθ : R → R in such a way that the image (Φθ ◦ Ψ)(X) that appears to the color- Figure6 shows multispectral to RGB transformations blind person has the structure of the original image X. We where the input space is the concatenation of RGB and NIR extend our problem formulation so that the transformed im- (d = 4). In this specific example, the NIR channel is used age maintains its structure both when seen by a color-blind to enhance the RGB image with an additional source of in- observer and when seen by a regular observer. In our opti- formation. Comparing our result with the method of Lau et mization problem, this translates into requiring the two pairs al. [LHM11], we can see that Laplacian colormap provides of Laplacians L ,L and L ,L to commute. an enhanced version of RGB while preserving the correct X (Φθ ◦Ψ)(X) X Φθ (X) The cost function is similar to the multiple Laplacians set- colors (e.g. trees on the mountains have more detail than in ting (6): RGB, but at the same time they do not present the blue-green 2 2 halo that appears in [LHM11]). Finally, in Figure7 we show min µ01k[LX,L ]k + µ02k[LX,L ]k n (Φθ ◦Ψ)(X) F Φθ (X) F a fusion of four photos of a city in different lighting condi- θ∈R tions into a single image, which looks visually plausible. +µ kL − L k2 + µ kL − L k2 11 X (Φθ ◦Ψ)(X) F 12 X Φθ (X) F +µ kθ −θ k2 (8) 2 0 2 5. Conclusions Figure4 shows Laplacian colormaps results for two dif- Laplacian colormaps address the problem of structure- ferent types of color blindness (protanopia and tritanopia). preserving color transformations by relying on Laplacians Qualitatively, our result appears to be much closer to the as image structure descriptors and using Laplacian commu- original image compared to [LHM11] (this is especially ap- tativity as a criterion for structure preservation. Given a para- parent in the tritanopia case) such that a ‘normal’ viewer metric colormap, we optimize for the parameters that pro- sees less distorted colors, while a color-deficient viewer can duce an image whose Laplacian commutes as much as pos- clearly see the structure structured in the image (digit 6 and sible with the one of the original image, thus preserving its different candies) which otherwise would disappear. Quanti- original structure. Since Laplacians can be defined in any tatively, we obtain smaller RWMS error, suggesting that our colorspace, our approach can be applied to different kinds mapping better preserves the original structure of the image, of colormaps (global or local, with any number of input and even in those areas that are critical for other approaches. output channels, and where part of the mapping is provided Eynard et al. / Structure-preserving color transformations using Laplacian commutativity 7

Original image Color-blind [LHM11] Laplacian −3 Lau et Al − mean d=0.0012 −3 −3 Protanope − mean d=0.0010 x 10 x 10 Optimized − mean d=0.0005 x 10

2.2 2.2 2.2

2 2 2

1.8 1.8 1.8

1.6 1.6 1.6

1.4 1.4 1.4

1.2 1.2 1.2

1 1 1

0.8 0.8 2.2x100.8 -3

0.6 0.6 0.6

0.4 0.4 0.40.4

0.98 1.23 0.50

−3 Lau et Al − RWMS=1.62 −3 −3 Tritanope − RWMS=1.27 x 10 x 10 Ibopt − RWMS=0.53 x 10

2 2 2

1.8 1.8 1.8

1.6 1.6 1.6

1.4 1.4 1.4

1.2 1.2 1.2

1 1 1

0.8 0.8 0.8

0.6 0.6 0.6 2x10-3 0.4 0.4 0.4

0.2 0.2 0.20.2

1.27 1.69 0.53

Figure 4: Color mapping for color-blind observers (top: protanope, bottom: tritanope). From left: original image, simulated color-blind, result from [LHM11], and our result, with their respective RWMS error images and mean RWMS values.

a priori). Moreover, Laplacians can be computed using sim- [BVM07]B RETTEL H.,VIÉNOT F., MOLLON J. D.: Comput- ilarity of local feature descriptors rather than individual pix- erized simulation of color appearance for dichromats. JOSA 14 els colors. Computationally, pixel-wise relationships is the (2007), 2647–2655.2 main bottleneck of our approach: a benchmark we ran on [Cad08]ˇ Cˇ ADÍK M.: Perceptual evaluation of color-to-grayscale all the 25 pictures of Cadík’s˘ dataset using a MacBook Pro image conversions. CGF 27, 7 (2008), 1745–1754.5 with 8GB RAM showed that the average computation time [CHRW10]C UI M.,HU J.,RAZDAN A.,WONKA P.: Color-to- for color-to-grayscale conversion was 117 seconds. This cost gray conversion using isomap. Visual Computer 26, 11 (2010), 1349–1360.2 can be alleviated by computing Laplacians on resized images, at a price of potentially lower accuracy. [CIE04] Guidelines for the evaluation of color gamut mapping algorithms. Tech. Rep. CIE 156:2004, 2004.2,6,8 Overall, we believe that our results show the promise in [dal] Daltonize. http://www.daltonize.org/.2 the use of Laplacians commutators to measure structure sim- [EGBB12]E YNARD D.,GLASHOFF K.,BRONSTEIN M.M., ilarity, and seem to be the ﬁrst application of rather theoreti- BRONSTEIN A. M.: Multimodal diffusion geometry by joint di- cal results on joint diagonalization of matrices to very practi- agonalization of Laplacians. arXiv:1209.2295 (2012).2,4 cal problems in image processing. In future works, we intend [GB13]G LASHOFF K.,BRONSTEIN M. M.: Matrix commuta- to explore additional applications such as image correspon- tors: their asymptotic metric properties and relation to approxi- dence and alignment. We believe that our approach will be mate joint diagonalization. LAA 438, 8 (2013), 2503–2513.2,3, especially useful when handling visually different but struc- 4 turally similar scenes. [GD07]G RUNDLAND M.,DODGSON N. A.: Decolorize: Fast, contrast enhancing, color to grayscale conversion. Pattern Recog- Acknowledgements nition 40, 11 (2007), 2891–2896.1,5,6, 10, 11 This research was supported by the ERC Starting Grant No. [GOTG05]G OOCH A.A.,OLSEN S.C.,TUMBLIN J.,GOOCH 307047 (COMET). B.: Color2gray: salience-preserving color removal. TOG 24, 3 (2005), 634–639.1,2,5,6, 10, 11 References [HJ90]H ORN R.A.,JOHNSON C.R.: Matrix Analysis. Cam- bridge University Press, 1990.4 [AF09]A LSAM A.,FARUP I.: Colour gamut mapping as a constrained variational problem. In Image Analysis. 2009, pp. 109– [HTWW07]H UANG J.-B.,TSENG Y.-C., WU S.-I.,WANG S.- 118.2 J.: Information preserving color transformation for protanopia [BD13]B ANSAL M.,DANIILIDIS K.: Joint spectral correspon- and deuteranopia. Signal Processing Letters 14, 10 (2007), 711– dence for disparate image matching. In Proc. CVPR (2013).1, 714.2 2,3 [JKDB11]J OO KIM S.,DENG F., BROWN M. S.: Visual en- [BdQEW00]B ALASUBRAMANIAN R., DE QUEIROZ R.L.,ES- hancement of old documents with hyperspectral imaging. Pattern CHBACH R.,WU W.: Gamut mapping to preserve spatial lumi- Recognition 44, 7 (2011), 1461–1469.2 nance variations. In Proc. Conf. Color Imaging (2000).2 [KAC10]K UK J.,AHN J.,CHO N.: A color to grayscale con- [BGL13]B RONSTEIN M.M.,GLASHOFF K.,LORING T. A.: version considering local and global contrast. In Proc. ACCV Making Laplacians commute. arXiv:1307.6549 (2013).2,3,4 (2010).1 [BSBB06]B ONNIER N.,SCHMITT F., BRETTEL H.,BERCHE [KBB∗13]K OVNATSKY A.,BRONSTEIN M.M.,BRONSTEIN S.: Evaluation of spatial gamut mapping algorithms. In Proc. A.M.,GLASHOFF K.,KIMMEL R.: Coupled quasi-harmonic Conf. Color Imaging (2006).2 bases. CGF 32 (2013), 439–448.2,4 8 Eynard et al. / Structure-preserving color transformations using Laplacian commutativity

Original image [LHM11] Laplacian HPMINDE [CIE04]

Figure 5: Gamut mapping results. Odd rows, left-to-right: original image, gamut mapping with method of Lau et al. [LHM11], our approach and HPMINDE [CIE04]. Even rows: gamut alerts for the images above (green shows the out-of-gamut pixels). NIR RGB [LHM11] Laplacian

Figure 6: Multispectral (RGB+NIR) fusion results.

[KJDL09]K IM Y., JANG C.,DEMOUTH J.,LEE S.: Robust [KSES05]K IMMEL R.,SHAKED D.,ELAD M.,SOBEL I.: color-to-gray via nonlinear global mapping. TOG 28, 5 (2009), Space-dependent color gamut mapping: a variational approach. 161:1–161:4.1 Trans. Image Processing 14, 6 (2005), 796–803.2 ∗ [KJY 12]K IM H.-J.,JEONG J.-Y., YOON Y.-J., KIM Y.-H., [LHM11]L AU C.,HEIDRICH W., MANTIUK R.: Cluster-based KO S.-J.: Color modiﬁcation for color-blind viewers using the color space optimizations. In ICCV (2011).1,2,6,7,8 dynamic color transformation. In Proc. ICCE (2012).2 [Lin97]L IN H.: Almost commuting selfadjoint matrices and ap- [KOF08a]K UHN G.R.,OLIVEIRA M.M.,FERNANDES L. plications. Fields Inst. Commun. 13 (1997), 193–233.4 A. F.: An efﬁcient naturalness-preserving image-recoloring method for dichromats. Trans. VCG 14, 6 (2008), 1747–1754. [LXJ12]L U C.,XU L.,JIA J.: Contrast preserving decoloriza- 2 tion. In Proc. ICCP (2012).1 [KOF08b]K UHN G.R.,OLIVEIRA M.M.,FERNANDES L. [MG88]M EYER G. W., GREENBERG D. P.: Color-defective A. F.: An improved contrast enhancing approach for color-to- vision and computer graphics displays. IEEE Comput. Graph. grayscale mappings. Vis. Comput. 24, 7 (2008), 505–514.1,5 Appl. 8, 5 (1988), 28–40.2 Eynard et al. / Structure-preserving color transformations using Laplacian commutativity 9

Morning Day Evening Night Fusion

Figure 7: Fusion of images of four different illuminations into a single RGB image (rightmost).

OROVICˇ Color Gamut Mapping W n [Mor08]M J.: . Wiley, 2008.2 the adjacency matrix WΦθ(X) , and by the number of parameters [NHU99]N AKAUCHI S.,HATANAKA S.,USUI S.: Color gamut θ of the colormap, respectively. The non-zero elements wi j > 0 w i mapping based on a perceptual image difference measure. Color are indexed as wθ = (w1,...,w|W|) = (wi1, j1 ,...,wi|W|, j|W| ). φθ : d Research & Application 24, 4 (1999), 280–291.2 R → R denotes the ith channel of the colormap, such that φθ (x) = 1 d0 T i [NJW02]N G A.,JORDAN M.I.,WEISS Y.: On spectral cluster- (φθ (x),··· ,φθ (x)) , and ∇θ φθ is its gradient w.r.t. θ. ing: Analysis and an algorithm. In Proc. NIPS (2002).3 We now derive the gradients of the cost function (5). The gradient ˇ [NvN07]N EUMANN L., CADÍK M.,NEMCSICS A.: An efficient of the µ -term is trivial, perception-based adaptive color to gray transformation. In Com- 2 2 putational Aesthetics (2007).1,5,6, 10, 11 ∇θ kθ −θ0k2 = 2(θ −θ0). [RGW05a]R ASCHE K.,GEIST R.,WESTALL J.: Detail preserving reproduction of color images for monochromats and dichro- Denote by Gi the matrix of size NM ×n, whose jth row is the Φθ (X) mats. IEEE Comput. Graph. Appl. 25, 3 (2005), 22–30.2 i T gradient of the ith channel at the jth pixel, ∇θ φθ (x j) , and define [RGW05b]R ASCHE K.,GEIST R.,WESTALL J.: Re-coloring 0 1 T d0 T T NMd ×n matrix GΦ (X) = ((G ) ,··· ,(G ) ) . Differen- images for gamuts of lower dimension. CGF 24, 3 (2005), 423– θ Φθ (X) Φθ (X) tiating the µ -term w.r.t θ gives 432.1,2,5,6, 10, 11 3 2 T ∇ kΦ (Xc) − Yck = 2GG (vec(Φ (Xc)) − vec(Yc)). [SF10]S ÜSSTRUNK S.,FREDEMBACH C.: Enhancing the visible θ θ F Φθ (X) θ with the invisible. In Proc. SID (2010).2 [SI07]S HECHTMAN E.,IRANI M.: Matching local self- The gradients of the first two terms of (5) are obtained by apply- similarities across images and videos. In Proc. CVPR (2007). ing the chain rule. First, we differentiate the terms w.r.t wθ , obtain- 3 ing a gradient of size |W| × 1. Next, we differentiate w.r.t. θ. The gradient of the adjacency matrix elements w w.r.t. θ is [SKM98]S OCHEN N.,KIMMEL R.,MALLADI R.: A general i j framework for low level vision. Trans. Image Processing 7, 3 d0 wi j (1998), 310–318.3 ∇ w = − (φk (x ) − φk (x ))(∇ φk (x ) − ∇ φk (x )). θ i j 2 ∑ θ i θ j θ θ i θ θ j σr k [SLTM08]S MITH K.,LANDES P.-E., THOLLOT J., =1 MYSZKOWSKI K.: Apparent greyscale: A simple and fast The gradient of the commutator (µ0-term) is: conversion to perceptually accurate images and video. CGF 27, ∂ 2 (2008), 193–200.1,5,6, 10, 11 k[L ,L ]k2 = X Φθ (X) F ∂wi j [SM00]S HI J.,MALIK J.: Normalized cuts and image segmentation. Trans. PAMI 22, 8 (2000), 888–905.3 T T − 2 O1 − LΦ (X)[LΦ (X),LX] −O2 + [LΦ (X),LX]LΦ (X) . θ θ θ θ i j [TG11]T SUKIDA K.,GUPTA M.R.: How to Analyze Paired Comparison Data. Tech. rep., DTIC Document, 2011.5 The gradient of the µ1-term is: [Thu27]T HURSTONE L. L.: A law of comparative judgment. ∂ kL − L k2 = 2 O + L − L . X Φθ (X) F X Φθ (X) Psychological Review 34, 4 (1927), 273–286.5 ∂wi j i j [VBM99]V IÉNOT F., BRETTEL H.,MOLLON J. D.: Digi- Here, O,Ok are matrices with equal columns given by tal video colourmaps for checking the legibility of displays by dichromats. Color Research & Application 24, 4 (1999), 243– O = (diag(LX),··· ,diag(LX)), 252.2 T T O1 = (diag(L [L ,LX]),··· ,diag(L [L ,LX])) [WK12]W ETZLER A.,KIMMEL R.: Efficient Beltrami flow in Φθ (X) Φθ (X) Φθ (X) Φθ (X) patch-space. In Proc. SSVM. 2012.3 T T HOU ENG O2 = (diag([L ,LX]L ),··· ,diag([L ,LX]L )). [ZF12]Z B.,F J.: Gradient domain salience-preserving Φθ (X) Φθ (X) Φθ (X) Φθ (X) color-to-gray conversion. In SIGGRAPH Asia (2012).1,2 [ZSM]Z HANG X.,SIM T., MIAO X.: Enhancing photographs Finally, the gradient of the colormap appearing in the expressions with Near InfraRed images. In CVPR 2008.2 above depends on the choice of the colormap. For all the experiments using the colormap φi (x) defined in Section4, the deriva- [ZT10]Z HAO Y., TAMIMI Z.: Spectral image decolorization. In θ Proc. ISVC (2010).2 tion of the gradient is straightforward. In the experiments simulating i color blindness, the colormap is (φθ ◦Ψ)(x), whose gradient is given i T i as ∇θ (φθ ◦ Ψ)(x) = JΨ∇θ φθ (x), where JΨ is the Jacobian of Ψ. In Appendix A: Gradients of the cost function our experiments, transformation Ψ simulating the deficient observer is linear Ψ(x) = Ax, and thus J = AA. Let L = D − W be the image Laplacian as defined Ψ Φθ (X) Φθ(X) Φθ(X) in (2). We denote by |W| the number of non-zero elements in 10 Eynard et al. / Structure-preserving color transformations using Laplacian commutativity

RGB CIECIE−Y Y [GOTG05Gooch05 ][RGW05bRasche05 ][Grundland07GD07][Neumann07NvN07][SLTM08Smith08 ] LaplacianOurs

RWMS=2.13 RWMS=1.42 RWMS=2.92 RWMS=1.49 RWMS=4.24 RWMS=2.02 RWMS=0.94

2.4x10-3

0.4 2.13 / -1.16 1.42 / 1.03 2.34 / -0.44 1.49 / 0.57 4.62 / -1.55 2.07 / 0.31 0.94 / 1.24

CIE−Y Gooch05 Rasche05 Grundland07 Neumann07 Smith08 Ours

RWMS=2.21 RWMS=1.99 RWMS=1.42 RWMS=1.36 RWMS=2.22 RWMS=2.16 RWMS=1.19

3x10-3

0.5 2.14 / -0.68 1.93 / 0.40 1.43 / -1.24 1.33 / 1.34 2.30 / -1.35 2.09 / 0.11 1.16 / 1.42

CIE−Y Gooch05 Rasche05 Grundland07 Neumann07 Smith08 Ours

RWMS=4.87 RWMS=3.29 RWMS=3.49 RWMS=3.35 RWMS=5.20 RWMS=4.90 RWMS=1.24

6x10-3

1 5.06 / -0.06 3.47 / -0.73 3.71 / -0.08 3.46 / 1.73 5.44 / -2.29 5.07 / 0.37 1.36 / 1.06

CIE−Y Gooch05 Rasche05 Grundland07 Neumann07 Smith08 Ours

RWMS=9.24 RWMS=7.08 RWMS=7.14 RWMS=7.27 RWMS=10.65 RWMS=9.10 RWMS=4.31 9x10-3

3 9.33 / -0.15 7.02 / -0.04 7.32 / 0.36 7.32 / 1.98 10.03 / -2.17 9.19 / -0.92 4.37 / 0.94

CIE−Y Gooch05 Rasche05 Grundland07 Neumann07 Smith08 Ours

RWMS=1.21 RWMS=1.33 RWMS=1.10 RWMS=1.25 RWMS=2.69 RWMS=1.08 RWMS=0.93

2x10-3

0.2 1.26 / 0.50 1.37 / -0.64 1.12 / 0.17 1.24 / 0.21 2.72 / -1.97 1.12 / 0.73 0.98 / 1.00 CIE−Y Gooch05 Rasche05 Grundland07 Neumann07 Smith08 Ours

RWMS=0.97 RWMS=1.25 RWMS=0.96 RWMS=1.02 RWMS=1.64 RWMS=1.05 RWMS=0.85

16x10- 4

2 0.98 / 0.29 1.24 / -0.84 0.99 / 0.33 1.02 / 0.92 1.65 / -1.99 1.07 / 0.38 0.87 / 0.92

Figure 8: Decolorization experiment results. Left: original RGB image, right: grayscale conversion results. Rows 2, 5, : RWMS error images and mean RWMS (the smaller the better) / z-score (the larger the better) values. Our Laplacian colormap method performs the best in most cases. (Continues in next page) Eynard et al. / Structure-preserving color transformations using Laplacian commutativity 11

RGBCIE−Y CIE Y [Gooch05GOTG05][Rasche05RGW05b][Grundland07GD07][Neumann07NvN07][Smith08SLTM08] LaplacianOurs

RWMS=1.09 RWMS=1.12 RWMS=1.57 RWMS=1.15 RWMS=2.09 RWMS=1.17 RWMS=1.03

x 10−3

1.6

1.4

1.2

0.8

0.6

0.4

1.09CIE−Y / 0.60 1.12Gooch05 / 0.56 1.57Rasche05 / -1.22 1.15Grundland07 / 0.35 2.09Neumann07 / -1.78 1.17Smith08 / 0.49 1.03Ours/ 1.01

RWMS=0.81 RWMS=0.88 RWMS=1.41 RWMS=0.68 RWMS=11.02 RWMS=1.82 RWMS=0.59

x 10−3

2.5

1.5

0.5 0.81 / 1.13 0.88 / 0.05 1.41 / -1.24 0.68 / 1.32 11.02 / -2.14 1.82 / -0.15 0.59 / 1.04

Figure 8: (Continues from previous page) Decolorization experiment results. Left: original RGB image, right: grayscale conversion results. RWMS error images and mean RWMS (the smaller the better) / z-score (the larger the better) values. Our Laplacian colormap method performs the best in most cases.