JOSA COMMUNICATIONS Communications are short papers. Appropriate material for this section includes reports of incidental research results, comments on papers previously published, and short descriptions of theoretical and experimental tech- niques. Communications are handled much the same as regular papers. Galley proofs are provided. Color constancy: surface color from changing illumination Michael D'Zmura Department of CognitiveSciences and Irvine Research Unit in Mathematical Behavioral Sciences, University of California, Irvine, Irvine, California 92717 Received September 5, 1991; revised manuscript received October 30, 1991; accepted November 13, 1991 Viewing the lights reflected by a set of three or more surfaces, a trichromatic visual system can recover three color-constant descriptors of reflectance per surface if the color of the surfaces' illuminant changes. This holds true for a broad range of models that relate photoreceptor, surface, and illuminant spectral properties. Changing illumination, which creates the problem of color constancy, affords its solution. INTRODUCTION spectral power distributions. Changing illumination both introduces the problem color constancy Land's best-known demonstrations of and per- of color constancy in- mits its solution. volve changing the illumination of a Mondrian display comprising many distinctly colored surfaces.",2 Although changing the chromatic properties of the Mondrian il- BILINEAR MODELS luminant alters the chromatic properties of the lights The following analysis depends reflected toward the viewer, the perceived colors of on knowledge of accu- rate finite-dimensional linear models for surface reflec- Mondrian surfaces hardly change at all. This constancy tances'15- and illuminants"-" and their relationship of surface color exhibits the ability of the visual system to photoreceptoral spectral sensitivity. Such models to derive, from reflected lights, accurate descriptors of exist and find much application in studies of color con- surface-color properties that do not depend on circum- 3 7 0 9 stancy. - 1 " -23 A particular stances of illumination. combination of photorecep- tors and models for ref lectances and illuminants provides Theoretical work has focused on the problem of estimat- a bilinear model,22 23 fundamental to the analysis, ing surface and illuminant chromatic properties that is in the derived as follows. case in which a set of surfaces is viewed under a single When an n-dimensional model for reflectances is used, illuminant.'' 0 Maloney and Wandell established a gen- a reflectance function R(A) that falls within the span of eral result for this situation6 : A trichromatic visual sys- the model is a linear combination of n linearly indepen- tem viewing surfaces under a single unknown illuminant dent basis functions Rj(A),j = 1, . , n: that is represented by three color descriptors'"- 3 can re- cover, at best, two color-constant descriptors per surface. n R(A) = >riR(A). The theory of color constancy rests incomplete, however, 1 (1) j= for neither surface-color percepts54 nor surface-color prop- erties' 5 ' 8 are described adequately by two-dimensional The n coefficients rj are the color-constant descriptors of models. Three descriptors per surface are required.1 2 surface-color properties provided by the model. An illu- The premise of the present study is that color constancy minant A(A) that falls within the span of an m-dimensional is not an issue for color perception unless the chromatic model for illuminants with linearly independent basis properties of illumination change. It is natural to ask functions Ai(A),i = 1,... , m, is expressed similarly: whether the lights from a set of surfaces viewed under two or more distinct illuminants, as in Land's demonstra- m A(A) = IaiAi(A). (2) tions, permit the visual system to estimate accurately i=1 both the unknown surface-color properties and the un- Light reflected from known illuminants. This is so; a trichromatic visual a surface with reflectance R(A) viewed under illuminant system can recover three accurate color descriptors for A(A) is the product of the two functions and has the following each unknown surface and for each illuminant when three model decomposition: or more surfaces are viewed under two or more lights in m n turn. This holds true for a variety of combinations of L(A) = A(A)R(A) = 2 ajrjA(A)Rj(A). (3) photoreceptor spectral sensitivities, three-dimensional i=1 j=1 models for describing the variation among surfaces' color- The quantum catch q of the kth photoreceptoral type in a constant reflectance properties, and three-dimensional p-chromatic system, k = 1,... ,p, is given by the integral models for describing the variation among illuminants' (over the visible spectrum) of the product of the kth mecha- 0740-3232/92/030490-04$05.00 ( 1992Optical Society of America JOSA Communications Vol. 9, No. 3/March 1992/J. Opt. Soc. Am. A 491 nism's spectral sensitivity Qk(A)and the light L(A): matic system that views s surfaces under v illuminants: 700 nm qk= Qk(A)L(A)dA d = E rt>Cja. (8) 400 nm j=1 700nm m n = |J400nmQk(A) [>> airjAi(A)Rj(A) dA. (4) The intuition behind this system of equations is like 400nm i=l j=l that underlying the results of Maloney and Wandell. In the case of a trichromatic system, that is, provided with To bring descriptors outside the integrals in Eq. (4), use two views of three surfaces (p = m = n = 3, = 2, s = the bilinear model matrices Bj, j = 1,..., n, with enti-ies 3), the three data vectors d, d , and d3 span a three- bjki given by the following integral: 2 dimensional subspace of R,6 the orientation of this sub- 700nm space in R6is determined uniquely by the two illuminants, biki =J Qk(A)Ai(A)Rj(A)dA. (5) and the position within this subspace of a data vector 400 nm from a particular surface corresponds uniquely to a par- Equation (4) in this form reads ticular reflectance. The aim is to invert this system [Eq. (8)] by using the n m data vectors and the known, model-dependent matrices C. qk = 2 rjbjkiai. (6) to provide both the s reflectances (represented by a ma- j=l i=1 trix R with entries rtj) and the v illuminants (represented This equation expresses the bilinear dependence of quan- by the vm-dimensional vector a) up to an arbitrary posi- tum catches on reflectance and illuminant descriptors.'9 2 tive scalar. With p = m = n = s, the matrix R holding the unknown reflectance descriptors and the matrices Cj are square; suppose further that these are nonsingular. RECOVERY FROM MULTIPLE VIEWS The reflectance matrix R then has an inverse with en- To include the effect of view, suppose that the sur- tries Pjt, where j = 1, . ., n, and t = 1, . , s, that satisfy face with descriptors rj is lit, in turn, by v illuminants s a,, w = 1,...,v, where a = [awl...awm] (T for trans- pjt dt = Ca. (9) pose). One may order the vp quantum catches to form t=l a data vector d = [ql... q1p,... .ql... q, T and the vm il- luminant descriptors to form an illumination vector a = Setting D = [d1... d,] and p = [pjl... pj,]T, the system [all. ...aim...avi... av.]T. Thedata are related to theillu- [Eq. (9)] entails Cj-1 Dpj = a. In terms of thepv x s ma- minants by a reflectance-weighted linear combination of trix Ej = Cj-'D, one has the n-matrix equations block-diagonal matrices Cj, in which each of v blocks along the diagonal is Bj: C = diag[Bj, ... ,Bj]. The re- Ejpj = a, j = 1,...,n. (10) lations among data, illuminants, and surface reflectance are then described by Taking differences for successive values of the indexj, one finds the n - 1 equations EhPh - Eh+l Ph+l = 0. To com- n bine these, define p = [lT* .PnT]T and construct the ma- d= >r 1Ca. (7) j=l trix F, which has n - 1 x n blocks, each of which matches the size of the Eh, namely, pv x s, by placing in the diago- Introducing s surfaces with reflectances rt, t = 1, . ., s, nal blocks F the matrices E>, placing in the adjacent provides s data vectors d, that hold the catches of ap-chro- blocks FU,,,, the matrices -EU+1, and placing zeros in Table 1. Tested Components Photoreceptor Illuminant Basis Reflectance Basis 2 5 6 5 7 Smith and Pokorny : L, M, S CIE daylight basis""'13: S0 , Sl S2 Cohen's basis' : I, II, III (natural surfaces,' 6 (human RGB; also CIE Judd-modified (daylight, Northern Hemisphere) Munsell chips' ) 1931 standard observer3) 26 8 Hurvich and Jameson : R-G, Y-B, VA Dixon's daylight basis2: E, V1,V 2 Parkkinen et al.'s basis' : 41, 02, 03 (human color-opponent model; (daylight, Southern -Hemisphere) (Munsell chips) also CIE 1931 standard observer'3) CIE 1964 supplemental observer3: CIE illuminants3: A, D65, F2 Fourier-series expansion on interval of visible xl0, Y1o,z10 (human large-field (indoor lighting: tungsten, daylight, and wavelengths: the first three functions'9 standard observer) fluorescent sources) Sony XC-711: R, G, B (CCD camera) Fourier-series expansion on interval of visible wavelengths: the first three functions'9 Sony XC-007: R, G, B (CCD camera) 492 J. Opt. Soc. Am. A/Vol. 9, No. 3/March 1992 JOSA Communications all the other entries. The single system that results is dimensional perception of a body undergoing motion.2 9 homogeneous: Likewise, while a single view of a Mondrian permits, without further assumptions,7 a two-dimensional repre- Fp= 0. (11) sentation of surface color,6 further views can lead, in principle, to the perception of stable, three-dimensional Thus the vector p, which is determined solely by the surface colors under changing lights.