Color Constancy: Surface Color from Changing Illumination

Home , Color constancy

JOSA COMMUNICATIONS Communications are short papers. Appropriate material for this section includes reports of incidental research results, comments on papers previously published, and short descriptions of theoretical and experimental tech- niques. Communications are handled much the same as regular papers. Galley proofs are provided.

Color constancy: surface color from changing illumination

Michael D'Zmura Department of CognitiveSciences and Irvine Research Unitin Mathematical Behavioral Sciences, University of California, Irvine, Irvine, California 92717

Received September 5, 1991; revised manuscript received October 30, 1991; accepted November 13, 1991 Viewing the lights reflected by a set of three or more surfaces, a trichromatic visual system can recover three color-constant descriptors of reflectance per surface if the color of the surfaces' illuminant changes. This holds true for a broad range of models that relate photoreceptor, surface, and illuminant spectral properties. Changing illumination, which creates the problem of color constancy, affords its solution.

INTRODUCTION spectral power distributions. Changing illumination both introduces the problem color constancy Land's best-known demonstrations of and per- of color constancy in- mits its solution. volve changing the illumination of a Mondrian display comprising many distinctly colored surfaces.",2 Although changing the chromatic properties of the Mondrian il- BILINEAR MODELS luminant alters the chromatic properties of the lights The following analysis depends reflected toward the viewer, the perceived colors of on knowledge of accurate finite-dimensional linear models for surface reflec- Mondrian surfaces hardly change at all. This constancy tances'15- and illuminants"-" and their relationship of surface color exhibits the ability of the visual system to photoreceptoral spectral sensitivity. Such models to derive, from reflected lights, accurate descriptors of exist and find much application in studies of color con- surface-color properties that do not depend on circum- 3 7 0 9 stancy. - 1 " -23 A particular stances of illumination. combination of photoreceptors and models for ref lectances and illuminants provides Theoretical work has focused on the problem of estimat- a bilinear model,22 23 fundamental to the analysis, ing surface and illuminant chromatic properties that is in the derived as follows. case in which a set of surfaces is viewed under a single When an n-dimensional model for reflectances is used, illuminant.'' 0 Maloney and Wandell established a gen- a reflectance function R(A) that falls within the span of eral result for this situation6 : A trichromatic visual sys- the model is a linear combination of n linearly indepen- tem viewing surfaces under a single unknown illuminant dent basis functions Rj(A),j = 1, . . , n: that is represented by three color descriptors'"- 3 can recover, at best, two color-constant descriptors per surface. n R(A) = >riR(A). The theory of color constancy rests incomplete, however, 1 (1) j= for neither surface-color percepts54 nor surface-color properties' 5 ' 8 are described adequately by two-dimensional The n coefficients rj are the color-constant descriptors of models. Three descriptors per surface are required.1 2 surface-color properties provided by the model. An illu- The premise of the present study is that color constancy minant A(A) that falls within the span of an m-dimensional is not an issue for color perception unless the chromatic model for illuminants with linearly independent basis properties of illumination change. It is natural to ask functions Ai(A),i = 1,... , m, is expressed similarly: whether the lights from a set of surfaces viewed under two or more distinct illuminants, as in Land's demonstra- m A(A) = IaiAi(A). (2) tions, permit the visual system to estimate accurately i=1 both the unknown surface-color properties and the un- Light reflected from known illuminants. This is so; a trichromatic visual a surface with reflectance R(A) viewed under illuminant system can recover three accurate color descriptors for A(A) is the product of the two functions and has the following each unknown surface and for each illuminant when three model decomposition: or more surfaces are viewed under two or more lights in m n turn. This holds true for a variety of combinations of L(A) = A(A)R(A) = 2 ajrjA(A)Rj(A). (3) photoreceptor spectral sensitivities, three-dimensional i=1 j=1 models for describing the variation among surfaces' color- The quantum catch q of the kth photoreceptoral type in a constant reflectance properties, and three-dimensional p-chromatic system, k = 1,... ,p, is given by the integral models for describing the variation among illuminants' (over the visible spectrum) of the product of the kth mecha- 0740-3232/92/030490-04$05.00 ( 1992Optical Society of America JOSA Communications Vol. 9, No. 3/March 1992/J. Opt. Soc. Am. A 491 nism's spectral sensitivity Qk(A)and the light L(A): matic system that views s surfaces under v illuminants:

700 nm qk= Qk(A)L(A)dA d = E rt>Cja. (8) 400 nm j=1 700nm m n = |J400nmQk(A) [>> airjAi(A)Rj(A) dA. (4) The intuition behind this system of equations is like 400nm i=l j=l that underlying the results of Maloney and Wandell. In the case of a trichromatic system, that is, provided with To bring descriptors outside the integrals in Eq. (4), use two views of three surfaces (p = m = n = 3, = 2, s = the bilinear model matrices Bj, j = 1,..., n, with enti-ies 3), the three data vectors d, d , and d3 span a three- bjki given by the following integral: 2 dimensional subspace of R,6 the orientation of this sub- 700nm space in R6is determined uniquely by the two illuminants, biki =J Qk(A)Ai(A)Rj(A)dA. (5) and the position within this subspace of a data vector 400 nm from a particular surface corresponds uniquely to a par- Equation (4) in this form reads ticular reflectance. The aim is to invert this system [Eq. (8)] by using the n m data vectors and the known, model-dependent matrices C. qk = 2 rjbjkiai. (6) to provide both the s reflectances (represented by a ma- j=l i=1 trix R with entries rtj) and the v illuminants (represented This equation expresses the bilinear dependence of quan- by the vm-dimensional vector a) up to an arbitrary posi- tum catches on reflectance and illuminant descriptors.'9 2 tive scalar. With p = m = n = s, the matrix R holding the unknown reflectance descriptors and the matrices Cj are square; suppose further that these are nonsingular. RECOVERY FROM MULTIPLE VIEWS The reflectance matrix R then has an inverse with en- To include the effect of view, suppose that the sur- tries Pjt, where j = 1, . . ., n, and t = 1, . . , s, that satisfy face with descriptors rj is lit, in turn, by v illuminants s a,, w = 1,...,v, where a = [awl...awm] (T for trans- pjt dt = Ca. (9) pose). One may order the vp quantum catches to form t=l a data vector d = [ql... q1p,... .ql... q, T and the vm illuminant descriptors to form an illumination vector a = Setting D = [d1... d,] and p = [pjl... pj,]T, the system [all. ...aim...avi... av.]T. Thedata are related to theillu- [Eq. (9)] entails Cj-1 Dpj = a. In terms of thepv x s ma- minants by a reflectance-weighted linear combination of trix Ej = Cj-'D, one has the n-matrix equations block-diagonal matrices Cj, in which each of v blocks along the diagonal is Bj: C = diag[Bj, ... ,Bj]. The re- Ejpj = a, j = 1,...,n. (10) lations among data, illuminants, and surface reflectance are then described by Taking differences for successive values of the indexj, one finds the n - 1 equations EhPh - Eh+l Ph+l = 0. To com- n bine these, define p = [lT* .PnT]T and construct the ma- d= >r 1Ca. (7) j=l trix F, which has n - 1 x n blocks, each of which matches the size of the Eh, namely, pv x s, by placing in the diago- Introducing s surfaces with reflectances rt, t = 1, . ., s, nal blocks F the matrices E>, placing in the adjacent provides s data vectors d, that hold the catches of ap-chro- blocks FU,,,, the matrices -EU+1, and placing zeros in

Table 1. Tested Components Photoreceptor Illuminant Basis Reflectance Basis 2 5 6 5 7 Smith and Pokorny : L, M, S CIE daylight basis""'13: S0 , Sl S2 Cohen's basis' : I, II, III (natural surfaces,' 6 (human RGB; also CIE Judd-modified (daylight, Northern Hemisphere) Munsell chips' ) 1931 standard observer3)

26 8 Hurvich and Jameson : R-G, Y-B, VA Dixon's daylight basis2: E, V1,V 2 Parkkinen et al.'s basis' : 41, 02, 03 (human color-opponent model; (daylight, Southern -Hemisphere) (Munsell chips) also CIE 1931 standard observer'3)

CIE 1964 supplemental observer3: CIE illuminants3: A, D65, F2 Fourier-series expansion on interval of visible xl0, Y1o,z10 (human large-field (indoor lighting: tungsten, daylight, and wavelengths: the first three functions'9 standard observer) fluorescent sources)

Sony XC-711: R, G, B (CCD camera) Fourier-series expansion on interval of visible wavelengths: the first three functions'9

Sony XC-007: R, G, B (CCD camera) 492 J. Opt. Soc. Am. A/Vol. 9, No. 3/March 1992 JOSA Communications

all the other entries. The single system that results is dimensional perception of a body undergoing motion.2 9 homogeneous: Likewise, while a single view of a Mondrian permits, without further assumptions,7 a two-dimensional repre- Fp= 0. (11) sentation of surface color,6 further views can lead, in principle, to the perception of stable, three-dimensional Thus the vector p, which is determined solely by the surface colors under changing lights. reflectance descriptors, lies in the kernel of F. If the di- mension of ker(F) is one, then the reflectances are determined uniquely [by Eq. (11)],as are the illuminants a [by ACKNOWLEDGMENTS Eq. (10)], up to an arbitrary positive scalar [by Eq. (8)]. I thank Geoff Iverson, Jack Yellott, Glenn Healey, Don The kernel of F is readily computed by using a singular- Hoffman,, Steve Shevell, and Brian Wandell for their help. value decomposition. 24

REFERENCES SIMULATION 1. E. H. Land, "Recent advances in retinex theory and some Whether a particular combination of photoreceptors and implications for cortical computations: color vision and the basis functions for illumination and reflectance actually natural image," Proc. Natl. Acad. Sci. USA 80, 5163-5169 (1983). works is a purely numerical matter. In Table 1 are listed 2. E. H. Land, "Recent advances in retinex theory," Vision Res. practical choices of photoreceptoral spectral sensitivities 26, 7-21 (1986). and basis functions for illuminants and reflectances. 3. P. Sllstrom, "Colour and physics: some remarks con- The 60 distinct combinations formed by choosing all pos- cerning the physical aspects of human colour vision," Univ. sible triples of receptors and illuminant and reflectance Stockholm Inst. Phys. Rep. 73-09 (1973). 4. M. H. Brill, 'A device performing illuminant-invariant as- bases in Table 1 all function robustly, as was revealed by sessment of chromatic relations," J. Theor. Biol. 71, 473-478 simulation of surface-color and illuminant-descriptor (1978). recovery in 128 randomly generated test cases per com- 5. G. Buchsbaum, 'A spatial processor model for object colour bination, through numerical determination2 4 of ker(F) perception," J. Franklin Inst. 310, 1-26 (1980). 6. L. T. Maloney, and B. A. Wandell, "Color constancy: a [Eq. (11)]. As expected, the technique fails if the two illu- method for recovering surface spectral reflectance," J. Opt. minants are linearly dependent or if the three surface re- Soc. Am. A 3, 29-33 (1986). flectances are linearly dependent. 7. M. D'Zmura and P. Lennie, "Mechanisms of color constancy," A variation suggested by Yellott27 handles situations J. Opt. Soc. Am. A 3, 1662-1672 (1986). in which scene illumination does not 8. A. Hurlbert, "Formal connections between lightness algo- change: One ef- rithms," J. Opt. Soc. Am. A 3, 1684-1693 (1986). fectively obtains a second view of a scene by imaging it 9. H.-C. Lee, 'A method for computing the scene illuminant through a color filter. Two pictures, taken with and chromaticity from specular highlights," J. Opt. Soc. Am. A 3, without a filter of known transmission properties, provide 1694-1699 (1986). the input to the altered recovery procedure {the two blocks 10. B. V. Funt, M. S. Drew, and J. Ho, "Color constancy from mu- tual reflection," in a Cj matrix [Eq. (8)] Int. J. Comp. Vis. 6, 5-24 (1991). differ}. The technique was simu- 11. D. B. Judd, D. L. MacAdam, and G. Wyszecki, "Spectral dis- lated successfully by using the 24 camera-based combina- tribution of typical daylight as a function of correlated color tions of Table 1 in conjunction with transmission spectra28 temperature," J. Opt. Soc. Am. 54, 1031-1040 (1964). for broadband (Corning 1-57 glass), orange (Schott FG9 12. E. R. Dixon, "Spectral distribution of Australian daylight," glass), and blue (Kodak Wratten J. Opt. Soc. Am. 68, 437-450 (1978). 80C) filters. 13. G. Wyszecki and W S. Stiles, Color Science. Concepts and Methods, Quantitative Data and Formulae, 2nd ed. (Wiley, New York, 1982). DISCUSSION 14. R. S. Hunter, The Measurement of Appearance (Wiley, New York, 1975). A trichromatic visual system can implement color con- 15. E. L. Krinov, "Spectral reflectance properties of natural for- stancy by comparing quantum catches from three (or mations," NRC Tech. Transl. TT-439 (1953). more) surfaces viewed under two (or more) lights. The 16. J. Cohen, "Dependency of the spectral reflectance curves of change in illumination can occur temporally or the Munsell color chips," Psychon. Sci. 1, 369-370 (1964). spatially 17. L. T. Maloney, "Evaluation of linear models of surface spec- or by some combination of the two. The modest require- tral reflectance with small numbers of parameters," J. Opt. ment is that correspondence between surfaces across two Soc. Am. A 3, 1673-1683 (1986). views be maintained. Although a dichromatic visual sys- 18. J. P. S. Parkkinen, J. Hallikainen, and T. Jaaskelainen, tem cannot take advantage of changing illumination in "Characteristic spectra of Munsell colors," J. Opt. Soc. Am. A 6, 318-322 (1989). this way, visual systems with more than three photorecep- 19. B. A. Wandell, "The synthesis and analysis of color images," toral types are like trichromatic ones: Successful simu- IEEE Trans. Pattern Anal. Mach. Intell. PAMI-9, 105-116 lations with four- and five-dimensional systems suggest (1987). that a p-chromatic system, forp 3, can use two views to 20. S. Tominaga and B. A. Wandell, "Standard surface- recover p color-constant reflectance model and illuminant estimation," J. Opt. Soc. descriptors per surface. Am. A 6, 576-584 (1989). At the heart of an extended analogy between color con- 21. P. Lennie and M. D'Zmura, "Mechanisms of color vision," stancy and structure from motion lies the following. Crit. Rev. Neurobiol. 3, 333-400 (1988). While a single view of points on a three-dimensional 22. D. Brainard and B. A. Wandell, 'A bilinear model of the illu- body (e.g., the corners of a cube) often leads to a flat, minant's effect on color appearance," in Computational Models of Visual Processing, M. Landy and J. A. Movshon, two-dimensional precept, further views of these points eds. (Massachusetts Institute of Technology, Cambridge, provided by the rigid motion of the body lead to the three- Mass., 1991), pp. 171-186. JOSA Communications Vol. 9, No. 3/March 1992/J. Opt. Soc. Am. A 493

23. D. H. Marimont, B. A. Wandell, and A. B. Poirson, "Predict- 26. L. M. Hurvich and D. Jameson, "Some quantitative aspects of ing receptor responses from low-dimensional descriptions of an opponent-colors theory. II. Brightness, saturation, and surface and illuminant spectra," in Society for Information hue in normal and dichromatic vision," J. Opt. Soc. Am. 45, Display 1991 Digest of Technical Papers (Society for Infor- 602-616 (1955). mation Display, Playa del Rey, Calif., 1991). 27. J. I. Yellott,Jr., University of California, Irvine, Irvine, Calif. 24. W H. Press, B. P. Flannery, S. A. Teukolsky, and W T. 92717 (personal communication, 1991). Vetterling,Numerical Recipes in C. The Art of Scientific 28. G. Wyszecki and W S. Stiles, Color Science. Concepts and Computing (Cambridge U. Press, New York, 1988). Methods, Quantitatiue Data and Formulas (Wiley,New York, 25. V C. Smith and J. Pokorny, "Spectral sensitivity of the foveal 1967). cone photopigments between 400 and 500 nm," Vision Res. 29. S. Ullman, The Interpretation of Visual Motion (Massa- 15, 161-171 (1975). chusetts Institute of Technology, Cambridge, Mass., 1979).