Chapter 3 Color Constancy In the previous chapter I gave an introduction to fundamental theories of color vision. These theories referred to stimuli that were viewed in isolation. A cen- tral issue within this framework was the assignment of primary color codes to isolated light stimuli. Hitherto we discussed the color appearance of such stimuli only marginally. In this chapter we will extend the examined stimulus to com- plex spatial patterns that contain more than one light. We will see that the color appearance of light stimuli in this context is not determined by corresponding primary color codes. Rather, the color appearance of a given light depends on temporal and spatial characteristics of the stimulus. We can study the influence of the context on the color appearance of a stimulus when we present two differ- ent lights either sequentially or spatially (as a center-surround configuration) to the observer. The corresponding phenomena are called successive contrast and simultaneous contrast. In this chapter we will focus on a related phenomenon that refers to color appearance of fixed objects under varying lighting conditions. This phenomenon is known as color constancy. Loosely speaking, color constancy is described as the ability of visual systems to assign stable color appearance to a fixed object under changing illuminant conditions. 3.1 The Problem of Color Constancy 3.1.1 Introduction Probably the most important function of human color vision is object recognition. In this sense, the color appearance of an object should persist with changes of the illumination. We can experience this phenomenon every day but we rarely notice it. For example, a leaf appears green to us under bright sunny daylight and inside under tungsten light as well. This phenomenon which is called color constancy is an ability of our visual system. Research over the last few decades has shown that the human visual system achieves a high degree of color constancy over a 28 CHAPTER 3. COLOR CONSTANCY 29 Figure 3.1: Formation of the color signal that reaches the eye of the observer. wide range of illumination conditions. In fact, this feature of our visual system is very impressive for at least two reasons. First, the physical stimulus can change dramatically with changes of the illuminant. Second, the information that our visual system uses is very little and noisy. We will now characterize the physical stimulus which will be referred to as the color signal that reaches the eye from an object (Figure 3.1).1 This stimulus consists of two components, the incident illumination and characteristics of the object’s surface. For reasons of simplicity we will assume that the light source is spatially uniform and that the objects are flat and matte. This environment is often referred to as the Mondrian- or Flat World (Maloney, 1999; Brainard, 2004). The illumination can be described by its spectral power distribution E(λ). In studies on color constancy illuminants are often characterized as natural day- lights. Appendix A gives an introduction to properties of natural daylights and describes a method to approximate their spectral power distributions. Objects in the scene differ in the way they reflect incident light at different wavelengths. This characteristic of a surface is represented by the spectral reflectance func- tion S(λ). For each wavelength, S(λ) gives the fraction of incident light that is reflected from the surface. The color signal C(λ), that is the spectral power dis- tribution of the light which is reflected from the surface to the eye of the observer, can then be characterized by: C(λ)= E(λ)S(λ). (3.1) Clearly, in the color signal informations about the illumination and the re- flectance characteristics of the surface are already confounded. We are only in- terested in the latter when we wish to maintain persistent object recognition. 1The term color signal might be misleading as it is defined physically but not psychologically. I use this term here because its use is common in the color constancy literature (see for example Maloney, 1999). CHAPTER 3. COLOR CONSTANCY 30 light sources surface color signal receptor codes Tungsten Surface Signal - Tungsten Tungsten EHΛL SHΛL CHΛL 1 × ⇒ ⇒ Λ Λ Λ 400 800 400 800 400 800 L M S D10 Surface Signal - D10 D10 EHΛL SHΛL CHΛL 1 × ⇒ ⇒ Λ Λ Λ 400 800 400 800 400 800 L M S Figure 3.2: A fixed surface which appears gray under white daylight is rendered under two different illuminations: tungsten and bluish daylight D10. The first column shows the spectral power distributions of the two light sources. The second column gives the reflectance function of the surface. The resulting color signals and receptor codes (arbitrary units) are shown in columns three and four respectively. A comparison of both lines reveals that the color signals and the receptor codes differ strongly. Hence, our visual system has to somehow discount the effect of the illuminant from the color signal. A second problem arises if we consider the limitations of our visual system. When the color signal reaches the cone photoreceptors a dramatic reduction of information results. In (2.16) we denoted the sensitivity functions of the three types of cones with R1(λ), R2(λ), R3(λ). The receptor codes φi(C) of the color signal are given with: φi(C)= E(λ) S(λ) Ri(λ) dλ, i =1, 2, 3. (3.2) Zλ In general, for a fixed surface S and two different illuminants E1 and E2, the color signals and corresponding receptor codes differ. In Figure 3.2, the effects of two different light sources on the color signal from a surface which appears achromatic to the observer under white daylight are shown. When seen under a bluish daylight D10 the three cone photoreceptors are excited almost equally. Under a yellowish tungsten illuminant the L-cone is much more strongly excited than M- and S-cones. If the surface is seen only locally, these different patterns of cone excitations induce very different color impressions. If the two surfaces are embedded in larger scenes instead, we are probably able to recognize that we are observing one and the same surface under two different illuminants. On the other hand, identical color signals do not necessarily lead to identical color impressions. For example, imagine two different surfaces S1 and S2 which appear yellowish and bluish under white daylight respectively. The surface S1 is illuminated with a bluish and surface S2 with a yellowish light source so that identical receptor excitations from both surfaces result. Edwin Land showed in his impressive demonstrations of color constancy that when surfaces S1 and S2 are seen in corresponding contexts very different color appearances may result CHAPTER 3. COLOR CONSTANCY 31 (Land, 1977; see also Lotto & Purves, 2004). But if we present both surfaces in isolation to an observer they are perceptually indistinguishable for him. Hence it is impossible for the observer to decide if he sees a yellow surface under blue illumination or a blue surface under yellow illumination. A demonstration of this phenomenon is shown in Color Plate D.2 (Appendix D). From this example we can see that the confounding of information from the illuminant and the surface in the color signal can not be resolved without further information. Several suggestions have been made as to what kind of information the visual system uses to arrive at constant estimations of surface characteristics with changes of illumination. We will discuss some of these suggestions in the next section. At this point I will try to define the concept of color constancy more precisely. In the simple lighting model that was just introduced we considered only effects of the illuminant on the color signal but ignored other variables that may influence stable perception of surface colors such as interreflections between surfaces in the scene or atmospheric haze. In the literature color constancy is often defined as the ability of a visual system to discount only the effect of the illuminant from the color signal (e.g. B¨auml, 1995). Maloney (1999) has pointed out correctly that a definition of color constancy should take into account the aforementioned additional factors as well. According to Maloney (1999) . an observer has (perfect) color constancy precisely when the color ap- pearance assigned to a small surface patch by the visual system is com- pletely determined by that surface’s local spectral properties (Maloney, 1999, p. 389). It should be noted that human performance in this sense is not perfect and that color constancy is achieved only to a certain degree. Throughout this work we will mainly concentrate on the ability of the visual system to discount the effect of the illuminant from the color signal and ignore other potential variables. In this sense, the objective of a color constant system is to provide stable estimates of reflectance characteristics of surfaces across different illuminations. Often this estimation problem is broken down into two subproblems. First, the system has to get an estimate of the illumination using different cues from the viewed scene.2 Second, this estimate is used to determine certain intrinsic properties of the surface that are invariant with changes of the illuminant. In the next section I will describe this estimation process in more detail. 3.1.2 Simultaneous and Successive Color Constancy In recent years, the distinction between simultaneous and successive color con- stancy has evoked wider interest (B¨auml, 1997; Brainard, 1998). Let us first 2The term ‘estimate’ refers to the estimation of photoreceptor excitations that are related with the illuminant. In Section 3.2.3 we will discuss models where it is assumed that the illuminant is estimated only up to a scaling factor.