Vision Research 46 (2006) 2659–2674 www.elsevier.com/locate/visres Gradient representation and perception in the early visual system—A novel account of Mach band formation Matthias S. Keil a,*, Gabriel Cristo´bal b, Heiko Neumann c a Computer Vision Center (Universitat Autono`ma), E-08193 Bellaterra, Spain b Instituto de O´ ptica (CSIC), Image and Vision Department, E-28006 Madrid, Spain c Universita¨t Ulm, Abteilung Neuroinformatik, D-89069 Ulm, Germany Received 23 January 2004; received in revised form 23 December 2005 Abstract Recent evidence suggests that object surfaces and their properties are represented at early stages in the visual system of primates. Most likely invariant surface properties are extracted to endow primates with robust object recognition capabilities. In real-world scenes, lumi- nance gradients are often superimposed on surfaces. We argue that gradients should also be represented in the visual system, since they encode highly variable information, such as shading, focal blur, and penumbral blur. We present a neuronal architecture which was designed and optimized for segregating and representing luminance gradients in real-world images. Our architecture in addition provides a novel theory for Mach bands, whereby corresponding psychophysical data are predicted consistently. Ó 2006 Elsevier Ltd. All rights reserved. Keywords: Gradient representation; Gradient segregation; Mach bands; Diffusion 1. Introduction neuronal representations are not simply luminance values as measured by retinal photoreceptors. In that case, for Luminance gradients on object surfaces constitute the example, changes in physical illumination would cause sig- integral elements of real-world scenes. We define luminance nificant changes in neuronal activity, and object recogni- gradients generally as smooth variations in intensity, in tion would strongly depend on illumination. Everyday contrast to surface edges showing step-like changes. Lumi- experience tells us that this is clearly not the case: any nance gradients may be engendered, for instance, through object can equally be recognized under bright sunlight, (i) curvature of object surfaces causing intensity variations and dim candle light, or whether or not a shadow is casted (shading), (ii) the limited depth-of-field representation of on it; in all, a profound invariance against illumination the eye’s passive optics (focal blur), which leads to blurring conditions is achieved. To realize robust object recognition, of objects which are not in the fixation plane, and (iii) soft the visual system extracts invariant scenic features, such as shadows (penumbral blur) (Elder & Zucker, 1998). color or lightness. Specifically, lightness constancy refers to Visual objects are composed of object surfaces. Object the observation that perceived surface reflectance does not recognition involves segregation of these surfaces from vary with illumination (e.g., Adelson, 2000), a property scene content, and their neuronal representation in the which is already established at an early neuronal level of visual system (e.g., Kinoshita & Komatsu, 2001; Komatsu, surface processing (MacEvoy & Paradiso, 2001). The latter Murakami, & Kinoshita, 1996; MacEvoy, Kim, & Parad- indicates that for the processing of surfaces, luminance gra- iso, 1998; Rossi, Rittenhouse, & Paradiso, 1996). However, dients are attenuated, To avoid spatial variations of surface representations with illumination, shadows, etc. (‘‘dis- * Corresponding author. counting the illuminant,’’ Grossberg & Todorovic´, 1988; E-mail address: [email protected] (M.S. Keil). Land, 1977). 0042-6989/$ - see front matter Ó 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.visres.2006.01.038 2660 M.S. Keil et al. / Vision Research 46 (2006) 2659–2674 Specular highlights represent another example of a lumi- (unless further mechanisms act to explicitly distinguish nance gradient. The occurrence of these highlights depends between surface and gradient information). In other words, on surface curvature. Moving an object normally changes typical multi-scale approaches mix information about sur- the positions of specular highlights at its surfaces, which faces and smooth gradients. should interfere with lightness constancy. Yet, this seems Filling-in models address the problem of ‘‘discounting not to be the case: the specular component of reflectance the illuminant’’ (Grossberg & Todorovic´, 1988; Land, is segregated from the diffuse one, and lightness is deter- 1977). To this end, information at sharp luminance discon- mined by the diffuse component of surface reflectance tinuities is used to interpolate surface appearance by means (Todd, Norman, & Mingolla, 2004). of a spatially isotropic diffusion mechanism (‘‘filling-in,’’ The aforementioned examples suggest that the visual e.g., Grossberg & Todorovic´, 1988; Grossberg & Pessoa, system suppresses luminance gradients for surface process- 1998; Grossberg & Howe, 2003; Kelly & Grossberg, ing (Biedermann & Ju, 1988; Marr & Nishihara, 1978). But 2000; Neumann, 1996; Pessoa & Ross, 2000). Since it has information on luminance gradients is unlikely to be fully been pointed out that filling-in mechanisms effectively discarded: for example, highlights provide information would average-out luminance gradients (Paradiso & about object shape (Lehky & Sejnowski, 1988; Ramachan- Nakayama, 1991), mechanisms were proposed which allow dran, 1988; Todd, 2003), and motion in the presence for representations of luminance gradients (Pessoa, Mingo- of specular highlights improves ‘‘shape-from-shading’’ lla, & Neumann, 1995; see also Grossberg & Mingolla, (Norman, Todd, & Orban, 2004). Luminance gradients 1987). However, the latter implies again a mixed represen- were also shown to be involved in the representation of tation of surfaces and gradients, as it is the case for the surface texture (Hanazawa & Komatsu, 2001). ‘‘traditional’’ multi-scale approaches (as mentioned above). Briefly summarizing, most luminance gradients do not As a solution to the amalgamation of information about interfere with lightness constancy, and it seems that the surfaces and gradients, we propose a two-dimensional (i.e., visual system uses them for computing different properties monocular) architecture for the segmentation and repre- (rather than discarding corresponding information). These sentation of smooth luminance gradients, called gradient properties include both object properties (such as shape), system, consistent with neurophysiological data and bio- but also scene properties (such as information about illumi- physical mechanisms, respectively. nation sources). Taken together, this is considered as evi- With the gradient system we address the problem of how dence that the processing of surfaces and gradients is to detect, segregate and represent luminance gradients at segregated at an early level in the visual system, implying an early level in the visual system, given foveal (i.e., high a neural representation for luminance gradients, which resolution) retinal responses. This problem exists because coexists with surface representations. Interactions between luminance gradients often extend over distances which both representations may occur in a dynamic fashion, and may exceed several times the receptive field sizes of foveal can be brought about by higher level or attentional ganglion cells. As opposed to multi-scale approaches, the mechanisms to increase robustness for object recognition. gradient system constitutes a mechanism to extract and Gradient representations can be considered as a further represent more global properties (luminance gradients) of low-level stimulus dimension, comparable with, for exam- visual scenes by analyzing strictly local information (foveal ple, feature orientation (as represented by orientation retinal responses). In this way a high spatial accuracy is maps). Just like the early representations of surfaces, gradi- preserved in gradient representations, as opposed to repre- ent representations should interact with brightness and senting gradients in responses of large filter kernels (e.g., lightness perception, respectively, and should be influenced Gabor wavelets, Gabor, 1946).1 We will see below that by other stimulus dimensions (e.g., depth information). by proceeding in this way we can successfully predict psy- However, at this level, neither surfaces nor gradients are chophysical data on Mach bands, and, with the same set associated yet with specific object representations. of parameter values, segregate luminance gradients from How are luminance gradients processed in other models real-world images. for brightness or lightness perception? We briefly consider multi-scale approaches and filling-in-based models. In mul- 2. Formulation of the gradient system ti-scale approaches, luminance gradients are encoded in the responses of large-scale filters (e.g. Koenderink & van The gradient system presented below is an improved Doorn, 1978; Koenderink & van Doorn, 1982; du Buf, version of the approach developed in Keil (2003) and Keil, 1994; du Buf & Fischer, 1995; Blakeslee & McCourt, Cristo´bal, and Neumann (2003). It consists of two subsys- 1997; Blakeslee & McCourt, 1999; Blakeslee & McCourt, tems. The first subsystem detects gradient evidence in a giv- 2001). Typically, band-pass filters are used in these en luminance distribution L. The second subsystem approaches to decompose a given luminance distribution. generates perceived gradient representations by means of Since brightness (and lightness) predictions are