Color-opponent mechanisms for local hue encoding in a hierarchical framework Paria Mehrani,∗ Andrei Mouraviev,∗ Oscar J. Avella Gonzalez, John K. Tsotsos [email protected], [email protected], [email protected] , [email protected] Abstract L cone A biologically plausible computational model for color rep- M cone resentation is introduced. We present a mechanistic hier- archical model of neurons that not only successfully en- codes local hue, but also explicitly reveals how the con- tributions of each visual cortical layer participating in the process can lead to a hue representation. Our proposed model benefits from studies on the visual cortex and builds a network of single-opponent and hue-selective neurons. Local hue encoding is achieved through gradually increas- ing nonlinearity in terms of cone inputs to single-opponent cells. We demonstrate that our model's single-opponent neurons have wide tuning curves, while the hue-selective neurons in our model V4 layer exhibit narrower tunings, Cone responses as a function of x -4 -2 0 2 4 resembling those in V4 of the primate visual system. Our x simulation experiments suggest that neurons in V4 or later layers have the capacity of encoding unique hues. More- Figure 1: Spatial profile of a single-opponent L+M- cell. over, with a few examples, we present the possibility of The receptive field of this cells receives positive contri- spanning the infinite space of physical hues by combining butions from L cones and negative ones from M cones. the hue-selective neurons in our model. The spatial extent of these cone cells are different and Keywords| Single-opponent, Hue, Hierarchy, Visual determines the mechanism for this cell. Figure adapted Cortex, Unique Hues from [3]. 1 Introduction lengths. These wavelengths, categorized according to the cone sensitivities to short (S), medium (M) and long (L) The goal of this study is to introduce a biologically- wavelengths, form the LMS color space. Color encoding arXiv:1706.10266v2 [cs.CV] 20 Feb 2018 plausible computational color processing model that as in the visual system gets more complicated in higher lay- Brown [1] argues, helps in \understand[ing] how the ele- ers. Cones send feedforward signals to LGN, whose cells ments of the brain work together to form functional units are characterized by their opponent inputs from differ- and ultimately generate the complex cognitive behaviors ent cones [2]. Reid and Shapley demonstrated that LGN we study". For this purpose, we propose a hierarchical cells with single-opponent receptive fields receive oppo- model of neurons, each layer of which corresponds to one nent cone inputs that overlap in the visual field with dif- processing layer in the brain. We show that the behavior ferent extents. An example of such a spatial profile is il- of our model neurons matches that of the corresponding lustrated in Figure 1. They also mapped the contributions layers in the brain. Our hierarchical framework suggests a of cones to LGN receptive fields. Color opponent mecha- computational mechanism for color representation in the nisms were the basis of the \Opponent Process Theory" ventral stream. of Hering [4], in which he also introduced unique hues, Studies on the human visual system confirm that color those pure colors unmixed with other colors. In his the- encoding starts in a three-dimensional space with three ory, there are four unique hues: red, green, yellow and types of cone cells, each sensitive to a certain band of wave- blue that were believed to be encoded by cone-opponent processes. Later studies [5, 6, 7], however, confirmed that ∗The two authors contributed equally to this work. the cone-opponent mechanisms of earlier processing stages 1 do not correspond to Hering's red vs. green and yellow vs. blue opponent processes. In fact, they observed that the color coding in these regions is organized along the two dimensions of the MacLeod and Boyton [8] diagram. That is, along L vs M and S vs LM axes. In a study on color appearance in normal vision subjects, Webster et al. [9] found that while unique red is close to the +L axis, unique green, yellow and blue hues cluster around intermediate directions in the MacLeod-Boyton diagram. A similar study [10] suggested that the encoding of unique (a) Combined map of hue (b) Larger view of a cluster. hues, unlike the tuning of LGN neurons, needs higher or- patches in three clusters. der mechanisms such as a piecewise linear model in terms of cone inputs. Figure 2: Color map of V4 neurons in three clusters of Even though most studies agree LGN tunings do not patches (adapted from [20]). encode unique hues, there exists no such unanimous agree- ment about V1 and V2 tunings. Lennie et al. [11] observed similar chromatic selectivity as LGN neurons in V1 cells ters is shown in Figure 2(a). In Figure 2(b) a larger view and suggested a rectified sum of the three cone types de- of one of the clusters is shown. A close investigation on scribing the responses of V1 neurons. Hanazawa et al.[12] these patches revealed that their sequential representation found both linear and nonlinear V1 and V2 responses with follows the color order in the HSL space, hence a \rain- respect to the three cone types, with gradual increase in bow" of patches is observed. In addition, they observed percentage of nonlinearity from LGN to higher layers. A that each color activated 1-4 overlapping patches and that similar finding by Kuriki et al. [13] suggested that there neighboring patches are activated for similar hues. More- are selectivities to both intermediate and cone-opponent over, they found that patches in each cluster have the same hues in areas V1, V2, V3 and V4, with more diverse se- local visual field and that same-hue patches across clusters lectivity in V4 compared to the other areas. De Valois et have a great overlap in their local visual fields. Following al.[6] suggested perceptual hues can be encoded in V1, ob- these observations, they concluded that any hue in the vi- tained by combining LGN responses in a nonlinear fashion. sual field is represented by 1-4 patches, and different hues Wachtler et al.[14] found that the tunings of V1 neurons are represented by different multi-patch patterns. Com- are different from LGN, and that the responses in V1 are parable findings in V2 were reported in [21]. They also affected by context as well as remote color patches in both propose a distributed and combinatorial color representa- suppressive and enhancement manners. tion as the solution for encoding the large space of phys- In a set of studies [15, 3], three different types of cells in ical colors, given the limited number of neurons in each V1 were identified: luminance, color-luminance, and color- cortical color map. Representation of the color space in preferring neurons. Johnson et al. [15] suggested that cells globs was also studied by Bohon et al. [22] who found with a single-opponent receptive field encode local hue, that glob populations result in a uniform representation while double-opponent cells signal color contrast. of the color space with bias toward \warm" colors. They Color encoding in higher layers is less understood. also observed these that cells have nonlinear narrow tun- Millimeter-sized islands in extrastriate cortex, called ings. Similarly, Schein and Desimnone [23] and Zeki [24] globs, were shown to have cells with luminance-invariant remarked on narrow-band tunings in V4 cells. color tunings in the Macaque extrastriate cortex [16]. Ad- Transformation mechanisms from cone-opponent re- ditionally, cells in different globs represent distinct visual sponses to unique hues is still unclear. From responses field locations. Conway and Tsao [17] observed that neu- of IT neurons, Zaidi et al. [25] could accurately decode rons in a single glob have various color preferences and colors without any reference to unique hues. They also those in adjacent globs showed similar color tunings. Con- observed no significance of unique hues in human subjects. sequently, they suggested that cells in a glob are clus- However, Stoughton and Conway [26], similar to Zeki [24] tered by color preference and form the hypothesized color and Komatsu et al.[27], observed that even though neu- columns of Barlow [18]. In contrast with previous find- rons in PIT show selectivities to all hue angles in color ings, Namima et al. [19] found that not only the responses space, there are more neurons selective to those close to of the neurons were luminance-dependent, but also that unique hues. Hence, they suggest an encoding for unique the effect of luminance in responses of neurons in V4, AIT hues in higher layers, and rejected such representations in and PIT varies from one stimulus color to another. V1 and V2. The choice of stimuli for recordings in [26] A recent study [20] on V4 examined how perceptual was challenged by Mollon [28] commenting that it is still colors are mapped in this layer. They identified clusters unclear whether or not unique hues are represented in IT. of hue-selective patches, which they called \rainbows of Among all the attempts to understand the neural pro- patches". An example of an identified map of three clus- cesses for transformation from cone-opponency to percep- 2 tual colors, a number of computational models tried to of [20]. Our proposed network differs from that of [32] suggest mechanisms for this problem and other aspects as it explicitly models neurons in each of LGN, V1, V2, of color representation in higher areas. For example, the and V4 areas.