University of Nevada, Reno Cortical Representation of Illusory and Surface Color a Dissertation Submitted in Partial Fulfillment

University of Nevada, Reno

Cortical Representation of Illusory and Surface Color

A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Psychology

By Andrew John Coia Dr. Michael A. Crognale/ Dissertation Advisor

August, 2016

THE GRADUATE SCHOOL

We recommend that the dissertation prepared under our supervision by

ANDREW COIA

Entitled

Cortical Representation Of Illusory And Surface Color

be accepted in partial fulfillment of the requirements for the degree of

DOCTOR OF PHILOSOPHY

Michael Crognale, Advisor

Michael Webster, Committee Member

Gideon Caplovitz, Committee Member

Grant Mastick, Committee Member

Thomas Nickles, Graduate School Representative

David W. Zeh, Ph. D., Dean, Graduate School

August, 2016

Abstract

In order to see the world as we do our visual system constantly performs many amazing feats that are largely unknown to us, the viewers. In order to create and update a seamless and coherent representation of the world, our eyes and brain have to deal with many obstacles that could interfere with object recognition such as occluding blood vessels in our eyes, layers of occluding retinal neurons, the blind spot that is devoid of photoreceptors, objects and shadows occluding other objects, and retinal inhomogeneities. Ultimately, we obtain information that encodes relative brightness and colors that allow us to recognize different objects and surfaces.

How is it that we obtain the information to assign color to object surfaces? It has been hypothesized that we predominantly extract information about the color of an object from the spectral contrast at the edges and fill in the remaining areas (if the edge information is consistent with that from a uniform surface). These ideas have been put forth to explain striking visual illusions such as the Cornsweet Illusion, neon color spreading, and the watercolor effect.

All of these illusions exist in both color and brightness domains. Thus, even though it may not be intuitively obvious, edges play a key role in our final percept of what color we see when viewing an object. It is said that we ‘perceptually fill in’ color from edges similarly to how our visual system fills in the blind spot in our eye. Of course it should be noted that spectral information may also be available from within the regions away from the edges and this information may also play an important role in the perception of surface color. The theory that edge information alone determines color and that internal surface information is discarded implies that there is no difference in neural computation of physical surface color and edge

ii induced color spreading. Initial studies of this question suggest that illusory colors and actual surface colors may in fact show important differences. One goal of the present study is to employ both psychophysics and electrophysiology to determine under what conditions illusory colors and surface colors are differentiated to provide insights into the fundamental processes involved in surface color perception.

If filling-in is indeed important for surface color perception, then how might color filling- in occur? One possibility is that information from the edges might be propagated in a feed- forward manner that passively spreads until another edge or contradicting information is encountered. Another possibility is that edge information could be relayed to higher order form centers and then surface colors reconstructed from feedback from those higher regions. The question of feed-forward vs. feedback mechanisms of perception has been debated by scientists since at least the time of Hering and Helmholtz, Hering favoring a bottom up interpretation while Helmholtz argued that our visual system performs ‘unconscious inferences.’ It is currently appreciated that a relatively sparse sampling of visual information from the world results in a rich visual percept, supporting the idea that much of our vision is actually reconstructed from past experience. Another goal of the present study is to apply electrophysiological methods

(hdEEG) to determine the relative importance of feed-forward and feedback mechanisms in surface color perception. This work expands on previous research which developed a method of measuring the watercolor illusion with single channel VEP.

This dissertation provides a literature review of the background and significance of the problems, presents preliminary data, outlines the series of proposed experiments, and lastly the results of the proposed experiments.

iii

Acknowledgements

I would like to thank my parents Anthony and Patricia-Radosevich Coia for supporting me through my graduate studies as well as my friends, other family members, and colleagues in the Cognitive and Brain Science program at the University of Nevada Reno.

Thank you to Michael Crognale, John Erik Vanston, Gideon Caplovitz, and Annica

Aguzzi for helping with the design, implementation, execution, and analysis of experiments.

This work was supported by the Bilinski Fellowship and use of equipment maintained by the Centre of Biomedical Research Excellence (COBRE).

Table of Contents

Abstract______i

Acknowledgments ______iii

Table of Contents ______iv

List of Tables ______vi

List of Figures ______vi

Chapter 1: Introduction

General Goals______1

Background and Significance______3

Taxonomies of Filling In______4

How and Why Does Filling in Happen? ______5

Chromatic Induction______7

Lightness, Brightness, and Color Constancy______8

Temporal Properties of Filling In______9

Spatial Frequency and Natural Scene Statistics______11

Afterimages, Adaptation, and Edges______12

Neurons and Visual Coding______13

Neurons and Filling In______14

Cortical Modularity of Color and Form______16

Color Centers in the Cortex______17

fMRI, Retinotopic Maps, and Filling In______19

The Visual Evoked Potential (VEP)______23

Chromatic Visual Evoked Potentials______26

Attentional Feedback in the VEP______27

Physiological Correlates of Watercolor Effect______28

Chapter 2: Preliminary Data______32

Materials and Methods______32

Subjects______33

Results (onset, electrodes) ______33

Results (SSVEPs, electrodes) ______35

Estimation of Cortical Activity______37

Results (onset, sources) ______39

Results: (SSVEPs, sources) ______47

Case Study______50

Conclusions______52

Chapter 3: Proposal______54

Proposed Studies______54

Behavioral Psychophysics______54

VEPs______55

Chapter 4: High Density Investigation of the Chromatic Visual Evoked Potential______57

Psychophysical Procedures______59

Participants______61

Results: Psychophysics______62

HdEEG Recordings______62

Stimulus Presentation______63

Data Pre-processing ______64

Results: hdEEG______64

Source Localization______66

SSVEPs______71

Chapter 5: High Density Investigation of the Watercolor Illusion______74

Psychophysics______76

Results: Psychophysics ______77

Results: EEG______78

Watercolor Source Localization______80

Watercolor SSVEPs______81

Discussion______83

Case Study: Deuteranomoly______83

General Discussion______84

References______87

List of Tables

Table 1: Chromaticities for Gabors______60

Table 2: Chromaticities for Watercolor Stimuli______76

List of Figures

Figure 1: Examples of illusions ______3

Figure 2: How filling in may be achieved by the brain ______6

Figure 3: SSVEP stimuli______28

Figure 4: Fundamental and 2nd Harmonic______30

Figure 5: Average onset topography ______34

Figure 6: Electrode Diagram ______35

Figure 7: SSVEP Topography______36

Figure 8: Average SSVEP amplitudes. ______37

Figure 9: Onset Cortical Activity______40

Figure 10: Regions of Interest______41

Figure 11: Latencies and Amplitudes for Regions of Interest.______42

Figure 12: T tests of illusion vs. control onset waveforms for ROIs______45

vii

Figure 13: T tests for filled control vs. control onset waveforms for ROIs______46

Figure 14: Cortical FFT amplitudes ______47

Figure 15: SSVEP 2 Hz amplitudes ______49

Figure 16: SSVEP 4 Hz amplitudes ______50

Figure 17: Polar Plots: illusion to filled control ______51

Figure 18: Polar plots: illusion and align/misalign conditions______52

Figure 19: Multi Gabor Fields ______59

Figure 20: Average heterochromatic flicker photometry matches ______62

Figure 21: Average waveforms for multi Gabor crVEPs______64

Figure 22: Timeline of topographic activity for crVEPs______65

Figure 23: Regions of Interest in Source Localization Study______66

Figure 24: Average source estimates for crVEPs ______68

Figure 25: Comparison of summed activity in early and ventral visual ROIs______69

Figure 26: Average SSVEP waveforms ______70

Figure 27: Significance level of T-Circ test results______71

Figure 28: Estimates of 2nd harmonic SSVEP ROI activity______72

Figure 29: Watercolor stimuli ______74

Figure30: Average psychophysical matches across observers ______77

Figure 31: Average onset responses across observers to watercolor stimuli ______78

Figure 32: analysis of latency and amplitude components of watercolor onset VEPs__78

Figure 33: Average source localization for watercolor stimuli______79

Figure 34: Average responses found in ventral visual area 3 (V3v) ______80

Figure 35: Average SSVEP watercolor responses______81

Figure 36: Watercolor SSVEP source estimates______82

Figure 37: Comparison of Deutan subject to group average______83

Chapter 1: Introduction

GENERAL GOALS

The aim of these studies is to test theories about perceptual filling in by observing both behavioral and electrophysiological responses in humans to physical surface color as well as stimuli that induce perceptual filling in and comparing them to control stimuli which have edges but no induced surface color.

AIM 1: Comparison of the strength of illusory hue and brightness

The watercolor illusion exists in both chromatic and achromatic (brightness) forms, and previous studies show that the illusory color can be matched to physical color. The chromatic matches generally produce a desaturated hue of the inducing contour while the achromatic matches have reduced luminance contrast compared to the inducer. Are there differences in perceptual magnitude between chromatic and achromatic illusions? Since the watercolor illusion can be induced in achromatic as well as chromatic form, this study aims to contrast match achromatic and chromatic inducing contours and apply those matches to create chromatic and achromatic watercolor illusions. To measure the strength of the induction, the illusions will be matched to physical colors.

AIM 2: Comparison of cortical activity during chromatic and achromatic visual evoked potentials

It has been known for some time that robust evoked potentials can be elicited from isoluminant chromatic stimuli. It is currently unknown whether the neural generators of the chromatic visual evoked potential (VEP) are the same as those of achromatic VEPs. This aim sets out to contrast match isoluminant chromatic stimuli (Gabor patches) to achromatic stimuli.

HdEEG will be used to record responses to these stimuli and dipole modeling techniques will be used to estimate the cortical sources of chromatic and achromatic visual responses.

AIM 3: Cortical activity during real and filling in of color and brightness

The aim here is to determine whether there are characteristic differences in cortical activity recorded from the visual response to physical and illusory surface color and brightness.

Are surface colors (real and illusory) generated in early visual areas in the brain, immediately after retinal signals are delivered? Or does the early visual system only respond to edges, with surfaces being filled in later on in image construction? In addition, are there differences in processing of real surface colors versus illusory ones as seen in the watercolor illusion?

The matched chromatic and achromatic stimuli measured for each individual in the behavioral experiments will be used to evoke responses for high density EEG (hdEEG) recordings. HdEEG allows for an analysis of activity measured at the level of the electrodes which can also be used to estimate the activity of different areas in the brain using dipole modeling. The responses to illusory color spreading and their matched physical counterparts will be compared to a control stimulus which has contours of the same color and luminance but not surface color. In addition to looking at neural activity of edge induced vs. actual physical surface color, we hope to compare color vs. brightness evoked responses for different regions of interest in the brain. Comparisons will be made in both distribution of the response as well as the time course of the response in different regions. These data are particularly interesting since the source of the different components of the chromatic VEP have not been determined with certainty.

Background and Significance

In order to construct a coherent visual scene, our visual system integrates many different cues such as borders and shading determining the brightness and color of objects in our surroundings. These cues help us recognize objects in different contexts. In certain instances different cues can send conflicting messages resulting in ambiguous percepts, as illustrated in the infamous ‘dress’ picture that was popularized on the internet recently. Some cues can be exploited in color and brightness illusions, in order to make the appearance of color experienced where really there should be none. Many brightness and color illusions (see figure

1) exploit the role of edges in our determining the color of an area, and it is said that we perceptually ‘fill in’ the surface color of objects from edges, causing shifts in appearance of brightness and color.

Figure 1: Examples of illusions where physical brightness/color differs from perceived brightness/color. A: The

Kanizsa triangle: an illusory white triangle is seen occluding three black circles and an outline of another triangle and appears brighter than the background. The perceived edges of the white triangle are created by the contextual cues.

B: Simultaneous Contrast: The surroundings of the two smaller identical grey squares causes the one in the black

square to look slightly brighter than the one in the lighter square. C: The Craik Cornsweet O’Brien Effect (CCOE): A sharp luminance edge going down the middle of the square causes the right side of the square to appear brighter than the left side. By covering the border region in the middle the illusion goes away. D: White’s Effect: The small grey rectangles can change brightness whether they are placed on the white or black tracks. E: The watercolor effect

(WCE): A thin black line traces a thin orange line which causes a bleeding of orange into the interior of each of the columns. F: Neon color spreading: The yellow lines in the middle are continuous with the surrounding black lines, making it appear as if there is a yellow filter covering that area, and we perceive a yellowish tint over the whole region.

Taxonomies for Filling In

Other examples of perceptual filling in involve filling of the blind spot (see Murakami,

2008, and Komatsu, 2006), filling in of real (Gerrits & Vendrik, 1970) and artificially created

(Ramachandran and Gregory, 1991) scotomas, peripheral fading (Troxler, 1804), and retinal stabilization (Krauskopf, 1963). Characteristics such as color, brightness, texture, motion, and depth can be filled in. In a study of filling in of both texture and color, it was shown that first color was filled in and then texture (Ramachandran and Gregory, 1991). This implies active processing of separate mechanisms.

Another type of filling in exists where we perceive illusory contours, delineating an object that is being partially occluded (Kanizsa (1976), figure 1A). Illusory contours can also induce color spreading (neon color spreading figure 1F) (see Bressan et al., 1997). This implies that illusory boundaries may be created that later give rise to surface properties. Although all these visual phenomena can be considered ‘filling in,’ their individual underlying neural substrates may be different.

Several authors have created taxonomies categorizing these different types of filling in

(Pessoa et al., 1998, Weil & Rees, 2011). Pessoa et al. designate different types of filling in into modal vs. amodal completion and boundary vs. featural completion. Amodal and modal completion can be seen in the Kanizsa triangle (figure 1A). Modal completion would be the illusory white triangle seen in the foreground, while amodal completion involves inferring that the black pacmen surrounding the triangle are partially occluded circles. Boundary vs. surface completion is also present in the Kanizsa triangle: the percept of a continuous boundary around the white triangle (boundary) causes the surface of the triangle to appear brighter. Weil and

Rees (2011) argue that filling in phenomena should be separated into groups for delayed vs. instantaneous and stimulus dependent vs. stimulus independent. While filling in at the blind spot is stimulus independent (it always happens no matter what is being looked at), the illusions discussed in this proposal are stimulus dependent (in order for color to spread in the watercolor illusion, two lines must be bordering each other), and both are instantaneous in the sense that filling in happens quickly. Examples of delayed filling in are retinal stabilization and Troxler fading (when eye movements are suppressed, objects fade from view), which can take 10-15 seconds to fade.

How and why does filling in happen?

A debate exists about whether the filling in processes is an active neural process or if the visual system is just ignoring the absence of sensory input. Some argue that filling in shouldn’t be thought of as the brain ‘painting’ over a surface, but it may just be ‘ignoring the absence’ of contradictory information. Dennet (2002) brings up the example of neon color spreading illusion and points out the question of whether this ‘mislabeling’ of color in an area is due to a spreading activity or if there is a ‘single label’ of color applied to the area. If the brain

were applying a single color label over an area of space which the retina (or retinotopic maps such as V1) have numerous receptors (or receptive fields), this could be thought of as some form of neural data compression. How does the brain accomplish filling in?

Figure 2: Possible explanations of how filling in may be achieved by the brain. A: Feed forward propagation of visual information from lower to higher levels. B: Lateral/horizontal connections combine information to infer context intermediate location. C: Feedback from higher level back to lower level causes filling in of intermediate location. (adapted from Pessoa et al., 1998)

Figure 2 illustrates 3 hypothetical models to account for the neural basis of perceptual filling in (see Pessoa et al., 1998, and Spillmann & Werner, 1996). The circles represent neuron receptive fields, and the arrows going from level 1 to level 2 in A represent feed forward activity, where filling in could be completed in a bottom up manner. Figure 2 B represents an alternate mechanism of horizontal connections within a single cortical area, and Figure 2 C illustrates a feedback mechanism where after information is brought from level 1 to level 2, level 2 feeds back information to level 1. The current proposal aims to test these theories using electrophysiological responses to the watercolor illusion.

In addition to these different neural models accounting for filling in, there is also a question of whether the induced area is isomorphically filled in (neurons with receptive fields

within the area are stimulated during filling in) or whether filling in happens on a more symbolic, or representational level, without the need to actually stimulate neurons within the retinotopic maps of the early visual cortex. These are referred to as the isomorphic and symbolic theories, respectively.

Chromatic Induction

Simultaneous contrast (figure 1 B) occurs with stimuli of low spatial frequencies, where the inducing and induced areas both cover a rather broad range but have a sharp high frequency border. This causes the test patch to contrast with its surrounding area. In most of the other illusions shown in figure 1, the opposite sensation of assimilation, or color spreading, occurs.

The test areas appear more similar to the inducing surrounding (the area adjacent to the luminance increment will appear brighter). Researchers have shown that one can cause a shift from contrast to assimilation by increasing the spatial frequency of a square wave grating

(Helson, 1963). Both contrast and assimilation are forms of chromatic induction.

Although assimilation and contrast are opposing percepts, many of the assimilation illusions are caused by borders with opposing contrast polarity, such as the Craik-Cornsweet

O’Brien Effect (CCOE) and watercolor illusions. Both of these illusions are caused by a sharp change in contrast polarity at high spatial frequency, usually causing a spreading of the neighboring luminance or color. One principle of the watercolor illusion is that the color spreading is stronger for the line that contrasts less with the background (Pinna 2004). If the inducing contours are equally salient, both can spread within their allotted boundaries. It has been shown that the color spreading of the line that contrasts more with the background can be in the opposed direction of the outer/farther contour (Kimura and Kuroki, 2014).

Another point to emphasize is the link between color and form. This can be seen in the watercolor effect. In addition to the color spreading more on the side of where the line contrasts less with the background, this side also adopts a figural percept, the other side acting as the background (Pinna, Werner, & Spillmann, 2003). In contrast to the ambiguous figure ground percept that can switch in the face vase illusion, the dominant coloration of one side of the watercolor effect strongly supports a figural percept.

Lightness, Brightness, and Color Constancy

Factors involved in the percept of surface color include physical properties of the surface as well as properties of the illuminant. To account for this, there are two proposed perceptual dimensions: lightness and brightness (Gilchrist, 2007). While lightness refers to perceived reflectance, brightness refers to perceived luminance. Observing a blank grey background, there are no cues as to the reflectance of the surface: it could be a white object in shade or a black object in light. While these two examples would have the same brightness, they would have different lightness.

Our ability to recognize colors of objects under different illuminants is known as color constancy. Cues such as edges, specular reflections, and shading give our visual system clues as to the lighting context, and can change the perceived lightness of a uniform grey area. Inferring the color of a surface from its edges may be beneficial for color constancy and being able to recognize uniformly reflective surfaces under a vast range of illuminations.

One explanation of the watercolor effect invokes visual mechanisms involved in processing depth, reflectance, and illumination. Although the watercolor effect appears vividly when the two lines differ in brightness and color, the illusion still exists by juxtaposing purely

chromatic lines equal in luminance to each other (Devinck, Delahunt, Hardy, Spillmann &

Werner, 2005). Conversely, an achromatic watercolor illusion can be created by juxtaposing black and white lines (Cao, Yazdanbakhsh, & Mingolla, 2011). For achromatic illusions spreading is along the dimension of brightness (Cao et al., 2011). Chromatic illusions resemble shifts in saturation of the inducing contour, but shifts in hue have also been observed (Devinck et al.,

2005).

Temporal properties of Filling In

Although the watercolor effect can be matched to physical color, there may be differences in the underlying processes for these different percepts of colors. By modulating the surrounding color or brightness of the simultaneous contrast illusion (figure 1B), one can modulate the color or brightness of the center region, making the illusion dynamically change in time. By comparing this modulation to actually modulating the center color itself, De Valois et al. (1986) found differences in temporal tuning matches for physical vs. induced modulation in both chromatic and brightness domains (De Valois, Webster, De Valois and Lingelbach, 1986).

When the test area itself was modulated, the modulation could still be seen at higher frequencies, while the illusory modulation was reported as being slower. At high temporal frequencies of surround modulation, no center modulation was seen. These temporal differences imply an induction process that is sluggish compared to physical contrast modulation.

Other researchers have hypothesized that it should take time for filling in to propagate from the edges to the center of the stimulus. By briefly presenting a white disk followed by a mask, Paradiso and Nakayama (1991) demonstrated that while the border of the white disk was

visible to observers, the interior was black. In other words, a ring was seen instead of a disk.

This implies a filling in process that takes time to travel from the edge to the surround for a normal uniform visual stimulus, which enforces the idea that filling in from edges is a general aspect of vision and not just specific to illusions. Increasing the spatial frequency of a modulating square wave grating has been shown to increase the critical flicker frequency of which brightness induction can be seen (i.e. the larger the induced area, the longer it takes to fill in) (Rossi and Paradiso, 1996).

Further studies tested the frequency at which reversing Cornsweet illusions (figure 1C) at different spatial frequencies could be viewed and found similar results of increasing critical flicker frequency with increasing spatial frequency (Davey, Maddess, & Srinivasan, 1998). A following study testing a larger range of spatial frequencies showed an inverted u function for achromatic CCOE patterns (Devinck, Hansen & Gegenfurtner, 2007). In other words, at very high spatial frequencies, it actually takes longer to fill in compared to intermediate spatial frequencies. For chromatic isoluminant CCOE gratings, the critical frequency decreases with increasing spatial frequency. This goes against the filling in hypothesis which would predict decreasing critical frequencies for lower spatial frequencies. These differences in results between chromatic and achromatic stimuli seem to be consistent with the differences seen in the achromatic vs. chromatic contrast sensitivity functions (Mullen, 1985). This suggests that if filling in is happening, it is happening at different spatial scales and by different neural mechanisms.

Looking at these studies, it appears that although both real surface and illusory brightness/color fill in from the edges, the current evidence shows contrast induced brightness/color (square wave gratings) to be spatially dependent (Rossi and Paradiso, 1996),

while illusory color (the CCOE effect) depends more on the contrast sensitivity function

(Devinck, Hansen & Gegenfurtner, 2007). However Devinck et al. (2007) point out that for one observer, Rossi and Paradiso (1996) found a lower cutoff frequency for the highest spatial frequency. Further studies are needed to establish whether different forms of brightness and color induction have different critical flicker frequencies.

Spatial Frequency and Natural Scene Statistics

Images of natural scenes like rocks, forests, and landscapes have been shown to obey orderly statistics (Field, 1987). Natural scene statistics show that low spatial frequencies have higher intensities than high spatial frequencies by a function of 1/f. Some have posited that for stimuli like the Cornsweet edge, the visual system may be amplifying low spatial frequencies

(Dakin & Bex, 2003). Since the spatial contrasts in the CCOE and WCE consist mainly of only high spatial frequencies, our visual system may be compensating for the lack of low spatial frequencies by filling in the closest color. Dakin & Bex (2003) also show that the CCOE can be nulled by scrambling low spatial frequency components of the image, but not high spatial frequency components. This challenges the filling in theory by proposing a filtering process rather than a spreading. Others have made multiscale filtering models of different spatial frequencies to account for illusions such as simultaneous contrast, grating induction, and

White’s effect (Blakeslee and McCourt, 1999).

Although edges have traditionally been associated with sharp changes in luminance, an analysis of natural color images has shown that there also exist isoluminant chromatic edges in natural scenes independent of luminance edges (Hansen and Gegenfurtner, 2009). Hansen and

Gegenfurtner analyzed a database of 700 chromatic images and found that isoluminant edges

were no less rare than luminance edges, and color information is independent of luminance in that it can be used to analyze images in a separate way. This agrees with other studies linking orientation with color (Clifford et al., 2003, Webster, Switkes, and DeValois, 1990), and also with the fact that isoluminant versions of the WCE and COCE exist (Devinck et al., 2005, Devinck,

Hansen & Gegenfurtner, 2007).

Afterimages, Adaptation, and Edges

The role of edges has also been shown to have major effects on the appearance of afterimages. The location of an afterimage can be modulated by placing the contours used to define the afterimage in different regions (Anstis, Vergeer, & Van Lier, 2012). Additionally, the contrast of a surface can be temporarily erased after adapting to a flickering outline of the surface, an effect known as contour adaptation (Anstis, 2013). These observations show that boundaries are made before filling in can happen (Cox & Maier, 2015). It has also been shown the prolonged viewing of the neon color spreading illusion produces an afterimage that is filled in (Shimojo, Kamitani, & Nishida, 2002). Other studies created filled in afterimages of regions by adapting to reversing watercolor illusion stimuli (Hazenberg & van Lier, 2013).These studies exemplify the importance of edges in surface color perception and edges and illusory surfaces can affect afterimages.

Expanding on the initial discovery of contour adaptation (Anstis, 2013), we have done some experiments on contour adaptation and the watercolor illusion (Coia and Crognale, in preparation). We replicated the general finding that adapting to flickering black and white outlines dulls out inner achromatic surfaces, and that the flickering black and white lines do not reduce the contrast (desaturate) of colored surfaces as they do achromatic surfaces. However,

adapting to black and white flickering lines does desaturate color matches for a chromatic watercolor illusion, while adapting to isoluminant chromatic lines increases the amount of saturation needed for a match. These findings indicate that adapting to black and white flickering lines can be effective for nulling illusory color but not physical surface color, while it is effective for both real and illusory achromatic surface brightness.

Neurons and Visual Coding

As mentioned earlier, many of the illusions presented in Figure 1 are a result of a sharp change in brightness or color. A starting point for the mechanism behind simultaneous contrast is thought to be lateral inhibition, a property of contrast enhancement stemming from the center surround receptive fields of neurons early on in the visual stream. A firing neuron with a receptive field on an edge will inhibit the activity of surrounding neurons with receptive fields in the surface. Recordings made from neurons in the optic nerve of a cat while viewing border contrast correlate nicely with the perception of border contrast (Baumgartner and Hakas, 1962, as discussed in Spillmann, 2009). Most of the physiological studies on assimilative filling in and filling in of the blind spot have examined early visual cortical areas, such as V1and V2.

The studies of Hubel and Wiesel (1962, 1965) recording from cortical cells of the cat created a model of hierarchical information processing of vision, where the functions of neurons in cortical areas exhibit increasing complexity. This is evident in the simple, complex, and hypercomplex cells they found in Brodmann areas 17, 18, and 19. These regions have since been linked to areas being functionally active during visual processes and are now known as V1,

V2, and visual association areas, respectively. These different neurons were stated to respond

primarily to edges but not uniform surfaces, and their initial discovery was in fact an accident when they noticed the cell firing to the edge of the projector slide while they were removing it.

The hierarchical model of vision motivated by these studies portrayed neural processes as occurring in a largely bottom up, or feed forward manner (Van Essen, 1983). The primary visual cortex (V1) is where the lateral geniculate nucleus first distributes the retinal signal to the cortex, corresponding to Brodmann area 17. V1 then projects the information to extrastriate visual areas (V2,V3/Brodmann’s areas 18 and 19) upstream.

An influential idea on the way that the brain processes visual information is that there are two separate cortical processing streams known as dorsal and ventral streams. The dorsal stream runs from the occipital area V1 to upper temporal and parietal areas which have been functionally associated with spatial vision and where things are in space, as well as how we interact with objects (reaching and grasping). The ventral stream goes from V1 into the temporal lobes and is associated with object vision, recognizing shape, form, and color

(Ungerleider, Mishkin, and Macko, 1983).

Neurons and Filling In

While cells in the LGN can respond to uniform surfaces, most of the cortical cells in areas V1 and V2 respond to edges (e.g. von der Heydt, Friedman, & Zhou, 2003). These neurons are also sensitive to orientation and contrast polarity. Von der Heydt and colleagues have searched for color filling in using stimuli that caused delayed filling in of a disk, i.e. Troxler fading. They trained monkeys to do color discrimination and were able to establish that they perceive filling in similarly to humans (Friedman, Zhou, & von der Heydt, 1999). They recorded from neurons in V1 and V2 whose receptive fields were on the edge and surface of the stimulus

during filling in of the surface. While both isomorphic and symbolic theories would predict a decrease in edge cell activity during filling in (the edge disappears), the isomorphic theory would also predict a change in activity for surface cells. While they found the expected decrease in edge cells during filling in, they failed to find activity in surface cells, which goes against isomorphic theory. They conclude that edge responses and border ownership contribute to surface color.

Although ‘surface’ cell firing does not seem to correlate with perceptual filling in during

Troxler fading (as discussed in von der Heydt et al., 2003), other single cell recording studies of brightness filling in have measured uniform surface responses in the absence of sharp edges

(Huang & Paradiso, 2008). By selectively studying cells in V1 that gave robust responses to surface brightness changes in time, Huang and Paradiso (2008) found that these neurons tend to have longer latencies when their receptive fields are farther from an edge in a visual stimulus.

This implies that the filling in process extracts information from borders and moves inwards.

This time course of activity relates to behavioral studies showing that filling in of edge induced illusory colors is a phenomenon that acts on a relatively slow timescale (DeValois et al., 1986,

Davey et al., 1998, Rossi and Paradiso, 1996).

A recent study has found a difference in the early processing of chromatic vs. achromatic surfaces. A study using a voltage sensitive dye technique measured responses to surface and edge color and brightness in monkey V1 (Zweig et al., 2015). They found an overall edge enhancement in responses for both chromatic and achromatic patterns, but only a later surface response for achromatic patterns. They refer to the isomorphic vs. symbolic debate and state that while their achromatic data could be argued as being isomorphic, the chromatic data cannot, agreeing with the previous findings of von der Heydt et al., 2003.

There has been at least one study recording from single cells in monkey while viewing physical and illusory brightness variation (square wave grating vs. illusory brightness (COCE))

(Roe, Lu, & Hung, 2005). Single cell recordings in anesthetized monkey show responses in V2 for illusory brightness but not V1, while V1 did respond to the real luminance grating. They conclude that V1 is not a suitable source of the COCE, but V2 may be. A similar finding was also found using illusory contours as stimuli, where responses in V2 were evoked by illusory contours, while responses from V1 were not (von der Heydt & Peterhans, 1989).

Anatomically there are feedback projections from extrastriate cortical areas projecting back to earlier visual areas, and the functional purposes of these feedback connections are being explored. While the parietal areas (dorsal stream) may be receiving input from subcortical areas such as the superior colliculus and pulvinar, temporal areas of the ventral stream are more dependent on signals from the V1 retinogeniculatestriate path (Lamme, Super, Spekreijse,

1998). Another study showed that response latencies to visual stimuli took longer to reach ventral areas V2 and V4 than V1 (Schmolesky et al., 1998). One of the motivations of the current proposal is to investigate whether perceptual filling in of surface colors are instigated by feedback activity from extrastriate areas back to early visual areas V1/V2.

Cortical Modularity of Color and Form

A separate debate exists concerning the cortical mechanisms of vision and whether there are modules in the brain that perform discrete functions and whether certain visual attributes such as color, orientation, motion, and size are processed in segregated manner.

Tissue staining studies using cytochrome oxidase have shown certain clusters of neurons stained better, which were called ‘blobs’ in V1 and ‘bands’ in V2. Neurons within these blobs and bands

seemed to have sensitivity to color stimuli, while neurons outside these areas tended to be more selective for orientation (Livingstone and Hubel, 1984). On average neurons in these regions may prefer one visual attribute or another, however the selectivity of neurons to orientation and color is best described as lying on a continuum where each neuron can have a certain amount of color and orientation selectivity (see Gegenfurtner, 2003).

Physiologists have categorized color cells in visual cortex as single and double opponent

(e.g. Shapely and Hawken, 2011). While single opponent neurons respond to uniform surfaces and are color selective, double opponent cells may be orientation selective, responding to color boundaries and textures. Both of these cell types are thought to exist throughout the cortical visual pathway in order to create the percepts of color and form, and may be analogous to edge and surface selective cells reported from other labs.

Color Centers in the Cortex

While early visual areas undoubtedly contribute to both chromatic and achromatic vision, ventral-occipital (VO) cortical regions have been shown to be necessary for the perception of color vision in humans. This has been known from neuropsychological studies of cerebral achromatopsia with brain damaged patients. Zeki (2002) refers to observations made by Verrey, describing a case where damage to the fusiform and lingual gyri in the right hemisphere of the brain caused achromatopsia for the entire hemifield. Since these brain lesions are located below the calcarine sulcus, one would expect that they may only affect the upper quadrants of the visual field, given what is known about the retinotopic map of V1. This implies that there is an area in the lower part of the cortex, independent of V1,that controls color vision for the whole hemisphere, implying a ‘color center’ in the cortex.

Others have contested the role of area V4 as being a ‘color center’ in the cortex. As discussed in Shapely and Hawken (2011). Lesion studies in macaque have shown the monkeys only have minor color discrimination deficits after lesions to V4 and found distinct areas in the inferotemporal cortex lesions which caused a monkey analogue to cerebral achromatopsia.

These authors speculate that these temporal areas may serve as a memory bank of colors that feeds back to V1. Grossberg has created a model (called FAÇADE: Form and color and depth) theorizing about how the brain accomplishes filling in (see Pinna and Grossberg, 2005). In this model a surface processing stream of neurons in visual cortex can be influenced by a separate boundary stream. These are processed in parallel by the visual system, while the surface stream is thought to travel through the ‘blob’ regions of the cortex, the boundary stream goes through the ‘interblob’ regions. These cortical pathways are distinguished from the previously proposed color and orientation pathways. The boundary stream pools over opposite polarities and in doing this lose their information about color. While the boundary stream is invisible, it delineates the areas to which the surface stream will apply color, and sends feedback which the surface stream uses.

One aspect of the watercolor illusion is that the edge of the colored boundary (inducing inner line) is weakened by the presence of the surrounding dark boundary, causing the color to spread inwards. This can be explained by spatial competition between the contours, in which the line contrasting more with the background suppresses the boundary signal of the less contrasting inducing contour, causing weakening and spreading of the inner color. This boundary process may be completing itself in V2, which will signal back to V1 to suppress the colored boundary signals (Pinna and Grossberg, 2005).

As previously mentioned, along with the illusory color and brightness seen in figure 1 examples, these illusions can be thought of as mechanisms for enhancing border ownership and figure-ground segregation. A neuron that shows border ownership is one which will respond differentially when an object (say a square) is placed on one side or another of its receptive field, even when contrast polarity is controlled for. Single neuron recordings in monkeys have found cells that code for figure ground organization and border ownership by somehow accessing information outside of their classic receptive field (Zhou, Friedman, and Von Der

Heydt, 2000). These neurons were prevalent in V4 and V2, but also found in V1.

Neurons in V1 are more responsive when on a figure, or the edge of a figure, compared to the receptive field being on the background of an image with figure ground properties

(Lamme, 1995). These types of neurons may be getting feedback from higher visual areas or have long range connections with other neurons far away on the retinotopic map. When extrastriate regions are lesioned, V1 retains its increased output when on the edge, but loses the increased response when on the surface. This implies some sort of feedback mechanism occurring, which may be responsible for the filling in of the figure.

fMRI, Retinotopic Maps, and Filling In

The spatial scene we see is imaged on our retina to some degree of precision, and the photoreceptors relay that information in the retina to cells with receptive fields, which can act as edge detectors. Retinotopic organization is present in the cortex, and can now be visualized using fMRI by seeing how different parts of the cortex prefer different eccentricities and areas of visual space. It has been shown that there are many different retinotopic maps laid out over the cortex (see Wandell & Winawer 2011). All of these visual areas have their own

representation of visual space. The map of the primary visual cortex (V1) is organized contralaterally (left cortex~right side of visual field), as well as upside down (below calcarine sulcus~upper visual field). While the foveal representation of areas V1, V2 and V3 all converge near one area of the cortex, there are other foveal representations, one in the ventral occipital cortex of humans shown to prefer color to achromatic stimuli (Wade, Brewer, Rieger, and

Wandell, 2002). Retinotopic maps are made up of neurons whose receptive fields occupy a portion of the visual field. With increases in the visual hierarchy, the receptive fields of cells gets larger, and there are some brain areas wherein neurons seem to respond to objects regardless of spatial location This implies that the neurons’ receptive fields are the whole visual field of the organism (Smith et al., 2001). In addition to the retinotopic map, the primary visual cortex is organized into columns representing different spatial frequencies and orientations.

Monocular information from both eyes converges in V1 forming ocular dominance columns ranging in preference of eye input ratios.

There have been several studies using fMRI investigating the activity of different brain regions upon viewing the Cornsweet illusion (CCOE). Perna et al. (2005) performed an fMRI imaging study on the CCOE and failed to differentiate illusory brightness edge from their control line in early visual areas. As a control stimulus, they turned the Cornsweet edges into lines which had a luminance gradient but did not reverse in polarity. This control stimulus created an illusion of depth without the illusion of brightness seen in the COCE. Perna et al. (2005) did find dorsal areas (caudal intraparietal and lateral occipital sulcus) responsive for brightness perception differences in theses stimuli, and brings up the possibility of some top down cognitive process from the dorsal stream involved in brightness computations.

Boyaci et al. (2007), using fMRI, did find activity in early cortical visual areas with the presence of the COCE. They compared a COCE stimulus to a square wave control with matched physical brightness. They found significantly greater activation in V1 for physically matched brightness than the illusion. For areas V2 and V3, both real and illusory responses were greater than those for the control stimuli with less perceptual surface brightness.

High resolution fMRI has shown that the lateral geniculate nucleus responses have been correlated with the CCOE, indicating neural correlates for low level visual processing (Anderson,

Dakin, & Rees, 2009). This study used the fact shown by Dakin and Bex (2003) that scrambling low spatial frequencies nulls the CCOE. They showed that a CCOE stimulus that had its low spatial frequencies scrambled produced a smaller BOLD activation in the LGN, compared to high spatial frequency scrambled (which doesn’t affect the illusion).

These different studies on the BOLD signal from CCOE stimuli all report different regions of enhanced activity, ranging from the LGN (Anderson et al., 2009), V2/V3 (Boyaci et al., 2007), to association areas in parietal/lateral occipital areas (Perna et al., 2005). While Anderson et al.

(2009) and Boyaci et al. (2007) argue for a response in early visual areas that corresponds to the percept of illusory brightness, feedback from higher to lower visual areas cannot be ruled out.

Although the illusions discussed above exist in both brightness and color domains, the neural correlates have been more thoroughly studied in the brightness domain. One study using fMRI has been done on the neon color spreading illusion (figure 1F) (Sasaki & Watanabe, 2004).

They found increased activity in the retinotopic area corresponding to the color spreading predominantly in area V1 when attention was controlled by having observers respond to a letter task. When attention was not controlled, increased activity for spreading was found in V1

through V4. The retinotopic areas corresponding to the inducing contours also activated areas

V1-V4 in both illusion and control, indicating a separate contour processing stream. This was seen for both chromatic and achromatic forms of the illusion, indicating a similar mechanism for chromatic and achromatic neon color spreading. It was inferred from this study that when attention is allocated to another task, V1 is still activated by the illusion. In other words, the activity in V1 was not due to attentional feedback, although preconscious feedback could be possible.

An fMRI study looking at peripheral (Troxler) fading of a disk showed a decrease in activation of early visual areas (V1 and V2), but an increase in more anterior regions (V3A and

V4v) (Mendola et al., 2006). This indicates that higher visual association areas are active while early visual areas may be suppressed during this type of perceptual filling in task. It should be noted that in this peripheral fading, luminance contrast is disappearing, while in other types of filling in (the watercolor illusion) color is spreading over a region. These authors also report increased activation of the parietal cortex during Troxler fading. They correlate this finding with fading being more effective behaviorally in the lower visual field, and the parietal cortex being more responsive to the lower visual field.

In Troxler fading the filled in area appears to take on the color of its surroundings. Some studies show that actually what is happening is the filled in area is actually a mixture of the surrounding and center colors (Hsieh and Tse, 2010). In other words, when a blue disc is filled in on a red background, the area will actually become purple. Using fMRI and pattern classification algorithms, Hsieh and Tse (2010) were able to make above chance predictions about whether an observer was viewing a blue, red, or purple disc, and applied this technique to obtain evidence of the perceptual feature mixing. This feature mixing could also explain why in the watercolor

effect the induced area is a mixture of the inducing contour and the physical surface. The results of Hsieh and Tse (2010) give evidence for filling in of early visual areas, their effects being strongest in V1.

One study failed to find fMRI activation correlated with illusory brightness/color perception in early visual cortex. Corniellesen et al. (2006) performed an fMRI study with temporally modulating contrast effect similar to DeValois et al. (1986) study and looked at activity of retinotopic areas while modulating center vs. surround brightness. Even though a brightness modulation was perceived in the center when the surround luminance was modulated, no fMRI activity was seen during surround modulation in the retinotopic area of V1 representing the center. This is evidence against isomorphic filling in, and they argue that other studies finding evidence for color filling in from edges in early visual areas may be confounded with figure ground and border ownership responses. They do however report extended edge responses and postulate about a mechanism such as filtering from edges that does not require a filling in mechanism.

The Visual Evoked Potential (VEP)

While fMRI studies provide some spatial resolution of specific brain regions, the blood oxygen level (BOLD) signal is slow in time. In other words, it has rather poor temporal resolution when considering the speed at which neurons in the visual stream communicate with each other. While the speed of nerve processes happens on the order of milliseconds, the BOLD signal is on the order of seconds. In order to achieve higher temporal resolution other methods which record electrical activity directly can be used. EEG and MEG, which are recorded non- invasively at the scalp, can give gross potentials of brain activity. The trade off with EEG and

fMRI is with timing and spatial resolution: EEG has rather poor spatial resolution: the exact location of the electrical source of neural activity is unknown. The temporal resolution for EEG is very high resulting in millisecond temporal precision.

A study using dense array EEG was done to look at activity generated in response to a brightness illusion known as White’s Effect (figure 1 D) (McCourt & Foxe, 2004). This study showed that the earliest measurable response (C1 component) was increased in White’s Effect compared to control stimuli, indicating that the illusion may be detected almost immediately in the cortical responses, suggesting feed forward activity. While White’s effect is an example of assimilation, where the induced area looks more similar to the inducer, the induced area is of short range compared to the long range spreading of the Cornsweet illusion and the watercolor effect.

Although the spatial resolution of EEG is poor, estimates can be made of the conductance of electrical activity on the cortex using dense arrays of EEG/MEG (many recording electrodes). Source modeling techniques use dipoles as model neurogenerators and create head models using structural MRI and estimating the conductivity of the skull and scalp. Then an inverse model can be built which best explains the electrical activity measured from the electrodes, given the head model. Estimates of activity in functional regions of interest which are mapped out using fMRI localizers can then be calculated (Cottereau, Ales, & Norcia, 2014).

Di Russo and colleagues have estimated the sources for both transient and steady state visual evoked potentials (Di Russo et al., 2001, Di Russo et al., 2006). Transient visual evoked potentials briefly show an image followed by a blank screen. This allows for a signal to return to baseline. The transient VEP can be split up into 3 components: C1, P1, and N1. While the

source of C1 was found to be V1, P1 and N1 components were found to be in extrastriate areas

(Di Russo et al., 2002). The steady state VEP (SSVEP) is an exchange or pattern reversal of two images (most commonly reversing black and white checkerboards). The resulting waveform elicited by this form of visual stimulation is sinusoidally shaped, never returning to baseline due to the repeated stimulation. Di Russo et al. (2006) state this as a disadvantage for SSVEP

(compared to transient/onset presentation) because many brain regions may be contributing to the sinusoidal waveform. SSVEP data can be objectively quantified in the frequency domain using the Fourier transform. SSVEPs induced by achromatic Gabor patterns were found to have two sources: one located in V1 and another located near MT/V5, which is an area thought to be involved in motion processing (Di Russo et al., 2006).

Using hdEEG in conjunction with dipole modeling, a study of cortical processing of figure ground segmentation showed that figure and background representations have different cortical activations (Appelbaum, Wade, Vildavski, Pettet, and Norcia, 2006). By modulating the figure and background of a stimulus at different frequencies, Appelbaum et al. (2006) were able to frequency tag separate responses for figure and background. Their source modeling showed that while the background frequency was strong over areas V1, V2, and V3, figure responses were dominant over lateral occipital areas (LOC). They cite other studies showing LOC activity in response to line drawings of objects and illusory contours, and state that their results are consistent with cortical feedback from high visual areas back to the early visual areas to enhance figure ground segmentation.

Chromatic Visual Evoked Potentials

Most natural scenes contain both chromatic and luminance contrast, but on a computer monitor under controlled conditions it is possible to create chromatic stimuli that are isoluminant, containing no brightness contrast with the surrounding. While strong visual evoked potentials (VEPs) can be measured using black and white checkerboards, isoluminant chromatic stimuli also elicit VEP signals, referred to as chromatic visual evoked responses

(CRVEPs) (see Rabin, Switkes, Crognale, Schneck, & Adams, (1994), Porciatti, & Sartucci, (1999).

These studies used sine wave grating stimuli which isolated S (tritan axis) and L-M cone opponent channels along cardinal directions in color space (Derrington, Krauskopf, and Lennie,

1984, Macleod and Boynton, 1979). Furthermore the CRVEP can be measured at low contrasts

(Porciatti, & Sartucci, 1999), which is important for trying to measure colors matched to the faint color spreading seen in the watercolor illusion.

Important differences have been noted in chromatic and achromatic VEPs. One is that

CRVEP responses seem to be lacking an early component that is present in achromatic stimuli

(Gerth, Delahunt, Crognale, & Werner, 2003, Rabin et al., 1994) and in the adult, the waveform recorded over the occipital lobe consists of a large negative wave usually followed by a positive component. To generate responses to achromatic stimuli, pattern reversal (exchanging two images repeatedly) stimulation is known to be effective. For isoluminant chromatic stimuli pattern onset (showing color followed by a neutral grey background) generates robust responses compared to achromatic stimuli. At low spatial frequencies, S cone stimuli have been shown to elicit stronger VEP responses than achromatic stimuli (Rabin et al., 1994). Similarly, blurring high spatial frequency stimuli causes a more drastic reduction in achromatic than S cone patterns (Gerth et al., 2003).

Studies of cortical dyschromatopsia, a condition of incomplete color loss, also imply that the percept of color may be disrupted by damage to ventral visual areas. One piece of evidence comes from a case study of a stroke patient with lesions in ventral occipital regions, but a relatively spared V1 (Crognale et al., 2013). The damaged brain regions were bilateral and included posterior parahippocampal, fusiform, and lingual gyri. This brain damage resulted in a loss of color sensation where colors once known to be vivid to the patient appeared lacking, dulled out sepia tones. An interesting finding was that the early components of the patients’ visual evoked potentials for chromatic stimuli were normal compared to control subjects.

While chromatic VEPs have previously been shown to be reduced in the case of retinal color deficiencies (Crognale et al., 1993) the dyschromatopic patient in Crognale et al. (2013) gave normal chromatic VEPs yet had a dysfunctional color percept due to ‘higher level’ cerebral damage. It was also noted that the later components of this patient’s CRVEP waveforms appeared to be lacking, implying activity occurring in later visual areas or a possible lack of neural feedback from the damaged regions. Importantly although there have been many studies using the CrVEP, the cortical origin of the major components in the response recorded over the occipital lobe is still uncertain.

Attentional Feedback in the VEP

One cognitive function that has been implicated with neural feedback is attention. By attending to a colored stimulus, its saliency and saturation are increased (Fuller and Carrasco,

2006). Attention has been shown to modulate the BOLD signal in ventral occipital areas more so than early visual areas (Kastner et al., 1998). Studies using the visual evoked potential found attentional effects for steady state achromatic stimuli, but failed to find attentional effects when

using transient isoluminant chromatic stimuli (Highsmith and Crognale, 2010, Gerth et al., 2003).

This implies that the initial wave of signal in the chromatic VEP is independent of attention.

Attentional effects on achromatic ERPs have also been reported in later components of the waveform occurring ~200ms after stimulus onset (Flevaris, Martinez, and Hillyard, 2014).

Figure 3: SSVEP stimuli. A: Control pattern reversal where two mirror image control stimuli with no surface color are exchanged. B: Illusion condition where illusion (containing illusory surface color) is exchanged with control stimulus. C: Filled control condition where control with physical surface color is exchanged with control with no surface color. D: Same as A without orange lines. E: same as B without orange lines (and hence without illusion as well).

Physiological Correlates of Watercolor Effect

By using a steady state VEP (SSVEP) presentation, Coia et al. (2014) developed a technique to measure the watercolor illusion. This study involved comparing the frequencies of the VEP response elicited by exchanging pairs of images (illusion vs. control, control 1 vs. control

2, control vs. filled control, see figure 3 A, B, and C). The illusion stimulus was a series of

columns made of two thin outlines, each having colored lines in the interior bordered by luminance lines on the exterior, while in the control the colored and luminance lines were braided. A consistently stronger fundamental frequency component was found when exchanging illusion with control than when exchanging control 1 with control 2 (a mirror image of itself). In addition, when physically filled color was placed into one of the controls (Figure 3 C) in the control condition, this also produced a measurable fundamental frequency. Each image was presented for 250 ms (4 Hz exchange rate), and the color in the illusion and filled control conditions would appear once per cycle of the two images (2 Hz fundamental). While all 3 conditions produced a strong 4 Hz component due to the image updating frequency, a 2 Hz component was also present in the illusion and filled control conditions, but not in the control

(see bottom of figure 4).

Figure 4 (top) shows example SSVEP responses for one subject for control and illusion conditions. For this subject, the control condition (figure 4, left) appear symmetric and to be dominated by the 4Hz component (indicated by the 4 uniform peaks and large 4Hz component highlighted in red). The illusion condition (figure 4, right) has a large 2 Hz component compared to the control (highlighted in red).

Figure 4: Fundamental and 2nd Harmonic. Top: SSVEP waveforms from one subject for illusion and control. Middle:

Frequency tagging the waveform using Fast Fourier Transform (FFT) to objectively quantify stimulus response.

Bottom: Results of Coia et al., (2014). Experimental (illusion) and filled control conditions produced larger 2Hz components than the control condition (left). There was no significant difference for the 4 Hz (right).

Coia et al. (2014) showed the fundamental 2Hz frequency elicited by the illusion SSVEP to be dependent on the luminance and color of the lines relative to each other and the background. When comparing the steady state VEP conditions, one discrepancy that could be a potential confound is that there is an alignment of the same colored lines in the illusion condition that is not present in the control condition. The control condition is a pattern reversal in the sense that the two images being exchanged are mirror images of each other. In contrast, in the illusion condition, the two images are not mirror images, but rather the lines exchanged go from an aligned to misaligned state. Previous research has used similar stimuli to measure

Vernier acuity in infants using SSVEP (Skoczenski & Norcia, 1999). They found that exchanging aligned and misaligned lines produced a strong fundamental harmonic, while exchanging

misaligned to misaligned did not. Therefore stimuli excluding the inducing lines but going from an aligned to misaligned state (Figure 3 E) should also produce a 2Hz signal. Including this condition may help to disambiguate additional activity hypothesized to be responsible for the illusory color from the alignment/misalignment (Vernier Acuity) response.

The watercolor effect can propagate over large distances, and some have proposed that the watercolor illusion may be mediated by long range horizontal connections within the cortex

(Werner, Pinna, Spillmann, 2007). Where in the cortex this takes place is unknown. While Coia et al. (2014) demonstrated that the illusion (along with filled surface color) are measureable using SSVEP, single electrode EEG could not give much insight about the possible difference in loci of these signals.

Chapter 2: Preliminary Data

We have collected preliminary data measuring the watercolor illusion using high density

EEG with two methods of stimulation: pattern onset (transient) VEPs and steady state (SSVEP).

In the pattern onset, stimuli were shown for 250 ms followed by a blank screen for 750ms.

There were 3 conditions: illusion, control and filled color. In these pilot studies the filled control and illusion were not contrast matched: a fixed contrast was used for every subject. Therefore any differences between illusion and filled control in these data may be due to differences in perceived contrast (the filled control was made to be more on the saturated side). The onset presentations are analogous to the stimuli in the left side of each column in figure 3(A-C), but instead of being exchanged with the control, were followed by a blank screen (same luminance as background) allowing the evoked potential to return to baseline. For the SSVEPs, two images per condition were exchanged at a rate of 2 cycles per second (see Figure 3). Three 60 second sessions for a total of 180 seconds were recorded for each condition in both onset and SSVEPs.

Materials and Methods

A 256 channel EGI (Electrical Geodesics, Eugene, Oregon) hydrocel system was used.

The data were acquired using NetStation software and stored offline. A photogrammetry system was used to take pictures of the electrode placement on each participant, allowing for warping of a general head model based on subject’s head size.

Initially, data were run through Net Station software which filtered (band-pass 1-40Hz), segmented, performed artifact detection, bad channel replacement, baseline correction, and averaging. A separate analysis was done using the Brainstorm software in order to perform source localization (Tadel et al., 2011). In order to eliminate blinks and eye movements, a signal

space projection (SSP) algorithm was used. This consisted of using electrodes under the eyes to detect blinks, which cause massive electric potentials contaminating the data. An algorithm then smoothed the data around the blink, removing these artifacts.

Subjects

While 8 people were run in pattern onset VEPs, the data from one subject was not included because of insufficient response amplitudes. 8 subjects participated in the color SSVEP conditions (Figure 3 A-C), while 4 participated in the alignment experiment (Figure 3D and E).

Results (onset, electrodes)

The transient onset presentation allowed us to analyze components of the event related potential (ERP) and compare waveforms of the three conditions. A grand average of selected time points in the topography of the evoked potentials for the 3 different conditions (control, illusion and filled control) are shown in figure 5. These points were chosen corresponding to the prominent components of the waveforms from the oz electrode, which is the active electrode in single channel VEP studies. There is an initial peak around 100 ms (P1) after onset and reaching a low point at around 150 (N1), then peaking at 300 (P2) and 400 (P3) ms.

Figure 5: Average onset topography of activity recorded for 3 different conditions (left). Average waveforms for Oz electrode over back of head where VEPs are typically recorded.

A subsequent analysis on the pattern onset data was done by taking an average of posterior electrodes on left and right hemispheres performing a paired sample t-test for illusion vs. control and filled control vs. control at each millisecond. These data are shown in figure 6.

The areas between the curves highlighted in yellow are the areas in which p<.05. Difference between illusion and control can be seen early on (around 100-200 ms after stimulus onset) in the waveforms for the left and right hemispheres, and in the right hemisphere for the filled control vs. control. The filled control vs. control also shows early difference in the right hemisphere and later differences (around 400 ms after onset).

Figure 6: Electrode Diagram. Diagram showing grouping of electrodes (top) and resulting waveforms averaged across participants (bottom). Areas highlighted in yellow are areas where paired sample t test p<.05.

Results (SSVEPs, electrodes)

Steady state VEPs were presented a rate of 2 cycles per second. The data were acquired in the same session and processed in the same way as the onset presentation. An additional frequency analysis was done on the SSVEPs performing a Fast Fourier Transform (FFT).

Figure 7 shows averaged 2 and 4 Hz topographies for the 5 SSVEP conditions shown in figure 3. These were derived by calculating average 2 and 4 Hz amplitudes for each individual’s electrodes and then averaging across subjects. The alignment and misalignment conditions were run in the same session but always after the first 3 conditions (priority was given to the color experiments). This later recording may have caused the data to be noisier because the electrodes may have dried out. As expected from Coia et al., (2014), the illusion and filled control produced greater 2 Hz activity than the control condition, which is reflected in the red

seen in the 2 Hz signals but not so much in the control 2 Hz. In line with previous studies on

Vernier acuity EEG responses (Skoczenski & Norcia, 1999), the align/misalign condition appears to have more 2 Hz activity than the misalign/misalign condition.

Figure 7: SSVEP Topography. Average topographies for 2 and 4 Hz amplitudes in different experimental conditions.

In order to quantify these data, the amplitudes of 2 and 4 Hz frequencies were averaged across the same subsets of posterior electrodes as in the onset data shown in figure 6. These data are shown in Figure 8. These data show greater 2 Hz activity for illusion, filled control, and align/misalign conditions compared to control and misalign/misalign conditions, while 4 Hz are approximately all similar in magnitude.

Figure 8: Average SSVEP amplitudes. Bar graphs show 2 and 4 Hz amplitudes calculating from averaging across electrodes and then across subjects. Error bars represent +- 1 standard error of the mean.

The results shown in figure 8 indicate that the 2 Hz component seen in the illusion condition may be due to alignment of the lines and not the percept of illusory color. The way that we propose to control for this confound of alignment is by changing the SSVEP stimuli: instead of exchanging from illusion to control, we will in the proposed study exchange two complementary illusions in a pattern reversal manner.

Estimation of Cortical Activity

While the activity recorded from the electrodes represents activity recorded from the scalp, estimates of cortical activity was made to see if different brain regions at different times and intensities for the different experimental conditions. Neural models of the watercolor effect have speculated that the illusion may be caused by neural feedback from higher visual areas to lower ones, which attributes color to regions that aren’t receiving bottom up color

signals (Pinna and Grossberg, 2005). This hypothesis could be tested by estimating the activity of regions of interest and looking at the time course of activity for the illusion and comparing it to the control and filled control conditions.

In order to estimate activity of cortical areas, a computer model of each individual’s head was created using Brainstorm as well as the OpenMEEG BEM software (Gramfort et al.,

2010 / Kybic et al., 2005). Since the experimenters did not have individual MRI data for 7 out of the 8 participants in this experiment, source estimation was done on a generalized head model

(Colin27: MNI brain with a 1mm resolution). Panoramic pictures of each subject were taken after each recording session while wearing the electrode cap and were used to generate a 3-D head model of the individual. This 3-D model was later used to morph the general head model to give a more precise individualized model.

The envelope of the cortex surface was represented by 15000 vertices, each having and

X Y and Z orientation. Building this head model is referred to as solving the forward problem of how the electric signals in the brain propagate through inner skull, and outer skull and scalp to the electrodes. Each of these three layers was modeled with 1922 vertices.

This head model assumes that the signals being measured by the electrodes come from pyramidal cells aligned parallel to the surface of the cortex. The surface of the cortex is not always perpendicular to the scalp, as can be seen by the presence of sulci and gyri, so activity of a certain brain region isn’t always going to be under the closest electrode, per se.

In order to estimate the noise coming from each sensor, chunks of baseline EEG recording were used to calculate a noise covariance matrix. This is used to solve the inverse problem, that is, estimate the most likely source of activity from the electrode recordings. This

experiment used a minimum norm estimate (Baillet et al., 2001) to approximate the activity of certain regions of the surface of the cortex. The measured electrode responses were transformed into a source space to give an estimated average evoked response for a given brain region in a 3/d model.

Results

Onset (sources)

The same data as presented earlier were used in the following analyses. In addition to the head models created for each individual subject, a grand average across subjects was made and fit to a generalized head model. Figure 9 shows average cortical activity on a generalized 3D head model. The red and green outlines represent Brodmann Areas 17 and 18 which correspond to V1 and V2. In figure 8, activity appears to begin in all 3 conditions in the right hemisphere, lateral to V1 and V2. It then propagates to these areas in the right then left hemispheres, respectively.

Although the standard model of cortical activity presumes visual activity to begin in V1, there are other subcortical visual pathways (one through the superior colliculus to the medial temporal lobe) that may be contributing to the early activity seen in the lateral occipital/parietal areas. This has been proposed as an explanation for blind sight in which patients have damage to V1 yet can still perform unconscious visual functions such as motion detection (see Cowey,

2010). For the N1 and P3 components the activity seems to be localized quite precisely to the

V1/V2 areas.

Figure 9: Onset Cortical Activity. View of posterior cortical activity at temporal points of interest for different onset VEP presentations

In addition to these early visual areas V1 and V2, some other regions of interest include areas in the dorsal and ventral streams thought to be associated with vision. Regions of interest

(ROIs) were selected using the Desikan-Killiany brain atlas (Desikan et al., 2006). In addition to

Brodmann areas V1 and V2, four other general regions of interest were picked: inferior parietal cortex, fusiform gyrus, lingual gyrus, parahippocampal gyrus (see figure 10). The inferior parietal cortex is in the dorsal stream and appeared to be where the activity in the grand

average source estimates started. The other three regions are in the ventral stream and are of interest because they are areas reported to be damaged in achromatopsia/dyschromatopsia patients (Zeki, 2002, Crognale et al., 2013) and are located in regions which are associated with color vision in fMRI studies (Wade et al., 2002, Brewer et al., 2005). The fusiform gyrus has also been shown to be activated more in people with color grapheme synesthesia (see Hubbard and

Ramachandran, 2005).

Figure 10: Regions of Interest. Posterior view of extrastriate regions of interest where analyses were performed.

Activity within an ROI can be averaged across dipoles to create a waveform of activity for that region. The amplitude and latency of the first three components (see figure 5) were recorded for each condition of each participant in each ROI. The averages of these values for

ROIs in both hemispheres are shown in figure 11. The top 3 rows of graphs show the latencies of the components for each ROI averaged across subjects. The lower two rows are normalized ratios of the differences in amplitude between P1 and N1 and N1 and P2 which were calculated using the following formulas:

If the amplitudes are above zero, it means that a given condition has a greater amplitude than the control condition on average.

Figure 11: Latencies and Amplitudes of onset waveforms for different regions of interest (ROIs). Error bars represent +-1 standard error of the mean.

Looking at the P1 latency for left hemisphere ROIs (top left), a gradual increase in latency can be seen going from left to right, indicating feed forward processing (Figure 11). The filled control left hemisphere P1 latencies appear slightly quicker than the control and illusion latencies, but overall the latencies of the three conditions appear similar within a given ROI. The

P1 latencies for the right hemisphere appear similar to the left, except for the latency of the fusiform region are faster for all three conditions, being closer to V1 and V2 than the parahippocampal regions. There doesn’t appear to be any significant differences in amplitudes of these early components for illusion or filled control in V1, V2, inferior parietal, or lingual gyrus. The illusion and filled control amplitudes do not on average differ from the control in these areas. Differences in P1 amplitude can be seen for the filled control condition in the fusiform and parahippocampal gyrus, indicating that these regions are responding to filled in color in the early component of the VEP. Both illusion and filled control show larger N1 to P2 amplitudes in the right fusiform gyrus. These data indicate that the filled color response elicits an increased early response (P1-N1) in the fusiform gyri, while the illusion is eliciting a larger later response (N2-P2) in the right hemisphere.

In addition to looking at the components of the waveforms for the different ROIs, t tests were also computed for each time point within the ROI waveforms, as was done for the electrode data in Figure 6. Because some of the onset data for different ROIs had flipped polarity, all waveforms were flipped on their axes so the 2nd component (n1) went negative, and then waveforms were averaged across subjects for each ROI. Figure 12 shows a comparison of illusion to control waveforms for the 5 different ROIs presented earlier. For V1 and V2, the illusion and control waveforms are nearly identical in the early stages (0-200 ms), which are consistent with the analysis of the early components for these areas in figure 11. There are

some differences in the later stages of the waveform (300-500 ms), indicative of later processing, possibly feedback. Significant differences can be seen early on in the right inferior parietal cortex. These differences start happening right at the stimulus onset, which may indicate either they are anticipatory or happening very late (~1000ms after the previous trial).

Figure 13 shows the comparison of filled control to control. In this comparison differences can be seen early on in right hemisphere V1, V2, and parietal cortex, as well as in the left parahippocampal cortex. Significant differences can also be seen later on in the waveforms, similar to the illusion vs. control analysis. These data indicate that (at least in the right hemisphere) the filled control is activating early visual areas V1 and V2 early on in the visual process, contrary to the illusion, which is only activating these areas preferentially over the control later on in time. The differences seen in the filled control in the left parahippocampal region are not present in the illusion condition either. It must be mentioned that these differences in effects could be due to increased perceived contrast of filled control than illusion.

One possibility is that the differences seen in the onset data at the electrode level for illusion vs. control could be coming from the parietal areas and not V1 and V2, while differences in the filled control vs. control could be coming from V1, V2, and/or the parietal regions.

Figure 12: Results of paired sample t tests of illusion vs. control onset waveforms for ROIs. Areas highlighted yellow represent p<.05.

Figure 13: Results of paired sample t tests for filled control vs. control onset waveforms for ROIs. Areas highlighted yellow represent p<.05.

Results: SSVEPs (sources)

Source estimation of the same regions of interest as the transient onset VEPs were investigated for the SSVEP data. The data were analyzed by binning each condition into one second intervals and taking the Fast Fourier Transform of each one second interval and averaging the amplitudes and phases of the fundamental and second harmonics of each ROI.

While amplitude corresponds to the peak to trough difference in voltage, phase is a measure of the relative latencies of the peaks and troughs.

Figure 14 shows grand average two and four Hz amplitudes representing the topography shown in figure 7. These cortical estimates reflect the relative activity seen in the topographical representation in figure 7 in that the control shows relatively no 2 Hz activity compared to the illusion and filled control conditions. As with the electrode data, the align/misalign condition does appear to have more 2Hz activity than the misalign/misalign condition. All conditions show 4 Hz cortical activity. This was to be expected since it represents the rate at which the image is updating.

Figure 14: Cortical activity estimates generated by inputting average FFT amplitudes for each electrode across subjects and performing source estimation on these average values on a general head model. Original electrode topography data are shown in figure 7.

While the 2 Hz activity for the filled control condition appears to be localized with V1 and V2 areas, the illusion and align/misalign 2Hz frequency appears to have more activity in the parietal/lateral occipital areas. In order to compare activity across conditions within the aforementioned ROIs, Fast Fourier transforms (FFTs) were computed on the raw SSVEP data after being transformed into source space. The 2 and 4 Hz amplitudes were extracted from the frequency analysis and averaged across participants, the amplitudes shown in figure 14. The alignment experiment data are plotted separately due to the different baseline levels of noise observed.

The results for the SSVEP frequency analysis (figure 14 an15) show increased 2 Hz amplitudes for the illusion and filled control compared to the control bilaterally in V1 and V2. At the lingual gyrus the 2 Hz appears to be greater in the illusion than the control as well. The 2 Hz signal for the alignment/misalignment condition appears to be consistently larger than the misalign/misalign condition across most ROIs.

While the heat maps in figure 14 were generated by importing 2 and 4 Hz amplitudes

(averaged across participants) from each electrode into one general head model, the graphs in figure 11 were calculated by running individual subjects’ raw data through the model and doing an FFT on the source data. This may be why increased 2 Hz parietal activity is seen in the heat map (figure 14) of the illusion condition but not in the graph (figure 15). Another possibility is that the inferior parietal ROI is much bigger than the small spots of activity within them and these isolated regions of activity are not sufficient to drive up the signal of the whole region.

Figure 15: SSVEP 2 Hz amplitudes for regions of interest. Error bars represent +-1 standard error of the mean.

In figure 16, the average 4 Hz amplitude across participants is plotted for the SSVEP illusion experiment. While the 2 Hz amplitudes showed an increase in the illusion and filled control for V1 and V2, no significant differences are seen in the 4 Hz amplitudes for these ROIs.

For the lingual and parahippocampal areas, the 4 Hz component seem slightly smaller for the illusion compared to filled control, but this result is not clearly interpretable.

Figure 16: Average amplitude of 4 Hz frequencies within ROIs. Error bars represent +-1 standard error of the mean.

To summarize these results, the illusion, filled control, and align/misalign conditions show larger 2 Hz activity in early visual areas V1, V2, and lingual gyrus, while the 4 Hz activity does not show any strong differences across conditions. The distribution of activity across ROIs for the three conditions showing greater 2 Hz activity are not clearly distinguishable from each other (i.e. the illusion, filled control, and align/misalign do not differ obviously from each other.

Case Study

One of the observers in the preliminary data set was part of the UNR database of people who had been retinotopically mapped using fMRI. This observer’s data was therefore used in an in depth analysis of activity in some functionally defined areas of visual cortex V1, V2, V3, hV4,

and ventral occipital areas 1 and 2 (as defined by Brewer et al., 2005). Figures 17 and 18 show polar plots comparing illusion to filled control (figure 17) and illusion to align/misalign (figure

18). These data are from the right hemisphere V1 as defined through retinotopic mapping, and are exemplary of all the early visual areas examined by this subject. T-circ tests were conducted to determine if the frequency components significantly differed between these conditions

(Victor and Mast, 1991). For the illusion and filled control, both 2 and 4 Hz components were significantly different in this ROI, while for illusion and alignment/misalignment, neither component significantly differed across conditions. The main point here is that significant differences are seen between illusory color and real color, but not illusion and alignment.

Figure 17: Polar Plots for one subject’s SSVEP 2 and 4 Hz components comparing illusion to filled control conditions. Note that the conditions show significant differences for both frequencies.

Figure 18: Polar plots for one subject SSVEP 2 and 4 Hz components for illusion and align/misalign conditions.

Note: neither component significantly differ from one another.

Conclusions

These data show a few interesting findings. For the onset VEPs there is no evidence for differences between illusion and control stimuli in early visual areas V1 and V2, although both conditions show strong response in these areas. Differences in the filled control vs. control can be seen in early components of the waveforms in the right hemisphere V1 and V2. The parietal area seems to be selectively activated early on for both illusion and real color, and this is a potential source for the illusion (see Perna et al. 2005). In addition, all 3 onset conditions seem to show a feed forward activity moving from early occipital areas to more ventral areas like the parahippocampal regions, which may be due to the contours and not necessarily the surface color. These ventral regions have been previously implicated in color processing (Zeki, 2004,

Crognale et al., 2013, Brewer et al., 2005).

The SSVEP data when analyzed in the frequency domain show larger 2 Hz amplitudes for illusion and filled control conditions compared to control, with the largest difference being in

early visual areas V1, V2 and lingual gyrus. This contrasts with effects seen in the onset data, but may be explained by the alignment response that is present in the illusion, which is present even when there is no illusory color spreading. The alignment response is just as strong in these early visual areas and with our current analyses we are unable to distinguish responses of illusion from those of alignment.

Chapter 3: Proposal

Proposed Studies

The proposed studies aimed to build off the methods used in the preliminary experiments and collect hdEEG data on participants who have been scanned and retinotopically mapped using MRI. This will allow us to constrain our regions of interest to functional visual areas as well as have a more accurate head model. There were 10 participants in the subject pool of scanned subjects and eight of these subjects were run in the following month. An additional question we wanted to ask in the proposed studies is whether differences in processing can be seen for chromatic and achromatic illusions, as well as their physically matched counterparts. Differences in VEP responses to chromatic and achromatic stimuli have been shown in previous studies (Rabin et al., 1994), and the source of these differences is unknown.

Behavioral Psychophysics

While the preliminary stimuli used the same filled control color for all subjects, the proposed study aimed to measure individual color matches to the illusion and use those individual matches in the VEP stimuli, as was done in Coia et al. (2014). A more accurate comparison of filled color to illusory color can be made if they are of equal perceptual magnitude.

Instead of using black and orange inducing lines, the proposed experiments aimed to test 3 different opponent pairs of contours: those differing only in luminance (achromatic), and those along the 2 cardinal directions of color space (S-(L+M) and +-(L-M) defined by Macleod and Boynton (1979) and Derrington, Krauskopf, and Lennie (1984). These patterns were

contrast matched for equal salience (Crognale & Switkes, 1999). This involved matching contrast of the opponent pairs to each other. Next once the contours had been contrast matched the illusions were matched to a physically filled in control pattern in a color matching experiment.

Inducing patterns were +S (violet) bordered by –S (chartreuse), + (L-M) (pink) bordered by – (L-M) (green/teal), and - (L+M+S) (black) bordered by + (L+M+S) (white). These three colored patterns have been previously shown to elicit illusions that can be behaviorally matched to physical color (Devinck et al., 2005, Cao et al., 2011, Coia et al., 2014).

In addition to contrast matching watercolor illusion stimuli, Gabor Patches were contrast matched along the same axes in color space in order to create stimuli for traditional chromatic VEPs (Rabin et al., 1994). Flicker photometry was employed to measure isoluminance settings for individual observers.

VEPs

The contrast matched contours and surface colors were used to create custom VEP stimuli for each individual. SSVEP stimuli were pattern reversals, which will avoid the Vernier acuity issue raised in the preliminary data. The control condition will be the same, but the illusion and filled control will be different in the sense that they will be exchanging complementary surface colors, instead of being exchanged with uncolored control stimuli. We will therefore only be examining the 2nd harmonic, as the surface color will be exchanged with the image update.

Stimuli will be presented in 60 s blocks with 3 illusion, 3 filled control, and 3 unfilled control, totaling 9 minutes. Pattern onset stimuli had 3 illusory color conditions, 3 filled

controls, and 3 non filled controls for a total of approximately 9 minutes recording time. The additional one line control experiment will have 6 conditions, taking 6 minutes. Traditional crVEPs of Gabor patches onset and reversals (3 each) will take 6 minutes. The total VEP time should be approximately 30 minutes.

The same analysis was performed as was done on the preliminary data set, except the

ROIs will be customized according to individuals’ retinotopic mapping. These studies will contribute to our understanding of the difference in neural processing involved in chromatic and achromatic stimuli as well as physical vs. illusory brightness and color.

Chapter 4: High Density Investigation of the Chromatic Visual Evoked Potential

The study of color vision combines the disciplines of physics, biology, psychology, and philosophy. Curiosity about mixing and matching colored light showed that most humans can combine amounts of three different lights to match any given colored light (Young, 1802). This then led us to find systematic differences between people in the sensation of different wavelengths of light, which later led to the discovery that different genetic sequences of photopigments in our photoreceptors can cause abnormal or color deficient vision at the level of the eye (Nathans et al. 1986). Other forms of color vision loss are caused by damage to regions of the cerebral cortex, distinct from the primary visual center (V1) (Meadows, 1974).

These are known as achromatopsia and dyschromatopsia, which are the complete or partial loss of color vision with a sparing of achromatic (black/white) vision.

Studies of color vision have characterized different levels of chromatic processing, one notably opponent processing, which is apparent in the experience of visual afterimages. Cone opponent cells exist in the eye and lateral geniculate nucleus going to the cerebral cortex.

Neuroscientists have employed various methodologies in the quest to understand the neural mechanisms underlying color vision and the brain regions involved in these computations.

Studying color vision at the level of spiking of individual neurons have revealed the existence of neurons with different properties towards chromatic stimuli (DeValois, Abramov and Jacobs,

1966, Livingstone and Hubel, 1988, Johnson, Hawken and Shapley, 2001, Wachtler, Sejnowski and Albright 2003). Others have used fMRI as a tool to investigate brain responses to color

(Engel, Zhang, and Wandell, 1997, Zeki and Bartels, 1999, Wade et al., 2002, Brewer et al.,

2005,).

Electroencephalography (EEG) is another useful tool for probing the neural mechanisms of color vision by recording the gross potential of brain activity from the scalp (Regan, 1973,

Kulikowski, Murray and Parry, 1989, Rabin et al., 1994). The EEG response to visual stimuli is called a visual evoked potential (VEP), ‘crVEP ‘specifically for color. While the spatial resolution of EEG is limited, advancements have been made in estimating where the sources of EEG activity are by modeling the surface of the cortex as a series of dipoles and focusing on functionally defined areas(Di Russo et al., 2002, Cottereau, Ales and Norcia, 2015). These emerging source localizing methodologies incorporate anatomical and functional magnetic resonance images into the analysis and source estimation of EEG activity.

This series of studies probed the inner workings of electrocortical responses to opponent colors using high density electroencephalography (hdEEG). 3 color combinations were used defined along a luminance axis (L+M, achromatic), a red green (L-M, cyan and magenta), and blue yellow (S-(L+M), violet chartreuse). The color space used to define these colors varied along cone opponent axes (see Derrington, Krauskopf and Lennie, 1984, MacLeod and Boynton, 1979). Although it does not appear that strict adherence to these cardinal mechanisms is maintained at the cortical level (see Shapely and Hawken, 2011), the L-M pathway is a good representation of the color response in general, and the S-(L+M) pathway may have interesting properties since it is known that the S cones do not feed into all pathways equally and may contribute little to motion and luminance computations.

Psychophysical Procedures

Before the electrical recordings were made, 2 psychophysical measures were taken in order to produce the stimuli. The first was heterochromatic flicker photometry (HFP). This was done on the two chromatic patterns in order to ensure that they were isoluminant. Participants were instructed to use computer keys to adjust the relative luminance of the opponent colors until the flickering was minimized. This procedure was done at 5 eccentricities and repeated 3 times for each color pair, then averaged at each eccentricity.

The stimuli were similar to those used in Skiba, Duncan, and Crognale (2014). They were full field (20 deg) ‘multi-Gabor fields,’ consisting of 59 horizontally oriented Gabor fields.

When viewed at 57 cm the Gabors subtended 2 cycles per degree in the center and 0.5 cycles per degree in the periphery (see figure 19). The colors were presented in the rings of five different eccentricity levels which covered the entire computer monitor. The average background chromaticity in CIE 1931 space was [x=.313, y= .322]. The luminance of the entire pattern was held at 18 cd. The chromaticity of the S-(L+M) Gabors was held constant and ranged from a max S [.292 .274] to min S [.343, .39] with individual applied isoluminant settings

(see table 1 for chromaticities).

Figure 19: Multi Gabor fields along three cardinal directions in color space

The second experiment was suprathreshold contrast matching (Switkes and Crognale,

1999). This involved having participants view 2 opponent color pairs in temporal sequence and making judgments about which pair, the 1st or 2nd, was more visually salient. While the S-(L+M) pattern was held as a constant reference for all participants, the L+M and L-M stimuli were modulated along their axes along fixed steps using the method of constant stimuli. There were

8 steps ranging along the L+M and L-M axes (see table 1). These steps were then counterbalanced with the S-(L+M) reference as being presented either first or second. The outputs of the counterbalanced responses were than averaged, then fit to a curve from which the point of subjective equality was derived from the psychometric function. Most participants produced a relatively smooth psychometric curve after one trial, but if necessary additional trials were run.

Table 1: Chromaticities for Chapter 1 Gabor Experiment

Participants

Eight people (1 female) participated in the study. One subject had abnormal color vision

(deuteranomaly) and was not included in the averages. All the others had normal color vision, as tested with the Ishihara plates. While some people produced much more pronounced waveforms than others, no data were excluded. All subjects were part of a database which had undergone anatomical as well as functional magnetic resonance imaging (MRI). The anatomical

MRI data was used to make a realistic head model of each participant. The fMRI data included retinotopy and was used to define areas of visual cortex known to exhibit retinotopic organization (this retinotopic analysis had been done previously in the lab of Professor

Caplovitz: https://sites.google.com/site/clabwiki/quick-reference). These included the primary visual cortex (V1), V2, and V3. In addition to the ‘early visual areas’ a dozen or so other areas along the dorsal and ventral visual streams were also mapped out for each participant.

Results:

Results from psychophysics include average HFP settings at the five different eccentricities and average contrast matches of L+M and L-M to S-(L+M) reference. Figure 20 shows average HFP settings across subjects for the two cardinal chromatic axes across different eccentricity levels.

Flicker Photometry l-m

5 s-(l+m)

0 1 2 3 4 5 -5

luminanc e amgle luminance -10 eccentricity

Figure 20: Average heterochromatic flicker photometry matches across observers. Error bars represent one standard error of the mean.

While L-M settings remain relatively flat across eccentricities, there is a large shift in the

S-(L+M) settings across eccentricities, people requiring more +S luminance in the foveal region than in the periphery. This is in line with previous studies (Skiba, Duncan and Crognale, 2014,

Parry and Robson, 2012) and is believed to be due to the varying levels of macular pigment throughout the retina. The reader is referred to table 1 for average contrast match values for the stimuli.

HdEEG Recordings

A 256 channel system was used to obtain the hdEEG recordings (EGI: Eugene, Oregon).

The circumference of the participant’s head was measured in order to obtain the proper cap size, and the midpoint from the inion to nasion and the left and right periauricular points were marked to locate the vertex of the scalp. The net was then soaked in a mixture of water, potassium chloride, and baby shampoo, before being put on the participant’s head. The electrodes were then set, impedances checked and corrected. After the recordings, panoramic

pictures were taken of the participant wearing the net (photogrammetry) and later co- registered with the head model.

Stimulus Presentation

Stimuli were presented in two forms: onset and steady-state (SSVEP). The onset presentation showed the full-field Gabors for 200ms, followed by 800ms of blank screen of equal average luminance to the Gabors. One minute of onset VEPs were recorded for each condition. The steady-state presentation showed a full-field Gabor for 200 ms followed by a phase-reversed Gabor for 200 ms (5 Hz 2nd harmonic), which repeated for 48 seconds (120 cycles). While the onset stimuli were randomly interleaved across color combinations, the

SSVEPs were presented in blocks.

Data Preprocessing

While the psychophysical data were processed on the fly to produce the custom EEG stimuli, the raw EEG data were stored off line for further analysis. This analysis was done with the Brainstorm software package (Tadel et al., 2011), and comprised bandpass filtering (1-40Hz), detection of blinks and other artifacts, removal of artifacts using signal space projection (SSP) projectors, and segmenting the data into 1 second blocks starting at stimulus onset. Chunks of data containing massive artifacts were removed altogether before averaging. Bad channel recordings that deviated drastically from the majority were deleted.

Results: hdEEG

After averaging the processed data it was noticed that for most subjects a cluster of electrodes over the occipital area showed a negative going component at around 180 ms after stimulus onset, so these electrodes were averaged together for each subject to form a representative waveform. This negative going peak for both chromatic and achromatic stimuli has been previously reported, along with the earlier positive peak for achromatic stimuli (Rabin et al., 1994).

Figure 21: Left: Average waveforms for multi Gabor crVEPs. Right: topographic representation of data showing strong negativity (blue) at the time of the first major negative component in waveform.

Figure 21 shows averaged waveforms for the 3 conditions (left), and a topographic map of averaged electrodes highlighted in red. These average electrodes are immersed in a blue which represents the negative going activity seen at approximately 180 ms after stimulus onset.

Figure 22: Timeline of topographic activity for crVEPs.

Figure 22 shows topography of three conditions across four points in time. While L+M shows increased activity in right posterior scalp (~133ms), all three conditions show strong negative component around 200 ms. The L+M negativity appears more laterally distributed around 200ms after onset compared to the more medial chromatic negativity.

Source Localization

Sources of the early component of the achromatic visual evoked potential have been largely attributed to the primary visual cortex (V1) (Di Russo et al., 2002). Di Russo and colleagues (2002) showed that the later n1 component of the VEP was localized to more ventral extrastriate regions in the fusiform gyrus. Our achromatic stimuli were low contrast due to contrast matching to isoluminant chromatic stimuli, and we did not see a very pronounced early component and choose here to focus on this later n1 component. We were particularly interested in comparing chromatic and achromatic responses in the dorsal and ventral streams

because fMRI studies have shown responses to color in other areas in addition to V1, notably areas in ventral visual cortex ((Engel, Zhang, and Wandell, 1997, Hadjikhani et al., 1998, Zeki and

Bartels, 1999, Wade et al., 2002, Brewer et al., 2005).

Figure 23 shows the different retinotopic maps analyzed in these studies. The ventral stream includes V1, V2v, V3v, hV4, VO1 (ventral-occipital), and VO2 (Sereno et al., 1995, DeYoe et al., 1996, Engel, Zhang, and Wandell, 1997, Wade et al. 2002, Brewer et al. 2005). The dorsal regions include V2D, V3D, V3A, V3B, IPSO (intraparietal sulcus, also known as V7), and IPS1

(Press et al. 2001, Konen and Kastner, 2008).

Figure 23: Regions of Interest in Source Localization Study: Regions of Interest were derived from retinotopic maps measured using fMRI. Diagram shows posterior view of inflated cerebral cortex with ventral (left) and dorsal

(right) retinotopic maps for one subject.

Brainstorm software was used for source localization. This required importing Free

Surfer folders with subjects MRI data and building a head model using the symmetric boundary element method and OpenMEEG software (Gramfort et al., 2005, Kybic et al., 2010). The

settings used were the default set by Brainstorm, estimating the layers of the skull, with outer and inner layers each having 1922 vertices.

Once this forward model was generated, a minimum-norm estimate constrained normal to the cortex (15000 dipoles) was applied to solve the inverse problem of source localization

(Hämäläinen and Ilmoniemi, 1994). This required the input of a noise covariance matrix which was generated using the last 300ms of raw data from each of the 1s onset VEP segments. The output converts scalp voltage to current density (amperes) and gives an estimate of activity at every dipole used to model the cortical surface.

The dipoles within each region of interest (ROI) were summed together to get an average waveform for each ROI. The amplitudes of the source estimates were converted to z scores within subjects with respect to baseline (baseline: 600-1000ms after stimulus onset) before being averaged across participants. This required taking the average of and standard deviation of the baseline, then at each time point taking the difference from the mean and dividing by the standard deviation. The z scored data were then rectified in order to avoid the issue of polarity flips: because the surface of the cortex undergoes many folds in the sulci and gyri, the polarity of the sources gets flipped depending on the dipole orientations. Without rectification inverted dipoles would cancel one another and produce little response. Figure 24 shows average source estimates for crVEPs in the dorsal and ventral ROIs.

Figure 24: Average source estimates for crVEPs. Left column shows responses in ventral stream, right column dorsal stream. Y axis is average z score of source estimates across participants

Figure 24 shows differences in amplitude within conditions through the dorsal and ventral streams. Achromatic amplitudes have an attenuated n1 response in early visual areas compared to ventral areas in the early portion of the signal (~200 ms after onset). These early areas in the achromatic response have a large later component (~370ms). The chromatic waveforms have early components in both early and ventral visual areas, and the L-M seems to have more of a pronounced spike than S-(L+M). This suggests that the electrode activity for the early component in the achromatic response may be coming more from ventral visual areas, while chromatic is also distributed in early and ventral visual areas

Figure 25: Comparison of summed activity in early and ventral visual ROIs. Top: Same data as figure 6 but collapsed across early visual areas (V1, V2, V3) and ventral (hv4, VO1, VO2). Bottom: Average latency (left) and amplitude (right) of n1 component across subjects for above waves.

Figure 25 shows average waveforms for early vs. ventral visual areas separately for each condition (top). Average latencies and amplitudes were averaged across observers (bottom), error bars represent standard error of the mean. A repeated measures ANOVA on the latencies revealed no main effect for condition or region, but a marginally significant interaction between region and condition F(2, 5) = 4.7, p = .089. While there doesn’t appear to be a significant difference in chromatic latencies between early and ventral areas (if anything chromatic latencies are slightly faster in ventral than early areas), achromatic information seems to be coming later to the ventral areas. Although the achromatic condition seems largely attenuated in the average waveform, a repeated measure ANOVA on the amplitudes produced no significant main effects or interactions between condition and region.

SSVEPs

Steady-state VEPs were also recorded in a continuous presentation where the same

Gabor fields as in the onset were exchanged with an antiphase version of itself in a reversal paradigm. Each image was displayed for 200ms, resulting in an exchange rate of 5 Hz. Each of the three conditions was presented for 48s.

These data were primarily analyzed in the frequency domain using the fast Fourier transform (FFT). This allows for objective quantification of amplitude of a particular frequency.

In this case we were interested in the 2nd harmonic, which is the rate at which the image is presented. Figure 26 shows average SSVEP waveforms across observers (left) and the average amplitude of the 2nd harmonic (right).

Figure 26: Average SSVEP waveforms (left) and the corresponding 2nd harmonic amplitudes calculated using the

FFT (right).

There is a difference between the waveforms produced by onset and SSVEP presentations: while L-M produced strong onset response, its SSVEP response was smaller relative to the other two conditions. This is apparent in the red line in the left of figure 8 being

of smaller amplitude than the blue and black lines, and the smaller L-M bar in the right graph of figure 26.

Significance Level SSVEP 2nd Harmonic 1

0.9

0.8

p value p 0.7

- 1 0.6 0.5 L-M L+M S-(L+M)

Figure 27: Significance level of T-Circ test results. Each subject’s value was an average of the 30 posterior electrodes’ p values. The y axis is 1-pvalue in order to display the data as increasing for more significant.

A T circ test was done on each of the electrodes as in the onset, and the average significance averaged across observers for each condition is plotted in figure 27 above (Victor and Mast, 1991). These p values are inversely correlated to the amplitudes (i.e. higher amplitudes usually result in lower p values). These t-circ tests were also done on the dipole source data and the average p values are plotted below in figure 28. This was done by inputting the raw data into the minimum norm estimate, while for the onset data only the averages were input. The SSVEP source data was not rectified to positive in order to maintain the frequency component.

Figure 28: Estimates of 2nd harmonic SSVEP ROI activity.

Looking at figure 28, there doesn’t appear to be any significant differences between the significance levels of the different colors in these different regions, although it is notable that L-

M seems to have less significance, reflecting the electrode results.

Chapter 5: High Density Investigation of the Watercolor Illusion

While the color we experience is correlated with the wavelength of light absorbed by photoreceptors, our visual system performs other computations which lead to color sensation.

This is apparent in many visual illusions such as simultaneous contrast, the Cornsweet illusion,

White’s effect, and neon color spreading. There are other long term aftereffects such as the

McCollough effect where achromatic patterns appear chromatic much longer than the fleeting afterimages of non-patterned stimuli.

The next series of experiments uses the same logic and methodology as the previous chapter but applies it to a color spreading illusion known as the watercolor effect (Pinna,

Brelstaff and Spillmann, 2001, Broerse, Vladusich and O’Shea, 1998). This illusion occurs when thin lines (of higher spatial frequency than the multi-Gabor fields) form a contrasting boundary.

The watercolor illusion has a similar contrast profile to the Cornsweet Illusion and has been shown psychophysically to be measurable for chromatic (Devinck et al. 2005, 2006, Devinck and

Knoblauch, 2012) and achromatic (Cao Yazdanbakhsh and Mingolla, 2011) contours.

One previous study demonstrated the watercolor illusion is measureable using SSVEP

(Coia et al., 2014). Coia et al. exchanged the illusion with a control pattern (see figure 29) and produced a reliably larger fundamental harmonic, indicating that the illusion may be of cortical origin. The current study set out to determine the cortical locus of this illusion using the source localization techniques outlined in the previous chapter.

Figure 29: Watercolor stimuli: Rows designate color axes and columns designate condition. Left to right: Filled

Control is control stimulus with physically filled in color. Middle is illusion (no physical inner surface color, just illusory spreading from contours. Right is control with braided contours and no surface color.

These stimuli are similar in spatial structure to those used in previous studies (Coia et al.

2014 a, Coia and Crognale, 2014 b). At a viewing distance of 114 cm, contours subtended 4.5 arcmin in width and the resulting columns were 0.66° s wide by 6.7° s long. Each group of patterns comprised seven columns, separated by 0.37 degrees. In the current study these contours are modulated exclusively along the three cardinal directions in color space.

Psychophysics

Participants completed HFP on the control contours for individual isoluminance settings and subsequently contrast matching of L+M and L-M axes to a set S-(L+M) control patterns. The chromaticities of the stimuli are given in table 2. Method of constant stimuli in temporal sequence was used to compare the visibility of L+M and L-M contours of varying contrasts to a fixed S-(L+M) pattern. This involved showing one control before another control and asking the observer which was more visible. Once the contrasts of the contours were matched to each other, the illusion was measured by comparing it to a range of filled color put into the control.

The same temporal method of constant stimuli was used. See table 2 for details about chromaticities of the patterns.

Table 2: Chromaticities for Chapter 2 Watercolor Experiment

Results: Psychophysics

Previous studies have measured the magnitude of the watercolor effect by the percentage of the inducing contour required to make a perceptual match to the illusion (Devinck et al., 2005, 2006, Coia et al., 2014 a, 2014 b). Coia et al. (2014 a) found a correlation between the perceptual matches to the illusion with the SSVEP amplitudes induced by different contour

pairs. Figure 30 shows the percentage of inducing colors required to make a match for the six different colors tested in this study. The percentages in figure 12 were calculated in MBDKL space. The average percentages in the bottom right column in table 2 were converted into cone contrast space. They are overall smaller percentages but match the overall trend of figure 30.

Psychophysical Magnitude of

50 Illusion increment decrement

Percent of of Percent 0

Inducing Contour Inducing L+M S-(L+M) L-M

Figure30: Average psychophysical matches across observers. Error bars represent +-1 standard error of the mean.

For L-M and L+M, decrements have slightly larger percentage, while for S-(L+M) increments have a slightly larger percentage. L+M has an overall larger percentage than both L-

M and S-(L+M) when poles are summed together.

Results: EEG

The data were processed in the same manner as chapter 1 concerning data acquisition and pre-processing. Data were segmented according to their specified color and condition.

Graphs in figure 31 below show the average electrode responses averaged across the same set of occipital electrodes chosen for analysis in chapter 1.

Figure 31: Average onset responses across observers to watercolor stimuli. Different graphs represent different chromatic axes and different lines within graphs represent different conditions.

While differences can be seen in the waves of the different conditions for achromatic

(L+M) and S-(L+M), the L-M conditions appear more similar to each other. In the achromatic graph, the negative component for the filled control is slightly faster than the illusion, while in the S-(L+M) graph the illusion latency is faster than the filled control. This indicates that physical achromatic contrast is being processed more quickly by the visual system than physical +S contrast, while the illusion latencies seem to be on a similar timescale

Figure 32: analysis of latency and amplitude components of watercolor onset VEPs.

Above figure 32 shows average latencies and amplitudes for electrode data. In general, chromatic latencies are a bit longer than achromatic, even though perceptually they have been contrast matched. This is especially apparent in the filled control condition. While small effects in amplitude can be seen for the L+M and S-(L+M) illusion and fill conditions compared to their

controls, the L-M control amplitudes is just as large as the illusion, and even slightly larger than the filled control.

Watercolor Source Localization

As was done in Chapter 1 for the Gabors, source localization for the ROIs in the ventral visual stream was performed on the watercolor data. Figure 33 shows average, rectified source estimates for ventral ROIs. The data are separated into rows of conditions (top: control, middle: illusion, bottom: filled control) and columns for color (left: L+M, middle: S-(L+M), right: L-M)).

Figure 33: Average source localization for watercolor stimuli.

Overall the responses in V1 are not as strong as upstream areas. Looking at the left column plotting L+M conditions, it is apparent that V3v has a large response for the illusion but

not for the filled control, but both conditions show large spikes in upstream ventral areas VO1 and VO2. This increase in V3v amplitude is less distinct in the chromatic conditions.

Figure 34: Average responses found in ventral visual area 3 (V3v) in response to illusion (red) and filled control

(black).

Figure 34 shows differences in filled control vs. illusion in visual area V3v. The illusion response is larger than the filled control in the achromatic (L+M) and S-(L+M), but not in L-M.

Watercolor SSVEPs:

SSVEPs were also collected for the watercolor illusion stimuli. There were three conditions for each of the same three color pairs. Pattern reversals exchanged contrast for the contours in the control and illusion conditions, and for the contours as well as filled contrast for the filled controls. This differed from Coia et al. (2014 a) in that the illusion was not exchanged with the control but the colors of the contours were simply swapped. This avoided issues with the alignment of the contours.

Figure 35: Average SSVEP watercolor responses.

Figure 35 provides evidence supporting the hypothesis that the illusion (as well as physical color) is measurable with SSVEP. The waveforms (35, left) show larger waves for illusion and filled control compared to control for achromatic stimuli. Figure 35 (right) shows averages of illusion and filled control amplitudes normalized to the control (illusion- control)/(illusion +control). For all color axes, the illusion index is above zero. For filled control, however, L-M is reliably less than the control. This is in agreement with the smaller SSVEP amplitudes for L-M Gabor stimuli in chapter 1.

Figure 36: Watercolor SSVEP source estimates.

Figure 36 shows SSVEP data in source space. While the error bars are largely overlapping, the largest difference between illusion and filled control in the achromatic ventral graph (figure 36, top left) is in area V3v, which is where the effect was found in the onset data.

This is also the only point in the S-(L+M) ventral (figure 36, top-middle) where the illusion is above filled control. This effect does not hold up in the L-M data, but nor did it in the onset either.

Discussion

The study on the watercolor illusion revealed a candidate for the cortical locus of the illusion: ventral visual area 3 (V3v), at least for L+M and S-(L+M). One possibility for not seeing the effect for L-M is because L-M stimuli seem to be already generating large V1 responses, while the L+M V1 responses were much smaller than ventral areas. Another recent study using hdEEG and fMRI found area V3v to be involved in texture perception of symmetrical texture patterns (Kohler et al., 2016). The implication of V3v with the watercolor illusion supports the

idea that this mid-level cortical area may be computing the surface properties attributed within boundaries.

Case Study: Deuteranomolous Subject

As was mentioned earlier, one deuteranomolous observer participated in this experiment. A comparison of the psychophysics and crVEP responses can be seen in figure 37.

Figure 37: Comparison of Deutan subject to group average. Top row show psychophysical results for contrast matching for Gabors (left), watercolor control contours (middle), and watercolor illusion(right). Bottom row shows crVEP responses to Gabor fields.

The graphs in figure 37 show differences in responses from deutan subject and group average. While the suprathreshold contrast matches for L-M stimuli (top left/middle, left side) do not differ from the group, this subject seems to require less L+M to match to S-(L+M). In the watercolor illusion matches, this subject requires much more red +(L-M) and green –(L-M) to match the illusions than controls, which would agree with having an elevated threshold which some of the matches were below. VEPs (bottom row) show longer latencies for L-M, while L+M latency looks similar.

General Discussion

This section relates the findings of the proposed studies back to the specific aims. Aim 1 was to compare the magnitude of watercolor illusion along the different chromatic axes.

Previous studies have quantified the watercolor effect for chromatic (Devinck et al. 2005, 2006,

Devinck and Knoblauch, 2012, Coia et al. 2014, Coia and Crognale, 2014)as well as achromatic

(Cao et al., 2011) inducers. This study compared chromatic and achromatic watercolor illusions by contrast matching cone opponent inducers along the three cardinal axes in color space. The results show that the achromatic inducers, when contrast matched to chromatic inducers, produce significantly larger illusion magnitudes, as measured by the percentage of inducing contour.

One previous study found the L-M axis to be stronger watercolor inducers than the S-

(L+M) axis (Devinck et al., 2006). Our results support this finding with the percentages being 34,

16 and 12 for the L+M, L-M, and S-(L+M) axes, respectively. These percentages were calculated in a version of the MBDKL color space and when the chromaticities were converted to the LMS color space in table 2, these percentages were all reduced equally in half to 16, 8, and 6%. The larger achromatic matches (L+M) outweigh any differences seen along the chromatic axes (L-M) or S-(L+M).

There have been reports of asymmetries in the magnitude of the watercolor illusion along the S-(L+M) axis, with the +S (violet/blue) being stronger than –S (chartreuse/yellow) (Coia et al. 2014, Coia and Crognale, 2014, Devinck et al., 2005, Pinna, Brelstaff, and Spillmann, 2001).

Non assimilative spreading in the watercolor illusion shows asymmetries along the S axis as well

(Kimura and Kuroki, 2014). In the current study the design was not intended to look for

asymmetries along a particular axis, as the inducing stimuli were bichromatic opponent pairs, and observers were instructed to consider spreading not only within the columns but outside of them as well. Other studies using assimiliative ‘pincushion’ stimuli showed asymmetries along the achromatic axis, similar to the slightly stronger luminance decrement compared to increment in the current results (De Weert and Spillmann, 1995).

Aim 2 set out to determine whether the cortical activity (as estimated from hdEEG source localization) differed for chromatic and achromatic patterns. With the chromatic onset presentation, we find the n1 component of the color response to be present in early as well as ventral cortical areas in the visual hierarchy, while L+M n1 components were more restricted to the ventral visual areas. Specifically, L-M n1 responses were the strongest out of the three axes tested in early visual areas, which is consistent with previous fMRI studies (Engel et al., 1997).

The SSVEP results showed attenuated L-M responses compared to the onset stimuli, consistent with previous VEP studies (Rabin et al., 1994). Rabin et al. (1994) also observed decreased amplitudes for isoluminant chromatic VEPs compared to luminance defined stimuli and showed that although adaptation may play a role in the diminished response it is likely other mechanisms are involved. Some other explanations could be reduced motion mechanisms, or that the L-M system is more linear than the L+M in the number of on and off cells and the symmetry of the on and off responses.

Aim 3 served to measure the cortical activity of illusory and surface color by comparing responses evoked from the watercolor illusion to matched filled control patterns. Results revealed noticeable differences between illusion and filled surfaces for waveforms along the

L+M and S-(L+M) axes, but not so much difference along the L-M axes. The largest differences in

onset amplitude between surface and illusory color were found in area V3v for the L+M and S-

(L+M) axes (illusion > filled surface). Small effects for L-M were seen in V2v for L-M.

Latencies for chromatic stimuli were later on average than achromatic stimuli, but while illusion and control latencies showed similar latency ratios when comparing the different colors, the filled control was much faster for achromatic than chromatic. This could be because the magnitude of the achromatic illusion was stronger and therefore faster, but then we would expect the achromatic illusion to be much faster. The responses in V1 were overall smaller than later visual areas which make it difficult to say whether information is feeding back to V1 from a higher area (V3v), but it seems unlikely that the differences seen in V3v are due to feedback because they are in the early component of the waveform. Additional analyses will be done on the data to address this question.

SSVEP data show increased 2nd harmonic amplitudes for illusory and surface color compared to control, except for L-M surface color which actually showed a decrease in 2nd harmonic compared to unfilled control. This is in line with the reduced L-M SSVEPs seen in the previous experiment and implies possibly some inhibitory mechanism involved with the filled color, or the responses to the reversing stimuli cancelling each other out. SSVEP data support the onset data with illusion being relatively more active than control in area V3v.

References

1.Anderson, E. J., Dakin, S. C., & Rees, G. (2009). Monocular signals in human lateral geniculate nucleus reflect the Craik–Cornsweet–O'Brien effect. Journal of Vision, 9(12), 14 2.Anstis, S. (2013). Contour adaptation. Journal of Vision, 13(2):25, 1–14, http://www.journalofvision.org/content/13/2/25, doi:10.1167/13.2.25. 3.Anstis, S., Vergeer, M., & Van Lier, R. (2012). Luminance contours can gate afterimage colors and ‘‘real’’ colors. Journal of Vision, 12(10):2, 1–13, www.journalofvision.org/content/12/10/2, doi:10.1167/12.10.2. 4.Appelbaum, L. G., Wade, A. R., Vildavski, V. Y., Pettet, M. W., & Norcia, A. M. (2006). Cue- invariant networks for figure and background processing in human visual cortex. The Journal of neuroscience, 26(45), 11695-11708. 5.Baillet, S., Mosher, J. C., & Leahy, R. M. (2001). Electromagnetic brain mapping. Signal Processing Magazine, IEEE, 18(6), 14-30. 6.Baumgartner, G., & Hakas, P. (1962). Die Neurophysiologie des simultanen Helligkeitskontrastes. Pflügers Archiv European Journal of Physiology,274(5), 489-510. 7.Blakeslee, B. & McCourt, M. E. 1999 A multiscale spatial filtering account of the White effect, simultaneous brightness contrast and grating induction. Vision Res. 39, 4361– 4377. 8.Boyaci, H., Fang, F., Murray, S. O., & Kersten, D. (2007). Responses to lightness variations in early human visual cortex. Current Biology, 17(11), 989-993. 9.Bressan, P., Mingolla, E., Spillmann, L., & Watanabe, T. (1997). Neon color spreading: a review. Perception, 26(11), 1353-1366. 10.Brewer, A. A., Liu, J., Wade, A. R., & Wandell, B. A. (2005). Visual field maps and stimulus selectivity in human ventral occipital cortex. Nature neuroscience, 8(8), 1102-1109. 11.Broerse, J., Vladusich, T., & O’Shea, R. P. (1999). Colour at edges and colour spreading in McCollough effects. Vision research, 39(7), 1305-1320. 12.Cao, B., Yazdanbakhsh, A., & Mingolla, E. (2011). The effect of contrast intensity and polarity in the achromatic watercolor effect. Journal of vision, 11(3), 18-18. 13.Clifford, C. W., Spehar, B., Solomon, S. G., Martin, P. R., & Qasim, Z. (2003). Interactions between color and luminance in the perception of orientation. Journal of Vision, 3(2), 1-1. 14.Coia, A. J., & Crognale, M. A. (2014). Asymmetric effects of luminance and chrominance in the watercolor illusion. Frontiers in human neuroscience, 8. 15.Coia, A. J., Jones, C., Duncan, C. S., & Crognale, M. A. (2014). Physiological correlates of watercolor effect. JOSA A, 31(4), A15-A22. 16.Cornelissen, F. W., Wade, A. R., Vladusich, T., Dougherty, R. F., & Wandell, B. A. (2006). No functional magnetic resonance imaging evidence for brightness and color filling-in in early human visual cortex. The Journal of Neuroscience, 26(14), 3634-3641. 17.Cornsweet, T. N. (1970). Vision perception. Vision perception. 18.Cottereau, B. R., Ales, J. M., & Norcia, A. M. (2014). How to use fMRI functional localizers to improve EEG/MEG source estimation. Journal of neuroscience methods. 19.Cowey, A. (2010). Visual system: how does blindsight arise?. Current biology, 20(17),

R702-R704. 20.Cox, M. A., & Maier, A. (2015). Serial versus parallel processing in mid-level vision: filling-in the details of spatial interpolation. Neuroscience of Consciousness, 2015(1), niv007. 21.Craik, K. J. (1966). The nature of psychology. At the University Press. 22.Crognale, M. A., Duncan, C. S., Shoenhard, H., Peterson, D. J., & Berryhill, M. E. (2013). The locus of color sensation: Cortical color loss and the chromatic visual evoked potential. Journal of vision, 13(10), 15. 23.Crognale, M. A., Switkes, E., Rabin, J., Schneck, M. E., Hægerström-Portnoy, G., & Adams, A. J. (1993). Application of the spatiochromatic visual evoked potential to detection of congenital and acquired color-vision deficiencies. JOSA A, 10(8), 1818- 1825. 24.Dakin, S. C., & Bex, P. J. (2003). Natural image statistics mediate brightness ‘filling in’. Proceedings of the Royal Society of London B, 270, 2341–2348 25.Davey, M. P., Maddess, T., & Srinivasan, M. V. (1998). The spatiotemporal properties of the Craik–O’Brien–Cornsweet effect are consistent with ‘filling-in’. Vision Research, 38(13), 2037-2046. 26.De Valois, R. L., Abramov, I., & Jacobs, G. H. (1966). Analysis of response patterns of LGN cells. JOSA, 56(7), 966-977. 27.De Valois, R. L., Webster, M. A., De Valois, K. K., & Lingelbach, B. (1986). Temporal properties of brightness and color induction. Vision Research,26(6), 887-897. 28.Dennett, D. (1992). Filling in versus finding out: A ubiquitous confusion in cognitive science. Cognition, conception, and methodological issues. 29.Derrington, A. M., Krauskopf, J., & Lennie, P. (1984). Chromatic mechanisms in lateral geniculate nucleus of macaque. The Journal of Physiology, 357, 241. 30.Desikan, R. S., Ségonne, F., Fischl, B., Quinn, B. T., Dickerson, B. C., Blacker, D., & Albert, M. S. (2006). An automated labeling system for subdividing the human cerebral cortex on MRI scans into gyral based regions of interest. Neuroimage, 31(3), 968-980. 31.Devinck, F., & Knoblauch, K. (2012). A common signal detection model accounts for both perception and discrimination of the watercolor effect. Journal of vision, 12(3), 19-19. 32.Devinck, F., Delahunt, P. B., Hardy, J. L., Spillmann, L., & Werner, J. S. (2005). The watercolor effect: quantitative evidence for luminance-dependent mechanisms of long-range color assimilation. Vision Research, 45(11), 1413-1424. 33.Devinck, F., Hansen, T., & Gegenfurtner, K. R. (2007). Temporal properties of the chromatic and achromatic Craik–O’Brien–Cornsweet effect. Vision research, 47(27), 3385-3393. 34.Devinck, F., Hardy, J. L., Delahunt, P. B., Spillmann, L., & Werner, J. S. (2006). Illusory spreading of watercolor. Journal of vision, 6(5), 7-7. 35.DeYoe, E. A., Carman, G. J., Bandettini, P., Glickman, S., Wieser, J., Cox, R., ... & Neitz, J. (1996). Mapping striate and extrastriate visual areas in human cerebral cortex. Proceedings of the National Academy of Sciences,93(6), 2382-2386. 36.De Weert, C. M., & Spillmann, L. (1995). Assimilation: Asymmetry between brightness and darkness?. Vision Research, 35(10), 1413-1419. 37.Di Russo, F., Martínez, A., Sereno, M. I., Pitzalis, S., & Hillyard, S. A. (2002). Cortical sources of the early components of the visual evoked potential. Human brain mapping, 15(2), 95-111.

38.Di Russo, F., Pitzalis, S., Spitoni, G., Aprile, T., Patria, F., Spinelli, D., & Hillyard, S. A. (2005). Identification of the neural sources of the pattern-reversal VEP. Neuroimage, 24(3), 874-886. 39.Engel, S., Zhang, X., & Wandell, B. (1997). Colour tuning in human visual cortex measured with functional magnetic resonance imaging. Nature, 388(6637), 68-71. 40.Field, D. J. (1987). Relations between the statistics of natural images and the response properties of cortical cells. JOSA A, 4(12), 2379-2394. 41.Flevaris, A. V., Martínez, A., & Hillyard, S. A. (2014). Attending to global versus local stimulus features modulates neural processing of low versus high spatial frequencies: an analysis with event-related brain potentials.Front. Psychol, 5(277), 10-3389. 42.Friedman, H. S., Zhou, H., & von der Heydt, R. (1999). Color filling-in under steady fixation: Behavioral demonstration in monkeys and humans.PERCEPTION-LONDON- , 28(11), 1383-1396. 43.Fuller, S., & Carrasco, M. (2006). Exogenous attention and color perception: Performance and appearance of saturation and hue. Vision research, 46(23), 4032- 4047. 44.Gerrits, H. J. M. & Timmerman, G. J. M. E. N. (1969). The filling-in process in patients with retinal scotoma. Vision Res. 9, 439–442 . 45.Gerrits, H. J. M., & Vendrik, A. J. H. (1970). Simultaneous contrast, Filling in process and information processing in man’s visual system. Experimental Brain Research, 11, 411– 430. 46.Gerth, C., Delahunt, P. B., Crognale, M. A., & Werner, J. S. (2003). Topography of the chromatic pattern-onset VEP. Journal of vision, 3(2), 5-5. 47.Gilchrist, A. L. (2007). Lightness and brightness. Current Biology, 17(8), R267-R269. 48.Gramfort, A., Papadopoulo, T., Olivi, E., & Clerc, M. (2010). OpenMEEG: open source software for quasistatic bioelectromagnetics. Biomed. Eng. Online, 9(1), 45. 49.Hadjikhani, N., Liu, A. K., Dale, A. M., Cavanagh, P., & Tootell, R. B. (1998). Retinotopy and color sensitivity in human visual cortical area V8. Nature neuroscience, 1(3), 235- 241. 50.Hämäläinen, M. S., & Ilmoniemi, R. J. (1994). Interpreting magnetic fields of the brain: minimum norm estimates. Medical & biological engineering & computing, 32(1), 35-42. 51.Hansen, T., & GEGENFURTNER, K. R. (2009). Independence of color and luminance edges in natural scenes. Visual neuroscience, 26(01), 35-49. 52.Hazenberg, S. J., & van Lier, R. (2013). Afterimage watercolors: an exploration of contour-based afterimage filling-in. Frontiers in psychology, 4. 53.Helson, H. (1963). “Studies of anomalous contrast and assimilation,” JOSA 53, 179-184. 54.Highsmith, J., & Crognale, M. A. (2010). Attentional shifts have little effect on the waveform of the chromatic onset VEP. Ophthalmic and Physiological Optics, 30(5), 525- 533. 55.Hsieh, P. J., & Tse, P. U. (2010). “Brain‐reading” of perceived colors reveals a feature mixing mechanism underlying perceptual filling‐in in cortical area V1. Human brain mapping, 31(9), 1395-1407. 56.Huang, X., & Paradiso, M. A. (2008). V1 response timing and surface filling-in. Journal of neurophysiology, 100(1), 539-547. 57.Hubbard, E. M., & Ramachandran, V. S. (2005). Neurocognitive mechanisms of synesthesia. Neuron, 48(3), 509-520.

58.Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology, 160(1), 106. 59.Hubel, D. H., & Wiesel, T. N. (1965). Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. Journal of neurophysiology, 28(2), 229- 289. 60.Johnson, E. N., Hawken, M. J., & Shapley, R. (2001). The spatial transformation of color in the primary visual cortex of the macaque monkey. Nature neuroscience, 4(4), 409- 416. 61.Kanizsa, G. (1976). Subjective contours. Scientific American, 234(4), 48-52. 62.Kastner, S., De Weerd, P., Desimone, R., & Ungerleider, L. G. (1998). Mechanisms of directed attention in the human extrastriate cortex as revealed by functional MRI. science, 282(5386), 108-111. 63.Kimura, E., & Kuroki, M. (2014). Assimilative and non-assimilative color spreading in the watercolor configuration. Frontiers in Human Neuroscience, 8, 722. http://doi.org/10.3389/fnhum.2014.00722 64.Kohler, P. J., Clarke, A., Yakovleva, A., Liu, Y., & Norcia, A. M. (2016). Representation of Maximally Regular Textures in Human Visual Cortex. The Journal of Neuroscience, 36(3), 714-729. 65.Komatsu, H. (2006). The neural mechanisms of perceptual filling-in. Nature reviews neuroscience, 7(3), 220-231. 66.Krauskopf, J. (1963). Effect of retinal image stabilization on the appearance of heterochromatic targets. JOSA, 53(6), 741-741. 67.Kulikowski, J. J., Murray, I. J., & Parry, N. R. A. (1989). Electrophysiological correlates of chromatic-opponent and achromatic stimulation in man. In Colour vision deficiencies IX (pp. 145-153). Springer Netherlands. 68.Kybic, J., Clerc, M., Abboud, T., Faugeras, O., Keriven, R., & Papadopoulo, T. (2005). A common formalism for the integral formulations of the forward EEG problem. Medical Imaging, IEEE Transactions on, 24(1), 12-28. 69.Lamme, V. A. (1995). The neurophysiology of figure-ground segregation in primary visual cortex. The Journal of Neuroscience, 15(2), 1605-1615. 70.Livingstone, M. S., & Hubel, D. H. (1984). Anatomy and physiology of a color system in the primate visual cortex. J Neurosci, 4(1), 309-356. 71.Livingstone, M., & Hubel, D. (1988). Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science, 240(4853), 740-749. 72.MacLeod, D. I., & Boynton, R. M. (1979). Chromaticity diagram showing cone excitation by stimuli of equal luminance. JOSA, 69(8), 1183-1186. 73.McCourt, M. E., & Foxe, J. J. (2004). Brightening prospects for early cortical coding of perceived luminance: a high-density electrical mapping study. Neuroreport, 15(1), 49- 56. 74.Meadows, J. C. (1974). Disturbed perception of colours associated with localized cerebral lesions. Brain, 97(4), 615-632. 75.Mendola, J. D., Conner, I. P., Sharma, S., Bahekar, A., & Lemieux, S. (2006). fMRI measures of perceptual filling-in in the human visual cortex. Journal of Cognitive Neuroscience, 18(3), 363-375. 76.Mishkin, M., Ungerleider, L. G., & Macko, K. A. (1983). Object vision and spatial vision: two cortical pathways. Trends in neurosciences, 6, 414-417.

77.Mullen, K. T. (1985). The contrast sensitivity of human colour vision to red-green and blue-yellow chromatic gratings. The Journal of Physiology, 359, 381. 78.Murakami I, 2008 “Perceptual filling-in” Encyclopedia of Neuroscience (Berlin: Springer) 79.Nathans, J., Piantanida, T. P., Eddy, R. L., Shows, T. B., & Hogness, D. S. (1986). Molecular genetics of inherited variation in human color vision. Science, 232(4747), 203-210. 80.O’BRIEN, V. I. V. I. A. N. (1958). Contour perception, illusion and reality. JOSA, 48(2), 112-119. 81.Paradiso, M. A., & Nakayama, K. (1991). Brightness perception and filling-in. Vision research, 31(7), 1221-1236. 82.Perna, A., Tosetti, M., Montanaro, D., & Morrone, M. C. (2005). Neuronal mechanisms for illusory brightness perception in humans. Neuron, 47(5), 645-651. 83.Pessoa, L., Thompson, E., & Noë, A. (1998). Filling-in is for finding out.Behavioral and brain sciences, 21(06), 781-796. 84.Pinna B. (2004). The role of the Gestalt principle of similarity in the watercolor illusion. Spat. Vis. 18, 185–207 10.1163/1568568053320639 85.Pinna, B., & Grossberg, S. (2005). The watercolor illusion and neon color spreading: a unified analysis of new cases and neural mechanisms. JOSA A, 22(10), 2207-2221. 86.Pinna, B., Brelstaff, G., & Spillmann, L. (2001). Surface color from boundaries: a new ‘watercolor’illusion. Vision research, 41(20), 2669-2676. 87.Pinna, B., Werner, J. S., & Spillmann, L. (2003). The watercolor effect: a new principle of grouping and figure–ground organization. Vision Research,43(1), 43-52. 88.Porciatti, V., & Sartucci, F. (1999). Normative data for onset VEPs to red-green and blue- yellow chromatic contrast. Clinical Neurophysiology, 110(4), 772-781. 89.Press, W. A., Brewer, A. A., Dougherty, R. F., Wade, A. R., & Wandell, B. A. (2001). Visual areas and spatial summation in human visual cortex. Vision research, 41(10), 1321- 1332. 90.Rabin, J., Gooch, J., & Ivan, D. (2011). Rapid quantification of color vision: the cone contrast test. Investigative ophthalmology & visual science, 52(2), 816-820. 91.Rabin, J., Switkes, E., Crognale, M., Schneck, M. E., & Adams, A. J. (1994). Visual evoked potentials in three-dimensional color space: correlates of spatio-chromatic processing. Vision research, 34(20), 2657-2671. 92.Ramachandran, V. S., & Gregory, R. L. (1991). Perceptual filling in of artificially induced scotomas in human vision. Nature, 350(6320), 699-702. 93.Regan, D. (1973). Evoked potentials specific to spatial patterns of luminance and colour. Vision Research, 13(12), 2381-2402. 94.Rossi, A. F., & Paradiso, M. A. (1996). Temporal limits of brightness induction and mechanisms of brightness perception. Vision Research,36(10), 1391-1398. 95.Sasaki, Y., & Watanabe, T. (2004). The primary visual cortex fills in color. Proceedings of the National Academy of Sciences, 101(52), 18251-18256. 96.Schmolesky, M. T., Wang, Y., Hanes, D. P., Thompson, K. G., Leutgeb, S., Schall, J. D., & Leventhal, A. G. (1998). Signal timing across the macaque visual system. Journal of neurophysiology, 79(6), 3272-3278. 97.Shapley, R., & Hawken, M. J. (2011). Color in the cortex: single-and double-opponent cells. Vision research, 51(7), 701-717. 98.Shimojo, S., Kamitani, Y. & Nishida, S. (2001). Afterimage of perceptually filled-in

surface. Science 293, 1677–1680. 99.Skiba, R. M., Duncan, C. S., & Crognale, M. A. (2014). The effects of luminance contribution from large fields to chromatic visual evoked potentials. Vision research, 95, 68-74. 100.Skoczenski, A. M., & Norcia, A. M. (1999). Development of VEP Vernier acuity and grating acuity in human infants. Investigative Ophthalmology and Visual Science, 40, 2411-2417. 101.Smith, A. T., Singh, K. D., Williams, A. L., & Greenlee, M. W. (2001). Estimating receptive field size from fMRI data in human striate and extrastriate visual cortex. Cerebral cortex, 11(12), 1182-1190. 102.Spillmann, L. (2009). Phenomenology and neurophysiological correlations: Two approaches to perception research. Vision Research, 49(12), 1507-1521. 103.Spillmann, L., & Werner, J. S. (1996). Long-range interactions in visual perception. Trends in neurosciences, 19(10), 428-434. 104.Switkes, E., & Crognale, M. A. (1999). Comparison of color and luminance contrast: apples versus oranges?. Vision Research, 39(10), 1823-1831. 105.Tadel, F., Baillet, S., Mosher, J. C., Pantazis, D., & Leahy, R. M. (2011). Brainstorm: a user-friendly application for MEG/EEG analysis. Computational intelligence and neuroscience, 2011, 8. 106.Troxler D. (1804). Ueber das Verschwinden gegebener Gegensta¨nde 107.Van Essen, D. C., & Maunsell, J. H. (1983). Hierarchical organization and functional streams in the visual cortex. Trends in neurosciences, 6, 370-375. 108.Victor, J. D., & Mast, J. (1991). A new statistic for steady-state evoked potentials. Electroencephalography and clinical neurophysiology, 78(5), 378-388. 109.Von der Heydt, R., Friedman, H. S., & Zhou, H. (2003). Searching for the neural mechanisms of color filling-in. Filling-in: From perceptual completion to cortical reorganization, 106-127. 110.Von der Heydt, R., Peterhans, E., & Baumgartner, G. (1984). Illusory contours and cortical neuron responses. Science, 224(4654), 1260-1262. 111.Wachtler, T., Sejnowski, T. J., & Albright, T. D. (2003). Representation of color stimuli in awake macaque primary visual cortex. Neuron, 37(4), 681-691. 112.Wade, A. R., Brewer, A. A., Rieger, J. W., & Wandell, B. A. (2002). Functional measurements of human ventral occipital cortex: retinotopy and colour. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 357(1424), 963-973. 113.Webster, M. A., Switkes, E., & De Valois, K. K. (1990). Orientation and spatial-frequency discrimination for luminance and chromatic gratings. JOSA A, 7(6), 1034-1049. 114.Weil, R. S., & Rees, G. (2011). A new taxonomy for perceptual filling-in. Brain research reviews, 67(1), 40-55. 115.Werner, J. S., Pinna, B., & Spillmann, L. (2007). Illusory Color & the Brain. Scientific American, 296(3), 90-95. 116.Young, T. (1802). The Bakerian lecture: On the theory of light and colours. Philosophical transactions of the Royal Society of London, 92, 12-48. 117.Zeki, S. "Improbable areas in color vision." The Visual Neurosciences 2 (2004): 1029- 1039. 118.Zeki, S., & Bartels, A. (1999). The clinical and functional measurement of cortical (in) activity in the visual brain, with special reference to the two subdivisions (V4 and V4α)

of the human colour centre. Philosophical Transactions of the Royal Society of London B: Biological Sciences, 354(1387), 1371-1382. 119.Zweig, S., Zurawel, G., Shapley, R., & Slovin, H. (2015). Representation of Color Surfaces in V1: Edge Enhancement and Unfilled Holes. The Journal of Neuroscience, 35(35), 12103-12115.