Misbinding of Color to Form in Afterimages

Misbinding of color to form in afterimages

STEVEN K. SHEVELL,1,2 REBECCA ST.CLAIR,1,3 and SANG WOOK HONG1,3 1Department of Psychology, University of Chicago, Chicago, Illinois 2Department of Ophthalmology & Visual Science, University of Chicago, Chicago, Illinois 3Department of Psychology, Vanderbilt University, Nashville, Tennessee (Received October 1, 2007; Accepted December 3, 2007!

Abstract Under dichoptic viewing conditions, rivalrous gratings that differ in both color and form can give the percept of the color from one eye in part of the form in the other eye. This study examined the afterimage following such misbinding of color to form. The first experiment established that afterimages of the misbound percept were seen. Two possible mechanisms for the misbound afterimage are ~1! persisting retinal representations that are rivalrous and subsequently resolved to give misbinding, as during rivalrous viewing, and ~2! a persisting response from a central neural representation of the misbound percept with the form from one eye and color from the other eye. The results support afterimage formation from a central representation of the misbound percept, not from resolution of rivalrous monocular representations. Keywords: Afterimage, Binocular rivalry, Color rivalry, Feature binding

Introduction work shows the afterimage is from a neural representation of a surface, not from representations of physical stimuli that cause An afterimage is a percept of a visual stimulus after the physical filling-in. The neural mechanism is very likely to be cortical but is stimulus has been removed. A basic question ~Loomis, 1972; thought to be monocular ~Kamitani & Shimojo, 2004!. Sakitt & Long, 1979; Virsu & Laurinen, 1977; Wilson, 1997! is These previous studies examine afterimages whose appear- whether the experience of an afterimage depends on a retinal ances depend on stimulation of only one eye, not on afterimage representation of the extinguished stimulus or, at least in part, on a appearance that arises from binocular combination. The present cortical representation of it. The experiments presented here inves- study uses dichoptic presentation of two orthogonal chromatically tigated whether persisting signals from a retinal ~or other monoc- different equiluminant gratings; rivalry between these stimuli is ular! neural mechanism can fully account for afterimages from known to cause misbinding of color to form ~Hong & Shevell, percepts that depend on combining signals from the two eyes. If 2006!. For example, if a vertical orange0gray grating is presented afterimages are purely retinal in origin, then percepts that depend to one eye and a horizontal blue0gray grating to the other eye, an on binocular combination should not influence the afterimage. If, orange0blue grating is often perceived. The orientation alternates however, afterimages are influenced by a neural representation that between vertical and horizontal but the color from the suppressed depends on binocular combination, then afterimages, at least in part, form is expressed in non-retinotopic regions of the dominant form are due to a cortical neural representation that incorporates signals ~thus misbinding of color to form!. An unexplored question, in- from both eyes. vestigated here, is the origin of the afterimage following such Behavioral studies show that cortical processes influence after- misbound percepts. images. Interocular suppression of a binocularly rivalrous stimulus The measurements showed that a misbound afterimage was per- alters the strength of its subsequent afterimage ~Tsuchiya & Koch, ceived but only if misbinding was seen during viewing of rivalrous 2005; Gilroy & Blake, 2005!. Afterimages, however, still might stimuli. This finding is consistent with a contribution to afterimages arise from a retinal representation that is regulated in strength by from a cortical mechanism encoding the observer’s percept while a cortical neural response driven by the percepts during rivalrous viewing rivalrous stimuli, even when the percept results from mis- viewing. Another important result concerns the afterimage from binding the color in one eye to the form in the other eye. viewing a perceptually filled-in surface: the afterimage includes the opponent color of the filled region without adaptation of the corresponding portion of the retina ~Shimojo et al., 2001!. This Materials and methods Apparatus

Address correspondence and reprint requests to: Steven K. Shevell, Two rivalrous gratings were presented using a Macintosh G4 Visual Science Laboratories, The University of Chicago, 940 E 57th Street, computer driving a Sony color display ~GDM-F520!. The pixel Chicago, IL 60637. E-mail: [email protected] resolution of the cathode ray tube ~CRT! was 1360 ϫ 1024 with a

355 356 Shevell et al. refresh rate of 75 Hz noninterlaced. Ten-bit lookup tables were of Fig. 1a!. The rivalrous stimuli were 2 cpd square-wave gratings used to linearize the red, green, and blue guns of the color CRT. within a circular aperture of diameter 28. Grating luminance was The stimuli on the left ~right! side of the CRT screen were 7.6 cd0m2 throughout the experiments. Three pairs of chromatic- presented to the left ~right! eye with a haploscope. ities were used in the experiments ~Table 1!. Chromaticities are given in cone-stimulus coordinates L0~LϩM! and S0~LϩM! ~MacLeod & Boynton, 1979!. The unit of S0~LϩM! is arbitrary Stimuli and normalized here to 1.0 for equal-energy-spectrum ~EES! Orthogonally oriented equiluminant gratings, one to each eye, “white.” The chromaticities were chosen toward the extremes of were different in chromaticity ~an example is shown in the left half the display’s gamut for L0~LϩM!, and to allow a comparison

Fig. 1. Top row: Schematic representation of continuously presented dichoptic stimuli. Middle row: Schematic representation of pulsed dichoptic stimuli. Bottom row: Four percepts observers could report during the experiments. Misbinding of color to form in afterimages 357

Table 1. Three pairs of chromaticities used in the experiments. second condition, the grating to the left eye was presented alone Chromaticities are given in coordinates L/(LϩM) and S/(LϩM) for 500 ms and then the grating to the right eye was presented (see text). The approximate color appearance of each alone for 500 ms. The guidelines and nonius lines were continu- chromaticity is shown in the rightmost column ously presented to each eye. This was continued for a total of 40 s. The stimulation to each eye was the same in both conditions but Color the gratings were not presented simultaneously ~and thus not Pair Chromaticity Appearance rivalrous! in the second condition. Perceptual alternation while viewing the gratings and during 1L0~LϩM! ϭ 0.708 S0~LϩM! ϭ 3.2 L0~LϩM! ϭ 0.621 S0~LϩM! ϭ 0.4 the subsequent period of afterimages was measured in each run. Observers pressed different buttons on a game pad to report their 2L0~LϩM! ϭ 0.621 S0~LϩM! ϭ 3.2 L0~LϩM! ϭ 0.708 S0~LϩM! ϭ 0.4 percept and held the button until the percept changed ~e.g., observers pressed one button when the stimulus from the left eye was 3L0~LϩM! ϭ 0.708 S0~LϩM! ϭ 1.0 L0~LϩM! ϭ 0.621 S0~LϩM! ϭ 1.0 perceived and pressed a different button when the stimulus from the right eye was perceived!. Buttons were assigned for the percept of the left-eye stimulus alone, the right-eye stimulus alone, misbinding with the orientation of the left-eye stimulus, and misbinding with the orientation of the right-eye stimulus ~Fig. 1c!. Observers were told not to press any button if they experienced piecemeal or between a large accompanying difference in S0~LϩM! versus no ambiguous percepts. Afterimage percepts were measured in the S0~LϩM! difference. Following the presentation of rivalrous grat- same manner. In the second experiment, however, in the pulsed ings for 20 s, a uniform achromatic field ~metameric to EES! was simultaneous viewing condition ~only! observers were asked to presented in the circular apertures to evoke an afterimage ~right report only misbound percepts ~to determine if and when misbind- side of Fig. 1a!. The luminance of the achromatic field was also 7.6 ing occurred! because briefly pulsed simultaneous ~and rivalrous! 2 cd0m . In the first experiment, the gratings were presented con- stimuli caused a very rapid change in percept. Pilot work showed tinuously and the background was dark. In the second experiment, the percept changed too rapidly to be accurately tracked with the gratings were pulsed on and off; the luminance of the back- different button-press responses. During the afterimage phase, they 2 ground was fixed at 5.2 cd0m ~Fig. 1b!. The non-zero background were told to report both monocular and misbound percepts, as in luminance was found to enhance the visibility of the afterimages other conditions. Observers repeated every condition five times. following pulsed stimuli. The total duration of each percept, averaged over the replications White circular guidelines surrounded each grating to aid stable ~all on different days!, was used for further analysis. fixation and maintain correct alignment of the two eyes. Top and left nonius lines were presented within the guideline to the left eye, and bottom and right nonius lines were presented within the Observers guideline to the right eye. With each eye properly fixated, the Three observers participated in the study. Each observer was tested observer perceived two horizontally aligned and two vertically for normal color vision using a Neitz anomaloscope and for normal aligned nonius lines. stereoscopic vision using the Titmus Stereo Test. Observer R.S. is one of the authors and had prior experience with binocular rivalry experiments. Observer P.K. had experience with binocular rivalry Procedure but was not knowledgeable about this study. Observer A.B. was Within each session, each chromaticity pair was presented in two naïve. Consent forms were completed in accordance with the separate trials with the pair reversed between the left and right eyes. policy of the University of Chicago’s Institutional Review Board. This controlled for eye dominance. In a given session, the orientation of the gratings remained the same ~e.g., the grating presented Results to the right eye was always vertical and the grating presented to the left eye was always horizontal!. In a separate session, the orienta- Experiment 1. Continuous presentation tions of the gratings were reversed so that each eye viewed each of rivalrous stimuli orientation in a counterbalanced design. Each session began with the presentation of the guidelines and nonius lines surrounding an During dichoptic presentation of two orthogonally oriented, equi- achromatic field in the area where the gratings would be presented luminant and chromatically rivalrous gratings ~e.g., vertical orange0 ~these stimuli were not rivalrous!. After achieving good fusion and gray and horizontal blue0gray gratings!, misbound percepts were alignment of the nonius lines, the observer pressed a button on a observed often ~e.g., an orange0blue grating!. Misbound after- game pad to begin an experimental run. Then the grating to the left image percepts also were observed following the presentation of eye was presented alone for 5 s followed by the grating to the right the rivalrous stimuli. The misbound afterimage was observed for eye alone for 5 s. This allowed the observer to identify each mon- every chromaticity pair and by each observer, though such after- ocular stimulus in the following test period. A 30 s dark period images were very brief for observer A.B. followed, and then the rivalrous gratings were presented for 20 s. The total duration of each percept during viewing ~Hollins, 1980; Following presentation of the rivalrous gratings, the uniform ach- Paffen et al., 2006!, known as exclusive visibility, was compared to romatic field was presented to each eye for 20 s. the duration of each percept in the afterimage ~Fig. 2!. The hori- In the second experiment, the gratings were pulsed on and off zontal axis shows the different percepts during viewing and in the prior to measuring the afterimage. In the first condition, the afterimage, and the vertical axis shows the amount of time of each rivalrous gratings flashed on and off simultaneously: on for 500 ms percept. During viewing, misbinding was perceived significantly and off for 500 ms. This continued for 40 cycles ~i.e., 40 s!.Inthe longer than the monocular stimuli for two of the three observers. In 358 Shevell et al.

Fig. 2. Duration of each percept ~vertical axis! during 20 s of continuous dichoptic viewing or in the afterimage ~horizontal axis!. Solid bars: Monocular percepts. Shaded bars: Misbound percepts. Each column shows results for a different chromaticity pair ~see Table 1! and each row is for a different observer. Error bars are standard errors based on repeated measurements on different days.

the afterimage, the misbound percept was seen but not for a longer When the stimulus to one eye was alternated with the stimulus period of time than monocular afterimages. to the other eye, observers never experienced misbinding during Note that the large difference in duration for perceived mis- viewing or in the afterimage ~Fig. 3b!. The horizontal axis again binding during viewing versus in the afterimage merely reflects the shows the different chromaticity pairs and the vertical axis shows 20-s viewing period used in the experiment. Stimulus duration was the amount of time each percept was experienced. In this condi- chosen so a possible neural representation of the misbound percept tion, observer R.S. perceived some overlap between the right-eye would be excited for a substantial duration. No afterimage in any and left-eye stimuli when each stimulus was alternated for 500 ms, condition here was several seconds long. presumably due to brief persistence at each stimulus offset, so for this observer each stimulus was on for 400 ms and off for 600 ms. This left a 100 ms dark gap between the alternating left- and Experiment 2. Pulsed presentation of rivalrous stimuli right-eye stimuli, which eliminated the problem for this observer. When the stimuli to both eyes were pulsed on and off simultaneously, the observers experienced misbinding during viewing and Discussion in the afterimage ~Fig. 3a!. Recall that observers responded to only misbound percepts during pulsed simultaneous viewing so the lack The percept of misbinding in an afterimage is a new observation. of monocular results in the left panels reflects the procedure, not Misbound percepts are known to occur during dichoptic viewing perception. The horizontal axis shows the different chromaticity ~Hong & Shevell, 2006! but the subsequent afterimage was not pairs and the vertical axis shows the amount of time each percept investigated. Studies of afterimages following dichoptic viewing was experienced. Misbound afterimage percepts were common ~Tsuchiya & Koch, 2005; Gilroy & Blake, 2005! do not consider ~right panels!. ~Observer R.S. had slightly different timing with percepts that combine features from both eyes. The results here 400 ms on and 600 ms off instead of 500 ms on and 500 ms off, show that the misbound afterimage is experienced but only after for reasons described below!. perceiving misbinding during dichoptic viewing. Misbinding of color to form in afterimages 359

Fig. 3. ~a! Duration ~vertical axis! of each percept during 40 s of pulsed presentation with the left and right eyes stimulated dichoptically and simultaneously ~left column! and in the afterimage following stimulation ~right column!. Solid bars: Monocular percepts. Shaded bars: Misbound percepts. Results are shown for the three chromaticity pairs ~horizontal axis!. Each row has results for a different observer. Error bars are standard errors based on repeated measurements on different days. ~b! As in ~a! but with the left and right eyes stimulated alternately instead of simultaneously ~see text!. 360 Shevell et al.

The neural representation mediating misbound ocular ~probably retinal! driven by light entering the eye and the afterimages other more central that depends on the percept while viewing rivalrous stimuli. A different type of central mechanism is revealed Misbound afterimages may be caused by a neural representation of by studies in which dichoptic stimuli suppress one eye’s stimulus the misbound percept during viewing. If so, afterimages are driven from awareness. The suppression may influence a central mecha- by a representation at or after a neural locus where signals from the nism that regulates afterimages driven by persisting retinal re- two eyes are combined because misbound percepts during viewing sponses but this is a separate question than afterimages that follow have the chromatic information from one eye expressed within the from binocularly combined percepts, as with misbinding. form presented to the other eye. An alternative, however, is that Others have proposed two neural sources for afterimages. retinal ~or otherwise monocular! neural representations mediate Afterimages are seen within an area not stimulated during viewing the misbound afterimages. On this view, the only persisting rep- but perceived to be part of a colored surface due to perceptual resentations are monocular; they are rivalrous and resolved cen- filling-in. Shimojo et al. ~2001! propose two sources for after- trally to give misbinding in the afterimage in the same way that images: ~1! adapted retinal mechanisms and ~2! a neural represen- dichoptically presented rivalrous stimuli result in a misbound tation of a perceptually filled-in surface; in fact, they observe percept during viewing. These alternatives were distinguished rivalry between afterimages from these two posited sources. Their using pulsed stimuli in Experiment 2. retinal process is comparable to the monocular one proposed here If misbound afterimages have their origin in only rivalrous but their surface mechanism cannot account for misbound after- monocular representations ~with subsequent misbinding at a cen- images because they localize it within a monocular neural pathway tral locus!, then the critical variable for the misbound afterimage is ~Kamitani & Shimojo, 2004!. Misbinding must follow binocular the stimulation delivered to each eye. When the stimuli were combination so the afterimage of a misbound percept follows from pulsed in Experiment 2, each eye received the same stimulation a persisting neural representation that combines signals from both regardless of whether the two eyes were stimulated simultaneously eyes. or alternately. On the monocular hypothesis, therefore, the simultaneous and alternating conditions should give the same misbound afterimages. If, however, misbound afterimages depend on the Acknowledgments persisting response from a central mechanism that represents the misbound percept during viewing, then the misbound afterimage Supported by NIH grant EY-04802. Publication supported in part by an should be experienced only when the left and right eyes are unrestricted grant to the Department of Ophthalmology & Visual Science stimulated simultaneously. Misbinding results from color-form from Research to Prevent Blindness. binocular rivalry but only the simultaneous condition, not the alternating condition, has rivalrous dichoptic stimuli. References During pulsed simultaneous viewing, the misbound percept was seen frequently ~left panel, Fig. 3a!. During pulsed alternating Gilroy, L.A. & Blake, R. ~2005!. The interaction between binocular viewing, a misbound percept was never seen ~left panel, Fig. 3b!. rivalry and negative afterimages. Current Biology 15, 1740–1744. These are the pre-conditions that imply the cortical-locus hypoth- Hollins, M. ~1980!. The effect of contrast on the completeness of binocular rivalry suppression. Perception and Psychophysics 27, 550–556. esis predicts misbound afterimages in only the simultaneous con- Hong, S.W. & Shevell, S.K. ~2006!. Resolution of binocular rivalry: dition, and the retinal0monocular-locus hypothesis predicts misbound Perceptual misbinding of color. Visual Neuroscience 23, 561–566. afterimages in both conditions. Kamitani, Y. & Shimojo, S. ~2004!. Global yet early processing of visual A misbound afterimage was experienced following only the surfaces. In The Visual Neurosciences, ed. Chalupa, L. & Werner, J., simultaneous-viewing condition ~right panels, Figs. 3a, 3b!. Thus, 1129–1138. Cambridge, MA: MIT Press. Loomis, J.M. ~1972!. The photopigment bleaching hypothesis of comple- only the condition giving misbound percepts during viewing re- mentary afterimages: A psychophysical test. Vision Research 12, sulted in misbound afterimages. Alternating stimulation to the two 1587–1594. eyes never caused misbound afterimages. These results support the MacLeod, D.I.A. & Boynton, R.M. ~1979!. Chromaticity diagram show- conclusion that the afterimage of a misbound grating is due to a ing cone excitation by stimuli of equal luminance. Journal of the Opical Society of America 69, 1183–1185. persisting, central neural representation that mediates the mis- Paffen, C.L.E., Tadin, D., te Pas, S.F., Blake, R. & Verstraten, F.A.J. bound percept during rivalrous viewing ~or by some other central ~2006!. Adaptive center-surround interactions in human vision revealed representation that drives the misbound percept during both view- during binocular rivalry. Vision Research 46, 599–604. ing and in the afterimage!. Misbound afterimages do not result Sakitt, B. & Long, G.M. ~1979!. Cones determine subjective offset of a from resolving rivalrous monocular afterimages. stimulus but rods determine total persistence. Vision Research 19, 1439–1441. Shimojo, S., Kamitani, Y. & Nishida, S. ~2001!. Afterimage of percep- Multiple, distinct neural processes that mediate tually filled-in surface. Science 293, 1677–1680. Tsuchiya, N. & Koch, C. ~2005!. Continuous flash suppression reduces afterimages negative afterimages. Nature Neuroscience 8, 1096–1101. Virsu, V. & Laurinen, P. ~1977!. Long-lasting afterimages caused by In the pulsed-simultaneous viewing condition, both monocular and neural adaptation. Vision Research 17, 853–860. misbound afterimages were perceived. This suggests that two Wilson, H.R. ~1997!. A neural model of foveal light adaptation and separate neural mechanisms contribute to afterimages, one mon- afterimage formation. Visual Neuroscience 14, 403–423.