Spatial Filtering, Color Constancy, and the Color-Changing Dress Department of Psychology, American University, Erica L

Journal of Vision (2017) 17(3):7, 1–20 1

Spatial filtering, color constancy, and the color-changing dress Department of Psychology, American University, Erica L. Dixon Washington, DC, USA

Department of Psychology and Department of Computer Arthur G. Shapiro Science, American University, Washington, DC, USA

The color-changing dress is a 2015 Internet phenomenon divide among responders (as well as disagreement from in which the colors in a picture of a dress are reported as widely followed celebrity commentators) fueled a rapid blue-black by some observers and white-gold by others. spread of the photo across many online news outlets, The standard explanation is that observers make and the topic trended worldwide on Twitter under the different inferences about the lighting (is the dress in hashtag #theDress. The huge debate on the Internet shadow or bright yellow light?); based on these also sparked debate in the vision science community inferences, observers make a best guess about the about the implications of the stimulus with regard to reflectance of the dress. The assumption underlying this individual differences in color perception, which in turn explanation is that reflectance is the key to color led to a special issue of the Journal of Vision, for which constancy because reflectance alone remains invariant this article is written. under changes in lighting conditions. Here, we The predominant explanations in both scientific demonstrate an alternative type of invariance across journals (Gegenfurtner, Bloj, & Toscani, 2015; Lafer- illumination conditions: An object that appears to vary in Sousa, Hermann, & Conway, 2015; Winkler, Spill- color under blue, white, or yellow illumination does not change color in the high spatial frequency region. A first mann, Werner, & Webster, 2015) and in popular approximation to color constancy can therefore be scientific press articles (Lafer-Sousa, 2015; Macknik, accomplished by a high-pass filter that retains enough 2015) featured variations of what we term the color low spatial frequency content so as to not to completely constancy hypothesis. The term color constancy refers to desaturate the object. We demonstrate the implications the fact that objects maintain a relatively stable color of this idea on the Rubik’s cube illusion; on a shirt placed appearance even when viewed under markedly different under white, yellow, and blue illuminants; and on illumination (Judd, 1940). For instance, a yellow ball spatially filtered images of the dress. We hypothesize viewed in sunlight and then fluorescent light is that observer perceptions of the dress’s color vary perceived as ‘‘yellow’’ in both conditions despite because of individual differences in how the visual illuminants with wildly different spectral compositions. system extracts high and low spatial frequency color Color constancy would not pose a theoretical problem content from the environment, and we demonstrate if the visual system were truly able to sense the spectral cross-group differences in average sensitivity to low reflectance of objects (i.e., the proportion of light an spatial frequency patterns. object reflects at each wavelength); however, the visual system has access only to the light reaching the eye. The ability to maintain color constancy despite the limited information in the environment has therefore led to the Introduction suggestion that observers infer reflectance by using their prior experience to make a best guess about scene On February 25, 2015, a phenomenon referred to as illumination, the nature of the material of the object, the color-changing dress went viral on the Internet. A and the physical configurations of the world (Brainard blogger on the website Tumblr posted a photo of a et al., 2006; Lotto & Purves, 2000). striped dress, asking her relatively small community of The color constancy hypothesis leads to a seemingly readers to vote on the perceived colors of the fabric; straightforward explanation for the color-changing some voters saw the dress as blue with black lace dress: Observers report color based on their estimation stripes, whereas others perceived it as white with gold of material reflectance, and individual differences in the lace stripes (‘‘Guys-please-help-me,’’ 2015). The sharp appearance of the dress arise because observers differ in

Citation: Dixon, E. L., & Shapiro, A. G. (2017). Spatial filtering, color constancy, and the color-changing dress. Journal of Vision, 17(3):7, 1–20, doi:10.1167/17.3.7.

This work is licensed under a Creative Commons Attribution 4.0 International License. Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 2

how they interpret the illumination and shadows in the model for brightness/lightness phenomena, shown in picture. According to this explanation, the material of Supplementary Movie 1. The model consists of a high- the dress typically appears blue and black even though pass filter with a single parameter; the parameter measured pixel values of the dress image show a range of adjusts the cutoff frequency so that when the parameter colors most easily characterized as blue and brown is small, only a narrow range of high-pass information (Lafer-Sousa et al., 2015). Observers who have good remains, and the filter acts like a high-pass edge color constancy are able to discount the yellowish detector. When the parameter is large, only a narrow illuminant and therefore perceive the dress as blue-black range of low spatial frequency content is removed, and (or perhaps blue and brown), whereas those who see the the filter acts like a process that removes blur from the dress as white and gold either have poor color constancy image. Remarkably, this simple filter can account for and are unable to discount the yellowish illuminant or the relative brightness of test patches in a natural image perceive the dress as if it is in shadow and are able to and for phenomena often attributed to unconscious discount the shadow. Still other observers who see the inference or to anchoring. In addition, Dixon et al. dress sometimes as blue and black and sometimes as (2014) showed that the size of the parameter adapts to white and gold have imperfect constancy and can switch the size of the objects in the scene. So, objects far from their interpretation of the illuminant. Another suggested an observer subtend a smaller visual angle than objects explanation is that observers are discounting shorter close to an observer; hence, a distant object would lead wavelengths based on prior experience and their to less low spatial frequency being removed. particular daytime chronotype (Lafer-Sousa et al., 2015) To our knowledge, no one has directly attempted to and that the variability in the perception of the dress account for color vision phenomena with the spatial may be accentuated because observers are more likely to filtering models frequently applied to lightness and interpret blue as a property of illumination rather than brightness (Blakeslee & McCourt, 2012). This is reflectance (Winkler et al., 2015). somewhat surprising because there are many reasons to speculate that such models would be useful for color perception. For instance, von Kries transformations and the gray world hypothesis suggest that much of the A spatial filtering approach effect of illumination is contained in the low spatial portion of the image spectra (see Smithson, 2005). The Although a framework based on illumination and response of double-opponent cells is spatially band pass reflectance has some intuitive appeal (particularly for and that of single-opponent cells is spatially low pass discussing phenomena with the media and public), (Conway, 2002; Shapley & Hawken, 2011). Color vision there are other ways of thinking about spatial aspects based on these channels would select from different of color vision that may be of relevance for under- portions of the images, and these portions likely contain standing the dress. For instance, many researchers have different types of chromatic information. Indeed, suggested that illumination can be estimated by models that propose that color ‘‘fills in’’ between averaging across portions of the scene (Buchsbaum, luminance edges in a scene (see Feitosa-Santana, 1980; Judd, 1940; Land, 1986); indeed, investigations D’Antona, & Shevell, 2011) in essence divide the image with ‘‘a gray world hypothesis’’ show that the average into high spatial frequency ‘‘edges’’ and low spatial of the scene can give reliable estimates of illuminant frequency ‘‘fill-in.’’ In addition, Werner (2014) recently chromaticity under many conditions (Barnard, Cardei, suggested two types of color constancy processes: a slow & Funt, 2002; Khang & Zaidi, 2004). Others have type, operating at a global scale for the compensation of suggested that many aspects of color contrast can be the ambient illumination, and a fast type that is locally accounted for by lateral inhibition-type processes restricted and well suited to compensation for region- (D’zmura & Singer, 2001; De Valois, Webster, De specific variations in the light field. Valois, & Lingelbach, 1986), spatial weighting func- Here, we explore how effective the Shapiro and Lu tions (Shapley & Reid, 1985; Zaidi, Yoshimi, Flanigan, (2011) and Dixon et al. (2014) spatial filtering model is & Canova, 1992), or other types of gain control that at explaining several aspects of color vision and how operate within color channels. Indeed, many bright- this can lead to an explanation about the dress. We ness/lightness phenomena previously attributed to show that adaptive high-pass filters can counteract the unconscious inference can be accounted for by spatial effects of illumination within many of the images filtering approaches that have similarities to older typically used to illustrate the effects of ‘‘discounting lateral inhibition models (Blakeslee, Pasieka, & the illuminant’’: (a) Lotto and Purves’s (2002) Rubik’s McCourt, 2005; Perna & Morrone, 2007; Robinson, cube illusion, (b) the image of the dress under three Hammon, & de Sa, 2007). different simulated illuminants, and (c) an image of a Following this approach, Shapiro and Lu (2011) and shirt under three different natural illuminants. The Dixon, Shapiro, and Lu (2014) presented a simple demonstrations suggest that we do not need to posit

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 3

that observers make assumptions about the illumi- Scientific American Mind blog post, which explained nant—a stage in models of color vision that often lacks the dress phenomenon with reference to a color quantitative specificity; instead, individual variations in constancy hypothesis (Macknik, 2015). In this version perception of the dress can arise from differences in of the Rubik’s cube illusion, the test squares are how observers’ visual systems filter the image. physically achromatic, and a yellow or blue overlay is We use the dress to investigate two hypotheses about added to the cube. the two potential roles for the high-pass channel in The color constancy explanation of the Rubik’s cube color vision. The main hypothesis assumes that the illusion, like the color constancy explanation for the high-pass filter simply acts as a way of discounting the dress, is that the perception of the color of the square illuminant, as described by Dixon et al. (2014). In this corresponds to the visual system’s estimate of the version of events, the filter can be considered a segment reflectance of the cube’s material. So, in Figure 1a and of the standard color constancy hypothesis: The filter c, the top test squares appear darker because observers serves as part of the discounting method (perhaps in infer them to represent a less reflective surface in bright conjunction with other information about specularity) light, whereas the bottom squares appear brighter so that the observer can make inferences about the because observers infer a highly reflective surface in reflectance of the material. The alternative hypothesis is shadow (Lotto, 2010). Figure 1e can be interpreted as a that the high spatial frequency content is invariant with particularly powerful extension of this approach illumination, and a low spatial frequency response because it seems to show that the test squares, like the encodes information that correlates with changes in dress, change appearance depending on the presumed global illumination. To achieve color constancy, illumination. That is, although the test squares are therefore, the visual system does not need to infer physically achromatic, the visual system treats the spectral reflectance of the material; rather, the visual square under blue illumination as having yellow system primarily needs to give greater weighting to the reflectance and the square under yellow illumination as higher bands of the spatial frequency spectrum. We having blue reflectance. emphasize that we are not suggesting that the low Here we explore an alternative explanation: All three spatial frequency content is completely discarded from variations of the Rubik’s cube illusion (Figure 1a,c, all future processing. Rather, observers seem to balance and e) can be accounted for by removing low spatial responses to the low and high spatial frequency color frequency content from the image. The particular information depending on the situation and, perhaps, technique for removing low spatial frequency does not the individual. In this way, we attempt to recast the seem to matter (Supplementary Movie 1 demonstrates discussion of the dress from perceptual interpretations three different techniques); we use Adobe Photoshop’s of reflectance and illumination to the visual system’s high-pass filter because it is widely accessible and weighting of color information that arises in the simple for others to use to replicate our findings. The environment at multiple spatial (and temporal) scales. filter has a single parameter that has the effect of changing the amount of low spatial frequency content removed from the image. For example, a filter radius of 1 removes most spatial frequency information so that Demonstration 1: A high-pass filter all that remains in the image are the thinnest edges; can account for the Rubik’s cube edge extraction of this form is what is usually thought of when high-pass filters are mentioned. Larger filter illusion values shift the filter cutoff so that more low spatial frequency information remains in the image; in effect, The Rubik’s cube illusion (Lotto & Purves, 2002), an the filter removes the blur but leaves the pictorial sense iconic image for illustrating the effect of illumination of the image roughly unchanged. on color appearance, has often been recruited to The results of the application of the high-pass filter explain the color constancy hypothesis with regard to to Figures 1a, c, and e are shown in Figures 1b, d, and the dress (Lafer-Sousa, 2015; Macknik, 2015). The f, in which the filter size is set to 60 pixels (1.5875 cm), illusion (shown in Figure 1a) depicts a cube with the width of each individual square. Although not multiple colored squares on its surface; the illusion standard, we use pixels as opposed to visual angle consists of two squares that appear radically different because we have previously demonstrated that the from each other (a brown square on the top of the cube optimal value of the high-pass filter is related to the and an orangeish square on the side), even though they relative size of the test object, not to the test object’s have the same pixel value. In an achromatic version of visual angle on the retina (Dixon, Shapiro, & Lu, 2014). the image (Figure 1c), the two squares appear to have The operations we describe should therefore work on different brightness levels. The variation of the Rubik’s any device or any rescaling of the image, as long as the cube demonstration in Figure 1e appeared in a filter is adjusted to the size of the objects in the scene. In

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 4

Figure 1. (a) The original cube image (copyright Beau Lotto). Two identical brown squares are placed in direct illumination (black bounding box) or shadow (white bounding box); bounding boxes are present only to identify the measured squares. The square in shadow appears lighter than the square in illumination, even though the two squares have identical pixel values; actual colors shown in two squares to the right of the image. (b) The cube image after filtering at size 60 pixels (the measured diagonal of the individual squares on the surface of the cube). The square in shadow is now physically different from the square in illumination; actual colors shown in two squares to the right of the image. (c) Gray-scale version of the cube. Both squares are identical, mid-luminance gray, and again, the square in shadow appears brighter. (d) The gray-scale cube image after filtering at size 60 pixels. The square in shadow is now physically brighter; actual colors to the right of the image. (e) The cube in the white bounding box and the cube in the black bounding box are both mid-luminance gray. The one under the blue overlay appears yellow, and the one under the yellow overlay appears blue; actual colors below the image. (f) Blue and yellow overlay images after filtering at size 60 pixels. The square in the white bounding box appears yellow postfiltering, and the square in the black bounding box appears blue postfiltering; actual colors below the image.

the filtered images, the pixel values of the test squares consider the high-pass filter as a process that subtracts are no longer equal to each other but instead more the global information from the original image (see closely mimic the perceived values in the original Supplementary Movie 2). The global information can image. What is particularly striking, however, is that be thought of as the result of a low-pass filter or, when the high-pass filter is applied to Figure 1e, the equivalently, a very large blur filter. So, Figure 1f can simulated illumination completely disappears: The be approximated by blurring the image in Figure 1e, high-pass filter removes the illuminant, and the gray subtracting the blurred image from the original image squares are now physically yellow and blue. in Figure 1e, and then adding a constant value so that One way to understand why the filter successfully the image values are all positive. The blur does not accounts for the Rubik’s cube phenomenon is to affect the overlay color in Figure 1e; hence, the overlay

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 5

Figure 2. Dresses under varying simulated illumination based on Wired magazine illustration. (a) Original dress photo in center, flanked by image under a yellow illuminant (left) and a blue illuminant (right). (b–h) Image in panel (a) filtered with high-pass filter sizes ranging from 256 pixels (b) to one pixel (edge-extracted image) (h). Photograph of the dress used with permission. Copyright Cecilia Bleasdale.

colors can be found both in the blurred image of Figure unconscious inference or Bayesian estimation in order 1e and in the original image and will disappear when to separate the image into material and illumination the images are subtracted from each other. The simple properties (see Brainard et al., 2006). filtering operation, therefore, approximates an elimi- nation of the illuminant and therefore allows the observer to estimate the reflectance of the material. Another way to understand the success of the filter is Demonstration 2: Application of a to consider that—at the appropriate spatial scale—the high-pass filter to the dress image tests patches are actually physically different from each other. A photometer placed in front of a high-pass filtered version of the Rubik’s cube illusion in Figures In this section, we demonstrate how a spatial filter 1a and c would record that the top square has a like that presented above can be extended to improve physically lower value than the bottom square. the understanding of the dress phenomenon. To do Therefore, if we desire a physical value of light that this, we will use stimuli modeled after the Wired corresponds to perception, our measurements should magazine image (Rogers, 2015) that appeared in the consider the energy at different spatial bandwidths as days following the Internet phenomenon, which con- well as different spectral wavelengths. We should not sists of the original image and two images using assume that the test patches are physically identical to different pixels for a white-balance point in order to each other just because they have the same pixel values. create both a white-gold percept and a blue-black In this interpretation, then, the patches are different percept. The image in Figure 2a re-creates the Wired from each other because the visual system simply photo by placing a semitransparent yellow (255, 255, 0 extracts the information in the image at the appropriate values) and a semitransparent blue (0, 0, 80) overlay on spatial scale; the visual system would not need to use top of the original dress image. The original dress

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 6

Figure 3. Colored squares from the dress image in all overlay and filter conditions: (a) yellow overlay, (b) original, and (c) blue overlay. Each row shows the results for a different size high-pass filter; the numbers indicate the radius size. The colors for the unfiltered condition are variable across overlay conditions and for filters of size 256 and 128, but relative color constancy occurs in the range of 64–8 radius filters. The squares for the filter size of 1 demonstrate that when the filter is very small, the filter works only as an edge detector, and the colors within the image are gray scale.

photo is in the center, the image on the left simulates a respectively), the shading from the overlay seems to very strong yellowish illuminant, and the image on the disappear, resulting in images that look similar to each right simulates a very strong bluish illuminant. If other. Figure 2h shows a filter with a radius of one observers had perfect color constancy based on an pixel; only in this limiting condition does the high-pass ability to discount the illuminant, then the dress should filter behave as an edge detector that completely appear the same in all three panels in Figure 2a. desaturates the image. However, the simulated illuminants serve to disambig- Figure 2 therefore continues the suggestion that uate the color for many observers: Under a yellow high-pass filters act to remove local averages from illuminant, most observers see the dress as white-gold, images and can be used to simulate the discounting of whereas under a blue illuminant, most observers see the the illuminant in many images. To illustrate this idea dress as blue-black (evidence for this is shown in the more directly, Figure 3 shows color patches taken from experiment below). The Wired magazine image has the dresses seen in Figure 2. In each column, the left therefore been used to illustrate the idea that observers square corresponds to the color of a pixel taken from a perceive the original dress differently depending on white (or blue) stripe of the dress, and the right square how they interpret the illumination in the original dress corresponds to the color of a pixel taken from the gold image. or black stripe of the dress. As with Figure 2, the left The effect of filtering the original image of the dress column shows the pixel values from the dress with the through a high-pass filter is shown in Figure 2b through blue overlay, the middle column shows the pixel values h; a dynamic demonstration can be seen in Supple- from the original dress image, and the right column mentary Movie 2. The dress within the image is shows the pixel values from the dress with the yellow approximately 170 pixels wide, and we used a range of overlay. As described by Lafer-Sousa, et al. (2015), the filter sizes from 256 pixels in radius down to one pixel. pixels from the original dress are measured as blue/ With large high-pass filters, as in Figure 2b and c brown, which can be seen in the color patches for the (radius equals 256 and 128 pixels), the resulting images original image in the unfiltered condition (Figure 3b, look fairly similar to the original image in Figure 2a. first row). With mid-size high-pass filters, as in Figure 2d through As can be seen, the diameter of the filter affects the g (radius equal to 64, 32, 16, and eight pixels, size of the local average and therefore affects the

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 7

amount of global information removed. So, in Figure 3, 256- and 128-pixel diameter filters remove the majority of overlay information; however, for all three overlays, the color of the patches does not match the color of the unfiltered dress image. The 64-, 32-, 16-, and eight-pixel diameter filters produce almost identical colors. This means that when viewed through filters of these sizes, the simulated illumination would not affect the color of the original patch. Hence, we can say that the patches would remain relatively color constant for these filter sizes and these illuminants (this point will be a focus of the following section). A common (and incorrect) assumption about high- pass filters is that they act only to desaturate the colors of images. Perhaps this assumption is so common because it holds for the extreme case (i.e., radius ¼1; Figure 2h), where high-pass filters act as edge detectors. For intermediate filter sizes, however, high-pass filters can have a variety of effects on color direction and sometimes even produce color saturation instead of desaturation. For example, in the Rubik’s cube illusion, the two achromatic test patches in Figure 1e become more colorful after high-pass filtering (Figure 1f). The reason high-pass filters sometimes produce saturation and sometimes produce desaturation of test patches is that high-pass filters subtract local averages. If the test patch is less saturated than the surround, then the tendency is to saturate the test patch, and vice versa. The effect of filter size on the saturation of the test patches in Figure 3 can be seen in Figure 4, which plots the CIE chromaticity of the patches in Figure 3. Panel a corresponds to the yellow overlay, panel b to the original illuminant, and panel c to the blue overlay. The lines connect the chromaticities of each successive filter size; hence, a straight line indicates that increasing filter size leads to desaturation, and a bent line indicates that the filter had a more complex effect. Curiously, decreasing the filter diameter tends to desaturate the blue overlay condition but not the yellow overlay condition. We do not think that this change indicates anything unique about the colors of the overlays but rather is pertinent to the specifics of the dress image. That is, it should be possible to create an image in which the yellow overlay desaturates with filter size Figure 4. CIE color space values for colored squares in Figure 3 whereas the blue does not. for all overlay and filter conditions: (a) yellow overlay, (b) original image, and (c) blue overlay. The values for the stripe perceived as white or blue are marked as white diamonds with blue borders, and the values for the stripe perceived as gold or Demonstration 3: Application of a black are marked as yellow diamonds with black borders. The high-pass filter to fabric under CIE values demonstrate that full desaturation occurs only for varying illuminants the filter size of 1; all other filter sizes do not overlay the achromatic (rgb ¼ 128) value marked on the color space.

The analysis described in Demonstration 2 is, in many respects, trivial: The simulated layers of illumination add low spatial frequency content to the image;

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 8

Figure 5. Purple checked shirt under varying illumination conditions. The images shown in the left column are the original unfiltered photographs of the shirt under three illumination conditions: (a) shirt under a white illuminant, (b) shirt under a yellow illuminant, and (c) shirt under a blue illuminant. The illuminants cause the fabric to appear dissimilar across conditions. The images shown in the right column have been high-pass filtered with a filter radius of 300 pixels, matching the width of the fabric in the image; illuminant information has largely been discarded, and the fabric now appears similar across conditions.

the high-pass filter operation removes low spatial condition—the white illuminant is seen in a, the yellow frequency content from the image. So, even though we illuminant in b, and the blue illuminant in c; the left have added simulated blue and yellow illumination to column shows the unfiltered images of the shirt and the image, it is fairly obvious that we can remove the background, whereas the right column shows the effects of illumination with a simple filter. Nonetheless, images after filtering with Adobe Photoshop’s high- this relatively simple operation may be able to explain pass filter with a radius of 300 pixels, approximately the the Rubik’s cube illusion (Demonstration 1) and many width of the shirt in the original photographs. other similar phenomena without requiring knowledge In the original images (Figure 5, left column), the of intrinsic object properties, as would be suggested by colors of the shirt and background look quite different proposals that posit cognitive or Bayesian estimation of from each other across illumination conditions; in the illumination. filtered images (Figure 5, right column), the colors To show that the operation performed in Demon- appear similar. We stress that the color content in the stration 2 is not just a trick performed by digital filtered versions has not been lost, as would be expected addition and subtraction, we demonstrate that the when an extreme high-pass filter is applied (like the same high-pass filter operation can be used to image in Figure 2h); instead, all three shirts have colors neutralize the effects of real changes in illumination. similar to that shown under the neutral illumination Figure 5 shows a shirt with purple and white checks, (Figure 5a, left column). The demonstration, therefore, photographed under three illuminant conditions shows that a simple high-pass filter can go a long way (General Electric A-19 white, yellow, and blue 25-W toward achieving color constancy, even when the low- light bulbs). Each row depicts a different lighting pass information is added with natural illumination.

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 9

representation of an object will not respond much to changes in illumination. (b) The visual system repre- sents color at two different spatial scales. The image output of a high-pass filter is relatively immune to changes in illumination; on the other hand, the converse filter operation (i.e., a low-pass filter) encodes only information about changes in illumination. By dividing the world into different spatial scales, therefore, the visual system can create one representation that is impervious to illumination and another representation that encodes only illumination. Such representations would be analogous to current descriptions of neural responses based on material reflectance and illumination. We will further discuss these hypotheses in the General discussion, but first we will examine the effects of filtering on the appearance of the dress.

Experiment 1: The effects of filtering on the perceived color of the dress

The demonstrations show two possibilities: first, that the removal of low spatial frequency content can neutralize the effects of the illuminant, and second, that high-spatial frequency content is invariant to changes in illumination. We therefore hypothesized that individual variation in the perception of the dress is related to the observer’s response to spatial properties of the image. We measured whether observers change their report of dress color when low spatial frequency content is removed from the image. Online observers viewed each dress from Figure 2a through g individ- ually for all possible illumination and filtering conditions (e.g., the dress under the blue illuminant at filter Figure 6. Labeling of dress under different illuminants. size 8, as in Figure 2g, rightmost dress image) and Percentage of observers labeling the dress as white-gold for (a) selected whether each dress appeared most like white- the original illuminant, (b) the blue illuminant, and (c) the gold or blue-black. yellow illuminant. Method So, although the use of a high-pass filter in Demon- strations 1 and 2 may seem trivial, because of the Participants addition and removal of digital illumination informa- A total of 203 observers completed the survey online tion, it can also be used to successfully remove real via Amazon’s Mechanical Turk and were compensated. scene illumination in natural images. We selected the sample size of 200 after running a pilot The demonstration suggests two intriguing hypoth- survey to determine response rates and used it for all eses: (a) Neural representations of objects have built following experiments. The survey closed after 200 into them some degree of color constancy. Any visual submissions. We had an N of 203 because Mechanical representation that does not include low spatial Turk allowed three observers who had started prior to frequency content will be relatively invariant to global closing to complete the survey. Thirty-two observers changes in illumination. Because objects are spatially who failed to respond correctly to catch trials were bounded, the neural representation of an object itself dropped from analysis. Results were analyzed from the does not include low spatial frequency content larger remaining 171 participants. All observers had normal than the spatial scale of the object. Hence, any neural or corrected-to-normal vision and provided informed

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 10

Figure 7. Dress similarity rankings. (a) Rankings of dress similarity with original illuminant and blue illuminant. (b) Rankings of dress similarity with original illuminant and yellow illuminant. (c–h) Stimuli used in Experiment 2: paired dresses with original on left and dress under blue illuminant on right, for all filter sizes. (j–p) Stimuli used in Experiment 3: paired dresses with original on left and dress under yellow illuminant on right, for all filter sizes. Photograph of the dress used with permission. Copyright Cecilia Bleasdale.

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 11

consent as approved by the American University white-gold, the yellow illuminant condition shows a Institutional Review Board. large decrease from the unfiltered condition to the size 256 filter. This drop makes sense, however, when compared with the color values seen in Figure 3,in Stimuli and procedure column a. Adding a yellow overlay causes the values of The images used were 21 individual photos of the the measured squares to appear white and gold, dress; each filter size (unfiltered, 256, 128, 64, 32, 16, whereas filtering to 256 brings the values into a blue eight) for each illuminant (original, yellow, and blue) and brown range. It is not until entering the midrange was represented. Each dress image was 450 3 300 pixels of filter sizes, 64 and below, that the filter appears to and was presented in isolation on the screen, so create a color constant range of values in the stimulus. observers could view only one image at a time. For each image, observers were asked to respond to the prompt, ‘‘The colors in this dress are most like’’ by selecting ‘‘white and gold’’ or ‘‘blue and black.’’ Experiments 2 and 3: The effects of Observers had no time limit for how long they spent on filtering and dress similarity each image. The 21 images were randomized for each observer so that the order was nonsequential, and each To measure color constancy between illumination image appeared one time, except for the original, conditions, we collected observer rankings of similarity unfiltered image, which appeared three times. These between sets of two dress images; the dress under the three trials were compared as catch trials; observers original illuminant paired with either the dress under who varied in response across conditions were excluded the blue illuminant (Experiment 2) or paired with the from analyses. dress under the yellow illuminant (Experiment 3); the pairings were done across all filter sizes. We hypoth- Results esized that regardless of whether the paired dresses were perceived as white-gold or blue-black by observ- For the unfiltered images, the observer reports of the ers, that rankings of similarity would increase as filter color of the dress were as follows: for the original image size decreases and more low spatial frequency content is (i.e., no simulated illumination), 70% reported white- removed from the images. We increased the opacity of gold and 30% blue-black (similar to informal online the overlays for the experimental conditions to simulate polls); for the simulated yellow illumination, 85% of more extreme blue and yellow illuminants. In the participants reported white-gold and 15% blue-black; original images, the opacity was set to 20%, which we for the simulated blue illumination, 16% white-gold increased to 40% for the experimental stimuli, as can and 84% blue-black. The results confirm the observa- be seen in Figure 7 below. tion in Demonstration 1 that adding a yellow illuminant creates a shift toward reports of white-gold and adding a blue illuminant creates a shift toward Method reports of blue-black. Removing low spatial frequency content increases Participants the percentage of observers who report seeing the dress In Experiment 2, 207 observers completed the survey as white-gold in both the original condition (an 11% online via Amazon’s Mechanical Turk and were increase) and blue illuminant condition (a 60% compensated. Thirty observers who failed to respond increase). Observers were classified into either a white- correctly to catch trials were dropped from analysis. gold group or blue-black group based on the colors Results were analyzed from the remaining 177 partic- assigned to the original, unfiltered dress image. Figure ipants. In Experiment 3, 205 observers completed the 6a shows the percentage of observers in each group survey online via Amazon’s Mechanical Turk and were who labeled the dress under the original illuminant as compensated. Eleven observers who failed to respond white-gold, and Figure 6b and c show the same for the correctly to catch trials were dropped from analysis. blue illuminant and yellow illuminant, respectively. For Results were analyzed from the remaining 194 partic- the white-gold group, the dress under both the original ipants. All observers had normal or corrected-to- illuminant and the yellow illuminant is labeled as white- normal vision and provided informed consent as gold by more than 98% of observers for each filter size, approved by the American University Institutional and the dress under blue illumination is labeled as Review Board. white-gold by 91% or more of observers after filtering at sizes 64 and smaller. Although the graphs for the original and illuminant Stimuli and procedure conditions show a generally steady increase of observ- The images used in each experiment were paired ers from the blue-black group labeling the dress as photos of the dress for each filter size (unfiltered, 256,

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 12

128, 64, 32, 16, eight). Experiment 2 presented the dress. If the high-pass filter acts to remove the original dress image and the image under the blue illuminant (as part of the color-constancy hypothesis), illuminant; Experiment 3 presented the original dress then high-pass filtering should allow observers to see image and the image under the yellow illuminant. In the dress more veridically (i.e., blue-black). We should each pairing, the dresses were marked as 1 and 2, and therefore expect that when low spatial frequency the combined image of the two dresses was 650 3 500 content is removed from the images, more observers pixels; the stimuli can be seen in Figure 7. would report the dress as blue-black. However, Each block of paired dresses and questions was Experiment 1 demonstrates that as more low spatial presented in isolation on the screen, so observers could frequency content is removed from the image, more view only one image set at a time. For each block, observers see the dress as white-gold. observers were asked to respond to the prompt, ‘‘The Experiments 2 and 3 set out to give empirical colors in dress 1 are most like’’ by selecting ‘‘white and documentation to the question asked in Demonstration gold’’ or ‘‘blue and black’’; observers answered the 2; that is, can high-pass filtering act to neutralize the same question for Dress 2. Observers then rated the effect of the simulated illuminant placed on the dress? similarity of the two dresses (1 ¼ very different,2¼ As would be expected from the demonstration, as more different,3¼ similar,4¼ very similar/identical). In low spatial frequency content is removed from the addition to the seven experimental conditions, there image, the dresses under simulated blue and yellow were two catch trial blocks: one with the unfiltered illumination appear more similar to the original dress dress under the original illuminant, repeated twice, and image. The results lead to the suggestion that individual one with the unfiltered dress under the blue illuminant differences in the perception of the dress could be (for Experiment 2) or yellow illuminant (for Experi- related to variation in the processing of low spatial ment 3), repeated twice; observers who did not select frequency content available within the image. matching colors for identical dresses were dropped. The nine blocks were randomized for each observer so that the order was nonsequential, and each block appeared once; observers had no time limit for how long they Experiment 4: Contrast sensitivity spent on each image pairing. and the dress

Results One obvious question is whether differences in the Regardless of group (white-gold or blue-black) or dress correlate with observers’ sensitivity in the low illumination condition (yellow or blue), removing a spatial frequency portion of the spectrum. The field of higher proportion of low spatial frequencies increased optometry has spent considerable effort examining how rankings of dress similarity. The results can be seen in individuals differ in the processing of high spatial Figure 7a for the blue illuminant and 7b for the yellow frequency content, because deficits in the high-fre- illuminant. For the blue illuminant, observers in the quency region of the spatial spectrum can prevent white-gold group averaged 1.92 for the original pairing, people from reading, driving, and a host of other social whereas those in the blue-black group averaged 2.81. activities. Individual differences in gain control at the As filter size decreases and the amount of low spatial other end of the spatial scale must exist but are less frequency information removed increases, both groups documented because they do not cause the same level increased in similarity rankings: white-gold averaged of social deficit as high spatial frequency deficits (see 3.77, and blue-black averaged 3.7. For the yellow McCourt, Leone, & Blakeslee, 2015, for a recent illuminant, observers in the white-gold group averaged investigation into the substantial individual differences 2.66 for the original pairing, whereas those in the blue- in low spatial frequency sensitivity). We therefore black group averaged 2.15. As filter size decreases and looked for differences between observers’ perception of the amount of low spatial frequency information the color of the original dress image and measurements removed increases, both groups increased in similarity of contrast sensitivity. rankings: white-gold averaged 3.51, and blue-black averaged 3.26. Method Summary and discussion of Participants Fifty-three observers (n ¼ 25 white-gold, n ¼ 28 blue- Experiments 1–3 black) completed an experimental measure of contrast sensitivity function (CSF) in the lab. Fifty-one were We have measured the effects of removing low American University undergraduates who participated spatial frequency content on the appearance of the in a larger study that included the CSF measurement,

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 13

Figure 8. Contrast sensitivity functions of white-gold and blue-black observers.

for course credit, and two were American University reversals. Participants ran the experiment twice during graduate students who are members of the lab. All the session; the results for each participant are the observers had normal or corrected-to-normal vision averages from the two runs. and provided informed consent as approved by the American University Institutional Review Board. Results The data are plotted in Figure 8 and reported in Stimuli and procedure Table 1. The results and shape of the functions are The stimuli were presented on a Sony Trimaster EL similar to other CSF functions reported in the literature OLED monitor. The luminance levels of the monitor (see Hou et al., 2016). Observers in the white-gold were measured using a Spectrascan 650 and gamma group have a higher average contrast sensitivity than corrected using the driver software packaged with the observers in the blue-black group at low spatial computer graphics card (Catalyst Control Center on frequencies; a two-tailed, independent-samples t test ATI Radeon HD 5970). Observers were seated 24 in. across all conditions showed significant differences for from the monitor. The stimuli were 458 tilted spatial 0.25 and 1 cycles per degree, as seen in Table 1. frequency gratings for six conditions of cycles per Although there was no significant difference in the degree (0.25, 0.5, 1, 2, 4, 8, 16), generated and presented location of peak sensitivity (average equal to 0.88 using PsychoPy (Peirce, 2008). Gratings appeared on a [white-gold] and 1.1 [blue-black], p ¼ 0.59), there was a gray background for 50 ms, and observers used the left significant difference for maximum contrast sensitivity and right arrow keys to indicate which direction the between groups, with white-gold observers having an grating was tilted; between responses, a fixation dot average of 3.95 to the blue-black observers’ average of appeared. The gratings were presented in three-down, 3.04 (p ¼ 0.04). one-up interleaved staircases with eight reversals per We also fit the results from individual observer data spatial frequency (step sizes of 4, 2, 2, 2, 1, 1, 1, 1 db), with a function described by Chung and Legge (2016). beginning at 50% contrast. For each condition, the The function had four parameters (the peak sensitivity threshold was expressed as the average of the last four frequency, the peak sensitivity, the width of the CSF

Cycles per degree 0.25 0.5 1 2 4 8 16 White-gold 2.59 (1.58) 3.51 (1.78) 3.23 (1.63) 2.00 (0.26) 1.76 (0.23) 1.36 (0.31) 0.47 (0.30) Blue-black 1.92 (0.83) 2.82 (1.61) 2.45 (1.19) 1.86 (0.46) 1.68 (0.20) 1.26 (0.26) 0.37 (0.26) T values 2.28 1.47 2.00 1.32 1.35 1.24 1.38 p values 0.03 0.15 0.05 0.19 0.18 0.22 0.17

Table 1. Contrast sensitivity (1/threshold contrast), standard deviation, T values, and p values for white-gold and blue-black observers.

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 14

function on the right, and the width on the left). Of spatial filters (Blakeslee et al., 2005; Perna & Morrone, these, only the peak sensitivity parameter showed a 2007; Robinson et al., 2007). We implemented the significant difference on a two-tailed, independent- removal of low spatial frequency content with the groups t test (mean blue-black ¼ 2.82; mean white-gold simple filter model of Shapiro and Lu (2011) and Dixon ¼ 3.51), t(49) ¼2.07; p ¼ 0.043. et al. (2014). We show that a simple spatial filter can account for (a) the Rubik’s cube illusion, an iconic demonstration frequently used to discuss how the visual system discounts illumination; (b) the differences Discussion produced by adding simulated illumination to the dress image; and (c) the effects produced by changing real Prior to the start of these experiments, we noted that scene illumination on a colored object. sensitivity measurements should not necessarily be As noted earlier, the basic principle underlying many correlated with individual perceptual differences of the of the demonstrations in this article is trivial: First, low dress. One reason for this is that the task of judging spatial frequency content is added to an image, and color is generally a superthreshold task and, as shown then the low spatial frequency content is removed with for contrasts greater than 5%–10%, many visual a high-pass filter. Trivial as this may seem, a simple psychophysical tasks are contrast independent (see filter can be useful because changes in global illumi- Sperling & Lu, 1998). A second reason is that we think nation primarily have their effects in the low spatial that the relevant spatial scales for this filtering process frequency portion of the visual image. The visual may be in some way related to the encoding of objects system contains many processes that act like high-pass or visual grouping. For example, if Observer A filters (e.g., spatial response of double-opponent cells, organizes the scene in a manner that encodes a small changes of ganglion cell response with eye movements, framework, and Observer B organizes the scene in a removal of motion blur, representation of objects). If manner that encodes a large framework, then Observer observers differ in one or more of these spatial A will exclude more low spatial frequency content than processes, it is likely that they will also differ in how Observer B. Two observers may have identical contrast they encode the effects of illumination in the image and, sensitivity, but Observer A’s representation of the subsequently, may therefore differ in their reports of framework would be different from Observer B’s in the the color of the dress. And although this may seem like low spatial frequency portion of the spatial spectrum. a reasonable first place to begin inquiry into individual Thus, there are many ways in which observers can have perceptual differences of the color of the dress, we call differences in spatial processing but not have any attention to the fact that the four most recent reviews of differences in threshold contrast sensitivity. Although color constancy (Foster, 2011; Shevell & Kingdom, the results show significant differences for CSF 2008; Smithson, 2005; Xiao, 2016) do not mention measurements for white-gold and blue-black observers, spatial filters or spatial scale representations. Hence, it the variability of individual scores within a group and should not be surprising that spatial filtering ap- overlapping scores between groups make it seem proaches were absent from other discussions of the unlikely that this is what is fully driving the group dress. differences.

Representation of the color at different spatial General discussion scales

Most explanations of the dress assume that a central The demonstrations in Figures 1, 2, and 3 illustrate task of color perception is to infer the reﬂectance of the idea that global changes in illumination can be surface material by way of discounting the illumination discounted simply by removing the low spatial fre- falling on the object. In this article, we pursue a quency content. To be clear, we are not suggesting that different approach based on the idea that many aspects the visual system completely discards low spatial of color constancy can be achieved by removing low frequency information, nor do we wish to imply that spatial frequency content from the image. The idea has high spatial frequency representations always are similarities to other models of color constancy invariant to illumination. Rather, the demonstrations (Buchsbaum, 1980; Judd, 1940; Land, 1986), to show the utility of considering color information at a physiological explorations of color vision that suggest range of spatial frequency scales: Color in the high that cortical double-opponent cells are spatially band- spatial frequency range will encode different functional pass (Conway, 2002; Shapley & Hawken, 2011), and to information than color in the low spatial frequency models of brightness/lightness perception based on range.

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 15

Figure 9. Filtering the dress on a gradient background. (a) Original image repeated across yellow-blue gradient background. (b) High-pass filtered version of (a) with filter diameter of 170 pixels. (c) Gaussian blurred low-pass version of (a) with blur diameter of 170 pixels. (d) Combined high-pass and low-pass images with alpha of 0.5. Photograph of the dress used with permission. Copyright Cecilia Bleasdale.

We present one more demonstration to illustrate frequency visual system would not encode the dress at these differences. In Figure 9a, we show multiple copies all. of the dress in front of a blue-yellow gradient Our visual system must somehow combine these background. The dress itself does not show induction, different types of spatial representations into a coherent so all copies of the dress appear fairly similar to each image. To simulate the synthesis, we create a new image other (whether white-gold, blue-black, or another in Figure 9d by overlaying 9c with a transparency of variation dependent on the observer). Figure 9b shows alpha equal to 0.5 over 9b. In the resulting image, the a high-pass filtered version of the image, and Figure 9c dresses (not surprisingly) again appear the same, as in shows a low spatial frequency version of the image. To 9a. So, although the dress images appear dissimilar at a arrive at these images, we calculated the width of the particular scale after spatial filtering, they appear dresses (about 170 pixels in our image) and then filtered identical again once low-pass information is added back into the image. Images contain information at the original image with a high-pass filter with a radius multiple spatial scales, and each scale may contribute a of 170 pixels (Figure 9b) and with a Gaussian blur of different interpretation of the world; any particular 170 pixels (Figure 9c). visual process could weight the information according Both Figures 9b and 9c can be thought of as back- to its needs. Variations in weighting of any level of pocket approximations of the information encoded by frequency would necessarily result in variations in color vision systems with two spatial resolutions. For perception (e.g., observers who perceive very little the high-pass image (Figure 9b), the dresses are illumination in the original dress image versus those physically different from each other. For the low-pass who perceive high illumination). image (Figure 9c), the dresses, rather remarkably, The idea that there are multiple spatial representa- disappear; the filtered image shows only the gradient tions is a hallmark of 20th-century psychophysics and background. An exclusively high spatial frequency physiology (Graham, 1989), and it seems likely that visual system would therefore encode four dresses of spatial filtering should play a role in color vision. In differing hue, whereas an exclusively low spatial general, we are agnostic to the particular quantitative

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 16

and physiological embodiment of spatial frequency suggest that either the filter has an effect that is attenuation and combination. However, Dixon et al. dissimilar from removing the illuminant or that (2014) showed that the cutoff frequency for the high- attenuating the illuminant with a physical filter has the pass filter adjusts to information in the scene and seems reverse effect of what happens when the brain discounts to be tuned to the size of objects. By tying the size of the illuminant. the spatial filter to objects rather than to a low-level Or, alternatively, the goal of the visual system is not response, we can posit adjustments to spatial channel to recover the ‘‘true’’ color; rather, color is encoded by weights based on other upstream factors that may a system that preferentially responds to the high spatial affect color or brightness, such as attention (Tse, 2005), frequency content extracted from the image, and there grouping (Gilchrist & Radonjic´, 2010; Xian & Shevell, is no process that infers the material property. We 2004), and motion (Werner, 2007). There has been sometimes think the visual system infers the material some evidence for objects being invariant to changes in property because the high-pass information (possibly distance for some tasks, thereby suggesting visual carried by the spatially band-pass double-opponent representation in terms of object spatial frequency and cells in the cortex) remains constant and therefore has not retinal spatial frequency (Peterzell, 1997; Peterzell, the same outcome as a putative cognitive function that Harvey, & Hardyck, 1989). estimates the material. Although such an approach seems to open new ways to consider individual differences, the predictions that it makes depend on the Individual differences and the dress specification of the linking assumption that connects responses to high spatial frequency content to color Many of the arguments about the dress have relied perception. For instance, observers could differ in their on demonstrations to illustrate possible ways of perception of the dress because they differ in channel interpreting the image of the dress and have primarily tuning (i.e., they differ in the amount of low spatial concluded that the visual system makes an inference frequency attenuated; observers, in effect, have differ- about some aspect of illumination, such as direction or ent gain controls for the low spatial frequency channel), color. We have written this article to show that an observers could construct the perception of the dress explanation for these demonstrations can also be cast through a comparison of the low and high spatial in terms of information in the image that the visual frequency responses, or observers could differ in how system can extract through spatial filtering. In a filter- their perceptual system encodes contrast within the based approach, individual differences in the percep- high spatial frequency system. At this time, we cannot tion of the dress would arise if observers differ in the differentiate between these specific types of linking amount of high spatial frequency content extracted assumptions. So, although our results are inconsistent from the image (or, conversely, the amount of low with the standard hypothesis (i.e., if the high-pass filter spatial frequency content attenuated) or if observers discounts the illuminant, then as low spatial frequency give different weights to processes that differently filter content is removed from the image, more people should high and low spatial frequency content. At the see the dress as blue-black), the data do not reject the suggestion of a reviewer and the editor, we measured alternative hypothesis. for correlations between observers’ contrast sensitivity functions (CSF) and the perception of the dress. We found that the white-gold observers had higher contrast Limitations sensitivities at lower spatial frequencies as well as a higher overall peak contrast sensitivity. The result is We have shown that a very simple model can consistent but not necessary because the differences in account for demonstrations related to the dress and spatial response between groups could arise at super- hopefully will give new insights into color constancy. threshold levels. To be clear, we are not suggesting that a simple high- We have shown that the dress tends to appear white- pass filter can explain all brightness illusions and all of gold when low spatial frequencies are filtered from the color constancy. First, it is unlikely that the filtering image. This result seems contrary to what would be process is unitary; spatial filters occur at many different predicted if the visual system were attempting to stages of visual processing and are likely to vary for ‘‘recover’’ an illuminant-independent description of the different functional pathways. The notion that the object’s reflection. That is, because the high-pass filter dress occurs along a yellow-blue line in color space in effect removes the illuminant, the prediction should (Winkler et al., 2015) may represent differences in be that after the filter is applied, observers should shift filtering for the ancient tritan S-(L-M) system com- the perception toward blue-black. The white-gold pared with other color systems. We note here that our response therefore argues against a theory based on the filter model could also be considered a blunt one- estimation of the material reflectance. The results dimensional version of von Kries adaptation (Buchs-

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 17

baum, 1980), because such von Kries adaptation switch their color categorization of the dress. The result implicitly examines color contrast over local spatial is consistent with the reports that increasing image size regions (Khang & Zaidi, 2004). Given our results, one leads to a higher proportion of white-gold responses might expect that expanding von Kries adaptation and that blurring the image of the dress leads to a lower models to account for different spatial scales would be proportion of white-gold responses (Lafer-Sousa et al., productive, but the testing of any hypotheses in this 2015). regard would require experiments beyond the scope of The results support the hypothesis that, under some this article. conditions, high spatial frequency content remains Second, we restate the caveat given by Shapiro and invariant to changes in illuminant. Furthermore, if Lu (2011) that some brightness illusions cannot be visual subsystems weight spatial information according accounted for by a simple filter. Illusions such as the to the task (e.g., object perception will give a greater Craik-O’Brien-Cornsweet effect, Long-range Argyles weight to high spatial frequency content than to low (Flynn & Shapiro, 2014), and the Watercolor Effect spatial frequency content), then any subsystem that (Pinna, Brelstaff, & Spillmann, 2001) show changes gives less weight to processes that respond to low that are generated at a thin edge but extend over very spatial frequency content will automatically have some large distances. In addition, it is clear that processes degree of color constancy. Conversely, any subsystem that generate brightness (and contrast) occur at several that responds primarily to low spatial frequency stages of visual processing (Flynn & Shapiro, 2013; content will respond primarily to changes in the global Shevell, Holliday, & Whittle, 1992). Furthermore, the illuminant. Of course, this hypothesis may not be true addition of low spatial frequency transparent layers to for complex scenes with multiple illuminants or large contrast illusions has curious perceptual consequences amounts of interreflection. (Dixon & Shapiro, 2014); a simple high-pass filter To be clear, we are not suggesting that a single cannot fully account for the additional strength of these spatial filter is a complete explanation for color illusions. constancy and the dress. We are saying, however, that Third, the filter used for our demonstrations has one measurements of luminance and color that disregard parameter: the amount of low spatial frequency content spatial frequency scale are often incomplete and can removed from the entire image. A curious fact is that mislead as to what the visual system is capable of the size of the parameter seems to be tied to the size of encoding. There is a wealth of information in the the objects in the images (Dixon et al., 2014). This stimulus that many standard proposals of color suggests that filtering is not passive and may be part of constancy seem to ignore; in addition to high spatial higher-order functional processes. It could be that frequency content and low spatial frequency content, other factors often associated with perceptual inferences and top-down processing—scission, grouping, there are other types of information that the visual object formation, and so forth—could affect the filters system can extract from the environment (see Adelson or actually are the filters. As we have noted elsewhere, & Bergen, 1991). For instance, the visual system can object representations necessarily exclude low spatial separate changes in color and changes in color contrast frequency content. (Shapiro, 2008; Whittle, 2003); color contrast information remains constant with illumination changes, whereas color information remains constant when backgrounds change (Brown, 2003). General conclusion Attention to these aspects of the stimulus may give insight to many other problems in the color constancy A long-standing divide in the vision science literature literature that tend to be explained by cognitive concerns the role of perceptual inferences about the mechanisms or Bayesian inferences (see issues raised by distal stimulus versus the role of early filters in shaping Arend & Reeves, 1986). It is likely that individual visual response (see Kingdom, 1997, for a concise differences in extracting any of these sources of summary of the historical division). To date, the information are related to individual differences in predominant theories for the dress phenomenon have visual perception. Nonetheless, as pointed out by relied on a strong role for perceptual inferences (i.e., the Brainard and Hurlbert (2015), a complete understand- approaches assume that observers infer material ing of individual differences and the dress would reflectance by estimating and discounting the illumi- necessitate measurements ranging from preretinal nant). Here, we have tried to show how the dress can, in mechanisms through neural mechanisms and onto principle, be accounted for by variations in a simple cognitive mechanisms for each observer. Although an spatial filter that extracts relevant information from the overexposed photo of a striped dress may not seem to environment. We have also shown that changing the be the ideal stimulus for understanding the visual spatial content in the image leads some observers to system and color constancy, the image may prove a

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 18

useful starting point to test theories of individual vision: Understanding #TheDress. Current Biology, differences in perception at varying spatial scales. 25, R551–R554, doi:10.1016/j.cub.2015.05.020. Keywords: lightness/brightness perception, color Brainard, D. H., Longere, P., Delahunt, P. B., appearance/constancy, contrast, spatial ﬁltering Freeman, W. T., Kraft, J. M., & Xiao, B. (2006). Bayesian model of human color constancy. Journal of Vision, 6(11):10, 1267–1281, doi:10.1167/6.11.10. [PubMed] [Article] Acknowledgments Brown, R. O. (2003). Backgrounds and Illuminants: The Yin and Yang of colour constancy. In R. The authors thank Laysa Hedjar and Divya Nigam Mausfeld & D. Heyer (Eds.), Colour perception (pp. for coding the experiment and running the observers in 247–272). Oxford, UK: University Press. the contrast sensitivity study, and Dr. Sherri Geller for Buchsbaum, G. (1980). A spatial processor model for editorial help. object colour perception. Journal of the Franklin Institute, 310, 1–26, doi:10.1016/0016- Commercial relationships: none. 0032(80)90058-7. Corresponding author: Erica L. Dixon. Chung, S. T. L., & Legge, G. E. (2016). Comparing the Email: [email protected]. shape of contrast sensitivity functions for normal Address: Department of Psychology, American and low vision. Investigative Ophthalmogy and University, Washington, DC, USA. Vision Science, 57, 198–207. [PubMed] [Article] Conway, B. (2002). Neural mechanisms of color vision: Double-opponent cells in the visual cortex. Compre- References hensive physiology. Retrieved from http:// onlinelibrary.wiley.com/doi/10.1002/cphy. cp010310/full%5Cnhttp://books.google.com/ Adelson, E. H., & Bergen, J. (1991). The plenoptic books?hl¼en&lr¼&id¼pFodUlHfQmcC&oi¼ function and the elements of early vision. In M. fnd&pg¼PA1&dq¼Neuralþmechanismsþofþcolorþ Landy & J. Movshon (Eds.), Computational models visionþDouble-OpponentþCellsþinþtheþVisualþ of visual perception (pp. 3–20). Cambridge, MA: Cortex&ots¼IgvDhASSHd&sig¼ MIT Press. RvLKe4vUARQ8D5qHlJXuzXM Arend, L., & Reeves, A. (1986). Simultaneous color De Valois, R. L., Webster, M. A., De Valois, K. K., & constancy. Journal of the Optical Society of Lingelbach, B. (1986). Temporal properties of America. A, Optics and Image Science, 3, 1743– brightness and color induction. Vision Research, 26, 1751. Retrieved from http://www.ncbi.nlm.nih.gov/ 887–897, doi:10.1016/0042-6989(86)90147-1. pubmed/3772637 Dixon, E., & Shapiro, A. G. (2014). Paradoxical effect Barnard, K., Cardei, V., & Funt, B. (2002). A of spatially homogenous transparent fields on comparison of computational color constancy simultaneous contrast illusions. Journal of the algorithms—Part I: Methodology and experiments Optical Society of America A, 31, A307. Retrieved with synthesized data. IEEE Transactions on Image from http://www.opticsinfobase.org/abstract. Processing, 11, 972–984, doi:10.1109/TIP.2002. cfm?URI¼josaa-31-4-A307 802531. Dixon, E., Shapiro, A., & Lu, Z.-L. (2014). Scale- Blakeslee, B., & McCourt, M. E. (2012). When is invariance in brightness illusions implicates object- spatial filtering enough? Investigation of brightness level visual processing. Scientific Reports, 4, 3900, and lightness perception in stimuli containing a doi:10.1038/srep03900. visible illumination component. Vision Research, D’zmura, M., & Singer, B. (2001). Contrast gain 60, 40–50, doi:10.1016/j.visres.2012.03.006. control. In K. Gegenfurtner & L. Sharpe (Eds.), Blakeslee, B., Pasieka, W., & McCourt, M. E. (2005). Color vision: From genes to perception (pp. 369– Oriented multiscale spatial filtering and contrast 386). Cambridge, UK: Cambridge University Press. normalization: A parsimonious model of brightness Feitosa-Santana, C., D’Antona, A. D., & Shevell, S. K. induction in a continuum of stimuli including (2011). What kinds of contours bound the reach of White, Howe and simultaneous brightness contrast. filled-in color? Journal of Vision, 11(2):2, 1–11, doi: Vision Research, 45, 607–615, doi:10.1016/j.visres. 10.1167/11.2.2. [PubMed] [Article] 2004.09.027. Flynn, O. J., & Shapiro, A. G. (2013). The separation Brainard, D. H., & Hurlbert, A. C. (2015). Colour of monocular and binocular contrast. Vision

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 19

Research, 93, 19–28, doi:10.1016/j.visres.2013.10. Lotto, R. B. (2010). The science of optical illusions. 006. Retrieved July 20, 2015, from http://www.bbc.co. Flynn, O., & Shapiro, A. G. (2014). A note concerning uk/news/magazine-11553099 the relationship between the Adelson’s Argyle Lotto, R. B., & Purves, D. (2000). An empirical illusion and Cornsweet edges. Psihologija, 47, 353– explanation of color contrast. Proceedings of the 358. Retrieved from http://www.doiserbia.nb.rs/ National Academy of Sciences, USA, 97, 12834– Article.aspx?id¼0048-57051403353F#. 12839, doi:10.1073/pnas.210369597. VbZTIEPUkY0.mendeley Lotto, R. B., & Purves, D. (2002). The empirical basis Foster, D. H. (2011). Color constancy. Vision Research, of color perception. Consciousness and Cognition, 51, 674–700, doi:10.1016/j.visres.2010.09.006. 11, 609–629, doi:10.1016/S1053-8100(02)00014-4. Gegenfurtner, K. R., Bloj, M., & Toscani, M. (2015). Macknik, S. (2015). The current biology of the dress. The many colours of ‘‘the dress.’’ Current Biology, Scientific American Blogs, https://blogs. 25, R543–R544, doi:10.1016/j.cub.2015.04.043. scientificamerican.com/illusion-chasers/ Gilchrist, A. L., & Radonjic´, A. (2010). Functional the-current-biology-of-the-dress/. Retrieved July frameworks of illumination revealed by probe disk 20, 2015 technique, Journal of Vision, 10(5):6, 1–12, doi:10. McCourt, M. E., Leone, L. M., & Blakeslee, B. (2015). 1167/10.5.6. [PubMed] [Article] Brightness induction and suprathreshold vision: Graham, N. V. S. (1989). Visual pattern analyzers. Effects of age and visual field. Vision Research, 106, Oxford, UK: Oxford University Press. Retrieved 36–46, doi:10.1016/j.visres.2014.10.028. from http://dx.doi.org/10.1093/acprof:oso/ Peirce, J. W. (2008, January). Generating stimuli for 9780195051544.001.0001 neuroscience using PsychoPy. Frontiers in Neuro- Guys-please-help-me. (2015). Retrieved from http:// informatics, 2, 10, doi:10.3389/neuro.11.010.2008. swiked.tumblr.com/post/112073818575/ Perna, A., & Morrone, M. C. (2007). The lowest spatial guys-please-help-me-is-this-dress-white-and frequency channel determines brightness percep- Hou, F., Lesmes, L. A., Kim, W., Gu, H., Pitt, M. A., tion. Vision Research, 47, 1282–1291, doi:10.1016/j. Myung, J. I., & Lu, Z.-L. (2016). Evaluating the visres.2007.01.011. performance of the quick CSF method in detecting Peterzell, D. H. (1997). Hemisphericsymmetries in the contrast sensitivity function changes. Journal of identification of band-pass filtered letters Reply to Vision, 16(6):18, 1–19, doi:10.1167/16.6.18. Christman et al. (1997). Psychonomic Bulletin & [PubMed] [Article] Review, 4, 285–287, doi:10.3758/BF03209407. Judd, D. B. (1940). Hue, saturation, and lightness of Peterzell, D. H., Harvey, L. O., & Hardyck, C. D. surface colors with chromatic illumination. Journal (1989). Spatial frequencies and the cerebral hemi- of the Optical Society of America, 30(1), 2–32. spheres: Contrast sensitivity, visible persistence, Khang, B.-G., & Zaidi, Q. (2004). Illuminant color and letter classification. Perception & Psychophys- perception of spectrally filtered spotlights. Journal ics, 46, 443–455, doi:10.3758/BF03210859. of Vision, 4(9):2, 680–692, doi:10.1167/4.9.2. Pinna, B., Brelstaff, G., & Spillmann, L. (2001). Surface [PubMed] [Article] color from boundaries: A new ‘‘watercolor’’ illu- Kingdom, F. (1997). Simultaneous contrast: the sion. Vision Research, 41, 2669–2676, doi:10.1016/ legacies of Hering and Helmholtz. Perception, 26, S0042-6989(01)00105-5. 673–677. Robinson, A. E., Hammon, P. S., & de Sa, V. R. Lafer-Sousa, R. (2015). What is the deal with this (2007). Explaining brightness illusions using spatial goddamned dress? Retrieved January 1, 2015, from filtering and local response normalization. Vision http://web.mit.edu/bcs/nklab/what_color_is_the_ Research, 47, 1631–1644, doi:10.1016/j.visres.2007. dress.shtml 02.017. Lafer-Sousa, R., Hermann, K. L., & Conway, B. R. Rogers, A. (2015). The science of why no on agrees on (2015). Striking individual differences in color the color of this dress. Retrieved July 20, 2015, perception uncovered by ‘‘the dress’’ photograph. from http://www.wired.com/2015/02/science-one- Current Biology, 25, R545–R546. https://doi.org/ agrees-color-dress/ 10.1016/j.cub.2015.04.053 Shapiro, A. G. (2008). Separating color from color Land, E. H. (1986). Recent advances in Retinex theory. contrast, Journal of Vision, 8(1):8, 1–18, doi:10. Vision Research, 26, 7–21, doi:10.1016/0042- 1167/8.1.8. [PubMed] [Article] 6989(86)90067-2. Shapiro, A., & Lu, Z.-L. (2011). Relative brightness in

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017 Journal of Vision (2017) 17(3):7, 1–20 Dixon & Shapiro 20

natural images can be accounted for by removing Vision Research, 45, 1095–1098. Retrieved from blurry content. Psychological Science, 22, 1452– http://www.ncbi.nlm.nih.gov/pubmed/15707917 1459, doi:10.1177/0956797611417453. Werner, A. (2007). Color constancy improves, when an Shapley, R., & Hawken, M. J. (2011). Color in the object moves: High-level motion influences color cortex: Single- and double-opponent cells. Vision perception. Journal of Vision, 7(14):19, 1–14, doi: Research, 51, 701–717, doi:10.1016/j.visres.2011.02. 10.1167/7.14.19. [PubMed] [Article] 012. Werner, A. (2014). Spatial and temporal aspects of Shapley, R., & Reid, R. C. (1985). Contrast and chromatic adaptation and their functional signifi- assimilation in the perception of brightness. Pro- cance for colour constancy. Vision Research, 104, ceedings of the National Academy of Sciences, USA, 80–89, doi:10.1016/j.visres.2014.10.005. 82, 5983–5986. Retrieved from http://www. pubmedcentral.nih.gov/articlerender.fcgi?artid¼ Whittle, P. (2003). Contrast colours. In R. Mausfeld & 390678&tool¼pmcentrez&rendertype¼abstract D. Heyer (Eds.), Colour perception (pp. 115–138). New York: Oxford University Press. Shevell, S. K., Holliday, I., & Whittle, P. (1992). Two separate neural mechanisms of brightness induc- Winkler, A. D., Spillmann, L., Werner, J. S., & tion. Vision Research, 32, 2331–2340, doi:10.1016/ Webster, M. A. (2015). Asymmetries in blue–yellow 0042-6989(92)90096-2. color perception and in the color of ‘‘the dress.’’ Current Biology, 25, R547–R548, doi:10.1016/j.cub. Shevell, S. K., & Kingdom, F. A. A. (2008). Color in complex scenes. Annual Review of Psychology, 59, 2015.05.004. 143–166, doi:10.1146/annurev.psych.59.103006. Xian, S. X., & Shevell, S. K. (2004). Changes in color 093619. appearance caused by perceptual grouping. Visual Smithson, H. E. (2005). Sensory, computational and Neuroscience, 21, 383–388. Retrieved from http:// cognitive components of human colour constancy. www.ncbi.nlm.nih.gov/pubmed/15518218 Philosophical Transactions of the Royal Society of Xiao, B. (2016). Color constancy. In R. Luo (Ed.), London. Series B, Biological Sciences, 360, 1329– Encyclopedia of color science and technology. New 1346, doi:10.1098/rstb.2005.1633. York: Springer. doi:10.1007/978-3-642-27851-8_ Sperling, G., & Lu, Z.-L. (1998). A systems analysis of 266-1. visual motion perception. In T. Watanabe (Ed.), Zaidi, Q., Yoshimi, B., Flanigan, N., & Canova, A. High-level motion processing (pp. 153–183). Cam- (1992). Lateral interactions within color mecha- bridge, MA: MIT Press. nisms in simultaneous induced contrast. Vision Tse, P. U. (2005). Voluntary attention modulates the Research, 32, 1695–1707. Retrieved from http:// brightness of overlapping transparent surfaces. www.ncbi.nlm.nih.gov/pubmed/1455741

Downloaded From: http://jov.arvojournals.org/pdfaccess.ashx?url=/data/journals/jov/936104/ on 05/02/2017