Journal of Experimental Psychology: General Copyright 2008 by the American Psychological Association 2008, Vol. 137, No. 1, 163–181 0096-3445/08/$12.00 DOI: 10.1037/0096-3445.137.1.163 Understanding the Function of Visual Short-Term Memory: Transsaccadic Memory, Object Correspondence, and Gaze Correction Andrew Hollingworth and Ashleigh M. Richard Steven J. Luck University of Iowa University of California, Davis Visual short-term memory (VSTM) has received intensive study over the past decade, with research focused on VSTM capacity and representational format. Yet, the function of VSTM in human cognition is not well understood. Here, the authors demonstrate that VSTM plays an important role in the control of saccadic eye movements. Intelligent human behavior depends on directing the eyes to goal-relevant objects in the world, yet saccades are very often inaccurate and require correction. The authors hypothesized that VSTM is used to remember the features of the current saccade target so that it can be rapidly reacquired after an errant saccade, a task faced by the visual system thousands of times each day. In 4 experiments, memory-based gaze correction was accurate, fast, automatic, and largely unconscious. In addition, a concurrent VSTM load interfered with memory-based gaze correction, but a verbal short-term memory load did not. These findings demonstrate that VSTM plays a direct role in a fundamentally important aspect of visually guided behavior, and they suggest the existence of previously unknown links between VSTM representations and the occulomotor system. Keywords: visual short-term memory, visual working memory, eye movements, saccades, gaze correc- tion Human vision is active and selective. In the course of viewing integrate sensory information presented on separate fixations (Ir- a natural scene, the eyes are reoriented approximately three times win, Yantis, & Jonides, 1983; O’Regan & Le´vy-Schoen, 1983; each second to bring the projection of individual objects onto the Rayner & Pollatsek, 1983). In recent work with naturalistic scene high-resolution, foveal region of the retina (for reviews, see Hen- stimuli, researchers arrived at a similar conclusion. Relatively derson & Hollingworth, 1998; Rayner, 1998). Periods of eye large changes to a natural scene can go undetected if the change fixation, during which visual information is acquired, are separated occurs during a saccadic eye movement or other visual disruption by brief saccadic eye movements, during which vision is sup- (Grimes, 1996; Henderson & Hollingworth, 1999, 2003b; Rensink, pressed and people are virtually blind (Matin, 1974). The input for O’Regan, & Clark, 1997; Simons & Levin, 1998), an effect that vision is therefore divided into a series of discrete episodes. To has been termed change blindness. For example, Henderson and span the perceptual gap between individual fixations, a transsac- Hollingworth (2003b) had participants view scene images that cadic memory for the visual properties of the scene must be were partially occluded by a set of vertical gray bars (as if viewing maintained across each eye movement. the scene from behind a picket fence). During eye movements, the bars were shifted so that the occluded portions of the scene became Transsaccadic Memory and Visual Short-Term Memory visible and the visible portions became occluded. Despite the fact (VSTM) that every pixel in the image changed, the participants were almost entirely insensitive to these changes, demonstrating that low-level In early theories, it was proposed that transsaccadic memory sensory information is not preserved from one fixation to the next. integrates low-level sensory representations (i.e., iconic memory) Although transsaccadic memory does not support low-level across saccades to construct a global image of the external world sensory integration, visual representations are nonetheless retained (McConkie & Rayner, 1976). However, a large body of research across eye movements. In transsaccadic object identification stud- demonstrated conclusively that this is false; participants cannot ies, participants are faster to identify an object when a preview of that object has been available in the periphery before the saccade (Henderson, Pollatsek, & Rayner, 1987), and this benefit is re- Andrew Hollingworth and Ashleigh M. Richard, Department of Psy- duced when the object undergoes a visual change during the chology, University of Iowa; Steven J. Luck, Center for Mind and Brain saccade, such as substitution of one object with another from the and Department of Psychology, University of California, Davis. same basic-level category (Henderson & Siefert, 2001; Pollatsek, The research was supported by National Institute of Health Grants Rayner, & Collins, 1984) and substitution of an object with its R01EY017356, R01MH063001, R01MH076226, and R03MH65456. Parts mirror reflection (Henderson & Siefert, 1999). In addition, object of this study were reported at the 45th Annual Meeting of the Psychonomic Society, Toronto, Ontario, Canada, 2005. priming across saccades is governed primarily by visual similarity Correspondence concerning this article should be addressed to An- rather than by conceptual similarity (Pollatsek et al., 1984). Fi- drew Hollingworth, Department of Psychology, University of Iowa, 11 nally, structural descriptions of simple visual stimuli can be re- Seashore Hall E, Iowa City, IA 52242-1407. E-mail: andrew- tained across eye movements (Carlson-Radvansky, 1999; Carlson- [email protected] Radvansky & Irwin, 1995). Thus, memory across saccades appears 163 164 HOLLINGWORTH, RICHARD, AND LUCK to be limited to higher level visual codes, abstracted away from & Irwin, 2000; Henderson & Hollingworth, 1999, 2003a; Irwin, precise sensory representation but detailed enough to specify in- McConkie, Carlson-Radvansky, & Currie, 1994; McConkie & dividual object tokens and viewpoint. Currie, 1996). Multiple lines of converging evidence indicate that visual mem- The issue of object correspondence across saccades has often ory across saccades depends on the VSTM system originally been framed as a problem of visual stability: How do people identified by Phillips (1974) and investigated extensively over the consciously perceive the world to be stable when the image pro- last decade (see Luck, in press, for a review).1 On all dimensions jected to the retina shifts with each saccade? Irwin, McConkie, and tested, transsaccadic memory exhibits properties similar to those colleagues (Currie et al., 2000; Irwin et al., 1994; McConkie & found for VSTM. Both VSTM and transsaccadic memory have a Currie, 1996; see also Deubel & Schneider, 1994) proposed a capacity of 3–4 objects (Irwin, 1992a; Luck & Vogel, 1997; saccade target theory to explain visual stability. In this view, an Pashler, 1988) and lower spatial precision than sensory memory object is selected as the saccade target before each saccade. Per- (Irwin, 1991; Phillips, 1974). Both VSTM and transsaccadic mem- ceptual features of the target are encoded into VSTM and main- ory maintain object-based representations, with capacity deter- tained across the saccade. After completion of the saccade, the mined primarily by the number of objects to remember and not by visual system searches for an object that matches the target infor- the number of visual features to remember (Irwin & Andrews, mation in VSTM, with the search limited to a spatial region around 1996; Luck & Vogel, 1997). Finally, both VSTM and transsac- the landing position. If the saccade target is located within the cadic memory are sensitive to higher order pattern structure and search region, the experience of visual stability is maintained. If grouping (Hollingworth, Hyun, & Zhang, 2005; Irwin, 1991; the saccade target is not found, the observer becomes consciously Jiang, Olson, & Chun, 2000). The logical inference is that VSTM aware of a discrepancy between pre- and postsaccade visual ex- is the memory system comprising transsaccadic memory. perience. Saccade target theory was proposed to account for the A final strand of evidence about VSTM and transsaccadic phenomenology of visual stability, but the evidence reviewed memory is particularly germane to the present study. Before a above, demonstrating preferential encoding of the saccade target saccade, visual attention is automatically and exclusively directed into VSTM, raises the possibility that VSTM supports the more to the target of that saccade (Deubel & Schneider, 1996; Hoffman general function of establishing correspondence between the sac- & Subramaniam, 1995). Attention also supports the selection of cade target object visible before and after the saccade. Such perceptual objects for consolidation into VSTM (Averbach & correspondence might produce the experience of visual stability, Coriell, 1961; Hollingworth & Henderson, 2002; Irwin, 1992a; but that need not be its only purpose. We argue that an important Schmidt, Vogel, Woodman, & Luck, 2002; Sperling, 1960). Thus, function of VSTM across saccades is to support the correction of the saccade target object is preferentially encoded into VSTM and gaze when the eyes fail to land on the target of a saccade. Such stored across the saccade. Indeed, memory performance is higher gaze corrections are required thousands of times each day. A for objects at or near the target position of an impending or VSTM representation of the saccade target allows the target to be just-completed saccade (Henderson & Hollingworth, 1999, 2003a; found after an inaccurate