A Direct Link Between Visual Working Memory and Saccade Execution Experimental Psychology, Helmholtz Institute, Martijn J
Total Page:16
File Type:pdf, Size:1020Kb
Journal of Vision (2017) 17(6):15, 1–20 1 The cost of making an eye movement: A direct link between visual working memory and saccade execution Experimental Psychology, Helmholtz Institute, Martijn J. Schut Utrecht University, Utrecht, The Netherlands $ Experimental Psychology, Helmholtz Institute, N. Van der Stoep Utrecht University, Utrecht, The Netherlands $ Experimental Psychology, Helmholtz Institute, A. Postma Utrecht University, Utrecht, The Netherlands $ Experimental Psychology, Helmholtz Institute, S. Van der Stigchel Utrecht University, Utrecht, The Netherlands $ To facilitate visual continuity across eye movements, the visual system must presaccadically acquire information Introduction about the future foveal image. Previous studies have indicated that visual working memory (VWM) affects To process visual information, the visual system saccade execution. However, the reverse relation, the redirects the high-acuity fovea to objects of interest in effect of saccade execution on VWM load is less clear. To the visual field by rapidly moving the eyes, i.e., saccades investigate the causal link between saccade execution (Purves, Augustine, & Fitzpatrick, 2001). Although and VWM, we combined a VWM task and a saccade saccades dramatically alter the visual input, we task. Participants were instructed to remember one, subjectively experience a stable visual world, that is, a two, or three shapes and performed either a No Saccade-, a Single Saccade- or a Dual (corrective) world in which items of interest are available for Saccade-task. The results indicate that items stored in processing both before and after a saccade. In what VWM are reported less accurately if a single saccade—or manner the visual system stitches together a continuous a dual saccade—task is performed next to retaining experience from visual input interrupted by saccades is items in VWM. Importantly, the loss of response still debated (Burr, Morrone, & Ross, 1994; Deubel, accuracy for items retained in VWM by performing a Schneider, & Bridgeman, 1996; Matin, 1974). saccade was similar to committing an extra item to One potential candidate to bridge the gaps in visual VWM. In a second experiment, we observed no cost of processing during saccades is an attentional selection executing a saccade for auditory working memory mechanism that is tightly coupled to saccade prepara- performance, indicating that executing a saccade tion, i.e., presaccadic shifts of attention (Deubel & exclusively taxes the VWM system. Our results suggest Schneider, 1996; Irwin & Gordon, 1998; Schneider & that the visual system presaccadically stores the Deubel, 1995). The coupling between attentional upcoming retinal image, which has a similar VWM load selection and saccades is shown by an increase in the as committing one extra item to memory and interferes discriminability of visual input near a saccade target with stored VWM content. After the saccade, the visual (relative to stimuli further away from the saccade system can retrieve this item from VWM to evaluate target), even when the stimuli are no longer available saccade accuracy. Our results support the idea that VWM is a system which is directly linked to saccade after the execution of the saccade (Deubel, 2008; Irwin execution and promotes visual continuity across & Andrews, 1996). Information at (and surrounding) saccades. the attended location is acquired by attentional processes and stored in a memory buffer from which information can subsequently be retrieved after the saccade (Deubel & Schneider, 1996; Irwin & Gordon, 1998). A presaccadic shift of attention may allow the visual system to determine whether the stored visual Citation: Schut, M. J., Van der Stoep, N., Postma, A., & Van der Stigchel, S. (2017). The cost of making an eye movement: A direct link between visual working memory and saccade execution. Journal of Vision, 17(6):15, 1–20, doi:10.1167/17.6.15. doi: 10.1167/17.6.15 Received January 9, 2017; published June 23, 2017 ISSN 1534-7362 Copyright 2017 The Authors Downloaded from jov.arvojournals.orgThis work ison licensed 10/03/2021 under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. Journal of Vision (2017) 17(6):15, 1–20 Schut, Van der Stoep, Postma, & Van der Stigchel 2 input matches the postsaccadic visual input. By tional shifts can be used to refixate a target object if it is determining congruence between pre- and postsaccadic displaced during a saccade (Hollingworth et al., 2008). visual input, motor errors and/or changes in the In the study by Hollingworth and colleagues (2008), external world can be detected and compensated for, participants were presented with a circular array of e.g., by corrective eye movements (Johnson, Holling- stimuli. The saccade target was indicated by a brief worth, & Luck, 2008; Miall & Wolpert, 1996; Richard, increase and decrease in size of one of the stimuli. Luck, & Hollingworth, 2008). Previous research When the participant executed a saccade to this cued indicates that the presaccadic shift of attention, target, the array rotated, such that the participant together with the corollary discharge (an efference copy landed in between the previously cued object and an of the motor command executed by the visual system), uncued (distractor) object. To correctly perform the underlies visual continuity across saccades (Bridgeman, corrective saccade task, the observers had to execute a 1995; Cavanaugh, Berman, Joiner, & Wurtz, 2016; corrective saccade based on the available visual Melcher, 2011; Tian, Ying, & Zee, 2013). information alone, as variance in the motor system was The storage of presaccadic visual input, or trans- uninformative in regards to the postsaccadic location of saccadic memory, is often thought to be dependent on the target. The study showed that participants consis- visual working memory (VWM; Deubel & Schneider, tently executed a secondary (corrective) saccade to the 1996; Irwin, 1991; Irwin & Andrews, 1996; Luck & displaced target. The authors thus showed that Hollingworth, 2008). VWM is a buffer, with a limited information about the saccade target is acquired before amount of storage capacity, which we can consciously execution of the saccade, allowing for corrective access and use to update visual information (Bays, saccades. Another study on corrective saccades showed Catalao, & Husain, 2009; Bays & Husain, 2008; Ma, that VWM content may bias and increase the latency of Husain, & Bays, 2014). As storage capacity in VWM is visual corrective saccades (Hollingworth & Luck, limited: When more information is stored in VWM, 2009). In the study by Hollingworth and Luck (2009), individual items can be represented with less detail, observers were tasked to remember the exact color of a resulting in a decrease in the precision of responses stimulus. The crucial manipulation was a distractor (Bays et al., 2009). Storing presaccadic visual input is object in a visual corrective saccade task that changed important, as it allows for comparison with postsac- into a color that was being maintained by the cadic visual input. For example, after the saccade has participant. Thus, corrective saccades were slower and been completed, the postsaccadic visual input can be biased towards the distractor object when compared to integrated with the visual input stored in VWM to a distractor with irrelevant features. These results estimate saccade accuracy and increase the precision of indicate that VWM is tightly coupled with motor information that is processed by the visual system execution in a visual corrective saccade task. (Ganmor, Landy, & Simoncelli, 2015; Oostwoud Overall, prior studies indicate that corrective sac- Wijdenes, Marshall, & Bays, 2015; Wolf & Schu¨tz, cades are guided by current VWM content. The 2015). Other possible mechanisms include transsaccadic opposite relation, hence how VWM content is affected learning, as several authors have shown that trans- by the execution of a saccade, is less clearly defined. To saccadic changes of stimuli (e.g., shape and spatial specify, the proposed interaction between saccade frequency) can alter how observers respond to pre- execution and VWM is that visual information at and saccadic retinal stimuli (Bosco, Lappe, & Fattori, 2015; surrounding the saccade target is obligatorily and Herwig & Schneider, 2014; Herwig, Weiß, & Schneider, preferentially encoded into VWM before a saccade is 2015; Valsecchi & Gegenfurtner, 2016). Transsaccadic executed. We hypothesize that, before the eye move- learning is thought to occur because the visual system ment is executed, a prediction of the visual features of learns to associate presaccadic peripheral visual input the ‘‘to-be-fixated’’ object is made. This prediction will and postsaccadic foveal visual input over the course of most likely be as accurate as possible, which means it many saccades (Weiß, Schneider, & Herwig, 2014). will have a VWM load comparable to actively Both transsaccadic learning and transsaccadic integra- committing one extra item to VWM (or possibly an tion could underlie comparisons of presaccadic and even higher load). We also hypothesize that the visual postsaccadic retinal input, potentially bridging the gap system will prioritize information acquired by presac- in visual processing during saccades. cadic attentional shifts, as it is essential for the The ability of the visual system to evaluate stored evaluation of saccade accuracy, which in turn is presaccadic visual information after a saccade has been important