Transsaccadic Memory: Working Memory in Action

Adam Frost

A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Department of Psychology

Transsaccadic Memory: Working Memory in Action

Adam Frost

Doctor of Philosophy

Department of Psychology University of Toronto

2020

Abstract

Transsaccadic memory is the process by which visual information is maintained and spatially updated across eye movements. Transsaccadic memory resembles general visuospatial working memory in many ways, but it is unclear whether the two functions do share common faculties, or if the transsaccadic functions are underpinned by specialized mechanisms given the unique demands of maintaining spatial constancy across eye movements. Six experiments are presented in this thesis to address this question. In Chapter 2, I found that transsaccadic memory performance is predicted by performance on a 2-Back task, demonstrating commonalities with general visuospatial working memory. I also found, however, that performance on a change detection task did not predict transsaccadic memory performance, indicating a dissociation between a well-established contemporary measure of visuospatial working memory and transsaccadic memory. Change detection performance also failed to predict performance on the

2-Back task, so in Chapter 3, four variants of the change detection task were created to test possible explanations for the dissociations. All change detection tasks correlated strongly with the standard change detection task, but a correlation with the 2-Back task was seen only with a change detection variant where the availability of contextual spatial information during the test phase was minimized. This version of the change detection task was then tested against the transsaccadic task in Chapter 4, along with 2-Back and a no-saccade baseline for the ii transsaccadic task to isolate saccade-related performance components. Doing so, two saccade- related components were found that predicted performance on the working memory measures, though only one of these components explained variance across all tasks. I conclude that transsaccadic memory does appear to share faculties with general visuospatial working memory, but that like visuospatial working memory itself, it is a multifaceted construct.

iii

Acknowledgements

This PhD thesis was made possible by the support of a broad community, and I would like to express my appreciation to some individuals whose contributions have been especially important. I'd first like to thank Matthias Niemeier for providing a lab where I could pursue ideas with all the material and intellectual resources that I could possibly need. Matthias has been an inexhaustible well of theoretical insights and is a role model for the breadth of knowledge and expertise that a person can attain, while also being a kind and generous mentor. I am very fortunate to have had the distinguished John Kennedy and Mark Schmuckler provide their guidance and feedback throughout this PhD; their insights and generous spirit helped the thesis find its form. I'd also like to thank Susanne Ferber for introducing me to working memory research and providing me with the tools to start working in that domain.

Matthias has extended his own expertise by building a lab of highly capable researchers, who have themselves been invaluable resources to me. Ada Le gave up many lab hours during her final year to help me debug MATLAB code, and generally helped me develop effective research practices. She also played a crucial role in introducing me to the broader community at the University of Toronto, and connected me to a number of great researchers through her role as president of the Psychology Graduate Students’ Association. I'd like to thank Jiaqing Chen for her statistics prowess, fastidious attention to detail, and general good nature, and Lawrence Guo for introducing me to powerful new machine learning methods through his collaborations with Adrian Nestor. I would also like to thank my newer lab mates, Nina Lee, Teddy Cheung, and Simar Moussaoui for bringing their fresh ideas with them. My research assistants contributed more hours than I can count towards data collection and data processing, so I’d like to express my appreciation to Harsh Parikh, Jagjot Kaur, George Tomou, Andrew Laughlin, Marija Zivcevska, Samreen Aziz, Aleem Ahmad, Mihilkumar Patel and Erind Alushaj.

The administrative team at the University of Toronto provided flawless support throughout my graduate school career, ensuring that deadlines were met and providing the means to make my studies financially viable. Ann Lang, Nina Dhir, Kathleen Cook and Ainsley Lawson were all especially noteworthy in that regard, and I am grateful for their contributions. I would like to thank the Natural Sciences and Engineering Research Council of Canada, and the University of Toronto for their financial support.

On a personal level, the process of completing this thesis was made possible through the patience and emotional support of Anita Nipen, who agreed to convert our living room into an office so that I could finish writing during the coronavirus pandemic. I’d like to thank my dear friends Vincent Man, Michael Weiss, Buddhika Bellana, Britt Caron, Olivia Podolak Lewandowska, Nick Diamond, Iva Brunec, Mike Armson and Ed O’Neil for their comradery and their wisdom. They are all formidable researchers in their respective domains, and I owe a debt of gratitude to the psychology department for building culture that attracts people of their quality. Finally, I’d like to thank my parents Larry and Wendy Frost for providing the foundation for this entire enterprise.

Table of Contents

Acknowledgements ...... iv

Table of Contents ...... vi

List of Tables ...... ix

List of Figures ...... x

Chapter 1 ...... 1

General Introduction ...... 1

Spatial stability across saccades ...... 1

The neurophysiology of transsaccadic spatial constancy ...... 2

Transsaccadic memory ...... 4

Visual Working Memory ...... 6

Visual Working Memory VS. Transsaccadic Working Memory ...... 11

Present Research ...... 12

Chapter 2 ...... 14

Working memory in action: Inspecting the systematic and unsystematic errors of spatial memory across saccades ...... 14

Abstract ...... 14

Introduction ...... 14

Experiment 1 ...... 17

2.3.2 Results and Discussion ...... 21

Experiment 2 ...... 26

2.4.1 Participants ...... 26

2.4.2 Apparatus ...... 27

2.4.3 Procedure ...... 27

2.4.4 Results ...... 30

Experiment 3 ...... 36

2.5.1 Participants ...... 36

2.5.2 Apparatus ...... 36

2.5.3 Procedure ...... 36

2.5.4 Results ...... 39

General Discussion ...... 46

Chapter 3 ...... 49

Resolving Dissociations between Working Memory Tasks: The Effects of Disruption and Context ...... 49

Introduction ...... 49

Experiment 4: Cognitive vs. Perceptual Interference ...... 52

3.2.1 Method ...... 52

3.2.2 Results ...... 56

Experiment 5: Spatial vs. Non-spatial Context ...... 60

3.3.1 Methods...... 60

3.3.2 Results ...... 62

General Discussion ...... 66

Chapter 4 ...... 69

Transsaccadic Memory: Saccadic and Non-Saccadic Error Sources ...... 69

Introduction ...... 69

Experiment 6 ...... 70

4.2.1 Method ...... 70

4.2.2 Results ...... 76

4.2.3 Discussion ...... 84

General Discussion ...... 86

Summary of results ...... 86

Implications and future directions ...... 88

vii

Conclusion ...... 89

References ...... 91

viii

List of Tables

Table 2.1 2-Way Repeated Measures ANOVAs for the measures of transsaccadic error ...... 24

Table 2.2 Correlations between 2-Back performance and measures of transsaccadic error ...... 25

Table 2.3 2-Way Repeated Measures ANOVAs for the measures of transsaccadic error ...... 33

Table 2.4 Correlations between change detection performance and measures of transsaccadic error ...... 35

Table 2.5 2-Way Repeated Measures ANOVAs for the measures of transsaccadic error ...... 42

Table 2.6 Correlations between 2-Back performance and measures of transsaccadic error ...... 44

Table 3.1 Correlations between 2-Back and Change Detection performance ...... 58

Table 3.2 Rotated component scores ...... 59

Table 3.3 Correlations between 2-Back and Change Detection performance ...... 65

Table 3.4 Rotated component scores ...... 66

Table 4.1 Repeated Measures ANOVAs for the TSM/TSM-Control Error Variables ...... 77

Table 4.2 Rotated component scores ...... 80

Table 4.3 Correlations between measures of transsaccadic error and working memory ...... 82

List of Figures

Figure 2.1 Experimental Paradigm. A. TSM task. B. 2-Back line orientation task ...... 20

Figure 2.2 Systematic and unsystematic errors. A. Errors after leftward saccades. B. Errors after rightward saccades. As a visual analog for standard error, ellipses were fitted to group-level responses (mouse clicks) for each of the 9 possible intersection locations, with radii shrunk by a factor of 10 for graphical clarity. Ellipse centres represent the mean response for a given intersection location. Black: low memory load condition, grey: high memory load condition. Data are superimposed onto the true intersection locations (3x3 grid of plus signs) ...... 21

Figure 2.3 Results of Experiment 1: TSM performance measures for the two saccade directions and memory loads separately. A & B horizontal and vertical unsystematic error. C-G measures of systematic error ...... 22

Figure 2.4 Results of Experiment 1: Correlations between 2-Back performance and measures of transsaccadic error ...... 25

Figure 2.5 Experimental Paradigm. A. Transsaccadic memory task. B. Change detection task ..29

Figure 2.6 Systematic and unsystematic errors. A. Errors after leftward saccades. B. Errors after rightward saccades. As a visual analog for standard error, ellipses were fitted to group-level responses (mouse clicks) for each of the 9 possible memory locations, with radii shrunk by a factor of 10 for graphical clarity. Ellipse centres represent the mean response for a given intersection location. Black: low memory load condition, grey: high memory load condition. Data are superimposed onto the true intersection locations (3x3 grid of plus signs) ...... 30

Figure 2.7 Results of Experiment 2: TSM performance measures. Bar colour indicates load level (black for low load, grey for high), and grouping indicates the saccade direction. Vertical axis labels indicate the variable displayed. A & B horizontal and vertical unsystematic error. C-G measures of systematic error ...... 31

Figure 2.8 Results of Experiment 2: Correlations between CD performance and measures of transsaccadic error ...... 35

Figure 2.9 Experimental Paradigm. A. Transsaccadic memory task. B. 2-Back task. C. Change detection task ...... 38

Figure 2.10 Systematic and unsystematic errors. A. Errors after leftward saccades. B. Errors after rightward saccades. As a visual analog for standard error, ellipses were fitted to group-level responses (mouse clicks) for each of the 8 possible memory locations, with radii shrunk by a factor of 10 for graphical clarity. Ellipse centres represent the mean response for a given intersection location. Black: low memory load condition, grey: high memory load condition. Data are superimposed onto the true intersection locations (crosses) ...... 39

Figure 2.11 Results of Experiment 3: TSM performance measure means. Bar colour indicates load level (black for low load, grey for high), and grouping indicates the saccade direction. Vertical axis labels indicate the variable displayed. A & B horizontal and vertical unsystematic error. C-G measures of systematic error ...... 40

Figure 2.12 Results of Experiment 3: Correlations between 2-Back performance and measures of transsaccadic error ...... 45

Figure 2.13 Results of Experiment 3: Correlations between CD performance and measures of transsaccadic error ...... 45

Figure 3.1 Experimental paradigm. CD-Remap. Change detection remap task. CD-Std. Standard change detection task. CD-Mask. Change detection mask task. 2-Back. 2-Back task...... 55

Figure 3.2 Results of Experiment 4: Working memory performance (d’) means. CD-Std: CD-Standard, CD-Rmp: CD-Remap, CD-Mask: CD-Mask. Error bars indicate standard errors...... 57

Figure 3.3 Rotated component scores of the principal component analysis...... 59

Figure 3.4 Experimental paradigm. CD-Single. Change detection single task. CD-Std. Standard change detection task. CD-Indi. Change detection indicate task (self-paced response during test). 2-Back. 2-Back task...... 62

Figure 3.5 Results of Experiment 5: A. Working memory performance d’ means. First bar. 2-Back task. Second bar. Change detection standard task. Third bar. Change detection single task. Error bars indicate the standard error. B. Mean working memory performance (percent correct) for the change detection indicate task. Error bar indicates the standard error...... 64

Figure 3.6 Rotated component scores of the principal component analysis ...... 66

Figure 4.1 Experimental Paradigm ...... 75

Figure 4.2 Systematic and unsystematic errors. Crosses represent the eight possible stimulus presentation locations. A. Errors after leftward saccades. B. Errors after rightward saccades. C. Errors after the no-saccade control. As a visual analog for standard error, ellipses were fitted to group-level responses (mouse clicks) for each of the 8 possible memory locations, with radii shrunk by a factor of 10 for graphical clarity. Ellipse centres represent the mean response for a given intersection location...... 77

Figure 4.3 Results of Experiment 6: TSM and control condition performance measure means. Dark bars indicate the eye movement conditions, light bars indicate the no-saccade control condition. Vertical axis labels indicate the variable displayed. A & B horizontal and vertical unsystematic error. C-G measures of systematic error ...... 78

Figure 4.4 Results of Experiment 6: Correlations between 2-Back performance and measures of transsaccadic error ...... 83

Figure 4.5 Results of Experiment 6: Correlations between CD performance and measures of transsaccadic error...... 83

xii

Chapter 1

General Introduction

Our experience of vision as a continuous visual stream that is stable, expansive, and highly detailed, is an illusion. In reality, only a small central area of the retina is capable of resolving fine details, so the eyes must be moved to place this high-resolution ‘fovea’ over points of interest to build up a detailed field of view. To achieve a functional implementation of this strategy, the brain overcomes several obstacles, and the research presented in this thesis examines mechanisms that are used to achieve this – namely, those supporting the retention and translation of information across quick ‘saccadic’ eye movements, and the possible involvement of general visual working memory faculties. In the first section of this chapter I introduce the problems facing spatial stability across saccades and mechanisms that contribute to their resolution from neurophysiology to transsaccadic working memory. Following that, I review research relating to general visual working memory, and detail how it both parallels and differs from transsaccadic working memory. The chapter is concluded by introducing the overarching goals of the studies presented in this thesis, and the means by which these goals were achieved.

Spatial stability across saccades

About three times per second, the eyes rotate at velocities up-to and beyond 700° per second to move the fovea from one environment point of interest to another. These rotations are pre- programmed and ballistic, stopping briefly to capture a detailed ‘snapshot’ at one point before jumping to the next. Most of the time these saccades are programmed and executed automatically, and indeed we are rarely aware of our saccades (e.g. Stevens et al., 1976). Given that the image of space projected onto the retina changes with each saccade, one might ask: How does the brain combine the ‘snapshots’ it acquires from each fixation to provide a ‘seamless’ experience of vision? The perception of a world that is stable in spite of sweeping changes to retinal images due to gaze shifts is known as spatial constancy and is achieved through remapping and transsaccadic memory.

1 2

The neurophysiology of transsaccadic spatial constancy

Maintaining spatial constancy across a saccade is a multi-stage process that begins before the eye starts to move (Duhamel, Colby & Goldberg, 1992; Colby, Duhamel & Goldberg, 1996; Sommer & Wurtz, 2006), then ends some time after refixation (Boon et al., 2018), and neural correlates of these processes have been observed at the single-cell and population level. Before a saccade, target candidates in space are represented retinotopically by activity, for example, in the superior colliculus (e.g. McPeek & Keller, 2002; Shen & Paré, 2007; Shen & Paré 2014), lateral intraparietal area (LIP; Duhamel, Colby & Goldberg, 1992; Ipata et al., 2006; Thomas & Paré, 2007; Ogawa & Komatsu, 2009) and frontal eye fields (FEF; Schall & Hanes, 1993), with target selection reflected by increased activity in a number of retinotopic neuronal populations in these regions (Colby, Duhamel & Goldberg, 1996; McPeek & Keller, 2002; Shen & Paré, 2007; Shen & Paré 2014; Ipata et al., 2006; Thomas & Paré, 2007; Ogawa & Komatsu, 2009; Schall & Hanes, 1993). Once a target is chosen, a corollary discharge of the saccadic motor command of the target selection is relayed to areas that represent stimuli in retinotopic coordinates (Sommer & Wurtz, 2004; 2006). In response to this corollary discharge, populations of neurons receiving these signals shift their receptive fields to the regions they will represent once the eye movement is complete. For instance, during fixation, neurons in FEF will respond to a visual stimulus should its image fall on specific retinotopic coordinates (i.e. its receptive field). However, just before a saccade, such a neuron will start to respond to stimuli that fall on a different part of the retina – specifically, to the position its fixational receptive field will occupy after the upcoming saccade has been completed. This ‘elastic’ relocation of the neurons’ receptive fields is controlled by corollary discharges of saccade motor control commands generated by a pathway projecting from the superior colliculus to the mediodorsal nucleus of the thalamus and from there to the frontal eye fields (Sommer & Wurtz 2004). That is, Sommer and Wurtz showed that disabling the mediodorsal pathway from the superior colliculus, through the mediodorsal nucleus of the thalamus and to the FEF caused monkeys to commit remapping errors (Sommer & Wurtz 2002) and receptive fields of FEF neurons to no longer shift before saccades (Sommer & Wurtz 2006).

Similar remapping behaviour can also be observed in other spatially selective visual neurons in LIP (Colby, Duhamel & Goldberg, 1996; Duhamel, Colby & Goldberg., 1992) and the superior colliculus (Walker, FitzGibbon & Goldberg, 1995). These pre-saccadic receptive field shifts

3 likely support spatial constancy by providing information about saccade error, possibly in the form of signals reflecting pre-and-post-saccadic correspondence (Wurtz, 2008). As another example of anticipatory effects, immediately before the eyes begin to move, visual perception is suppressed through active and passive mechanisms, to minimize the impact of distorted visual percepts due to self-motion (Campbell and Wurtz, 1978; Judge, Wurtz & Richmond, 1980; Mackay, 1970; Matin, Clymer & Matin, 1972; Matin and Pearce, 1965; Honda, 1989, 1993; Cai et al., 1997; Morrone, Ross & Burr, 1997; Ross, Morrone & Burr, 1997; Lappe Awater & Krekelberg, 2000; Kaiser and Lappe, 2004; Kis, Singh & Niemeier, 2009; Frost & Niemeier, 2015).

Anticipatory visual mechanisms likely serve an important role in maintaining spatial constancy since they operate on a timeline congruent with the saccade’s speed and frequency, whereas other sources of feedback like extraocular proprioception are relatively slow and can only convey information about eye movements once the eye has started to move. The proprioceptive signals that follow a saccade are, therefore, likely relegated to producing calibratory feedback for the visuomotor system (Lewis et al., 2001; Wurtz, 2008).

Saccades are ballistic, and the unavailability of immediate online feedback means that the saccade target is often either overshot or undershot, but the process of matching the desired result with the actual result allows corrections to be made quickly and unobtrusively. When a mismatch is detected (within a certain threshold), corrective saccades are generated automatically to achieve the intended final position (e.g. Deubel, Bridgeman & Schneider, 1998; Harris, 1995) which helps correct for motor error when visual feedback is available (Tian, Ying & Zee, 2013). Under experimental conditions, when a continuously visible saccade target is made to jump during a saccade, a corrective saccade typically is made automatically so that the displaced target comes to be foveated with little delay. When corrected automatically like this, large displacements of the target often go unnoticed by participants. However, these automatic corrections require pre-post saccadic correspondence, and this can be broken by introducing a brief, 100 ms blanking period before the jump is made. With correspondence broken, corrective saccades are generally not triggered, and small displacements of the target are both identified and consciously recognized as such (Deubel, Bridgeman & Schneider, 1998). Without being accounted for, motor error would cause ambiguity about whether jitter in a saccade landing position is caused by self-motion or motion in the world, presenting a possible threat to spatial

4 stability that is overcome by incorporating prior probabilities reflecting the low likelihood that the world will suddenly shift during a saccade (Niemeier, Crawford & Tweed, 2003; 2007). There is still some uncertainty about how visual feedback is used to generate corrective saccades, but remapping is a possible explanatory factor (Collins et al., 2009; Tian, Ying & Zee, 2013).

Transsaccadic memory

Thus far, remapping has been discussed with respect to stimuli that are continuously visible, but many of these processes can be observed to operate even when salient stimuli are no longer visible, which demonstrates the existence of a form of transsaccadic memory. Duhamel, Colby and Goldberg (1992) for instance, found that a stimulus flashed up to one second before a saccade in the future field (the position at which its receptive field will fall following the upcoming saccade) of a neuron in LIP, initially did not evoke a response. After the saccade however, the neuron produced a burst of activity in response to the previously flashed stimulus, which demonstrates the persistence of visual information and its compensatory translation in the absence of ongoing stimulation. More evidence for the retinotopic remapping of remembered coordinates has come from experiments using a ‘double-step’ task, where either humans or macaques perform a sequence of saccades to remembered spatial locations (Hallett & Lightstone, 1976; Sommer & Wurtz, 2002). In the task used by Sommer and Wurtz (2002) for example, two saccade targets were flashed in sequence, and monkeys made saccades to these targets after they were extinguished. Though this task could be accomplished without retinotopic remapping, e.g. by pre-programming both saccades or using a more stable spatial map (head-centered or body- centered for example), evidence supports the presence of a remapping mechanism. In Sommer & Wurtz (2002), this was tested in primates performing the aforementioned double-step task, while saccade-related corollary discharge signals from the superior colliculus were kept from reaching FEF by interrupting a pathway through the mediodorsal thalamus. With a non-remapping mechanism, the interruption of the corollary discharge should either distort each saccade individually or have no systematic effect. On the other hand, if a retinotopic remapping mechanism is present, then displacements caused by the first saccade should fail to be accounted for, and the second saccade should be programmed as though it is originating from the original fixation point. The observed result was that vectors for the second saccade followed an incorrect but consistent trajectory, as though they were originating in the vicinity of the initial fixation

5 point. This suggests that remembered positions are updated in retinotopic coordinates following the completion of a saccade, as guided by corollary discharge signals.

Though transsaccadic memory helps support spatial constancy when stimuli are no longer visible, it is not perfect, and a lack of visual feedback causes random error to accumulate. For instance, when saccades are made to a remembered location, they are significantly less precise than when the target is available (White, Sparks & Stanford, 1994). When visual landmarks are provided though, the precision of saccades to remembered locations is significantly restored (Burke, Clarke & Hedley, 2010). That said, the presentation of a visual landmark at retrieval only partially corrects the growing imprecision caused by multiple intervening saccades, suggesting information loss in both allocentric and egocentric reference frames (Karn, Møller & Hayhoe, 1997). Some systematic distortions are also observed in remapped remembered spatial coordinates, such as overshoot when stimuli are remapped into the periphery and indicated by pointing (Enright, 1995; Henriques et al., 1998). Transsaccadic memory appears to differ in its efficacy depending on where in space information is remapped to: For instance, recall performance for remapped remembered items has been found to be better when a rightward saccade is made as compared to leftward saccades (Vasquez & Danckert, 2008). Later contributions to this line of research found better information integrity when stimuli are remapped within left visual hemispace, which could explain the previous finding since remapping within left hemispace is more likely when making a rightward eye movement (Ten Brink et al., 2019). These findings are consistent with a specialization of the right cortical hemisphere for spatial representations and operations (Jonides et al., 1993; Prime, Vesia & Crawford, 2008; 2010), in spite of the fact that the right hemisphere generally represents left visual space and the left hemisphere represents right space (although a right-hemisphere dominance arises with higher working memory loads, Sheremata et al., 2010).

Though the process of remapping and recalling remembered information across an eye movement is imperfect, but in practice these small deviations from the original spatial coordinates represent relatively minor information losses, which contrast starkly against the fact that nearly all other retinal information is lost during a saccade. The scarcity of information retained across an eye movement was demonstrated in early change blindness experiments, where dramatic changes in a scene would go unnoticed by participants when the changes coincided with a saccade (Grimes, 1996). In one experiment, an image of two men changed

6 during a saccade such that their differently coloured hats were swapped, and only 50% of participants noticed in spite of having been warned that such changes might happen. Transsaccadic memory is not, therefore, a pictographic, ‘pixel-by-pixel’ information store. Rather, it is highly selective and carries only a sliver of the available information across an eye movement. It is apparent though, that this near-complete loss of information across eye movements has not significantly impeded humankind’s survival (nor that of other vertebrates that use similar strategies). This is less surprising when one considers that most visual information available before the saccade will become available again upon refixation, albeit at shifted spatial coordinates.

Soon after this limitation of transsaccadic memory was identified, other visual interruptions were found that could similarly cause change blindness to occur. For example, change blindness is exhibited when the changes are introduced while a participant is blinking (O’Regan et al., 2000), when a screen is blanked between presentations (Rensink, O’Regan & Clark, 1997), or when a change to a scene coincides with a briefly flashed visual distraction elsewhere in the image (O’Regan, Rensink & Clark, 1999). Whatever the source of the interruption, these experiments show that visual working memory carries only sparse information from one moment to the next, and explicit representation is limited to a surprisingly sparse set of features.

Visual Working Memory

The vast majority of a visual scene is not explicitly represented from one moment to the next, but a few key visual elements are maintained through visual working memory. The relationship between this general, stable vision form of visual working memory and transsaccadic memory is the primary focus of this thesis.

At one time, researchers debated whether or not working memory was distinct from long term memory, but an accumulation of evidence demonstrating behavioural and neurological differences between the two provided support for their separation (e.g. patients can show profound deficits in one but not the other; Baddeley & Warrington, 1970; Drachman & Arbit, 1966; Scoville & Milner, 1957; Shallice & Warrington, 1970). Baddeley & Hitch (1974) proposed a model which further subdivided working memory into a ‘central executive’ supplemented by a ‘phonological loop’ and ‘visuospatial sketchpad’ for verbal and visual working memory, respectively. Visual working memory was modelled as a separate faculty from

7 verbal working memory since the two were observed to be relatively resource independent. If visuospatial memory and verbal memory were simply different applications of the same finite- capacity system, one would expect to observe a drop in the capacity of one when the other is concurrently engaged. This is not the case; experimental results have shown that capacity of visuospatial working memory is not greatly impacted by adding a concurrent verbal memory load and vice versa (Luck & Vogel, 1997, Scarborough, 1972; Vogel, Woodman & Luck, 2001). Further, visuospatial working memory is much more limited in capacity than verbal working memory. Verbal working memory is characterized by a capacity of approximately seven items which, if exceeded, results in information loss (Miller, 1956). The analog in visuospatial working memory is a limit of approximately three to four items (Luck & Vogel, 1997; Vogel & Machizawa, 2004; Todd & Marois, 2005; Xu & Chun, 2006). This visuospatial working memory capacity has frequently been measured using a ‘change detection’ task, where an array of geometric shapes (e.g., coloured squares) to be remembered is briefly flashed, then after a delay of about one second, is shown again, at which point participants indicate whether or not one of the items has changed in colour. When the number of coloured squares is within a participant’s capacity limit, a colour change in the recall array is reliably noticed. But, if a participant’s capacity limit has been exceeded, the probability of noticing a change is decreased to approximately . That said, more sophisticated formulae exist that better capture observed performance by incorporating other explanatory variables to account for factors like response biases and guessing (e.g. Rouder et al., 2011). Put simply, a person’s error rate increases proportionally to the number of items in the array that exceed that person’s working memory capacity.

Performance on the change detection task is robustly associated with characteristic neural activity. In the time between the offset of the memory array and recall, ‘contralateral delay activity’ proportional to the number of items held in visual working memory can be observed in both ERPs (e.g. Vogel & Machizawa, 2004; Ikkai, McCollough & Vogel, 2010) and BOLD activity (Todd & Marois, 2004; Xu & Chun, 2006). When the number of items that must be held exceeds a person’s working memory capacity, contralateral delay activity reaches a plateau or logarithmic saturation point, beyond which additional items have little effect on its amplitude (Bays, 2018; Todd & Marois, 2004; Vogel & Machizawa, 2004; Xu & Chun, 2006), and these properties predict behaviourally measured capacity limits (Vogel & Machizawa, 2004; Todd &

Marois, 2005; Xu & Chun, 2006). Performing visual working memory tasks produces broad cortical activation from early visual areas like striate and extrastriate cortex (Greenlee, Magnussen & Reinvang, 2000) up to those associated with higher cognition in prefrontal cortex (Cohen et al., 1997; Goldman-Rakic, 1995; Curtis & D’Esposito, 2003). Within these, a small subset produces activity that reflects visual working memory load independently of visual stimulation. Specifically, activity in lateral occipital cortex and the intraparietal sulci show load- dependent modulations (Sheramata, Beetencourt & Somers, 2010; Todd & Marois, 2004; Vogel & Machizawa, 2004; Xu & Chun, 2006). The capacity of visuospatial working memory was initially conceptualized as a limited number of discrete ‘slots.’ In a basic sense, this conceptualization suggests two things. It argues for a system where information is stored in discrete units until all slots are occupied, from which point no further information can be retained. It also suggests that capacity is not limited by how simple or relatively complex the individual items are (Awh, Barton & Vogel, 2007; Luck & Vogel, 1997; Luck & Vogel, 2013). Indeed, stimulus arrays with feature-sets more complex than the classic space-colour associations (e.g. complex shapes that can change both colour and orientation instead of simple coloured squares, Luck & Vogel, 1997) used to measure working memory can be held in similar numbers to single-feature items.

However, alternative explanations for both properties of the slot model exist. First, observations suggested that working memory is a fluid resource that can be allocated in a non-discrete fashion. Evidence for this has accumulated through the use of variants of the change detection paradigm that employ, instead of binary correct/incorrect responses, recall along a continuous spectrum in ‘delayed estimation’ tasks. In one such experimental design, participants remember an array of coloured squares, just as they would in the classic change detection paradigm, but then at recall one of the squares is spatially cued, and the participant must indicate that square’s colour from a continuous spectrum of hues presented as a colour wheel (e.g. Zhang & Luck, 2008; Bays, Catalo & Hussain, 2009). These responses are evaluated in terms of their angular distance from the square’s true colour, which can then be analyzed using ‘mixture models’ to gain further insights into the sorts of errors a participant tends to make (Zhang & Luck, 2008; Bays, Catalo & Husain, 2009; Ma, Husain & Bays, 2014). Specifically, these models quantify random error, systematic offset, colour swap-in from one of the other items, and wholesale information loss. Therefore, responses on these delayed estimation tasks provide information

9 beyond whether a given response suggests that an item is or is not held in working memory. That is, they provide a measure of the fidelity of the probed item’s representation. Behavioural results from these tasks have led to the observation that recall precision falls as more items are stored in working memory (Bays, Catalo & Husain, 2009; Bays & Husain, 2008), which is inconsistent with the ‘all or nothing’ representations that would be expected from a slot-like storage system. Using these tasks, it has also been observed that the precision of recall for a given item is dependent on several factors beyond the load-capacity relationship.

Experimental results suggest that stimulus features can be stored in an efficient way, where similarities between items or contextual continuity are exploited to increase recall precision across a number of stimulus features such as orientation (Sims, Jacobs & Knill, 2012), spatial distribution (Jiang, Olson & Chun, 2000), colour (Brady & Alvarez, 2015; Sun et al., 2017) and others (e.g. Jiang et al., 2016). For example, memory array items with colours drawn from a range limited to ‘greenish’ hues, can be recalled with greater fidelity than colours randomly drawn from a broad range of hues, where no one carries information about the others (Brady & Alvarez, 2015; Sun et al., 2017). Since increasing load decreases precision, this enhancement of precision can be interpreted as a decrease in the resources required for representing the same number of stimuli, or an effective increase in capacity. Though this tendency is inconsistent with a system comprised of slots, where each ‘slot’ provides a discrete high-fidelity representation of a single item regardless of the item’s complexity, it can actually explain some of the aforementioned ‘slot-like’ properties of visual working memory. For example, in a previously mentioned experiment, adding featural complexity to items in a memory array had little effect on the number of items that could be stored (Luck & Vogel, 1997). This observation of little-to-no capacity loss with additional feature complexity could be compelling evidence for ‘siloed’ resources for a fixed number of slots if it is assumed that items and their features are stored in an inefficient way. The relatively small impact on capacity from adding additional feature complexity could, however, be expected if working memory is conceptualized as an efficient system: In an efficient system where a simple object (e.g. a coloured line and its location in space) requires n bits of information to be represented with sufficient fidelity for later recognition (via its colour & position), adding features that must be remembered (e.g. a specific orientation) would not require an additional n bits of information to be stored since the range of spatial locations that the new feature could occupy is constrained (e.g. by using the first feature’s

10 spatial location as an informative prior in a Bayesian sense). With the range of possibilities constrained, features can be represented with fewer bits of information. A number of studies have demonstrated that behavioural data for visual working memory performance is well modelled using an information theory conceptualization of a relatively efficient store (Sims, Jacobs & Knill, 2012; Taylor & Bays, 2018).

It should also be stated that working memory capacity is not a pure measure of the maximum number of information units that is afforded by an individual’s neuropsychological faculties, whether those faculties are implemented in the form of discrete slots or a shared resource pool. A true maximum capacity for information retention is inherently difficult to measure since inevitably some of those resources are misallocated to irrelevant information, so behavioural measures should be considered to reflect capacity mixed with an individual’s resistance to distraction. Model fits to visual working memory performance data can be improved by accounting for error sources like lapses of attention (Rouder et al., 2008; Sims, Jacobs & Knill, 2012), and a person’s ability to reject distracting information can account for significant variability in their working memory capacity scores (Vogel, McCollough & Machizawa, 2005). An arguably more direct measure of an individual’s working memory resistance to interference is the spatial 2-Back task. In this task, a stream of visual stimuli must be compared not simply against the previously seen stimulus, but the stimulus preceding the previously seen stimulus. To perform this task then, participants must retain not only information about the previous two stimuli, but also resist intrusions from the previously seen item on the comparison between the present stimulus and the 2-Back stimulus. This taxes executive control functions, and one’s ability to remember sequences. Bearing this in mind, the 2-Back task can be conceptualized according to Baddeley’s working memory model (Baddeley & Hitch, 1974; Baddeley, 1992; 2003), with the ‘central executive’ managing attentional control, and the ‘visuospatial sketchpad’ storing and manipulating information. Contemporary visual working memory research has moved away from Baddeley’s model (Baddeley, 2003) arguably because its visual components are loosely defined, but the concept of subdivisions within working memory has been borne out using data-driven approaches. More recent research has, for instance, identified visual working memory subcomponents that each contribute unique explanatory power related measures like fluid intelligence (Unsworth et al., 2014). According to the Unsworth model, working memory is well characterized by latent components derived from tasks that capture measures of capacity,

11 attention control, and ‘secondary memory’, which is participants’ ability to exploit resources outside of primary working memory to enhance performance. A given visual working memory task will likely load onto all three components, but when aggregated together with tasks measuring similar abilities, each component comes to explain a pattern of variance distinct from the other two.

Visual Working Memory VS. Transsaccadic Working Memory

The contemporary line of visuospatial working memory research stems from the study of transsaccadic change blindness (Grimes, 1996; Luck & Vogel, 1997; Luck & Vogel, 2013). Transsaccadic visual working memory is known to have limitations similar to those observed in visual working memory (e.g. Irwin, 1991; Prime, Tsotsos, Keith, and Crawford, 2007), and given the similarity of their roles, it seems reasonable to assume that transsaccadic memory is the same as visual working memory.

That said, the similarities could be coincidental: Transsaccadic memory might use neural resources that are rather separate from those of visual working memory – especially because transsaccadic memory faces unique obstacles. The planning and execution of eye movements could, for instance, interfere with stored information. When eye movements are made, information in spatial memory is translated according to its retinotopic coordinates, and items crossing the visual midline are transferred across cortical hemispheres (Merriam, Genovese, and Colby, 2003; 2007). Error can accumulate during this process, as observed in patients suffering from left hemispatial neglect: Memory performance was found to be poor when items were remapped into or within the impaired left hemifield, even when the items were remapped back into the right hemifield before recall (Vuilleumier et al., 2007). Further, remapping in general can incur significant costs to visual working memory performance relative to a no-saccade baseline (Vasquez and Danckert, 2008; Ten Brink et al., 2019). Transsaccadic memory also depends on some cortical structures differently than general visuospatial working memory: In healthy people, transcranial magnetic stimulation of posterior parietal cortex (Prime, Vesia & Crawford, 2008) or frontal eye fields (Prime, Vesia, and Crawford, 2010) was found to be disproportionately disruptive to visual working memory when a saccade is made before retrieval, suggesting that these areas are important specifically for the spatial updating components of transsaccadic memory.

It is clear that some saccade-specific processes are involved in the maintenance of transsaccadic memory, but it is not clear whether or not transsaccadic memory is just visuospatial working memory plus remapping. The remapping processes described in this chapter conceptualize remapping largely as translations of specific spatial coordinates fuelled by non-cognitive, sensorimotor processes with potentially large computational capacities (Medendorp et al., 2003). On the other hand, with visuospatial working memory, recent findings suggest that it is a fluid resource distributed across stimuli with variable precision depending on the number of stimuli to be stored (Bays, Catalo & Husain, 2009; Bays & Husain, 2008) and the characteristics of the set (Brady & Alvarez, 2015; Sims, Jacobs & Knill, 2012; Sun et al., 2017). In this framework, working memory performance is the product of group-level representations alongside individual spatiotopic representations, which provides a more nuanced account of the underpinnings that support visual working memory, and possibly transsaccadic memory. To date though, there is a scarcity of direct comparisons between ability on transsaccadic memory tasks and general visuospatial working memory tasks, so whether or not the two faculties share the same underpinnings is an open question, and the subject of this thesis.

Present Research

Therefore, across six experiments collected into three chapters, I test hypotheses about the systems that support transsaccadic memory and general visuospatial working memory. The questions tested are the following:

(1) Is transsaccadic memory a distinct system unto itself, independent from general visuospatial working memory? or does it share substantial underpinnings with general visuospatial working memory? In Chapter 2, I developed a transsaccadic memory task similar to a change detection task (e.g., Vogel & Luck, 1997) that measures systematic and unsystematic spatial errors. I tested whether these measures are sensitive to working memory load, and I probed whether the measures correlate with established tests of working memory that require no eye movements, i.e. a visuo-spatial version of the 2-Back task and the change detection task. I observed that several measures of the transsaccadic memory task, especially unsystematic errors, correlated with the 2- Back task. Surprisingly however, they did not correlate with the change detection task. This could indicate that transsaccadic memory is distinct from core working memory processes as measured with the change detection task. On the other hand, it was also possible that this

13 dissociation comes from surface features of the particular change detection task as used in Chapter 2, given that it did not correlate with the 2-Back task either. Therefore, I proceeded to determining task features that might account for dissociations between the tasks.

(2) Are there task features that are capable of establishing commonalities between the different working memory tasks? Or are the tasks fundamentally different from one another? In Chapter 3, I tested whether commonalities between the working memory tasks could be established by disrupting visual or cognitive processes, or by disrupting spatial or non-spatial context. To test these possibilities efficiently I first focused on the change detection task and the 2-Back task. Neither disrupting visual nor cognitive processes in the change detection task produced correlations with the 2-Back task. Disrupting non-spatial context also yielded no significant correlations. However, removing spatial contextual information from the change detection task (called CD-Single) revealed a predictive relationship with 2-Back. This shared variance suggests common underpinnings and demonstrates that change detection and 2-Back are not fundamentally different working memory tasks. It remained to be seen whether this newfound predictive relationship would extend to the transsaccadic memory task.

(3) Does transsaccadic memory share a single source of variance with fixational working memory tasks, or is variance shared along multiple dimensions? Chapter 4 addressed this question by comparing transsaccadic memory performance with the CD-Single and 2-Back tasks. In addition, a no-eye movement version of the transsaccadic task served as a control task. A principal component analysis identified two components. A first component related to all fixational working memory tasks. Given its explanatory power across tasks and its correlation with CD-Single it suggests that working memory capacity is an important determinant of transsaccadic memory. A second component only predicted performance for 2-Back. This shows that the eye movement component recruits an additional facet of working memory in a manner similar to 2-Back, perhaps reflecting dynamic working memory processes or attention control. Together, the results from these experiments help clarify how transsaccadic memory relates to visuospatial working memory, how visuospatial working memory is subdivided, and through this lens provides insights about how the brain constructs the experience of vision.

Chapter 2 Working memory in action: Inspecting the systematic and unsystematic errors of spatial memory across saccades

- Adapted from Frost et al., 2019

Abstract

Our ability to interact with the world depends on memory buffers that flexibly store and process information for short periods of time. Current working memory research, however, mainly uses tasks that avoid eye movements whereas in daily life we need to remember information across saccades. Because saccades disrupt perception and attention the brain might use special transsaccadic memory systems. Therefore, to compare working memory systems between and across saccades the current study devised transsaccadic memory tasks that evaluated the influence of memory load on several kinds of systematic and unsystematic spatial errors, and it tested whether these measures predicted performance in more established working memory paradigms. Experiment 1 used a line-intersection task that had people integrate lines shown before and after saccades, and it administered a 2-Back task. Experiments 2 and 3 asked people to point at one of several locations within a memory array flashed before an eye movement, and we tested change detection and 2-Back performance. We found that unsystematic transsaccadic errors increased with memory load, and they were correlated with 2-Back performance. Systematic errors produced similar results although effects varied as a function of the geometric layout of the memory arrays. Surprisingly, transsaccadic errors did not predict change detection performance despite the latter being a widely accepted measure of working memory capacity. Our results suggest that working memory systems between and across saccades share, in part, similar neural resources. Nevertheless, our data highlight the importance of investigating working memory across saccades.

Introduction

We make three or four saccadic eye movements per second to take highly resolved foveal snapshots of the world, and we use transsaccadic integration (Niemeier et al. 2003; 2007;

14 15

Deubel, Bridgeman and Schneider, 1998) and remapping (Henriques et al. 1998; Medendorp, Tweed, and Crawford, 2003) to combine information across saccades. Yet the foveal information quickly fades and cannot be joined into rich, ‘photographically’ detailed images of our visual surround – contrary to what subjective impressions might suggest. For example, we are surprisingly poor at spotting dramatic changes to scenes if the changes coincide with saccades or other brief visual interruptions and transients (e.g. Grimes, 1996; Phillips, 1974; Simons and Levin, 1998; Simons and Rensink, 2005). That is, we retain only a small fraction of the available visual information across eye movements.

Our limited ability to retain transsaccadic information resembles visual working memory, with its ability to store information for about four items of a given set over a span of seconds (e.g. Luck and Vogel, 1997). Trying to remember more information places increased demands on our working memory’s limited resources such that whole items are dropped from memory (Luck and Vogel, 1997), or their representation becomes gradually less precise (Bays, Catalao, and Husain, 2009).

Transsaccadic visual working memory is now known to have limitations similar to those observed in visual working memory (e.g. Irwin, 1991; Prime, Tsotsos, Keith, and Crawford, 2007). Thus, it seems reasonable to assume that transsaccadic memory is the same as visual working memory.

However, the similarities could be coincidental. Transsaccadic memory (TSM) might use neural resources that are rather separate from those of visual working memory – especially because TSM faces challenges that are rather different from those memory functions that are more commonly probed in visual working memory paradigms.

One obvious difference is that in usual visual working memory tests, participants keep their eyes steady at a fixation point, whereas transsaccadic tasks require the eyes to move. So, the intervening planning and execution of eye movements could affect stored information. When lateral eye movements are made, information in spatial memory is translated according to its retinotopic coordinates, and items crossing the visual midline are transferred across cortical hemispheres (Merriam, Genovese, and Colby, 2003; 2007). Vuilleumier et al., (2007) reported that patients suffering from left hemispatial neglect showed deficits in memory performance when items were remapped into or within the impaired left hemifield, even when the items were

16 remapped back into the right hemifield before recall. Transcranial magnetic stimulation of posterior parietal cortex (Prime, Crawford, and Vesia, 2008) or frontal eye fields (Prime, Vesia, and Crawford, 2010) is disproportionately disruptive to visual working memory when a saccade is made before recall, likely reflecting that functions related to spatial remapping are disrupted. Further, remapping in general can incur significant costs to visual working memory performance relative to a no-saccade baseline (Vasquez and Danckert, 2008; Brink, Nijboer, Fabius, and Stigchel, 2019).

The saccade-related drop in recall could reflect drops in accuracy or precision. In other words, they could reflect systematic or unsystematic memory errors resulting from representational noise that accumulates as memory buffers compete with attentional and oculomotor processes, or errors could come with an imperfect remapping process. Further, saccades could interfere more with greater memory loads.

In the present study, we addressed two central questions: First, how is the integrity of spatial information held in TSM affected when larger compared to smaller set sizes are consuming more or less memory resources? That is, do participants commit more unsystematic errors in their responses, or are there systematic effects that change the topography of TSM representations? Second, to what degree does TSM share underpinnings with visual working memory as captured in classic working memory paradigms?

Experiment 1

2.3.1.1 Participants

Twenty-six healthy undergraduate students (19 females with a median age of 18 years) gave their informed and written consent to participate in the experiment. All participants had normal or corrected-to-normal vision. All procedures were approved by the Human Participants Review Sub-Committee of the University of Toronto and were performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.

2.3.1.2 Apparatus

Participants performed the experiments in a dark room with their heads stabilized by a chin rest. Stimuli were presented on an LCD monitor centered at eye level that subtended 57.5° of visual angle (115cm wide at an eye-screen distance of 107cm). Stimuli were generated and presented using Matlab (MathWorks) and the Psychophysics and EyeLink Toolbox extensions (Brainard, 1997; Kleiner et. al, 2007; Pelli, 1997; our experimental code for this and the other two experiments is available upon request). One task in this experiment required gaze tracking, and for it participants were fitted with a head-mounted eye tracker which sampled the position of the left eye at 500 Hz (EyeLink II, SR Research, Ottawa).

2.3.1.3 Procedure

Transsaccadic memory task. The TSM task constitutes an extension of a previous transsaccadic integration task (Prime et al., 2006). Trials of the task began with fixation on a point (0.15° across) located at the vertical screen centre and 10° to the left or right of screen centre. Participants fixated this point, then clicked a computer mouse. 900ms to 1100ms later, a memory display of either 1 or 2 coloured lines (any pair of red, green or blue, 1 pixel across) was flashed on screen for 400ms. The lines always continued through to the edges of the display. They were positioned and angled so that they intersected both the fixation point and 1 of 9 points in a virtual 3-by-3 square grid (10° wide) centred horizontally between the initial fixation point and the saccade target, and vertically 8° above the fixation point (see Fig. 2.1). Immediately after the offset of the memory display, a mask showed 40 randomly distributed and oriented lines of the same widths and colours as those in the memory array for 200ms. Five-hundred milliseconds after the offset of the mask, a new fixation point appeared 20° to the opposite side of the screen

18 relative to the first fixation point and participants were asked to move their eyes to it. After 900 to 1100 ms a new angled line appeared that matched the colour of one of the previously presented lines. Crucially, the postsaccadic line was placed so that it crossed the postsaccadic fixation point as well as the same intersection point in the virtual 3-by-3 square grid that had been crossed earlier by the initial memory line of the same colour. Four hundred milliseconds later a second mask (same parameters as the first mask) followed, after which a cursor was placed at the fixation point. Participants used the cursor to indicate the position where the recall line would have intersected the colour-matched memory line, had they appeared on the screen at the same time. A mouse click at the estimated intersection location concluded the trial. Blocks were comprised of 126 trials, and the task was comprised of 3 blocks for a total of 378.

Data from all blocks were screened for eye movement errors (eye movements during stimulus presentation, refixations further than 2° away from the target and/or trials with blinks - an average of 79%, SD = 16%, of trials per participant were included in the final analysis). The remaining data were used to calculate systematic and unsystematic transsaccadic errors as reflected in the mouse clicks. To this end we collected the horizontal and vertical coordinates of all clicks for each working memory load level and saccade direction separately. As shown in Fig. 2 this created nine clusters of mouse clicks, each pertaining to one of the nine line intersection locations. From the clusters we first removed outliers. To identify outliers, for each participant we calculated a ‘global’ standard deviation of the distance of each of their responses from its respective intersection location. Then, each response that deviated by more than 1.96 ‘global’ standard deviations from its respective cluster was labelled as an outlier and removed from subsequent analyses (0.49% of responses on average). Note that it is possible that some of the outliers reflected working memory errors. For example, participants might have forgotten the true intersection location and simply guessed. Other outliers might have been due to confusion errors where participants aimed for an intersection location other than the correct one. However, a third kind of outliers were probably due to simple motor errors where participants pressed the mouse key before moving the mouse to the intended location. With the current paradigm we were unable to tease apart memory and motor errors.

Next we recalculated the horizontal and vertical standard deviations as measures of unsystematic error. To calculate systematic errors we recalculated the average coordinates of the nine clusters of mouse clicks and submitted the data to a Procrustes analysis to obtain quantitative measures of

19 all linear transformations required to overlay the cluster locations with the 3-by-3 grid of the original line intersections. That is, we obtained measures of how much the nine clusters were systematically shifted horizontally and vertically, how much they were rotated, and how much they differed in scale relative to the original grid. As a fifth output, Procrustes analysis provided a measure of nonlinear distortion.

2-Back line orientation task. Participants’ working memory ability was assessed using a 2-Back task. Here participants were presented with a series of white lines (3° in length, 0.06° wide) on a grey background, and they compared the angle line on screen with the angle of the line presented prior to the previously presented one (see Fig. 2.1). Participants fixated a central fixation point (0.2° across) for the entirety of the task, and the stimulus lines were each presented, extending from the fixation point outwards, for 400ms. The angles of the lines were one of four possibilities, either 55°, 90°, 125°, or 160°. The probability of a match trial was 1/6. The angled line was followed immediately by a mask comprised of 1000 lines placed at random across the screen, which was displayed for 200ms, and followed by a blank screen and central fixation point. One second later, the next trial began. If the participant believed that the present line was a match, they responded with a key press on a computer keyboard. Blocks were comprised of 62 trials, and participants completed 3 blocks. Hit rates and false alarm rates were used to calculate d’ values as measures of sensitivity free of any response bias.

A B

Fixation - 900-1100ms Fixation - 1000ms

Memory array - 400ms Memory item - 400ms

Mask - 200ms Mask - 200ms

Fixation - 500ms Fixation - 1000ms

Saccade - 900-1100ms Memory item - 400ms

Memory Probe - 400ms

Mask - 200ms

Response

Figure 2.1 Experimental Paradigm. A. TSM task. B. 2-Back line orientation task

2.3.2 Results and Discussion

Group averages of systematic and unsystematic errors are visualized in Fig. 2.2 for each memory load condition and saccade direction separately. The figure shows that participant responses were distorted in a centrifugal manner away from the saccade target. Also, responses in the low load condition covered a larger area than those of the high load condition. Finally, close inspection suggested that ellipses in the high load condition were slightly larger than in the low load condition. To quantify systematic errors we converted data, we flipped the leftward saccade data horizontally to register coordinates relative to the saccade, and submitted them to Procrustes analysis (see Methods). Unsystematic errors were calculated using standard deviations (Fig. 2.3).

A Leftward B Rightward

A B C D 3 3 0.5 1 Low Load Low Load Low Load Low Load High Load High Load High Load High Load 0.45 0.9

2.5 2.5 0.4 0.8

0.35 0.7 2 2 ) ) D D g n S S 0.3 0.6 i ( ( l

D a

X Y c

s c c e S i i

t t t s s a 1.5 a 1.5 0.25 0.5 u e t r m m s c e e t t u o r r s s c y y P o s s 0.2 r 0.4 n n P U U 1 1 0.15 0.3

0.1 0.2 0.5 0.5

0.05 0.1

0 0 0 0 Leftward Rightward Leftward Rightward Leftward Rightward Leftward Rightward

E F G 3.5 3.5 0.5 Low Load Low Load Low Load High Load High Load High Load 0.45 3 3

0.4

2.5 2.5 0.35 n o

i 0.3 t a

2 2 t o X Y t t R e e s

s s 0.25 e f f f f t s O O u r

1.5 1.5 c o

r 0.2 P

0.15 1 1

0.1

0.5 0.5 0.05

0 0 0 Leftward Rightward Leftward Rightward Leftward Rightward

To test which of these measures of TSM performance reflect a working memory-like sensitivity to load, we conducted a series of seven 2-way repeated measures ANOVAs with Memory load (one or two lines) as a first factor. As a second factor we included Saccade direction (left vs. right) (Tab. 2.1). We found that an increase in memory load resulted in decreased performance with respect to unsystematic error (both horizontal and vertical), Procrustes D and scaling (ps < 0.005). Although the centrifugal distortions resulted in large values of horizontal and vertical offset opposite to the saccade as well as large Procrustes rotation (clockwise and counter- clockwise after rightward and leftward saccades, respectively), these measures were not

23 significantly influenced by load (ps > 0.1). Also, the tests showed no significant effect of saccade direction on any of the dependent variables, neither in terms of its main effects (p’s > 0.05) nor its interactions with load (ps > 0.05).

Table 2.1 2-Way Repeated Measures ANOVAs for the measures of transsaccadic error

Load Saccade Load X Saccade

2 2 2 Error F(1,25) p ηp F(1,25) p ηp F(1,25) p ηp

Unsystematic X 10.20* 0.004 0.29 4.17 0.052 0.14 1.47 0.237 0.06

Unsystematic Y 9.98* 0.004 0.29 0.22 0.642 0.01 1.73 0.200 0.07

D 31.90*** <0.001 0.56 0.06 0.807 0.00 3.13 0.089 0.11

Scale 19.60*** <0.001 0.44 0.65 0.427 0.03 0.31 0.584 0.01

Offset X 0.03 0.853 0.00 1.83 0.188 0.07 0.58 0.452 0.02

Offset Y 0.57 0.457 0.02 1.73 0.200 0.07 0.22 0.647 0.01

Rotation 0.94 0.341 0.04 4.09 0.054 0.14 0.54 0.470 0.02

Note. Unsystematic error is comprised of the mean horizontal and vertical standard deviation. Procrustes D, scaling, offset and rotation are all derived from Procrustes analysis (without reflection). Greenhouse-Geisser corrections produced identical values for all ANOVAS. Asterisks denote significance after serial Bonferroni corrections for multiple comparisons. * p < 0.05, ** p < 0.01 *** p < 0.001.

Subsequently, to test whether the TSM task captures performance features comparable to those reflected in classic working memory tasks without eye movements, we conducted a multiple regression analysis with stepwise inclusion of the TSM error variables (averaged across both saccade directions and loads) to predict participants’ performance in the 2-Back task (average d- prime = 1.13, SD = 0.81). The resultant model included only horizontal unsystematic error as a significant predictor for performance in the 2-Back task (greater unsystematic error associated with poorer 2-Back performance, β = -0.58, F(1, 24) = 12.11, p = 0.002), with an R2 of .335. In addition, we calculated individual correlations that revealed that the TSM error measures that were significantly affected by load, also significantly predicted performance on the 2-Back task (two-tailed bootstrapped 95% confidence intervals did not include zero) in the transsaccadic task (see Tab. 2.2).

Table 2.2 Correlations between 2-Back performance and measures of transsaccadic error

CI(95%) CI Corrected

Error r p Lower Upper Lower Upper

Unsystematic X -0.58* 0.002 -0.84 -0.23 -0.89 -0.07

Unsystematic Y -0.39 0.051 -0.66 -0.10 -0.73 0.02

D -0.51* 0.008 -0.73 -0.24 -0.80 -0.11

Scaling 0.48* 0.014 0.20 0.70 0.08 0.76

Offset X 0.07 0.719 -0.26 0.38

Offset Y 0.04 0.828 -0.29 0.40

Rotation -0.32 0.112 -0.71 0.05

Note. Median r and 95% confidence intervals derived from bootstrapped regression analysis (100000 resampling iterations). * zero falls outside of the Bonferroni-corrected 95% confidence interval.

TSM Unsystematic X vs 2-Back TSM Unsystematic Y vs 2-Back TSM Procrustes D vs 2-Back TSM Procrustes Scaling vs 2-Back 4 4 1 1.5

) 3.5 ) 3.5 D D 0.8 g n S S i ( ( l

D a

X Y c

3 3 s 1 c c S e i i

t t t

0.6 s s a a e u t r m m s c e 2.5 e 2.5 t t u o r s s r c y y P o s s 0.4 r n n M P

U 2 U 2 0.5 S

T M M M S S S T

T T 0.2 1.5 1.5

1 1 0 0 -1 0 1 2 3 4 -1 0 1 2 3 4 -1 0 1 2 3 4 -1 0 1 2 3 4 2-Back (d') 2-Back (d') 2-Back (d') 2-Back (d')

TSM Offset X vs 2-Back TSM Offset Y vs 2-Back TSM Procrustes Rotation vs 2-Back 6 6 0.8

5 5 0.6 n o i

4 4 t

a 0.4 t o X Y

R t 3 t 3

e e s

s 0.2 s e f f t f f

2 2 s O O u

r c

M 0 M o S S 1 1 r T T P -0.2 0 0 M S T -0.4 -1 -1

-2 -2 -0.6 -1 0 1 2 3 4 -1 0 1 2 3 4 -1 0 1 2 3 4 2-Back (d') 2-Back (d') 2-Back (d')

Figure 2.4 Results of Experiment 1: Correlations between 2-Back performance and measures of transsaccadic error

In sum, Experiment 1 indicates that greater memory load increased unsystematic errors as well as contracted and nonlinearly distorted topography in TSM. Further, the same measures, except for vertical unsystematic errors, correlated with the 2-Back test, suggesting that TSM relies on similar cognitive resources as spatial working memory. However, these results come with several limitations. First, the TSM task as used here requires transsaccadic integration of one presaccadic and one postsaccadic line, and, so, it probably measures more than purely transsaccadic memory functions. Second, the influence of transsaccadic memory (and integration) could have been lessened if participants had pursued a certain “line search strategy” to respond; they could have used the postsaccadic fixation point as a reference and then travelled with the mouse cursor along the remembered postsaccadic line, more or less so depending on how steep the presaccadic line had been. Indeed, we noticed that, except for the top row of responses, systematic errors tended to be biased approximately along an imaginary line extending from the postsaccadic fixation point beyond the correct intersection location (Prime et al., 2006 for a similar observation). In other words, it is difficult to judge the extent to which participants memorized categorical information rather than spatial information across saccades. Furthermore, the “line searches” likely skewed systematic errors. Third, the 2-Back task, although commonly used as a measure of working memory, has been found to correlate poorly with some other measures of working memory (Kane et al., 2007; Jaeggi et al., 2010) and appears to represent a less common facet of working memory-related executive function. Therefore, we devised a second experiment in which we simplified the TSM task and also replaced the 2-Back task with a change detection paradigm that is well accepted as a measure of working memory capacity (Vogel, McCollough, and Machizawa, 2005).

Experiment 2

2.4.1 Participants

Twenty-two healthy undergraduate students (13 females with a median age of 19 years) gave their written and informed consent to participate in the experiment. All participants had normal or corrected-to-normal vision. All procedures were approved by the Human Participants Review Sub-Committee of the University of Toronto and were performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.

2.4.2 Apparatus

Participants were tested using the same set-up described in Experiment 1.

2.4.3 Procedure

Transsaccadic memory task. This task (Fig. 2.5) was structurally similar to that in Experiment 1 (Fig. 2.1) with fixation point and saccade target set apart by 20° horizontally and the same virtual 3-by-3 grid of possible memory locations positioned between and 8° above them. However, all stimuli were shifted down by 5° to leave more space for responses in the upper part of the screen, and we jittered positions horizontally so that from trial to trial all stimuli might appear 5.75° further to the left or right, or anything in between. Also, gaze position was processed online. During a trial, if the participant’s gaze deviated by 2° from the initial fixation point or 3° from the target fixation point after saccade, the participant was alerted with a beep tone and blue flash, and the trial was recycled. Crucially, to avoid any “line searches” the presaccadic memory arrays showed one or three squares (red, green and blue, 0.6° across) that marked positions randomly selected from the 3-by-3 grid. After the saccade, one of the squares would re-appear at a random position within the 3-by-3 grid, and participants used the computer mouse to try to restore its correct location. In addition, we used masks that contained 1500 randomly distributed squares of the same dimensions and colours as those in the memory array. Blocks were comprised of 112 trials, and there were 3 blocks in total. We extracted the same systematic and unsystematic errors as dependent variables as in Experiment 1.

Change detection task. We adopted the paradigm from Vogel and colleagues (Vogel, McCollough, and Machizawa, 2005; Fig. 2.5). Participants fixated centrally on a white fixation point (0.2° across) on a grey background, and a set of 2, 3 or 4 uniquely coloured (red, orange, green, blue, purple) squares (0.65° in width) was presented either to the left or to the right of fixation. The squares were assigned random locations within a rectangular area (4° wide, 7.3° high), centred 5° away from fixation on the horizontal axis, with a minimum spacing of 1.7° between squares. The memory array remained on screen for 100ms, followed by a blank display (save for the fixation point) for 900ms, after which the squares reappeared at their original spatial locations. However, on 50% of trials, one square’s colour was changed. Participant used the “d” and “s” key on a computer keyboard to indicate whether they believed the colours to be different or the same, respectively. The recall array remained on screen for 1.5s or until a response was

28 given, and then the next trial began after a 1.5s inter-trial interval. Participants completed two blocks of 150 trials (one trial was lost due to a programming glitch). Based on the data we calculated d-prime values (data from set sizes 3 and 4 merged together), just like for the 2-Back task in Experiment 1.

A B

Fixation - 900-1100ms Fixation - 1500ms

Memory array - 400ms Memory array - 100ms

s Mask - 900ms

00ms Recall array - 1500ms

Saccade - 900-1100ms

00ms

Response

Figure 2.5 Experimental Paradigm. A. Transsaccadic memory task. B. Change detection task

2.4.4 Results

Group averages of systematic and unsystematic errors for each memory load condition and saccade direction separately are visualized in Fig. 2.6. This shows that participant responses were much less distorted than in Experiment 1, consistent with our intention to dissuade participants from using a line search strategy during their responses. Low load data formed a slightly skewed parallelogram. High load data formed a smaller, trapezoid that was slightly rotated clockwise after leftward saccades and counter-clockwise for rightward saccades. Also, ellipses were noticeably larger in the high load condition than in the low load condition.

A Leftward B Rightward

A B C D 3 3 0.1 1 Low Load Low Load Low Load Low Load High Load High Load High Load High Load 0.09 0.9

2.5 2.5 0.08 0.8

0.07 0.7 2 2 ) ) D D g n S S 0.06 0.6 i ( ( l

D a

X Y c s c c e S i i

t t t s s a 1.5 a 1.5 0.05 0.5 u e t r m m s c e e t t u o r r s s c y y P o s s 0.04 r 0.4 n n P U U 1 1 0.03 0.3

0.02 0.2 0.5 0.5

0.01 0.1

0 0 0 0 Leftward Rightward Leftward Rightward Leftward Rightward Leftward Rightward

E F G

2 0 0.05 Low Load Low Load High Load High Load 1.8 -0.1 0.04

1.6 -0.2 0.03

1.4 -0.3 0.02 n o

1.2 -0.4 i 0.01 t a t o X Y t t R e e s

s 1 s -0.5 0 f f e t f f s O O u r c o

0.8 -0.6 r -0.01 P

0.6 -0.7 -0.02

0.4 -0.8 -0.03

0.2 -0.9 -0.04 Low Load High Load 0 -1 -0.05 Leftward Rightward Leftward Rightward Leftward Rightward

Just like in Experiment 1 we converted the data relative to saccade direction and tested the influence of memory load and saccade direction on the systematic and unsystematic errors in a series of seven 2-way repeated measures ANOVAs (Tab. 2.3). Again we found that higher load resulted in greater unsystematic error. Some measures of systematic error were affected as well: scaling again declined with higher load (p < 0.05). Rotation now showed a significant load- dependent effect as well (p < 0.001). As with Experiment 1, there was no significant effect of load on systematic offset (horizontal and vertical) (ps > 0.5), but unlike before, Procrustes D now also showed no effect of load (p = 0.72). Also consistent with Experiment 1, the tests produced

32 no significant effect of saccade direction on any of the dependent variables, neither in terms of its main effects (ps > 0.05) nor its interactions with load (ps > 0.05).

Table 2.3 2-Way Repeated Measures ANOVAs for the measures of transsaccadic error

Load Saccade Load X Saccade

2 2 2 Error F(1,21) p ηp F(1,21) p ηp F(1,21) p ηp

Unsystematic X 21.02*** <0.001 0.50 5.91* 0.024 0.22 2.97 0.099 0.12

Unsystematic Y 41.90*** <0.001 0.67 0.24 0.627 0.01 0.07 0.793 0.00

D 0.14 0.716 0.01 2.24 0.150 0.10 1.59 0.221 0.07

Scale 8.77* 0.007 0.30 0.81 0.380 0.04 0.55 0.467 0.03

Offset X 4.87 0.039 0.19 0.48 0.496 0.02 1.00 0.330 0.05

Offset Y 1.15 0.296 0.05 0.75 0.397 0.03 0.21 0.650 0.01

Rotation 45.78*** <0.001 0.69 0.29 0.596 0.01 0.46 0.505 0.02

Next, we conducted a multiple regression analysis with stepwise inclusion of the TSM error variables (averaged across both saccade directions and loads) to predict participant performance in the change detection task (mean d’ = 2.72, SD = 0.71). However, none of the TSM error measures were significant predictors. Of the individual correlations (Tab. 2.4) only vertical translation showed a trend but failed to reach significance when corrected for multiple comparisons. In addition, it is of note that the bootstrapped confidence intervals were often very large, with a span width of r values exceeding 1 in four out of the seven regressions. This could suggest that a small number of influential data points concealed true relationships present within the majority of the data. To rule out this possibility, we computed robust regressions (Maronna, Martin and Yohai, 2006), which reweights data points such that individual cases do not have disproportionate influence over the regression model. Nevertheless, the resulting correlations were equivalent-to or smaller-than those that resulted from the initial regression tests. Finally, we found that converting change detection performance into measures of Kowan’s k (mean k = 2.75 SD = 0.47) produced no correlations either.

In sum, once again the TSM task showed load effects on unsystematic errors and on scaling. Unlike in Experiment 1, here we also observed a load effect on rotation but none on Procrustes D, likely due to differences in the format of the two transsaccadic tests. Further, the correlational analyses of the current experiment suggest that TSM bears little functional relevance to working memory as measured in the change detection task. This is surprising given that change detection is a well-researched and robust measure of visual working memory (Cohen et al., 1994; Courtney et al., 1998; Gevins et al., 1990; Gevins & Cutillo, 1993, Smith, Jonides & Koeppe, 1996). However, in this experiment we also used a novel TSM task, so to confirm that the new TSM task truly shows no correlation with change detection and, further, to test whether the new task correlates with the 2-Back task, in Experiment 3 we tested both 2-Back and change detection in conjunction with TSM.

Table 2.4 Correlations between change detection performance and measures of transsaccadic error

CI(95%)

Error r p Lower Upper r robust p robust

Unsystematic X -0.23 0.297 -0.71 0.51 0.14 0.648

Unsystematic Y -0.19 0.389 -0.66 0.45 0.09 0.886

D -0.23 0.297 -0.66 0.52 0.25 0.244

Scaling 0.14 0.522 -0.40 0.62 0.16 0.597

Offset X 0.08 0.717 -0.33 0.45 0.07 0.730

Offset Y -0.36 0.104 -0.68 -0.01 -0.37 0.070

Rotation -0.22 0.325 -0.47 0.03 -0.22 0.109

Note. 95% confidence intervals derived from bootstrapped regression analysis (100,000 resampling iterations). Robust r and p were computed using robust linear regression analysis (Maronna, Martin and Yohai, 2006). * zero falls outside of the Bonferroni-corrected 95% confidence interval.

TSM Unsystematic X vs CD TSM Unsystematic Y vs CD TSM Procrustes D vs CD TSM Procrustes Scaling vs CD 3 3 1 1.5

) 2.5 ) 2.5 D D

0.8 g S S n i ( ( l

D a

X Y c

2 2 s 1 c c S e i i

t t t

0.6 s s a a e u t r m m s c e 1.5 e 1.5 t t u o r s s r c y y P o s s 0.4 r n n M P

U 1 U 1 0.5 S

T M M M S S S 0.2 T T 0.5 T 0.5

0 0 0 0 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 CD (d') CD (d') CD (d') CD (d')

TSM Offset X vs CD TSM Offset Y vs CD TSM Procrustes Rotation vs CD 4 4 0.8

3 3 0.6 n o i

2 2 t

a 0.4 t o X Y

R t 1 t 1

e e s

s 0.2 s e f f t f f

0 0 s O O u

M c 0 M o S S

-1 -1 r T T P -0.2 -2 -2 M S T -0.4 -3 -3

-4 -4 -0.6 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 CD (d') CD (d') CD (d')

Figure 2.8 Results of Experiment 2: Correlations between CD performance and measures of transsaccadic error

Experiment 3

2.5.1 Participants

Twenty-four healthy undergraduate students (15 females with a median age of 21 years) gave their informed and written consent to participate in the experiment. All participants had normal or corrected-to-normal vision. All procedures were approved by the Human Participants Review Sub-Committee of the University of Toronto and were performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.

2.5.2 Apparatus

Participants were tested using the set-up as described in Experiment 1.

2.5.3 Procedure

Transsaccadic memory task. The TSM task for this experiment (Fig. 2.9) was quite similar to that of Experiment 2 (Fig. 2.5). However, we re-designed its display to probe memory in both the upper and lower visual field. Moreover, with the new design participants could no longer anticipate the direction of the saccade and perhaps bias their attention accordingly. To remove all directional cues, we always showed the fixation point in the screen centre (trial by trial with +/- 6.9° horizontal jitter) and the saccade target 15° to the left or right of it with equal probability. Also, we abandoned the virtual 3-by-3 grid of possible positions. Instead, we used a virtual octagon (10° diameter) that was always centred on the fixation point. That is, the (one or three) squares of the memory array would appear on positions randomly sampled from the octagon’s 8 vertices.

Change detection task. We used the same task as in Experiment 2, except we only tested set sizes 3 and 4 and so there were 200 trials in total (one trial was lost due to a programming glitch).

2-Back task. Participants’ working memory, was also assessed using a 2-Back task. To correspond with the change from line stimuli to squares in the TSM task, here participants fixated a central fixation point (0.2° across; white on a middle-grey background) for the entirety of the task, and then monitored a series of white squares (0.6° in width) that appeared, one at a time, for 400ms at one of eight evenly-spaced locations of a circular array (12° in diameter), followed by a 200-ms mask showing 1000 random squares, and followed by a blank screen and

37 central fixation point. 1 second later, the next trial began. Every time a square appeared at the same position as two trials earlier participants were asked to respond with a key press on a computer keyboard. There were 186 trials tested in three blocks.

A B C

Fixation - 900-1100ms Fixation - 1000ms Fixation - 1500ms

Memory array - 400ms Memory item - 400ms Memory array - 100ms

Mask - 200ms Mask - 200ms Mask - 900ms

Fixation - 500ms Fixation - 1000ms Recall array - 1500ms

Saccade - 900-1100ms Memory item - 400ms

Mask - 200ms

Fixation - 900-1100ms

Response

Figure 2.9 Experimental Paradigm. A. Transsaccadic memory task. B. 2-Back task. C. Change detection task

2.5.4 Results

Group averages of systematic and unsystematic errors are visualized in Fig. 2.11 for each memory load condition and saccade direction separately. To confirm and extend the results of Experiments 1 and 2, we again converted the data relative to saccade direction and then tested systematic and unsystematic errors for load-dependent and saccade-direction effects in a series of seven 2-way repeated measures ANOVAs. Once again, unsystematic errors increased significantly with load (corrected p’s < 0.001).

A Leftward B Rightward

A B C D 3 3 0.1 1 Low Load Low Load Low Load Low Load High Load High Load High Load High Load 0.09 0.9

2.5 2.5 0.08 0.8

0.07 0.7 2 2 ) ) D D g n S S 0.06 0.6 i ( ( l

D a

X Y c s c c e S i i

t t t s s a 1.5 a 1.5 0.05 0.5 u e t r m m s c e e t t u o r r s s c y y P o s s 0.04 r 0.4 n n P U U 1 1 0.03 0.3

0.02 0.2 0.5 0.5

0.01 0.1

0 0 0 0 Leftward Rightward Leftward Rightward Leftward Rightward Leftward Rightward

E F G

2 0 0.05 Low Load Low Load High Load High Load 1.8 -0.1 0.04

1.6 -0.2 0.03

1.4 -0.3 0.02 n o

1.2 -0.4 i 0.01 t a t o X Y

R t t e e s

s 1 s -0.5 0 f f e t f f s O O u r c o

0.8 -0.6 r -0.01 P

0.6 -0.7 -0.02

0.4 -0.8 -0.03

0.2 -0.9 -0.04 Low Load High Load 0 -1 -0.05 Leftward Rightward Leftward Rightward Leftward Rightward

In terms of systematic errors, the F-tests for scaling and horizontal offset yielded load effects with p-values of p = 0.035 or smaller, though these failed to reach significance after correcting for multiple comparisons. Procrustes D showed a trend (uncorrected p = 0.027) towards greater distortions with smaller loads that contrasted with Experiment 1 and reflected that high load responses formed less regular octagons with shorter oblique edges compared to horizontal and vertical ones. This effect was not significant after serial Bonferroni correction though. As with

41 the two previous experiments, there was no significant effect of load on vertical offset, nor was there any effect of load on Procrustes rotation. Consistent with both previous experiments, the tests showed no significant effect of saccade direction on any of the dependent variables, neither in terms of its main effects nor its interactions with load (ps > 0.05 after serial Bonferroni correction).

Table 2.5 2-Way Repeated Measures ANOVAs for the measures of transsaccadic error

Load Saccade Load X Saccade

2 2 2 Error F(1,23) p ηp F(1,23) p ηp F(1,23) p ηp

Unsystematic X ***58.92 <0.001 0.72 0.06 0.816 0.00 7.99 0.010 0.26

Unsystematic Y ***34.32 <0.001 0.60 1.03 0.320 0.04 0.05 0.826 0.00

D 5.56 0.027 0.20 0.00 0.979 0.00 0.00 0.962 0.00

Scale 5.01 0.035 0.18 0.38 0.544 0.02 1.11 0.302 0.05

Offset X 5.78 0.025 0.20 4.43 0.046 0.16 3.43 0.077 0.13

Offset Y 0.25 0.619 0.01 1.84 0.188 0.07 3.29 0.083 0.13

Rotation 1.05 0.316 0.04 4.88 0.037 0.18 0.04 0.852 0.00

We also verified the regression findings from the previous two experiments by testing TSM in conjunction with both 2-Back and change detection performance within the same participants. To do so, we first conducted multiple regression analyses with stepwise inclusion of the TSM error variables (averaged across both saccade directions and loads) as predictors of participants’ performance in the 2-Back task and subsequently the change detection task. As was the case in Experiment 1, 2-Back performance (mean d’ = 1.16, SD = 0.96) was predicted by error in the TSM task, with a model comprised of horizontal unsystematic error (R2 = 0.414) as before, and horizontal translation as well (R2 = 0.123) for a whole-model R2 of 0.613 (horizontal systematic error β = -0.71, horizontal translation β = 0.45). As with Experiment 2, none of the TSM error variables significantly predicted performance on the change detection task (mean d’ = 2.40, SD = 0.92). Therefore, once again, it appears that the relationship between TSM and 2-Back is stronger than the relationship between TSM and change detection.

Next we proceeded to examine each of the TSM error variables individually for relationships with performance on the two traditional working memory tasks (Tab. 2.6). We found that 2-Back performance was significantly predicted by TSM unsystematic error (both horizontal and

43 vertical), and there was a trend towards a negative correlation with Procrustes D that, although failing to reach significance after correcting for multiple comparisons, was consistent with our finding in Experiment 1.The correlation reflected that participants with high D values produced octagons that were more irregular than participants with low D values. As for change detection, we found no significant predictors amongst our TSM error variables, which is consistent with our results from Experiment 2. In addition, there was no significant correlation between 2-Back and change detection performance (r = 0.13, 95% CI = [-0.28, 0.50]).

Table 2.6 Correlations between 2-Back performance and measures of transsaccadic error

CI(95%) Corrected CI

TSM Error r p Lower Upper Lower Upper

2 Back Unsystematic X -0.64* 0.001 -0.83 -0.40 -0.87 -0.28

Unsystematic Y -0.51* 0.011 -0.74 -0.25 -0.80 -0.15

D -0.57 0.004 -0.83 -0.13 -0.88 0.06

Scaling 0.28 0.185 -0.15 0.67

Offset X 0.35 0.096 -0.04 0.63

Offset Y 0.24 0.261 -0.11 0.61

Rotation -0.17 0.433 -0.48 0.20

CD Unsystematic X -0.03 0.893 -0.33 0.24

Unsystematic Y -0.21 0.315 -0.60 0.17

D -0.28 0.188 -0.61 0.05

Scaling 0.31 0.145 0.01 0.56 -0.13 0.64

Offset X 0.17 0.417 -0.40 0.56

Offset Y 0.40 0.056 -0.04 0.69

Rotation -0.24 0.251 -0.56 0.12

Note. 95% confidence intervals derived from bootstrapped regression analysis (100,000 resampling iterations). * zero falls outside of the serial Bonferroni corrected confidence interval.

TSM Unsystematic X vs 2-Back TSM Unsystematic Y vs 2-Back TSM Procrustes D vs 2-Back TSM Procrustes Scaling vs 2-Back 3 3 1 1.5

) 2.5 ) 2.5 D D

0.8 g S S n i ( ( l

D a

X Y c

2 2 s 1 c c S e i i

t t t

0.6 s s a a e u t r m m s c e 1.5 e 1.5 t t u o r r s s c y y P o s s 0.4 r n n M P

U 1 U 1 0.5 S

M T M M S S S 0.2 T T 0.5 T 0.5

0 0 0 0 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 2-Back (d') 2-Back (d') 2-Back (d') 2-Back (d')

TSM Offset X vs 2-Back TSM Offset Y vs 2-Back TSM Procrustes Rotation vs 2-Back 4 4 0.8

3 3 0.6 n o i

2 2 t

a 0.4 t o X Y

t R 1 t 1

e e s

s 0.2 s e f f t f f

0 0 s O O u

M c 0 M o S S

-1 -1 r T T P -0.2 -2 -2 M S T -0.4 -3 -3

-4 -4 -0.6 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 2-Back (d') 2-Back (d') 2-Back (d')

Figure 2.12 Results of Experiment 3: Correlations between 2-Back performance and measures of transsaccadic error

TSM Unsystematic X vs CD TSM Unsystematic Y vs CD TSM Procrustes D vs CD TSM Procrustes Scaling vs CD 3 3 1 1.5

) 2.5 ) 2.5 D D

0.8 g n S S i ( ( l

D a

X Y c

2 2 s 1 c c S e i i

t t t

0.6 s s a a e u t r m m s c e 1.5 e 1.5 t t u o r s s r c y y P o s s 0.4 r n n M P

U 1 U 1 0.5 S

T M M M S S S T

T T 0.2 0.5 0.5

0 0 0 0 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 CD (d') CD (d') CD (d') CD (d')

TSM Offset X vs CD TSM Offset Y vs CD TSM Procrustes Rotation vs CD 4 4 0.8

3 3 0.6 n o i

2 2 t

a 0.4 t o X Y

R t 1 t 1

e e s

s 0.2 s e f f t f f

0 0 s O O u

r c

M 0 M o S S -1 -1 r T T P -0.2 -2 -2 M S T -0.4 -3 -3

-4 -4 -0.6 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 CD (d') CD (d') CD (d')

Figure 2.13 Results of Experiment 3: Correlations between CD performance and measures of transsaccadic error

In sum, the results of this third experiment largely replicate the findings from Experiments 1 and 2, and intriguingly suggest a dissociation between 2-Back and TSM on the one hand – and change detection performance on the other hand. We found that greater Procrustes D values predicted poorer 2-Back performance whereas higher load did not increase D values. Thus, in

46 part between-subject and within-subject effects were governed by different mechanisms. The latter probably included perceptual effects. For example, participants might have misperceived memory squares as closer to an oblique position but less so when multiple squares appeared.

General Discussion

The current study asked how transsaccadic memory (TSM) relates to other forms of working memory. This is of significance given the fact that saccadic eye movements are extremely common in daily life while potentially having a substantial impact on memory functions. Yet they are systematically eliminated from standard working memory paradigms. In three experiments, we found that unsystematic TSM errors were sensitive to working memory load. Load also exacerbated certain systematic types of TSM errors, although these errors varied, probably depending on the different spatial layouts of our tests and on people’s response strategies. Crucially, we observed that those TSM errors that were sensitive to load also correlated with the 2-Back task. However, we found no such correlations with the change detection task (CD). Our results show that TSM shares mechanisms at least with certain other types of working memory and that the representations of spatial information maintained across eye movements are impacted by representational noise as well as by imperfect remapping across eye movements.

The fact that TSM correlates with 2-Back but not CD is surprising. Our TSM and CD paradigms were quite similar at the time of encoding, and both were rather different from the serialized presentation schedule of the 2-Back task. It is very likely then that this dissociation can be attributed to differences between CD and TSM that occur after encoding, namely: remapping as one form of information manipulation, interference, and/or the degree to which gestalt-like or allocentric information is available at recall.

Remapping is clearly a potential error-source for TSM in contrast to CD. That is, in the TSM the remembered stimulus location must be translated to its pre-saccadic position in eye-centred coordinates in order to accurately indicate its position on the display. Essentially these translations constitute a form of manipulation of working memory content (e.g., Baddeley, 2003) and so, there is a possible connection to the 2-Back task where manipulation of memory content is required to compare the present stimulus to the stimulus from two trials earlier. It is therefore

47 possible that working memory manipulation is one source of errors, arguably systematic errors, that 2-Back and TSM have in common, but not the CD task.

As a (perhaps related) commonality, TSM and 2-Back, in contrast to the current CD task, include more distracting events that could interfere with working memory. Being able to suppress interference is a part of what is measured with visuospatial working memory tasks (Vogel, McCollough, and Machizawa, 2005; Oberauer and Lin, 2017; Sun et al., 2017). For example, in both the TSM and the 2-Back task visual stimuli appear and must be attended in between encoding and recall. However, other interferences are quite different from one another: TSM requires intervening saccades while 2-Back affords a back-and-forth of supressing and re- activating working memory content, which (if relevant) would be difficult to reconcile with the observed correlations. Finally, the masks in TSM and 2-Back task represent a common visual form of distraction, but then it remains unclear why other visual commonalities between TSM and CD task did not produce correlations.

Another possibility is that the CD task differs from the other two because it provides allocentric information during recall. That is, when the original memory array reappears participants only have to decide whether within the gestalt formed by the squares one has changed its colour (participants could also compare lists of colour names but that is unlikely given the time pressure of the task). In contrast, with the TSM and the 2-Back task only a single item is presented at the time of recall and, thus, little allocentric information remains. Allocentric recall seems to recruit different neural mechanisms than egocentric recall (e.g. Feigenbaum and Morris, 2004). However, this degree of dissociation based only upon the presence or absence of allocentric information would, to our knowledge, be a newly identified phenomenon.

Whichever of these explanations will hold true in the future, with regard to TSM itself, across the present three experiments we found that information is preserved across eye movements in terms of their spatial positions and topography, though this information was sensitive to working memory load and degraded by systematic and unsystematic errors. Contrary to the findings of Vasquez and Danckert (2008), we did not observe any saccade direction-specific working memory performance costs, consistent with a recent report (Brink et al., 2019).

In conclusion, our current study shows that several measures of TSM are sensitive to load and predict performance at least in one other working memory task. Remarkably, we found that TSM

48 does not correlate with the CD task as employed here, although both tasks provide quite similar information at the time of encoding and although working memory capacity correlates with many other cognitive functions (Fukuda et al., 2010; Gold et al., 2010; Johnson et al., 2013; Luck and Vogel, 2013; Unsworth et al., 2014). In contrast, the 2-Back task and TSM have more in common than either one does with CD, even though stimulus presentation in the two tasks is rather different and even though 2-Back is sometimes considered as a specialized and non- general measure of working memory compared to other tasks that preclude any eye movements (Kane et al., 2007; Jaeggi et al., 2010). Our study therefore demonstrates the significance of testing working memory functions across saccades.

Chapter 3 Resolving Dissociations between Working Memory Tasks: The Effects of Disruption and Context Introduction

The n-back task is widely used. It shows series of objects, one at a time, and each time it asks participants to indicate whether an object or one of its features is a repetition from before. Different repetitions matter in different versions of the task. The 1-back version requires participants to compare each current “target” object with that of the immediately preceding trial, often to ensure that participants track objects attentively (e.g., Douglas et al., 2017; Martin et al., 2018; Nemrodov et al., 2016). More complex versions of the task, 2-Back, 3-back, 4-back etc. have participants compare each object with the one that was shown 2, 3, 4 etc. trials earlier, such that objects continuously change in their roles, from targets to distractors that need to be remembered, to comparison objects, and finally to distractors that can be forgotten (Watter, Geffen & Geffen, 2001). Thus, memory functions must be involved in the n-back task, and so, the n-back task is frequently used to probe working memory (e.g. Cohen et al., 1994; Courtney et al., 1998; Gevins et al., 1990; Gevins & Cutillo, 1993, Smith, Jonides & Koeppe, 1996).

However, it is not clear which aspects of working memory function are measured by the n-back task. Although the task correlates with measures of simple memory span (Haatveit et al., 2010; Jaeggi et al., 2010; Roberts & Gibson, 2002) it has been found to correlate only moderately with complex memory span (Jaeggi et al., 2010, Kane et al., 2007). On the other hand, in Chapter 2 (Frost et al., 2019) I have shown that visuo-spatial versions of the n-back task (more specifically the 2-Back task) correlated well with dynamic forms of working memory where the position of one object within an array of objects has to be remembered across a saccadic eye movement.

This transsaccadic memory task, as well as the 2-Back task, were found to be poorly correlated with the change detection task (Frost et al., 2019). The latter task shows arrays of objects before and after a blank screen, and participants have to spot possible changes in the features of the objects. Given that change detection is a well accepted measure of working memory capacity (Alvarez & Cavanagh, 2004; Vogel & Luck, 1997; Vogel, McCollough, and Machizawa, 2005; Unsworth, Fukuda, Awh & Vogel, 2014), a question arises: Do the n-back task and the

50 transsaccadic memory task comprise a group of tasks that recruit memory functions that are entirely separate from those captured by the change detection task (and other tasks), or can variations of the tasks create an overlap in functions?

At first blush, the lack of a correlation, especially between change detection and transsaccadic memory, is puzzling: In both tasks, participants must remember arrays of coloured squares, then after a brief delay, remember information about them. That said, there are differences between change detection on the one hand and the 2-Back and the transsaccadic task on the other that could potentially call upon different neural processes. Differences might arise from visual or cognitive processes that were disrupted in the 2-Back and the transsaccadic task, but not the change detection task. Or they might arise from spatial or non-spatial context that was only useful in the change detection task.

To elaborate, as a first possibility, 2-Back and transsaccadic memory performance might place greater demands on cognitive control and require more active maintenance of working memory’s contents because working memory content has to be transformed in different ways. In the case of transsaccadic memory, remapping must be performed to align the retinotopic positions of memorized objects with the post-saccadic reference frame (Henriques et al., 1998; Merriam, Genovese, and Colby, 2003; 2007; Prime, Niemeier & Crawford, 2006; Sommer & Wurtz, 2002). Closely related to these processes, participants must compute eye movements which engages mechanisms that overlap with working memory (e.g. Ikkai & Curtis, 2011; Postle et al., 2006; Sommer & Wurtz, 2001; Srimal & Curtis, 2008; Theeuwes, Olivers & Chizk, 2005).

Equivalently, the 2-Back task with its stream of objects requires that working memory be continuously updated with new information (Owen et al., 2005; Wager & Smith, 2003; Watter, Geffen & Geffen, 2001) while also preventing old information from interfering. Further, participants must maintain the 1-back object without allowing it to interfere with the comparison between the present object and the object from 2 trials earlier. Finally, the 1-back object might cause spatial attention to reorient (e.g. Jonides, 1981; Lambert, Spencer & Mohindra, 1987; Posner & Cohen, 1984; Yantis & Jonides, 1990), which could also distort working memory representations (Awh, Jonides & Reuter-Lorenz, 1998; Corbetta & Shulman, 2002; Smyth & Scholey, 1994; Van Der Stigchel, Merten, Meeter & Theeuwes, 2007). In contrast, the change detection task entails no comparable forms of spatial transformations, intermediate updating,

51 reorientations of attention away from the remembered stimulus array, or any other type of manipulation of working memory content. Given the possible differences in active maintenance, it is conceivable that the correlations observed between 2-Back and transsaccadic memory (Frost et al, 2019) were due to the tasks both loading onto this facet of working memory ability.

A second possible explanation for the dissociation of change detection performance from the 2- Back and transsaccadic memory tasks is an absence of visual disruptions or distractions between encoding and test. In both the transsaccadic task and the 2-Back task, I used visual masks (Frost et al., 2019). However, the change detection task was modelled after Luck & Vogel’s (1997) version of the task (called ‘standard’ version henceforth) and so, no mask was used. Perhaps then, the working memory faculty common to 2-Back and the transsaccadic task is participants’ ability to protect remembered information from visual disruption or distraction.

A third possible explanation for the change detection dissociation is that the standard version of the task (e.g. Luck & Vogel, 1997) provides spatial context during its test phase. That is, at the end of each trial it returns the entire memory array of objects such that participants can make comparisons based on the gestalt of presented objects. It is known that reinstating spatial (allocentric) context in this manner tends to improve performance (Brady & Alvarez, 2015; Jiang, Olson & Chun, 2000; Sims, Jacobs & Knill, 2012). Neither the transsaccadic task nor the 2-Back task provided spatial context information during their test phases; in both, participants were presented only with a single object that offered minimal information about the encoding context.

The fourth hypothetical explanation for the difference between the three working memory tasks relates to a non-spatial form of context or gist. Specifically, the standard change detection task might allow participants to compute holistic summary statistics of the colours of the displayed objects. Summary statistics for colours and other object features are extracted from arrays of stimuli even when presentation is brief (e.g. Brady & Alvarez, 2011; Maule & Franklin, 2015), and numbers of items largely exceed explicit working memory capacity (Maule & Franklin, 2015; Ward, Bear & Scholl, 2016). This could help in the standard version of the change detection task because the task does not demand that a participant know precisely where a change occurred in a display - rather, a vague sense of average colour similarity or incongruence between memory and test array could aid change detection. On the other hand, neither the

52 transsaccadic memory task nor the 2-Back task allowed participants to exploit summary statistics in this way.

To test which of the above four possible causes (two types of disruptions and two types of useful contexts) contributed to the dissociation between the different working memory tasks, in the current chapter I used the flexibility of the change detection paradigm to administer several versions of the task and to explore their correlations with the 2-Back task. Two separate experiments (to limit total test time) tested whether cognitive or visual interference (Experiment 4) as well as removing spatial or non-spatial context (Experiment 5) created correlations with 2- Back. Any correlation between one of the versions of the change detection task and 2-Back would add to the validity of the 2-Back task as a measure of working memory capacity. Furthermore, any finding of correlations in the current chapter would then be carried forward for a follow-up experiment to be tested against the transsaccadic task in the next chapter.

Experiment 4: Cognitive vs. Perceptual Interference

3.2.1 Method

3.2.1.1 Participants

Frost et al. (2019) observed correlations of r = 0.51 or larger for 22 to 26 participants. Therefore, in the current study, to reliably detect correlations above r = 0.45 (with a power of 0.8) I would need to test at least 29 participants. Thus, thirty healthy students (18 females) with a median age of 19 years (range from 18 to 32) participated in the experiment, and all participants had normal or corrected-to-normal vision. All procedures were approved by the Human Participants Review Sub-Committee of the University of Toronto and were performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.

3.2.1.2 Apparatus and Procedure

In this experiment I measured participants’ performance on the 2-Back task along with different versions of the change detection task where I disrupted either visual or cognitive processes. Participants sat in a dark room with their heads stabilized by a chin rest. Stimuli were presented on an LCD monitor centered at eye level that subtended 66 degrees of visual angle (115 cm wide at a distance of 89 cm). Stimuli were generated using Matlab (MathWorks) and the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997).

3.2.1.3 2-Back

Participants fixated a central point (0.2 degrees across) on a middle-grey background while monitoring a randomized sequence of white stimulus squares (0.6 degrees in width) presented one at a time, at one of eight evenly spaced locations of a circular array (12 degrees in diameter). Each square appeared for 400 ms, followed by a 200-ms mask (1000 similar white squares placed at random across the screen) and a 1-second blank screen before the next stimulus square appeared. Whenever a square’s location matched the location of the square presented two instances earlier (16.7% probability) participants were asked to press a key on a computer keyboard. Responses from 186 trials (tested in 3 blocks) were then converted into d-prime scores (z-transformed hit rate minus z-transformed false alarm rate).

3.2.1.4 Change detection - Standard

A modified version of the original change detection task (Vogel & Luck, 1997) as used in the previous chapter served as a baseline measure. Participants fixated a central point (0.2 degrees across) on a grey background. 1500 milliseconds later a memory array of 3 or 4 uniquely coloured squares (randomly selected from red, orange, green, blue or violet, 0.65 degrees in width) appeared to the left or to the right of fixation (Vogel & Luck, 1997, presented stimulus arrays on both sides and cued participants to disentangle electrophysiological signals due to visual stimulation and cognitive processes; this was not necessary here). The squares were randomly placed at locations on a rectangular array measuring 4 degrees in width by 8 degrees in height (see Fig. 3.1), centred 5 degrees away from fixation on the horizontal axis. The area of possible locations was subdivided into a grid of 5 by 9 positions with 1 degree spacings, and from these locations, the stimulus squares were assigned up to 0.1 degrees of random jitter to minimize any appearance of positional regularity. The memory array remained on screen for 100 ms, and was followed by a blank display (save for the fixation point) for 800 ms. Next, the squares reappeared at their original spatial locations, but with a 50% probability that one square’s colour changed. If the participant believed there was a change, they pressed the ‘up’ arrow key on a computer keyboard, otherwise they pressed the ‘down’ arrow key. The test array remained on screen until a response was given or until a 1.5 s time-out at which point trials were counted as invalid (on average 2.1% of all trials). The next trial began after a 1.5 s inter-trial interval. A total of 199 trials was tested in two blocks (one trial was lost due to a programming glitch).

3.2.1.5 Change detection - Remap

CD-Remap was identical to CD-Standard, except the test array appeared on the horizontally opposite side of where the memory array had appeared. That is, the squares within the array retained their spatial configuration relative to one another, but all occupied new egocentric coordinates within the opposing visual hemifield. This way CD-Remap required participants to transform working memory content by way of spatial remapping. Participants completed 199 trials of this task with a break given mid-way through. An average of 1.5% of all trials were invalid due to time-out.

3.2.1.6 Change detection - Mask

CD-Mask was the same as the CD-Standard task, but incorporated a mask. Specifically, 150 ms after the offset of the memory array, coloured squares of the average spatial density as the memory array appeared for 400 ms across the screen, followed by a blank display (save for the fixation point) lasting 250 ms. Participants completed 199 trials of this task with a break given mid-way through. An average of 1.6% of all trials were invalid due to time-out.

CD-Remap CD-Std CD-Mask 2-Back

Fixation - 1500ms Fixation - 1000ms

Memory array - 100ms Memory item - 400ms Fixation - 150ms

Fixation - 800ms Mask - 400ms Mask - 200ms

Fixation - 250ms Test array - 1500ms Test array - 1500ms Fixation - 1000ms

Memory item - 400ms

Figure 3.1 Experimental paradigm. CD-Remap. Change detection remap task. CD-Std. Standard change detection task. CD-Mask. Change detection mask task. 2-Back. 2-Back task

3.2.2 Results

Before reporting the correlational structure of the different tasks, I will first report descriptive statistics and explore possible effects of the experimental manipulations on task performance. To calculate performance, hit rates and false alarm rates were converted into d’ values for the four tasks. Average d’ values ranged between 1.50 and 2.47 (Fig. 3.2) and were significantly larger than guessing rate (t’s ≥ 13.59, p’s < 0.001). All skewness values were positive but not significantly different from zero (0.370 ≥ z’s ≤ 1.288), inconsistent with ceiling effects, making the planned correlational analyses meaningful.

To compare differences between tasks, I next conducted a one-way repeated-measures ANOVA that yielded a significant influence of task on performance (Greenhouse-Geisser corrected F(1.81, 52.37) = 34.14, p < 0.001). Orthogonal linear contrasts showed that the 2-Back task was significantly more difficult than the three CD tasks (t(29) = 5.26, p < 0.001). Also, CD-Mask was significantly more difficult than CD-Standard and CD-Remap on average (t(29) = 12.16, p < 0.001), suggesting that the mask substantially interfered with change detection performance. CD- Remap was not significantly more difficult than CD-Standard (t(29) = 1.24, p = 0.225).

2.5

1.5

Mean Performance (d') 1

0.5

0 2-Back CD-Std CD-Rmp CD-Mask Task

Figure 3.2 Results of Experiment 4: Working memory performance (d’) means. CD-Std: CD-Standard, CD-Rmp: CD- Remap, CD-Mask: CD-Mask. Error bars indicate standard errors

Inspecting the correlational structure of the data showed that performance in the 2-Back and the CD-Standard task were weak predictors for one another (r = 0.27, Tab. 3.1), consistent with the findings of Experiment 3. The correlations of 2-Back with CD-Mask and CD-Remap were similarly weak (r = 0.29 and r = 0.19, respectively). Even including all of the change detection tasks as predictors for performance on the 2-Back task in a multiple regression model failed to identify a significant relationship (r = 0.342, p = 0.117). In contrast, correlations between the different change detection tasks were strong (r’s ≥ 0.79) with confidence intervals that did not overlap with the intervals of the CD vs. 2-Back correlations.

Table 3.1 Correlations between 2-Back and Change Detection performance

CI(95%) Corrected CI

r p Lower Upper Lower Upper

2-Back vs. CD-Standard 0.27 0.154 0.01 0.48 -0.04 0.51

2-Back vs. CD-Mask 0.29 0.118 0.03 0.54 -0.05 0.59

2-Back vs. CD-Remap 0.19 0.327 -0.07 0.43 -0.07 0.43

CD-Standard vs. CD-Remap 0.79* <0.001 0.61 0.90 0.54 0.92

CD-Standard vs. CD-Mask 0.84* <0.001 0.69 0.92 0.61 0.93

CD-Remap vs. CD-Mask 0.89* <0.001 0.73 0.95 0.66 0.96

Note. 95% confidence intervals derived from bootstrapped regression analysis (100,000 resampling iterations). * significant correlations where zero falls outside of the serial Bonferroni corrected confidence interval.

Finally, to study the relationship between 2-Back and CD tasks more comprehensively I conducted a principal component analysis with preplanned extraction of 2 factors (producing eigenvalues of 2.79 and 0.90, respectively; cumulative explained variance: 69.61% and 92.22%) and Varimax rotation. The solution confirmed the stark difference between 2-Back and CD tasks with very similar loads for all CD tasks (Tab. 3.2 & Fig. 3.3), despite the significant differences in performance. Variability in all CD tasks was almost exclusively described by the first component, whereas the second component only described variability in the 2-Back task. Therefore, the next experiment sought to alter correlations with the 2-Back task using other strategies.

Table 3.2 Rotated component scores

Task Component 1 Component 2

2-Back 0.130 0.991

CD-Standard 0.912 0.159

CD-Remap 0.949 0.050

CD-Mask 0.948 0.172

1 2-Back

0.8

0.6

0.4 Component 2 0.2 CD-Std CD-Mask CD-Remap -0.2 0 0.2 0.4 0.6 0.8 1

-0.2 Component 1

Figure 3.3 Rotated component scores of the principal component analysis

Experiment 5: Spatial vs. Non-spatial Context

3.3.1 Methods

3.3.1.1 Participants

Once again, thirty healthy students (22 females) with a median age of 18.5 years participated (range from 17 to 32) in the experiment, and all participants had normal or corrected-to-normal vision. All procedures were approved by the Human Participants Review Sub-Committee of the University of Toronto and were performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.

3.3.1.2 Apparatus and Procedure

In this experiment I measured participants’ performance on the 2-Back along with different versions of the change detection task where I made spatial or non-spatial context less useful. Participants performed the experiments while seated in a dark room. Stimuli were presented on an LCD monitor centred at eye level that subtended 66 degrees of visual angle (115 cm wide at a distance of 89 cm). Stimuli were generated using Matlab (MathWorks) and the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997).

3.3.1.3 2-Back task

Participants’ n-back performance was assessed using a 2-Back task, and its design was identical to that used in Experiment 4 (see Fig. 3.1 & Fig. 3.4).

3.3.1.4 Change detection task - Standard

Once again, the previously employed change detection task was used as a baseline measure in this experiment (see Experiment 4, Fig. 3.1, & Fig. 3.4). It consisted of 2 blocks testing 199 trials in total, with an average of 1.3% invalid trials due to time-out.

3.3.1.5 Change detection - Indicate

To prevent participants from using summary statistics of the entire array at the test phase (Brady & Alvarez, 2015; Jiang, Olson & Chun, 2000; Oberauer & Lin, 2017; Sims, Jacobs & Knill, 2012; Whitney & Yamanashi Leib, 2018), this variant of the change detection task had participants indicate precisely where in the memory array a colour change happened, rather than

61 simply judging whether or not a change was present somewhere (see Fig. 3.4). The CD-Indicate task proceeded identically to the CD-Standard task up to the test phase, at which point the entire memory array was returned to the display with a 100% probability that one of the array items changed in colour. Participants then used a computer mouse to click on the square that they believed had changed. The test array remained on screen until a click was exactly on one of the squares to avoid ambiguity. No time-out was imposed to minimize the confounding influence of motor skills on performance. After the click the screen was blanked. Participants completed 199 trials of this task with a break given mid-way through.

3.3.1.6 Change detection - Single

Participants completed a version of the CD-Standard task that removed all colour ensemble as well as gestalt information during the test phase. That is, during test only a single square from the memory array reappeared at its original spatial location, with a 50% probability that the square’s colour changed to that of one of the other squares in the array (see Fig. 3.4). In total 199 trials were tested in two blocks; an average of 1.5% were invalid due to time out.

62 CD-Single CD-Std CD-Indi 2-Back

Fixation - 1500ms Fixation - 1000ms

Memory array - 100ms Memory item - 400ms

Fixation - 800ms Mask - 200ms

Test item - 1500ms Test array - 1500ms Test array - s.p. Fixation - 1000ms

Memory item - 400ms

3.3.2 Results

Once again, I will first report descriptive statistics before reporting the correlational structure of the different tasks. Responses for 2-Back, CD-Standard, and CD-Single were converted into d’

63 values. Performance on 2-Back, CD-Standard and CD-Single ranged from 1.42 to 2.17 on average (Fig. 3.5 A). For CD-Indicate the proportion of correct responses (0.76) was calculated (Fig. 3.5 B; no d’ values could be calculated because of the absence of no-change trials). All performance measures were significantly different from guessing rate (t’s ≥ 13.09, p’s < 0.001). Skewness values were not significantly different from zero (-0.948 ≤ z’s ≥ 0.726).

I next conducted a one-way repeated-measures ANOVA including all tasks except for CD- Indicate, and this yielded a significant difference in d’ values (Greenhouse-Geisser corrected F(1.7, 49.3) = 19.8, p < 0.001). Orthogonal linear contrasts showed that 2-Back was significantly more difficult than the mean of CD-Standard and CD-Single (t(29) = 3.47, p = 0.002), and that CD-Single was significantly more difficult than CD-Standard (t(29) = 5.82, p < 0.001).

A B A2.5 B 1

2 0.75

1.5

0.5

1 Mean Performance (d’) Mean Performance

0.25 0.5

0 0 2-Back CD-Std CD-Single CD-Indi Task

Next, inspecting the correlations between tasks (Tab. 3.3) showed that, once again, there was no statistically significant relationship between 2-Back and CD-Standard (r = 0.02). CD-Indicate correlated marginally with 2-Back (r = 0.35). However, there was a significant correlation between CD-Single and 2-Back (r = 0.40). Also, there were significant correlations between the three different change detection tasks (CD-Standard vs. CD-Single (r = 0.51), CD-Standard vs. CD-Indicate (r = 0.71), and CD-Single vs CD-Indicate (r = 0.53)).

Table 3.3 Correlations between 2-Back and Change Detection performance

CI(95%) Corrected CI

r p Lower Upper Lower Upper

2-Back vs. CD-Standard 0.02 0.910 -0.37 0.44 -0.37 0.44

2-Back vs. CD-Single 0.40* 0.030 0.19 0.58 0.13 0.62

2-Back vs. CD-Indicate 0.35 0.055 -0.01 0.65 -0.08 0.68

CD-Std. vs. CD-Single 0.51* 0.004 0.19 0.75 0.09 0.80

CD-Std. vs. CD-Indicate 0.71* 0.000 0.50 0.86 0.40 0.90

CD-Single vs. CD-Indicate 0.53* 0.003 0.17 0.81 0.05 0.87

Finally, a principal component analysis inspected the relationship between the four tasks with respect to 2 factors (eigenvalues: 2.32 and 1.00, respectively; cumulative explained variance: 57.97% and 82.98%). After Varimax rotation, all CD tasks loaded on the first component whereas CD-Single and 2-Back loaded on the second component (Tab. 3.4 & Fig. 3.6).

Table 3.4 Rotated component scores

Task Component 1 Component 2

2-Back 0.070 0.966

CD-Standard 0.945 -0.109

CD-Single 0.662 0.487

CD-Indicate 0.848 0.282

1 2-Back

0.8

0.6 CD-Single 0.4

Component 2 CD-Indi 0.2

-0.2 0 0.2 0.4 0.6 0.8 1 CD-Std -0.2 Component 1 Figure 3.6 Rotated component scores of the principal component analysis General Discussion

It has been debated whether the n-back task is a valid measure of working memory functions such as complex working memory span (Jaeggi et al., 2007; Kane et al., 2007; Oberauer, 2005). Seemingly consistent with this, Experiments 2 and 3 (Chapter 2) found no correlation between visuospatial versions of the 2-Back and the change-detection task, a commonly used measure of working memory capacity (Vogel & Luck, 1997; Unsworth et al., 2014). Curiously however, Experiments 1 and 3 did identify a robust predictive relationship between 2-Back and transsaccadic memory, whereas change detection performance did not correlate with transsaccadic memory (Experiment 3). Therefore, the aim of the present study was to identify the features of the change detection task that would produce correlations with the 2-Back task (to

67 then test this version of the task with the transsaccadic memory task in the next chapter). In two experiments I used four variations of the change detection task, together with a standard version of the original task (CD-Standard; Vogel & Luck, 1997) as well as a 2-Back task as references. I found that 2-Back only correlated with CD-Single, that is, when all allocentric (as well as non- spatial) contextual information was removed during the test phase. At the same time, performance during CD-Single was significantly lower than during CD-Standard. This suggests that removing spatial (together with non-spatial) context at test prevented participants from using task strategies that exploited, for instance, gestalt information about the memory array. In the absence of gestalt strategies, 2-Back and change detection performance were correlated with one another.

None of the other features tested revealed a significant predictive relationship between 2-Back and change detection. Removing non-spatial context from change detection (but maintaining spatial context) established no predictive relationship, although summary statistics (here we tested colour) could have helped with CD-Standard (Brady & Alvarez, 2015; Jiang, Olson & Chun, 2000; Oberauer & Lin, 2017; Sims, Jacobs & Knill, 2012; Whitney & Yamanashi Leib, 2018), they were less useful with CD-Indicate. Even so, the correlational relationship of CD- Indicate with CD-Standard and 2-Back changed only marginally.

Likewise, I found no evidence that cognitive or perceptual interference during change detection taxed similar resources as those afforded in the 2-Back task. CD-Remap clearly required spatial remapping of egocentric coordinates. But this did not make CD-Remap more difficult. Thus, egocentric coordinates seem to be not particularly important for working memory fidelity so long as allocentric context exists. Also, remapping produced no correlations with 2-Back. Furthermore, CD-Mask effectively disrupted working memory encoding and resulted in lower performance compared to CD-Standard. Nevertheless, no correlation with 2-Back emerged. In sum, though resistance to interference from distracting information can be a strong predictor of working memory performance (Kane & Engle, 2002; Vogel, McCollough & Machizawa, 2005), introducing performance-impacting distractions into the change detection task played no particular role in creating covariability with 2-Back.

Finally, none of the poor correlations between 2-Back and change detection tasks were due to ceiling effects. Performance scores showed little skew, and there were robust correlations between all versions of the change detection task.

As for the correlational structure, inspecting the results of the principal component analyses reveals an interesting pattern where interference (CD-Remap and CD-Mask) yielded principal component loads that were very similar to those of CD-Standard regardless of task difficulty. However, removing spatial context (CD-Single), and less so non-spatial context (CD-Indicate), produced loads that drifted towards the dimension defined by 2-Back. Nonetheless, no version of the CD task drifted entirely towards 2-Back, perhaps given the more dynamic nature of the latter task.

To conclude, I found that change detection in the presence of spatial contextual information hides individual differences that would otherwise be predictive of performance on the 2-Back task. By removing context from the test phase of the CD-Single task, a predictive relationship with 2-Back performance emerged. Other manipulations of variables that could plausibly explain the previous lack of correlations did not, in the end, reveal such a relationship. Their results served to rule out other hypothetical explanations, such as visual distraction, remapping, and egocentric coordinate disruptions. Taken together, the present chapter further adds to the idea that the 2-Back task is a valid measure of working memory capacity while at the same time the task seems to include additional facets, perhaps as a consequence of its dynamic nature. Therefore, the next step will be to test whether performance on the context-deprived CD-Single task also predicts transsaccadic memory scores and whether 2-Back and CD-Single performance will produce a predictive relationship with the same or different aspects of transsaccadic memory.

Chapter 4 Transsaccadic Memory: Saccadic and Non-Saccadic Error Sources Introduction

Transsaccadic memory is required to retain visual information across eye movements. Visual information seen before an eye movement can be retrieved following a saccade (Frost et al., 2019; Irwin, 1991; Prime, Niemeier & Crawford, 2006; Prime, Tsotsos, Keith, and Crawford, 2007), demonstrating a form of working memory. It is not clear, however, to what extent transsaccadic memory is the same or different from fixational working memory. Crucially, when visual information is retrieved following a saccade, it must be localized according to its correct spatiotopic coordinates, even though the retinotopic coordinates have shifted (Henriques et al., 1998; Merriam, Genovese, and Colby, 2003; 2007; Prime, Niemeier & Crawford, 2006; Sommer & Wurtz, 2002). With fixational working memory tasks however, coordinates do not shift. Therefore, transsaccadic memory likely involves at least some processes unique unto itself (Prime, Vesia & Crawford, 2008; Prime, Vesia, and Crawford, 2010; Ten Brink, Nijboer, Fabius, and Van Der Stigchel, 2019; Vasquez and Danckert, 2008).

Indeed, in Chapter 2 I showed that transsaccadic memory performance could not be predicted by performance on a change detection task, although it was predicted by performance on a 2-Back task. 2-Back (or more generally n-Back) tasks have long been used as a measure of working memory ability (Cohen et al., 1994; Courtney et al., 1998; Gevins et al., 1990; Gevins & Cutillo, 1993, Smith, Jonides & Koeppe, 1996). However, it has been debated whether the n-Back task is a valid measure of working memory (e.g., Jaeggi et al., 2010, Kane et al., 2007). The change detection task, on the other hand, is well-established as a measure of working memory capacity (Alvarez & Cavanagh, 2004; Luck & Vogel, 1997; Vogel, McCollough, and Machizawa, 2005; Unsworth, Fukuda, Awh & Vogel, 2014). In Experiment 3 I found that the change detection task, as based on Luck and Vogel (1997), did not correlate with the 2-Back task. Also, in Experiments 2 and 3, transsaccadic memory performance was found to share little variance with performance on the change detection task. This lack of a correlation suggests that transsaccadic memory might not be a general form of visual working memory, and that further experiments are needed

70 to characterize the working memory faculties that transsaccadic memory as well as the 2-Back task recruit.

Importantly, Chapter 3 showed that presenting only a single test item, so as to remove spatial context, resulted in a significant correlation between the change detection task and 2-Back, helping to reconcile the dissociation between the tasks. It is not known, however, whether the correlation of CD-Single with 2-Back will extend to transsaccadic memory.

Therefore, the aim of the present chapter was to advance our understanding of the processes underpinning transsaccadic memory and establish external validity for it by evaluating how its components relate to the change detection-single (CD-Single) task along with 2-Back. Using this approach, I sought to answer the question as to whether transsaccadic memory shares a single source, or multiple sources of variance with fixational working memory tasks. In addition, to directly contrast fixational and saccadic working memory processes with one another, I included a control task that was identical to the transsaccadic memory task except participants were not required to make an eye movement. If the no-eye movement control task and the transsaccadic memory task involved faculties that are distinct from one another, some components of the two tasks should differ from one another, and correlate differently with CD-Single and 2-Back.

Experiment 6

4.2.1 Method

4.2.1.1 Participants

22 healthy students (14 females) with a median age of 21 years (range: 18 to 23 years) participated in the experiment, and had normal or corrected-to-normal vision. All procedures were approved by the Human Participants Review Sub-Committee of the University of Toronto and were performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.

4.2.1.2 Apparatus and Procedure

In this study I measured the correlational structure of performance on the transsaccadic memory task (as established in Chapter 2) together with performance on the 2-Back task and the CD- Single task that was found to share variability with the 2-Back task in Chapter 3. In addition, I

71 devised a no-saccade control version of the transsaccadic task. Participants performed the experiments while seated in a dark room with their heads stabilized by a chin rest. Stimuli were presented at a distance of 89 cm on an LCD monitor centered at eye level that was 115 cm wide (66 degrees of visual angle). Stimuli were generated and presented using Matlab (MathWorks) and the Psychophysics and EyeLink Toolboxes (Brainard, 1997; Kleiner et. al, 2007; Pelli, 1997). Two tasks in this experiment required gaze tracking, and for them participants were fitted with a head-mounted eye tracker which sampled the position of the left eye at 500 Hz, which was calibrated to produce readings with less than 1 visual degree of error (EyeLink II, SR Research, Ottawa).

4.2.1.3 Transsaccadic memory (TSM) task

The TSM task for this experiment (Fig. 4.1) was based upon the task used in Experiment 3 (Fig. 2.9), with the difference being that only the high-load condition was used. Eye movements were recorded throughout and gaze position was processed online to ensure correct fixations and eye movements. Trials of this task had participants first fixate on a point (0.2 deg across) located at the screen’s vertical center and an average of 15 degrees horizontally from screen center (with up to +/- 6.9 degrees of horizontal random jitter) and then click a mouse to indicate that they were ready. Nine hundred to 1100 ms after the mouse click, a memory array of three coloured squares (red, green and blue, 0.7 degrees across, Fig. 4.1) was briefly presented. To facilitate the analysis of systematic errors, each of the memory array squares was located at one of eight possible positions in an evenly spaced circular array (10 degrees in diameter) centred around initial fixation point forming an imaginary octagon (e.g., Fig. 4.2). Four hundred milliseconds later a mask composed of 1500 randomly positioned squares of the same dimensions and colours as those in the memory array was displayed for 200 ms. Five hundred milliseconds after the offset of the mask, a saccade target was presented 15 degrees to the left or to the right of the fixation point, and participants were instructed to make a saccade to it. After 900 to 1100 ms, with participants now fixated on the saccade target, another mask was presented for 200 ms, and 900 to 1100 ms later a cursor with the colour (and dimensions) of one of the memory array items was placed at a random location within the circular region in which the memory array had appeared earlier. Participants then moved the cursor with a computer mouse to indicate the remembered location of the memory item of the matching colour. A mouse click at that location concluded the trial. During a trial, if the participant’s eyes deviated by 2 visual degrees from the initial

72 fixation point or 3 visual degrees from the target fixation point after saccade, the participant was alerted that fixation had been lost, and the trial was recycled. A total of 192 trials were completed across 3 blocks (one block of data for one participant was lost due to a computer error, but all others produced 192 useable trials).

Mouse click responses were processed to calculate measures of systematic and unsystematic transsaccadic error. As shown in Fig. 4.2, this created eight clusters of mouse clicks, each pertaining to one of the eight stimulus locations. From the clusters I first removed outliers. To identify outliers, for each participant I calculated a ‘global’ standard deviation of the distance of each of their responses from its respective intersection location. Then, each response that deviated by more than 1.96 ‘global’ standard deviations from its respective cluster was labelled as an outlier and removed from subsequent analyses (1.7% of responses on average across the transsaccadic and control conditions). Note that it is possible that some of the outliers reflected working memory errors. For example, participants might have forgotten the true location of the memory item and simply guessed. Other outliers might have been due to confusion errors where participants aimed for a location other than the correct one. However, a third type of outlier was probably due to simple motor errors where participants pressed the mouse key before moving the mouse to the intended location. With the current paradigm I was unable to tease apart memory and motor errors.

Next I recalculated the horizontal and vertical standard deviations as measures of unsystematic error. To calculate systematic errors, we recalculated the average coordinates of the eight clusters of mouse clicks and submitted the data to a Procrustes analysis to obtain quantitative measures of all linear transformations required to overlay the cluster locations with the octagonal arrangement of the original stimulus positions. That is, we obtained measures of how much the eight clusters were systematically shifted horizontally and vertically, how much they were rotated, and how much they differed in scale relative to the original grid. As a fifth output, Procrustes analysis returned a value ‘D’ that reflects the amount of divergence between the response positions and the true stimulus positions after correcting for the four other linear transformations. All measures of systematic error that could vary in two different directions (+/- horizontal offset, +/- vertical offset, +/- rotation angle, scaling up vs. scaling down) were converted into absolute deviations from zero error, since the size of deviation from the correct

73 values are more likely to capture information about working memory performance than deviation in one particular direction.

For later analyses where data from the two eye movement conditions (leftward and rightward saccades) are combined, the average response coordinates for each of the stimulus locations from leftward trials were averaged with their corresponding responses from rightward trials, with the rightward locations horizontally inverted to match the directionality of any saccade-related horizontal offsets across eye movements. From these combined data, the same measures of systematic error described above were extracted.

4.2.1.4 TSM-Control Task

In the no-saccade control version of transsaccadic memory task trials proceeded identically to the transsaccadic task, except that no saccade target was presented, and participants remained fixated at the initial fixation point throughout the trial (Fig. 4.1). Eye movements were recorded throughout and gaze position was processed online to ensure correct fixations (trials with fixation errors were recycled), and to mirror the experience from the transsaccadic task. A total of 192 trials were completed across 3 blocks, and the data were analyzed in the same manner as the transsaccadic conditions.

4.2.1.5 2-Back

Participants’ working memory ability was also assessed using a 2-Back task using the same design employed in Experiments 3, 4 and 5 (see Fig. 4.1). Participants fixated a central fixation point (0.2 degrees across) for the entirety of the task, and then monitored a series of white squares (0.6° in width) that appeared, one at a time, for 400 ms at one of eight evenly-spaced locations of a circular array (12° in diameter), followed by a 200-ms mask showing 1000 random squares, and then by a blank screen and central fixation point. One second later, the next trial began. Every time a square appeared at the same position as two trials earlier participants were asked to respond with a key press on a computer keyboard. Participants completed 368 trials of this task (double the number collected in Experiments 3, 4 and 5 to maximize the accuracy of the d’ estimate).

4.2.1.6 Change detection - Single

The change detection-single task (CD-Single) designed for Experiment 5 was used once again in this experiment. That is, trials began with fixation upon a central point (0.2 degrees across) on a grey background. 1500 milliseconds later a memory array of 3 or 4 uniquely coloured squares (randomly selected from red, orange, green, blue or violet, 0.65 degrees in width) appeared to the left or to the right of fixation (Vogel & Machizawa, 2004, presented stimulus arrays on both sides and cued participants to disentangle electrophysiological signals due to visual stimulation and cognitive processes; this was not necessary here). The squares were randomly placed at locations on a rectangular array measuring 4 degrees in width by 8 degrees in height (see Fig. 4.1), centred 5 degrees away from fixation on the horizontal axis. The area of possible locations was subdivided into a grid of 5 by 9 positions with 1 degree spacings, and from these locations, the stimulus squares were assigned up to 0.1 degrees of random jitter to minimize any appearance of positional regularity. The memory array remained on screen for 100 ms, and was followed by a blank display (save for the fixation point) for 800 ms. Next, a single square from the memory array reappeared at its original spatial location. But half the time its colour changed to a new colour not previously shown in the array. If the participant believed there was a change, they pressed the ‘up’ arrow key on a computer keyboard, otherwise they pressed the ‘down’ arrow key. The test array remained on screen until a response was given or until a 1.5 s time-out at which point trials were counted as invalid (on average 1.9% of all trials). The next trial began after a 1.5 s inter-trial interval. A total of 199 trials was tested in two blocks (one trial was lost due to a programming glitch).

TSM TSM-Control CD-Single 2-Back

Fixation - 900-1100ms Fixation - 900-1100ms Fixation - 1500ms Fixation - 1000ms

Memory array - 400ms Memory array - 400ms Memory array - 100ms Memory item - 400ms

Mask - 200ms Mask - 200ms Fixation - 800ms Mask - 200ms

Fixation - 500ms Fixation - 500ms Test array - 1500ms Fixation - 1000ms

Saccade - 900-1100ms + Fixation - 900-1100ms Memory item - 400ms

Mask - 200ms Mask - 200ms Mask - 200ms

Fixation - 900-1100ms Fixation - 900-1100ms Fixation - 1000ms

Response Response Memory item - 400ms

Figure 4.1 Experimental Paradigm

4.2.2 Results

I will first report descriptive and inferential statistics before exploring the correlational structure of the different tasks. Group averages of systematic and unsystematic errors are visualized in Fig. 4.2 for both saccade directions and the transsaccadic control condition separately. As with the transsaccadic experiments from Chapter 2, we converted the transsaccadic memory responses relative to saccade direction and then computed systematic and unsystematic errors, which were then tested for saccade-related effects (leftward vs. rightward vs. no-saccade) in a series of fourteen repeated measures ANOVAs (Tab. 4.1). For unsystematic error, a significant difference was identified for the horizontal component (F(2,42) = 43.19, p < 0.001; the vertical component was not significant after Bonferroni correction), and follow-up linear contrasts found that this effect was driven by better accuracy in the no-saccade control condition compared to the two eye movement conditions’ mean accuracy (t(21) = 7.84, p < 0.001). Between the two saccade directions, there was no significant difference in horizontal accuracy (p > 0.05 after serial Bonferroni correction). With regard to systematic errors, significant differences were identified for horizontal translation (F(2,42) = 13.78, p < 0.001), vertical translation (F(2,42) = 10.18, p < 0.001), and rotation (F(2,42) = 10.25, p = 0.001). Follow-up linear contrasts showed greater error in the eye movement conditions for all variables (offset X: t(21) = 6.19, p < 0.001; offset Y: t(21) = 3.96, p = 0.001; rotation: t(21) = 3.48, p = 0.002). Unexpectedly, I found significantly smaller rotational error for leftward than rightward eye movements (t(21) = 3.07, p < 0.006; the same difference was observed for non-rectified data). No other variable yielded significant differences between the two saccade conditions, consistent with Chapter 2.

A B C

1°

Table 4.1 Repeated Measures ANOVAs for the TSM/TSM-Control Error Variables

2 Error F(2,42) p ηp

Unsystematic X 43.19*** < 0.001 0.67

Unsystematic Y 4.67 0.037 0.18

D 3.01 0.082 0.13

Scale 1.20 0.290 0.05

Offset X 13.78*** < 0.001 0.40

Offset Y 10.18*** < 0.001 0.33

Rotation 10.25** 0.001 0.33

A B C D 2 2 0.05 0.25

1.8 1.8 0.045

1.6 1.6 0.04 0.2

1.4 1.4 0.035

1.2 1.2 0.03 0.15

1 1 0.025 Procrustes D 0.8 0.8 0.02 0.1 Procrustes Scaling Unsystematic X (SD) Unsystematic Y (SD)

0.6 0.6 0.015

0.4 0.4 0.01 0.05

0.2 0.2 0.005

0 0 0 0 Left Right No EM Left Right No EM Left Right No EM Left Right No EM E F G 1.5 1.5 0.1

0.09

0.08

0.07 1 1

0.06

0.05 Offset X Offset Y

0.04 Procrustes Rotation 0.5 0.5 0.03

0.02

0.01

0 0 0 Left Right No EM Left Right No EM Left Right No EM

To evaluate the relationships between error variables from the transsaccadic task and its no eye movement control, the 14 variables were entered into a principle components analysis. This analysis identified four components with eigenvalues greater than 1 (cumulative explained variance: 73.81%), with its component loadings Varimax rotated for interpretability (Tab. 4.2). The first component was loaded onto strongly (> 0.5) by unsystematic error in general (vertical and horizontal, for both the transsaccadic task and the no-eye movement control), and for systematic error, by the control condition’s Procrustes D and scaling measures. This pattern of loadings is consistent with a general ‘working memory degradation’ component, with little loading on saccade-specific systematic error variables. The second component was strongly loaded onto only by measures of systematic error from the eye movement task in terms of Procrustes’ D and scaling, which represent participants’ ability to accurately reproduce spatial topography following a saccade. Component 3 was strongly loaded onto by systematic offset (horizontal and vertical) for the no-saccade task, which suggests that participants who tend to systematically shift responses horizontally when mouse pointing also tend to systematically shift responses vertically. Finally, component 4 was strongly loaded onto by horizontal offset and rotation from the eye movement task, and by rotation from the no-saccade movement control, though in the opposite direction as the transsaccadic conditions. Note that conducting the principle component analysis on the unrectified variables yielded the same sets of substantial loadings for components 1 and 2, while redistributing components 3 and 4 into three parts.

Table 4.2 Rotated component scores

Component

Error 1 2 3 4

TSM Unsystematic X 0.953 0.044 0.063 0.095

Unsystematic Y 0.654 0.350 0.228 0.395

Procrustes D 0.177 0.891 -0.102 0.020

Scaling -0.032 0.916 -0.092 0.028

Offset X 0.112 -0.080 -0.362 0.752

Offset Y 0.138 0.395 0.415 0.012

Rotation -0.085 0.048 0.361 0.556

TSM-Control Unsystematic X 0.966 -0.023 0.097 -0.162

Unsystematic Y 0.969 0.053 0.158 -0.051

Procrustes D 0.948 -0.014 0.142 -0.152

Scaling 0.750 0.382 0.060 -0.252

Offset X 0.211 -0.035 0.826 -0.093

Offset Y 0.127 -0.189 0.835 0.013

Rotation 0.151 -0.038 0.023 -0.536

Note. Component scores in bold type are greater than 0.5 or smaller than -0.5.

For a better understanding of the components and to establish external validity as measures of working memory, I computed factor scores for the four components of the PCA. Then I used the

81 factor scores as predictors in two multiple regression analyses (with stepwise inclusion) to explain the performances in the 2-Back and CD-Single tasks, respectively. I considered component 1 a likely candidate for predicting working memory performance given that unsystematic error in general loaded strongly onto it, and that unsystematic error on the transsaccadic task previously predicted working memory performance (2-Back, Experiments 1 and 3) while also being sensitive to working memory load (Experiments 1, 2 and 3). The 2-Back task was predicted by components 1 and 2 (Factor 1 R2 = 0.216, β = -0.359, p = 0.029; Factor 2 R2 = 0.151, β = -0.300, p = 0.047; whole-model R2 = 0.367, p = 0.013; negative beta weights indicated that larger errors as captured with components 1 and 2 predicted poorer performance in 2-Back, as expected). In contrast, only component 1 was included in the regression analysis for the CD-Single task (R2 = 0.280, p = 0.011, β = -0.285; once again, the negative beta weight indicates that larger component 1 scores predict poorer CD-Single performance). Neither component 3 nor 4 provided significant additional predictive value for either 2-Back or CD- Single (correlations with 2-Back and CD-Single: r’s ≤ 0.311, p’s ≥ 0.159). This suggests that the two components reflected functions unrelated to working memory, perhaps to do with the visuomotor aspects of the mouse clicking.

Finally, there was a significant correlation (r = 0.55, p = 0.007) between 2-Back performance (mean d’ = 1.68, SD = 0.77) and CD-Single performance (mean d’ = 1.52, SD = 0.54), consistent with the correlation observed in Experiment 5. Individual correlations between the two working memory tasks and the 14 TSM/TSM-Control error variables were computed and are outlined in Tab. 4.3, with plots for the TSM correlations in Fig. 4.4 and Fig 4.5. Given the large number of comparisons, the only correlations that were significant after serial Bonferroni correction were transsaccadic Procrustes D vs. 2-Back (r = -0.52, p = 0.014), and Procrustes scaling from the no- saccade control vs. CD-Single (r = -0.62, p = 0.002).

Table 4.3 Correlations between measures of transsaccadic error and working memory

TSM TSM-Control

Error Variable r p Lower Upper r p Lower Upper

2-Back Unsystematic X -0.51 0.016 -0.76 -0.1 -0.43 0.043 -0.68 -0.14

Unsystematic Y -0.58 0.005 -0.8 -0.24 -0.48 0.024 -0.71 -0.19

D -0.52* 0.014 -0.74 -0.26 -0.46 0.032 -0.70 -0.16

Scaling -0.25 0.264 -0.57 0.13 -0.41 0.058 -0.68 -0.09

Offset X 0.12 0.603 -0.32 0.57 -0.24 0.280 -0.56 0.09

Offset Y -0.26 0.238 -0.66 0.19 -0.20 0.362 -0.51 0.15

Rotation 0.00 0.995 -0.45 0.42 -0.49 0.021 -0.71 -0.18

CD- Unsystematic X -0.51 0.015 -0.81 -0.03 Single -0.57 0.006 -0.83 -0.14

Unsystematic Y -0.35 0.112 -0.6 -0.08 -0.51 0.015 -0.80 -0.06

D -0.23 0.307 -0.48 0.04 -0.54 0.009 -0.82 -0.05

Scaling -0.18 0.422 -0.59 0.23 -0.62* 0.002 -0.85 -0.22

Offset X 0.30 0.176 -0.11 0.6 -0.05 0.810 -0.38 0.27

Offset Y -0.25 0.268 -0.56 0.05 -0.04 0.874 -0.36 0.29

Rotation 0.17 0.444 -0.27 0.54 -0.15 0.491 -0.62 0.28

Note. 95% confidence intervals derived from bootstrapped regression analysis (100,000 resampling iterations). * zero falls outside of the serial Bonferroni corrected confidence interval.

TSM Unsystematic X vs 2-Back TSM Unsystematic Y vs 2-Back TSM Scale Abs vs 2-Back TSM Procrustes D vs 2-Back 0.5 4 4 0.1

3.5 3.5 0.08 0.4 3 3

2.5 2.5 0.06 0.3 2 2 0.04 0.2 1.5 1.5 TSM Scale Abs

1 1 TSM Procrustes D 0.02 0.1 TSM Unsystematic X (SD) TSM Unsystematic Y (SD) 0.5 0.5

0 0 0 0 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 2-Back 2-Back 2-Back 2-Back

TSM Offset X Abs vs 2-Back TSM Offset Y Abs vs 2-Back TSM Procrustes Rotation Abs vs 2-Back 2 1.5 0.1

0.08 1.5 1 0.06 1 0.04 0.5 TSM Offset X Abs 0.5 TSM Offset Y Abs 0.02 TSM Procrustes Rotation Abs

0 0 0 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 2-Back 2-Back 2-Back

Figure 4.4 Results of Experiment 6: Correlations between 2-Back performance and measures of transsaccadic error. Envelopes indicate the correlation coefficients’ bootstrapped 95% confidence intervals, with red sections indicating boundaries where a threshold of r = 0 has been crossed

TSM Unsystematic X (SD) vs CD-Single TSM Unsystematic Y (SD) vs CD-Single TSM Procrustes D vs CD-Single TSM Scale Abs vs CD-Single 4 4 0.1 0.5

3.5 3.5 0.08 0.4 3 3

2.5 2.5 0.06 0.3 2 2 0.04 1.5 1.5 0.2 TSM Scale Abs 1 1 TSM Procrustes D 0.02 0.1 TSM Unsystematic X (SD) TSM Unsystematic Y (SD) 0.5 0.5

0 0 0 0 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 CD-Single CD-Single CD-Single CD-Single

TSM Offset X Abs vs CD-Single TSM Offset Y Abs vs CD-Single TSM Procrustes Rotation Abs vs CD-Single 2 1.5 0.1

0.08 1.5 1 0.06 1 0.04 0.5 TSM Offset X Abs 0.5 TSM Offset Y Abs 0.02 TSM Procrustes Rotation Abs

0 0 0 0 1 2 3 0 1 2 3 0 1 2 3 CD-Single CD-Single CD-Single

Figure 4.5 Results of Experiment 6: Correlations between CD-Single performance and measures of transsaccadic error. Envelopes indicate the correlation coefficients’ bootstrapped 95% confidence intervals, with red sections indicating boundaries where a threshold of r = 0 has been crossed

4.2.3 Discussion

The present experiment sought to extend our understanding of the processes underlying transsaccadic memory by establishing external validity. To this end I explored the correlational structure of the transsaccadic memory (TSM) task by extracting its principal components, and I evaluated how the components related to two established working memory tasks. These tasks were a visuospatial version of the 2-Back task that in Chapter 2 was found to predict transsaccadic task performance, as well as the change detection single (CD-Single) task, a variant of the change detection task (Luck & Vogel, 1997) where contextual information is removed from the retrieval phase, thereby establishing a correlation with the 2-Back task (see Chapter 3). Further, I included a no-saccade version of the TSM task to contrast fixational task performance with performance after left- and rightward saccades.

Comparing performance with respect to leftward and rightward eye movements, there was little difference between the two. There was, however, one exception: I observed significantly greater rotation errors following rightward compared to leftward saccades. This contrasts with Chapter 2 where none of the experiments produced a significant saccade direction effect. Furthermore, neither Chapter 2 nor the present study found rotation errors to correlate with the 2-Back or the change detection task, and rotation errors did not load on any principal component capturing working memory, which suggests that rotation errors are not related to working memory performance. In sum, the present observations are consistent with Chapter 2 (also see Ten Brink et al., 2019; in contradiction to Vasquez & Danckert, 2008) in that saccades have no direction- specific effect on transsaccadic memory.

However, saccades, regardless of direction, did impact performance in the TSM task. Compared to the no-saccade control task, participants performed worse after saccades. They showed greater horizontal unsystematic errors, and their systematic errors increased in terms of horizontal and vertical translational errors, as well as rotations.

To understand whether these effects were due to saccades impacting working memory functions, I submitted the 14 error variables of the TSM task (averaged for leftward and rightward saccades) and its no-saccade control version to a principal component analysis followed by multiple regressions with the 2-Back and CD-Single scores. This produced two components that were related to working memory. Component 1 especially captured unsystematic errors in the

85 saccade as well as the no-saccade version of the TSM task (although it also encompassed contributions from scaling and nonlinear errors during the no-saccade test). The component was correlated with both the CD-Single task and the 2-Back task. Taken together this suggests that component 1 measured working memory capacity (similar to change detection, Luck & Vogel, 1997; Unsworth et al., 2014) and that this aspect of working memory is engaged regardless of whether or not participants made eye movements.

In contrast, component 2 only measured saccadic aspects of working memory. That is, it captured systematic errors of scaling and nonlinear distortions observed in the TSM task, but only when participants made a saccade and not when they did not move their eyes. Nevertheless, saccades were not necessary to trigger this aspect of working memory given that component 2 correlated with the 2-Back task where participants maintain fixation throughout. Thus, component 2 appears to reflect a dynamic aspect of working memory in tasks that require information to be remapped spatially or cognitively. As an alternative or related possibility, component 2 might be a measure of attentional control (e.g., Unsworth et al., 2014). Two additional components were extracted by the principal component analysis but neither of them was found to measure working memory functions as they did not correlate with the 2-Back or the CD-Single task. Also, it is unlikely that they captured presently unknown memory functions that are specific to transsaccadic memory because neither of the two components featured loads from the saccadic version of the TSM task alone: component 3 mostly captured translational errors in the no-saccade version of the TSM task, and component 4 captured a mixed pattern of errors from the saccadic as well as the no-saccadic versions of the task. Therefore, it is quite possible that components 3 and 4 reflected visuomotor abilities of moving the mouse cursor.

In conclusion, the present experiment identified two components of transsaccadic memory. A first component appeared to reflect working memory capacity and was relied upon no matter whether participants made eye movements or not. A second component was more specifically associated with remembering information across saccades, which seems to reflect dynamic processes of remapping. Although more research is required to clarify the sources of non- overlapping variability, these present results are, to my knowledge, the first to identify the similarities and differences between fixational and saccadic working memory. They highlight the significance of studying working memory in a more ecologically valid manner including saccadic eye movements.

General Discussion

Since storing visual information over short delays is the core function of both transsaccadic memory and visuospatial working memory, a single set of faculties might provide the underpinnings for both. That said, visuospatial working memory is typically measured under conditions where stimuli are retinotopically stable, whereas transsaccadic memory necessarily entails spatial transformations, attention shifts and visual disruptions. Identifying the similarities and differences between these two systems is the central aim of this thesis. Do both systems use the same cognitive faculties? Or are they independent systems? Also, based on previous research (Vasquez & Danckert, 2008; but see Ten Brink et al., 2019) I sought to test whether transsaccadic memory differs depending on saccade direction.

Summary of results

In Chapter 2, transsaccadic memory performance was predicted by performance on a 2-Back task, which is a measure of visual working memory ability (Cohen et al., 1994; Courtney et al., 1998; Gevins et al., 1990; Gevins & Cutillo, 1993, Smith, Jonides & Koeppe, 1996). Specifically, better 2-Back performance predicted lower unsystematic error, Procrustes D and scaling on the transsaccadic task, suggesting that common faculties support performance on both tasks. Further, across experiments, an increase in transsaccadic memory load was found to increase error across a number of measures, which is consistent with the kinds of resource limitations that characterize working memory (Luck & Vogel, 1997; Vogel & Machizawa, 2004; Todd & Marois, 2005; Xu & Chun, 2006). However, I found no difference in transsaccadic performance between leftward and rightward eye movements. Also, performance on a change detection task produced no significant correlations with any of the transsaccadic tasks’ error variables, suggesting that transsaccadic memory recruits faculties that are quite different from those that support visuospatial working memory in the change detection task. That said, change detection performance also failed to significantly predict 2-Back performance, which prompted the question of whether the eye movement component of the transsaccadic task was the critical factor dissociating it from change detection, or whether this is attributable to a feature common to both 2-Back and transsaccadic memory but absent from the change detection task.

In Chapter Three, the dissociation between change detection and the other tasks was investigated further (focusing first on the 2-Back task instead of the time-consuming transsaccadic task). Four possible explanations for the dissociation were tested by comparing performance on variants of the change detection task to performance on the 2-Back task, and a correlation was identified using a change detection task that minimizes the availability of contextual spatial information during its retrieval phase (CD-Single) thereby also making the task more difficult. Three other manipulations of the change detection task failed to produce significant correlations with 2-Back performance, but these failures provided useful insights. Specifically, disrupting egocentric coordinates during the recall phase of the change detection task did not reveal a predictive relationship with the 2-Back task. It did however correlate strongly with the standard change detection task, and the disruption of the egocentric coordinates did not have a significant effect on its difficulty level. This indicates that working memory represents information in the form of higher-level relational properties of the array, such as gestalts, rather than egocentric coordinates. Another variant of the change detection task where participants used a mouse to indicate changes made it difficult to use ensemble statistics or colour gist, yet the test failed to produce a significant correlation with 2-Back, suggesting that removing non-spatial context was not sufficiently effective to establish a correlation between the task and 2-Back. That said, the task showed a trend for a correlation that could be investigated in the future. Finally, inserting a visual distraction between the encoding and retrieval phases of the change detection task made it significantly more difficult, but this did not produce a correlation with 2-Back, though a strong correlation with the standard change detection task was maintained. As such, I concluded that difficulty alone is insufficient to explain the correlation uncovered with CD-Single and is unlikely to explain the dissociation from the other tasks. In sum, Chapter 3 showed strong internal consistency amongst change detection tasks as long as the whole array was presented during the retrieval phase, in spite of substantial changes to other task features. By removing contextual spatial information from the retrieval phase of the change detection task though, a significant correlation with 2-Back was identified, which provided a candidate for comparison with the transsaccadic task.

In Chapter 4 then, I tested the hypothesis that the dissociation between transsaccadic memory and change detection performance arose from the same feature that caused the dissociation between 2-Back and change detection performance. If this were the case, then removing

88 contextual information from the retrieval phase of the change detection task should result in substantial shared variance with both transsaccadic memory performance and 2-Back. Transsaccadic memory and CD-Single were therefore tested along with 2-Back, as well as a no- saccade control version of the transsaccadic task to isolate error directly related to saccades. The 14 error measures derived from transsaccadic memory performance and its no-eye movement baseline were entered into a principal component analysis that extracted four latent components, of which two components related to performance on the working memory tasks. Component 1 reflected loadings from both transsaccadic and no-saccade baseline error measures, importantly reflecting unsystematic error for both conditions. This component produced significant correlations with both CD-Single and 2-Back, so given its ability to predict performance across tasks, component 1 might reflect participants’ general ability to maintain units of information in working memory, i.e., their working memory capacity. Component 2 featured loadings relating to systematic error from the transsaccadic task, but not the no-saccade control task. It only correlated with the 2-Back task and not the CD-Single task. This suggests that component 2 reflects an aspect of working memory that is specific to dynamic working memory tasks where information needs to be spatially or cognitively remapped. In general, the results from Chapter 4 support a conceptualization of transsaccadic memory that recruits general working memory faculties to maintain information across saccades.

Implications and future directions

The central question motivating the research in this thesis was: “Does transsaccadic memory recruit the faculties that support general visual working memory?”, and in general, the results suggest that it does. That is, the 2-Back task, CD-Single task, and the no-saccade baseline share substantial variance with the transsaccadic task. However, there were also substantial differences in how the different visual working memory tasks related to transsaccadic memory performance, and how they related to each other. The results support a model of transsaccadic working memory performance that covaries with visuospatial working memory along two dimensions. The first of these dimensions explains most of the variability in unsystematic error for the transsaccadic task as well as the no-saccade baseline, and its explanatory power also extended to predict performance on the 2-Back and change detection-single tasks. The pattern of explained variance across all of the tasks seems to reflect the general ability to hold units of visuospatial information in working memory, in the form of working memory capacity (e.g. Luck & Vogel,

1997; Vogel & Machizawa, 2004; Todd & Marois, 2005; Xu & Chun, 2006; Unsworth et al., 2014). The second dimension captured systematic errors observed in the transsaccadic task and predicted 2-Back performance but did not explain substantial variance in either the change detection task or the no-saccade baseline. The unique relationship between transsaccadic memory and 2-Back highlighted by this component might reflect the ability of participants to dynamically manage the contents of working memory in the face of spatial or cognitive remapping, perhaps akin to the faculties captured by attention control. A definitive accounting of the faculties captured by this component is beyond the scope of this thesis but should be the subject of future experiments testing canonical measures of attention control alongside the transsaccadic task.

In addition to the two components explaining variance in transsaccadic task performance, a possible third working memory component emerged from Chapters 2 and 3. Once again, in Chapter 2 the CD-Standard task was not correlated with transsaccadic memory performance. Furthermore, in Chapter 3 three additional, substantially different versions of the change detection task yielded performance measures that were, nevertheless, strongly interrelated with one another. This seems to indicate that working memory, whenever given the chance, exploits higher level representations like summary statistics or gestalts. This could reflect people’s ability to encode features of the working memory array by attaching them to objects or other templates stored in long term memory so as to boost working memory performance (akin to the well- known benefits of chunking, e.g. Miller, 1956; Brady, Konkle & Alvarez, 2009). If this is the case, then the dissociation between performance on the standard change detection task and transsaccadic memory is unlikely to arise from the eye movement component of the transsaccadic task, but rather because the current transsaccadic task does not permit for higher level summary representations to be exploited. Therefore, future versions of the transsaccadic task could include, for example, landmarks to aid performance. It is possible that this would establish correlations with the standard change detection task.

Conclusion

In this thesis I investigated transsaccadic memory using a task that provided separate measures of precision (reflected in unsystematic errors) and accuracy (reflected in several systematic errors) across saccadic eye movements and during fixation. I found that during fixation, precision

90 reflected working memory capacity similar to the change detection task. Across saccades, precision continued to rely on capacity, yet its measures of accuracy now made the task more similar to the 2-Back task, perhaps because saccadic remapping recruited processes similar to those subserving attentional control. In general, the patterns of interrelation across the tasks are consistent with views of working memory being a multifaceted construct, with different tasks exploiting some facets more than others. Together, these findings provide a clear path for future experiments to identify and characterize the specific faculties that support transsaccadic memory as a special, yet very common form of visual working memory.

References

Alvarez, G. A., & Cavanagh, P. (2004). The capacity of visual short-term memory is set both by visual information load and by number of objects. Psychological Science, 15(2), 106– 111. https://doi.org/10.1111/j.0963-7214.2004.01502006.x Awh, E., Barton, B., & Vogel, E. K. (2007). Visual working memory represents a fixed number of items regardless of complexity. Psychological Science, 18(7), 622–628. https://doi.org/10.1111/j.1467-9280.2007.01949.x Baddeley, A. D., & Warrington, E. K. (1970). Amnesia and the distinction between long- and short-term memory. Journal of Verbal Learning and Verbal Behavior, 9(2), 176–189. https://doi.org/10.1016/S0022-5371(70)80048-2 Baddeley, A. (1992). Working memory. Science, 255(5044), 556–559. Baddeley, A. (2003). Working memory: Looking back and looking forward. Nature Reviews Neuroscience, 4(10), 829–839. https://doi.org/10.1038/nrn1201 Baddeley, A. D., & Hitch, G. (1974). Working memory. In G. Bower (Ed.), Psychology of Learning and Motivation: Advances in Research and Theory (pp. 47–89). Bays, P. M. (2018). Reassessing the evidence for capacity limits in neural signals related to working memory. Cerebral Cortex, January, 1–7. https://doi.org/10.1093/cercor/bhx351 Bays, P. M., Catalao, R. F. G., & Husain, M. (2009). The precision of visual working memory is set by allocation of a shared resource. Journal of Vision, 9(7), 1–11. https://doi.org/10.1167/9.10.7 Bays, P. M., & Husain, M. (2008). Dynamic shifts of limited working memory resources in human vision. Science, 321(5890), 851–854. https://doi.org/10.1126/science.1158023 Boon, P. J., Zeni, S., Theeuwes, J., & Belopolsky, A. V. (2018). Rapid updating of spatial working memory across saccades. Scientific Reports, 8(1), 1–11. https://doi.org/10.1038/s41598-017-18779-9 Brady, T. F., & Alvarez, G. A. (2011). Hierarchical encoding in visual working memory: Ensemble statistics bias memory for individual items. Psychological Science, 22(3), 384– 392. https://doi.org/10.1177/0956797610397956 Brady, T. F., & Alvarez, G. A. (2015). Contextual effects in visual working memory reveal hierarchically structured memory representations. Journal of Vision, 15(15), 1–24. https://doi.org/10.1167/15.15.6 Brady, T. F., Konkle, T., & Alvarez, G. A. (2009). Compression in visual working memory: Using statistical regularities to form more efficient memory representations. Journal of Experimental Psychology: General, 138(4), 487–502. https://doi.org/10.1037/a0016797 Brainard, D. (1997). The psychophysics toolbox. Spatial Vision, 10(4), 433–436. https://doi.org/10.1163/156856897X00357 Burke, M. R., Clarke, J. B., & Hedley, J. (2010). Effect of retinal and/or extra-retinal information on age in memory-guided saccades. Experimental Brain Research, 205(1), 87–94. https://doi.org/10.1007/s00221-010-2335-y

Cai, R., Pouget, A., Schlag-Rey, M., & Schlag, J. (1997). Perceived geometrical relationships affected by eye-movement signals. Nature, 386(6625), 601–604. https://doi.org/10.1038/386601a0 Campbell, F., & Wurtz, R. (1978). Saccadic omission: Why we do not see a grey-out during a saccadic eye movement. Vision Research, 18(10), 1297–1303. https://doi.org/10.1016/0042-6989(78)90219-5 Cohen, J. D., Perlstein, W. M., Braver, T. S., Nystrom, L. E., Noll, D. C., Jonides, J., & Smith, E. E. (1997). Temporal dynamics of brain activation during a working memory task. Nature, 386(April), 604–608. Cohen, J. D., Forman, S. D., Braver, T. S., Casey, B. J., Servan-Schreiber, D., Noll, D. C., Trainor, R. J., Giedd, J., Kaysen, D., Hertz-Pannier, L., & Rapoport, J. L. (1994). Activation of prefrontal cortex in children during a nonspatial working memory task with functional MRI. Human Brain Mapping. Colby, C. L., Duhamel, J.-R., & Goldberg, M. E. (1996). Visual, presaccadic, and cognitive activation of single neurons in monkey lateral intraparietal area. Journal of Neurophysiology, 76(5), 2841–2852. Collins, T., Rolfs, M., Deubel, H., & Cavanagh, P. (2009). Post-saccadic location judgments reveal remapping of saccade targets to non-foveal locations. Journal of Vision, 9(5), 1–9. https://doi.org/10.1167/9.5.29 Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the brain. Nature Reviews. Neuroscience, 3(3), 201–215. https://doi.org/10.1038/nrn755 Courtney, S. M., Petit, L., Haxby, J. V., & Ungerleider, L. G. (1998). The role of prefrontal cortex in working memory: Examining the contents of consciousness. Philosophical Transactions of the Royal Society B: Biological Sciences, 353(1377), 1819–1828. https://doi.org/10.1098/rstb.1998.0334 Curtis, C. E., & D’Esposito, M. (2003). Persistent activity in the prefrontal cortex during working memory. Trends in Cognitive Sciences, 7(9), 415–423. https://doi.org/10.1016/S1364-6613(03)00197-9 Deubel, H., Bridgeman, B., & Schneider, W. X. (1998). Immediate post-saccadic information mediates space constancy. Vision Research, 38(20), 3147–3159. https://doi.org/10.1016/S0042-6989(98)00048-0 Douglas, D., Thavabalasingam, S., Chorghay, Z., O’Neil, E. B., Barense, M. D., & Lee, A. C. H. (2017). Perception of impossible scenes reveals differential hippocampal and parahippocampal place area contributions to spatial coherency. Hippocampus, 27(1), 61– 76. https://doi.org/10.1002/hipo.22673 Drachman, D. A., & Arbit, J. (1966). Memory and the hippocampal complex: II. Is memory a multiple process? Archives of Neurology, 15(1), 52–61. https://doi.org/10.1001/archneur.1966.00470130056005 Duhamel, J., Colby, C., & Goldberg, M. (1992). The updating of the representation of visual space in parietal cortex by intended eye movements. Science, 255, 90–92. Enright, J. T. (1995). The non-visual impact of eye orientation on eye-hand coordination. Vision Research, 35(11), 1611–1618.

Feigenbaum, J. D., & Morris, R. G. (2004). Allocentric versus egocentric spatial memory after unilateral temporal lobectomy in humans. Neuropsychology, 18(3), 462–472. https://doi.org/10.1037/0894-4105.18.3.462 Frost, A., & Niemeier, M. (2015). Suppression and reversal of motion perception around the time of the saccade. Frontiers in Systems Neuroscience, 9(October), 143. https://doi.org/10.3389/fnsys.2015.00143 Frost, A., Tomou, G., Parikh, H., Kaur, J., Zivcevska, M., & Niemeier, M. (2019). Working memory in action: inspecting the systematic and unsystematic errors of spatial memory across saccades. Experimental Brain Research, 237(11), 2939–2956. https://doi.org/10.1007/s00221-019-05623-x Fukuda, K., Vogel, E., Mayr, U., & Awh, E. (2010). Quantity, not quality: The relationship between fluid intelligence and working memory capacity. Psychonomic Bulletin and Review, 17(5), 673–679. https://doi.org/10.3758/17.5.673 Gevins, A. S., Bressler, S. L., Cutillo, B. A., Illes, J., Miller, J. C., Stern, J., & Jex, H. R. (1990). Effects of prolonged mental work on functional brain topography. Electroencephalography and Clinical Neurophysiology, 76(4), 339–350. https://doi.org/10.1016/0013-4694(90)90035-I Gevins, A., & Cutillo, B. (1993). Spatiotemporal dynamics of component processes in human working memory. Electroencephalography and Clinical Neurophysiology, 87(3), 128– 143. https://doi.org/10.1016/0013-4694(93)90119-G Gold, J. M., Hahn, B., Zhang, W. W., Robinson, B. M., Kappenman, E. S., Beck, V. M., & Luck, S. J. (2010). Reduced capacity but spared precision and maintenance of working memory representations in schizophrenia. Archives of General Psychiatry, 67(6), 570–577. https://doi.org/10.1001/archgenpsychiatry.2010.65 Goldman-Rakic, P. S. (1995). Cellular basis of working memory. Neuron, 14(3), 477–485. https://doi.org/10.1016/0896-6273(95)90304-6 Greenlee, M. W., Magnussen, S., & Reinvang, I. (2000). Brain regions involved in spatial frequency discrimination: Evidence from fMRI. Experimental Brain Research, 132(3), 399–403. https://doi.org/10.1007/s002210000385 Grimes, J. (1996). On the failure to detect changes in scenes across saccades. In Perception (pp. 89–110). Oxford University Press. https://doi.org/10.1093/acprof Haatveit, B. C., Sundet, K., Hugdahl, K., Ueland, T., Melle, I., & Andreassen, O. A. (2010). The validity of d prime as a working memory index: Results from the Bergen n-back task. Journal of Clinical and Experimental Neuropsychology, 32(8), 871–880. https://doi.org/10.1080/13803391003596421 Hallett, P. E., & Lightstone, A. D. (1976). Saccadic eye movements towards stimuli triggered by prior saccades. Vision Research, 16(1), 99–106. https://doi.org/10.1016/0042- 6989(76)90083-3 Harris, C. M. (1995). Does saccadic undershoot minimize saccadic flight-time? A Monte-Carlo study. Vision Research, 35(5), 691–701. https://doi.org/10.1016/0042-6989(94)00163-G

Henriques, D. Y., Klier, E. M., Smith, M. A., Lowy, D., & Crawford, J. D. (1998). Gaze- centered remapping of remembered visual space in an open-loop pointing task. Journal of Neuroscience, 18(4), 1583–1594. Honda, H. (1989). Perceptual localization of visual stimuli flashed during saccades. Perception & Psychophysics, 45(2), 162–174. https://doi.org/10.3758/BF03208051 Honda, H. (1993). Saccade-contingent displacement of the apparent position of visual stimuli flashed on a dimly illuminated structured background. Vision Research, 33(5–6), 709– 716. https://doi.org/10.1016/0042-6989(93)90190-8 Ikkai, A., McCollough, A. W., & Vogel, E. K. (2010). Contralateral delay activity provides a neural measure of the number of representations in visual working memory. Journal of Neurophysiology, 103(4), 1963–1968. https://doi.org/10.1152/jn.00978.2009 Ipata, A. E., Gee, A. L., Goldberg, M. E., & Bisley, J. W. (2006). Activity in the lateral intraparietal area predicts the goal and latency of saccades in a free-viewing visual search task. Journal of Neuroscience, 26(14), 3656–3661. https://doi.org/10.1523/JNEUROSCI.5074-05.2006 Irwin, D. E. (1991). Information integration across saccadic eye movements. Cognitive Psychology, 23, 420–456. https://doi.org/10.1167/14.1.28. Jaeggi, S. M., Buschkuehl, M., Perrig, W. J., & Meier, B. (2010). The concurrent validity of the n-back task as a working memory measure. Memory, 18(4), 394–412. https://doi.org/10.1080/09658211003702171 Jiang, Y. V., Lee, H. J., Asaad, A., & Remington, R. (2016). Similarity effects in visual working memory. Psychonomic Bulletin and Review, 23(2), 476–482. https://doi.org/10.3758/s13423-015-0905-5 Jiang, Y., Olson, I. R., & Chun, M. M. (2000). Organization of visual short-term memory. Journal of Experimental Psychology: Learning Memory and Cognition, 26(3), 683–702. https://doi.org/10.1037/0278-7393.26.3.683 Johnson, M. K., McMahon, R. P., Robinson, B. M., Harvey, A. N., Hahn, B., Leonard, C. J., Luck, S. J., & Gold, J. M. (2013). The relationship between working memory capacity and broad measures of cognitive ability in healthy adults and people with schizophrenia. Neuropsychology, 27(2), 220–229. https://doi.org/10.1037/a0032060 Jonides, J., Smith, E. E., Koeppe, R. A., Awh, E., Minoshima, S., & Mintun, M. A. (1993). Spatial working-memory in humans as revealed by PET. Nature, 363(6430), 623–625. https://doi.org/10.1038/363623a0 Jonides, J. (1981). Voluntary vs. automatic control of the mind’s eye’s movement. In Attention and performance (pp. 187–204). Judge, S., Wurtz, R., & Richmond, B. (1980). Vision during saccadic eye movements. I. Visual interactions in striate cortex. Journal of Neurophysiology, 43(4), 1133–1155. Kaiser, M., & Lappe, M. (2004). Perisaccadic mislocalization orthogonal to saccade direction. Neuron, 41(2), 293–300. https://doi.org/10.1016/S0896-6273(03)00849-3

Kane, M. J., & Engle, R. W. (2002). The role of prefrontal cortex in working-memory capacity, executive attention, and general fluid intelligence: An individual-differences perspective. Psychonomic Bulletin & Review, 9(4), 637–671. Kane, M. J., Conway, A. R. A., Miura, T. K., & Colflesh, G. J. H. (2007). Working memory, attention control, and the n-back task: A question of construct validity. Journal of Experimental Psychology: Learning Memory and Cognition, 33(3), 615–622. https://doi.org/10.1037/0278-7393.33.3.615 Karn, K. S., Møller, P., & Hayhoe, M. M. (1997). Reference frames in saccadic targeting. Experimental Brain Research, 115(2), 267–282. https://doi.org/10.1007/PL00005696 Kis, A., Singh, V., & Niemeier, M. (2009). Short- and long-term plasticity of eye position information: Examining perceptual, attentional, and motor influences on perisaccadic perception. Journal of Vision, 9(6), 11.1-20. https://doi.org/10.1167/9.6.11 Kleiner, M., Brainard, D. H., & Pelli, D. G. (2007). What’s new in Psychtoobox-3? Perception. Lambert, A., Spencer, E., & Mohindra, N. (1987). Automaticity and the capture of attention by a peripheral display change. Current Psychology, 6(2), 136–147. https://doi.org/10.1007/BF02686618 Lappe, M., Awater, H., & Krekelberg, B. (2000). Postsaccadic visual references generate presaccadic compression of space. Nature, 403(6772), 892–895. https://doi.org/10.1038/35002588 Lewis, R. L., Zee, D. S., Hayman, & Tamargo, R. (2001). Oculomotor function in the rhesus monkey after deafferentation of the extraocular muscles. Experimental Brain Research, 141(3), 349–358. https://doi.org/10.1007/s002210100876 Luck, S. J., & Vogel, E. K. (1997). The capacity of visual working memory for features and conjunctions. Nature, 390(6657), 279–281. https://doi.org/10.1038/36846 Luck, S. J., & Vogel, E. K. (2013). Visual working memory capacity: From psychophysics and neurobiology to individual differences. Trends in Cognitive Sciences, 17(8), 391–400. https://doi.org/10.1016/j.tics.2013.06.006 Ma, W. J., Husain, M., & Bays, P. M. (2014). Changing concepts of working memory. Nature Neuroscience, 17(3), 347–356. https://doi.org/10.1038/nn.3655 Mackay, D. (1970). Elevation of visual threshold by displacement of retinal image. Nature, 225(5227), 90–92. https://doi.org/10.1038/225090a0 Maronna, R. A., Martin, R. D., & Yohai, V. J. (2006). Robust Statistics: Theory and Methods. Wiley. https://doi.org/10.1002/0470010940 Martin, C. B., Douglas, D., Newsome, R. N., Man, L. L. Y., & Barense, M. D. (2018). Integrative and distinctive coding of visual and conceptual object features in the ventral visual stream. ELife, 7, 1–29. https://doi.org/10.7554/eLife.31873 Matin, E., Clymer, A., & Matin, L. (1972). Metacontrast and saccadic suppression. Science, 178(57), 179–182. https://doi.org/10.1126/science.178.4062.796 Matin, L., & Pearce, D. (1965). Visual perception of direction for stimuli flashed during voluntary saccadic eye movements. Science, 148(3676), 1485–1488. https://doi.org/10.1126/science.148.3676.1485

Maule, J., & Franklin, A. (2015). Effects of ensemble complexity and perceptual similarity on rapid averaging of hue. Journal of Vision, 15(4), 1–18. https://doi.org/10.1167/15.4.6 McPeek, R. M., & Keller, E. L. (2002). Saccade target selection in the superior colliculus during a visual search task. Journal of Neurophysiology, 88(4), 2019–2034. Medendorp, W. P., Tweed, D. B., & Crawford, J. D. (2003). Motion parallax is computed in the updating of human spatial memory. In Journal of Neuroscience (Vol. 23, Issue 22). http://www.ncbi.nlm.nih.gov/pubmed/12954876 Merriam, E. P., Genovese, C. R., & Colby, C. L. (2007). Remapping in human visual cortex. Journal of Neurophysiology, 97, 1738–1755. https://doi.org/10.1152/jn.00189.2006. Merriam, E. P., Genovese, C. R., & Colby, C. L. (2003). Spatial updating in human parietal cortex. Neuron, 39(2), 361–373. https://doi.org/10.1016/S0896-6273(03)00393-3 Miller, G. A. (1956). The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological Review, 63(2), 81–97. https://doi.org/10.1037/h0043158 Morrone, M., Ross, J., & Burr, D. (1997). Apparent position of visual targets during real and simulated saccadic eye movements. The Journal of Neuroscience, 17(20), 7941–7953. https://doi.org/10.1038/nn1488 Nemrodov, D., Niemeier, M., Mok, J. N. Y., & Nestor, A. (2016). The time course of individual face recognition: A pattern analysis of ERP signals. NeuroImage, 132, 469–476. https://doi.org/10.1016/j.neuroimage.2016.03.006 Niemeier, M., Crawford, J. D., & Tweed, D. (2007). Optimal inference explains dimension- specific contractions of spatial perception. Experimental Brain Research, 179(2), 313– 323. https://doi.org/10.1007/s00221-006-0788-9 Niemeier, M., Crawford, J. D., & Tweed, D. (2003). Optimal transsaccadic integration explains distorted spatial perception. Nature, 422(6927), 76–80. https://doi.org/10.1038/nature01439 O’Regan, K. J., Deubel, H., Clark, J. J., & Rensink, R. A. (2000). Picture changes during blinks: Looking without seeing and seeing without looking. Visual Cognition, 7(1–3), 191–211. https://doi.org/10.1080/135062800394766 O’Regan, K. J., Rensink, R. A., & Clark, J. J. (1999). Change-blindness as a result of ‘mudsplashes.’ Nature, 398(March). Oberauer, K. (2005). Binding and inhibition in working memory: Individual and age differences in short-term recognition. Journal of Experimental Psychology: General, 134(3), 368– 387. https://doi.org/10.1037/0096-3445.134.3.368 Oberauer, K., & Lin, H. Y. (2017). An interference model of visual working memory. Psychological Review, 124(1), 21–59. https://doi.org/10.1037/rev0000044 Ogawa, T., & Komatsu, H. (2009). Condition-dependent and condition-independent target selection in the macaque posterior parietal cortex. Journal of Neurophysiology, 101(2), 721–736. https://doi.org/10.1152/jn.90817.2008

Owen, A. M., McMillan, K. M., Laird, A. R., & Bullmore, E. (2005). N-back working memory paradigm: A meta-analysis of normative functional neuroimaging studies. Human Brain Mapping, 25(1), 46–59. https://doi.org/10.1002/hbm.20131 Pelli, D. (1997). The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spatial Vision, 10(4), 437–442. https://doi.org/10.1163/156856897X00366 Phillips, W. A. (1974). On the distinction between sensory storage and short-term visual memory. Perception & Psychophysics, 16(2), 283–290. https://doi.org/10.3758/BF03203943 Posner, M. I., & Cohen, Y. (1984). Components of Visual Orienting. In D. Bouma & D. Bonwhuis (Eds.), Attention and performance X: Control of language processes (pp. 531– 556). Erlbaum. Prime, S. L., Vesia, M., & Crawford, J. D. (2008). Transcranial magnetic stimulation over posterior parietal cortex disrupts transsaccadic memory of multiple objects. Journal of Neuroscience, 28(27), 6938–6949. https://doi.org/10.1523/jneurosci.0542-08.2008 Prime, S. L., Niemeier, M., & Crawford, J. D. (2006). Transsaccadic integration of visual features in a line intersection task. Experimental Brain Research, 169(4), 532–548. https://doi.org/10.1007/s00221-005-0164-1 Prime, S. L., Tsotsos, L., Keith, G. P., & Crawford, J. D. (2007). Visual memory capacity in transsaccadic integration. Experimental Brain Research, 180(4), 609–628. https://doi.org/10.1007/s00221-007-0885-4 Prime, S. L., Vesia, M., & Crawford, J. D. (2010). TMS over human frontal eye fields disrupts trans-saccadic memory of multiple objects. Cerebral Cortex, 20(4), 759–772. https://doi.org/10.1093/cercor/bhp148 Rensink, R. A., O’Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8(5), 368–373. https://doi.org/10.1111/j.1467-9280.1997.tb00427.x Roberts, R., & Gibson, E. (2002). Individual differences in sentence memory. Journal of Psycholinguistic Research, 31(6), 573–598. https://doi.org/10.1023/A:1021213004302 Ross, J., Morrone, M., & Burr, D. (1997). Compression of visual space before saccades. Nature, 386(6625), 598–601. https://doi.org/10.1038/386598a0 Rouder, J. N., Morey, R. D., Cowan, N., Zwilling, C. E., Morey, C. C., & Pratte, M. S. (2008). An assessment of fixed-capacity models of visual working memory. Proceedings of the National Academy of Sciences of the United States of America, 105(16), 5975–5979. https://doi.org/10.1073/pnas.0711295105 Rouder, J. N., Morey, R. D., Morey, C. C., & Cowan, N. (2011). How to measure working memory capacity in the change detection paradigm. Psychonomic Bulletin and Review, 18(2), 324–330. https://doi.org/10.3758/s13423-011-0055-3 Scarborough, D. L. (1972). Stimulus modality effects of forgetting in short-term memory. Journal of Experimental Psychology, 95(2), 285–289. https://doi.org/10.1037/h0033667 Schall, J. D., & Hanes, D. P. (1993). Neural basis of saccade target selection in frontal eye field during visual search. Nature, 366(6454), 467–469. https://doi.org/10.1038/366467a0

Scoville, W. B., & Milner, B. (1957). Loss of recent memory after bilateral hippocampal lesions. Journal of Neurology, Neurosurgery, and Psychiatry, 20(11), 11–21. Shallice, T., & Warrington, E. K. (1970). Independent functioning of verbal memory stores: a neuropsychological study. The Quarterly Journal of Experimental Psychology, 22(2), 261–273. https://doi.org/10.1080/00335557043000203 Shen, K., & Paré, M. (2007). Neuronal activity in superior colliculus signals both stimulus identity and saccade goals during visual conjunction search. Journal of Vision, 7(5), 1– 13. https://doi.org/10.1167/7.5.15 Shen, K., & Paré, M. (2014). Predictive saccade target selection in superior colliculus during visual search. Journal of Neuroscience, 34(16), 5640–5648. https://doi.org/10.1523/JNEUROSCI.3880-13.2014 Sheremata, S. L., Bettencourt, K. C., & Somers, D. C. (2010). Hemispheric asymmetry in visuotopic posterior parietal cortex emerges with visual short-term memory load. Journal of Neuroscience, 30(38), 12581–12588. https://doi.org/10.1523/JNEUROSCI.2689- 10.2010 Simons, D. J., & Levin, D. T. (1998). Failure to detect changes to people during a real-world interaction. Psychonomic Bulletin & Review, 5(4), 644–649. https://doi.org/10.3758/bf03208840 Simons, D. J., & Rensink, R. A. (2005). Change blindness: Past, present, and future. Trends in Cognitive Sciences, 9(1), 16–20. https://doi.org/10.1016/j.tics.2004.11.006 Sims, C. R., Jacobs, R. A., & Knill, D. C. (2012). An ideal observer analysis of visual working memory. Psychological Review, 119(4), 807–830. https://doi.org/10.1037/a0029856 Smith, E. E., Jonides, J., & Koeppe, R. A. (1996). Dissociating verbal and spatial working memory using PET. Cerebral Cortex, 6(1), 11–20. https://doi.org/10.1093/cercor/6.1.11 Smyth, M. M., & Scholey, K. A. (1994). Interference in immediate spatial memory. Memory & Cognition, 22(1), 1–13. http://www.ncbi.nlm.nih.gov/pubmed/9080789 Sommer, M. A., & Wurtz, R. H. (2002). A pathway in primate brain for internal monitoring of movements. Science, 296(5572), 1480–1482. https://doi.org/10.1126/science.1069590 Sommer, M. A., & Wurtz, R. H. (2006). Influence of the thalamus on spatial visual processing in frontal cortex. Nature, 444(7117), 374–377. https://doi.org/10.1038/nature05279 Sommer, M. A., & Wurtz, R. H. (2004). What the brain stem tells the frontal cortex. I. Oculomotor signals sent from superior colliculus to frontal eye field via mediodorsal thalamus. Journal of Neurophysiology, 91(3), 1381–1402. https://doi.org/10.1152/jn.00738.2003 Srimal, R., & Curtis, C. E. (2008). Persistent neural activity during the maintenance of spatial position in working memory. NeuroImage, 39(1), 455–468. https://doi.org/10.1016/j.neuroimage.2007.08.040 Sun, S. Z., Fidalgo, C., Barense, M. D., Lee, A. C. H., Cant, J. S., & Ferber, S. (2017). Erasing and blurring memories: The differential impact of interference on separate aspects of forgetting. Journal of Experimental Psychology: General, 146(11), 1606–1630. https://doi.org/10.1037/xge0000359

Taylor, R., & Bays, P. M. (2018). Efficient coding in visual working memory accounts for stimulus-specific variations in recall. Journal of Neuroscience, 38(32), 7132–7142. https://doi.org/10.1523/JNEUROSCI.1018-18.2018 Ten Brink, A. F., Nijboer, T. C. W., Fabius, J. H., & Van der Stigchel, S. (2019). No direction specific costs in trans-saccadic memory. Neuropsychologia, 125, 23–29. https://doi.org/10.1016/j.neuropsychologia.2019.01.014 Theeuwes, J., Olivers, C. N. L., & Chizk, C. L. (2005). Remembering a location makes the eyes curve away. Psychological Science, 16(3), 196–199. https://doi.org/10.1111/j.0956- 7976.2005.00803.x Thomas, N. W. D., & Paré, M. (2007). Temporal processing of saccade targets in parietal cortex area LIP during visual search. Journal of Neurophysiology, 97(1), 942–947. https://doi.org/10.1152/jn.00413.2006 Tian, J., Ying, H. S., & Zee, D. S. (2013). Revisiting corrective saccades: Role of visual feedback. Vision Research, 89, 54–64. https://doi.org/10.1016/j.visres.2013.07.012 Todd, J. J., & Marois, R. (2005). Posterior parietal cortex activity predicts individual differences in visual short-term memory capacity. Cognitive, Affective, & Behavioral Neuroscience, 5(2), 144–155. Todd, J. J., & Marois, R. (2004). Capacity limit of visual short-term memory in human posterior parietal cortex. Nature, 428(6984), 751–754. https://doi.org/10.1038/nature02466 Unsworth, N., Fukuda, K., Awh, E., & Vogel, E. K. (2014). Working memory and fluid intelligence: Capacity, attention control, and secondary memory retrieval. Cognitive Psychology, 71, 1–26. https://doi.org/10.1016/j.cogpsych.2014.01.003 Van der Stigchel, S., Merten, H., Meeter, M., & Theeuwes, J. (2007). The effects of a task- irrelevant visual event on spatial working memory. Psychonomic Bulletin & Review, 14(6), 1066–1071. https://doi.org/10.3758/BF03193092 Vasquez, B., & Danckert, J. (2008). Direction specific costs to spatial working memory from saccadic and spatial remapping. Neuropsychologia, 46, 2344–2354. https://doi.org/10.1016/j.neuropsychologia.2008.03.006 Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature, 428(6984), 748–751. https://doi.org/10.1038/nature02447 Vogel, E. K., McCollough, A. W., & Machizawa, M. G. (2005). Neural measures reveal individual differences in controlling access to working memory. Nature, 438(7067), 500– 503. https://doi.org/10.1038/nature04171 Vogel, E. K., Woodman, G. F., & Luck, S. J. (2001). Storage of features, conjunctions, and objects in visual working memory. In Journal of Experimental Psychology: Human Perception and Performance (Vol. 27, Issue 1, pp. 92–114). https://doi.org/10.1037/0096-1523.27.1.92 Vuilleumier, P., Sergent, C., Schwartz, S., Valenza, N., Girardi, M., Husain, M., & Driver, J. (2007). Impaired perceptual memory of locations across gaze-shifts in patients with unilateral spatial neglect. Journal of Cognitive Neuroscience, 19(8), 1388–1406. https://doi.org/10.1162/jocn.2007.19.8.1388

100

Wager, T. D., & Smith, E. E. (2003). Neuroimaging studies of working memory: A meta- analysis. Cognitive, Affective, & Behavioral Neuroscience, 3(4), 255–274. Walker, M. F., Fitzgibbon, E. J., & Goldberg, M. E. (1995). Neurons in the monkey superior colliculus predict the visual result of impending saccadic eye movements. Journal of Neurophysiology, 73(5), 1988–2003. https://doi.org/10.1152/jn.1995.73.5.1988 Ward, E. J., Bear, A., & Scholl, B. J. (2016). Can you perceive ensembles without perceiving individuals?: The role of statistical perception in determining whether awareness overflows access. Cognition, 152, 78–86. https://doi.org/10.1016/j.cognition.2016.01.010 Watter, S., Geffen, G. M., & Geffen, L. B. (2001). The n-back as a dual-task: P300 morphology under divided attention. Psychophysiology, 38(6), 998–1003. https://doi.org/10.1111/1469-8986.3860998 White, J. M., Sparks, D. L., & Stanford, T. R. (1994). Saccades to remembered target locations: an analysis of systematic and variable errors. Vision Research, 34(1), 79–92. https://doi.org/10.1016/0042-6989(94)90259-3 Whitney, D., & Yamanashi Leib, A. (2018). Ensemble perception. Annual Review of Psychology, 69(1), 105–129. https://doi.org/10.1146/annurev-psych-010416-044232 Wurtz, R. H. (2008). Neuronal mechanisms of visual stability. Vision Research, 48(20), 2070– 2089. https://doi.org/10.1016/j.visres.2008.03.021 Xu, Y., & Chun, M. M. (2006). Dissociable neural mechanisms supporting visual short-term memory for objects. Nature, 440(7080), 91–95. https://doi.org/10.1038/nature04262 Yantis, S., & Jonides, J. (1990). Abrupt visual onsets and selective attention: Voluntary versus automatic allocation. Journal of Experimental Psychology: Human Perception and Performance, 16(1), 121–134. https://doi.org/10.1037/0096-1523.16.1.121 Zhang, W., & Luck, S. J. (2008). Discrete fixed-resolution representations in visual working memory. Nature, 453(May), 233–235. https://doi.org/10.1038/nature06860