Sensory is allocated exclusively to the current event-segment

Item Type Article

Authors Tripathy, Srimant P.; Ögmen, H.

Citation Tripathy SP and Ögmen H (2018) Sensory memory is allocated exclusively to the current event-segment. Frontiers in Psychology. 9: 1435.

Rights (c) 2018 The Authors. This is an Open Access article distributed under the Creative Commons CC-BY license (http:// creativecommons.org/licenses/by/4.0/)

Download date 28/09/2021 02:42:01

Link to Item http://hdl.handle.net/10454/16722 ORIGINAL RESEARCH published: 07 September 2018 doi: 10.3389/fpsyg.2018.01435

Sensory Memory Is Allocated Exclusively to the Current Event-Segment

Srimant P. Tripathy 1* and Haluk Ögmenˇ 2

1 School of Optometry and Vision Science, University of Bradford, Bradford, United Kingdom, 2 Department of Electrical and Computer Engineering, University of Denver, Denver, CO, United States

The Atkinson-Shiffrin modal model forms the foundation of our of human memory. It consists of three stores (Sensory Memory (SM), also called iconic memory, Short-Term Memory (STM), and Long-Term Memory (LTM)), each tuned to a different time-scale. Since its inception, the STM and LTM components of the modal model have undergone significant modifications, while SM has remained largely unchanged,

Edited by: representing a large capacity system funneling information into STM. In the laboratory, Qasim Zaidi, visual memory is usually tested by presenting a brief static stimulus and, after a delay, University at Buffalo, United States asking observers to report some aspect of the stimulus. However, under ecological Reviewed by: Ronald A. Rensink, viewing conditions, our visual system receives a continuous stream of inputs, which University of British Columbia, Canada is segmented into distinct spatio-temporal segments, called events. Events are further Stephen Emrich, segmented into event-segments. Here we show that SM is not an unspecific general Brock University, Canada funnel to STM but is allocated exclusively to the current event-segment. We used a *Correspondence: Srimant P. Tripathy Multiple-Object Tracking (MOT) paradigm in which observers were presented with disks [email protected] moving in different directions, along bi-linear trajectories, i.e., linear trajectories, with a single deviation in direction at the mid-point of each trajectory. The synchronized deviation Specialty section: This article was submitted to of all of the trajectories produced an event stimulus consisting of two event-segments. Science, Observers reported the pre-deviation or the post-deviation directions of the trajectories. a section of the journal Frontiers in Psychology By analyzing observers’ responses in partial- and full-report conditions, we investigated the involvement of SM for the two event-segments. The hallmarks of SM hold only for the Received: 14 October 2017 Accepted: 23 July 2018 current event segment. As the large capacity SM stores only items involved in the current Published: 07 September 2018 event-segment, the need for event-tagging in SM is eliminated, speeding up processing Citation: in active vision. By characterizing how memory systems are interfaced with ecological Tripathy SP and Ögmenˇ H (2018) Sensory Memory Is Allocated events, this new model extends the Atkinson-Shiffrin model by specifying how events Exclusively to the Current are stored in the first stage of multi-store memory systems. Event-Segment. Front. Psychol. 9:1435. Keywords: memory, sensory memory, iconic memory, short-term memory, modal model of memory, event doi: 10.3389/fpsyg.2018.01435 segmentation, tracking, multiple-object tracking

Frontiers in Psychology | www.frontiersin.org 1 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments

INTRODUCTION Herzog, 2016). According to this model, SM has two parallel components, one based on a retinotopic reference-frame (rSM) The classical multi-store (or modal) model of human memory and a second one based on motion-grouping based (non- (Atkinson and Shiffrin, 1968, 1971) posits that information is retinotopic) reference frames (nrSM). encoded and stored in three memory systems (Figure 1A): First, Another recent modification to the multi-store functional a large-capacity but rapidly decaying SM stores information for architecture of memory systems has been the introduction of a few 100 ms. A subset of the contents of SM is transferred into a new visual memory component, intermediate between SM a more durable STM, which can store items for a few seconds. and STM (Sligte et al., 2008; van Moorselaar et al., 2015). However, STM’s capacity is severely limited. Finally, some items This component is proposed to have large capacity and a in STM are transferred into LTM whose contents can last as long retention time in the order of several seconds. It was suggested as a lifetime. that this memory component lacks robustness with respect Since its inception, the STM and LTM components of to changing inputs and hence was termed fragile visual STM. the modal model have undergone significant modifications Evidence suggests that the erasure of the contents of the (Baddeley and Hitch, 1974; Klatzky, 1980; Cowan, 2001; Alvarez fragile STM depends on both location-specific and object-specific and Cavanagh, 2004; Baddeley, 2007; Bays and Husain, 2008; matches between the contents and the new input, leading to Zhang and Luck, 2008; Van den Berg et al., 2012), while the the proposal that this memory component holds higher object- classical view of SM as a large capacity system funneling a static level representations but that these representations are anchored snapshot of visual inputs into STM has persisted for several to specific locations (Pinto et al., 2013). However, other studies decades (Sperling, 1960; Averbach and Coriell, 1961; Coltheart, have questioned the existence of fragile VSTM (Matsukura and 1983; Haber, 1983; Breitmeyer and Ögmen,ˇ 2006). Recent models Hollingworth, 2011; Makovski, 2012). extended SM in terms of its architecture and function. In the laboratory, visual memory is usually tested by In terms of how SM is implemented in the brain, recently presenting a brief static stimulus (e.g., shapes with different a multi-layer approach reflecting the hierarchy of the visual colors) and, after a delay, asking observers to report some cortex has been proposed (Rensink, 2014). It was found that aspect of the stimulus. However, under normal ecological viewing the retention-duration in SM depends on the nature (but not conditions, due to the movements of the observer and those on the difficulty) of the task that SM is called to subserve of objects in the environment, our visual system receives a (Rensink, 2014). For change detection, target-identity report, continuous stream of dynamic inputs. Perceptually, the visual and static detection tasks, the corresponding SM retention- system segments the continuous stream of visual information durations were 120, 190, and 240ms, respectively. This finding into distinct spatio-temporal segments, called events (Johansson was interpreted to reflect a multi-layer cortical implementation et al., 1980; Warren and Shaw, 1985). For example, flipping a of SM, according to which SM can store information and make it coin can be considered as an event, which in turn can have sub- available both within and across the cortical layers by relying on segments (Zacks and Swallow, 2007), such as the launching of horizontal and vertical feedback, respectively. Hence, according the coin, following the movement of the coin in the air, and to this model, the architecture of SM is distributed across the catching the coin, all occurring within a second. An important cortex following the hierarchy of visual cortical processes. This question is then how events and their segments are stored in distributed implementation suggests that information flow from SM. Previous studies examined how information embedded in SM to STM and to other visual processes occurs within and across events is stored in STM and LTM. It has been reported that cortical layers because, architecturally, SM is an integral part of objects that were present around event boundaries were better various networks. recalled than other objects in the stimulus (Swallow et al., Another aspect of cortical organization is retinotopy: Via 2009). This finding has been interpreted as evidence that event the optics of the eyes, neighboring points in the environment boundaries structure the contents of STM and LTM (Swallow are mapped to neighboring points on the and these et al., 2009). Given this background, our goal was to analyse the neighborhood relations are preserved in early visual cortical relationship between event segments and memory mechanisms, areas. Under ecological viewing conditions, the observer (eyes, in particular for relatively short events that span the first two head, body) and many objects in the environment are in stages of the modal model, namely SM and STM. We focused motion. While a retinotopically encoded SM can store and on these two memory stores because, due to their time-scales, process information that is stable (i.e., static) with respect they are inherently involved in real-time ecological aspects of to its retinotopic reference-frame, the and processing perception and and because they can be distinguished of dynamic information is problematic (Haber, 1983): Any from each other by well-established experimental techniques relative motion between the observer’s retinae and the external (Sperling, 1960; Averbach and Coriell, 1961; Coltheart, 1983; environment will cause a shift in retinotopic coordinates Haber, 1983; Breitmeyer and Ögmen,ˇ 2006). for the stimulus received by SM. These shifts, in turn, will Stimuli used to probe events can range in complexity cause spatiotemporal blurring and inappropriate integration from a few geometric shapes in motion (Zacks et al., 2006) of information over space and time (Ögmen,ˇ 2007; Ögmenˇ to complex movie scenes (Swallow et al., 2009). Here, in and Herzog, 2010). Based on a series of experiments that order to probe memory mechanisms underlying events, we show non-retinotopic storage and processing of information, used relatively simple stimuli, namely a of moving disks recently a modified SM model has been suggested (Ögmenˇ and undergoing a synchronized trajectory-change in mid-course

Frontiers in Psychology | www.frontiersin.org 2 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments

FIGURE 1 | Models of human memory. (A) The modal model: The environmental input is received by SM whose contents are transferred to STM and LTM. (B) Working of the modal model in ecological viewing where the environmental input is a continuous stream of signals that are segmented into events containing sub-events, i.e., segments. The observer encodes in memory not only stimuli present within the event but for each stimulus a tag that indicates to which segment it belongs. The contents of the memory are depicted by triangles whose color indicates segment tags. (C) An alternative model where SM is exclusively allocated to the current segment of the event. When event segmentation is driven by low-level stimulus changes, transients may be used to mask and reset SM (Averbach and Coriell, 1961; Breitmeyer and Ögmen,ˇ 2006) so that it can now be allocated to the current event segment. Additional mechanisms may be involved when event segmentation is based on higher-level criteria. The model in (B) has the advantage that the large capacity SM can be used for all segments of an event within the time-span of SM, while the model in (C) can use the capacity of SM only for the current event. However, the advantage of this model is that it requires less information to be stored in SM and requires much fewer tagging operations since the contents of the large capacity SM are exclusively allocated to the current event-segment and do not need tagging. The tables to the right of (B,C) show the predictions of these models and the experimental support for the predictions. The model in (B) involves SM for both pre- and post-deviation segments and thus, as indicated by “YES” entries, predicts that properties of SM should be found for both segments. The model in (C), on the other hand predicts that properties of SM should be found only for the current event segment. Thus, the models differ in their predictions for the pre-deviation segment. The green and red colors in each box indicate experimental agreement and disagreement, respectively, with the predictions.

Frontiers in Psychology | www.frontiersin.org 3 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments

(Figure 2 and demos). This synchronized trajectory-change these features and objects belong to. In other words, contents formed an event boundary clearly splitting the event into two of memory need to be “tagged” or associated with segments temporally successive pre- and post-deviation segments. of events. We consider two alternatives for the operation of When an event contains multiple segments, observers need to SM. According to one model (Figure 1B), which we call the remember not only features and objects, but also which segment “Shared Model”, tagging can take place both in SM and STM allowing the brain to use both mechanisms simultaneously for different segments of an event. While this has the advantage of providing two types of memory systems to each event segment, its disadvantage is that the complexity of both SM and STM will be high since they will both require tagging operations. The cost of tagging is especially critical for SM compared to STM since it has a much higher capacity than STM and thus requires extensively more tagging operations. That tagging operations take place in SM appear to be supported by a recent study where observers were presented with two temporally successive arrays of letters and were cued to selectively report the first or the second array (Smith et al., 2011). However, as we show in Supplementary Information, the findings of that study can also be explained by an alternative model (Figure 1C) that restricts tagging operations only to STM and uses SM as an untagged buffer for the current event segment. We call this model the “Exclusive Model.” In addition to reducing the complexity for tagging, using SM as an untagged buffer for the current event segment enhances the speed of retrieval. If the only items in SM are from the current event segment, then when a response is needed for an item in the current event segment, there is no need to check tags. This speed advantage is especially critical for SM since its contents decay relatively fast, and any time spent on tagging and tag-checking operations would lead to severe information loss. The models illustrated in Figures 1B,C can be distinguished by examining the involvement of SM in the of information available from the previous event-segment (i.e., before the deviations in the trajectories) and from the current event- segment (i.e., after the deviation in the trajectories). The Shared Model in Figure 1B predicts the involvement of SM for the recall of information related to the previous and the current event- segment, whereas the Exclusive Model in Figure 1C predicts the involvement of SM for the recall of information related to the current event-segment only. To test these sets of predictions we looked for the key signatures for the involvement of SM when recalling information. These signatures are: (i) a decay of recalled information with time; (ii) a better recall of information FIGURE 2 | Schematic of a 800 ms stimulus in Experiment 1. Each trial during single-report, compared to during full-report; and (iii) started with three randomly positioned stationary disks presented on the single-report performance that is comparable to the performance screen. The observer indicated their readiness with a mouse-click, which was for reporting the first item during full-report. The tables in followed with the sequence shown in the figure. After a static viewing period of 1,000 ms, the disks moved along straight lines for 400 ms, deviated by Figures 1B,C predict the presence (“YES”) and absence (“NO”) 30–180◦ clockwise or counter-clockwise, and then moved along straight lines of these signatures for the two models. for a further 400 ms. Blue arrows show the directions of motion of the disks The experimental paradigm used is a variation of the and dashed blue lines show their recent trajectories. Dotted lines illustrate one Multiple Object Tracking (MOT) paradigm (Pylyshyn and set of potential trajectories of the disks, but were not shown in the actual Storm, 1988; Tripathy and Barrett, 2003, 2004; Tripathy et al., stimulus. After 800 ms the motion ceased and the observer was cued to report the direction(s) of motion. The cue could query one of the following: 2007; Narasimhan et al., 2009; Shooner et al., 2010; Tripathy and post-deviation SR (illustrated in figure), post-deviation FR, pre-deviation SR or Howard, 2012; Ögmenˇ et al., 2013; Huynh et al., 2015, 2017). pre-deviation FR (see text and demos). The observer reported the direction(s) The stimuli consisted of a set of moving disks that underwent of motion using a mouse-controlled vector at each disk cued. After all reports a synchronized change in trajectories mid-way through their were recorded, feedback was provided using red and blue vectors for the trajectories, yielding two event-segments, referred to as the actual and reported directions of motion for each cued disk. previous and the current event-segments. Observers reported the

Frontiers in Psychology | www.frontiersin.org 4 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments direction(s) of motion of the trajectories in each event-segment, presented on the screen. Observers indicated their readiness with in conditions of single-report and full-report. The errors in a mouse-click, which was followed after a gap of 1 s with the reporting the direction of motion in the different conditions were sequence shown in Figure 2. Each disk moved along a straight transformed to performance measures and these were compared line, deviated by 30–180◦ clockwise or counter-clockwise at the to look for the signatures of SM as listed in the above paragraph. mid-point of its trajectory, and then moved along a straight The different experiments varied parameters of the stimuli, such line for the remainder of the trajectory. The deviations of all as: duration of motion, trajectory-length, cue-delay and set-size. trajectories occurred synchronously exactly midway through the In each case the role of SM was investigated to distinguish trajectory, with the deviations for the different trajectories on between the two models proposed. A summary of the models a trial varying randomly over the above range. The starting can be visualized in the color-coded tables in Figures 1B,C, points of the trajectories did not overlap and the trajectories where green indicates a match and red a mismatch between were constrained so that they terminated before the disks reached model predictions and experimental results. The findings largely the edge of the display, i.e., the disks did not disappear outside support the Exclusive Model presented in Figure 1C and the the visible area of the screen, nor did they bounce off the edge proposition that sensory memory is allocated exclusively to the of the display. When the motion had ceased the observer was current event-segment. cued to report the direction(s) of motion. To minimize confusion when reporting directions none of the 2N directions of motion ◦ MATERIALS AND METHODS present in each stimulus were within 20 of one another. When trajectories intersected, disks moved across each other without Observers changing direction or speed. Ten observers participated in the experiments, with some observers participating in more than one experiment. This study Reporting Conditions was carried out in accordance with the recommendations of Each trial involved one of four possible reporting conditions University of Bradford Committee for Ethics in Research with (see Figure 2). Observers could be queried on the direction(s) written informed consent from all subjects. All subjects gave of motion of: a randomly selected pre-deviation trajectory, written informed consent in accordance with the Declaration a randomly selected post-deviation trajectory, all of the pre- of Helsinki. They had normal visual acuity. They were highly deviation trajectories, or all of the post-deviation trajectories. trained, undergoing a few thousand trials in each task before These are referred to as the pre-deviation single report (SR), post- actual data collection. This ensured that any absence of evidence deviation SR, pre-deviation full report (FR), or post-deviation FR of use of SM was not simply because performance was poor on conditions. The reporting condition was cued immediately after account of lack of adequate familiarity with the task. In addition, the termination of stimulus motion as follows: all observers were younger than 32 years at the time of testing. Pre-deviation SR Trials This ensured that the age-related decline in performance for tracking that kicks in very early in adulthood (Kennedy et al., To provide an effective position cue for the pre-deviation 2009) did not compromise performance in our observers in the segment, each disk disappeared at the end of the trajectory and current study. reappeared at the point of its deviation. The disk marked for report was blue in color, the other disks being gray as before.

Equipment Post-deviation SR Trials Experiments used programs written in C++, running on a Each disk remained visible and stationary when motion Dell Precision 670 computer, along with Cambridge Research terminated. The disk marked for report turned red. System’s ViSaGe Visual Stimulus Generation hardware. Stimuli ◦ were viewed from a distance of 1m on a 22.5 × 17 CRT Sony Pre-deviation FR Trials FD Trinitron computer with a frame rate of 100 Hz. The visible To provide an effective position cue for the pre-deviation area of the monitor screen was 39.5cm (800 pixels) wide and segment, each disk disappeared at the end of the trajectory and 29.5cm (600 pixels) high. Each screen pixel subtended 1.694min reappeared at the point of its deviation. All of the disks were of arc in the horizontal and vertical directions. The background marked blue to indicate that they were to be reported. luminance of the screen was 64.9 cd/m2. Chin-and-forehead rests stabilized the head and the viewing distance. A computer mouse Post-deviation FR Trials was used to initiate each trial and to report the direction(s) All disks remained stationary and turned red to indicate that they of motion. The room was illuminated with regular fluorescent were to be reported. lighting in order to ensure that the screen-persistence could not All four reporting conditions were equally likely within a be utilized when responding to stimuli. block, which consisted of 10 trials of each condition randomly interleaved. Experiments 1a,b and 3 involved 1600 trials (4 Stimuli and Procedure stimulus conditions × 10 blocks × 40 trials/block) for each The stimuli consisted of N (= 1, 2, 3, or 4) gray disks of diameter observer. Experiment 2 with 6 stimulus conditions involved 2400 1◦ moving along random bilinear trajectories at a constant speed, trials. which was typically 5◦/s (Figure 2 and demos). The duration When the mouse-cursor was moved close to a marked disk at of motion was typically 800 ms, but was varied in Experiment the end of a trial a pointer appeared linking the disk to the cursor. 1b. Each trial started with randomly positioned stationary disks The direction of the pointer could be adjusted and reported by

Frontiers in Psychology | www.frontiersin.org 5 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments moving and clicking the mouse. This resulted in the reported disk were violated, a Greenhouse-Geisser correction was applied and turning gray. If more items remained to be reported, as in full- the correction-factor (ε) is shown in the degrees of freedom report, the reporting cycle was repeated. When all marked items column in Table 1. had been reported feedback was provided in the form of color- coded vectors attached to each marked item showing the actual Experimental Conditions and reported directions of motion. On full-report trials observers In the first experiment, we fixed the cue delay for the post- were permitted to report the marked items in the order of their deviation segment to 0ms (i.e., the cue appeared immediately at choosing and the direction of motions reported and the order of the termination of motion). For the pre-deviation segment, this reporting were both recorded. corresponded to a cue delay equal to half the stimulus duration. Data Analysis To obtain different cue delays for the pre-deviation segment, we varied stimulus duration. In general, with increased cue delay, Each error, δ, was transformed (Shooner et al., 2010; Ögmenˇ et al., SM performance decays (Sperling, 1960; Averbach and Coriell, 2013; Huynh et al., 2015, 2017) using the equation: 1961; Coltheart, 1983; Haber, 1983; Breitmeyer and Ögmen,ˇ 2006; Shooner et al., 2010; Ögmenˇ et al., 2013; Huynh et al., 2015, Transformed Performance TP = 1 −|δ|/180 2017). In our stimulus, however, since stimulus duration co- varies with cue delay, this decay may be countered by an increase This linear transformation converts the error into a probability- in signal strength through the increase in stimulus duration. To like number, with perfect performance and chance performance disentangle the two effects, first we examined performance for represented by 1.0 and 0.5 respectively. TP was averaged the post-deviation segment to assess signal strength as a function separately for each of the 4 reporting conditions for each stimulus of duration. For the post-deviation segment, memory is not condition. For the full-report trials, averages were also obtained expected to decay as a function of duration since cue delay is for FR1, FR2, FR3, and FR4 (where applicable), the first, second, fixed. Thus, any increase in signal strength should be reflected as third and fourth items to be reported, respectively. improved performance as a function of duration. Two conditions Some studies have used RMS error to analyse their results were run, one with constant motion speed (Experiment 1a) and and a question that arises is would our results be any different one with constant motion trajectory length (Experiment 1b). had we used RMS error instead of TP. We simulated 1,000 trials with an RMS error of 1◦ and calculated the TP for the same Constant Speed Condition (Experiment 1a) 1,000 data. We repeated the simulations for 21 different values of Three disks moved along bilinear trajectories at a constant speed RMS error in the in the range 1–90◦. The corresponding values of 5◦/s for a total duration of 200, 400, 800 or 1200ms. The of TP ranged from 0.995 to 0.597 and the co-efficient of linear different durations were tested in different blocks in randomized correlation between the RMS errors and their corresponding TP order. values was −0.9999. While it might be theoretically possible to generate non-random data sets where the relationship between the two measures is not that linear, for most data encountered in Constant Trajectory Length Condition practical situations, we expect an almost perfect linear correlation (Experiment 1b) between RMS error and TP. As for the results of statistical tests The three disks moved for the same durations as above, but conducted, TP is a linear transformation of RMS error (and vice the speed of the disks co-varied with the duration of motion so ◦ versa) and any statistical conclusion reached by analysing data that the disks moved through a constant trajectory length of 4 , using one measure should be valid if the other measure were to regardless of the duration or speed of motion. be used instead. We preferred to use TP as a measure of error as its use permits Post-deviation Delay Experiment (Experiment 2) comparison of our data with that from our earlier studies that A delay was introduced between the termination of the motion used the same measure (e.g., Shooner et al., 2010). Additionally of the disks and the appearance of the cue (i.e., the change of the use of TP is intuitively appealing as it is a normalized measure color of the marked disks) which indicated to the observers which that permits comparisons across experiments and also because trajectory or trajectories to report. The stimuli were the same as of its similarity to probability, with 1.0 and 0.5 representing in Experiment 1 except that the stimuli had a fixed duration of perfection and chance levels. 800ms and on the post-deviation trials there was a cue-delay of 0, 100, 200, 400, 800, or 1,600ms. This cue-delay on the post- Statistical Analysis deviation trials was fixed within a block and varied between Each experiment had 4 or 5 observers, in line with sample blocks. The cue on the pre-deviation trials appeared immediately sizes used in similar studies (Shooner et al., 2010; Ögmenˇ at the termination of motion, corresponding to a cue-delay of et al., 2013; Huynh et al., 2015, 2017). All observers within an 400ms for the pre-deviation segment. experiment participated in all of the experimental conditions for that experiment. Most statistical analyses were performed Set-Size Experiment (Experiment 3) on TP by using RM-ANOVA with significance threshold set One, two, three, or four disks moved at a speed of 5◦/s for 800 ms. to 0.05 (see Table 1). Mauchly’s Test was used to check if The different set-sizes were tested in different blocks in random sphericity assumptions were met. When sphericity assumptions order.

Frontiers in Psychology | www.frontiersin.org 6 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments

TABLE 1 | Statistical results.

2 Statisticaltest Row Comparison Experiment Degrees of F-value p-value ηP Shared Exclusive number freedom model model (Figure 1B) (Figure 1C)

TPSR > TPFR 2-way Repeated 1 Post-deviation Exp 1a 1,3 76.683 0.003 0.962 √ √ Measures ANOVA 2 Exp 1b 1,3 72.519 0.003 0.96 √ √ 3 Exp2 1,4 15.963 0.016 0.8 √ √ 4 Exp3 1,3 43.284 0.007 0.935 √ √ 5 Exp3 3 t = 5.23 0.013 √ √ SS4

6 Pre-deviation Exp 1a 1,3 35.118 0.01 0.921 √ X 7 Exp1b 1,3 4.002 0.139 0.572 X √ 8 Exp2 4 t =0.153 0.886 X √ 9 Exp3 1,3 4.535 0.123 0.602 X √ 10 Exp3 3 t = 1.821 0.166 X √ SS4

TPSR = TPFR1 11 Post-deviation Exp1a 1,3 2.661 0.201 0.47 √ √ 12 Exp1b 1,3 0.965 0.398 0.243 √ √ 13 Exp2 1,4 17.987 0.013 0.818 √ √ 14 Exp3 1,3 0.772 0.444 0.205 √ √ 15 Exp3 3 t = 1.038 0.376 √ √ SS4

16 Pre-deviation Exp 1a 1,3 29.667 0.012 0.908 X √ 17 Exp1b 1,3 16.892 0.026 0.849 X √ 18 Exp2 4 t = 3.389 0.028 X √ 19 Exp3 1,3 10.744 0.047 0.782 X √ 20 Exp3 3 t = 3.453 0.041 X √ SS4

Duration and Cue-delay 21 Post-deviation Exp 1a 3,9 91.411 < 0.001 0.968 22 Exp1b 3,9 6.567 0.012 0.686 23 Exp2 5,20 14.418 <0.001 0.783 √ √

24 Pre-deviation Exp 1a 3,9 23.603 <0.001 0.887 25 Exp1b 3,9 1.728 0.23 0.366 X √

Duration 3-way Repeated 26 Exp 1a 3,9 107.998 <0.001 0.973 (ε = 0.411) Measures ANOVA 27 Exp 1b 3,9 3.077 0.083 0.506 (ε = 0.394) 28 Exp3 3,9 201.348 <0.001 0.985 (ε = 0.607)

EventSegment 29 Exp1a 1,3 64.38 0.004 0.955 30 Exp1b 1,3 80.758 0.003 0.964 31 Exp3 1,3 967.696 <0.001 0.997

ReportType (SR, FR) 32 Exp1a 1,3 75.189 0.003 0.962 33 Exp1b 1,3 27.866 0.013 0.903 34 Exp3 1,3 20.743 0.02 0.874 EventSegment*ReportType (SR, FR) 35 Exp1a 1,3 50.145 0.006 0.944 X √

(Continued)

Frontiers in Psychology | www.frontiersin.org 7 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments

TABLE 1 | Continued

2 Statisticaltest Row Comparison Experiment Degrees of F-value p-value ηP Shared Exclusive number freedom model model (Figure 1B) (Figure 1C)

36 Exp1b 1,3 23.677 0.017 0.888 X √ 37 Exp3 1,3 47.186 0.006 0.94 X √ EventSegment*ReportType (SR, FR1) 38 Exp1a 1,3 10.95 0.045 0.785 X √ 39 Exp1b 1,3 18.013 0.024 0.857 X √ 40 Exp3 1,3 3.863 0.144 0.563 √ X

EventSegment*Duration 41 Exp1a 3,9 1.562 0.299 0.342 X √ (ε = 0.358) 42 Exp1b 3,9 1.703 0.275 0.362 X √ (ε = 0.461)

EventSegment 2-wayRepeated 43 Exp3 1,3 42.322 0.007 0.934 SS4 (SR,FR) Measures ANOVA 44 Exp 3 1,3 60.352 0.004 0.953 SS4 (SR,FR1) ReportType

45 Exp3 1,3 19.505 0.022 0.867 SS4 (SR,FR) 46 Exp3 1,3 8.585 0.061 0.741 SS4 (SR,FR1)

EventSegment*Report Type 47 Exp3 1,3 21.729 0.019 0.879 X √ SS4 (SR,FR) 48 Exp3 1,3 5.849 0.094 0.661 √ X SS4 (SR,FR1)

The rightmost two columns indicate agreement (√) or disagreement (X) with the predictions of the model shown in Figure 1B (“shared model) and in Figure 1C (“exclusive model”); blank entries indicate absence of a strong prediction of the models. ε, Greenhouse-Giesser correction factor when assumptions of sphericity were violated; t, t-statistic where ANOVA was inappropriate; SS4, When statistical tests were conducted only on data of set-size = 4.

RESULTS for a decay in performance as a function of cue delay that would indicate the involvement of SM. No such decay can be The right panels of Figures 3A,B show performance for the seen for cue delays ranging from 100 to 600ms (Table 1, row post-deviation trajectory as a function of duration when the 25), suggesting that SM is not involved in storing pre-deviation speed (Figure 3A) or the trajectory length (Figure 3B) is kept segment information. fixed. While there is substantial increase in post-deviation Since SM is a rapidly decaying store, single-report performance as a function of duration when the speed is kept performance should be better than full-report performance fixed (Figure 3A, right panel; Table 1, row 21), this increase (TPSR > TPFR). Furthermore, since SM is a large capacity store, is significantly reduced (from increases of 19.8 and 9.2% to all items should be stored with approximately equal precision increases of 4.5 and 1.5% for FR and SR respectively relative and therefore performance for the single report condition to the performance for the stimulus with duration of 200 ms), should be approximately equal to the performance for the first yielding a relatively constant performance when the trajectory item reported in the full-report condition (TPSR = TPFR1) length is kept fixed (significant but with a negligible average (Shooner et al., 2010). A three-way repeated-measures ANOVA increase in magnitude of 3.0%: Figure 3B, right panel; Table 1, (RM-ANOVA) with factors report-type (SR, FR), event-segment row 22). (pre- and post-deviation), and duration shows significant main We then examined the pre-deviation segment, in particular effects of report-type and event-segment (Table 1, rows 30, 33; for the constant trajectory length case (Figure 3B, left panel), duration was significant in Expt. 1a but not 1b: rows 26,27) as

Frontiers in Psychology | www.frontiersin.org 8 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments

A

B

FIGURE 3 | Effect of stimulus duration. The left and right columns show data for pre- and post-deviation conditions, respectively. Transformed performance (TP) averaged across observers (mean ± SEM, N = 4) is shown as a function of stimulus duration when the speed of the dots was constant (A) and when the trajectory length was constant (B). Each panel shows performance for the SR condition, for the three reports in order of report in the FR condition, and for the FR condition (average of the three reports). well as significant two-way interactions between event-segment segment. At a single pre-deviation cue-delay of 400 ms, pre- and report types (Table 1, rows 35, 36, 38, 39) but not between deviation results (left panel of Figure 4) confirm the findings event-segment and duration (Table 1, rows 41,42). of Experiment 1, i.e., both SM tests fail (Table 1, rows 8, 18). For the first test (TPSR > TPFR), a two-way RM-ANOVA Results for the post-deviation segment (right panel of Figure 4) shows a significant single-report advantage for the post-deviation support strongly the involvement of SM. Performance decays trajectory (Table 1, rows 1, 2). On the other hand, for the pre- as a function of cue-delay (Table 1, row 23). TPSR > TPFR is deviation segment, there was a single-report advantage for the evident at 0ms cue-delay, but systematically decays so that TPSR fixed speed condition, where signal strength varied, but this was approaches TPFR when cue-delay is increased (Table 1, row 3). not significant for the fixed trajectory length condition (Table 1, Similarly, TPSR = TPFR1 is evident at 0 ms cue-delay but becomes rows 6,7). violated as cue-delay increases (Table 1, row 13). Taken together, Considering the second SM test (TPSR = TPFR1), for the results of this experiment also favor the model in Figure 1C over post-deviation segment, TPSR was not significantly different from the model in Figure 1B. TPFR1 (Table 1, rows 11, 12). In contrast, for the pre-deviation In general, performance for both SM and STM decreases segment, TPSR was significantly different from TPFR1 (Table 1, monotonically as set size increases (Shooner et al., 2010; Ögmenˇ rows 16, 17). Taken together, evidence from this experiment et al., 2013). However, since STM’s capacity is significantly lower favors strongly the model shown in Figure 1C over the model than the capacity of SM, this decay is much more pronounced for shown in Figure 1B. STM (Ögmenˇ et al., 2013). To test how performance varies as a In Experiment 2, we kept the cue-delay for the pre-deviation function of set-size, in Experiment 3, 1–4 disks moved at a speed segment fixed and varied the cue delay for the post-deviation of 5◦/s for 800ms and the cue was delivered immediately at the

Frontiers in Psychology | www.frontiersin.org 9 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments

shapes, etc.) and stimulus features (e.g., color, orientation, etc.) can be stored in SM, much less is known on how events and event segments are stored in SM. Time is a fundamental dimension for events and motion information can play a critical role in segmenting events (Zacks and Swallow, 2007). Several studies showed that motion information itself can be stored in SM in a way similar to other stimulus properties (Treisman et al., 1975; Demkiw and Michaels, 1976; Blake et al., 1997; Narasimhan et al., 2009; Shooner et al., 2010; Bradley and Pearson, 2012; Ögmenˇ et al., 2013; Huynh et al., 2015, 2017). Changes in motion information can serve as strong cues for event segmentation (Zacks and Swallow, 2007). Here, we used a simple change in motion information (change of direction) to create an event with two segments and analyzed how SM is allocated to these event- FIGURE 4 | TP (mean ± SEM, N = 5) as a function of cue-delay. The left and segments. The study was guided by two competing models shown right panels show data for pre- and post-deviation conditions, respectively. in Figure 1, viz., Shared and Exclusive Models. Inspection of Note that in the pre-deviation conditions the cue-delay was always 400 ms, the last two columns of Table 1 shows that, taken together, our i.e., the cue was presented as soon as motion of the dots was terminated and 400 ms after the deviation of the trajectories. Some of the symbols in the left results favor overwhelmingly the Exclusive Model (Figure 1C) panel have been offset by 50 or 100 ms to enhance their visibility. over the Shared Model (Figure 1B). By using the capacity of SM exclusively for the current event segment, this model eliminates tagging operations so that active vision can operate in end of motion. The results are shown in Figure 5. A three-way real-time. RM-ANOVA shows significant effect of all main factors (Table 1, A question that needs to be addressed is whether the cue at rows 28, 31, 34), and significant interaction for event-segment the end of the pre-deviation segment is sufficient for retrieving and report-type for (SR, FR) but not for (SR, FR1) (Table 1, the appropriate tag. The end point of the pre-deviation segment rows 37, 40). TP dropped as the number of disks to be tracked is also the starting point for the post-deviation segment. Could increased, both for pre-deviation and post-deviation trajectories. the cue override this ambiguity? The instructions given to the For post-deviation trajectories, when set-size increased, the drop observers were that they should report the direction of the pre- for FR was steeper than that for SR. deviation segment when the color of the cue was blue. However, The tests for the difference between SM and STM become it is conceivable that the pre-deviation directions were over- more meaningful at larger set-sizes. Therefore, we ran the SM written by the post-deviation directions and observers were mis- tests for the largest set-size. For the first SM test (TPSR > reporting these in place of the pre-deviation directions. This TPFR), a two-way RM-ANOVA showed significant main effects might explain why the performance was poorer for reporting of segment-type, and report-type, and significant interaction pre-deviation directions. This is very unlikely, because the (Table 1, rows 43, 45, 47). For the pre-deviation segment TPSR large deviation (30–180◦, with mean of 105◦) between the two was not significantly different than TPFR, while it was for the segments would produce a mean difference in directions of 75◦. post-deviation segment (Table 1, rows 5, 10). If the observers were systematically and accurately reporting the For the second SM test (TPSR = TPFR1), a two-way post-deviation direction when asked to report the pre-deviation RM-ANOVA showed significant main effect of segment-type, direction, then the errors would be large and, theoretically, the marginally significant effect of report-type, but no significant TP should be no better than 0.583 and simulations suggest that interaction (Table 1, rows 44, 46). Although the interaction was the TP values obtained in the different experimental conditions not significant (Table 1, row 48), application of the second test tested would lie between 0.584 and 0.522, which is well below to individual segment-types indicate that TPSR was significantly most of the TP values presented in Figures 3–5. In addition, different from TPFR1 for pre-deviation segment, but not for the if the errors were consistently as large as 75◦, this would have post-deviation segment (Table 1, rows 15, 20). been very obvious in the visual feedback that was provided at the end of each trial. It is therefore unlikely that the post- DISCUSSION deviation directions systematically over-wrote the pre-deviation directions. It is possible that over-writing did occur in a small The notion of a two-component memory system goes back proportion of the trials and this would have added some noise to to nineteenth century. In the 1960s, Atkinson and Shiffrin the performance curves. However, over-writing of directionsisan formalized these components into a model and added control unlikely explanation for the pattern of results across the different processes for information transfer. Sperling’s seminal work conditions. (Sperling, 1960) around the same time-period provided the An alternative to the Exclusive Model proposed here is that missing interface between the visual inputs and the two- SM simply has no event tagging at all: it simply contains the component memory system, namely a sensory memory with trajectory information collected over the last few 100 ms or so, high capacity, but of limited duration. Whereas there have been without any need for event boundaries. A possible interpretation extensive studies on how different stimuli (e.g., letters, digits, of the post-deviation results in Figure 3A is that earlier contents

Frontiers in Psychology | www.frontiersin.org 10 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments

FIGURE 5 | TP (mean ± SEM, N = 4) as a function of set-size. The left and right panels show data for pre- and post-deviation conditions, respectively. Each panel shows performance for the SR condition, for up to four reports in order of report in the FR condition, and for the FR condition (average of up to four reports). Note that FR3 and FR4 are not defined when set-size is 2 and FR4 is not defined when set-size is 3. of SM were flushed out by later ones as more spatial information the pre-deviation trajectories. This suggests that the trajectory was collected. However, this is contradicted by the pre-deviation information in pre- and post-deviation segments are not simply results in Figure 3B. Suppose SM contains trajectory information integrated; the preferred explanation is that the visual system collected over 400 ms and flushes out all earlier information. recognizes that the two segments are to be processed as belonging Then when the total stimulus duration is 200 or 400ms, both to separate events. pre- and post-deviation information should be available in SM The definition and segmentation of events may depend not and so these data points in both panels of Figure 3B should show only on stimulus characteristics but also on the goals of the evidence of involvement of SM. While the right panel shows observer (Zacks and Swallow, 2007). The goals of the observer this evidence, the left panel clearly does not (no SR advantage may guide which items are stored in memory. For example, if is evident, nor does SR data equal FR1 data, when the stimulus an observer is looking for a particular set of objects, regardless duration is 200 or 400 ms). Similar arguments can be made whether these objects are in the first or second segment of an against collection durations of 800 or 1,200 ms in SM. Therefore event, she does not need to attach event-segment tags to items to this alternate hypothesis that there is no event tagging is not achieve her goal. Whereas goal-directed flexibility is a signature supported by the data, at least for the durations tested in our of STM, future research needs to address to which extent study. observer’s goals can determine whether and how tagging takes A further analysis of the data was conducted to rule out place in SM. The fact that observers can resolve the ambiguity the alternate hypothesis that SM simply contains the trajectory in the cue for the pre-deviation trajectories, as discussed above, information collected over the last few 100 ms and is not indicates that goal-directed flexibility may also be a characteristic influenced by event boundaries. In Experiment 1b the duration of SM. of the stimulus was varied, with the briefest duration tested If indeed SM stores exclusively the current event segment, being 200 ms. If event boundaries do not exist, then for this then the question arises as to how SM is reset at the termination brief stimulus duration, one would expect that the trajectory of each event segment so as to eliminate all stored items information in the pre-deviation segments would be integrated from the prior segment. When event-segmentation is driven with that in the post-deviation segments. Therefore, it would be by low-level stimulus changes, transients may be used to mask expected that the error in reporting the direction of the post- and reset SM (Averbach and Coriell, 1961; Bachmann, 1994; deviation segment would be highly correlated with the angle Breitmeyer and Ögmen,ˇ 2006) making it exclusively available to between the pre- and post-deviation segments. For each observer the current event segment. However, a transient signal does not in the 200ms duration, single-report condition in Experiment automatically mask and erase all contents of SM: Many factors— 1b, we plotted the post-deviation error (in degrees) against the such as target-mask spatio-temporal proximity and featural angle (in degrees) between the two segments for the trajectory similarity—determine the effectiveness of a transient in erasing that was reported for each of the 100 trials. We determined the the contents of SM (Bachmann, 1994; Breitmeyer and Ögmen,ˇ co-efficient of linear correlation (r) between the post-deviation 2006). In our stimulus, the synchronous deviations in trajectories error and the angle between the pre- and post-deviation segments create a transient signal. This transient signal may be how the for each set of 100 trials. The co-efficients for the four observers contents of SM are erased to allocate SM exclusively to the current were 0.23, 0.09, 0.36, and 0.24 (respective regression slopes were event segment. While we have not measured the effectiveness of 0.08, 0.02, 0.11, and 0.04) and the resulting mean of the r2-values these transients in masking the preceding motion information, was 0.06. In other words, only about 6% of the variability in the two lines of reasoning suggest that visual transients may be reported post-deviation errors is explained by the direction of sufficient reset-signals in some contexts but they are not likely

Frontiers in Psychology | www.frontiersin.org 11 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments to be necessary in setting event boundaries and simultaneously AUTHOR CONTRIBUTIONS erasing the contents of SM: First, as mentioned above, spatiotemporal proximity and ST designed the experiments and collected the data. ST and HÖ featural similarity determine the effectiveness of a mask. In our analyzed and interpreted the data and wrote the paper. Several stimulus, post-deviation trajectories are spatially segregated from undergraduate students also assisted by acting as observers and pre-deviation trajectories and hence they are not likely to be collating data and performing preliminary analysis of the data effective masks. The local transients created at the deviation and writing reports for parts of the project, as part of their Final points are not likely to be effective masks for motion trajectories Year Research Project module at the School of Optometry and either. This is because of featural dissimilarity (flicker vs linear Vision Science, University of Bradford. motion), or because of the fact that pre-deviation motion duration is long and is not likely to be masked effectively with ACKNOWLEDGMENTS a brief transient. Second, whereas visual transients are in general associated The contribution of several undergraduate students in observing with event boundaries, events can be segmented without visual and collating the data obtained is gratefully acknowledged. transients as well (Zacks and Swallow, 2007). Segments can be We are grateful to Profs. Harold Bedell and Brendan Barrett created based on cross modal information and/or can be context for suggestions and discussions during the study and to Dr. dependent: For example, a whistle sound coming from a referee Chris Shooner for writing the original computer program in a basketball game may generate a segment boundary whereas, that was modified for the current study. Many thanks to Dr. for the identical visual stream, a whistle sound coming from Duong Huynh for putting together the video demos in the spectators may not generate a segment boundary. Hence, in Supplementary Information. addition to “traditional” masking mechanisms, we speculate the existence of other mechanisms to control the storage of event SUPPLEMENTARY MATERIAL information in SM. Recent research suggests a key role for SM in cognition by The Supplementary Material for this article can be found showing that it requires (Persuh et al., 2012; Ögmenˇ online at: https://www.frontiersin.org/articles/10.3389/fpsyg. et al., 2013), the read-out from SM and selective spatial attention 2018.01435/full#supplementary-material share similar neural correlates (Ruff et al., 2007), and SM Video 1 | Simulation of one trial for the post-deviation full-report condition. dynamics correlate with psychological intelligence (Miller et al., Video 2 | Simulation of one trial for the post-deviation single-report condition, with 2010) and cognitive impairments (Lu et al., 2005). The structure a delay between end of stimulus presentation and presentation of the cue. of the Exclusive Model in Figure 1C dovetails nicely with these Video 3 | Simulation of one trial for the post-deviation single-report condition. findings by showing how SM is allocated dynamically (and possibly with attentional guidance) under ecological viewing Video 4 | Simulation of one trial for the pre-deviation full-report condition. conditions. Video 5 | Simulation of one trial for the pre-deviation single-report condition.

REFERENCES Bradley, C., and Pearson, J. (2012). The sensory components of high- capacity iconic memory and visual working memory. Front. Psychol. 3:355. Alvarez, G. A., and Cavanagh, P. (2004). The capacity of visual short-term memory doi: 10.3389/fpsyg.2012.00355 is set both by visual information load and by number of objects. Psychol. Sci. 15, Breitmeyer, B. G., and Ögmen,ˇ H. (2006). Visual Masking: Time Slices through 106–111. doi: 10.1111/j.0963-7214.2004.01502006.x Conscious and Unconscious Vision, 2nd Edn. Oxford, UK: Oxford University Atkinson, R. C., and Shiffrin, R. M. (1968). “Human memory: a proposed system Press. and its control processes,” in The Psychology of and Motivation, Vol. Coltheart, M. (1983). Iconic memory. Philos. Trans. R. Soc. B 302, 283–294. 2, eds K. W. Spence and J. T. Spence (New York, NY: Academic Press), 89–195. doi: 10.1098/rstb.1983.0055 Atkinson, R. C., and Shiffrin, R. M. (1971). The control of short term memory. Sci. Cowan, N. (2001). The magical number 4 in short-term memory: a Am. 225, 82–90. doi: 10.1038/scientificamerican0871-82 reconsideration of mental storage capacity. Behav. Brain Sci. 24, 87–114. Averbach, E., and Coriell, A. (1961). Short-term memory in vision. Bell Syst. Tech. doi: 10.1017/S0140525X01003922 J. 40, 309–328. doi: 10.1002/j.1538-7305.1961.tb03987.x Demkiw, P., and Michaels, C. F. (1976). Motion information in iconic memory. Bachmann, T. (1994). Psychophysiology of Visual Masking: The Fine Structure of Acta Psychol. 40, 257–264. doi: 10.1016/0001-6918(76)90029-9 Conscious Experience. New York, NY: Nova Science Publishers, Inc. Haber, R. N. (1983). The impending demise of the icon: a critique of the concept Baddeley, A. (2007). Working Memory, , and Action. Oxford, UK: Oxford of iconic storage in visual information processing. Behav. Brain Sci. 6, 1–54. University Press. doi: 10.1017/S0140525X0001428X Baddeley, A. D., and Hitch, G. (1974). “Working memory,” in The Psychology of Huynh, D., Tripathy, S. P., Bedell, H. E., and Ögmen,ˇ H. (2017). The reference Learning and Motivation: Advances in Research and Theory, Vol. 8, ed G. H. frame for and retention of motion depends on stimulus set size. Atten. Bower (New York, NY: Academic Press), 47–89. Percept. Psychophys. 79, 888–910. doi: 10.3758/s13414-016-1258-5 Bays, P. M., and Husain, M. (2008). Dynamic shifts of limited working memory Huynh, D. L., Tripathy, S. P., Bedell, H. E., and Ögmen,ˇ H. (2015). Stream resources in human vision. Science 321, 851–854. doi: 10.1126/science.1158023 specificity and asymmetries in feature binding and content-addressable access Blake, R., Cepeda, N. J., and Hiris, E. (1997). Memory for visual in visual encoding and memory. J. Vis. 15:14, doi: 10.1167/15.13.14 motion. J. Exp. Psychol. Human Percept. Perform. 23, 353–369. Johansson, G., von Hofsten, C., and Jansson, G. (1980). Event perception. Annu. doi: 10.1037/0096-1523.23.2.353 Rev. Psychol. 31, 27–63. doi: 10.1146/annurev.ps.31.020180.000331

Frontiers in Psychology | www.frontiersin.org 12 September 2018 | Volume 9 | Article 1435 Tripathy and Ögmenˇ Sensory Memory Allocation to Event-Segments

Kennedy, G. J., Tripathy, S. P., and Barrett, B. T. (2009). Early age-related decline Sligte, I. G., Scholte, H. S., and Lamme, V. A. F. (2008). Are there in the effective number of trajectories tracked in adult human vision. J. Vis. 9:21, multiple visual short-term memory stores? PLoS ONE 3:e0001699. doi: 10.1167/9.2.21 doi: 10.1371/journal.pone.0001699 Klatzky, R. L. (1980). Human Memory: Structures and Processes. New York, NY: Smith, W. S., Mollon, J. D., Bhardwaj, R., and Smithson, H. E. (2011). Is there brief Freeman. temporal buffering of successive visual inputs? Q. J. Exp. Psychol. 64, 767–791. Lu, Z. L., Neuse, J., Madigan, S., and Dosher, B. A. (2005). Fast decay doi: 10.1080/17470218.2010.511237 of iconic memory in observers with mild cognitive impairments. Sperling, G. (1960). The information available in brief visual presentations. Psychol. Proc. Natl. Acad. Sci. U.S.A. 102, 1797–1802. doi: 10.1073/pnas.0408 Monogr. Gen. Appl. 74, 1–29. doi: 10.1037/h0093759 402102 Swallow, K. M., Zacks, J. M., and Adams, R. A. (2009). Event boundaries in Makovski, T. (2012). Are multiple visual short-term memory storages perception affect memory encoding and updating. J. Exp. Psychol. Gen. 138, necessary to explain the retro-cue effect? Psychon. Bull. Rev. 19, 470–476. 236–257. doi: 10.1037/a0015631 doi: 10.3758/s13423-012-0235-9 Treisman, A., Russell, R., and Green, J. (1975). “Brief visual storage of shape and Matsukura, M., and Hollingworth, A. (2011). Does visual short-term movement,” in Attention and Performance, Vol. 5, eds P. M. A. Rabbitt and S. memory have a high-capacity stage? Psychon. Bull. Rev. 18, 1098–1104. Dornic (London: Academic Press), 699–721. doi: 10.3758/s13423-011-0153-2 Tripathy, S. P., and Barrett, B. T. (2003). Gross misperceptions in the perceived Miller, R., Rammsayer, T. H., Schweizer, K., and Troche, S. J. (2010). trajectories of moving dots. Perception 32, 1403–1408. doi: 10.1068/p5161 Decay of iconic memory traces is related to psychometric intelligence: Tripathy, S. P., and Barrett, B. T. (2004). Severe loss of positional information a fixed-links modeling approach. Learn. Individ. Differ. 20, 699–704. when detecting deviations in multiple trajectories. J. Vis. 4, 1020–1043, doi: 10.1016/j.lindif.2010.08.010 doi: 10.1167/4.12.4 Narasimhan, S., Tripathy, S. P., and Barrett, B. T. (2009). Loss of Tripathy, S. P., and Howard, C. J. (2012). Multiple-trajectory tracking. Scholarpedia positional information when tracking multiple moving dots: the role 7:11287. doi: 10.4249/scholarpedia.11287 of visual memory. Vis. Res. 49, 10–27. doi: 10.1016/j.visres.2008. Tripathy, S. P., Narasimhan, S., and Barrett, B. T. (2007). On the effective number 09.023 of tracked trajectories in normal human vision. J. Vis. 7:2. doi: 10.1167/7.6.2 Ögmen,ˇ H. (2007). A theory of moving form perception: synergy between Van den Berg, R., Shin, H., Chou, W. C., George, R., and Ma, W. masking, perceptual grouping, and motion computation in retinotopic J. (2012). Variability in encoding precision accounts for visual short- and non-retinotopic representations. Adv. Cogn. Psychol. 3, 67–84. term memory limitations. Proc. Natl. Acad. Sci. U.S.A. 109, 8780–8785. doi: 10.2478/v10053-008-0015-2 doi: 10.1073/pnas.1117465109 Ögmen,ˇ H., Ekiz, O., Huynh, D., Bedell, H. E., and Tripathy, S. P. (2013). van Moorselaar, D., Olivers, C. N. L., Theeuwes, J., Lamme, V. A. F., and Sligte, I. G. Bottlenecks of motion processing during a visual glance: the leaky flask model. (2015). Forgotten but not gone: retro-cue costs and benefits in a double-cueing PLoS ONE 8:e83671. doi: 10.1371/journal.pone.0083671 paradigm suggest multiple states in visual short-term memory. J. Exp. Psychol. Ögmen,ˇ H., and Herzog, M. H. (2010). The geometry of : Learn. Mem. Cogn. 41, 1755–1763. doi: 10.1037/xlm0000124 retinotopic and non-retinotopic representations in the human Warren, W. H., and Shaw, R. E. (eds.). (1985). “Persistence and Change,” in visual system. Proc. IEEE Inst. Electr. Electron. Eng. 98, 479–492. Proceedings of the First International Conference on Event Perception (Mahwah, doi: 10.1109/JPROC.2009.2039028 NJ: Lawrence Erlbaum Associates). Ögmen,ˇ H., and Herzog, M. H. (2016). A new conceptualization of human visual Zacks, J. M., and Swallow, K. M. (2007). Event segmentation. Curr. Dir. Psychol. sensory-memory. Front. Psychol. 7:830. doi: 10.3389/fpsyg.2016.00830 Sci. 16, 80–84. doi: 10.1111/j.1467-8721.2007.00480.x Persuh, M., Genzer, B., and Melara, R. D. (2012). Iconic memory requires attention. Zacks, J. M., Swallow, K. M., Vettel, J. M., and McAvoy, M. P. (2006). Visual Front. Hum. Neurosci. 6:126, doi: 10.3389/fnhum.2012.00126 motion and the neural correlates of event perception. Brain Res. 1076, 150–162. Pinto, Y., Sligte, I. G., Shapiro, K. L., and Lamme, V. A. F. (2013). Fragile visual doi: 10.1016/j.brainres.2005.12.122 short-term memory is an object-based and location-specific store. Psychon. Zhang, W., and Luck, S. J. (2008). Discrete fixed-resolution representations in Bull. Rev. 20, 732–739. doi: 10.3758/s13423-013-0393-4 visual working memory. Nature 453, 233–235. doi: 10.1038/nature06860 Pylyshyn, Z. W., and Storm, R. W. (1988). Tracking multiple independent targets: evidence for a parallel tracking mechanism. Spat. Vis. 3, 1–19. Conflict of Interest Statement: The authors declare that the research was doi: 10.1163/156856888X00122 conducted in the absence of any commercial or financial relationships that could Rensink, R. A. (2014). Limits to the usability of iconic memory. Front. Psychol. be construed as a potential conflict of interest. 5:971. doi: 10.3389/fpsyg.2014.00971 Ruff, C. C., Kristjánsson, A., and Driver, J. (2007). Readout from iconic memory Copyright © 2018 Tripathy and Ögmen.ˇ This is an open-access article distributed and selective spatial attention involve similar neural processes. Psychol. Sci. 18, under the terms of the Creative Commons Attribution License (CC BY). The use, 901–909. doi: 10.1111/j.1467-9280.2007.01998.x distribution or reproduction in other forums is permitted, provided the original Shooner, C., Tripathy, S. P., Bedell, H. E., and Ögmen,ˇ H. (2010). High-capacity, author(s) and the copyright owner(s) are credited and that the original publication transient retention of direction-of-motion information for multiple moving in this journal is cited, in accordance with accepted academic practice. No use, objects. J. Vis. 10:8, doi: 10.1167/10.6.8 distribution or reproduction is permitted which does not comply with these terms.

Frontiers in Psychology | www.frontiersin.org 13 September 2018 | Volume 9 | Article 1435