What You See Is What You Remember: Visual Chunking by Temporal Integration Enhances Working Memory Elkan G. Akyürek, Nils Kappelmann, Marc Volkert, and Hedderik van Rijn Downloaded from http://mitprc.silverchair.com/jocn/article-pdf/29/12/2025/1786754/jocn_a_01175.pdf by MIT Libraries user on 17 May 2021 Abstract ■ Human memory benefits from information clustering, which ing attentional processing, depended on the actual number of can be accomplished by chunking. Chunking typically relies on successive targets. The memory-related CDA and P3 compo- expertise and strategy, and it is unknown whether perceptual nents instead depended on the perceived number of targets clustering over time, through temporal integration, can also irrespective of their actual succession. The report of two sepa- enhance working memory. The current study examined the rate targets was associated with elevated amplitude, whereas attentional and working memory costs of temporal integration integrated as well as actual single targets exhibited lower ampli- of successive target stimulus pairs embedded in rapid serial tude. Temporal integration thus provided an efficient means of visual presentation. ERPs were measured as a function of behav- processing sensory input, offloading working memory so that ioral reports: One target, two separate targets, or two targets the features of two targets were consolidated and maintained reported as a single integrated target. N2pc amplitude, reflect- at a cost similar to that of a single target. ■ INTRODUCTION et al., 1995), space is special in that neighboring regions Regardless of modality, virtually all information perceived in the visual field are represented by neighboring neu- by our senses is eventually clustered into larger struc- rons. Because proximal neurons are more likely to be tures before being stored in our memory systems. This closely connected than distal ones, neural activity in re- so-called chunking process increases memory capacity, sponse to a coherent shape or figure will also show sim- as more bits of information can be retained if clustering ilar coherency, facilitating object perception. An object is is possible (Miller, 1956). Chunking is thought to exist in indeed frequently perceived and later maintained in two forms: The first is deliberate, often employed as a memory as a discrete spatial entity, that is, as a single co- conscious mnemonic strategy, and the second is more herent chunk (e.g., Luria, Balaban, Awh, & Vogel, 2016), automatic and continuously acts on perceptual represen- even though it may carry different features (color, tations (Gobet et al., 2001). Common to both is that shape). These features are quickly bound together at chunking seems to take place at a relatively high level an early perceptual stage, often by virtue of their common of abstraction, involving elaborate cognitive processes location (Wheeler & Treisman, 2002). Larger patterns in a as is evident when memorizing a sequence of single, in- visual scene can also often be recognized rapidly, parti- dividual words by generating a sentence that contains cularly if they form a wholesome global figure: a Gestalt these words. Even when more automatic, perceptual (for a review, see Wagemans et al., 2012). Our ability to chunking occurs, links to existing knowledge or expertise quickly extract and later recall the gist of a visual scene, seem needed, as reflected by the classic memory advan- even within a single feed-forward sweep through the visual tage of expert chess players over novices for game cortex (VanRullen & Koch, 2003; Thorpe, Fize, & Marlot, positions. Neuroimaging research on chunking has fur- 1996), probably relies similarly on such visuospatial thermore also implicated the involvement of the lateral clustering. frontal cortex, a region thought to be involved in the Here we test the hypothesis that perceptual clustering high-level organization and control of working memory across time can similarly afford chunking. Unlike the (Bor, Duncan, Wiseman, & Owen, 2003). clear spatial organization of the visual cortex, its temporal Under the right circumstances, however, effortless organization is more implicit and emerges from variable clustering of perceptual information in space is also pos- neural activity patterns that might be generated endoge- sible. Because the visual cortex is organized along spatial nously as well as induced exogenously, such as oscilla- dimensions (e.g., retinotopically in areas V1–V3; Tootell tions (Buzsáki & Draguhn, 2004) and neural response latency (Lamme & Roelfsema, 2000). Forming a temporal representation, a so-called event, is thus intrinsically University of Groningen different from representing an object defined in space. © 2017 Massachusetts Institute of Technology Journal of Cognitive Neuroscience 29:12, pp. 2025–2036 doi:10.1162/jocn_a_01175 Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/jocn_a_01175 by guest on 29 September 2021 Functionally, such a temporal event can be defined as a optimal interval may thus rather be ecologically con- relatively brief time interval, within which information is strained, so that we need to constantly appraise the pos- integrated, but which is segregated from other events sible efficiency gained from having fewer events (chunks) (Dixon & Di Lollo, 1994). The phenomenon of temporal due to temporal integration against the higher informa- integration has been observed in various tasks, but one tion fidelity without integration. particularly illustrative example comes from the rapid se- In this study, we used electrophysiological recordings rial visual presentation (RSVP) paradigm. In the RSVP task to directly test the hypothesis that temporal integration is developed by Akyürek et al. (2012), visually compatible associated with increased cognitive efficiency, in particu- Downloaded from http://mitprc.silverchair.com/jocn/article-pdf/29/12/2025/1786754/jocn_a_01175.pdf by MIT Libraries user on 17 May 2021 but discrete individual stimuli were successively shown lar during attentional selection, and during the encoding at the same location, for about 100 msec each. These and maintenance of information in working memory. We stimuli could be perceived individually, but also as an measured the ERP during an RSVP task in which partici- integrated whole (similar to a “0” and a “–” image in a pants had to detect either a single or two sequentially traditional alarm clock display, or the integrated “8”). presented targets among distractor stimuli, and we deter- Observers frequently reported the integrated percept mined whether the ERPs support the hypotheses that comprising two or more individually presented stimuli temporal integration is more efficient. The RSVP task is (Akyürek & Wolff, 2016; Akyürek et al., 2012). known to result in a commensurate report rate of single, Temporal integration has been associated exclusively dual, and integrated dual targets while keeping stimulus with basic, early perceptual functions, rather than cogni- conditions constant (Wolff, Scholz, Akyürek, & van Rijn, tive ones, such as memory. For instance, early studies fo- 2015; Akyürek et al., 2012), which enables a balanced cused on the perceived simultaneity of stimuli, whether measure of the ERP. Furthermore, because participants or not they were part of the same perceptual moment are always explicitly asked to identify the two targets (Allport, 1968; Efron, 1967). Later work targeted the lin- individually, the observed frequency of integration re- gering impression made by stimuli, referred to as visible sponses is as unbiased as possible. persistence, hypothesizing that this might provide a ERP component amplitude can be assessed to reflect bridge to succeeding stimuli (Hogben & Di Lollo, 1974; on the degree to which functional processes related to Eriksen & Collins, 1967). A well-known conceptualization temporal integration are being engaged. Here, we will of such persistence is iconic memory (Neisser, 1967). specifically focus on the N2pc component that is associ- This kind of fleeting memory is thought to be limitless, ated with attentional processing of stimulus features and so that any change in temporal integration rate should on the CDA (contralateral delay activity) and the P3 com- not impact upon later, more cognitive processing and ponent that are associated with consolidation and main- storage. Recent studies, however, give indications that tenance in working memory. this is unlikely. The N2pc is a relatively early, lateralized component, One important clue comes from the observation that which tracks the attentional processing of visually pre- the integration of sensory input over time seems to de- sented stimulus features across the hemifields (Kiss, pend on one’s current perceptual pace, that is, the inter- van Velzen, & Eimer, 2008; Eimer, 1996; Luck & Hillyard, val across which integration occurs scales in response to 1994). Lower N2pc amplitude should correspond to less perceived task demands (Akyürek, Toffanin, & Hommel, attentional effort and/or higher efficiency. As the N2pc re- 2008; Akyürek, Riddell, Toffanin, & Hommel, 2007). This flects relatively early attentional processes, we expect that adaptive nature of temporal integration suggests that the N2pc indexes the number of presented individual something can be gained from slowing down one’s per- RSVP targets, irrespective of whether the later report ceptual pace. At first glance, integration seems primarily on the presented stimuli indicates that the elements lossy,