bioRxiv preprint doi: https://doi.org/10.1101/2020.07.08.193326; this version posted July 11, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 1 Sensing and seeing associated with overlapping occipitoparietal activation in simultaneous EEG-fMRI Catriona L. Scrivener MRC Cognition and Brain Sciences Unit, University of Cambridge, Cambridge, UK Centre for Integrative Neuroscience and Neurodynamics, University of Reading, UK Asad Malik, Michael Lindner, Etienne B. Roesch Centre for Integrative Neuroscience and Neurodynamics, University of Reading, UK Author Note Correspondence regarding this article should be addressed to C. L. Scrivener at the MRC Cognition and Brain Sciences Unit, 15 Chaucer Rd, Cambridge, CB2 7EF (email: [email protected]) bioRxiv preprint doi: https://doi.org/10.1101/2020.07.08.193326; this version posted July 11, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 2 Abstract The presence of a change in a visual scene can influence brain activity and behaviour, even in the absence of full conscious report. It may be possible for us to sense that such a change has occurred, even if we cannot specify exactly where or what it was. Despite existing evidence from electroencephalogram (EEG) and eye-tracking data, it is still unclear how this partial level of awareness relates to fMRI BOLD activation. Using EEG, functional magnetic resonance imaging (fMRI), and a change blindness paradigm, we found multi-modal evidence to suggest that sensing a change is distinguishable from being blind to it. Specifically, trials during which participants could detect the pres- ence of a colour change but not identify the location of the change (sense trials), were compared to those where participants could both detect and localise the change (localise or see trials), as well as change blind trials. In EEG, late parietal positivity and N2 amplitudes were larger for localised changes only, when compared to change blindness. However, ERP-informed fMRI analysis found no voxels with activation that significantly co-varied with fluctuations in single-trial late positivity amplitudes. In fMRI, a range of visual (BA17,18), parietal (BA7,40), and midbrain (anterior cingulate, BA24) areas showed increased fMRI BOLD activation when a change was sensed, compared to change blindness. These visual and parietal areas are commonly implicated as the storage sites of visual working memory, and we therefore argue that sensing may not be explained by a lack of stored representation of the visual display. Both seeing and sensing a change were associated with an overlapping occipitoparietal network of activation when compared to blind trials, suggesting that the quality of the visual representation, rather than the lack of one, may result in partial awareness during the change blindness paradigm. Keywords: EEG-fMRI, change blindness, sensing, conscious awareness bioRxiv preprint doi: https://doi.org/10.1101/2020.07.08.193326; this version posted July 11, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 3 Sensing and seeing associated with overlapping occipitoparietal activation in simultaneous EEG-fMRI 1 Introduction 2 It is common for us to overestimate the amount of information that we can process 3 and store about the world around us. Although we may assume that we would notice a 4 cyclist entering the path of our car, or if a building on our street changed in colour, we 5 very often miss these occurrences (Simons, 2000; Simons & Levin, 1997). The failure to 6 detect changes between visual scenes is known as change blindness, and is used as evidence 7 to suggest that our internal representation of the outside world is not as complete as once 8 thought (Rensink, 2004; Noe et al., 2000). When changes to an image are disrupted in 9 some way, for example by a distractor image or a visual saccade, we cannot use visual 10 transients (or motion) to detect them, and are often blind to the difference (Rensink 11 et al., 1997; Kanai & Verstraten, 2004). 12 It was previously assumed that if we are blind to a change then we cannot provide 13 any information about it, and that the change should not influence our behaviour in any 14 way. Blindness to changes is thought to result from a lack of detailed representation 15 about the pre- and post-change scenes, or an inability to successfully compare the two 16 (Simons, 2000). If this is the case, then our knowledge when we are blind to changes 17 should be equivalent to that when there is no change at all. Anecdotally, this does not 18 align with the experience of observers in a change blindness experiment; it is common for 19 them to remark that they suspected something had changed, but that they were not sure 20 about its nature or location. This experience appears to be phenomenologically different 21 from complete change blindness, but how this difference is reflected in behavioural and 22 neuroimaging data is unclear. 23 In the domain of visual consciousness, there is a recurring debate on the nature 24 of visual awareness; whether it is graded, or dichotomous, or a combination determined 25 by the context. In the Global Neuronal Workspace Theory (Dehaene & Naccache, 2001; 26 Dehaene et al., 2006), it is posited that awareness arises when inputs cross a threshold for 27 ‘ignition’, resulting in the distribution and maintenance of information within a ‘global bioRxiv preprint doi: https://doi.org/10.1101/2020.07.08.193326; this version posted July 11, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 4 28 workspace’. Based on this proposal, conscious awareness is a dichotomous state, as only 29 inputs selected by attention can spark the activation of the global workspace. This 30 consists of a large network of connected regions, including prefrontal and parietal regions 31 as well as the thalamic nuclei and basal ganglia (Dehaene & Changeux, 2011). Therefore, 32 conscious awareness requires directed attention and activation of a distributed frontal- 33 parietal network, in an ‘all-or-nothing’ fashion. 34 In accordance with this, fMRI studies specifically investigating change blindness 35 report that detected changes are associated with greater activation in the parietal lobe, 36 dorsolateral prefrontal cortex, and fusiform gyrus, when compared to changes that are 37 missed (Beck et al., 2001). Further, detected changes compared with correctly categorised 38 no-change trials revealed activation in a wider network including the inferior, superior, 39 and medial temporal gyrus, anterior interparietal sulcus, precuneus, central sulcus, infe- 40 rior frontal gyrus, anterior cingulate cortex, putamen, pulvinar, and cerebellum (Pessoa, 41 2004). A similar pattern was identified for false alarm trials, where participants reported 42 a change when no change occurred, suggesting that activity was related to the partici- 43 pants’ perception of the change rather than properties of the visual stimulus. Overall, 44 few regions were specifically activated when participants exhibited change blindness. 45 However, this ‘all-or-nothing’ explanation of visual awareness does not align with 46 our subjective experience of the world. Based on participants’ report of a sense for 47 something changing, we might conclude that awareness is graded. This allows for a level, 48 or levels, of awareness lying somewhere on a continuum between full and absent awareness. 49 In an early experiment, Rensink (2004) suggested the presence of a sense condition, in 50 which observers could detect a change without fully identifying it. Observers were asked 51 to indicate when they ‘thought’ that something had changed, and then again when they 52 were certain of it. He argued that this sense condition is both phenomenologically and 53 perceptually distinct from the traditionally reported see condition in which participants 54 are fully aware of what change occurred. 55 This definition has been extended and explored using electroencephalogram (EEG) 56 and eye-tracking, with a range of results suggesting a richer visual experience than either bioRxiv preprint doi: https://doi.org/10.1101/2020.07.08.193326; this version posted July 11, 2020. The copyright holder for this preprint (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under aCC-BY-NC-ND 4.0 International license. 5 57 “yes I saw a change” or “no I didn’t see anything” (Busch et al., 2009; Fernandez-Duque 58 & Thornton, 2003; Kimura et al., 2008; Lyyra et al., 2012; Thornton & Fernandez-Duque, 59 2001; Howe & Webb, 2014; Chetverikov et al., 2018; Reynolds & Withers, 2015; Lyyra 60 et al., 2012; Galpin et al., 2008). The distinction could be described by the ‘partial aware- 61 ness hypothesis’ (Kouider et al., 2010; Kouider & Dehaene, 2007). While the mechanism 62 of awareness can still be considered dichotomous and dependent on an ignition threshold, 63 the level of detail contained within the workspace is variable. Stimuli can be represented 64 with varying detail, based on factors such as stimulus strength, therefore giving rise to 65 graded knowledge of its contents.