Cognition 138 (2015) 132–147 Contents lists available at ScienceDirect Cognition journal homepage: www.elsevier.com/locate/COGNIT Visual, haptic and bimodal scene perception: Evidence for a unitary representation ⇑ Helene Intraub , Frank Morelli 1, Kristin M. Gagnier 2 University of Delaware, USA article info abstract Article history: Participants studied seven meaningful scene-regions bordered by removable boundaries Received 3 July 2014 (30 s each). In Experiment 1 (N = 80) participants used visual or haptic exploration and Revised 21 January 2015 then minutes later, reconstructed boundary position using the same or the alternate mod- Accepted 25 January 2015 ality. Participants in all groups shifted boundary placement outward (boundary extension), Available online 25 February 2015 but visual study yielded the greater error. Critically, this modality-specific difference in boundary extension transferred without cost in the cross-modal conditions, suggesting a Keywords: functionally unitary scene representation. In Experiment 2 (N = 20), bimodal study led to Scene representation boundary extension that did not differ from haptic exploration alone, suggesting that Spatial memory Multisensory (visual and haptic) bimodal spatial memory was constrained by the more ‘‘conservative’’ haptic modality. In Cross-modal representation Experiment 3 (N = 20), as in picture studies, boundary memory was tested 30 s after Boundary extension viewing each scene-region and as with pictures, boundary extension still occurred. Results suggest that scene representation is organized around an amodal spatial core that orga- nizes bottom-up information from multiple modalities in combination with top-down expectations about the surrounding world. Ó 2015 Elsevier B.V. All rights reserved. 1. Introduction of the view, in the absence of any corresponding sensory input (boundary extension; Intraub & Richardson, 1989). Multiple sensory modalities provide the perceiver with This can occur very rapidly, across intervals as brief as a rich information about the surrounding world. In spite of saccadic eye movement (Dickinson & Intraub, 2008; this, similar to other areas of perception, research on scene Intraub & Dickinson, 2008). Boundary extension may be perception has typically been studied through a modality- an adaptive error that facilitates integration of successive specific lens (usually vision; Intraub, 2012; O’Regan, views of the world (Hubbard, Hutchison, & Courtney, 1992). Yet, even when perception is limited to the visual 2010; Intraub, 2010, 2012). Indeed, research has shown modality alone participants frequently remember seeing that boundary extension can prime visual perception of the continuation of the scene just beyond the boundaries upcoming layout, when that layout is subsequently presented (e.g., Gottesman, 2011). What leads to this spatial error? Intraub (2010, 2012) ⇑ Corresponding author at: Department of Psychological and Brain and Intraub and Dickinson (2008) suggested that rather Sciences, University of Delaware, Newark, DE 19716, USA. than a visual representation, representation of visual scenes E-mail address: [email protected] (H. Intraub). is actually a multisource representation in that it incorpo- 1 Address: Dismounted Warrior Branch, Human Research and Engineer- rates information from both the sensory source (vision) ing Directorate, U.S. Army Research Laboratory, Aberdeen Proving Ground, as well as top-down sources of information that place the MD 21005, USA. 2 Address: Spatial Intelligence and Learning Center, Department of studied view within a likely surrounding spatial context. Psychology, Temple University, Philadelphia, PA 19122, USA. Potential top-down sources include amodal continuation http://dx.doi.org/10.1016/j.cognition.2015.01.010 0010-0277/Ó 2015 Elsevier B.V. All rights reserved. H. Intraub et al. / Cognition 138 (2015) 132–147 133 of the surface beyond the boundaries (Fantoni, Hilger, Intraub, 2002, 2010), and following haptic perception of Gerbino, & Kellman, 2008), general scene knowledge based the same scene regions (Intraub, 2004; Mullally et al., upon scene classification (Greene & Oliva, 2009), and 2012). The multisource model provided the same explana- object-to-context associations (Bar, 2004). The purpose of tion for visual and haptic boundary extension, but included our research was to determine if boundary extension no commitment as to whether they draw on a single scene following visual or haptic perception of the same scene-re- representation or on distinct modality-specific representa- gion is supported by a single multimodal scene representa- tions. The evidence for boundary extension in 3D space tion or by two functionally independent modality-specific was based on experiments in which meaningfully related scene representations. objects were arranged on natural backgrounds (e.g., ‘‘k- Boundary extension is a spatial error in which a swath itchen scene’’), bounded by a ‘‘window frame’’ to limit of anticipated space just beyond the boundaries of the visual or haptic exploration. view is remembered as having been perceived. Neuroimag- In haptic studies, blindfolded participants explored the ing and neuropsychological research have shown that bounded regions right up to edges of the display, and min- boundary extension is associated with neural activation utes later, after the boundaries were removed, participants of brain regions thought to play important roles in spatial reconstructed boundary placement. They set the bound- cognition: the hippocampus, parahippocampal cortex, aries outward, including a greater expanse of space that and retrosplenial complex (Chadwick, Mullally, & had originally been included in the stimulus. This occurred Maguire, 2013; Mullally, Intraub, & Maguire, 2012; Park, in spite of the fact that there was always an object 2–3 cm Intraub, Yi, Widders, & Chun, 2007). The hippocampus from the boundary, forcing participants to squeeze their has long been associated with spatial representation and hands into a tightly constrained space. As in the case of navigation (Burgess, 2002; Maguire & Mullally, 2013; vision (Gagnier et al., 2013) a seemingly clear marker of O’Keefe & Nadel, 1978). The parahippocampal cortex and boundary placement did not prevent boundary extension. retrosplenial complex have been associated with percep- A comparison of boundary extension following visual or tion of spatial layout, and with the integration of local haptic exploration of the same regions showed that vision spaces within larger spatial contexts, respectively yielded the more expansive error (Intraub, 2004). This was (Epstein, 2008). Recent research has shown that the the case whether visual boundary extension was compared parahippocampal cortex responds similarly to visual and to haptic boundary extension in sighted participants who haptic perception of layout (Wolbers, Klatzky, Loomis, were blindfolded for the experiment, or in a woman who Wutte, & Giudice, 2011; also see Epstein, 2011), underscor- had been deaf and blind since early life (a ‘‘haptic expert’’). ing the spatial rather than modality-centric role of this Why might vision have yielded a greater anticipatory brain area. spatial error? Intraub (2004) speculated that such a differ- It has been suggested that scene representation is fun- ence, if reliable, might be related to the different character- damentally an act of spatial cognition (Dickinson & istics and spatial scope of the two modalities. Vision is a Intraub, 2008; Gagnier, Dickinson & Intraub, 2013; distal modality with a small high acuity foveal region Gagnier & Intraub, 2012; Intraub, 2010, 2012; Intraub & (about 1° of visual angle) and a large low-acuity periphery. Dickinson, 2008). In their multisource model Intraub and Together these encompass a relatively large spatial area. In Dickinson (2008; Intraub, 2010, 2012) proposed that an contrast, the haptic modality encompasses multiple high amodal spatial structure organizes multiple sources of acuity regions (the fingertips) and a relatively small knowledge (bottom-up and top-down) into a coherent periphery. In the case of vision, a greater amount of the scene representation (see Maguire & Mullally, 2013, for a visually imagined continuation of the view might be con- similar view from the perspective of hippocampal func- fusable with visual memory for the stimulus than in the tion). The idea is that the observer brings to any view of case of haptic exploration. This explanation conforms to a scene a sense of surrounding space (the space ‘‘in front the notion of boundary extension as a source monitoring of’’, ‘‘to the left and right’’, ‘‘above’’, ‘‘below’’ and ‘‘behind’’ error (Intraub, 2010, 2012; Intraub, Daniels, Horowitz, & the observer (see Bryant, Tversky, & Franklin, 1992; Wolfe, 2008; Intraub & Dickinson, 2008; Seamon, Franklin & Tversky, 1990; Tversky, 2009). This provides Schlegel, Hiester, Landau, & Blumenthal, 2002). According the scaffolding that supports not only the bottom-up visual to the source monitoring framework (Johnson, information but the anticipated continuation of the scene Hashtroudi, & Lindsay, 1993) as the similarity between beyond the boundaries of the view. This underlying spatial representations drawn from two different sources (e.g., structure is similar to the ‘‘spatial image’’ proposed by perception and imagination) increases, so too does the Loomis, Klatzky, and Giudice (2013), in that it is a sur- likelihood of source misattributions (as, for example, when rounding spatial representation