This Accepted Manuscript has not been copyedited and formatted. The final version may differ from this version.
Research Articles: Systems/Circuits
Dynamic causal modelling of active vision
Thomas Parr1, M. Berk Mirza1, Hayriye Cagnan2,3 and Karl J Friston1
1Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London, WC1N 3BG, UK. 2MRC Brain Network Dynamics Unit at the University of Oxford 3Nuffield Department of Clinical Neurosciences, University of Oxford https://doi.org/10.1523/JNEUROSCI.2459-18.2019
Received: 24 September 2018
Revised: 8 March 2019
Accepted: 11 March 2019
Published: 10 June 2019
Author contributions: T.P. and K.F. designed research; T.P. performed research; T.P. analyzed data; T.P. wrote the first draft of the paper; T.P., M.B.M., H.C., and K.F. edited the paper; M.B.M., H.C., and K.F. wrote the paper.
Conflict of Interest: The authors declare no competing financial interests.
TP is supported by the Rosetrees Trust (Award Number 173346). KJF is a Wellcome Principal Research Fellow (Ref: 088130/Z/09/Z). MBM is a member of PACE (Perception and Action in Complex Environments) supported by the European Union's Horizon 2020 (Marie Sklodowska-Curie Grant Agreement 642961). HC is supported by the MRC (MR/R020418/1). We are grateful to David Bradbury and Eric Featherstone for their technical support and to the WCHN MEG group for helpful discussions.
Correspondence: Thomas Parr, The Wellcome Centre for Human Neuroimaging, Institute of Neurology, 12 Queen Square, London, UK WC1N 3BG, [email protected]
Cite as: J. Neurosci 2019; 10.1523/JNEUROSCI.2459-18.2019
Alerts: Sign up at www.jneurosci.org/alerts to receive customized email alerts when the fully formatted version of this article is published.
Accepted manuscripts are peer-reviewed but have not been through the copyediting, formatting, or proofreading process.
Copyright © 2019 Parr et al. This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International license, which permits unrestricted use, distribution and reproduction in any medium provided that the original work is properly attributed.
Dynamic causal modelling of active vision
Thomas Parr1*, M. Berk Mirza1, Hayriye Cagnan2,3, Karl J Friston1
1 Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London, WC1N 3BG, UK.
2 MRC Brain Network Dynamics Unit at the University of Oxford
2 Nuffield Department of Clinical Neurosciences, University of Oxford
[email protected], [email protected], [email protected], [email protected]
Correspondence: Thomas Parr
The Wellcome Centre for Human Neuroimaging
Institute of Neurology
12 Queen Square, London, UK WC1N 3BG
Abbreviated title: Active vision
Pages: 25
Figures: 8
Wordcount – Abstract: 213; Introduction: 371; Discussion: 1333
Disclosure statement: The authors have no disclosures or conflict of interest.
Acknowledgements: TP is supported by the Rosetrees Trust (Award Number 173346). KJF is a Wellcome
Principal Research Fellow (Ref: 088130/Z/09/Z). MBM is a member of PACE (Perception and Action in
Complex Environments) supported by the European Union's Horizon 2020 (Marie Sklodowska-Curie
Grant Agreement 642961). HC is supported by the MRC (MR/R020418/1). We are grateful to David
Bradbury and Eric Featherstone for their technical support and to the WCHN MEG group for helpful
discussions.
1 Active vision
2
3 Dynamic causal modelling of active vision
4
5 Thomas Parr1*, M. Berk Mirza1, Hayriye Cagnan2,3, Karl J Friston1
6 1 Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London, WC1N 3BG, UK.
7 2 MRC Brain Network Dynamics Unit at the University of Oxford
8 2 Nuffield Department of Clinical Neurosciences, University of Oxford
9 [email protected], [email protected], [email protected], [email protected]
10
11 Correspondence: Thomas Parr
12 The Wellcome Centre for Human Neuroimaging
13 Institute of Neurology
14 12 Queen Square, London, UK WC1N 3BG
16
17
18 Abstract
19 In this paper, we draw from recent theoretical work on active perception that suggests the brain
20 makes use of an internal (i.e., generative) model to make inferences about the causes of sensations.
21 This view treats visual sensations as consequent on action (i.e., saccades) and implies that visual
22 percepts must be actively constructed via a sequence of eye movements. Oculomotor control calls
23 on a distributed set of brain sources that includes the dorsal and ventral frontoparietal (attention)
24 networks. We argue that connections from the frontal eye fields to ventral parietal sources
25 represent the mapping from ‘where’, fixation location to information derived from ‘what’
26 representations in the ventral visual stream. During scene construction, this mapping must be
1
Active vision
27 learned, putatively through changes in the effective connectivity of these synapses. Here, we test
28 the hypothesis that the coupling between the dorsal frontal cortex and the right temporoparietal
29 cortex is modulated during saccadic interrogation of a simple visual scene. Using dynamic causal
30 modelling for magnetoencephalography (MEG) with (male and female) human participants, we
31 assess the evidence for changes in effective connectivity by comparing models that allow for this
32 modulation with models that do not. We find strong evidence for modulation of connections
33 between the two attention networks; namely, a disinhibition of the ventral network by its dorsal
34 counterpart.
35
36 Keywords: Dynamic causal modelling; Active vision; Magnetoencephalography; Eye-tracking;
37 Attention; Visual neglect
38
39 Significance statement
40 This work draws from recent theoretical accounts of active vision and provides empirical evidence
41 for changes in synaptic efficacy consistent with these computational models. In brief, we used
42 magnetoencephalography in combination with eye-tracking to assess the neural correlates of a form
43 of short-term memory during a dot cancellation task. Using dynamic causal modelling to quantify
44 changes in effective connectivity, we found evidence that the coupling between the dorsal and
45 ventral attention networks changed during the saccadic interrogation of a simple visual scene.
46 Intuitively, this is consistent with the idea that these neuronal connections may encode beliefs about
47 “what I would see if I looked there”, and that this mapping is optimised as new data are obtained
48 with each fixation.
49
50 Introduction
2
Active vision
51 Perception is a fundamentally active process. While this is true across modalities, it is especially
52 obvious in the visual system, where what we see depends upon where we look (Andreopoulos &
53 Tsotsos, 2013; Ognibene & Baldassarre, 2014; Parr & Friston, 2017a; Wurtz, McAlonan, Cavanaugh,
54 & Berman, 2011). In this paper, we consider the anatomy that supports decisions about where to
55 look, and the fast plastic changes that underwrite effective saccadic interrogation of a visual scene.
56 We appeal to the metaphor of perception as hypothesis testing (Gregory, 1980), treating each
57 fixation as an experiment to garner new information about states of affairs in the world (Mirza,
58 Adams, Mathys, & Friston, 2018; Mirza, Adams, Mathys, & Friston, 2016; Parr & Friston, 2017c).
59 Building upon recent theoretical work (Parr & Friston, 2017b), which includes a formal model of the
60 task used here, we hypothesised that the configuration of a visual scene is best represented in terms
61 of expected visual sensations contingent upon a given saccade (‘what I would see if I looked there’)
62 (Zimmermann & Lappe, 2016). This implies a form of short-term plasticity following each fixation, as
63 the mapping from fixation to observation is optimised.
64 The purpose of this study is not to evaluate whether we engage in active vision, as there is already
65 substantial evidence in favour of this (Mirza et al., 2018; Yang, Lengyel, & Wolpert, 2016), but to try
66 to understand how the underlying computations manifest in terms of changes in effective
67 connectivity. Our aim is to establish whether there is neurobiological evidence in favour of
68 optimisation of a generative model (Yuille & Kersten, 2006) that represents visual consequences of
69 fixations as a series of eye-movements are performed.
70 In the following, we describe our experimental set-up, including our gaze-contingent cancellation
71 task. Through source reconstruction, we demonstrate the engagement of frontal, temporal, and
72 parietal sources, and note the right-lateralisation of the temporal component. We then detail the
73 hypothesis in terms of network models or architectures and use DCM to adjudicate between models
74 that do and do not allow for plastic changes in key connections. This model comparison revealed a
3
Active vision
75 decrease, from early to late fixations, in the inhibition of neuronal populations in the ventral
76 network by those in the dorsal network.
77
78
79 Network structure
80 The cortical anatomy of oculomotor control has been investigated through functional neuroimaging,
81 neuropsychological, and structural connectivity studies. Figure 1 summarises how their findings
82 converge upon a system that can be separated into a bilateral dorsal frontoparietal network, and a
83 right lateralised ventral network. In brief, functional imaging experiments (Corbetta & Shulman,
84 2002; Vossel, Weidner, Driver, Friston, & Fink, 2012) during visuospatial tasks reveal activation of the
85 frontal eye fields (FEF) and the intraparietal sulcus (IPS) in both hemispheres, but greater
86 involvement of the right temporoparietal junction (TPJ) than its contralateral homologue. The
87 volumes of the white matter tracts connecting the components of the dorsal attention network are
88 comparable, while those connecting the ventral network sources are of a significantly greater
89 volume in the right hemisphere (Thiebaut de Schotten et al., 2011). Neuropsychological asymmetries
90 reinforce this network structure, with right hemispheric lesions much more likely than left to give
91 rise to visual neglect (Halligan & Marshall, 1998).
92 Neglect is a syndrome that manifests as a failure to attend to, or perform exploratory saccades to
93 (Karnath & Rorden, 2012), one side of visual space and (often) appears to be a consequence of a
94 disconnection between the ventral and (right) dorsal networks (Bartolomeo, Thiebaut de Schotten,
95 & Doricchi, 2007; He et al., 2007). Given the dorsal frontoparietal origins of cortico-collicular axons
96 (Fries, 1984, 1985; Gaymard, Lynch, Ploner, Condy, & Rivaud-Péchoux, 2003; Künzle & Akert, 1977),
97 frontal control of eye position (Bruce, Goldberg, Bushnell, & Stanton, 1985; Sajad et al., 2015), and
98 the representation of visual stimulus identity in the ventral visual (‘what’) stream (Goodale & Milner,
4
Active vision
99 1992; Ungerleider & Haxby, 1994), this is consistent with the idea that the connection between
100 these networks is the neural substrate of an embodied (oculomotor) map of visual space. It is worth
101 noting that the temporo-parietal component of the ventral attention network is not within the
102 ventral visual stream. However, it has been associated with target-detection operations (Chica,
103 Bartolomeo, & Valero-Cabré, 2011; Corbetta & Shulman, 2002; Serences et al., 2005) that rely upon
104 a simple form of visually derived stimulus identity. Although our focus is in the visual domain, we
105 note that similar networks appear to be involved in auditory attention and neglect (Dietz, Friston,
106 Mattingley, Roepstorff, & Garrido, 2014).
107 Synthesising these theoretical and neuroanatomical constructs, we hypothesised that the coupling
108 between the dorsal and ventral attention networks changes with successive fixations in a saccadic
109 task. This hypothesis is based upon the idea that, as an internal model of the task is optimised, the
110 relationship between fixation locations and their visual consequences should become more precise
111 (see (Parr & Friston, 2017b) for a simulation that demonstrates this). If this is the case, this could
112 manifest in one of two ways. The effective connectivity from the temporoparietal cortex to the
113 frontal eye-fields could increase over time. Alternatively, plastic changes in connections in the
114 opposite (dorsal-to-ventral) direction could decrease their effective connectivity to relieve
115 descending inhibition of the ventral-to-dorsal projections arising from superficial pyramidal cells.
116 Ultimately, both of these would enhance the influence of ventral parietal over dorsal frontal regions.
117 We used an oculomotor cancellation paradigm, based upon the classic pen-and-paper line
118 cancellation task used to assess visual neglect (Albert, 1973; Fullerton, McSherry, & Stout, 1986). In
119 this task, patients with neglect tend to cancel (by crossing out) lines on the right side of a piece of
120 paper but miss those on the left. Using magnetoencephalography (MEG) and dynamic causal
121 modelling (DCM) for evoked responses (David et al., 2006) we assessed changes in effective
122 connectivity between dorsal frontal and ventral temporoparietal sources during early and late
123 cancellations (fixations) in healthy participants. Our task involved performing saccades to targets on
124 a screen that, once fixated, changed colour and were considered cancelled.
5
Active vision
125
126 Methods - Experimental Design and Statistical Analyses
127 Imaging and behavioural task
128 We recruited 14 healthy right-handed participants (8 females and 6 males) between the ages of 18
129 and 35 from the UCL ICN subject pool under minimum risk ethics. Participants were seated in the
130 MEG scanner (whole-head 275-channel axial gradiometer system, 600 samples per second, CTF
131 Omega, VSM MedTech, Coquitlam, Canada), with a screen about 64cm in front of them, showing the
132 stimulus display (size 40 x 29.5cm). This was presented using Cogent 2000 (developed by the Cogent
133 team at the FIL and the ICN and Cogent Graphics developed by John Romaya at the LON at the
134 Wellcome Department of Imaging Neuroscience).
135 The sequence of stimuli is illustrated in Figure 2. Following a fixation cross, a set of 16 black dots
136 appeared on the screen, simultaneously, in pseudo-random (using the Matlab random number
137 generator) locations. When a dot was fixated, it changed from black to red (i.e. was ‘cancelled’).
138 Participants were asked to look at the black dots, but to avoid looking at the red dots. We tracked
139 the eyes of the participants while the dots were on screen using an SR Research eye-tracker (Eyelink
140 1000 – operated using Psychtoolbox) sampling at a frequency of 1kHz. We divided the cancellation
141 events into two categories: early (first 8) and late (last 8).
142 Although almost all perceptual tasks call upon some sort of engagement with the sensorium, this
143 task emphasises the active nature of visual processing through making the visual element of the task
144 as simple as possible. This still calls upon optimisation of beliefs under an internal model, as
145 formalised in (Parr & Friston, 2017b). As outlined above, this has some validity in relation to
146 disorders in which active vision is impaired. However, it is worth noting that other approaches to
147 studying these processes, particularly those that focus on behavioural (as opposed to
148 neurophysiological) measures (Mirza et al., 2018; Yang et al., 2016), make use of more complicated
6
Active vision
149 visual stimuli – so that different saccades afford different levels of information gain about a
150 particular scene category.
151 Our pre-processing steps (using SPM 12 - http://www.fil.ion.ucl.ac.uk/spm/software/spm12/) are
152 specified in Figure 2. As participants generally had no trouble in cancelling all 16 dots, we rejected all
153 trials for which they were unable to do so (assuming these were due to eye-tracker calibration
154 errors). We merged the epoched data from all participants, and averaged the epochs corresponding
155 to the first 8, and the last 8, cancellations over all participants to create a grand average. This meant
156 we averaged over fixations preceded by saccades from all possible directions, ensuring any
157 directional eye movement induced artefacts following cancellation were averaged away. Using
158 robust averaging provides an additional protection against artefactual signals, as this iterative
159 procedure rejects those trials that deviate markedly from the mean response. The average eye-
160 speed is shown in Figure 2 (black dotted line) to illustrate that it falls to its minimum at about the
161 same time as the target is cancelled. The first principal component, across spatial channels, of the
162 averaged evoked response (to a cancellation) in each condition is shown on the same plot. To
163 further interrogate the changes in effective connectivity, we additionally constructed grand
164 averaged responses to each of the 16 cancellations in a trial. These were used for the more detailed
165 model of (parametric) time-dependent responses described below.
166
167 Source reconstruction
168 In Figure 3, we show the reconstructed source activity obtained using multiple sparse priors (Karl
169 Friston et al., 2008). This scheme tries to infer the sources in the brain that generated the data
170 measured at the sensors. There are an infinite number of possible solutions to this problem, but
171 Bayesian methods attempt to find the simplest of these. Our results – using standard settings (Litvak
172 et al., 2011) – show a relatively symmetrical distribution of frontal and posterior cortical sources,
173 and a right lateralised (asymmetrical) temporal component. While the inferred locations are more
7
Active vision
174 ventromedial than we might expect – based upon Figure 1 – (likely due to the ill posed nature of the
175 MEG inverse problem). It is encouraging that we can recover sources that are broadly consistent
176 with the known anatomy, and lateralisation, of the attention networks (Corbetta & Shulman, 2002)
177 from these data.
178
179 Dynamic causal modelling
180 Dynamic causal modelling (DCM) tries to explain measured electrophysiological data in terms of
181 underlying neuronal (i.e., source) activity (K. J. Friston, Harrison, & Penny, 2003). This rests upon
182 optimising the model evidence (or free energy) for a biophysically plausible neural mass model. The
183 (log) evidence that data y affords a model m is
lnpmE (yyx | )t [ln p ( , ,θxθ | m ) ln q ( , | m ) ] q 144442 44443 14442 4443 Generative model Approx. posterior 184 Ep[ln (yx | ,θxθxθ )] Dqmpm [ ( , | ) || ( , | )] 14444442qKL 4444443 1444444444442 444444444443 Accuracy Complexity
185 DCM makes use of a Variational Laplace procedure (Karl Friston, Mattout, Trujillo-Barreto,
186 Ashburner, & Penny, 2007) to optimise beliefs ( q ) about neuronal activity ( x ) and the parameters
187 (θ ) that determine this activity (e.g., connection strengths) and the (likelihood) mapping (e.g., lead
188 field) from x to y . The lead field matrix maps source activity to the measured sensor data on the
189 scalp (Kiebel, David, & Friston, 2006). In maximising model evidence, DCM finds the most accurate
190 explanation for the data that complies with Occam’s principle; i.e., is minimally complex (as
191 measured by the KL-Divergence between posteriors and priors). By comparing different generative
192 models, we can test hypotheses about biologically grounded model parameters; here, condition-
193 specific changes in connectivity under a particular network architecture.
194 The generative model we used is the canonical microcircuit model (Bastos et al., 2012; Moran,
195 Pinotsis, & Friston, 2013), which incorporates four distinct neuronal populations (Figure 4). These
8
Active vision
196 are spiny stellate cells, superficial and deep pyramidal cells, and inhibitory interneurons. The
197 connections associated with each of these populations conforms to known patterns of laminar
198 specific connectivity in the cerebral cortex (Daniel J. Felleman & Van Essen, 1991; Shipp, 2007; Zeki &
199 Shipp, 1988), allowing us to distinguish between ascending and descending extrinsic (i.e. between
200 source) connections. This accounts for the prior probability density pm(,x θ | ) that, supplemented
201 with a lead-field provides a likelihood p(|,)yxθ and completes the forward or generative model.
202 As we were interested in changes in the coupling of the dorsal and ventral attention networks, we
203 specified our generative model as in Figure 5; incorporating the bilateral dorsal network and the
204 right lateralised temporoparietal contribution to the ventral network (consistent with the source
205 reconstruction above). The connections between the right temporoparietal junction (rTPJ) and the
206 left frontal eye field (FEF) probably involve an intermediate thalamic relay (Guillery & Sherman,
207 2002; Halassa & Kastner, 2017), but this was omitted for simplicity. Our hypothesis was that the
208 connections between the rTPJ and each FEF would change between early and late target
209 cancellations (Parr & Friston, 2017b). Figure 5 highlights these ascending and descending
210 connections. After fitting the full model (with modulation of all four connections) to our empirical
211 data, we used Bayesian model reduction (K. J. Friston et al., 2016) to evaluate the evidence for
212 models with every combination of these condition-specific effects (early versus late) enabled or set,
213 a priori, to zero.
214
215 Results
216 Figure 6 reports the results of a model comparison between 16 (24) models that allowed for different
217 patterns of search-dependent changes in the forward and backward connections between each FEF
218 and the rTPJ. Given our grand average data, model 8 has a posterior probability of 0.827. This model
219 allows for changes in backwards connections, and the forward connection from rTPJ to the right FEF,
220 but not to the left FEF. This provides evidence in favour of changes in the efficacy of dorsal-ventral
9
Active vision
221 connections. Acknowledging that other models, although improbable, were found to be plausible,
222 we averaged our results across models, weighting each model by its posterior probability. Following
223 this Bayesian model averaging, we still found striking changes in the backward connections, which
224 show a decrease in effective connectivity for late compared to early cancellations. As backwards
225 connections are (net) inhibitory, this corresponds to a disinhibition of the superficial pyramidal cells
226 – the origin of ascending connections – in the TPJ. In other words, the effective connectivity during
227 the later stages of the trial changed, compared to that during the first few cancellations, to relieve
228 the inhibitory effect of the dorsal attention network on the source of its input from the ventral
229 network.
230
231 The effects of this disinhibition can be seen in the reconstructed neuronal activities shown in Figure
232 7. During later cancellations, the activity of the superficial pyramidal cells in the rTPJ has a greater
233 amplitude than evoked during earlier fixations. Figure 5 shows that this is the population inhibited
234 by the descending connections (labelled 3 and 4). These are the connections that show the greatest
235 change (both relative and absolute, despite being slightly weaker at baseline than the forward
236 connections). Although the change in ascending synapses is small or absent, the increase in activity
237 in these forward projecting TPJ cells has driven an increase in the amplitude of responses in all
238 populations in each FEF. The most dramatic effect is in the deep pyramidal layer, which receives
239 direct input from the superficial TPJ cells. Figure 7 additionally shows the resemblance between the
240 activity in deep pyramidal cells in FEF and the simulated rate of belief updating obtained under a
241 Markov Decision Process model of the same behavioural task: for details, please see (Parr & Friston,
242 2017b). This model represents a formalisation of the ideas raised in the introduction; namely, that
243 representations of visual space depend upon beliefs about the sensory consequences of actions. In
244 brief, the differences in the rate of belief updating from early to late fixations are due to the
10
Active vision
245 optimisation of the mapping from fixation location to the presence or absence of a target. More
246 precise beliefs later in the task enable faster and more confident belief updates.
247 To explore the changes in coupling demonstrated above in a more parametric way, we inverted a
248 DCM that was identical to that described above, but treated each cancellation as a separate event.
249 This meant that, in place of the relatively coarse division into ‘early’ and ‘late’, we could test
250 hypotheses about parametric changes in connection strength over 16 sequential cancellations.
251 Figure 8 illustrates a model comparison that tests these hypotheses, endorsing the pattern of
252 changes found in Figure 6. Due to the implicit model of time-dependent effects, this enables us to
253 plot the estimated changes in coupling throughout the trial, as shown in Figure 8. These show a
254 progressive decrease in the strength of inhibitory backwards connections, with a modest increase
255 over time in excitatory forward connections.
256 Discussion
257 The results presented here provide evidence in favour of short term plastic changes in the
258 connections between the dorsal and ventral attention networks during the active interrogation of a
259 simple visual scene. This supports an enactive perspective on visual cognition (Bruineberg, 2017;
260 Hohwy, 2007; Vernon, 2008), as it is consistent with the idea that we represent visual sensations as
261 the consequences of action – and that these contingencies may be learned over a short time period.
262 While these results have interesting implications for active vision, they also constrain the way in
263 which cortical neuronal circuits might implement inferential computations. That the descending
264 connections appear to change the most is consistent with the idea that ascending signals in the brain
265 carry evidence for or against hypotheses represented in higher areas. While this appears
266 counterintuitive, the evidence afforded to a hypothesis about one variable (e.g. location on a
267 horizontal axis) depends upon beliefs about other variables (e.g. location in the vertical axis). In
268 other words, dorsally represented beliefs about eye position, if represented in any factorised
269 coordinate system, must act to contextualise the ascending signals from the ventral to dorsal
11
Active vision
270 network. As this is learned over successive fixations, this contextualisation (i.e., interaction between
271 factors) leads to increasingly precise mappings between eye position and its visual consequences –
272 consistent with the disinhibition we observed here. This is analogous to the increase in amplitude of
273 evoked responses following cueing in working memory paradigms (Lenartowicz, Escobedo-Quiroz, &
274 Cohen, 2010) that can be reproduced in silico by appealing to beliefs about the context of ascending
275 signals (Parr & Friston, 2017d).
276 An interesting question that arises from this is what type of coordinate system the frontal eye fields
277 might employ. The argument given above applies regardless of the choice of coordinate system but
278 depends upon there being some factorisation (Parr & Friston, 2017a). This factorisation could be
279 representation of a horizontal and a vertical axis (McCloskey & Rapp, 2000) or could be closer to a
280 wavelet decomposition – used in computational visual processing (Antonini, Barlaud, Mathieu, &
281 Daubechies, 1992). The latter separates an image into different spatial scales and resolutions. For
282 example, we might represent which quadrant of space we are looking at, and which sub-quadrant
283 within that quadrant. Either of these systems requires far fewer neurons than we would need if we
284 were to independently represent each location in visual space. This is an important aspect of the
285 normative (active inference) theory on which the simulations in Figure 7 were based. In brief, the
286 sorts of generative models used by the brain to infer the causes of its sensory input are subject to
287 exactly the same imperatives used in Bayesian model comparison; namely, the brain's generative or
288 forward models must provide an accurate account of sensations with the minimum complexity.
289 Reducing the number of parameters via factorisation is, in theory, an important aspect of minimising
290 complexity or redundancy (H. Barlow, 1961; H. B. Barlow, 1974; K. Friston & Buzsaki, 2016;
291 Tenenbaum, Kemp, Griffiths, & Goodman, 2011). We used a decomposition of location into
292 quadrants to simulate the belief updating shown in the lower left of Figure 7, which enables us to
293 reproduce visual neglect at different spatial scales (Parr & Friston, 2017b), consistent with
294 neuropsychological observations (Grimsen, Hildebrandt, & Fahle, 2008; Medina et al., 2009; Ota,
295 Fujii, Suzuki, Fukatsu, & Yamadori, 2001; Verdon, Schwartz, Lovblad, Hauert, & Vuilleumier, 2009).
12
Active vision
296 Visual neglect is increasingly recognised as a disconnection syndrome (He et al., 2007). Specifically, it
297 can arise through damage to the white matter tracts that link right dorsal frontal sources to ventral
298 temporoparietal areas (Doricchi & Tomaiuolo, 2003; Thiebaut de Schotten et al., 2005). A
299 disconnection of this sort would preclude the changes we have observed in these connections. From
300 the perspective of active inference, this means that saccades to the left side of space represent poor
301 perceptual experiments, as the capacity to learn from them is diminished (Denzler & Brown, 2002;
302 Lindley, 1956; MacKay, 1992; Yang et al., 2016). We have previously argued that syndromes in which
303 active scene construction is impaired – visual neglect being an important example – may result from
304 pathological prior beliefs about these action-sensation mappings (Parr, Rees, & Friston, 2018). An
305 inability to change this mapping following observation, perhaps due to white matter disconnection
306 (Catani & ffytche, 2005; Geschwind, 1965), means that actions that would otherwise engage (and
307 modify) a given connection afford a smaller opportunity for novelty resolution (Parr & Friston,
308 2017b). The failure to update this mapping is consistent with the impairments in spatial working
309 memory that have been elicited in saccadic tasks in neglect patients (Husain et al., 2001). In future
310 work, we aim to follow up this idea by temporarily disrupting changes in these (dorsal-ventral)
311 connections using transcranial magnetic stimulation. We hypothesise that this will induce saccadic
312 scan paths consistent with those observed in visual neglect (Fruhmann Berger, Johannsen, &
313 Karnath, 2008; Karnath & Rorden, 2012). Encouragingly, this approach has previously been used to
314 induce other features of visual neglect (Ellison, Schindler, Pattison, & Milner, 2004; Platz, Schüttauf,
315 Aschenbach, Mengdehl, & Lotze, 2016), including changes in line bisection and visual search
316 performance following stimulation of the right TPJ.
317 An additional direction for future research concerns the use of more complex visual environments.
318 In this study we kept the visual stimuli as simple as possible. However, many interesting phenomena
319 in active vision can be elicited using more sophisticated, and often dynamic, manipulations. An
320 advantage to using stationary targets is that they induce scanning saccades as opposed to reactive
321 saccades – of the sort associated with a suddenly appearing target. The former are accompanied by
13
Active vision
322 greater involvement of the frontal part of the dorsal network, while the latter implicates the parietal
323 part (Pierrot-Deseilligny, Rivaud, Gaymard, Müri, & Vermersch, 1995). Given that our hypothesis
324 concerned the frontal regions of the dorsal network, the use of static targets facilitated the
325 involvement of these regions. However, the inclusion of a second condition in which targets
326 suddenly appeared would help us to further interrogate the respective contributions of the frontal
327 and parietal cortices to these processes. We hope to pursue this in future work.
328 Specifically, it would be interesting to probe the computational mechanisms that underwrite
329 differences between scanning and reactive saccades for both perception and neurobiological
330 measurements (Zimmermann & Lappe, 2016). This may relate to the time required for belief-
331 updating, which itself is likely to depend upon the sorts of beliefs that are updated. Typically, cortical
332 areas that sit higher in the anatomical hierarchy (Daniel J. Felleman & Van Essen, 1991; Shipp, 2007;
333 Zeki & Shipp, 1988) are thought to represent stimuli that evolve over longer time-periods (Hasson,
334 Chen, & Honey, 2015; Hasson, Yang, Vallines, Heeger, & Rubin, 2008; Kiebel, Daunizeau, & Friston,
335 2008; Murray et al., 2014), in relation to early sensory cortices. Given that the frontal eye fields are
336 engaged in control of scanning saccades, which occur at about 3-4 Hz, it is plausible that the time-
337 scale for updating beliefs about ‘where I am looking’ corresponds to this frequency. Speculatively,
338 short-latency reactive saccades may be driven by lower cortical regions (e.g. parietal cortex) that
339 represent the locations of fast-changing stimuli and may not leave enough time for completion of
340 belief updating in frontal areas. As noted by one of our reviewers, this might account for the changes
341 in spatial perception of stationary stimuli that follow adaptive changes in saccadic amplitude, but the
342 absence of this phenomenon when dynamic stimuli induce reactive saccades. This is because, under
343 the view that we represent visual space in terms of the visual consequences of saccades, a failure to
344 complete belief updating – in brain regions representing alternative saccades – may preclude the
345 sort of changes in coupling between frontal and temporoparietal areas observed here. Intuitively,
346 this is sensible when constructing a motor map of visual space: there is little point in including
14
Active vision
347 transient stimuli, as they are unlikely to be there on looking back. This idea predicts that there
348 should be a diminished inhibition of return following a reactive, as opposed to a scanning, saccade.
349
350 Conclusion
351 In this paper, we tested the hypothesis that the coupling between dorsal and ventral frontoparietal
352 networks is altered during visual exploration. To do so, we used dynamic causal modelling based
353 upon a network motivated by pre-existing structural, functional, and neuropsychological data. We
354 found greatest evidence for a model that allowed for modulation in connections from the dorsal to
355 the ventral network. Bayesian modelling averaging revealed a decrease in the effective connectivity
356 of these connections, resulting in a disinhibition of ventral sources by the dorsal attention network.
357 These results are consistent with the idea that the visual data obtained following a saccade drive
358 plastic changes, optimising beliefs about the sensory consequences of a given saccadic fixation. This
359 has potentially important implications for syndromes in which visual exploration is disrupted –
360 notably, visual neglect. We hope that understanding (and measuring) these changes in effective
361 connectivity in health will yield insights into the pathophysiology of disconnection syndromes.
362
363 Acknowledgements
364 TP is supported by the Rosetrees Trust (Award Number 173346). KJF is a Wellcome Principal
365 Research Fellow (Ref: 088130/Z/09/Z). MBM is a member of PACE (Perception and Action in
366 Complex Environments) supported by the European Union's Horizon 2020 (Marie Sklodowska-Curie
367 Grant Agreement 642961). HC is supported by the MRC (MR/R020418/1). We are grateful to David
368 Bradbury and Eric Featherstone for their technical support and to the WCHN MEG group for helpful
369 discussions.
370
15
Active vision
371 Author contributions
372 TP and KJF designed the experiment. TP collected data and performed the analysis. TP, MBM, HC,
373 and KJF wrote the manuscript.
374
375 Disclosure statement
376 The authors have no disclosures or conflict of interest.
377 Figure Legends
378 Figure 1 – The anatomy of attention. This figure summarises the functional, neuropsychological, and
379 structural characterisations of attention networks in the brain. The upper left image shows the
380 components of the dorsal and ventral frontoparietal attention networks, as derived through
381 functional imaging studies. The dorsal sources (blue) are bilaterally activated during visual attention
382 tasks, while the ventral (orange) network is lateralised to the right hemisphere. The lower left image
383 summarises lesion studies that demonstrate that lesions to the ventral network in the right
384 hemisphere are associated with visual neglect. The lower right image shows the three branches of
385 the superior longitudinal fasciculus – a white matter tract that connects the sources of the attention
386 networks. The plot on the upper right indexes the lateralisation of these tracts by their relative
387 volumes in each hemisphere. Notably, the third branch – that connects the ventral sources – is
388 significantly right lateralised. The images on the left are reprinted by permission from Springer
389 Nature: Nature Reviews Neuroscience from (Corbetta & Shulman, 2002), and those on the right
390 reprinted by permission from Springer Nature: Nature Neuroscience from (Thiebaut de Schotten et
391 al., 2011). The material in this figure is not included in the CC BY license for this article.
392 Figure 2 – Oculomotor cancellation task and pre-processing. The graphic on the upper left
393 illustrates the sequence of events for a given trial. First, a fixation cross is presented for 2 seconds.
394 After this, a display with 16 black dots is randomly generated and presented for 15 seconds. This is
395 followed by a blank screen for 3 seconds. The dots were placed within an 8x8 grid (not visible to the
16
Active vision
396 participants), as shown in the lower left schematic. When the dots were visible on screen, we
397 tracked the eyes of the participant. Whenever their gaze entered a square containing a black dot,
398 this changed from black to red and remained red for the rest of the trial. Participants were
399 instructed to look at the black dots, and to avoid looking at red dots. Events were defined as the
400 time at which the eye crossed into the square, causing a change in colour (i.e. a cancellation). There
401 were 15 of these trials per block, with 6 blocks per participant. The lower left plot shows a histogram
402 of the time intervals between saccadic dot cancellations, to give a sense of the latency between
403 saccades. These latencies are reported using a (natural) logarithmic time scale (with time in seconds)
404 over the first 2.5 standard deviations above and below the mean. The mean here is -1.0597,
405 corresponding to roughly 3 cancellations per second (consistent with the 3-4Hz frequency of
406 saccadic sampling (Hoffman et al., 2013)). On the right, we show the sequence of pre-processing
407 steps used and the first principal component of the ensuing evoked response. The evoked response
408 to early cancellations is averaged from 6738 events, and the response to late cancellations from
409 6571. Superimposed upon this is a trace of the eye speed in peristimulus time in arbitrary units. This
410 is aligned so that zero corresponds to the average speed during the time in which the fixation cross
411 was present.
412 Figure 3 – Source reconstruction with multiple sparse priors. These images show the Bayes optimal
413 source reconstruction under multiple sparse priors (and following application of a temporal Hanning
414 window) for the first 8 cancellations (left) and the second 8 cancellations (right) in a trial. This
415 reveals a set of symmetrical sources in both the frontal and posterior cortical sources, with a right
416 lateralised temporal component. The striking asymmetry of these temporal sources (dashed circles)
417 is encouraging, considering the known rightward lateralisation of the ventral attention network.
418 While we might expect the frontal sources to be more dorsal, this may reflect the ill-posed nature of
419 MEG source localisation – there are many possible combinations of sources in 3D space that could
420 give rise to the same pattern of activation over the 2D sensory array. The estimated responses show
421 the greatest amplitude at around 100ms. In the left plot (showing the maximal response for the first
17
Active vision
422 condition), the red lines indicate the reconstructed activity from the early cancellations and grey
423 from the late cancellations. In the right plot (maximal response for the second condition), red is late
424 and grey is early. Bayesian credible intervals are shown as dotted lines for each response. The
425 confidence associated with the posterior probability maps (PPM) (K. J. Friston & Penny, 2003), in
426 addition to the variance explained, are included in the upper left of each plot, and the location at
427 which the response is estimated is given in the lower right.
428 Figure 4 – The canonical microcircuit. The equations on the left of this schematic describe the
429 dynamics of the generative model that underwrites the dynamic causal modelling in this paper. The
430 x vectors represent population specific voltage (odd subscripts) and conductance (even subscripts).
431 Each element of the x vectors represents a distinct cortical source. The notation abb means the
432 element-wise product of a and b . The matrix A determines extrinsic (between-source)
433 connectivity (here illustrated as connections between a lower source i and a higher source i+1),
434 while G determines the intrinsic (within-source) connectivity. Subscripts for these matrices indicate
435 mappings between specific cell populations. For example, A1 describes ascending connections from
436 superficial pyramidal cells (source i) to spiny stellate cells (source i+1), while A3 describes
437 descending connections from deep pyramidal cells (source i+1) to superficial pyramidal cells (source
438 i). Experimental inputs – in our case, the cancellation of the target on fixation – are specified by u .
439 On the right, we illustrate the neuronal message passing implied by these equations. Red arrows
440 indicate excitatory connections and blue inhibitory. Superficial pyramidal cells give rise to ascending
441 connections that target spiny stellate and deep pyramidal cells in a higher cortical source.
442 Descending connections arise from deep pyramidal cells that target superficial pyramidal cells and
443 inhibitory interneurons.
444 Figure 5 – Network architecture. This schematic illustrates the form of the network model we used
445 to test our hypothesis. The dorsal network is present bilaterally (FEF and IPS) and is connected to the
446 ventral network – represented by the temporoparietal junction (TPJ) – on the right. The TPJ receives
18 Active vision
447 input as it sits lower in the visual hierarchy than the FEF (Daniel J Felleman & Van, 1991). Our
448 hypothesis concerns the (highlighted) connections between the two networks. We compared
449 models that allowed for changes or visual search-dependent plasticity in connections from the TPJ to
450 left FEF (1), from the TPJ to right FEF (2), from the left FEF to TPJ (3), from the right FEF to TPJ (4),
451 and every combination of the above. The matrices on the right illustrate the specification of these
452 connections. The A matrices are the same as those in Figure 4 and represent extrinsic connections
453 between sources (with subscripts indicating which specific cell populations in those sources). B
454 specifies the connections that can change between the early and late cancellations and C specifies
455 which sources receive visual (i.e., geniculate) input. To ensure that the signs of the A (and C )
456 connections do not change during estimation, their logarithms are treated as normally distributed
457 random variables. This ensures an excitatory connection cannot become an inhibitory connection
458 and vice versa.
459 Figure 6 – Model comparison and Bayesian model averaging. This figure shows the results of
460 comparing models with different combinations of condition-specific effects on the forward and
461 backward connections between the right TPJ and the frontal eye fields. We performed this
462 comparison using Bayesian model reduction (K. J. Friston et al., 2016), which involves fitting a full
463 model that allows all four connections to change and analytically evaluating the evidence for models
464 with combinations of these changes switched off. The upper plots show the log posterior
465 probabilities associated with each model, and the posterior probabilities. The winning model
466 (number 8) allows for modulation in connections 2, 3, and 4 (see Figure 5). The lower plots show
467 that, for the later fixations, there is a modest increase in the effective connectivity in connection 2,
468 but a decrease in 3 and 4. These values correspond to log scaling parameters, such that a value of
469 zero means no change. The lower left plot shows these parameter (maximum a posteriori) estimates
470 for the full model (that allows for all connections). The lower right plot shows the Bayesian model
471 average of these estimates (weighted by the probability of each reduced model to account for
472 uncertainty over models). Bayesian 90% credible intervals are shown as pink bars.
19
Active vision
473 Figure 7 – Estimated neuronal activity. These plots show the estimated activity in each excitatory
474 cell population. Dashed lines indicate the superficial pyramidal cells that give rise to ascending
475 connections and are inhibited by higher cortical sources. Ascending connections target the spiny
476 stellate cells (dotted lines), and the deep pyramidal cells (unbroken lines). The latter give rise to
477 descending connections. The activity here is shown for early (blue) and late (red) cancellations, for
478 each of the cortical areas shown in Figure 5. The lower left plot (highlighted) shows the simulated
479 evoked responses obtained from the Markov Decision Process model described in (Parr & Friston,
480 2017b), drawing from the process theory associated with active inference (Karl Friston, FitzGerald,
481 Rigoli, Schwartenbeck, & Pezzulo, 2016). It is computed by taking the absolute rate of change of the
482 sufficient statistics of posterior beliefs about the current fixation location, summed over spatial
483 scales (please see the discussion for details). While the y-axis here is arbitrary, the x-axis extends to
484 250ms, consistent with the theta frequency of saccadic eye movements. There is a striking
485 resemblance between the simulated rate of belief updating and the frontal eye field neuronal
486 activity estimated from our empirical data.
487 Figure 8 – Time-dependency of modulatory changes. The plots on the right of this Figure are the
488 same as those in Figure 6, but modelling a parametric effect of number of previous cancellations. For
489 this model, in place of the ‘early’ and ‘late’ conditions, we treated each sequential cancellation as a
490 separate event. Because the model is parameterised in terms of log-scaling parameters, linear (i.e.,
491 [0,1,…,15]) parametric effects of time (number of previous cancellations) correspond to a
492 monoexponential change in coupling (starting from a strength of exp(0), corresponding to 100%).
493 The two most probable models are the same as in Figure 6, and the overall pattern of changes
494 shown in the MAP estimates is the same (but with some evidence in favour of a small change in
495 connection 1). The plots on the left show the estimated changes in each connection with successive
496 cancellation events, as a percentage of their initial values. These indicate an increase in the strength
497 of forward excitatory connections over time, and a decrease in backward inhibitory connections.
20
Active vision
498
499 References
500 Albert, M. L. (1973). A simple test of visual neglect. Neurology, 23(6), 658. doi:10.1212/wnl.23.6.658 501 Andreopoulos, A., & Tsotsos, J. (2013). A computational learning theory of active object recognition 502 under uncertainty. International journal of computer vision, 101(1), 95-142. 503 Antonini, M., Barlaud, M., Mathieu, P., & Daubechies, I. (1992). Image coding using wavelet 504 transform. IEEE Transactions on image processing, 1(2), 205-220. 505 Barlow, H. (1961). Possible principles underlying the transformations of sensory messages. In W. 506 Rosenblith (Ed.), Sensory Communication (pp. 217-234). Cambridge, MA: MIT Press. 507 Barlow, H. B. (1974). Inductive inference, coding, perception, and language. Perception, 3, 123-134. 508 Bartolomeo, P., Thiebaut de Schotten, M., & Doricchi, F. (2007). Left Unilateral Neglect as a 509 Disconnection Syndrome. Cerebral Cortex, 17(11), 2479-2490. doi:10.1093/cercor/bhl181 510 Bastos, Andre M., Usrey, W. M., Adams, Rick A., Mangun, George R., Fries, P., & Friston, Karl J. 511 (2012). Canonical Microcircuits for Predictive Coding. Neuron, 76(4), 695-711. 512 doi:https://doi.org/10.1016/j.neuron.2012.10.038 513 Bruce, C. J., Goldberg, M. E., Bushnell, M. C., & Stanton, G. B. (1985). Primate frontal eye fields. II. 514 Physiological and anatomical correlates of electrically evoked eye movements. Journal of 515 Neurophysiology, 54(3), 714-734. 516 Bruineberg, J. (2017). Active inference and the primacy of the ‘I can’. 517 Catani, M., & ffytche, D. H. (2005). The rises and falls of disconnection syndromes. Brain, 128(10), 518 2224-2239. doi:10.1093/brain/awh622 519 Chica, A. B., Bartolomeo, P., & Valero-Cabré, A. (2011). Dorsal and Ventral Parietal Contributions to 520 Spatial Orienting in the Human Brain. The Journal of Neuroscience, 31(22), 8143. 521 Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention in the 522 brain. Nat Rev Neurosci, 3(3), 201-215. 523 David, O., Kiebel, S. J., Harrison, L. M., Mattout, J., Kilner, J. M., & Friston, K. J. (2006). Dynamic 524 causal modeling of evoked responses in EEG and MEG. NeuroImage, 30(4), 1255-1272. 525 Denzler, J., & Brown, C. M. (2002). Information theoretic sensor data selection for active object 526 recognition and state estimation. IEEE Transactions on pattern analysis and machine 527 intelligence, 24(2), 145-157. doi:10.1109/34.982896 528 Dietz, M. J., Friston, K. J., Mattingley, J. B., Roepstorff, A., & Garrido, M. I. (2014). Effective 529 connectivity reveals right-hemisphere dominance in audiospatial perception: implications 530 for models of spatial neglect. Journal of Neuroscience, 34(14), 5003-5011. 531 Doricchi, F., & Tomaiuolo, F. (2003). The anatomy of neglect without hemianopia: a key role for 532 parietal–frontal disconnection? Neuroreport, 14(17), 2239-2243. 533 Ellison, A., Schindler, I., Pattison, L. L., & Milner, A. D. (2004). An exploration of the role of the 534 superior temporal gyrus in visual search and spatial perception using TMS. Brain, 127(10), 535 2307-2315. doi:10.1093/brain/awh244 536 Felleman, D. J., & Van, D. E. (1991). Distributed hierarchical processing in the primate cerebral 537 cortex. Cerebral cortex (New York, NY: 1991), 1(1), 1-47. 538 Felleman, D. J., & Van Essen, D. C. (1991). Distributed Hierarchical Processing in the Primate Cerebral 539 Cortex. Cerebral Cortex, 1(1), 1-47. 540 Fries, W. (1984). Cortical projections to the superior colliculus in the macaque monkey: A retrograde 541 study using horseradish peroxidase. The Journal of Comparative Neurology, 230(1), 55-76. 542 doi:10.1002/cne.902300106 543 Fries, W. (1985). Inputs from motor and premotor cortex to the superior colliculus of the macaque 544 monkey. Behavioural Brain Research, 18(2), 95-105. doi:10.1016/0166-4328(85)90066-X
21
Active vision
545 Friston, K., & Buzsaki, G. (2016). The Functional Anatomy of Time: What and When in the Brain. 546 Trends Cogn Sci. doi:10.1016/j.tics.2016.05.001 547 Friston, K., FitzGerald, T., Rigoli, F., Schwartenbeck, P., & Pezzulo, G. (2016). Active Inference: A 548 Process Theory. Neural Computation, 29(1), 1-49. doi:10.1162/NECO_a_00912 549 Friston, K., Harrison, L., Daunizeau, J., Kiebel, S., Phillips, C., Trujillo-Barreto, N., . . . Mattout, J. 550 (2008). Multiple sparse priors for the M/EEG inverse problem. NeuroImage, 39(3), 1104- 551 1120. doi:https://doi.org/10.1016/j.neuroimage.2007.09.048 552 Friston, K., Mattout, J., Trujillo-Barreto, N., Ashburner, J., & Penny, W. (2007). Variational free energy 553 and the Laplace approximation. NeuroImage, 34(1), 220-234. 554 Friston, K. J., Harrison, L., & Penny, W. (2003). Dynamic causal modelling. NeuroImage, 19(4), 1273- 555 1302. doi:https://doi.org/10.1016/S1053-8119(03)00202-7 556 Friston, K. J., Litvak, V., Oswal, A., Razi, A., Stephan, K. E., van Wijk, B. C. M., . . . Zeidman, P. (2016). 557 Bayesian model reduction and empirical Bayes for group (DCM) studies. NeuroImage, 558 128(Supplement C), 413-431. doi:https://doi.org/10.1016/j.neuroimage.2015.11.015 559 Friston, K. J., & Penny, W. (2003). Posterior probability maps and SPMs. NeuroImage, 19(3), 1240- 560 1249. doi:https://doi.org/10.1016/S1053-8119(03)00144-7 561 Fruhmann Berger, M., Johannsen, L., & Karnath, H.-O. (2008). Time course of eye and head deviation 562 in spatial neglect. Neuropsychology, 22(6), 697-702. doi:10.1037/a0013351 563 Fullerton, K. J., McSherry, D., & Stout, R. W. (1986). Albert's Test: a Neglected Test of Perceptual 564 Neglect. The Lancet, 327(8478), 430-432. doi:http://dx.doi.org/10.1016/S0140- 565 6736(86)92381-0 566 Gaymard, B., Lynch, J., Ploner, C. J., Condy, C., & Rivaud-Péchoux, S. (2003). The parieto-collicular 567 pathway: anatomical location and contribution to saccade generation. European Journal of 568 Neuroscience, 17(7), 1518-1526. doi:10.1046/j.1460-9568.2003.02570.x 569 Geschwind, N. (1965). DISCONNEXION SYNDROMES IN ANIMALS AND MAN1. Brain, 88(2), 237-237. 570 doi:10.1093/brain/88.2.237 571 Goodale, M. A., & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in 572 Neurosciences, 15(1), 20-25. doi:http://dx.doi.org/10.1016/0166-2236(92)90344-8 573 Gregory, R. L. (1980). Perceptions as Hypotheses. Philosophical Transactions of the Royal Society of 574 London. B, Biological Sciences, 290(1038), 181. 575 Grimsen, C., Hildebrandt, H., & Fahle, M. (2008). Dissociation of egocentric and allocentric coding of 576 space in visual search after right middle cerebral artery stroke. Neuropsychologia, 46(3), 577 902-914. doi:http://dx.doi.org/10.1016/j.neuropsychologia.2007.11.028 578 Guillery, R. W., & Sherman, S. M. (2002). Thalamic Relay Functions and Their Role in Corticocortical 579 Communication. Neuron, 33(2), 163-175. doi:10.1016/S0896-6273(01)00582-7 580 Halassa, M. M., & Kastner, S. (2017). Thalamic functions in distributed cognitive control. Nature 581 neuroscience, 20(12), 1669-1679. doi:10.1038/s41593-017-0020-1 582 Halligan, P. W., & Marshall, J. C. (1998). Neglect of Awareness. Consciousness and Cognition, 7(3), 583 356-380. doi:http://dx.doi.org/10.1006/ccog.1998.0362 584 Hasson, U., Chen, J., & Honey, C. J. (2015). Hierarchical process memory: memory as an integral 585 component of information processing. Trends in cognitive sciences, 19(6), 304-313. 586 doi:10.1016/j.tics.2015.04.006 587 Hasson, U., Yang, E., Vallines, I., Heeger, D. J., & Rubin, N. (2008). A Hierarchy of Temporal Receptive 588 Windows in Human Cortex. The Journal of neuroscience : the official journal of the Society for 589 Neuroscience, 28(10), 2539-2550. doi:10.1523/JNEUROSCI.5487-07.2008 590 He, B. J., Snyder, A. Z., Vincent, J. L., Epstein, A., Shulman, G. L., & Corbetta, M. (2007). Breakdown of 591 Functional Connectivity in Frontoparietal Networks Underlies Behavioral Deficits in Spatial 592 Neglect. Neuron, 53(6), 905-918. doi:http://dx.doi.org/10.1016/j.neuron.2007.02.013 593 Hoffman, K. L., Dragan, M. C., Leonard, T. K., Micheli, C., Montefusco-Siegmund, R., & Valiante, T. A. 594 (2013). Saccades during visual exploration align hippocampal 3-8 Hz rhythms in human and
22
Active vision
595 non-human primates. Frontiers in Systems Neuroscience, 7, 43-43. 596 doi:10.3389/fnsys.2013.00043 597 Hohwy, J. (2007). The sense of self in the phenomenology of agency and perception. Psyche, 13(1), 598 1-20. 599 Husain, M., Mannan, S., Hodgson, T., Wojciulik, E., Driver, J., & Kennard, C. (2001). Impaired spatial 600 working memory across saccades contributes to abnormal search in parietal neglect. Brain, 601 124(5), 941-952. doi:10.1093/brain/124.5.941 602 Karnath, H.-O., & Rorden, C. (2012). The anatomy of spatial neglect. Neuropsychologia, 50(6), 1010- 603 1017. doi:10.1016/j.neuropsychologia.2011.06.027 604 Kiebel, S. J., Daunizeau, J., & Friston, K. J. (2008). A Hierarchy of Time-Scales and the Brain. PLoS 605 Computational Biology, 4(11), e1000209. doi:10.1371/journal.pcbi.1000209 606 Kiebel, S. J., David, O., & Friston, K. J. (2006). Dynamic causal modelling of evoked responses in 607 EEG/MEG with lead field parameterization. NeuroImage, 30(4), 1273-1284. 608 doi:https://doi.org/10.1016/j.neuroimage.2005.12.055 609 Künzle, H., & Akert, K. (1977). Efferent connections of cortical, area 8 (frontal eye field) in Macaca 610 fascicularis. A reinvestigation using the autoradiographic technique. The Journal of 611 Comparative Neurology, 173(1), 147-163. doi:10.1002/cne.901730108 612 Lenartowicz, A., Escobedo-Quiroz, R., & Cohen, J. D. (2010). Updating of context in working memory: 613 An event-related potential study. Cognitive, Affective, & Behavioral Neuroscience, 10(2), 614 298-315. doi:10.3758/CABN.10.2.298 615 Lindley, D. V. (1956). On a Measure of the Information Provided by an Experiment. Ann. Math. 616 Statist., 27(4), 986-1005. doi:10.1214/aoms/1177728069 617 Litvak, V., Mattout, J., Kiebel, S., Phillips, C., Henson, R., Kilner, J., . . . Friston, K. (2011). EEG and MEG 618 data analysis in SPM8. Comput Intell Neurosci, 2011, 852961. doi:10.1155/2011/852961 619 MacKay, D. J. C. (1992). Information-Based Objective Functions for Active Data Selection. Neural 620 Computation, 4(4), 590-604. doi:10.1162/neco.1992.4.4.590 621 McCloskey, M., & Rapp, B. (2000). Attention-referenced visual representations: evidence from 622 impaired visual localization. Journal of Experimental Psychology: Human Perception and 623 Performance, 26(3), 917. 624 Medina, J., Kannan, V., Pawlak, M. A., Kleinman, J. T., Newhart, M., Davis, C., . . . Hillis, A. E. (2009). 625 Neural Substrates of Visuospatial Processing in Distinct Reference Frames: Evidence from 626 Unilateral Spatial Neglect. Journal of cognitive neuroscience, 21(11), 2073-2084. 627 doi:10.1162/jocn.2008.21160 628 Mirza, M. B., Adams, R. A., Mathys, C., & Friston, K. J. (2018). Human visual exploration reduces 629 uncertainty about the sensed world. PLOS ONE, 13(1), e0190429. 630 doi:10.1371/journal.pone.0190429 631 Mirza, M. B., Adams, R. A., Mathys, C. D., & Friston, K. J. (2016). Scene Construction, Visual Foraging, 632 and Active Inference. Frontiers in Computational Neuroscience, 10(56). 633 doi:10.3389/fncom.2016.00056 634 Moran, R., Pinotsis, D. A., & Friston, K. (2013). Neural masses and fields in dynamic causal modeling. 635 Frontiers in Computational Neuroscience, 7, 57. doi:10.3389/fncom.2013.00057 636 Murray, J. D., Bernacchia, A., Freedman, D. J., Romo, R., Wallis, J. D., Cai, X., . . . Wang, X.-J. (2014). A 637 hierarchy of intrinsic timescales across primate cortex. Nature neuroscience, 17(12), 1661- 638 1663. doi:10.1038/nn.3862 639 Ognibene, D., & Baldassarre, G. (2014). Ecological Active Vision: Four Bio-Inspired Principles to 640 Integrate Bottom-Up and Adaptive Top-Down Attention Tested With a Simple Camera-Arm 641 Robot. Paper presented at the IEEE Transactions onAutonomous Mental Development. 642 Ota, H., Fujii, T., Suzuki, K., Fukatsu, R., & Yamadori, A. (2001). Dissociation of body-centered and 643 stimulus-centered representations in unilateral neglect. Neurology, 57(11), 2064-2069. 644 doi:10.1212/wnl.57.11.2064
23
Active vision
645 Parr, T., & Friston, K. J. (2017a). The active construction of the visual world. Neuropsychologia, 104, 646 92-101. doi:http://dx.doi.org/10.1016/j.neuropsychologia.2017.08.003 647 Parr, T., & Friston, K. J. (2017b). The Computational Anatomy of Visual Neglect. Cerebral Cortex, 1- 648 14. doi:10.1093/cercor/bhx316 649 Parr, T., & Friston, K. J. (2017c). Uncertainty, epistemics and active inference. Journal of The Royal 650 Society Interface, 14(136). 651 Parr, T., & Friston, K. J. (2017d). Working memory, attention, and salience in active inference. Sci 652 Rep, 7(1), 14678. doi:10.1038/s41598-017-15249-0 653 Parr, T., Rees, G., & Friston, K. J. (2018). Computational Neuropsychology and Bayesian Inference. 654 Frontiers in Human Neuroscience, 12(61). doi:10.3389/fnhum.2018.00061 655 Pierrot-Deseilligny, C., Rivaud, S., Gaymard, B., Müri, R., & Vermersch, A.-I. (1995). Cortical control of 656 saccades. Annals of Neurology, 37(5), 557-567. doi:doi:10.1002/ana.410370504 657 Platz, T., Schüttauf, J., Aschenbach, J., Mengdehl, C., & Lotze, M. (2016). Effects of inhibitory theta 658 burst TMS to different brain sites involved in visuospatial attention–a combined 659 neuronavigated cTBS and behavioural study. Restorative Neurology and Neuroscience, 34(2), 660 271-285. 661 Sajad, A., Sadeh, M., Keith, G. P., Yan, X., Wang, H., & Crawford, J. D. (2015). Visual–Motor 662 Transformations Within Frontal Eye Fields During Head-Unrestrained Gaze Shifts in the 663 Monkey. Cerebral Cortex, 25(10), 3932-3952. doi:10.1093/cercor/bhu279 664 Serences, J. T., Shomstein, S., Leber, A. B., Golay, X., Egeth, H. E., & Yantis, S. (2005). Coordination of 665 Voluntary and Stimulus-Driven Attentional Control in Human Cortex. Psychological Science, 666 16(2), 114-122. doi:10.1111/j.0956-7976.2005.00791.x 667 Shipp, S. (2007). Structure and function of the cerebral cortex. Current Biology, 17(12), R443-R449. 668 doi:10.1016/j.cub.2007.03.044 669 Tenenbaum, J. B., Kemp, C., Griffiths, T. L., & Goodman, N. D. (2011). How to grow a mind: statistics, 670 structure, and abstraction. Science, 331(6022), 1279-1285. doi:10.1126/science.1192788 671 Thiebaut de Schotten, M., Dell'Acqua, F., Forkel, S. J., Simmons, A., Vergani, F., Murphy, D. G. M., & 672 Catani, M. (2011). A lateralized brain network for visuospatial attention. Nat Neurosci, 673 14(10), 1245-1246. doi:10.1038/nn.2905 674 http://www.nature.com/neuro/journal/v14/n10/abs/nn.2905.html#supplementary-information 675 Thiebaut de Schotten, M., Urbanski, M., Duffau, H., Volle, E., Lévy, R., Dubois, B., & Bartolomeo, P. 676 (2005). Direct Evidence for a Parietal-Frontal Pathway Subserving Spatial Awareness in 677 Humans. Science, 309(5744), 2226. 678 Ungerleider, L. G., & Haxby, J. V. (1994). ‘What’ and ‘where’ in the human brain. Current Opinion in 679 Neurobiology, 4(2), 157-165. doi:http://dx.doi.org/10.1016/0959-4388(94)90066-3 680 Verdon, V., Schwartz, S., Lovblad, K.-O., Hauert, C.-A., & Vuilleumier, P. (2009). Neuroanatomy of 681 hemispatial neglect and its functional components: a study using voxel-based lesion- 682 symptom mapping. Brain, 133(3), 880-894. doi:10.1093/brain/awp305 683 Vernon, D. (2008). Cognitive vision: The case for embodied perception. Image and Vision Computing, 684 26(1), 127-140. doi:https://doi.org/10.1016/j.imavis.2005.08.009 685 Vossel, S., Weidner, R., Driver, J., Friston, K. J., & Fink, G. R. (2012). Deconstructing the Architecture 686 of Dorsal and Ventral Attention Systems with Dynamic Causal Modeling. The Journal of 687 Neuroscience, 32(31), 10637. 688 Wurtz, R. H., McAlonan, K., Cavanaugh, J., & Berman, R. A. (2011). Thalamic pathways for active 689 vision. Trends Cogn Sci., 5(4), 177-184. 690 Yang, S. C.-H., Lengyel, M., & Wolpert, D. M. (2016). Active sensing in the categorization of visual 691 patterns. eLife, 5, e12215. doi:10.7554/eLife.12215 692 Yuille, A., & Kersten, D. (2006). Vision as Bayesian inference: analysis by synthesis? Trends in 693 cognitive sciences, 10(7), 301-308. doi:https://doi.org/10.1016/j.tics.2006.05.002 694 Zeki, S., & Shipp, S. (1988). The functional logic of cortical connections. Nature, 335(6188), 311-317.
24
Active vision
695 Zimmermann, E., & Lappe, M. (2016). Visual Space Constructed by Saccade Motor Maps. Frontiers in 696 Human Neuroscience, 10(225). doi:10.3389/fnhum.2016.00225 697
698
25