The Frontal Eye Field As a Prediction

CHAPTER 5.2

The frontal eye ﬁeld as a prediction map

Trinity B. CrapseÃ and Marc A. Sommer

Department of Neuroscience and the Center for the Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA

Abstract: Predictive processes are widespread in the motor and sensory areas of the primate brain. They enable rapid computations despite processing delays and assist in resolving noisy, ambiguous input. Here we propose that the frontal eye ﬁeld, a cortical area devoted to sensorimotor aspects of eye movement control, implements a prediction map of the postsaccadic visual scene for the purpose of constructing a stable percept despite saccadic eye movements.

Keywords: Bayesian; forward model; saccadic eye movements; macaca mulatta; corollary discharge; shifting receptive ﬁelds; visual stability; feedback connections

Introduction consider the saccade as an event comprised of three constituent parts: presaccadically, a target is The brain is an inferential machine. Both its motor selected; intrasaccadically, the world as sensed areas and sensory networks engage in predictive actually shifts; and postsaccadically, the target is computations (Miall and Wolpert, 1996; Bullier, foveated. The intrasaccadic component is the 2001; Friston, 2005). Hierarchical models suggest problem. Were the animal to perceive the world that visual cortical neurons fire predominately to exactly as sensed during the intrasaccadic epoch, signal deviations from predicted inputs (Rao and the visual scene would seem to leap from place- Ballard, 1999; Lee and Mumford, 2003). Studies to-place dozens of times per minute. Yet it is in the motor realm, too, suggest that predictions not perceived as such, and the brain can even are used for adaptive control (Miall and Wolpert, distinguish what aspects of the jumpy visual 1996; Hwang and Shadmehr, 2005). Here we inflow are artefactual (due to saccades) as opposed propose that predictive operations for both the to real (due to changes in the world). How does sensory and motor domains find unification in the it do it? primate visuosaccadic system for the purpose of An important cortical gaze control component constructing a stable transaccadic percept. is the frontal eye field (FEF) (Fig. 2; Sommer and The primate visuosaccadic system faces a serious Wurtz, 2008). Located within the anterior bank of problem (Fig. 1). It must generate a stable percept the arcuate sulcus, the FEF contains topographi- despite the frequent disruptions in gaze induced by cally arranged neurons with response proper- saccadic eye movements. To appreciate this, ties that span the continuum from purely visual to purely movement. The FEF receives input from many cortical and subcortical areas inclu- ÃCorresponding author. Tel.: +1 412 268 7229; ding the superior colliculus (SC) via thalamic Fax: +1 412 268 5060; E-mail: [email protected] relay neurons. We recently demonstrated that

DOI: 10.1016/S0079-6123(08)00656-0 383 384

Presaccadic Full Field Representation

m Fov

Intrasaccadic

v=-m v

Postsaccadic

FEF Fov

Fig. 1. The saccadic eye movement as a visual problem. Each saccade is composed of three elements: a presaccadic component when a target is selected and the saccade initiated, an intrasaccadic component during which time the world as sensed actually shifts, and a postsaccadic component when the target is reached. The intrasaccadic component is the most disruptive element for the visual system. The fact that the world shifts with Ips Con each saccade, and yet is not perceived to do so, implies a SC SC corrective mechanism within the visual system. This mechanism is rooted in predictive operations. Fov, fovea; m, movement vector; v, retinal vector emphasizing direction and magnitude of the shift. Fig. 2. Full-field representation in the frontal eye field (FEF). The FEF has information about all of visual space and all eye movements via pathways from ipsilateral (Ips) and contral- ateral (Con) superior colliculus (SC). convergent inputs from both SCs provide each FEF with a full-field representation of all saccades and all of visual space (Crapse and a shifting RF is dynamic and starts sampling Sommer, 2007). the new RF location even before a saccade. Like several other primate areas, the FEF Such neurons depend on corollary discharge contains neurons that shift their response fields (CD) from the midbrain to trigger the shift and (RFs) before eye movements (a ‘‘shifting RF’’) are thought to contribute to a percept of visual (Umeno and Goldberg, 1997). While a typical RF stability (Sommer and Wurtz, 2006, 2008). is firmly retinotopic and samples a new part of the How might these neurons influence the rest of the visual field (the new RF) only after the eye moves, brain? 385

A prediction map in primate frontal eye field prior and conditional probability distributions to be utilized to generate a posterior distribution, i.e., We propose that shifting neurons of the FEF are the outcome expected on the balance of known components of a larger FEF inferential architec- probabilities and other inputs (in other words, ture that engages in predictive coding. That is, the a prediction; Fig. 3). For the case of a single FEF is a prediction map. This scheme assigns a saccade, the proposed Bayesian computation would causal role to the FEF in the construction of a express the conditional expectation or probability stable visual percept despite saccadic interruptions. of observing the postsaccadic scene (p) given the According to this conception, predictions of the relative probability of two independent pieces of future scene (postsaccadic) are generated based on evidence: the visual structure of the current scene extrapolations from the current scene (presacca- (c), and various saccadic parameters (s), written: dic). Neurons with shifting RFs, informed spatio- PðcjpÞPðsjpÞPðpÞ temporally of the imminent change in gaze by Pðpjc; sÞ¼ ¼ lr CD, collect data about what will fall in their RF PðcÞPðsÞ after the saccade. These data are convolved with estimates of an internal model of the visual world. Bayesian priors about the structure of the visual What follows is a prediction of the visual structure scene P(c) and how the world behaves during a of the postsaccadic scene. saccade P(s) could be learned and updated through How would the prediction be used? We experience and stored in FEF networks. The priors hypothesize that around the time of the saccade, would permit calculation of local conditional prob- the predictive signals are sent from the FEF to the abilities relating the postsaccadic scene to the current posterior lobes. The visual lobes are primed with scene P(c|p) and the postsaccadic scene to the saccade P(s|p). Ultimately, these computations would activity-constraining expectations. Deviations r from the predicted inputs (prediction errors) result in a global conditional expectation l that would be convolved with a generative model C to are reported back to the FEF. The prediction r errors therefore are ultimately manifest in the generate a prediction of the postsaccadic scene C(l ). The prediction would be compared with the actual visual responses of FEF neurons. This residual a a r alerts the FEF to unpredicted events. This may input l to yield a prediction error e=l C(l ). entail calibration/updating if the error is related to a miscalculation/noise or it may provide useful information about the environment that was Physiological mechanism of the prediction map unexpected, i.e., perhaps something moved or suddenly appeared. Either way, the final result of Mechanistically, frontal modulatory control of the the iterative signalling would be the perception of posterior lobes could be implemented through a stable world despite the change in gaze. In the imposed patterns of synchronization mediated by rest of this article we will formalize these ideas. cortico-cortical connections (Womelsdorf et al., 2007). Visually evoked activity of single neurons is surprisingly quite deterministic (Arieli et al., Computational basis of the prediction map 1996). The oft-encountered variability in single neuron responses seems to emerge from the The prediction map may be grounded in an dynamics of ongoing network activity. This fact inferential process based on empirical Bayesian could be exploited by the prediction map for principles. Neurons throughout the brain seem to purposes of ensuring transaccadic perception. engage in probabilistic and inferential computa- The imposed predictions could alter the spatio- tions (Gold and Shadlen, 2007). The probabilistic temporal properties of visual cortical network aspect seems necessary because of the noise and activity to influence the specific visually evoked ambiguity intrinsic to neural computation (Knill responses that are ultimately integrated into and Pouget, 2004). Bayesian inference would allow the network and expressed as spikes. The FEF 386

presaccadic

m Fov

current scene saccade

Fov c s v m postsaccadic

Fov

P(c|p)P(s|p)P(p) ρ conditional P(p|c,s)= = λ P(c)P(s) expectation

ρ Ψ(λ ) prediction α λ actual input

α ρ prediction ε =−λ Ψ(λ ) error

Fig. 3. Computational basis of the prediction map. A prediction map could be based on Bayesian computations. See text for details. 387 would initiate a speciﬁc synchronization pattern The sequence of events among the relevant visual neurons (those activated by features of the current scene) and this would A detailed account of what we propose occurs serve to amplify behaviourally relevant signals in during each voluntary saccade is as follows (Fig. 4). the cortex. The imminent saccade would initiate an iteration

D(h)

ƒhD(h) (t)

A ƒhD(h) (t)=x, ifChD(h) (t) = 0 ChD(h) (t)

IhD(h) (t) h ƒhD(h) (t) = I hD(h) (t)C hD(h) (t), if C hD(h) (t) > 0

x ...P(h)

Feedback

FeedForward FEF B CD Visual Input MD

Prediction Ψ(λ ρ)

Prediction Error Visual FEF α ρ ελ= − Ψ (λ ) Areas CD SC x λα y Visual Input Retina

Fig. 4. Mechanics of the prediction map. (A) A centrepiece of the prediction map would be neurons with shifting receptive ﬁelds.

These would serve as network hubs (h) that selectively route visual information fhD(h)(t) through the network based on the information IhD(h)(t) provided by upstream neurons P(h) and CD from the midbrain ChD(h)(t). This information would then be forwarded to neurons downstream D(h) that perform additional computations for the sake of prediction. Equations are explained in text. (B) Predictions C(lr) would exit the FEF as feedback that targets various areas of the visual brain. Errors e=laC(lr) between the predicted visual input C(lr) and actual visual input la would be then routed back to the FEF. 388 of recurrent processing between the FEF and an stream components seem most likely for two assortment of visual cortical areas, beginning with reasons. First, the temporal structure of informa- receipt in the FEF of CD (leftmost line) from tion flow through the primate visual system points the SC. This information represents the when and to a dorsal stream speed advantage over the where (a vector quantity) of the imminent change ventral stream (Bullier, 2001; Bar, 2007). Dorsal in gaze and induces a transient alteration in local stream components exhibit activation latencies functional topology of the FEF network. At the that often trail V1 responses by a meagre 10 ms. centre of the alteration would be FEF network Some even activate before V1 (Schmolesky et al., hubs (h) (Sporns et al., 2007). FEF network hubs 1998; Bullier et al., 2001). This speed advantage have access to information about the entire visual would be optimal for the rapid detection and field (IhD(h)(t)) and are considered to be neurons routing of prediction errors back to the FEF. with shifting RFs. The CD serves a gating function Second, the dorsal stream is known as the where (ChD(h)(t)) and routes the appropriate visual pathway (Ungerleider and Mishkin, 1982). Com- information through the hub ( fhD(h)(t)) to down- ponents are concerned with motion and the stream neurons D(h). [If CD is not present physical location of objects in the visual scene. (ChD(h)(t)=0), the hub simply passes on the visual Since the primary concern of the visual system information provided to it from node x.] These during a saccade is to ensure a stable world, the relationships may be written as: dorsal stream would seem to be particularly important in controlling the illusory percept f hDðhÞðtÞ¼IhDðhÞðtÞChDðhÞðtÞ; if ChDðhÞðtÞ40 of motion concomitant to saccade generation. Otherwise : f hDðhÞðtÞ¼x; if ChDðhÞðtÞ¼0 Any errors related to the prediction could be a consequence of an object that actually moved on The RF is said to shift and corresponds to a its own accord and thus was not predicted. The collection of visual information about the portions object would be with high probability worthy of of space that the RF will encompass after the further inspection. The FEF would then direct the saccade. This information then would be com- allocation of attentional and targeting mechanisms bined with the causal estimates of the generative to these objects via retrojections of activity to model embedded within the FEF network. A stations of the ventral stream such as V4 and IT prediction would be generated and exported, (Moore and Armstrong, 2003; Hamker, 2005). manifested as a round of perisaccadic synchronized activity between the frontal and posterior lobes. The synchronized activity would alter the Relation to previous FEF studies visual cortical network dynamics and effect how the visual stimulus-driven activity is integrated Previous studies have uncovered a number of FEF into the network. The synchronized activity is response properties consistent with the notion of equivalent to a prediction, the output of the a prediction map. One implication of prediction generative model. If the actual visual input does error in general is that single neurons should not match this prediction, then residuals (predic- not respond if the stimulus falling in its RF is tion errors) are routed downstream back to the predicted. Burman and Segraves (1994) found that FEF and iterative processes are continued. when monkeys rescanned a previously scanned image, visual activity was virtually unaffected by the contents of the image that fell within the RF. Site of prediction error calculation In contrast, these same neurons fired vigorously during the initial scan and when a target light The prediction errors could be calculated at any suddenly appeared in the RF. After the initial number of visual cortical depots. Virtually every scanning, a memory trace of the image was likely portion of the cortical mantle exhibits some degree formed, and only deviations from the predicted of saccadic modulation (Baker et al., 2006). Dorsal image components were signalled as visual bursts. 389

This suggests that the visual response of FEF In a more speculative sense, the frontal lobes are neurons signals prediction error, the unexpected. implicated in processes related to mental imagery An additional implication of a prediction map (Goebel et al., 1998). When a subject is asked to applies to the movement-related aspects of the ‘‘imagine’’ a particular image the frontal lobes FEF, generally. If a function of the FEF is to invariably come on line. Feedback connections generate predictions, then movement responses, from the frontal lobes to the posterior lobes are too, may be viewed probabilistically. In fact, some thought to activate the same circuits that would be pure movement cells have been found that activated if the actual image had been presented to generate vigorous bursts even when the monkey the retinas (Kosslyn et al., 1995). Furthermore, the errs and does not perform the saccade (Bruce and eye movements of subjects asked to imagine a Goldberg, 1985). Other movement neurons fail to particular scene reflect the content and spatial fire even when a saccade is launched into its RF. arrangement of objects populating the scene This is evidence that the movement signal in (Mast and Kosslyn, 2002). Again this may involve FEF is closely akin to a prediction or probability the same circuits, as ancestral primates were likely estimate of a saccade being generated into its making saccades and requiring mechanisms of movement field. It bears little relationship to transsaccadic stability long before they were saccade dynamics (Segraves and Park, 1993). We engaging in mental imagery and imaginative suggest that the forced manner in which FEF musings. movement signals have been studied classically, counting spikes while monkeys make required eye movements into a neuron’s RF, may mask this probabilistic aspect of FEF function and redirect References focus on alleged deterministic aspects of movement generation. Arieli, A., Sterkin, A., Grinvald, A. and Aertsen, A. (1996) Dynamics of ongoing activity: explanation of the large variability in evoked cortical responses. Science, 273: 1868–1871. Other frontal lobe functions Baker, J.T., Patel, G.H., Corbetta, M. and Snyder, L.H. (2006) Distribution of activity across the monkey cerebral cortical The prediction map is consistent with a number of surface, thalamus and midbrain during rapid, visually guided other phenomena involving the frontal lobes and saccades. Cereb. Cortex, 16: 447–459. Bar, M. (2007) The proactive brain: using analogies and visual function. Among other things, the frontal associations to generate predictions. Trends Cogn. Sci., 11: lobes are thought to play a role in resolving visual 280–289. ambiguity. Visual scenes are often ambiguous; Bruce, C.J. and Goldberg, M.E. (1985) Primate frontal eye the sensory data are consistent with multiple fields. I. Single neurons discharging before saccades. interpretations. This often results in illusions and J. Neurophysiol., 53: 603–635. Bullier, J. (2001) Integrated model of visual processing. Brain multistable percepts, i.e., depth reversals, binocu- Res., 36: 96–107. lar rivalry, ambiguous figures, etc. A host of Bullier, J., Hupe, J.M., James, A.C. and Girard, P. (2001) studies point to a role of the frontal lobes, FEF The role of feedback connections in shaping the included, in resolving visual ambiguity (Lumer responses of visual cortical neurons. Prog. Brain Res., 134: et al., 1998). Some contend that the frontal lobes 193–204. Burman, D.D. and Segraves, M.A. (1994) Primate frontal bias or control the posterior lobes in an attempt to eye field activity during natural scanning eye movements. resolve the ambiguity and arrive at a stable percept J. Neurophysiol., 71: 1266–1271. (Miller and Cohen, 2001). We submit that the Crapse, T.B. and Sommer, M.A. (2007) Frontal eye field same frontal circuits that are responsible for neurons receiving input from both superior colliculi: are their resolving visual ambiguity and arriving at a stable receptive fields tuned or untuned? Program No. 19.10. 2007 Neuroscience Meeting Planner, San Diego, CA, Society for percept of the present scene are part of the Neuroscience (online). prediction map and are utilized by the oculomotor Friston, K. (2005) A theory of cortical responses. Philos. Trans. system for transsaccadic stability. R. Soc. Lond., 360: 815–836. 390

Goebel, R., Khorram-Sefat, D., Muckli, L., Hacker, H. and Moore, T. and Armstrong, K.M. (2003) Selective gating of Singer, W. (1998) The constructive nature of vision: direct visual signals by microstimulation of frontal cortex. Nature, evidence from functional magnetic resonance imaging studies 421: 370–373. of apparent motion and motion imagery. Eur. J. Neurosci., Rao, R.P. and Ballard, D.H. (1999) Predictive coding in the 10: 1563–1573. visual cortex: a functional interpretation of some extra- Gold, J.I. and Shadlen, M.N. (2007) The neural basis of classical receptive-field effects. Nat. Neurosci., 2: 79–87. decision making. Annu. Rev. Neurosci., 30: 535–574. Schmolesky, M.T., Wang, Y., Hanes, D.P., Thompson, K.G., Hamker, F.H. (2005) The reentry hypothesis: the putative Leutgeb, S., Schall, J.D. and Leventhal, A.G. (1998) Signal interaction of the frontal eye field, ventrolateral prefrontal timing across the macaque visual system. J. Neurophysiol., cortex, and areas V4, IT for attention and eye movement. 79: 3272–3278. Cereb. Cortex, 15: 431–447. Segraves, M.A. and Park, K. (1993) The relationship of monkey Hwang, E.J. and Shadmehr, R. (2005) Internal models of limb frontal eye field activity to saccade dynamics. J. Neurophysiol., dynamics and the encoding of limb state. J. Neural Eng., 2: 69: 1880–1889. S266–S278. Sommer, M.A. and Wurtz, R.H. (2006) Influence of the Knill, D.C. and Pouget, A. (2004) The Bayesian brain: the thalamus on spatial visual processing in frontal cortex. role of uncertainty in neural coding and computation. Trends Nature, 444: 374–377. Neurosci., 27: 712–719. Sommer, M.A. and Wurtz, R.H. (2008) Brain circuits for Kosslyn, S.M., Thompson, W.L., Kim, I.J. and Alpert, N.M. the internal monitoring of movements. Annu. Rev. Neurosci., (1995) Topographical representations of mental images in 31: 317–338. primary visual cortex. Nature, 378: 496–498. Sporns, O., Honey, C.J. and Kotter, R. (2007) Identification Lee, T.S. and Mumford, D. (2003) Hierarchical Bayesian and classification of hubs in brain networks. PLoS ONE, 2: inference in the visual cortex. J. Opt. Soc. Am., 20: 1434– p. e1049. 1448. Umeno, M.M. and Goldberg, M.E. (1997) Spatial processing Lumer, E.D., Friston, K.J. and Rees, G. (1998) Neural in the monkey frontal eye field. I. Predictive visual responses. correlates of perceptual rivalry in the human brain. Science, J. Neurophysiol., 78: 1373–1383. 280: 1930–1934. Ungerleider, L.G. and Mishkin, M. (1982) Two cortical visual Mast, F.W. and Kosslyn, S.M. (2002) Eye movements during systems. In: Ingle D.J., Goodale M.A. and Mansfield R.J.W. visual mental imagery. Trends Cogn. Sci., 6: 271–272. (Eds.), Analysis of Visual Behavior. The MIT Press, Miall, R.C. and Wolpert, D.M. (1996) Forward models for Cambridge, MA. physiological motor control. Neural Netw., 9: 1265–1279. Womelsdorf, T., Schoffelen, J.M., Oostenveld, R., Singer, W., Miller, E.K. and Cohen, J.D. (2001) An integrative Desimone, R., Engel, A.K. and Fries, P. (2007) Modulation theory of prefrontal cortex function. Annu. Rev. Neurosci., of neuronal interactions through neuronal synchronization. 24: 167–202. Science, 316: 1609–1612.