Prefrontal Mechanisms Underlying Sequential Tasks By Feng-Kuei Chiang A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Psychology in the Graduate Division of the University of California, Berkeley Committee in charge: Professor Joni Wallis, Chair Professor Richard B. Ivry Professor Ming Hsu Summer 2017 Prefrontal Mechanisms Underlying Sequential Tasks Copyright c 2017 by Feng-Kuei Chiang Abstract Prefrontal Mechanisms Underlying Sequential Tasks by Feng-Kuei Chiang Doctor of Philosophy in Psychology University of California, Berkeley Professor Joni Wallis, Chair For decades, mechanisms of cognitive behaviors have been studied in a simple form of stimulus- and action-outcome associations. Those seminal studies serve as funda- mental frameworks and enable us to explore how neural activities in brain represent the increasingly complex and temporarily-extended associations in sequential tasks. The prefrontal cortex (PFC) has long been suspected to play an important role in cognitive control, in the ability to orchestrate thought and action in accordance with internal representation. In particular, PFC \top-down" processes serve as an internal signal to guide high-level cognitive functions, such as working memory (WM), abstract rules, or goal-directed decision-making. Several models have described how task information, in- cluding supra- and super-ordinate information, is organized in prefrontal cortex, but it remains unclear precisely how this cognitive information maps onto neurophysiological functions. To explore these issues, we devised primate versions of two tasks that tax sequential behavior: a spatial self-ordered search task and a hierarchical reinforcement learning (HRL) task. These tasks examine how sequential behavior interacts with WM and reinforcement learning (RL), respectively. We examined how prefrontal neurons en- coded task-related information across these two cognitive tasks. Our results show that prefrontal neurons are capable of adaptively regulating the precision with which infor- mation is encoded. In the spatial self-ordered search task, lateral prefrontal neurons have spatiotemporal mnemonic fields, in that their firing rates are modulated both by the spatial location of future selection behaviors and the temporal organization of that behavior. Furthermore, the precision of this tuning can be dynamically modulated by the demands of the task. In the HRL task, prefrontal neurons are involved in inte- grating the abstract subject values. Especially, we found that the firing rate of a small population of neurons encoded pseudo-reward prediction errors and these neurons were restricted to anterior cingulate cortex. Taken together, our findings suggest that pre- frontal neurons encode not only basic information associated with external stimuli, but also high-level information that is used to organize task relevant behaviors. 1 To my parents, family, and friends All is Well i Contents 1 Introduction 1 1.1 Outline of thesis . 2 1.2 Prefrontal cortex (PFC) . 3 1.2.1 Lateral prefrontal cortex (LPFC) . 3 1.2.2 Orbitofrontal cortex (OFC) . 5 1.2.3 Anterior cingulate cortex (ACC) . 7 1.2.4 Closing remarks on anatomy . 9 1.3 The role of PFC in working memory . 9 1.4 The role of PFC in reinforcement learning . 10 1.4.1 Historical perspective . 11 1.4.2 Reinforcement learning and hierarchical behaviors . 11 2 General Methods 19 2.1 Overview . 19 2.2 Behavioral training materials and methods . 19 2.2.1 Subjects . 19 2.2.2 Behavioral training . 20 2.2.3 Materials and methods for training . 20 2.3 Neurophysiological techniques . 21 2.3.1 Isolation of recording sites . 21 2.3.2 Surgery . 22 2.4 Recordings . 23 2.4.1 Materials and methods for neurophysiology . 23 2.5 Statistical analysis . 24 3 Spatiotemporal encoding of search strategies by prefrontal neurons 28 3.1 Introduction . 28 3.2 Methods . 29 3.3 Results . 30 3.3.1 Task performance . 30 3.3.2 Behavioral strategies . 31 3.3.3 Neurophysiological analysis . 32 3.3.4 Encoding of spatial information . 33 ii 3.3.5 Effects of behavioral strategy on spatial encoding . 34 3.4 Discussion . 35 3.4.1 Role of prefrontal cortex in the sequential organization of behavior . 35 3.4.2 Contribution of PFC to working memory . 36 3.4.3 Role of PFC in cognitive control . 37 4 Neuronal encoding in prefrontal cortex during hierarchical reinforcement learning 48 4.1 Introduction . 48 4.2 Methods . 49 4.2.1 Subjects and behavioral task . 49 4.2.2 Neurophysiological procedures . 51 4.2.3 Statistical methods . 51 4.3 Results . 54 4.3.1 Behavioral task performance . 54 4.3.2 Neural encoding . 55 4.4 Discussion . 55 5 Conclusion 67 5.1 Summary of results . 67 5.2 Remaining questions . 68 5.2.1 Errors in sequential tasks . 68 5.2.2 Adaptation from prediction errors . 69 5.3 Future directions . 69 5.4 Closing remarks . 70 6 Bibliography 71 iii List of Figures 1.1 Hierarchical representation of a routine sequential task. From Humphreys and Forde, 1998. 14 1.2 (A) Flow of control in one step of a sequential task, with blue representing the increased involvement of supervisory control and red representing increased involvement of schematic control during a single step. (B) Representation of the multiple, hierarchical levels that can characterize sequences. Each step in more concrete motor sequences or more abstract task sequences may engage supervisory or schematic control and the interaction between them. From Desrochers et al. (2015). 15 1.3 The medial, lateral, and orbital surfaces of the prefrontal cortex. Monkey outlines taken from Carmichael and Price (1996) and Petrides and Pandya (1999); human outline taken from Ongur and Price (2000). 16 1.4 Medial and orbital networks of the prefrontal cortex taken from Carmichael and Price (1996). Adapted and re-printed with permission from Wallis (2012). 17 1.5 Overview of Content-Specific Activity during Working Memory Delays in the Macaque (Left) and Human (Right) Brain. Icons indicate persistent stimulus- selective activity for each stimulus type indicated by the icon (see legend) at the respective locations. Both left- and right-sided effects are shown on the left hemisphere. A full list of individual studies is reported in the supple- mental information from Christophel, et al., 2017. Brain areas are identified by abbreviations: AC, auditory cortex; ERC, entorhinal cortex; EVC, early visual cortex; FEF, frontal eye fields; FG, fusiform gyrus; hMT+, human analog to MT/MST; IPS, intraparietal sulcus; IT, inferior temporal cortex; LOC, lateral occipital complex; lPFC, lateral prefrontal cortex; PM, premotor cortex; PPC, posterior parietal cortex. 18 2.1 Experimental set up used to control behavioral events and record neurophys- iological data. 25 2.2 Magnetic resonance imaging (MRI) scans illustrating the coronal slice from the middle of our recording locations illustrating potential electrode paths. Example on the top is from subject R in WM task. Example on the bottom is from subject Q in HRL task. Possible electrode tracks are highlighted in white. Brain areas recorded from are highlighted in red (LPFC), green (ACC), and blue (OFC). 26 iv 2.3 Cluster isolation of neuronal waveforms. On the left are 32 channels, of which many have neurons on them, i.e., detected waveforms. A sample channel is zoomed in in the middle panel to reveal three distinct waveforms. Those waveforms are decomposed into components in Plexon's online sorter, and each waveform is plotted as a single dot in the right panel. Clusters of wave- forms are then isolated manually and entered into neuronal analyses. 27 3.1 (A) Spatial self-ordered search task. (B) Each configuration consisted of 6 targets (green filled circle), which were selected from 36 possible locations (gray filled circle). We ensured that targets were approximately balanced across the display by requiring that the centroid of the configuration (red cross) was located within ± 3◦ fixation window (black circle). The inter- target distances were spaced at least 6◦ to avoid overlap of the eye position detection window around the target. (C) Number of unique sequences of target selection per configuration. 39 3.2 Behavioral performance. (A) Distribution of the number of incorrect saccades per trial. (B) Observed and expected error rates plotted as a function of which saccade in the sequence was performed. The observed error rate was significantly higher than the expected error rate for the last saccade in both subjects (binomial test, ∗ ∗ ∗p < 0:001). (C) Reaction times as a function of saccade. Boxplots indicate the 25th, 50th, and 75th percentiles of the sRT distribution. Red dots indicated the mean sRT. 40 3.3 (A) For each block of trials, we calculated how many times a particular target was selected by a particular saccade. We then plotted this data according to the most common sequence by which targets were selected. (B) Representative blocks, illustrating how the pseudocolor plots changed as a function of the stereotype index. Blocks above the line are from subject R, while blocks below the line are from subject Q. (C) Increased values of the stereotype index were correlated with better behavioral performance (fewer incorrect saccades) in both subjects. 41 3.4 Recording locations. (A) MRI of a coronal slice through the frontal lobe of subject R. Red region in each hemisphere denotes the area of the LPFC inves- tigated. White lines depict electrode paths. (B) We measured the anterior- posterior position from the interaural line (x-axis), and the lateral-medial position relative to the lip of the ventral bank of the principal sulcus (0 point on y-axis). Gray shading indicates unfolded sulci. Diameter of the circles in- dicate the number of recordings from a given location. SA = superior arcuate sulcus; IA = inferior arcuate sulcus; P = principal sulcus. 42 3.5 (A) Percentage of selective neurons that encoded different predictors in each epoch.