Neural dynamics and interactions in the human ventral visual pathway Yuanning Li December 2018 Center for the Neural Basis of Cognition, Dietrich College of Humanities and Social Sciences and Machine Learning Department, School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee: Avniel Singh Ghuman (Co-Chair), Max G. G’Sell (Co-Chair), Robert E. Kass, Christopher I. Baker (National Institute of Mental Health) Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy. Copyright c 2018 Yuanning Li This work was supported by the National Institute on Drug Abuse Predoctoral Training Grants under R90DA023426 and R90DA023420, the National Institute of Mental Health under awards R01MH107797 and R21MH103592, and the National Science Foundation under award 1734907. The content is solely the responsibility of the author and does not necessarily represent the official views of the National Institutes of Health or the National Science Foundation. Keywords: Cognitive neuroscience, computational neuroscience, machine learning, statisti- cal methods, decoding, functional connectivity, neuroimaging, intracranial electroencephalogra- phy, visual perception To all the beautiful souls that lit up my life iv Abstract The ventral visual pathway in the brain plays central role in visual object recog- nition. The classical model of the ventral visual pathway, which poses it as a hier- archical, distributed and feed-forward network, does not match the actual structure of the pathway, which is highly interconnected with reciprocal and non-hierarchical projections. Here we address three major consequences of this non-classical struc- ture with regard to neural dynamics and interactions: (i) the model does not consider any extended information processing dynamics; (ii) the model does not allow for adaptive and recurrent interactions between areas; (iii) the model only character- izes evoked-response with no state-dependence from the neural context. To begin to address these gaps in the classical model, we focus on the categorical-selective regions in the ventral pathway and study the neural dynamics and interactions us- ing intracranial electroencephalography (iEEG), which overcomes the limitations of spatiotemporal resolution in current non-invasive human neuroimaging techniques. With respect to the first consequence, we applied multivariate pattern analysis (MVPA) methods to the iEEG signal to analyze the dynamic roles of the word and face sensitive areas. We found that both areas demonstrated a similar multi-stage information processing dynamic wherein the representation in category-selective fusiform gyrus evolves from a gist category-level and similarity-based representa- tion to an invariant and highly detailed individual representation over the course of 500 ms. In addition, our results also suggest a dissociation between structural and motion in the face processing streams. Regarding to the second consequence, we introduced a novel method termed Multi-Connection Pattern Analysis (MCPA) to extract the discriminant information about cognitive states solely from the shared activity between neural populations from the interacting brain areas. Our results on iEEG and fMRI data with MCPA support the hypothesis that individual-level exemplar information is not only en- coded by the population activity within certain brain populations, but also repre- sented through recurrent interactions between multiple distributed populations at the network level. Finally, to address the third consequence, we designed a two-stage generalized linear model to study the relationship between category tuning and the ongoing neu- ral activity in category selective cortical areas. We used this model to demonstrate that endogenous activity modulates the category selective tuning in the post-stimulus evoked response, and the same aspects of endogenous activity that modulate tuning also predict perceptual behavior. Taken together, in this thesis we develop and apply statistical methods to assess the properties of the non-classical structure in the ventral visual stream, and high- light contributions of regions to multiple stages of processing through interactive and distributed computation that is influenced by ongoing neural context. vi Acknowledgments I would like to thank my advisor, Avniel Ghuman, for all his guidance and sup- port over the past five years. This thesis would not be possible without him taking me into his lab five years ago, when I was struggling to find my way toward a Ph.D. in computational neuroscience. I was fortunate to join the lab as the first grad student, and to be involved the process of shaping and developing a research program in the exciting intersection between cognitive and computational neuroscience. From him I learned not only knowledge and experience in doing cognitive neuroscience, but also pure enthusiasm about science and nature. I still remember vividly the countless times of brief conversation at the end of a day turning into hours of deep discussions on sparking research ideas. I am always amazed by his insights and ability to identify the critical scientific questions behind the experiments and data. I would like to thank my co-advisor, Max G’Sell, for teaching me how to think in statistics and how to applied statistics in science. From him I learned to always think about the validity of my model assumptions, and always be cautious about the interpretations. Working with Avniel and Max together is the best thing a graduate student could ask for. They would challenge me with both the scientific merit and methodological rigorousness. I also got to learn how to communicate neuroscience to statistics and machine learning community, and how to talk about statistics and machine learning with the neuroscience community. I also would like to thank my committee members, Rob Kass and Chris Baker, for their help over the course of writing up this thesis. It is always a pleasure to interacting with Chris. Everyone in the PNC program owes a lot to Rob. Rob cared about my progress throughout my Ph.D. years and introduced me to Max. Every time we talked, Rob would always offer me his wise advice. I would like to thank my collaborators on the projects presented in this thesis: Mark Richardson, Julie Fiez, Michael Ward, Elizabeth Hirshorn, and Nicolas Brunet. None of these would be possible without their great intelligence and hard work. I want to thank Mark for always making time from his tight neurosurgeon’s schedule to keep track of the progress of our projects and to make insightful feedbacks. And I want to thank Mike for helping to collect and maintain the tremendous amount of experimental data and records over the years, and his artistic suggestions on many of the graphics. Many people in CMU and Pitt community have helped and influenced me in var- ious ways over the years. I want to thank Byron Yu for introducing me into the realm of computational neuroscience, and for being an inspiring role model to me. I want to thank many amazing professors who have taught me neuroscience, statistics and machine learning, and have given me numerous suggestions and feedbacks on my re- search: Marlene Behrmann, Michael Tarr, David Plaut, Carl Olson, Ryan Tibshirani, Larry Wasserman, Tom Mitchell, Aarti Singh, Barnabas´ Poczos´ and many others. I want to thank my current and former lab mates: Matt Boring, Brett Bankson, Arish Alreja, Shahir Mowlaei, Maxwell Wang, Zachary Jessen, Roma Konecky, Ellyanna Kessler, Shawn Walls and Laura Morett, and thank my brilliant fellow graduate stu- dents and postdocs, Shupeng Sun, Fa Wang, Pengcheng Zhou, Ying Yang, Josue Orellana, Mariya Toneva, Tong Liu, Elissa Aminoff, John Pyles, Witold Lipski and many others, for their help and support along the journey. I am also very grateful to the excellent staff members of the Center for the Neural Basis of Cognition and the Machine Learning Department, especially Melissa Stupka and Diane Stidle. I am very lucky to have a few dear long time friends to share all the joys and sorrows in our lives. I want to thank Shi Gu, Chen Zhang and Zhan Su for the years we stuck together through the ups and downs in our lives towards doctorate from different corners of the planet. I am also thankful for my friends in the Internet communities, 67373 and 710676. Many of them I may have never met in person, but they have inspired me and lit up my life in many different ways. Last but not least, I want to thank my family, especially my parents and grand- parents for their love and support. I want to thank my girlfriend, Xiaochen Zhang, for the years we lived through and for the years to come. viii Contents 1 Introduction 1 1.1 Background . .1 1.2 Overview of Thesis Contributions and Structure . .4 1.2.1 Information processing dynamics . .4 1.2.2 The representational structure of the neural interactions . .6 1.2.3 State-dependence of neural coding . .6 1.2.4 Methodological summary . .7 2 Temporal dynamics in human fusiform underlying word individuation 9 2.1 Introduction . .9 2.2 Methods . 11 2.2.1 Subjects . 11 2.2.2 Experimental paradigm . 11 2.2.3 Category localizer . 11 2.2.4 Electrical brain stimulation . 12 2.2.5 Covert Naming: Sensitivity to Bigram Frequency . 12 2.2.6 Word Individuation . 13 2.2.7 Data Preprocessing . 13 2.2.8 Electrode Selection . 13 2.2.9 Multivariate classification analysis . 14 2.2.10 Permutation test . 15 2.3 Results . 16 2.3.1 Verification of orthographic selectivity at lmFG electrode sites . 16 2.3.2 Disrupting lmFG Activity Impairs Both Lexical and Sublexical Ortho- graphic Processing . 18 2.3.3 Electrophysiological Evidence for a Visual Word Form Representation in the lmFG . 20 2.3.4 Temporal Dynamics of Word Individuation in lmFG . 21 2.4 Discussion . 22 2.5 Appendix: Supplement Methods and Results . 24 3 Temporal dynamics in human fusiform underlying face individuation 33 3.1 Introduction .