Neural construction of conscious perception Thesis by Janis Karan Hesse In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy CALIFORNIA INSTITUTE OF TECHNOLOGY Pasadena, California 2020 (Defended May 28th, 2020) ii 2020 Janis Hesse ORCID: 0000-0003-0405-8632 iii ACKNOWLEDGEMENTS I'd like to thank Doris Tsao for being an incredible advisor and undepletable source of ideas, advice and inspiration, for sharing her passion for science with me, and giving me the courage to ask big questions. I am very happy about the choice of my thesis committee. Markus Meister, Ueli Rutishauser, and Ralph Adolphs contributed substantially by providing useful ideas, discussions, and criticisms throughout this thesis. I want to thank current and past members of the Tsao lab, including Varun Wadia, Nicole Schweers, Audo Flores, Pinglei Bao, Liang She, Steven Le Chang, Xueqi Cheng Shay Ohayon, Tomo Sato, Joseph Wekselblatt, Francisco Luongo, Lu Liu, Anne Martin, Jessa Alexander, Erin Koch, Jialiang Lu, Yuelin Shi, Alex Farhang, Irene Caprara, Frank Lanfranchi, Lindsay Salay, Hongsun Guo, Abriana Sustaita, and Sebastian Moeller, who have all been very willing to offer me help whenever I needed, taught me the different techniques in the lab, and gave me great comments, ideas and discussions. I would also like to note that the work on human epilepsy patients described in Chapter VI is as much of Varun Wadia's work as it is mine. I am grateful to have started my PhD with such a lovely cohort. My PhD would not have been as fun without Mason McGill, Vineet Augustine, Gabriela Tavares, and Ryan Cho. I am very grateful for having met MACS, who strongly motivated me while writing this thesis. I want to thank Angela Hesse for gifting me this life and raising me in it in such a loving way. iv ABSTRACT Out of a myriad of sensory stimulations, our brain constructs a unified, self- consistent reality that we consciously experience. Little is known about how or where in the brain’s processing stream of physical input a conscious percept emerges into awareness. A remarkable property of conscious perception is that even though external input is often ambiguous, the perceptual interpretation of the world that our brain generates is consistent across multiple layers of representation, e.g., figure-ground segmentation and object identity. We thus set out to study how the interaction between different nodes in the brain generates and propagates new conscious percepts. Since the code of object identity is already well-understood, in particular for faces as reviewed in this thesis, we decided to get a handle on segmentation signals first. It turned out that consistent segmentation signals are hard to find, however, we found functionally defined modules in the brain that contained consistent cells from which figure-ground signals can be decoded. We next investigated whether face cells in object recognition areas actually encode the conscious percept of a face or are just passive filters of visual input. To distill conscious perception from other cognitive processes, such as decision making, introspection, and reporting of the percept, which often accompany new conscious percepts, we developed a no-report binocular rivalry paradigm that relies on an active fixation task rather than report, and therefore eliminates these confounding factors. We found that face patches in inferotemporal cortex indeed encode the conscious percept of a face. Using novel high-yield electrodes, we were able to decode what the animal was consciously perceiving at a given time. Preliminary and future experiments of population recordings from multiple nodes of the cortical hierarchy simultaneously promise to go beyond correlates of consciousness and reveal the mechanisms of how and where conscious percepts are constructed. v PUBLISHED CONTENT AND CONTRIBUTIONS Hesse, Janis K., and Tsao, Doris Y. (2016). "Consistency of border-ownership cells across artificial stimuli, natural stimuli, and stimuli with ambiguous contours." In: Journal of Neuroscience 36.44 (2016): 11338-11349. DOI:10.1523/JNEUROSCI.1857-16.2016 J.K.H. conducted the experiments and analyzed the data. J.K.H. and D.Y.T. designed the experiments, interpreted the data, and wrote the paper. Hesse, Janis K. and Tsao, Doris Y. (2020). " Representation of conscious percept without report in the macaque face patch network". In : bioRxiv. DOI: 10.1101/2020.04.22.047522. Submitted to eLife. J.K.H. conducted the experiments and analyzed the data. J.K.H. and D.Y.T. designed the experiments, interpreted the data, and wrote the paper. Hesse, Janis K. and Tsao, Doris Y. (2020). " The macaque face patch system: a turtle’s underbelly for the brain". Submitted to: Nature Reviews Neuroscience. J.K.H. and D.Y.T. wrote the paper. vi TABLE OF CONTENTS Acknowledgements………………………………………………………... iii Abstract ……………………………………………………………...………iv Published Content and Contributions…………………………….….........v Table of Contents………………………………………………….………. vi List of Illustrations and/or Tables………………………………….………vii Nomenclature………………………………………………………………..ix Chapter I: Introduction ............................................................................ 1 Motivation .......................................................................................... 1 Bistable phenomena ......................................................................... 3 Outline ............................................................................................... 5 Chapter II: The macaque face patch system .......................................... 8 Introduction ........................................................................................ 8 Face cell discovery ............................................................................ 9 Anatomical organization .................................................................. 14 Coding principles ............................................................................. 20 Causal role in behavior ................................................................... 35 Interaction with other brain areas .................................................... 40 Conclusion and boxes ..................................................................... 46 Chapter III: Figure-ground segmentation .............................................. 55 Consistency of border-ownership cells ........................................... 55 Segmentation patches .................................................................... 87 Chapter IV: Mooney faces .................................................................... 95 Chapter V: A philosophical treatise on consciousness....................... 103 Definitions of consciousness ......................................................... 103 Review of previous paradigms for specific consciousness ........... 106 Confounding factors ...................................................................... 109 No-report paradigms ..................................................................... 113 Conclusions ................................................................................... 116 Chapter VI: Binocular rivalry ............................................................... 117 Representation of conscious percept in macaque face patches .. 117 Recordings in human epilepsy patients ........................................ 146 Chapter VII: Future directions ............................................................. 151 Are conscious switches feedforward or feedback? ....................... 151 Future experiments on mechanisms of consciousness ................ 156 Summary of thesis main results .................................................... 158 Bibliography ........................................................................................ 159 vii LIST OF ILLUSTRATIONS AND/OR TABLES Number Page 1. Figure I-1: Frontal-profile illusion ................................................. 2 2. Figure I-2: Bistable phenomena .................................................. 5 3. Figure II-1. Face-selective patches in macaque cortex ............. 13 4. Figure II-2. Anatomical connectivity of face patches ................. 18 5. Figure II-3. Probing feature selectivity of face cells ................... 21 6. Figure II-4. Increasing view invariance across the face patch hierarchy .................................................................................... 26 7. Figure II-5. The neural code for facial identity ........................... 29 8. Figure II-6. The neural code for object identity in general ......... 32 9. Figure II-B1. The Thatcher illusion ............................................ 48 10. Figure II-B2. Face patches across different primate species .... 50 11. Figure III-1. Segmentation can be ambiguous based on low-level cues ........................................................................................... 58 12. Figure III-2. An example border-ownership cell ......................... 59 13. Table III-1. Consistency of cells for border-ownership stimuli ... 62 14. Figure III-3. Comparison of population analysis to Qiu and Von Der Heydt (2007). ................................................. 65 15. Figure III-4. Quantification of response consistency across artificial and natural stimuli ........................................................ 67 16. Figure III-5. Responses of border-ownership cells to simple and natural stimuli ............................................................................