Social Saliency: Visual Psychophysics and Single-Neuron Recordings in Humans ! ! Thesis by Shuo Wang ! ! In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy ! ! ! ! ! ! CALIFORNIA INSTITUTE OF TECHNOLOGY Pasadena, California 2014 (Defended May 27, 2014) !ii ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! © 2014 Shuo Wang All Rights Reserved !iii ACKNOWLEDGEMENTS My life at Caltech started in 2008. I was fortunate to get a SURF fellowship and worked with Naotsugu Tsuchiya and Ralph Adolphs over the summer. I really enjoyed the research environment at Caltech and learnt a lot from Nao. It was a productive and determinative summer for me, after which I decided to return to Caltech. Again, I was fortunate to get a chance to work with Nao and Ralph as a research assistant until I started officially as a graduate student in 2010. I came to Ralph’s lab with a passion for science. Ralph gave me the anomalous freedom to decide what I would like to work on and to do the projects in my own way. Ralph let me explore all the fields of my interest while always offering me inspiring guidance when I was wandering around with puzzled mind. Freedom came from his trust in the student, while guidance resulted in the trust of the student in him. I have never been intimidated by any hassle or difficulty during my PhD, because he is always supportive as an advisor. Besides the academic advice, Ralph is also broad-minded and outgoing, and always ensures that our PhD study is not boring, with numerous lab parties, camping trips and movies. I cannot imagine such a great advisor anywhere else. This thesis would not have been written without the advice and support of Ralph. Nao was my mentor for the first few projects. I remember all the exciting moments when our experiments worked out. I learnt a lot of visual psychophysics and consciousness from Nao and was strongly influenced by his way of doing research. Without Nao’s help, I could not have published my first paper. I returned to Singapore and worked with Annett Schirmer in the summer of 2009 and have been collaborating with Annett since then. I learnt how to do EEG experiments and analysis, which is a very valuable experience. Christof Koch was my co-advisor. Nao introduced me to Christof to work on a psychophysics project of optic flow. I am very happy that this project eventually turned !iv out to be a paper and became a chapter of my thesis. I would like to thank Christof for all his support and advice. During my rotation, I am very grateful to have had an experience to work on monkeys with Doris Tsao, even though our idea of using fMRI to probe unconscious pressing of faces in face patches failed eventually after struggling for half a year. I thank Shay Ohayon for helping me to set up the experiment and Ty Basinger for helping me train the monkey very hard. I am also grateful to work with John O’Doherty and Mimi Liljeholm on human fMRI, which is a very valuable experience for me. I thank Mimi for teaching me the basics of analyzing fMRI data. I have been working with Ueli Rutishauser on single-unit recordings. I am very happy that our paper has finally been accepted to PNAS after a long struggle. I thank Ueli for teaching me all the single unit analysis and discussing with me all the interesting ideas. I am very impressed by his passion in science and his meticulousness. I thank Qi Zhao for teaching me the computational modeling of saliency and I have been collaborating with her on various saliency projects. I hope our current work will be published in the near future. I have been fortunate to be surrounded my colleagues and talented peers from the CNS program, especially my classmates Bo Chen, Nikki Sullivan, Soyoung Park, Yong-Jun Lin and Dan McNamee. We had many mentally stimulating discussions and I have very cheerful memories of hanging out with my classmates. I thank all of my colleagues in the Adolphs lab and Koch lab, past and present, for their generous help. I am grateful to my thesis committee: Ralph Adolphs, Christof Koch, John O’Doherty, Shinsuke Shimojo, Doris Tsao and Ueli Rutishauser. Finally, I would like to thank my parents for their generous support and encouragement. !v ABSTRACT My thesis studies how people pay attention to other people and the environment. How does the brain figure out what is important and what are the neural mechanisms underlying attention? What is special about salient social cues compared to salient non- social cues? In Chapter I, I review social cues that attract attention, with an emphasis on the neurobiology of these social cues. I also review neurological and psychiatric links: the relationship between saliency, the amygdala and autism. The first empirical chapter then begins by noting that people constantly move in the environment. In Chapter II, I study the spatial cues that attract attention during locomotion using a cued speeded discrimination task. I found that when the motion was expansive, attention was attracted towards the singular point of the optic flow (the focus of expansion, FOE) in a sustained fashion. The more ecologically valid the motion features became (e.g., temporal expansion of each object, spatial depth structure implied by distribution of the size of the objects), the stronger the attentional effects. However, compared to inanimate objects and cues, people preferentially attend to animals and faces, a process in which the amygdala is thought to play an important role. To directly compare social cues and non-social cues in the same experiment and investigate the neural structures processing social cues, in Chapter III, I employ a change detection task and test four rare patients with bilateral amygdala lesions. All four amygdala patients showed a normal pattern of reliably faster and more accurate detection of animate stimuli, suggesting that advantageous processing of social cues can be preserved even without the amygdala, a key structure of the “social brain”. People not only attend to faces, but also pay attention to others’ facial emotions and analyze faces in great detail. Humans have a dedicated system for processing faces and the amygdala has long been associated with a key role in recognizing facial emotions. In Chapter IV, I study the neural mechanisms of emotion perception and find that single neurons in the human amygdala are selective for subjective judgment of others’ emotions. Lastly, people typically pay special attention to faces and people, but !vi people with autism spectrum disorders (ASD) might not. To further study social attention and explore possible deficits of social attention in autism, in Chapter V, I employ a visual search task and show that people with ASD have reduced attention, especially social attention, to target-congruent objects in the search array. This deficit cannot be explained by low-level visual properties of the stimuli and is independent of the amygdala, but it is dependent on task demands. Overall, through visual psychophysics with concurrent eye- tracking, my thesis found and analyzed socially salient cues and compared social vs. non- social cues and healthy vs. clinical populations. Neural mechanisms underlying social saliency were elucidated through electrophysiology and lesion studies. I finally propose further research questions based on the findings in my thesis and introduce my follow-up studies and preliminary results beyond the scope of this thesis in the very last section, Future Directions. !vii TABLE OF CONTENTS Acknowledgements ...........................................................................................................iii Abstract ...............................................................................................................................v Table of Contents .............................................................................................................vii List of Figures ..................................................................................................................xii List of Tables ...................................................................................................................xiv Chapter I: General Introduction ......................................................................................1 1.1 Overview ........................................................................................................................1 1.2 Socially salient cues .......................................................................................................2 1.2.1 Eye gaze ......................................................................................................................3 1.2.2 Faces ...........................................................................................................................5 1.2.3 Head direction .............................................................................................................7 1.2.4 Finger gestures ............................................................................................................7 1.2.5 Postures, actions and biological motion ......................................................................8 1.2.6 Personal distance .........................................................................................................9 1.2.7 Social touch .................................................................................................................9 1.2.8 Social rewards ...........................................................................................................10