Computation and Psychophysics of Sensorimotor Integration by Zoubin Ghahramani BSE Computer Science University of Pennsylvania BA Cognitive Science University of Pennsylvania Submitted to the Department of Brain and Cognitive Sciences in partial fulllment of the requirements for the degree of Do ctor of Philosophy at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY Septemb er c Massachusetts Institute of Technology All rights reserved Author Department of Brain and Cognitive Sciences July Certied by Michael I Jordan Professor Thesis Sup ervisor Certied by Tomaso Poggio Uncas and Helen Whitaker Professor Thesis CoSup ervisor Accepted by Emilio Bizzi Eugene McDermott Professor Chairman Department of Brain and Cognitive Sciences Computation and Psychophysics of Sensorimotor Integration by Zoubin Ghahramani Submitted to the Department of Brain and Cognitive Sciences on July in partial fulllment of the requirements for the degree of Do ctor of Philosophy Abstract All higher organisms are able to integrate information from multiple sensory mo dalities and use this information to select and guide movements In order to do this the central nervous system CNS must solve two problems Converting information from distinct sensory representations into a common co ordinate system and integrating this information in a sensible way This dissertation prop oses a computational framework based on statistics and information theory to study these two problems The framework suggests explicit mo dels for b oth the co ordinate transformation and integration problems which are tested through human psychophysics The exp eriments in Chapter suggest that Spatial information from the visual and auditory systems is integrated so as to minimize the variance in lo calization When the relation b etween visual and auditory space is articially remapp ed the spatial pattern of auditory adaptation can b e predicted from its lo calization variance These studies suggest that multisensory integration and intersensory adaptation are closely related through the principle of minimizing lo calization variance This principle is used to mo del sensorimotor integration of proprio ceptive and motor signals during arm movements Chapter The temp oral propagation of errors in estimating the hands state is captured by the mo del providing supp ort for the existence of an internal mo del in the CNS that simulates the dynamic b ehavior of the arm The co ordinate transformation problem is examined in the visuomotor system which mediates reaching to visuallyp erceived ob jects Chapter The pattern of changes in duced by a lo cal remapping of this transformation suggests a representation based on units with large functional receptive elds Finally the problem of converting information from disparate sensory representations into a common co ordinate system is addressed computa tionally Chapter An unsup ervised learning algorithm is prop osed based on the prin ciple of maximizing mutual information b etween two top ographic maps What results is an algorithm which develops multiple mutuallyaligned top ographic maps based purely on correlations b etween the inputs to the dierent sensory mo dalities Thesis Sup ervisor Michael I Jordan Title Professor Thesis CoSup ervisor Tomaso Poggio Title Uncas and Helen Whitaker Professor Acknowledgments I thank Daniel Wolp ert for reasons to o numerous to mention here Without his men torship encouragement patience and creativity there would b e no psychophysics in this thesis He made hard work enjoyable and exciting and I doubt I will nd such a combination of friend and collab orator in the future I thank Michael Jordan for providing an unparalleled training environment I learned more in his grueling threehour lab meetings than in the rest of my graduate coursework combined Without his mentorship there would b e no computation in this thesis I thank Dick Held Emilio Bizzi and Tomaso Poggio for kindly serving as memb ers of my thesis committee Peter Dayan provided helpful comments on the manuscript and I b enetted from engaging discussions with Geo Hinton who also made it p ossible for me to write this thesis without worrying ab out searching for a p ostdo ctoral p osition All the memb ers of Jordan lab provided an excellent environment for research I esp ecially thank Flip Sab es for critical comments on the manuscript and friendly moral supp ort during the last weeks of writing Lawrence Saul for tutorials on sta tistical mechanics and Tommi Jaakkola for letting me measure his head Carlotta Domeniconi provided many hours of assistance in conducting exp eriments Adee Matan who earned Jordan lab memb ership by always using our computers deserves a sp ecial thanks for keeping track of my progress and mental sanity I thank David Po epp el for egging me on to defend early and for b eing just as stressed as I was during our last few days John Houde was my dietary advisor during the thesis and travel companion throughout graduate scho ol I thank James Thomas for distracting me late at night with horror stories and go o d music while I was trying to write my dissertation Thanks to Gregg Solomon for advice on writing Acknowledgment sections I thank Jan Ellertsen for guiding me through the tortuous road of academic re quirements I am grateful to Marney Smyth for dedicating many hours to helping me with slides and gures Ellie Bonsaint provided sup erb administrative supp ort throughout my graduate education and Pat Claey tracked down many obscure ar ticles for me I have enjoyed graduate scho ol immensely mostly due to the wonderful environ ment provided by the students in the program I thank each of them for their indi vidual gift in making this department unique I am grateful to the McDonnellPew Foundation for supp orting my studies in this department I esp ecially want to thank Azita Ghahramani for letting me crash at her place for years while doing a PhD and for b eing the most wonderful sister and housemate and Monica Biagioli for moral supp ort and making my last few months here very sp ecial Biographical note Although my family is originally from Shiraz Iran I was b orn in Moscow on February th After four years in Russia my family moved back to Iran for one year and then to Madrid Spain I lived in Spain and attended the American Scho ol of Madrid from until my high scho ol graduation in I then went to Philadelphia to study at the University of Pennsylvania where I obtained a BA in Cognitive Science and a BSE in Computer Science In I entered the do ctoral program in Brain and Cognitive Sciences at MIT To my father for al l the joy he brought me Contents Intro duction Outline of the Thesis I Integration Integration and Adaptation of Visual and Auditory Maps Intro duction Background Psychophysics Neuroscience The Computational Mo del Integration Adaptation Related Mo dels Summary Overview of the Exp eriments Exp eriment Lo calization of Visual Auditory and Visuoauditory Stimuli Metho d Results Discussion Exp eriment Adaptation to a VisuoAuditory Remapping CONTENTS Metho d Results Discussion Exp eriment Adaptation to VisuoAuditory Variance Metho d Results Discussion Exp eriment Generalization of the VisuoAuditory Map Metho d Results Discussion Controls Alternative Cues to Auditory Stimulus Lo cation Pointing with the Left Hand Discussion Empirical ndings Implications Directions for future work Conclusion An Internal Mo del for Sensorimotor Integration Intro duction Exp eriment Propagation of Errors in Sensorimotor Integration App endix A Paradigm App endix B Simulation II Co ordinate Transformations Representation of the Visuomotor Co ordinate Transformation CONTENTS Intro duction The Visuomotor Co ordinate Transformation Spatial Generalization Contextual Generalization Exp erimental Aims and Overview Exp eriment Visuomotor Generalization to a OnePoint Displacement Metho d Results Discussion Exp eriment Visuomotor Generalization to a TwoPoint Displacement Metho d Results Discussion Exp eriment Contextual Generalization of the Visuomotor Map A Metho d Results Exp eriment Contextual