RICE UNIVERSITY Integration of Sight, Hearing and Touch in Human Cerebral Cortex by Nafi Yaşar A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE Doctor of Philosophy APPROVED, THESIS COMMITTEE: __________________________________ Michael S. Beauchamp, Assistant Professor, Director Department of Neurobiology and Anatomy University of Texas Health Science Center at Houston __________________________________ Tomasz Tkaczyk, Assistant Professor, Chair Department of Bioengineering Rice University __________________________________ Robert M. Raphael, Associate Professor Department of Bioengineering Rice University __________________________________ Tatiana T. Schnur, Assistant Professor Department of Psychology Rice University HOUSTON, TEXAS APRIL 2009 Abstract Integration of Sight, Hearing and Touch in Human Cerebral Cortex by Nafi Yaşar While each individual sensory modality provides us with information about a specific aspect about our environment, our senses must be integrated for us to interact with the environment in a meaningful way. My thesis describes studies of the interactions between somatosensation, vision and audition using functional Magnetic Resonance Imaging (fMRI) of normal human subjects as the primary method. In order to study somatosensation with fMRI we first built an MRI-compatible tactile-stimulation apparatus. This apparatus was then used for four separate studies. In the first study, we investigated tactile responses in lateral occipital lobe, a brain region traditionally considered "visual" cortex. We found that visual area MST, but not visual area MT, responded to tactile stimulation. In the second study we investigated a possible homologue to a macaque multisensory area that integrates visual, auditory and tactile information, called the Superior Temporal Polysensory area (STP). We found responses to tactile stimuli co-localized with auditory and visual responses in posterior superior temporal sulcus. This is likely to be a human homologue to macaque STP. In the third study, we used Multi Voxel Pattern Analysis (MVPA) to demonstrate that this homologue of macaque STP (along with traditional "somatosensory" areas) can predict the location of tactile stimulation from fMRI data. In the fourth study we used psychophysical techniques to analyze the effects of auditory stimuli on tactile perception. We found that auditory stimuli can influence detection, frequency perception, and the perception of the spatial location of vibrotactile stimuli. Two additional projects are also briefly described. The results of an effort to develop an MRI compatible Transcranial Magnetic Stimulation (TMS) device are included. Also a project I worked on during my summer internship in which I debugged a system capable of both stimulating and recording from cortical tissue at the same time is also discussed. Table of Contents Abstract ...................................................................................................................................... ii Table of Contents ....................................................................................................................... iv Preface ....................................................................................................................................... v Chapter 1: Providing Computer Controlled Tactile Stimulation in an MRI Environment ............... 1 Chapter 2: Human MST But Not MT Responds to Tactile Stimulation ........................................ 17 Chapter 3: Touch, Sound and Vision in Human Superior Temporal Sulcus.................................. 42 Chapter 4: Distributed Representation of Single Touches in Somatosensory and Visual Cortex . 68 Chapter 5: Sound Enhances Touch Perception .......................................................................... 88 Chapter 6: Providing Transcranial Magnetic Stimulation (TMS) in an MRI Environment ........... 112 Chapter 7: Industrial Internship: Simultaneous Intracranial or Intracortical Recording and Stimulation ............................................................................................................................. 127 Summary ................................................................................................................................ 146 References .............................................................................................................................. 149 Preface Nearly every minute of our waking lives is dedicated to interpreting and interacting with our surrounding environment. Individually our senses give us information about specific aspects of our environment; our sense of vision inform us of the intensities and frequencies of light that is being reflected off surrounding objects, our senses of touch and hearing alert us to displacements or vibrations in the matter around us caused by movement, and our senses of smell and taste give us information about the chemical composition of what we inhale or imbibe. Individually each sense provides a very limited view of our surrounding, however by incorporating all of our senses together we are provided with a robust model of our immediate environment, allowing us to engage in complex interactions. Primary cortical areas have been identified for all five senses; however where and how the senses are integrated largely remains a mystery. In the traditional model of the brain all cortical areas communicate in a feed-forward manner, with lower cortical areas passing information to higher areas, and all areas ultimately terminating in an undiscovered cortical area where they are integrated to form the human mind. More recent research suggests that the brain operates in a more parallel fashion, with cortical areas having many lateral connections and higher cortical areas frequently passing information back to lower areas. This “parallel processing” is especially evident in sensory cortex, where many cortical areas previously thought to be dedicated to a single sensory modality have been shown to activate to two or more different modalities (Ghazanfar and Schroeder 2006; Grefkes, et al. 2001), and projections vi between cortical areas of differing modalities have been mapped. Therefore if the mechanics of multisensory integration can be elucidated it will have a great impact on our understanding of the brain as a whole. Since more than half of the cortex is devoted to primary and associative sensory areas it is highly desirable to be able to monitor activity in the entire cerebrum while investigating multimodal interactions. Additionally, since our research is focused on human cortex, invasive techniques that require direct access to the brain or injection of radionucliotides or contrasting agents are either impractical or impossible with ethical and legal constraints. For this reason we used BOLD fMRI as our principle tool since it is safe and noninvasive, and allows us to monitor activity of the entire brain with the level of resolution that is necessary to resolve different functional brain areas. For those unfamiliar with the technology, BOLD fMRI (Blood-Oxygenation Level Dependant functional Magnetic Resonant Imaging) relies on the hemodynamic response, a biological response to neural activity observed over a hundred years ago (Roy and Sherrington 1890). When a signal is passed from one neuron to another, the postsynaptic cell releases nitric oxide (NO) into the synaptic cleft. The NO diffuses into the surrounding tissue and causes the smooth muscle in any blood vessels it encounters to relax, increasing blood flow to that area. The greater the number of neurons activated, or the greater the degree of individual activation, the greater the amount of NO release and the greater the hemodynamic response. Therefore measuring the hemodynamic response provides a strong correlate to the level of neural activity. vii In 1936 Pauling and Coryell discovered that hemoglobin is paramagnetic by itself, but loses its magnetic moment when bound to oxygen (Pauling and Coryell 1936). Many years later Ogawa would realize that since an MRI signal depends on magnetic alignment of protons, deoxygenated hemoglobin’s paramagnetic qualities would interfere with the MRI signal where it was abundant (Ogawa and Lee 1990). He realized it would therefore be possible to use MRI to measure the hemodynamic response by taking a series of MRI images and looking for voxels to become brighter as fresh blood rushed in and displaced oxygen-poor blood (Ogawa, et al. 1990). His theory has been confirmed by a large amount of data showing a positive correlation between the amplitude of somatosensory evoked potentials and fMRI BOLD signal (Arthurs, et al. 2000; Backes, et al. 2000; Heeger and Ress 2002; Ogawa, et al. 1998) During an fMRI experiment, the subject is first given an anatomical scan providing a single 3D image at very high resolution. The subject is then provided stimulus, or asked to perform a task, or both depending on the experiment. During this time the subject is continuously scanned forming a series of low-resolution 3D images. The amount of time between each image is called the repetition time (TR), and is limited by how quickly a scanner can produce an image at the desired resolution (generally around 2 seconds for our purposes). These images are then overlaid on top of the anatomical image and allow activation to be measured by looking for changes in signal intensity in each voxel over time. These changes in signal intensity can then be correlated to the