INVESTIGATING THE NEURAL CORRELATES OF CROSSMODAL FACILITATION AS A RESULT OF ATTENTIONAL CUEING: AN EVENT-RELATED fMRI STUDY

by

Zainab Fatima

A thesis submitted in conformity with the requirements for the degree of Masters of Science Institute of Medical Science University of Toronto

© Copyright by Zainab Fatima, 2008

Investigating the Neural Correlates of Crossmodal Facilitation as a Result of Attentional Cueing: an Event-Related fMRI Study

Zainab Fatima

Masters of Science

Institute of Medical Science University of Toronto

2008 Abstract

Attentional cueing modulated neural processes differently depending on input modality. I used event-related fMRI to investigate how auditory and visual cues affected reaction times to auditory and visual targets. Behavioural results showed that responses were faster when: cues appeared first compared to targets and cues were auditory versus visual. The first result was supported by an increase in BOLD percent signal change in sensory cortices upon cue but not target presentation. Task-related activation patterns showed that the auditory cue activated auditory and visual cortices while the visual cue activated the visual cortices and the fronto-polar cortex. Next, I computed brain-behaviour correlations for both cue types which revealed that the auditory cue recruited medial visual areas and a fronto-parietal attentional network to mediate behaviour while the visual cue engaged a posterior network composed of lateral visual areas and subcortical structures. The results suggest that crossmodal facilitation occurs via independent neural pathways depending on cue modality.

ii

Acknowledgments

I am still in awe of the fact that I have now officially completed my Masters degree. This is partly because writing the thesis appeared to be such an arduous task at the beginning. Nevertheless, with time and patience, the page numbers increased and the quality of the writing improved. In retrospect, I realize that the one person who constantly supported me and pushed me to produce the best possible work is my supervisor – Dr. Anthony Randal McIntosh. His relentless critiques of my thesis and constant insistence on getting tasks completed have brought me to this pinnacle in my life. I am deeply grateful to him. I admire his strength of character and his ability to always be innovative in the light of academic adversity. He regards criticisms of his work as challenges and is truly an inspiration when it comes to scientific knowledge. If my scientific career is an inkling of what Randy’s is, I would be quite satisfied with my progress. So, thank you Randy for being the person that you are and for keeping me motivated throughout this whole process.

I would like to thank past and present members of the McIntosh lab who have provided valuable feedback in various forms during the preparation of my graduate work. These people include Maria Tassopoulos, Jordan Poppenk, Grigori Yourganov, Tanya Brown, Roxane Itier, Vasily Vakorin, Diana Khrapatch, Anjali Raja, Antonio Vallesi, Wilkin Chau, Michele Korostil, Signe Bray, Jeremy Caplan and Mackenzie Glaholt. I would like to especially thank Natasa Kovacevic – for her insights with regards to my experiment, Andreea Diaconescu and Bratislav Misic – for allowing me to vent and fret at any given time during my writing episodes, Andrea Protzner – for constantly keeping me caffeinated, Sandra Moses – for her gentle way of conveying criticisms, and Hana Burian – for the sushi runs and bear hugs.

I would like to thank my committee members – Drs. Adam Anderson and Claude Alain for their support and for providing me with thesis-related feedback so promptly. I would also like to thank Karen Davis, graduate coordinator at the Institute of Medical Science (IMS), for allowing me to defend my thesis within such stringent time constraints.

Last but not least, I would like to thank my parents and brother for their faith in my ability to accomplish any goal that I have set for myself and for taking care of me. I would like to thank my husband and best friend, Ali - without his foot-rubs, back massages and constant

iii coaxing – I would be far from completing any graduate work. And finally, I would like to thank my baby girl who has so patiently stayed in my tummy till I have completed my academic responsibilities. This is one journey we’ve already shared and I can’t wait to meet you, my darling.

iv

Table of Contents

Abstract...... ii

Acknowledgments...... iii

Table of Contents...... v

List of Tables ...... viii

List of Figures...... ix

List of Appendices ...... x

List of Abbreviations ...... xi

Chapter 1: Literature Review...... 1

1.1 Overview...... 1

1.2 Classifying Different Attentional Mechanisms ...... 1

1.2.1 Orienting ...... 1

1.2.2 Endogenous vs. Exogenous Shifts ...... 2

1.3 Influence of Crossmodal Stimuli on Attentional Mechanisms...... 4

1.4 Crossmodal Asymmetry Reported In Cognitive, Physiological, and Developmental Studies...... 7

1.4.1 Auditory to Visual (A-V) Interactions...... 8

1.4.2 Visual to Auditory (V-A) Interactions...... 10

1.5 General Anatomy of Central Auditory and Visual Pathways...... 11

1.5.1 The Auditory Pathway ...... 11

1.5.2 The Visual Pathway ...... 13

1.6 From Anatomy to Function - A Dynamic Systems’ Perspective...... 14

1.7 An Overview of fMRI...... 16

1.7.1 Basic MRI Physics...... 16

1.7.2 Physiological Basis of BOLD fMRI...... 18

1.7.3 Coupling of Neuronal Activity & BOLD ...... 19 v

Chapter 2: Aims and Hypotheses...... 22

Chapter 3: Attentional Cueing Modulates Multisensory Interactions in Human Sensory Cortices ...... 24

3.1 Introduction...... 24

3.2 Materials and Methods...... 26

3.2.1 Participants...... 26

3.2.2 Stimuli...... 27

3.2.3 Apparatus ...... 27

3.2.4 Procedure ...... 28

3.2.4.1 Trial Structure...... 28

3.2.4.2 Task Types...... 28

3.2.4.3 fMRI Session ...... 29

3.2.5 fMRI Scanning Parameters...... 30

3.2.6 Data Analysis...... 30

3.2.6.1 Pre-processing Pipeline...... 30

3.2.6.2 Statistical Analysis...... 32

3.3 Results...... 34

3.3.1 Behavioural Performance...... 34

3.3.2 fMRI Results...... 35

3.4 Discussion...... 36

Tables...... 40

Figures...... 46

Chapter 4: The Interplay of Cue Modality and Response Latency in Neural Networks Supporting Crossmodal Facilitation...... 60

4.1 Introduction...... 60

4.2 Methods...... 62

4.1.1 Data Analysis...... 62 vi

4.3 Results...... 63

4.1.2 Behavioural Performance...... 63

4.1.3 fMRI Results...... 63

4.4 Discussion...... 64

Tables...... 68

Figures...... 75

Chapter 5: General Discussion...... 87

5.1 A Convergent Model of Audio-Visual Interactions...... 90

5.2 Dynamic Processing in Sensory-specific Cortices ...... 91

5.3 Limitations ...... 92

5.4 Future Directions ...... 93

References...... 95

Appendices...... 115

vii

List of Tables

Table 3.1: Mean Reaction Times by Condition...... 40

Table 3.2: Local Maxima from AC-VT: AT-VC Task ST-PLS...... 41

Table 3.3: Local Maxima from VC-AT: VT-AC Task ST-PLS...... 43

Table 4.1: Local Maxima from AC-VT: AT-VC Behavioural ST-PLS...... 68

Table 4.2: Local Maxima from VC-AT: VT-AC Behavioural ST-PLS...... 71

viii

List of Figures

Figure 3.1. Experimental design schematic for auditory cue-visual target tasks...... 46

Figure 3.2. Experimental design schematic for visual cue-auditory target tasks...... 48

Figure 3.3. Behaviour measures for all four experimental tasks...... 50

Figure 3.4. BOLD HRFs for cue and target – auditory modality...... 52

Figure 3.5. Singular image and design scores differentiating cue from target for auditory modality ...... 54

Figure 3.6. BOLD HRFs for cue and target – visual modality...... 56

Figure 3.7. Singular image and design scores differentiating cue from target for visual modality...... 58

Figure 4.1. Correlation profiles for the auditory cue...... 75

Figure 4.2. Brain scores plotted by participants for the auditory cue...... 77

Figure 4.3. Singular images of brain areas that facilitate reaction time for the auditory cue...... 79

Figure 4.4. Correlation profiles for the visual cue...... 81

Figure 4.5. Brain scores plotted by participants for the visual cue...... 83

Figure 4.6. Singular images of brain regions that facilitate reaction time for the visual cue...... 85

ix

List of Appendices

Appendix A. fMRI Screening Form...... 115

Appendix B. MRI Screening Form...... 119

Appendix C: Information and Consent Form...... 120

x

List of Abbreviations

Abbreviation Region Cu Cl claustrum Ga angular GC cingulate gyrus GF GFd GFi GFm GFs Gh GL GOi inferior occipital gyrus GOm middle occipital gyrus GOs superior occipital gyrus GPoC (sensory cortex) GPrC precentral gyrus () GTi GTm GTs Gsm INS insula LPc LPi inferior LPs superior parietal lobe Pcu Th thalamus

Note: Definition of the region abbreviations by reference to Talairach and Tournoux (1988). Other abbreviations used in tables include L and R which stand for left and right respectively. Ant refers to anterior and Post refers to posterior.

xi 1

Chapter 1: Literature Review

1.1 Overview

The past few decades have marked a dramatic change in the realm of cognitive neuroscience. Neuroimaging techniques such as functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) have provided scientists with tools to examine brain function across spatial and temporal domains. Prior to the use of functional neuroimaging, attentional theories about the brain were based primarily on animal work or observations of deficits in patients with brain damage or disease. These theories of attention are now informed both by animal work and human neuroimaging studies; researchers are actively trying to bridge the gap between brain function and attentional models.

This section will begin with an overview of some common attentional mechanisms and how these processes can form the basis of behavioural crossmodal facilitation. Crossmodal facilitation occurs when a cue in one sensory modality, for example auditory, speeds responses to a target in a different sensory modality, such as vision. The motivation for the current experiment is to study the effects of crossmodal facilitation, in audition and vision, using the fMRI technique.

A survey of the behavioural literature on crossmodal facilitation will be followed by an outline of the neuroanatomy that may support audio-visual processes in the brain. The implications of neuroanatomical studies on the organization of brain function will then be described continued by a brief review of the fMRI technique.

1.2 Classifying Different Attentional Mechanisms 1.2.1 Orienting

Experiments on selective attention have shown that when participants are provided with a cue for the location of an upcoming event, they direct their attention to the event (Posner, 1980; Posner, Inhoff, Friedrich, & Cohen, 1987). Posner developed an attentional cueing paradigm to study such visual shifts of attention. The paradigm consisted of a central cross and two peripheral boxes. The central cross was replaced by a plus sign or an arrow that pointed in the direction of one of the boxes. The plus sign indicated that there was an equiprobable chance of a target

2

appearing in either of the two boxes while the direction of the arrow correctly identified the location of the target in eighty-percent of the trials (valid trials) and did not cue correct target location in twenty-percent of the trials (invalid trials). Responses were faster to valid trials compared to invalid trials. Posner (1980) postulated that facilitated responses in valid conditions occurred as a result of attentional engagement at the target location.

Subsequently, Posner and Petersen (1990) proposed a model of selective attention based on neuropsychological studies where three loci in the brain – the parietal lobe, the superior colliculus and the thalamus - seemed to contribute to different aspects of attentional shifts. Individuals with unilateral parietal lobe damage showed normal responses when cues and targets were presented on the side that was contra-lateral to their lesion but their performance was impaired when cues appeared on the ipsa-lesional side and the target appeared on the contra- lesional side (Posner, Walker, Friedrich, & Rafal, 1984). Patients that had degenerate superior colliculi, structures involved in eye movement coordination, such as those suffering from supranucluear palsy were slow to move their attention from one location to the next (Rafal et al., 1988). Lastly, patients with thalamic damage responded poorly at spatial locations that were contra-lateral to their lesion irrespective of whether trials were valid or invalid (Rafal & Posner, 1987). In summarizing the neuropsychological work that supported the selective attention model, it can be stated that the parietal lobe was critical in the disengagement of attention, the superior colliculus was implicated in moving attention from one point in space to another and the thalamus was essential for re-engaging attention at particular location.

1.2.2 Endogenous vs. Exogenous Shifts

The model of attentional cueing postulated by Posner was based on experiments in which cues were centrally presented (at fixation) and targets occurred in the periphery (either side of fixation). Experimenters have subsequently claimed that the location of cue presentation, central or peripheral, can tap into two different attentional mechanisms – endogenous and exogenous (see Ruz & Lupianez, 2002 for a review). Endogenous shifts of attention occur following an instruction to attend to a target location where participants have to use their own volition to orient attention. In contrast, exogenous shifts of attention are more reflexive in that participants can automatically orient to the stimulus that is presented at a target location (see Gazzaniga, Ivry, & Mangun, 2002 for review). When cues are presented centrally, participants have to move their

3

attention according to the instructional content of the cue (Ruz & Lupianez, 2002). On the other hand, peripheral cues are able to capture attention exogenously at a particular location without any instructional content (Ruz & Lupianez, 2002). Therefore, informative central and peripheral cues have been used to study endogenous shifts of attention while spatially non-predictive peripheral cues have been used to study exogenous shifts of attention (Chica, Sanabria, Lupianez, & Spence, 2007).

Researchers that have attempted to separate the effects of endogenous and exogenous shifts of attention have found that endogenous cueing requires a longer delay interval between cue and target presentation (Eimer, 2000; Muller and Findley, 1988). These studies imply that different attentional mechanisms may be mediating cue processing. Corbetta and Shulman (2002) have claimed that endogenous versus exogenous shifts of attention may invoke different behavioural patterns and neural activations. In a meta-analysis of studies conducted on shifts of attention, Corbetta and Shulman (2002) ascribed voluntary (endogenous) attention to a network of areas that include the dorsal posterior parietal and frontal cortex with transient activity in occipital areas. The functional attributes of this fronto-parietal network include integration of prior experience with expectations and goals to result in volitional shifts in attention. The temporo-parietal cortex and ventral frontal regions comprise a separate network that has been implicated in reflexive (exogenous) attentional shifts. This reflexive network is thought to focus attention on salient events in the environment.

A similar but less specific distinction in attentional processes was suggested earlier by Posner (1992) and Posner and Petersen (1990). The authors claimed that an anterior attentional network consisting of frontal areas and the anterior cingulate was responsible for extracting the relevance of a selected item while a posterior attentional network involving parietal areas selected information based on sensory attributes. Posner argued that this anterior-posterior attention system functioned to coordinate activity at a supramodal (modality-independent) level and was able to regulate data processing that was specific to particular cognitive tasks.

While Corbetta and Shulman (2002) and to some extent, Posner (1992) have made an attempt to categorize exogenous and endogenous attentional shifts into distinct processing streams, a study by Rosen and colleagues (1999) suggested a fair bit of overlap in brain activations in response to voluntary and reflexive attention. In both endogenous and exogenous

4

cueing tasks, activations were seen in the dorsal premotor region, the and the superior parietal cortex. Evidence to date remains inconclusive about the exact nature of interactions between endogenous and exogenous shifts of attention at the neural level.

The mechanisms of attention discussed in this section are not directly explored in the current experiment but provide the context for understanding experiments on crossmodal cueing outlined below.

1.3 Influence of Crossmodal Stimuli on Attentional Mechanisms

The research mentioned thus far has focused on visual spatial orienting while experiments conducted in the 1960s also found attentional modulations of response times to cross-modal (auditory, visual) stimuli (Bertelson & Tisseyre, 1969; Davis & Green, 1969). In one experiment, Bertelson and Tisseyre found that an auditory click decreased reaction time to a subsequent visual flash while a visual flash did not have the same effect on an auditory click. A more specific investigation into crossmodal cueing was conducted by Buchtel & Butter (1988) who used lateralized auditory and visual stimuli. These stimuli were spatially significant (spatially neutral conditions were used as controls) in comparison to Bertelson & Tisseyre’s non- spatial stimuli (centered flash, binaural click: neutral spatial significance), Two cross-modal cueing cases were devised, auditory to visual (A-V) and visual to auditory (V-A), as well as the intra-modal cases, visual to visual (V-V) and auditory to auditory (A-A). Previous findings were reinforced with respect to reaction time: cross-modal cueing led to faster responses compared to intra-modal cueing, and audio cues were more effective in facilitating responses compared to visual cues. In the cross-modal conditions, visual target stimuli seemed much easier to cue than audio target stimuli, regardless of the cue modality. In fact, Buchtel and Butter reported virtually no cueing effect for auditory target stimuli.

Farah, Wong, Monheit, and Morrow (1989) performed a cross-modal cueing study on patients with lesions circumscribed unilaterally to the parietal lobe to try to understand the nature of crossmodal cueing effects in the brain. Prior to this study, Posner and colleagues (1984) had shown that patients with unilateral parietal lobe lesions had difficulty disengaging attention in a standard visual spatial cueing paradigm. Farah and colleagues (1989) extrapolated on Posner’s work and examined A-V and V-V conditions. In the V-V condition there was a 50ms reduction in reaction time when the target appeared on the ipsa-lesional side of space, and a considerable

5

increase in reaction times for targets that appeared on the contra-lesional side of space. In the A- V condition, response times were faster than in the visual cue condition. Again, there was a large increase in reaction time for targets that appeared contra-lateral to the parietal lesion. This study was in agreement with its predecessors in a general way: cross-modal A-V cueing produced facilitated reaction times.

Contrary to the observation of these past studies was a paper by Ward published in 1994. Ward (1994) used a crossmodal spatial discrimination task where participants made speeded left- right responses to visual or auditory targets following the presentation of an auditory or a visual non-predictive cue, both auditory and visual cues, or no cues. Reaction times were measured for all conditions at different inter-stimulus intervals between the cue and target. The results indicated that visual cues facilitated reaction times to auditory targets that were presented on the same side of the cue (compatible) at short inter-stimulus intervals. Auditory cues, in contrast, did not facilitate reaction times to visual targets shown on either side of cue presentation or at any inter-stimulus interval. Auditory cues did facilitate reaction times to compatible auditory targets at short inter-stimulus intervals. Ward’s findings were directly opposite to those found by previous crossmodal studies (Bertelson & Tisseyre, 1969; Buchtel & Butter, 1988; Farah, Wong, Monheit, & Morrow, 1988).

In a subsequent study by Spence and Driver (1997), an orthogonal spatial cueing paradigm (Spence and Driver, 1994) was used to examine crossmodal cueing effects. In the orthogonal spatial cueing experiment, participants had to discriminate the elevation of a target sound rather than its laterality. The targets were preceded by an uninformative cue on the same side in fifty-percent of the trials and on the opposite side in the rest of the trials. The results of multiple manipulations of the orthogonal spatial cueing paradigm replicated the finding that auditory cues facilitate reaction times to visual targets. The authors claimed that Ward’s contradictory findings could be explained in light of spatial compatibility effects and response priming since Ward’s stimuli were lateralized.

Ward and his collaborators have subsequently demonstrated that the cross-modal asymmetry in favour of visual cueing still holds when all methodological confounds are removed (McDonald & Ward, 1999; Ward, McDonald, & Golestani, 1998; Ward, McDonald, & Lin, 2000). Ward, McDonald and Golestani (1998) showed that when cues and targets were presented

6

from exactly the same lateral eccentricity, responses to visual cues were faster, ruling out Spence and Driver’s criticism about Ward’s initial findings being based on spatial compatibility effects that arise as a result of cue-target lateralization differences. In another study by Ward, McDonald and Lin (2000), an implicit spatial discrimination paradigm was used to study cross-modal asymmetry and visual cue superiority was once again found eliminating effects of response priming on Ward’s early findings (1994).

Mondor and Amirault (1998) reacted to discrepancies in the direction of the crossmodal cueing (auditorily or visually driven effects) by conducting a comprehensive investigation into the impact of experimental set-up on cueing effects. The investigators manipulated cue and target modalities, stimulus onset asynchrony (SOA), and target location given cue location. In their first experiment, they tried to maximize uncertainty for the subject: an auditory or visual spatial cue preceded an auditory or visual lateralized target by either 150 or 300ms. The results showed that valid trials were faster than invalid trials (also known as the cue validity effect) only for intra- modal tasks (A-A, V-V) and only for an SOA of 150ms. Cueing for cross-modal cases did not reach significance. These results seemed inconsistent with past findings (Bertelson & Tisseyre, 1969), but Mondor and Amirault (1998) fostered a hypothesis that could address the discrepancy. They suspected that endogenous mechanisms were responsible for cross-modal cueing effects. That is, rather than pure perceptual-motor reactivity, it was expectation or anticipation that allowed cross-modal attention to be efficient. In their second experiment, Mondor and Amirault (1998) decreased the uncertainty: seventy-five percent of the cues were valid, while the modality of cue and stimulus was unpredictable and the SOA was a constant 150ms. A significant cue validity effect for A-V and V-A resulted even though the A-A and V-V effect was stronger. In a third experiment, Mondor and Amirault (1998) administered the conditions in blocks, thus rendering modality of cue and target predictable. This increased intra-modality cueing effects, though cross-modal cueing effects were unaffected. Mondor and Amirault (1998) concluded that cross-modal cueing was effective when endogenous mechanisms were at work, and those mechanisms were spurred by predictability (predictably of modality seemed irrelevant) or lack of uncertainty.

Although, Mondor and Amirault’s (1998) third experiment did not report strong crossmodal cueing effects, a subsequent study by Schmitt, Postma and de Haan (2000) found facilitated reaction times to both auditory and visual cues in an experimental set-up where cue

7

and target modalities were fixed within a block. Symmetric audio-visual cueing effects have subsequently been reported in other studies (McDonald & Ward, 1999; 2003). At present, facilitation effects have been shown for both auditory and visual cues but the exact neural underpinnings of these cueing effects have yet to be isolated.

On a side note, most crossmodal cueing experiments mentioned previously have manipulated SOA in addition to cue and target modalities (Bertelson & Tisseyre, 1969; Spence & Driver, 1997; Ward, 1994) to try to understand the type of attentional mechanisms that may underlie the integration of cue-target information. When Bertelson and Tisseyre (1969) varied stimulus onset asynchrony between 0 and 700 ms and measured the effect of SOA manipulation on responses to a target, they found that an auditory click decreased reaction time to a subsequent visual flash at the smallest SOA. At larger SOAs (greater than 70ms), in the case of the visual flash that preceded the auditory click, some facilitation of reaction time was noted but it was greatly attenuated in comparison to the auditory click – visual flash (A-V) case. The primary purpose of carrying out such behavioural experiments was to determine if there was an attentional refractory period that followed processing of the first stimulus. The logic was that if attentional mechanisms which may be governed by higher-order cognitive systems in the brain were required to process the first stimulus, than this processing would utilize resources that would not be available to process a subsequent stimulus till a certain time had elapsed (Davis, 1959). This elapsed time was known as the attentional refractory period. However, experiments by Bertelson and Tisseyre found that there was no attentional refractory period if the two stimuli were presented in two different sensory modalities (auditory and visual). When stimuli were presented in the same modalities (visual-visual: V-V), the facilitation was very weakly represented in the behavioural data. By using a substantially long SOA, both auditory and visual cueing effects can potentially be delineated.

1.4 Crossmodal Asymmetry Reported In Cognitive, Physiological, and Developmental Studies

The majority of behavioural studies conducted on some aspect of crossmodal cueing allude to the idea that there may be cognitive, physiological and developmental bases for auditory and visual interactions. Some of these studies will be described next.

8

1.4.1 Auditory to Visual (A-V) Interactions

Investigators examining multisensory interactions have found that a sudden sound can enhance the detection of a subsequent flash of light in the same location (McDonald, Teder- Salejarvi, & Hillyard, 2000). Abrupt sounds synchronized with visual search arrays can improve the identification of visual targets embedded in a series of distracters (Vroomen & de Gelder, 2000). These studies suggest that auditory events can influence visual processing.

According to Neumann, Van der Heijden, & Allport (1986), shifting visual attention to auditory events has evolutionary significance. They claim that auditory events that occur distally can be registered in the brain resulting in appropriate action whereas; by the time visual events come into view proximally, a response may not be viable. Since auditory events in the world are transient and intermittent relative to visual events which appear continuous in time, it is more beneficial to have an asymmetry in audio-visual cueing. Building on concepts proposed by Neumann and colleagues’ ideas, some researchers (Brown & May, 1989; Harrison & Irving, 1966; Spence & Driver, 1997) have suggested that the primary function of sound localization in animals is to the direct the eyes towards auditory events. In this way, sound localization may be necessary for the control of orienting towards significant distal events which occur outside an animal’s field of view.

Visual orienting with respect to sound localization has been explored in physiological studies involving the superior colliculus (King, 1993; Stein & Meredith, 1993; Stein, Wallace, & Meredith, 1995). In order to understand how the superior colliculus may contribute to the integration of audio-visual stimuli, it is imperative to consider the anatomical organization of this structure. The first three layers of the superior colliculus are purely visual and organized spatio- topically (de Monasterio, 1978a, 1978b; Dubois & Cohen, 2002; Leventhal et al., 1981; Perry and Cowey, 1984; Rodieck and Watanabe, 1993; Schiller and Malpeli, 1977; Sparks, 1988; Sparks & Hartwich-Young, 1989). There is no influence of audition on these first three superficial layers (King, 1993). The three layers beneath the visual layers are often referred to as the deep layers of the superior colliculus. These layers are considered polymodal because they receive ascending inputs from brainstem nuclei such as the inferior colliculus and the trigeminal nucleus that represent modalities such as audition, vision and touch (Wickelgren, 1971). Descending inputs from the temporal cortex and postcentral gyrus also culminate in these deep

9

layers (Wickelgren, 1971). Cells within the deep layers that receive afferents from auditory, visual pathways and multimodal cortical sites are known as multisensory integrative cells (MSI, term coined in Calvert, 2001). These MSI cells are not only capable of generating responses to different modalities but are also able to transform separate sensory inputs into an integrated product. In conclusion, the anatomy of the superior colliculus makes it an ideal candidate for carrying out multisensory integration; that is the assimilation of information from multiple sensory modalities into a motor outcome.

According to Wickelgren, the superior colliculus is able to track stimuli in either vision or audition that are moving laterally away from an animal to ensure appropriate avoidance or approach behaviour. This structure, in a variety of species ranging from amphibians to primates, is able to produce orienting movements linked to the eyes (saccades), head and body as well as approach, freezing or running responses (Sahibzada, Dean, & Redgrave, 1986; Sparks, 1999; Vargas, Marques, & Schenberg, 2000). Researchers have hypothesized that the superior colliculus may be vital in generating motoric responses to salient sensory events (Sparks, 1999; Stein, 1998).

Other neurophysiological investigations of the functional architecture of the superior colliculus have shown that there is a two-dimensional representation of auditory target location represented within deep layers of this structure (King, 1993; Stein & Meredith, 1993). Stein and Meredith (1993) however, are careful in noting that the deep layers are multimodal rather than purely auditory suggesting that any shifts of attention could implicate involvement of other modalities. King (1993) advocates further that there is no spatio-topic map of auditory space in the brain therefore; it unlikely that vision could guide auditory localization. Clarey, Barone and Imig (1992) have found that in structures like the inferior colliculus and the primary , there is a tonotopic organization with lateralized auditory receptive fields but there is no indication of spatial tuning or spatiotopy. Therefore, the only spatio-topic map of auditory space in the brain is the one found in polymodal deep layers of the superior colliculus.

Developmental studies of the superior colliculus have shown that the representation of auditory space in deep layers of the superior colliculus can be altered by varying the visual environment (King & Carlile, 1993; King, Hutchings, Moore and Blakemore, 1988; Withington, 1992). In contrast, manipulations of auditory experience have no influence on the spatiotopically

10

organized superficial layers of the superior colliculus (Knudsen, Esterly & Knudsen, 1984; Withington-Wray et al., 1990). This asymmetry in the direction of audio-visual interactions is also prominent in studies that have examined the development of senses in infants. According to Gottlieb (cited in Lewkowicz & Kraebel, 2004), different sensory modalities develop in the following sequence: tactile, vestibular, chemical, auditory and visual. The first four modalities mentioned are functional prenatally while vision develops most postnatally. Lewkowicz (1988a, 1988b) claims that sensory processing hierarchies evident in developmental profiles of infants may contribute to the different degrees of how input from a modality is transmitted to different unisensory and multisensory sites in the brain. For example, inputs from auditory cortical neurons could possibly project to areas of the that are still underdeveloped at birth however; neurons within the visual cortex may be unable to innervate the developed neuronal structure of the auditory cortex. Thus, the cellular architecture that supports a particular sensory modality may result from temporal differences in the development of the senses.

1.4.2 Visual to Auditory (V-A) Interactions

Over the years, researchers have reported some effects of vision influencing audition. These studies are not as extensive as the ones mentioned previously for the auditory capture of visual attention but are worth considering.

The famous McGurk effect provides evidence for vision altering speech perception (McGurk & MacDonald, 1976). For example, a sound of /ba/ is perceived as /da/ when it is coupled with a visual lip movement associated with /ga/. The McGurk effect suggests that sound can be misperceived when it is coupled with different visual lip movements. fMRI studies by Calvert and colleagues (1997, 1999, 2000) have investigated brain activity in relation to incongruent audio-visual linguistic information (similar to the McGurk effect). Calvert et al. (1997) found that lip-reading (visual speech) activated areas of the auditory cortex that were previously considered to be unimodal. In a subsequent study, Calvert and colleagues (1999) showed an enhancement in sensory-specific cortices when participants saw and heard speech. These neuroimaging studies uphold the view that vision can influence audition.

Another well-known effect of vision on audition is the ventriloquist effect first reported by Howard and Templeton (1966). The ventriloquist’s illusion is caused when the perceived location of a sound shifts towards the location of the visual source. Ventriloquists are able to

11

produce speech without visible lip movements while simultaneously moving a puppet. This leads to a conflict between visual and auditory localization that culminates in vision dominating audition. A recent study by Guttman, Gilroy and Blake (2005) has suggested that in audio-visual conflict situations, vision dominates spatial processing while audition directs temporal processing. The effects stated in this section suggest that vision can alter audition in very specific cases such as those of conflicting information from multiple modalities. The influence of vision on audition in terms of spatial localization and cueing still remains an area that has potential for exploration.

1.5 General Anatomy of Central Auditory and Visual Pathways

An exploration of the characteristics of audio-visual interactions is incomplete without a consideration for the general organization of auditory and visual pathways in the brain. This section will briefly review some general areas that are involved in the processing of auditory and visual stimuli.

1.5.1 The Auditory Pathway

Sound waves from the environment are captured and focused by the auricle – the external ear – into the auditory canal. The auditory canal is a hollow, air-filled tube that transmits sound waves to three bones located in the middle ear (the malleus, the incus and the stapes). These three bones amplify the auditory signal enabling it to travel through the fluid-filled inner ear structure called the cochlea. The cochlea is the primary site of the conversion of sound energy into a neural code. This process is called signal transduction (Noback, 1967). Outer hair cells, also known as auditory sensory receptors, in the cochlea display motility converting mechanical sound energy into receptor potentials. Subsequently, these outer hair cells transmit receptor potentials to inner hair cells which provide frequency and intensity information to cochlear ganglion cells (Miller & Towe, 1979). The innervation of hair cells by ganglion cells in the cochlea is the first part of the auditory neural pathway where information is encoded both in terms of frequency and intensity of sound. Neurons within the cochlea respond best to stimulation at characteristic frequencies of contiguous cells. In this way, tonotopy – organization of tones that share similar frequencies into topologically neighbouring neurons – begins postsynaptic to inner hair cells (Spoendlin, 1974).

12

Axons from the cochlear ganglion cells form the cochlear component of the eighth cranial nerve synapsing in the cochlear nuclear complex of the medulla-pontine junction. The cochlear nucleus sends auditory inputs via three different pathways – dorsal acoustic stria, intermediate acoustic stria and trapezoid body – to the pons. The trapezoid body projects to the superior olivary nuclei where information from both ears (binaural) is processed. Localization of sounds in space occurs in the medial and lateral portions of the superior olivary nucleus. Efferents from the superior olivary nucleus, dorsal and intermediate acoustic stria terminate in the inferior colliculus by way of the lateral lemniscus nuclei. The inferior colliculus in turn sends outputs to the medial geniculate nucleus of the thalamus which end up in the primary auditory cortex (area A1, or ; Brodmann, 1909) located on Heschl’s gyrus in the superior (adapted from Brodal, 1981, Hudspeth, 2000). The auditory cortex is organized in concentric bands with the primary auditory cortex in the centre and auditory association areas forming the periphery (see Pandya, 1995 for review). The auditory neural pathway maintains its tonotopic organization from the cochlear ganglion cells to the primary auditory cortex

For decades, it was assumed that the auditory neural pathway processed exclusively acoustic information. In recent years, studies have shown that outputs from midbrain structures such as the inferior colliculus and the medial geniculate nucleus of the thalamus can contain some visual information (Komura et al., 2005; Porter, Metzger, & Groh, 2007). Porter, Metzger and Groh (2007) have demonstrated that the inferior colliculus in monkeys carries visual, specifically saccade-related information in addition to auditory responses. This study suggests that the inferior colliculus, predominantly considered to be a unisensory structure responsive to only auditory stimuli, may have the capacity to integrate specific types of audio-visual information. In another study by Komura and colleagues (2005), rats were trained to perform an auditory spatial discrimination task with auditory or auditory-visual cues. The auditory-visual cues were presented in such a manner that the visual cues were either congruent with auditory cues or provided conflicting information. The results showed that almost fifteen percent of auditory thalamic neurons were modulated by visual cues. Responses in these neurons were enhanced in congruent conditions and suppressed when auditory and visual cues conflicted. Studies by Porter, Metzger and Groh, and Komura et al. imply that the auditory cortex receives some visual input via subcortical structures within the central auditory pathway; the extent of

13

which is not clear. Thus, unisensory sites in the brain may have the ability to perform some multisensory functions.

1.5.2 The Visual Pathway

Light entering the eye is detected by the retina which is composed of photoreceptors (cones and rods) that transduce light into electrical signals that can be decoded by the brain. Information from rods and cones feeds into a network of interneurons composed of horizontal, bipolar and amacrine cells which in turn project to ganglion cells. Axons from the retinal ganglion cells form the optic nerve (Wurtz & Kandel, 2000a; 2000b). The optic nerve carries information from both eyes till it reaches the optic chiasm where inputs from the left side of each eye (left hemifield) and the right hemifield cross to the contra-lateral hemisphere (Guillery, 1982). From the optic chiasm, visual inputs flow through optic tracts segregated by eye (left or right) to the lateral geniculate nucleus of the thalamus. The primate lateral geniculate nucleus contains six layers. Layers 1 and 2 are called the magnocellular layers and layers 3-6 are termed the parvocellular layers (Kaas, Guillery, & Allman, 1972; Sherman, 1988). Both magno- and parvo-cellular layers project to the primary visual cortex (area V1 or 17; Brodmann, 1909). V1 sends outputs to V2/V3 from which point the visual information is split into two streams – temporal and parietal (see Desimone & Ungerleider, 1989). The visual association areas include areas such as V4, V5 and MT (middle temporal area; adapted from Wurtz & Kandel, 2000a; 2000b). The plethora of connections between primary visual areas and visual association cortices are beyond the scope of this discussion but can be reviewed in a paper by Felleman and Van Essen (1991). Despite the multitude of levels in the central visual pathway, information is maintained retinotopically - adjacent areas in the visual field are encoded by slightly different but overlapping neuronal receptive fields.

Apart from the major retinogeniculostriate pathway, the retina also projects to other subcortical areas such as the superior colliculus (Leventhal, Rodieck, & Drehkr, 1985; Magalhaes-Castro, Murata, & Magalhaes-Castro, 1976; Rodieck & Watanabe, 1993; Wassle & Iling, 1980) and the pulvinar nucleus of the thalamus (Grieve, Acuna, & Cudeiro, 2000; Itoh, Mizuno, & Kudo, 1983; Mizuno et al., 1982; Nakagawa & Tanaka, 1984) as seen in primates and cats. The superior colliculus and the pulvinar nucleus are also extensively interconnected (Grieve, Acuna & Cudeiro, 2000; Stepniewska, Qi, & Kaas, 2000). The anatomical organization

14 of both these subcortical structures and their interactions with auditory and visual pathways places them in a unique position to integrate multisensory information.

1.6 From Anatomy to Function - A Dynamic Systems’ Perspective

As discussed in previous sections, the anatomical architecture of subcortical structures in auditory and visual pathways supports both unisensory and multisensory processing. At the cortical level, Felleman and Van Essen (1991) have documented extensive feed forward, feedback and horizontal connections between visual and multisensory brain areas. Connections between unimodal auditory cortex and primary visual areas have been mapped in primates by Rockland and Ojima (2003) and Falchier et al. (2002). Cognitive neuroscience studies have also successfully identified multisensory sites that include cortical regions like the posterior parietal cortex (Bushara, Grafman, & Hallett, 2001; Iacoboni, Woods, & Mazziotta, 1998), intraparietal (Macaluso & Driver, 2005), premotor areas (Dassonville et al., 2001; Kurata et al., 2000; Tanji & Shima, 1994), anterior cingulate (Laurienti et al., 2003; Taylor et al., 1994) and (Asaad, Rainer, & Miller, 1998; Bushara, Grafman, & Hallett, 2001; Laurienti et al., 2003; Tanji & Hoshi, 2001; Taylor et al., 1994) and subcortical regions such as the thalamus (Grieve, Acuna,& Cudeiro, 2000; Porter, Metzger, & Groh, 2007), superior colliculus (Calvert, 2000; King, 1993; Stein & Meredith, 1993; Stein, Wallace, & Meredith, 1995), cerebellum (Allen, Buxton, Wong, & Courchesne, 1997; Bense et al., 2001; Kim et al., 1999) and parts of the basal ganglia (Chundler, Sugiyama, & Dong, 1995; Nagy et al., 2006). The abundance in areas that process multisensory information suggests that there are large degrees of functional overlap and redundancy in the brain. This idea was initially asserted by Mountcastle (1979) who noticed that proximal areas showed similar functional characteristics and that there were many structural and functional redundancies in sensorimotor systems.

Theories about how the brain’s structural capacity contributes to its functional properties can be classified into a framework that considers the brain to be a dynamic system. This framework has been defined by researchers in numerous ways, some of which will be discussed below (Bressler, 1995; Bressler, 2002; Bressler & McIntosh, 2007; Bressler & Tognoli, 2006; Fuster, 1997; Goldman-Rakic, 1988; Mesulam, 1981, 1990; 1998; McIntosh, 1999; Price & Friston, 2002).

15

Bressler and Tognoli (2006) suggest that the functional expression of a particular cognitive operation results from the co-activation of specific interconnected local area networks. Mesulam (1990) has also stated that a cognitive or behavioural operation can be subserved by several interconnected brain areas each of which is capable of multiple computations. Therefore, a cognitive operation such as attention can be controlled by a diffuse cortical network that is redundant and specialized at the same time (Mesulam, 1981). According to his view, each region within a cortical network has some specialization because of its anatomical connections but this specialization is not absolute in that lesions to different areas in a network could have similar behavioural consequences. Goldman-Rakic, in a review paper (1988) comparing the literature on different models of cortical organization claims that the brain’s association cortices (areas that are largely responsible for complex processing that occurs between sensory input to primary sensory cortices and motor output elicited by primary motor areas) interact to form a finite number of dedicated networks that are reciprocally interconnected. These networks are capable of communicating with sensorimotor systems to produce integrated behaviour.

One dynamical systems’ perspective that has gained momentum in the past few decades is the notion that regions of the brain that share similar structural properties can contribute to a multitude of functions by way of their interactions with other regions (Bressler, 1995; Bressler, 2002; McIntosh, 1999; Bressler & McIntosh, 2007, Price & Friston, 2002). These interactions could be represented via direct or indirect connections. In addition, brain areas that have very different neural connections can contribute to the same functional output. This framework of distributed function in the brain has been defined operationally by McIntosh (1999, 2004) using the term neural context. Neural context represents the local processing environment of a given brain area that results from modulatory influences from other brain areas (Bressler & McIntosh, 2007; McIntosh, 1999). Therefore, cognitive function may not be localized in an area or network of the brain but may emerge from dynamic large-scale neural interactions between different brain areas that change as a function of task demands. Task demands, in broader terms, can be the situational context in which an event occurs. Situational context refers to environmental factors such as sensory input and response processing in which a task is performed. In most cases, neural context is elicited from changes in situational context (Bressler & Tognoli, 2006; Protzner & McIntosh, 2007).

16

While dynamical models incorporating large-scale neural interactions, neural and situational contexts have been explored for cognitive processes such as learning and memory (Lenartowicz & McIntosh, 2005; McIntosh & Gonzalez-Lima, 1994; 1998), they have seldom been applied to investigate interactions between cognitive processes such as attention and sensorimotor systems. Brain areas that display attention-sensorimotor interactions can be dissociated using techniques like functional resonance imaging (fMRI). The use of fMRI to study brain function will be discussed in the next section.

1.7 An Overview of fMRI 1.7.1 Basic MRI Physics

The MRI concepts presented here are summarized from Brown and Semelka (1999) and Huettel, Song and McCarthy (2004).

The proton of a hydrogen atom has a magnetic spin that is denoted by a spherical, distributed positive charge. This charge rotates about an axis at high speeds producing a small magnetic field called a magnetic moment. In addition to the magnetic moment, a proton’s mass in combination with the rotating charge produces angular momentum. The magnetic moment and the angular momentum form the basis of the spin properties of a proton. When a proton is exposed to a uniform magnetic field (B0), it can assume one of two types of spins – parallel

(same direction as B0) and anti-parallel (opposite direction to B0) – forming the equilibrium state. Parallel spins are lower in energy and more stable compared to anti-parallel spins. In order to convert a proton in a parallel spin state to an anti-parallel state, the proton needs to absorb electromagnetic energy. By the same token, a proton in a high energy state emits electromagnetic energy as it returns to a low energy state. The electromagnetic radiation frequency required to excite a proton from a low-energy state to a high-energy state can be calculated for an MR scanner. This frequency is called the Larmor frequency and is needed to change spins from parallel to anti-parallel orientations.

Apart from the changes in spin states induced by B0, a proton’s motion about its axis can also be influenced by B0. In the presence of B0, the axis of rotation for a proton can rotate around the direction of B0. The motion of a proton in B0 is referred to as spin precession. The physical characteristics of spin precession are exploited in MRI.

17

In an MRI set-up, a participant is placed in the centre of a uniform magnetic field. Protons within brain tissue assume their equilibrium states (parallel or anti-parallel). Using the Larmor frequency to generate an electromagnetic radiation pulse, protons can be excited and their rotational axis perturbed to generate an MR signal. Once the electromagnetic pulse is removed, the MR signal starts to decay. MR signal decay, also known as spin relaxation, is of two types – longitudinal and transverse. Longitudinal relaxation (T1 recovery) corresponds to the return of net magnetization in the same plane as B0. It is caused by protons in high-energy, anti- parallel spins (excited state) returning to low-energy, parallel, relaxed states. In order to understand transverse relaxation (T2 decay), consider the following: an electromagnetic pulse tips the axis about which a proton precesses in the transverse plane such that all protons precess in the same phase. When the pulse is removed, this phase-locking gradually subsides and protons return to their original out of phase states. This is referred to as transverse relaxation. The rates of longitudinal and transverse relaxation are constant for particular substances such as water, bone or fat. The constants that describe longitudinal and transverse relaxation are called T1 and

T2, respectively.

While both the T1 and T2 constants are important for MR, a third constant, T2* is essential for functional MR. T2* includes transverse relaxation due to phase differences in spin precessions as well as local magnetic field inhomogeneities. The latter can be described by considering an inhomogeneous external magnetic field where the strength of the magnetic field at a particular location influences the spin precessional frequency. Perturbed protons at different locations within the magnetic field lose coherence in their spin precessions at different rates contributing to the decay of net magnetization in the transverse plane.

An MR signal can highlight different parts of the brain depending on the contrast used. A

T1-weighted image of the brain shows high signal (bright) for fat content and low signal for cerebrospinal fluid (CSF; dark) while a T2-weighted image shows the opposite contrasts for fat and CSF.

Two parameters that are critical to the amount of MR signal recorded and the contrast expressed are repetition time (TR) and echo time (TE). The interval between successive electromagnetic pulses that excite protons is referred to as TR. The TR influences the rate of longitudinal recovery after an excitation pulse is removed. TE corresponds to the interval

18

between the excitation pulse and the acquisition of data and affects the rate of transverse decay. By varying the length of the TR or TE, image intensity at each spatial location can be made more or less sensitive to differences in T1 and T2.

1.7.2 Physiological Basis of BOLD fMRI

Functional MRI (fMRI) can be used to estimate the spatial locations of neural activations and associated changes in metabolic demands when a person performs a certain task. These metabolic changes include variations in concentration of deoxygenated hemoglobin, blood flow and blood volume, (Buxton, Wong, & Frank, 1998; Kwong et al., 1992) all of which play a role in producing the blood-oxygenation-level dependent (BOLD) response recorded in fMRI. To understand the BOLD response, the magnetic properties of oxygenated and deoxygenated hemoglobin must first be considered. Oxygenated hemoglobin is diamagnetic (does not affect magnetic field strength) while deoxygenated hemoglobin is paramagnetic (distorts local

magnetic fields). The paramagnetic properties of deoxygenated hemoglobin decrease T2* values (Thulborn et al., 1982) which allows measurement of changes in brain activity as indexed by changes in the amount of deoxygenated hemoglobin.

The BOLD response in MR was first discovered by Ogawa and colleagues in 1990. Anaesthetised rats were placed in an MR scanner. The experiment was performed to try to investigate brain physiology with MRI. It was known by that point that deoxygenated

hemoglobin decreased blood’s T2* values (Thulborn et al., 1982). Ogawa et al. (1990) used this finding to demonstrate that varying the proportion of blood oxygenation in rats could lead to different MR image characteristics. In cases where rats breathed in oxygen at a hundred percent concentration, Ogawa et al., noticed that T2* images of the brain showed structural differences and very little vasculature. As the amount of oxygen in rats decreased, brain vasculature became more prominent. To test this interesting effect, a saline-filled container that had test-tubes of oxygenated and deoxygenated blood within it was imaged using at T2* contrast (Ogawa & Lee,

1990). T2*-weighted images of oxygenated blood showed a dark outline around the test-tube’s diameter. In contrast, deoxygenated blood showed a greater signal loss that spilled over into the area filled with saline. Ogawa and others concluded that functional changes in brain activity could be studied using what was to be known as the BOLD contrast.

19

Following Ogawa’s work, Belliveau and colleagues (1991) observed the first functional images of the brain by using a paramagnetic contrast agent called Gd (DTPA)2-. Subsequently, Ogawa and others (1992) and Kwong et al., (1992) published the first functional images using the BOLD signal.

The physiological basis of the BOLD signal can be summarized as follows. Relative amounts of oxygenated and deoxygenated hemoglobin in the capillary bed of a brain region depend on the ratio of oxygen consumption and supply. When neural activity increases, the amount of oxygenated blood delivered to that area also increases while levels of deoxygenated hemoglobin decrease. The BOLD signal captures the displacement of deoxygenated hemoglobin by oxygenated blood since the former has the capability of affecting magnetic fields but the latter does not. In other words, changes in oxygenation levels lead to the modulation of microscopic field gradients around blood vessels which in turn affect T2* values for tissue water that produce differences in signal strength (Huettel, Song, & McCarthy, 2004).

A gradient echo-planar imaging sequence that is sensitive to the paramagnetic properties of deoxygenated hemoglobin can be used to display tomographic maps of brain activation (Brown & Semelka, 1999; Kwong et al., 1992; Huettel, Song, & McCarthy, 2004).

1.7.3 Coupling of Neuronal Activity & BOLD

The majority of functional neuroimaging studies assume that the physiological changes underlying the BOLD response are capturing neuronal activity. However, the nature of neuronal activity represented by BOLD responses is still an actively debated topic.

Neuronal activity that would predict fMRI BOLD response could include multiple factors such as the average firing rate of a sub-population of neurons also referred to as multi-unit activity (MUA; Legatt, Arezzo, & Vaughan, 1980); synchronous spiking activity across a neuronal population; the local field potential (LFP) which represents the synchronization of dendritic currents (Mitzdorf, 1987), or some measure of the sub-threshold electrical activity as measured by single-unit recordings (SUA, measured by Hubel & Wiesel, 1959). The size of a neuronal population whose activity is indexed by fMRI signals may also be an issue. Scannell and Young (1999) postulate that changes in fMRI BOLD responses could be caused by large

20

changes in firing rates in small neuronal populations or small changes in firing rates in larger neuronal sub-populations.

Logothetis and colleagues (2001) attempted to investigate the relationship between neuronal firing rates in the brain and subsequent BOLD responses picked up by fMRI. Monkeys were presented with rotating checkerboard patterns and both fMRI BOLD responses and electrophysiological signals were measured from primary visual cortex. The electrophysiological data that was acquired consisted of SUA, MUA and LFP recordings. The results showed a transient increase in BOLD at the onset of the visual stimulus which persisted until the visual stimulus went offline. Approximately twenty-five percent of MUA showed a transient increase in activity and subsequent return to baseline, while LFPs were sustained throughout the stimulus duration. The authors claimed that increased LFPs during stimulation were significantly stronger than MUA and were maintained over longer intervals therefore; LFPs give a better estimate of BOLD responses than MUA. Logothetis et al.’s (2001) paper also suggested that LFPs resembled integrative activity at neuronal dendritic sites while MUA corresponded to the axonal firing rate of a small population of neurons. Hence, BOLD seemed to reflect incoming input and local processing in an area more that spiking activity.

The experiment by Logothetis and colleagues (2001) assumed a somewhat linear relationship between BOLD and LFPs. A subsequent study by Mukamel and colleagues (2005) suggested that BOLD responses may be comprised of more complex neuronal activity that does not necessarily follow a linear pattern. In Mukamel et al.’s experiment (2005), SUA and LFPs were recorded from two neurosurgical patients. fMRI BOLD signals were collected from healthy participants. Both patients and participants viewed a movie segment during measurements of neural activity. The results showed a high, significant correlation (r = 0.75) between SUA from neurosurgical patients and fMRI BOLD signals in healthy controls. The authors claimed that fMRI BOLD responses were reliable measures of firing rates in human cortical neurons.

While the findings of Logothetis et al. (2001) and Mukamel et al., (2005) do not directly contradict each other; it does appear though that neuronal activity that is represented in the BOLD response may be a complex milieu of input and output processing. According to a review article on coupling between BOLD and neuronal activity, Heeger and Rees (2002) state that LFPs could be dominated by activity in proximal neurons which would suggest that local spiking

21 activity, synaptic activity and dendritic currents are all co-varying. The authors emphasize that fMRI BOLD response may be capturing both presynaptic and postsynaptic activity within a particular region. For experiments that are designed to investigate global changes in brain activity, having the resolution of single-unit recordings in fMRI may not be necessary.

22

Chapter 2: Aims and Hypotheses

The primary objective of the present study is to understand crossmodal facilitation effects in the brain. There are discrepancies in behavioural research about the direction of crossmodal facilitation. To recap, some scientists have found that auditory cues are superior to visual cues in producing fast responses (Bertelson & Tisseyre, 1969; Buchtel & Butter, 1988; Farah, Wong, Monheit, & Morrow, 1988; Spence & Driver, 1997) while other researchers have shown the opposite effect – visual cues facilitate reaction times to auditory targets (McDonald & Ward, 1999; Ward, McDonald, & Golestani, 1998; Ward, McDonald, & Lin, 2000; Ward, 1994). There have been no studies that I am aware of that have investigated the effects of audio-visual crossmodal facilitation in the brain using fMRI.

The hypotheses for the current experiment are as follows. Firstly, the magnitude of facilitation for auditory cues will be larger compared to visual cues given the greater neuroanatomical capacity for audition to influence vision. Secondly, auditory and visual cueing will be represented by distinct patterns of brain activation given the differences in behavioural performance. Lastly, brain areas that respond to cue processing (input) may not be the same areas that coordinate behaviour (output).

The first two hypotheses will be tested in Chapter 3 and the last hypothesis will be focused on in Chapter 4. In order to test the hypotheses mentioned, participants will perform a crossmodal version of the spatial stimulus-response compatibility task while BOLD fMRI responses are recorded. In spatial stimulus-response compatibility tasks, a cue signals the response rule to a lateralized target. Responses (button presses) are made to the same (compatible) or opposite (incompatible) side of target presentation. Reaction times are faster in compatible conditions than in incompatible conditions (Simon, 1969; Fitts & Seeger, 1953). In this experiment, the cue and targets will be presented in both auditory and visual modalities.

A unique aspect of the current experimental design that I would like to emphasize is the manipulation of cue-target order. Tasks can be of two types – cues appearing first followed by targets or targets being presented first followed by cues. Previous behavioural studies that have investigated crossmodal facilitation did not manipulate order however, in my fMRI study I can

23 use this order manipulation to examine changes in neural activity based exclusively on cue or target processing.

24

Chapter 3: Attentional Cueing Modulates Multisensory Interactions in Human Sensory Cortices

3.1 Introduction

Experiments on selective attention have shown that when participants are provided with a cue, they shift their attention to the cued location (Posner, Inhoff, Friedrich, & Cohen, 1987). Attentional tasks where cues and targets are manipulated have been adapted to study crossmodal facilitation effects (Spence and Driver, 1997; Ward, 1994). Crossmodal facilitation occurs when a cue in one sensory modality elicits a speeded response in another sensory modality. A study conducted by Ward (1994) investigated the effects of crossmodal facilitation using a spatial discrimination task. Participants made speeded left-right responses to visual or auditory targets following the presentation of an auditory or a visual non-predictive cue, auditory and visual cues or no cues. Reaction times were measured for all conditions at different inter-stimulus intervals between the cue and target. The results indicated that visual cues facilitated reaction times to auditory targets presented on the same side of the cue (compatible) at short inter-stimulus intervals. Auditory cues, in contrast, did not facilitate reaction times to visual targets on either side of cue presentation or at any inter-stimulus interval. Auditory cues did facilitate reaction times to compatible auditory targets at short inter-stimulus intervals. Ward’s findings were met with skepticism because previous crossmodal studies that had presented non-predictive auditory cues had shown response time facilitation to visual targets ipsa-lateral to the cue (Buchtel & Butter, 1988; Farah, Wong, Monheit, & Morrow, 1989). A crossmodal study conducted after Ward’s study also showed an asymmetrical facilitation of reaction times for auditory cues in comparison to visual cues (Spence & Driver, 1997).

Spence and Driver (1997) explained the asymmetrical auditory cue facilitation as having evolutionary significance. Auditory events in the world are transient and intermittent whereas visual events are continuous in time. Also, auditory events that occur distally could be registered in the brain resulting in appropriate action whereas; by the time visual events come into view proximally, a response may not be viable. Therefore, it is more beneficial to shift visual attention to auditory events rather than the other way around (Neumann, Van der Heijden, & Allport, 1986). Evidence from neuropsychological studies also indicates that orienting to auditory events is usually followed by visual localization in areas like the superior colliculus (Stein & Meredith,

25

1993; Stein, Wallace, & Meredith, 1995). McDonald, Teder-Salejarvi & Hillyard (2000) showed that sudden sound can improve the detection of a subsequent flash of light at the same location. Abrupt sounds synchronized with visual search arrays can improve the identification of visual targets embedded in a series of distracters (Vroomen & de Gelder, 2000). The evidence presented thus far implies that auditory events can influence the processing of subsequent visual events.

However, there has also been some evidence in favour of visual facilitation of reaction time. Schmitt et al. (2000) found facilitated reaction times to both auditory and visual cues in an experimental set-up where cue and target modalities were fixed within a block. Symmetric audio-visual cueing effects have subsequently been reported in other studies (McDonald & Ward, 1999; 2003). The famous McGurk effect also provides evidence for vision altering speech perception (McGurk & MacDonald, 1976). For example, a sound of /ba/ is perceived as /da/ when it is coupled with a visual lip movement associated with /ga/. The McGurk effect shows that sound can be misperceived when it is coupled with different visual lip movements. These studies suggest that vision can also alter audition in some cases.

The lack of consensus about the direction of reaction time facilitation in response to auditory and visual cues is further compounded by attempts to understand the exact nature of cue-target processing in the brain. Some researchers argue that salient sensory information contained in the cue is integrated with target information via separate, modality-specific sub- systems (Bertelson & Tisseyre, 1969; Bushara et al., 1999; Cohen, Cohen, & Gifford, 2004; Posner, Inhoff, Friedrich, & Cohen, 1987; Spence & Driver, 1997; Ward, 1994). Alternatively, other scientists argue that synthesis of information from different sensory modalities is achieved through a supramodal network that involves parts of the prefrontal cortex and parietal areas (Andersen, 1995; Andersen, Snyder, Bradley, & Xing, 1997; Bedard, Massioui, Pillon, & Nandrino, 1993; Downer, Crawley, Mikulis, & Davis, 2000; Eimer & Driver, 2001; Farah, Wong, Monheit, & Morrow, 1989; Iacoboni, Woods, & Mazziotta, 1998; Laurens, Kiehl, & Liddle, 2005; Snyder, Batista, & Andersen, 1997). Convergent theories, advocated by Macaluso and others suggest that while supramodal attentional networks may guide sensorimotor integration, reverberating loops that link sensory-specific cortices to each other can also integrate sensory information across modalities (Corbetta & Shulman, 2002; Ettlinger & Wilson, 1990; Macaluso, 2006; Macaluso & Driver, 2005). Macaluso and colleagues have derived a convergent

26

model for visual and tactile modalities but this model has not been applied to audio-visual cue- target processing.

In order to reconcile discrepancies found in behavioural crossmodal facilitation data and exact brain mechanisms responsible for multisensory processing; I designed a task that attempted to capture audio-visual interactions between a cue and a target in an event-related functional neuroimaging study. A stimulus-response compatibility paradigm, traditionally used to study response selection and cue-target processing (Bertelson and Tisseyre, 1969; Rodway, 2005), was modified to investigate crossmodal cue-target interactions. A general version of the stimulus- response compatibility paradigm has within it a cue that signals a response rule to a lateralized target. Response times are faster when responses are made to the same side as the presentation of a target (compatible responses) compared to when responses are made to the opposite side of target presentation (incompatible responses). This robust behavioural finding is known as the stimulus-response compatibility effect (Simon, 1969; Fitts & Seeger, 1953).The cues and targets in my experiment were presented in auditory and visual modalities.

The aims of the first part of my study are to determine the neural correlates that underlie reaction time facilitation in response to auditory and or visual cues. Given the discrepancies in behavioural findings, I am unsure about the direction of reaction time facilitation however, I hypothesize that auditory cues will facilitate reaction times to visual targets because of the structure of auditory and visual neural pathways in the brain (see Chapter 1 for details).

3.2 Materials and Methods 3.2.1 Participants

Twenty-four (12 female) healthy, right-handed individuals between the ages of 19 and 35 (mean age - 23.08 ± 3.87 years) were voluntarily recruited through undergraduate psychology courses to partake in the study. All individuals were screened for any history of medical, neurological, psychiatric, substance abuse-related problems (see Appendix A and B for screening forms) prior to their participation in the study. All participants signed an informed consent (see Appendix C) and were reimbursed $100.00 for two sessions. The experiment was conducted in the fMRI Suite at Baycrest upon approval from the Research Ethics Board at Baycrest.

27

3.2.2 Stimuli

Two types of auditory stimuli that were matched in amplitude but varied in frequency (250 Hz and 4000 Hz) were used in the experiment. Each participant adjusted the volume of all auditory tones using the left and right buttons on the response box such that the tones appeared perceptually identical. The volume adjustment was conducted at the beginning of the experiment with the scanner turned on so that participants could adjust the volume of the stimuli according to the noise produced by the scanner.

The visual stimuli used in the experiment were of two types and were matched for luminance and contrast. The first visual stimulus was a black and white checkerboard pattern and the second visual stimulus was also a black and white checkerboard pattern, but rotated at a 45 degree angle. Stimulus presentation was controlled and documented by Presentation software (version 10.2, Neurobehavioural Systems Inc.)

3.2.3 Apparatus

Participants viewed visual stimuli on a translucent screen via a mirror that was mounted on the head coil (used for acquiring images) in the MRI scanner. The total distance between the participants’ eyes and the screen was approximately 52 inches. The size of the image on the screen was 13.75 inches by 11 inches with an image resolution of 800 x 600 x 60 Hz. The field of view (FOV) was 12 degree vertical and 15 degree horizontal. The majority of the participants wore their own contact lenses for vision correction. MR safe glasses made by SafeVision (Webster Groves, MO, USA) with a range from +/- 6 dioptres in increments of 0.5 dioptres were provided to those participants that did not have prescription contact lenses. Auditory stimuli were presented using the Avotec Audio System (Jensen Beach, FL, USA). A button-press response was made by participants with either the left or the right index finger using Fiber-Optic Response Pad System developed by Current Designs Inc. (Philadelphia, PA, USA). The response pad system had two paddles – one for each hand and each paddle had four buttons. The first button used for the left hand (left paddle) was the first button in the four buttons starting from the right going left. The first button used for the right hand (right paddle) was the first button out of four buttons going from left to right. The other three buttons on both hands were taped to prevent responses. Participants rested their hands gently on top of the taped buttons. The signal transmission of the paddles was less than 1ms.

28

3.2.4 Procedure

3.2.4.1 Trial Structure

A trial had the following sequence: presentation of the first stimulus (S1) for 250ms, a 4- second inter-stimulus interval (ISI), presentation of the second stimulus (S2) for 250ms and a response window of 1500ms. The inter-trial interval (ITI) was jittered randomly at 3, 5, 7, 9 seconds. Response times were recorded using Presentation software from the onset of the second stimulus.

3.2.4.2 Task Types

The study consisted of two scanning sessions on different days. Each session was 1.5 hours in length. Two different tasks were performed in a session. Each session comprised of six runs (9 mins, 28 seconds in duration, three runs per task type) with 40 trials (20 compatible, 20 incompatible) within each run. A trial was compatible when the cue instructed participants to press a button on the same side (left or right) of a lateralized target. The lateralized target was presented on either the left or the right side of a fixation cross. In an incompatible trial, participant’s pressed the button the opposite side of the lateralized target as indicated by the cue. Tasks fell into the following two categories: auditory cue-visual target and visual cue-auditory target.

Auditory cue – Visual target. The pitch of a binaural auditory tone cued the response rule to a lateralized visual target, a square checkerboard. For the compatible conditions, the low pitched tone (250 Hz) cued participants to respond with the hand ipsa-lateral to the side of the visual target. For the incompatible conditions, a high pitched tone (4000 Hz) cued participants to respond with the hand contra-lateral to the visual target. Cue-target order was manipulated resulting in two types of tasks: presentation of the auditory cue first followed by the visual target (AC-VT) and presentation of the visual target first followed by the auditory cue (VT-AC; see Figure 3.1). Each task type (AC-VT, VT-AC) had both compatible and incompatible conditions randomly interspersed.

Visual cue – Auditory target. The shape of a visual stimulus signalled the response rule to a lateralized, monaural tone of 250 Hz. For compatible conditions, a square checkerboard cued participants to respond with the hand ipsa-lateral to the side of target presentation. For

29

incompatible conditions, a checkerboard rotated 45 degrees cued participants to respond with the hand contra-lateral to the side of target presentation. Cue-target order was manipulated resulting in two types of tasks: presentation of the visual cue first followed by the auditory target (VC-AT) and presentation of the auditory target first followed by the visual cue (AT-VC; see Figure 3.2). Each task type (VC-AT, AT-VC) had both compatible and incompatible conditions interspersed at random.

Half the participants performed the cue-first tasks (AC-VT, VC-AT) in one session and the target-first tasks (VT-AC, AT-VC) in the other session. The other half of the participants performed the auditory cue-visual target tasks (AC-VT, VT-AC) in one session and the visual cue-auditory target tasks (VC-AT, AT-VC) in the other session. The rationale behind having this counterbalanced, dual-session design was to maximize data collection.

3.2.4.3 fMRI Session

A participant arrived thirty minutes prior to his or her scheduled scan time. He or she was escorted from the research building to the MRI suite by the experimenter. Upon arrival into the MRI suite, the participant was ushered into a waiting room where he or she filled out the MRI screening form for the second time. The MRI screening form had been filled out over the phone prior to the scan date nevertheless; the form was verified and signed on the actual scan date. The experimenter then explained the tasks to the participant and obtained informed consent. Subsequently, ten practice trials for each task were given to the participant on a laptop computer. There were no learning-related delays in performing the task. Participants had an accuracy of ninety-nine percent on average across all tasks on the practice trials. After the practice trials were complete, the MRI technologist spoke privately to the participant to verify all MRI screening- related information. Afterwards, the participant changed into hospital pants and a gown and his or her personal belongings were secured in a locker.

The MRI technologist was responsible for setting up the participant in the MRI scanner. Following the preliminary MRI set-up, acquisition of the structural scan took place. Then the experimenter waited for the study to start since the MRI triggered the Presentation program that delivered stimuli to the participant. At the end of all functional scans, participants were removed from the MRI scanner and reimbursed for their time.

30

3.2.5 fMRI Scanning Parameters

Participants were scanned on a Siemens Magnetrom Trio Tim Whole Body 3T MR scanner with a matrix 12-channel head coil. A structural MRI was obtained for each participant

at the beginning of each scanning session, consisting of a 3D T1-weighted pulse sequence [repetition time (TR), 2.63ms; echo time (TE), 2.6; flip angle, 9º; 256 x 256 acquisition matrix; field of view (FOV), 256 x 192; 160 oblique axial slices; voxel size,1.0 x 1.0 x 1.0 mm]. Twenty-eight oblique axial slices with full brain coverage were obtained [TR, 2s; TE, 30; flip angle, 70º; 64 x 64 acquisition matrix; FOV, 200 x 200; voxel size, 3.125mm x 3.125mm x 5.0

mm] using T2*-weighted echo-planar image (EPI) sequence. Oblique axial slices were acquired to minimize sinus-related artefacts that occur in MRI sequences. These oblique axial slices were restored to the normal axial plane during reconstruction procedures (will be discussed below).

3.2.6 Data Analysis

Brain images from the Siemens 3T scanner were stored in Digital Imaging and Communications in Medicine (DICOM) format. The DICOM file is comprised of a header and image data. The header stores patient information, the type of scan, dimension of the image etc. while the image data contains the three-dimensional information. The coordinate space coded in DICOM files uses the radiological convention – left side of the image is the right side of the brain. In the current experiment, images were transformed from DICOM to ANALYZE format using LONI Debabeler software (available at http://www.loni.ucla.edu/Software/). LONI Debabeler converts images from radiological to neurological convention – left side of the image is the left side of the brain. Also, the ANALYZE format is different from DICOM in that it contains header and image information in two separate files. Statistical Parametric Mapping (SPM5; http://www.fil.ion.ucl.ac.uk/spm/, Frackowiak et al., 1995) package was used for the pre-processing of images. Data files in ANALYZE format are recognized by SPM5.

3.2.6.1 Pre-processing Pipeline.

The term pre-processing refers to computational procedures that reduce variability in fMRI data before these data can be statistically tested. Four standard procedures - slice acquisition timing correction, head motion correction, spatial normalization and spatial smoothing were used to prepare images for analysis. The descriptions of these pre-processing steps are summarized from Huettel, Song, and McCarthy (2004). An extra pre-processing step – independent

31

components analysis filtering – was inserted between motion correction and spatial normalization, the details of which will be discussed below.

Slice Acquisition Timing Correction. Brain slices that form a brain volume can be acquired from the bottom of the brain moving upward (ascending), from the top of the brain moving downward (descending) or interleaved (odd slices acquired first and then even slices). When data is acquired in an interleaved fashion, there are small differences in timing associated with acquiring the first slice and the second slice since these slices are acquired half a TR apart. Such small temporal differences can be corrected by estimating the hemodynamic response function (HRF) from neighbouring time points. fMRI images from the present study were acquired in an interleaved fashion and were corrected using default settings in SPM5.

Head Motion Correction. fMRI assumes that data collected from a voxel is in a fixed location in the brain. When participants’ move their heads, this location loses its uniqueness and therefore, the signal generated from a voxel cannot be attributed to a particular brain area. Motion correction tries to adjust the brain images so that the voxels are aligned to the same position in each image. This process is known as co-registration. I co-registered my images in each run to a reference volume – an image at the beginning. Standard SPM5 default settings were used for co-registration.

Independent Components Analysis Filtering. Noise-related artefacts such as residual head motion, MR system noise, and changes in the signal intensity acquired by the MR scanner are some of many factors that can contribute to increased variability in fMRI data. These artefacts however, can be identified easily because of their distinct spatial and temporal profiles. I used independent components decomposition (http://www.fmrib.ox.ac.uk/fsl/melodic/, Beckmann & Smith, 2004) to denoise my fMRI data.

Spatial Normalization. Once the data is free of timing differences and head motion-induced artefacts, the brain volumes need to be co-registered to an anatomical template. Each participant’s brain morphology can defer from the next participant’s therefore, it is imperative to align all participants to an averaged brain template. There are a variety of brain templates used by different researchers. A popular template created by the Montreal Neurological Institute (MNI) consists of 152 human brains averaged together to provide common anatomical reference points. The size and gross anatomical features of an individual participant are warped into a

32

common space shared by all participants in an experiment. Images in my study were normalized to the MNI template using a 9 parameter affine (linear) transformation with 4D-spline interpolation. The brain volumes were also re-sliced to obtain a voxel size of 4mm x 4mm x 4mm.

Spatial Smoothing. The last step in pre-processing fMRI data is to smooth brain images by using a Gaussian kernel that is larger than the acquired voxel size. Smoothing across voxels results in some spatial smearing but in general, provides localized clusters of activation in brain images. An 8 mm (twice the voxel size of my data) Gaussian kernel was used to smooth images.

All pre-processing steps were performed within a run for a participant to avoid overcorrection for between-run motion. However, all runs were concatenated into one run for statistical analysis.

3.2.6.2 Statistical Analysis.

Data analysis was conducted using Task Spatio-Temporal Partial Least Squares (Task ST- PLS; http://www.rotman-baycrest.on.ca/, McIntosh, Bookstein, Haxby, & Grady, 1996; McIntosh, Chau, & Protzner, 2004; McIntosh & Lobaugh, 2004). Task ST-PLS is a multivariate technique that delineates patterns of brain activity as a result of changing task demands. The task demands in the current experiment were cue/target conditions as well as sensory modality. Task ST-PLS is explained further below.

Task Spatio-Temporal PLS. Task ST-PLS identifies patterns of brain voxels whose signal change covaries with the experimental conditions in the same way. This multivariate approach is similar to a canonical correlation analysis with some modifications. Task ST-PLS assumes that brain function emphasizes the distributed activity of the brain rather than the independent activity of any single brain region. In this technique, all task conditions can be entered simultaneously into the analysis, thus facilitating the identification of common patterns of brain activity across conditions, as well as patterns unique to specific conditions. All data are inputted in matrix form where rows of the matrix are arranged as follows: condition blocks are stacked and each participant has a row of data within each block. With n participants and k conditions, there are n * k rows in the matrix. The signal intensity measure at each voxel and each time point is contained in the columns of the data matrix such that the first column is composed of the

33 signal intensity for the first voxel at the first timepoint and the second column codes intensity for the first voxel at the second timepoint and so forth. There are m * t columns in the matrix where m are the voxels and t are the timepoints. Each voxel is centred on the grand mean that is deviations from the grand mean are coded in the data matrix. The HRF for any given condition normally lasts for several scans; thus, a “lag-window” is defined as a short signal segment within a trial that represents the response of each voxel averaged across trials. In the current experiment, the lag-window size is 5 (TR = 2, 10 seconds), beginning at the onset of second stimulus in each trial. The HRF for each trial is expressed as the intensity difference from trial onset. The data matrix undergoes singular value decomposition (SVD) to yield a set of latent variables (LVs); spatio-temporal patterns reflecting cohesive brain activity related to the experimental design. SVD is a technique that is similar to principle components’ analysis (PCA) whereby an LV accounts for a proportion of the total variance in the data matrix. The LV is similar to loading on a factor in PCA. The primary difference between SVD and PCA is that each LV is mutually uncorrelated and explains an independent proportion of the variance whereas, the first factor in PCA accounts for the maximal total variance and the second factor accounts for the next maximal total variance and so forth.

Apart from the LVs, SVD also results in two types of scores – brain and design. Brain scores index how much a participant contributes to a particular LV. Design scores show a pattern of contrasts between experimental conditions. In the present study, the design scores represent “true effects” (data-driven) since there were no a priori contrasts assigned.

There are two levels of statistical analysis performed in PLS. The significance for each LV as a whole is determined by using a permutation test (McIntosh, Bookstein, Haxby, & Grady, 1996). The permutation test is performed on singular values which are the covariances between the brain and the design (set of experimental conditions). The purpose of conducting a permutation test is to evaluate whether an LV is significantly different from noise. The number of permutation performed is proportional to the precision of the alpha critical value. For the current experiment, LVs were considered significant when permuted 200 times which corresponds to an alpha critical of 0.02. Permutation tests usually reach asymptote by a 100 permutations.

34

In addition to the permutation test, a second and independent step is to determine the reliability of the saliences (or weights) for the brain voxels characterizing each pattern identified by the LVs. To do this, all data for each timepoint are submitted to a bootstrap estimation (calculated over 200 samples) of the standard errors (Efron & Tibshirani, 1986). Peak voxels with a salience/standard error ratio > 3.5 (99% confidence interval) were considered to be reliable (Sampson, Streissguth, Barr, & Bookstein, 1989). The number of bootstraps necessary to reach asymptote is usually around 50.

3.3 Results 3.3.1 Behavioural Performance

Mean reaction times across all conditions are displayed in Table 3.1. A 2 (cue modality: auditory, visual) x 2 (cue order: first, second) x 2 (compatibility: compatible, incompatible) repeated measures ANOVA was performed on reaction time measures. For simplicity, hand of response (right or left) was removed from the repeated measures design. I did find faster overall reaction times for the dominant hand, that is the right hand since all participants were right- handed. The main effects of compatibility (F(1,23) = 88.73, p < 0.001; see Figure 3.3A), cue-target

order (F(1,23) = 20.17, p < 0.001; see Figure 3.3B) and modality (F(1,23) = 9.35, p < 0.01; see Figure 3.3C) were significant. Participants responded faster when the cue was auditory than when it was visual. Reaction times were also facilitated when the cue was presented first as opposed to when the cue was presented second. Participants had shorter response latencies in compatible conditions when compared to incompatible conditions. The interactions between

modality and order (F(1,23) = 11.77, p < 0.001; see Figure 3.3D) and order by compatibility (F(1,23) = 8.46, p < 0.001; see Figure 3.3E) were significant. All other interactions were statistically non- significant.

All standard errors calculated for behavioural performance were adjusted for a within- subjects design in accordance with the method proposed by Loftus & Masson (1994). Average accuracy across all tasks and participants was at ceiling – 0.9976 ± 0.0014 (standard error of the mean).

35

3.3.2 fMRI Results

The first Task ST-PLS analysis included VC-AT and VT-AC tasks which were further categorized into compatible and incompatible conditions. The two tasks (VC-AT and VT-AC) selected in this Task ST-PLS analysis had the same stimulus presentation structure, visual stimulus presented first followed by an auditory stimulus. Therefore, any activation differences found were due to attentional cueing rather than stimulus presentation. In VC-AT and VT-AC, presentation of the auditory stimulus second ensured that the full extent of the BOLD response was captured in the middle of the temporal lag window. Two significant LVs (p < 0.001) showed different patterns across the four conditions. The first LV contrasted VC-AT from VT-AC conditions while the second LV dissociated compatible conditions from incompatible conditions. Since cueing effects were evident in the first LV, only this LV will be considered in the current discussion. The BOLD response in auditory areas (peak voxel MNI coordinates: -48.0, -80.0, 4.0 (xyz in mm)) showed a significant increase in BOLD percent signal change with respect to cue conditions but not target conditions (see Figure 3.4).

The percent signal change for the BOLD response was largest for compatible cue conditions. The compatible / incompatible difference found in the BOLD response was

statistically significant (F(1,23) = 48.35, p < 0.00). The correlation coefficient computed between reaction time and peak amplitude of the BOLD response for compatible conditions was -0.46 (significant at p < 0.05) and for incompatible conditions was -0.0036 (did not reach significance).

Apart from cue-induced changes in the BOLD response, I also found time-dependent brain activity changes. The AC-VT: AT-VC Task ST-PLS showed differences in the time course of activations following auditory cue onset (see Table 3.2 for list of local maxima). I chose these tasks where the auditory stimulus was presented first to look at the entire time course of brain activation patterns. Stimulus presentation and timing were held constant in both tasks. Dominant positive saliences (related to increased activation during AC-VT tasks) were located in auditory (BA 41/42) and visual (BA 17/18) cortices during the first six seconds (lags 1-3). Dominant negative saliences (related to increased activation during AT-VC conditions) were seen in visual cortices (BA 17/18) eight to ten seconds after the onset of the second stimulus (lags 4-5). The overall brain activity pattern suggested that auditory cortices were active at the same time as

36

visual cortices at early time intervals and this activation pattern was sustained for a few lags (see Figure 3.5).

The second Task ST-PLS analysis included AC-VT and AT-VC tasks. These conditions were subdivided into compatible and incompatible conditions. Both AC-VT and AT-VC tasks had the same stimulus presentation structure; auditory stimulus first followed by the visual stimulus. The consistent timing of the stimuli in both tasks ensured that activation differences were a result of cueing rather than stimulus presentation. These sets of tasks (auditory stimulus first) were used to centre the visual stimulus in the middle of the temporal lag window in order to obtain the full extent of the BOLD response. Two significant LVs (p < 0.001) distinguished each of the four conditions. The first LV contrasted AC-VT tasks from AT-VC tasks and the second LV differentiated compatible conditions from incompatible conditions. Only the first LV was of interest in determining cueing effects. Figure 3.6 shows the HRF for a peak visual voxel (MNI coordinates: -28.0, -92.0, 8.0 (xyz in mm) in each task. The results indicate a significant increase in BOLD percent signal change in cue conditions when compared to target conditions irrespective of stimulus-response compatibility. The enhancement of the BOLD response in the visual voxel was smaller in size than the magnitude of BOLD percent signal change in the auditory voxel.

As in the case of the auditory cue, I found differences in the time course of activations following the visual cue (see Table 3.3 for list of local maxima). The set of tasks used for this analysis were VC-AT and VT-AC where the visual stimulus was presented first. Dominant positive saliences (related to increased activation during VC-AT tasks) were observed in visual areas (BA 17/18) and the medial frontal gyrus (BA 10) during the first three lags with no activation in auditory cortices (BA 41/42) (see Figure 3.7). Dominant negative saliences (related to increased activation during VT-AC tasks) were seen in visual cortices (BA 17/18) two to four seconds post-stimulus onset followed by auditory cortices (BA 41/42) during lags 3-4.

3.4 Discussion

In the current experiment, I found that auditory cues facilitated reaction times to subsequent targets but visual cues did not. The counterbalanced design of the experiment enabled me to investigate all possible audio-visual interactions between cues and targets. Maintaining the same temporal pattern of stimulus presentation, I determined that when auditory and visual

37

stimuli were cues, they showed increased BOLD responses in comparison to when auditory and visual stimuli were targets. This enhancement of BOLD responses may be driven by both attentional mechanisms and sensory-specific ones because BOLD magnitude varied with modality but the effect was present across both modalities. I did not find any deactivations in response to cue conditions in my tasks.

There were some anomalies found in the fMRI data with regards to the auditory cue conditions. The magnitude of the BOLD response was higher in compatible conditions than in incompatible conditions. There was also a significant negative correlation between the amplitude of the BOLD response in compatible conditions with reaction times whereas this negative correlation was minimal for incompatible conditions. One probable explanation for this difference in the magnitude of the BOLD response and reaction time-BOLD coupling may have to do with how auditory cues are processed. Regions that facilitated compatible responding were usually sensory cortices which reflect the automatic route from perception to action (Hommel, 2005). The automatic route from perception to action refers to minimal sensorimotor transformations performed on initial sensory input in producing a motor output. On the contrary, incompatible responding followed a slightly more deliberate route because attention needed to be disengaged from the cued location and re-engaged at the target location before a response could be made (Hommel, 2005). There are more sensorimotor transformations performed in incompatible conditions. Differences in sensorimotor transformations required for compatible versus incompatible conditions could be one possible reason why there was a smaller increase in BOLD incompatible conditions than in compatible conditions. Nevertheless, both compatible and incompatible conditions related to the auditory cue showed a reliable, significant increase in BOLD responses when compared to compatible and incompatible auditory target conditions. Visual cue and target conditions showed no difference in the BOLD response between compatible and incompatible conditions. The distinction between automatic and strategic processing may not relate to visual processing and may be an exclusive property of auditory localization.

The fMRI results also revealed the spatio-temporal dynamics of auditory and visual cue processing. The results illustrated that when the auditory cue was presented first, auditory and visual cortices responded by increasing in activation and sustained this activation pattern for a few seconds before activation was visible in visual areas related to target processing. The tight

38

coupling between auditory and visual cortices at the onset of the auditory cue suggested that these regions may be co-activating. This was not observed in the visual cue conditions. Neuroanatomical studies conducted on macaque monkeys have determined that regions of the auditory cortex project directly to certain parts of the visual cortex (Falchier, Clavagnier, Barone, & Kennedy, 2002; Felleman and Van Essen, 1991; Rockland & Ojima, 2003). Rockland and Ojima (2003) injected tracers in the auditory cortex and labeled terminations in V1 and V2. Falchier and colleagues (2002) found that when tracers were injected in V1, they were retrogradely transported to neurons in parts of the auditory cortex. However, projections from the auditory cortex to the visual cortex were very sparse (Falchier, Clavagnier, Barone, & Kennedy, 2002; Felleman and Van Essen, 1991; Rockland & Ojima, 2003). Thus, it may be the case that the brain’s structural architecture allows the auditory cortex to send inputs to the visual cortex preparing the visual cortex for upcoming visual targets. This priming process may be responsible for speeded reaction times in the auditory cue tasks.

On the contrary, presentation of the visual cue first activated the visual cortex and parts of the middle and medial frontal gyrus (corresponding to BA 10). There was a weak activation in a small cluster in the left auditory cortex but this activation was not robust or reliable at the onset of the visual cue or during the inter-stimulus interval. Since there have been no major anatomical projections from the visual cortex to the auditory cortex documented to date, the visual areas may be forced to tap into other areas to integrate cue-target information. The results showed that the visual cortex did not prime the auditory cortex when a visual cue was presented but rather activated the anterior prefrontal cortex (BA 10). The anterior prefrontal cortex (BA 10) is referred to as the fronto-polar cortex and is involved in a variety of complex cognitive tasks including the Tower of London (Baker et al., 1996), the Wisconsin Card Sort Task (WCST: Berman et al., 1995; Goldberg et al., 1998; Nagahama et al., 1996) and other memory tasks that involve episodic retrieval (Cabeza et al., 1997; Nyberg, Cabeza, & Tulving, 1996). Christoff and Gabrieli (2000) argue that the fronto-polar cortex plays an important role in evaluating, monitoring and manipulating information held in working memory that is internally generated. The anatomical connections from different multisensory sites in the brain converge on this prefrontal area making it ideal for integrating sensorimotor information (Fuster, 1997). In my experiment, the visual cue may recruit the fronto-polar cortex to maintain and manipulate an internally generated percept of the cue to aid target processing. Perhaps crossmodal facilitation

39

involving the visual cue occurs via a separate pathway than the priming mechanism used by the auditory cue.

The differential pattern of activation found in my experiment with respect to the auditory and visual cues suggested that the modality of the cue interacts with different processing streams. The auditory cue elicits activation in auditory and visual cortex which in turn may facilitate reaction times to subsequent visual targets. My results are in line with Macaluso’s theory in that multisensory processing can occur across sensory cortices as is the case with the auditory cue. Activation of the fronto-polar cortex in response to the visual cue indicates that visual cue processing recruits higher-order cognitive areas that are capable of sensorimotor integration. Parts of the visual cortex appear to be co-active with the fronto-polar cortex demonstrating that areas outside the sensory cortices are also capable of modality-specific processing.

By using Task ST-PLS to delineate the spatial and temporal dynamics of the areas involved in auditory and visual cue processing, I was able to separate the contributions of supramodal and modality-specific processing in crossmodal cueing. The supramodal processing was primarily an attentional modulation of BOLD responses within sensory-specific cortices. Auditory and visual cues elicited higher BOLD responses than auditory and visual targets. Modality-specific processes were evident when the spatio-temporal profiles of sensory cortices were considered. Overall, these results emphasize that crossmodal facilitation can occur via multiple routes. My study supports a convergent model of audio-visual interactions where cue modality affects which processing stream is activated to produce a response.

40

Tables

Table 3.1: Mean Reaction Time by Condition.

Conditions Mean Reaction Time (ms) Standard Error (SE)

AC-VT_C 581.87 10.67

AC-VT_IC 645.92 9.04

VT-AC_C 682.48 10.71

VT-AC_IC 714.26 13.15

VC-AT_C 658.32 12.00

VC-AT_IC 718.21 13.26

AT-VC_C 691.00 10.72

AT-VC_IC 734.77 10.09

Note: AC/VC refers to auditory/visual cue; AT/VT corresponds to auditory/visual target and C/IC stand for compatible/incompatible conditions. The order of the cue and target is reflected in the naming (1st-2nd).

41

Table 3.2: Local maxima from AC-VT: AT-VC Task ST-PLS.

Lag x(mm) y(mm) z(mm) BSR Cluster size Region

(voxels)

Cue-Target LV

1 28 -96 0 6.8276 68 R GOm, BA 18

1 -36 -92 -4 5.5778 46 L GOm, BA 18

1 -44 -28 8 6.2148 37 L GTs, BA 41

1 52 -20 0 6.2094 115 R GTs, BA 22

1 44 -68 12 -4.9822 15 R GTm, BA 39

2 -12 -96 0 4.4898 14 L Cu, BA 17

2 24 -92 0 4.5934 39 R GOm, BA 18

2 60 -12 -8 5.6278 44 R GTm, BA 21

2 -44 -28 4 5.4953 31 L GTs, BA 22

2 -28 -48 56 -4.6140 20 L LPi, BA 40

2 40 -40 56 -4.4780 33 R GPoC, BA 2

2 -48 -20 36 -5.4302 31 L GPrC, BA 4/6

2 64 -12 28 -5.3233 84 R GPrC, BA 4

2 20 0 60 -4.3260 21 R GFm , BA6

3 -12 -96 0 6.0335 21 L Cu, BA 17

3 -52 -36 40 -5.6991 25 L LPi, BA 40

42

3 -44 -16 32 -5.1842 25 L GPrC, BA4/6

3 56 -4 32 -4.6209 43 R GPrC, BA 6

4 44 -64 8 5.0358 31 R GTm, BA 39

4 60 -52 -4 4.4301 12 R GTm, BA 37

4 36 -88 -8 -7.9365 77 R GOi, BA 18

4 -28 -96 -8 -6.9297 52 L GOi, BA 18

4 -40 -52 -20 -5.5071 12 L GF, BA 37

4 52 -56 -16 -4.2459 12 R GTi, BA 37

4 -48 -20 32 -4.7645 11 L GPoC, BA 1/2/3/4

4 52 0 28 -4.6530 20 R GPrC, BA 6

5 32 -92 0 -9.1891 106 R GOm, BA 18

5 -28 -104 0 -5.6607 40 L Cu, BA 17

5 -48 -36 44 -4.9904 12 L LPi, BA 40

Note: Lag is the period in TRs (TR = 2 s) after the onset of the second stimulus during which the peak occurred. X, Y, and Z refer to MNI voxel coordinates. BSR is the bootstrap ratio which represents Task ST-PLS parameter estimate divided by standard error. The cluster size is the number of contiguous voxels included in the cluster. Region shows the location and its corresponding Brodmann Area as referenced by Talairach and Tournoux (1988).

43

Table 3.3: Local maxima from VC-AT: VT-AC Task ST-PLS

Lag x(mm) y(mm) z(mm) BSR Cluster size Region

(voxels)

Cue-Target LV

1 -32 -96 -8 6.9677 55 L GOi, BA 18

1 60 0 -8 5.9230 16 R GTm, BA 21

1 64 -8 0 4.8534 11 R GTs, BA 22

1 12 -92 0 -7.2106 19 R GL, BA 17

1 -8 -96 0 -7.0568 74 L Cu, BA 17

1 -44 -72 20 -6.6647 25 L GTm, BA 39

1 -32 -16 56 -5.3227 11 L GPrC, BA 4

1 -16 -24 16 -5.0443 15 L Th (pulvinar)

2 -28 -100 0 7.1796 99 L GOm, BA 18/19

2 36 -88 -12 6.8608 124 R GOi, BA 18

2 -48 -64 16 -4.8764 15 L GTs, BA 39

2 32 -24 12 -5.0981 18 R INS, BA 13

2 12 -28 60 -5.3298 32 R LPc, BA 7

2 12 -8 48 -5.3704 11 R GC, BA 24

2 -32 -20 56 -5.9221 84 L GPrC, BA 4

2 40 -8 56 -6.5354 134 R GPrC, BA 4

44

2 12 44 20 -6.2351 11 R GFd, BA 9

3 -40 -88 -8 6.0282 57 L GOi, BA 18

3 40 -88 0 5.4601 122 R GOm, BA 19

3 32 56 16 5.7698 14 R GFm, BA 10

3 32 48 28 5.5572 15 R GFs, BA 10

3 52 -16 0 -8.4339 178 R GTs, BA 21/22

3 -48 -20 4 -7.1391 94 L GTs, BA 22

3 -52 -20 32 -5.0007 26 L GPoC, BA 1/2/3/4

3 40 -28 52 -5.2226 29 R GPoC, BA 3

3 -32 -24 52 -5.3656 21 L GPrC, BA 4

3 52 -12 40 -4.6104 13 R GPrC, BA 6

3 -52 0 20 -5.1408 16 L GFi, BA 44

4 60 -12 -8 -8.2228 242 R GTm, BA 21

4 -52 -20 0 -7.4800 209 L GTs, BA 21/22

4 -56 -16 40 -5.2214 57 L GPrC, BA 4

4 -48 16 16 -5.5414 14 L GFi, BA 45

5 0 56 8 4.5672 12 GL, BA 17/18

5 64 -12 -8 -4.8240 11 R GTm, BA 21

5 60 -28 4 -6.4331 34 R GTs, BA 22

5 -56 -24 4 -5.2470 46 L GTs, BA 22

45

5 -52 -16 48 -4.7982 18 L GPoC, BA 3

5 -44 28 8 -4.1253 13 L GFi, BA 45

5 40 -20 -12 -4.7268 11 R hippocampus

5 0 -32 -36 -4.6061 16 R brainstem (pons)

Note: Lag is the period in TRs (TR = 2 s) after the onset of the second stimulus during which the peak occurred. X, Y, and Z refer to MNI voxel coordinates. BSR is the bootstrap ratio which represents Task ST-PLS parameter estimate divided by standard error. The cluster size is the number of contiguous voxels included in the cluster. Region shows the location and its corresponding Brodmann Area as referenced by Talairach and Tournoux (1988).

46

Figures

Figure 3.1. Experimental design schematic for auditory cue-visual target tasks.

The first stimulus (S1) is presented followed by the inter-stimulus interval (ISI) and the second stimulus (S2) presentation. A button-press response is made after S2. Trials are jittered at variable intervals to optimize blood oxygenation-level dependent (BOLD) signals. In AC-VT, the auditory cue is presented first and the visual target is presented second. In VT-AC, the visual target is presented first followed by the auditory cue. Timing and general structure of a trial is identical for AC-VT and VT-AC tasks. Compatible and incompatible trials are randomly presented within each task.

47

Figure 3.1

48

Figure 3.2. Experimental design schematic for visual cue-auditory target tasks.

The first stimulus (S1) is presented followed by the inter-stimulus interval (ISI) and the second stimulus (S2) presentation. A button-press response is made after S2. Trials are jittered at variable intervals to optimize blood oxygenation-level dependent (BOLD) signals. In VC-AT, the visual cue is presented first followed by auditory target presentation. In AT-VC, the auditory target is presented first and the visual cue follows. Timing and general structure of a given trial is identical for VC-AT and AT-VC tasks. Compatible and incompatible trials are randomly presented within each task.

49

Figure 3.2

50

Figure 3.3. Behaviour measures for all four experimental tasks.

The bar-graphs represent the mean reaction times in milliseconds for five significant behavioural effects. The first column (A-C) shows the main task effects. A. Compatible vs. Incompatible. B. Order manipulation. C. Cue modality. The second column (D-E) indicates the interactions. D. Modality by order. E. Order by compatibility.

51 Figure 3.3

52

Figure 3.4. BOLD HRFs for cue and target – auditory modality.

The displayed brain image is obtained from Task ST-PLS thresholded with a bootstrap ratio of 3.5. The colour yellow shows increased activation in the auditory cortex. The plot shows the BOLD percent signal change averaged across a cluster of three voxels in primary auditory cortex that respond to the auditory cue being presented as the second stimulus. The lines in blue represent auditory cue (VT-AC) conditions and the lines in red/orange represent auditory target (VC-AT) conditions. Compatible and incompatible conditions are kept separate and are specified by a C or an IC at the end of the task name, respectively. The y-axis indicates the magnitude of BOLD percent signal change and the x-axis shows the progression of time in TRs from the onset of the first stimulus. Stimulus timing is presented at the bottom of the figure – visual stimuli were presented first followed by auditory stimuli. The errors bars are calculated as standard errors about the mean.

53

Figure 3.4

54

Figure 3.5. Singular image and design scores differentiating cue from target for auditory modality.

The brain images on the left represent activation patterns from AC-VT: AT-VC Task ST- PLS. In these tasks, the auditory stimulus is presented first followed by the visual stimulus (this timing is shown on the upper right corner). The only parameter that is varied in this analysis is the property of the stimuli (cue or target) while presentation timing is kept constant. The y-axis corresponds to time in seconds and the x-axis shows the equivalent axial slice in Talairach space. Voxels in the image are highlighted according to the magnitude of the ratio of their parameter estimate to the bootstrap- estimated standard error (bootstrap ratio) of 3.5. The singular image is superimposed on a T1-weighted MRI template. The yellow areas show an increase in activation for cue conditions while the blue areas show an increase in activation for target conditions. The design scores that correspond to the contrast are presented on the bottom right. Compatible and incompatible conditions are denoted by the letters C and IC, respectively.

55

Figure 3.5

56

Figure 3.6. BOLD HRFs for cue and target – visual modality.

The displayed brain image is obtained from Task ST-PLS thresholded with a bootstrap ratio of 3.5. The colour yellow shows increased activation in the visual cortex. The plot shows the BOLD percent signal change averaged across a cluster of three voxels in visual cortex that respond to the visual cue being presented as the second stimulus. The lines in blue represent visual cue (AT-VC) conditions and the lines in red/orange represent visual target (AC-VT) conditions. Compatible and incompatible conditions are kept separate and are specified by a C or an IC at the end of the task name, respectively. The y-axis indicates the magnitude of BOLD percent signal change and the x-axis shows the progression of time in TRs from the onset of the first stimulus. Stimulus timing is presented at the bottom of the figure – auditory stimuli were presented first followed by visual stimuli. The errors bars are calculated as standard errors about the mean.

57

Figure 3.6

58

Figure 3.7. Singular image and design scores differentiating cue from target for visual modality.

The brain images on the left represent activation patterns from VC-AT: VT-AC Task ST- PLS. In these tasks, the visual stimulus is presented first followed by the auditory stimulus (this timing is shown on the upper right corner). The only parameter that is varied in this analysis is the property of the stimuli (cue or target) while presentation timing is kept constant. The y-axis corresponds to time in seconds and the x-axis shows the equivalent axial slice in Talairach space. Voxels in the image are highlighted according to the magnitude of the ratio of their parameter estimate to the bootstrap- estimated standard error (bootstrap ratio) of 3.5. The singular image is superimposed on a T1-weighted MRI template. The yellow areas show an increase in activation for cue conditions while the blue areas show an increase in activation for target conditions. The design scores that correspond to the contrast are presented on the bottom right. Compatible and incompatible conditions are denoted by the letters C and IC, respectively.

59

Figure 3.7

60

Chapter 4: The Interplay of Cue Modality and Response Latency in Neural Networks Supporting Crossmodal Facilitation

4.1 Introduction

Behavioural analyses on the crossmodal cueing tasks (see Chapter 3) showed an enhanced facilitation of reaction time for the auditory cue in comparison to the visual cue. Previously, I demonstrated that this asymmetry may be influenced by different spatio-temporal dynamics of regions involved in processing cues that are dependent on input modality. While I classified differences in activation in spatial and temporal domains, most fMRI studies only consider spatial distributions of brain activity at the onset of a given task. Regions classified by such fMRI studies may be involved in various stages of cue processing, such as initial sensory processing, translating the cue into a meaningful neural representation, transforming representations into motor outcomes that include orienting to a particular location, maintaining sensorimotor transformations across cue-target intervals and preparing and executing a response (Giesbrecht et al., 2006). Only a subset of regions that are initially involved in cue processing may be responsible for producing a behavioural response. In order to dissociate brain areas that respond to initial cue processing from areas that mediate behaviour, I can use behavioural spatio- temporal PLS (b ST-PLS). This technique correlates brain activity with reaction time across participants. b ST-PLS outputs patterns of brain-behaviour correlations that change as a result of input modality and task type (cue or target). Regions that work in concert to facilitate reaction time (brain activity negatively correlates with reaction time) can be easily distinguished from regions that may hinder performance (brain activity positively correlates with reaction time).

Computing brain-behaviour correlations to understand how crossmodal facilitation leads to faster responses is an area that is seldom explored. Experiments to date have shown only brain activation patterns that correspond to attentional processing and multisensory integration. These studies have identified a specific fronto-parietal network that may be involved in attentional cue processing (Corbetta & Shulman, 2002; Hopfinger, Buonocore, & Mangun, 2000). Hopfinger and others (2000) presented participants with an arrow cue which pointed to a peripheral location. Participants had to attend to the peripheral location to complete a target discrimination task. The arrow cue always pointed towards the correct peripheral location; that is the cue was a hundred percent valid. The fMRI results for the study showed greater activations for the cue

61

compared to the target in superior frontal, superior and inferior parietal, superior temporal and posterior cingulate cortices. The centrally presented cue also activated extrastriate regions such as V4 before the onset of the target stimulus. Hopfinger and colleagues (2000) claimed that a fronto-parietal network may be modulating cue processing in a top-down fashion. Kastner and colleagues (1999) have also suggested that top-down processes may increase excitability of neurons that are specialized for processing an upcoming target.

The findings of Hopfinger and colleagues (2000) would allude to the idea that if an auditory cue was presented instead of a visual cue, the fronto-parietal network would be activated since this network modulates cue processing in a top-down manner. Nevertheless, there has been a repertoire of studies that have shown that auditory (Alain et al., 2001; Belin & Zatorre, 2000) and visual systems (Ungerleider & Mishkin, 1992) are organized into separate processing pathways in the brain. The dorsal stream originates from the primary visual cortex and projects to the posterior parietal cortex while the ventral stream starts in the primary visual cortex and projects to the infero-temporal cortex. The two streams carry out different functions with the dorsal stream mediating spatial processing (referred to as the “where” pathway) and the ventral stream guiding object recognition (known as the “what” pathway). Goodale and Milner (1992) suggested that this “what” / “where” classification proposed by Ungerleider and Mishkin using the macaque brain should actually be a “what” / “how” classification in the . According to Goodale and Milner (1992), the dorsal stream mediates visual control of action directed at objects that are represented by the ventral stream. The authors claim that the dorsal stream is responsible for action while the ventral stream is more perceptual. How the fronto- parietal network interacts with dorsal and ventral streams and how it relates to behavioural outcomes are questions that have not been answered.

My goals in the present study are to correlate brain activation with individual differences in reaction time to ascertain the relationship brain patterns that correspond to processing task- related information (cue-driven input processing) from brain regions involved in coordinating behaviour (output). In addition, I stipulate that crossmodal facilitation may occur as a result of interactions between distinct functional networks that are differentially engaged with respect to cue modality.

62

4.2 Methods 4.1.1 Data Analysis

Data analysis was conducted using behavioural spatio-temporal Partial Least Squares (b ST-PLS; www.rotman-baycrest.on.ca, McIntosh, Bookstein, Haxby, & Grady, 1996; McIntosh, Chau, & Protzner, 2004; McIntosh & Lobaugh, 2004) which identifies brain-behaviour correlations to determine areas that are negatively or positively correlated in activity with reaction time measures in cue/target conditions and across different sensory modalities. b ST- PLS is explained in the next section.

Behavioural ST-PLS. b ST-PLS identifies LVs that show changes in brain activity that are correlated with reaction time. The reaction time correlations are computed across participants within each task resulting in within-task brain-behaviour correlations. Singular value decomposition is then performed on the brain-behaviour correlation matrix to produce three matrices: voxel saliences, task saliences and singular values. Task salience variations indicate whether a given LV shows a similar or different brain-behaviour correlation pattern across tasks. The correlations are computed between brain scores, which correspond to a dot-product of voxel saliences and fMRI data, and reaction times for each task and participant. The spatio-temporal activity pattern is captured by voxel saliences which are displayed as a singular image. The singular image shows voxels that are weighted in proportion to the strength and direction (positive or negative) of their brain-behaviour correlation.

The reaction times that are used for behavioural ST-PLS are expressed as z-scores from each participant's mean and standard deviation computed for all experimental conditions. The z- score transformations provide reaction time measures without being influenced by large differences in mean reaction time across participants (Ben-Shakhar, 1985). The correlations between these z-scores and the tasks-dependent fMRI data were computed for all participants and served as the input for behavioural ST-PLS.

Statistical assessments for behavioural ST-PLS are similar to task ST-PLS. Permutation tests are performed on LVs to determine if the LVs are significantly different from random noise. Bootstrap estimates of standard errors for the voxel saliences evaluate the reliability of the non- zero saliences on significant LVs.

63

4.3 Results 4.1.2 Behavioural Performance

A 2 (cue modality: auditory, visual) x 2 (cue order: first, second) x 2 (compatibility: compatible, incompatible) repeated measures ANOVA was performed on reaction time data (see Chapter 3 for details). The significant effect of auditory cues eliciting faster reaction times compared to visual cues will be the considered in this part of the experiment.

4.1.3 fMRI Results

The first set of brain-behaviour correlations were computed using b ST-PLS for AC-VT and AT-VC tasks to determine areas that facilitate reaction times for an auditory cue in comparison to an auditory target (see Table 4.1 for list of local maxima). The two tasks (AC-VT and AT-VC) selected in this PLS analysis had the same stimulus presentation structure, auditory stimulus presented first followed by a visual stimulus. Therefore, any brain-behaviour differences found were related to cue-target processing rather than stimulus presentation. The overall correlation pattern of brain scores and reaction times showed significant negative correlations with cue conditions (Figure 4.1). There were no outliers in terms of brain scores (participants contributed to particular brain patterns consistently) or reaction times (all participants were within two standard deviations of the mean reaction time). Relationship between brain scores and reaction times are illustrated in Figure 4.2. The correlation pattern was expressed in the brain in the form of dominant positive and negative saliences. Dominant positive weights (related to faster reaction times in cue conditions) were found in medial visual areas (BA 18/19), brainstem, posterior parts of the cerebellum, posterior cingulate (BA 30), inferior parietal cortex (BA 40) and frontal areas (see Figure 4.3). Different parts of the frontal cortex were correlated with reaction time early and late in stimulus processing. While middle frontal areas (BA 9/10) were negatively correlated with reaction time early in stimulus processing, inferior frontal areas (BA 45) were negatively correlated with reaction time later in stimulus processing. I only considered the subset of regions that were involved in reaction time facilitation. There were no regions that corresponded to delays in reaction time however; there were a set of regions that were negatively correlated with target processing (areas showing dominant negative weights that is brain activity indexed by colours in the blue-cyan range).

64

These regions were not considered because target-driven brain activity is not a topic that has been explored in this dissertation.

Brain-behaviour correlations for the visual cue compared to the visual target were calculated using VC-AT and VT-AC tasks (see Table 4.2 for list of local maxima). In these tasks, the visual stimulus was presented first followed by the auditory stimulus. A similar pattern of correlations was observed for the visual cue with cue conditions negatively correlating with reaction time (Figure 4.4). No outliers were found for brain scores or reaction times (see Figure 4.5). Brain areas that promoted reaction time facilitation included lateral visual areas (BA 19/37), claustrum, and anterior and posterior parts of the cerebellum (showed dominant positive weights, see Figure 4.6). Reaction time was negatively correlated with activity in the hippocampal region early in stimulus processing.

4.4 Discussion

I computed brain-behaviour correlations to identify patterns of brain activation that corresponded to facilitated or delayed reaction time measures. In the case of the auditory cue, speeded reaction times were correlated with activity in primary and secondary visual areas (BA 17/18/19), the posterior cingulate, inferior parietal cortex (BA 40), middle frontal areas such as the dorso-lateral prefrontal area (DLPFC: BA 9) and fronto-polar cortex (BA 10) and inferior frontal areas (BA 45). I found a temporal dissociation between the activation of middle versus inferior frontal sites. The middle frontal areas were correlated negatively with reaction time early in stimulus processing, possibly because of their interactions with the posterior cingulate and parietal areas. It has been shown by Small and colleagues (2003) that activation in the posterior cingulate and middle frontal areas is correlated with reaction time facilitation in valid cue trials. The authors suggest that interactions of the posterior cingulate with middle frontal sites and reaction time may be a result of anticipatory allocation of spatial attention. Reaction time was negatively correlated with inferior frontal areas late in stimulus processing. Activation in the inferior frontal cortex has been reported when participants have to choose task-relevant representations from among other competing alternatives (Thomson-Schill, D’Esposito, & Kan, 1999). In my experiment, the middle frontal sites may initially be required to allocate attention and then activity within inferior frontal areas would allow appropriate response selection at the time of target presentation. The inferior parietal cortex (BA 40) was also negatively correlated

65

with reaction time. This region is capable of deploying spatial attention and integrating crossmodal information (Bushara et al., 1999). The similar correlation profiles shared by medial visual areas and the fronto-parietal network suggest that these areas may be important in modulating auditory cue-driven behaviour. The fronto-parietal network is known for its role in visuo-spatial attention and multisensory integration in many studies (Bremmer et al., 2001; Corbetta & Shulman, 2002; Driver & Spence, 1998; Hopfinger, Buonocore, & Mangun, 2000; Kastner et al., 1999; Rosen et al., 1999; Shulman et al., 2002) nevertheless; my results suggest that auditory cueing can also recruit this network when processing cue-relevant information.

The results from Task ST-PLS (see Chapter 3) showed that auditory cortices may impact visual areas. By considering brain-behaviour correlations, I can juxtapose that these visual areas may interact with parts of the fronto-parietal network to produce facilitated reaction times. Electrophysiological evidence suggests that medial visual areas do interact with fronto-parietal networks (Foxe & Simpson, 2002). Foxe and Simpson (2002) conducted an experiment to explore the time frame for the initial trajectory of activation flow along dorsal and ventral visual processing streams. The authors found that high density electrical mapping showed that the dorsal stream areas are active before ventral stream areas and that primary visual areas (such as V1) feed into dorsal stream areas which include frontal and parietal sites.

Brain-behaviour correlations for the visual cue implicated the involvement of a posterior network in facilitating reaction time. Areas such as the lateral occipito-temporal region (BA 19/37), claustrum, cerebellum, and hippocampal region (BA 36) were negatively correlated with reaction time. The lateral occipital complex (LOC) has been traditionally associated with complex visual object processing (Grill-Spector et al., 1998; Malach et al., 1995). Moreover, there have been studies that implicate the LOC in multisensory integration (Amedi et al., 2001; James et al., 2002). There have been no studies thus far that have found a direct relationship between LOC activation, crossmodal integration and attentional cue processing. In my experiment, LOC involvement may be crucial to encoding the salient, complex properties (such as shape and orientation) of the visual cue stimulus. Apart from the LOC, the claustrum may also play a role in multisensory convergence; showing greater activation in a crossmodal shape matching task compared to intra-modal conditions (Hadjikhani & Roland, 1998). However, anatomical localization of the claustrum using standard coordinate taxonomy may not be precise since it is a relatively small region in the brain. The cerebellum was another region that was

66

correlated with speeded reaction times. Activation in the cerebellum has been associated with preparation of motor plans in visuo-motor imagery tasks (Deiber et al., 1998).

The hippocampal region (BA 36) is often classified as a memory structure that is capable of binding events with their contexts, a form of relational memory (see review by Eichenbaum, 2004). Some researchers have also argued for the hippocampus’ role in integrating sensorimotor information (Bast & Feldon, 2003; Tesche & Karhu, 1999). Tesche and Karhu (1999) suggest that the hippocampus is recruited by large-scale cognitive networks to keep sensorimotor information online in order to make a speeded response. In rats, Bland and Oddie (2001), Dypvik and Bland (2004), and Hallworth and Bland (2004) have found that inputs from the red nucleus in the brainstem and parts of the basal ganglia converge on parts of the hippocampus for sensorimotor integration. These researchers claim that the hippocampus is in an ideal position to provide voluntary motor systems with feedback on performance relative to changing sensory conditions. I can extrapolate the evidence to date about the hippocampal region’s involvement in sensorimotor integration to my results by suggesting that in visual cue tasks, the hippocampal region helps maintain mental representations of the visual cue in light of changing task demands. The cerebellum could also be interacting with the hippocampus in planning motor actions after the LOC has processed salient aspects of the cue.

The different brain-behaviour correlation profiles of the auditory cue compared to the visual cue may be indicative of the difference in reaction times across the two cue modalities. The results show that the auditory cue is able to engage anterior and posterior sites; that is the fronto-parietal network, and interacts with the dorsal stream, to produce speeded reaction times. In contrast, the visual cue is only able to recruit posterior sites which may explain less reaction time facilitation for visual cues when compared to auditory cues. It may be the case that producing fast reaction times requires the contribution of fronto-parietal sites which are not fully engaged in visual cue tasks. In explaining my results in light of Hopfinger et al.’s findings, I suggest that the involvement of frontal areas in visual cue tasks may be specialized for early cue processing as seen in task-dependent activation in the fronto-polar cortex for the visual cue (see Chapter 3). However, activations in frontal areas while reflective of visual cue processing may not contribute to producing behavioural responses. Perhaps the interaction of the posterior network with the ventral stream which shows a delay in object processing (Foxe & Simpson,

67

2002) in comparison to the dorsal stream may also contribute to delayed reaction times in the case of the visual cue.

Overall, these results emphasize that sensorimotor integration can occur via multiple routes contingent on the identity of the cue modality. Sensory-specific cortices, supramodal sites such as the fronto-parietal network and different processing streams (dorsal and ventral) can all contribute to a particular behavioural profile.

68

Tables

Table 4.1: Local maxima from AC-VT: AT-VC Behavioural ST-PLS

Lag x(mm) y(mm) z(mm) BSR Cluster size Region

(voxels)

Cue-Target LV

1 -4 -80 12 5.6899 93 L Cu BA 18

1 20 -96 20 4.1988 12 R Cu, BA 19

1 -8 -100 12 3.9760 15 L GOm, BA 18

1 4 -72 24 6.6291 28 R PCu, BA 31

1 52 -56 36 3.2869 14 R Gsm, BA 40

1 56 -12 44 4.2583 12 R GPrC, BA 4/6

1 -48 12 36 3.9835 16 L GFm, BA 9

1 40 48 0 3.8207 11 R GFm, BA 10

1 52 24 32 3.6588 14 R GFm, BA 9

1 8 48 44 4.4151 35 R GFs, BA 8

1 32 -100 -4 -5.3068 18 R GOm

1 44 -4 -16 -3.6387 13 R GTm, BA 21

1 12 -48 -20 -4.1784 11 R cerebellum (ant)

1 36 -24 0 -3.9091 24 R Th

69

2 0 -92 32 4.5253 65 Cu, BA 19

2 16 -100 20 4.3788 14 R Cu, BA 19

2 16 -72 4 3.9254 17 R Cu, BA 17

2 4 -60 4 3.8961 35 R GC (post), BA 30

2 -48 -48 44 3.5129 11 L LPi, BA 40

2 4 40 44 4.0533 17 R GFd, BA 8

2 -40 -68 16 -5.6613 19 L GTm, BA 39

2 44 -28 4 -3.9903 12 R GTs, BA 22

2 16 -28 36 -4.6087 11 R GC, BA 24 /32

2 32 32 0 -4.4895 28 R GFi, BA 47

2 44 12 4 -4.2962 25 R INS, BA 13

2 -28 -12 -28 -3.9492 12 L Gh, BA 35/36

3 4 -68 36 3.7249 14 R PCu, BA 7

3 24 -56 44 -3.9761 12 R PCu, BA 7

3 40 32 -4 -5.1861 22 R GFi, BA 47

3 -40 4 -8 -4.4290 12 L INS (post), BA 13

3 -16 -28 20 -4.3067 15 L caudate (tail)

4 -44 -44 36 3.9284 16 L Gsm, BA 40

4 16 -64 -40 3.704 15 R cerebellum (post)

4 40 32 -4 -4.9318 11 R GFi, BA 47

70

4 -28 64 16 -4.6129 23 L GFm, BA 10

4 -28 12 52 -3.9842 11 L GFm, BA 6

5 -60 -12 -32 4.9460 13 L GF, BA 20

5 -68 -36 4 4.8408 32 L GTm, BA 22

5 -60 -44 -16 3.9506 13 L GTm, BA 20

5 36 -36 28 3.8360 17 R LPi, BA 40

5 4 -12 48 3.8113 11 R LPc

5 -64 12 8 5.5292 18 L GPrC, BA 6

5 36 4 28 4.5921 24 R GFi, BA 9

5 -36 32 8 5.7928 31 L GFi, BA 45

5 4 -32 -40 5.7581 13 R brainstem (pons)

5 -24 -56 -40 3.8774 14 L cerebellum (post)

5 12 -72 -20 5.0182 14 R cerebellum (post)

5 36 32 40 -5.0850 18 R GFm, BA 9

5 -36 56 20 -4.1808 18 L GFs, BA 10

Note: Lag is the period in TRs (TR = 2 s) after the onset of the second stimulus during which the peak occurred. X, Y, and Z refer to MNI voxel coordinates. BSR is the bootstrap ratio which represents Task ST-PLS parameter estimate divided by standard error. The cluster size is the number of contiguous voxels included in the cluster. Region shows the location and its corresponding Brodmann Area as referenced by Talairach and Tournoux (1988).

71

Table 4.2: Local maxima from VC-AT: VT-AC Behavioural ST-PLS

Lag x(mm) y(mm) z(mm) BSR Cluster size Region

(voxels)

Cue-Target LV

1 44 -60 -12 5.5828 30 R GOm / GF, BA 37/19

1 -40 -60 -4 4.3632 13 L GOm / GTi, BA 37/19

1 52 -4 -8 5.2033 22 R GTs, BA 38

1 20 -40 -24 5.0860 37 R cerebellum (ant)

1 -16 -76 -44 4.8517 11 L cerebellum (post)

1 -32 -32 -16 3.6921 14 L Gh, BA 36

1 -40 -84 24 -4.8411 13 L GOs, BA 19

1 -44 -20 28 -4.9123 23 L GPoC, BA 1/2/3/4

1 36 4 12 -4.4243 13 R INS, BA 13

1 -60 -44 28 -4.1852 12 L LPi, BA 40

1 0 -40 -52 -4.4293 13 L brainstem (medulla)

2 -12 -84 4 3.4006 12 L Cu, BA 17

2 44 -60 -12 6.2031 53 R GOm, BA 37/19

2 -24 -84 20 3.7091 11 L GOm / Cu, BA 18/19

2 -32 -96 -16 5.7588 12 L GOi, BA 18

72

2 32 -84 -20 6.0081 30 R GF, BA 19

2 28 -48 -24 3.9617 11 R cerebellum (ant)

2 56 12 0 -3.5465 14 R GTs, BA 22

2 -60 -44 28 -4.4747 37 L LPi, BA 40

2 -56 4 20 -4.3442 11 L GFi, BA 44

2 -44 16 16 -4.0443 12 L GFi, BA 44/45

2 8 -48 -40 -4.6353 14 R cerebellum (post)

3 36 -88 -20 3.6590 12 R GOi, BA 19

3 -32 -92 -12 5.7223 19 L GOi, BA 18

3 -24 -92 12 3.8850 15 L GOm, BA 19

3 -32 -64 56 4.7123 21 L LPs, BA 7

3 8 -60 -16 4.1184 12 R cerebellum (post)

3 -60 -40 -8 -5.8941 18 L GTm, BA 21

3 56 4 4 -4.7897 20 R GTs, BA 22

3 -44 48 -16 -5.8737 67 L GFm, BA 11

3 -56 -36 36 -4.0342 16 L LPi, BA 40

3 -64 16 12 -4.8674 12 L GFi, BA 44

3 20 -16 72 -4.6748 12 R GFs, BA 6

3 -36 -4 8 -4.3914 11 L INS, BA 13

4 -16 -96 4 5.1237 19 L Cu, BA 18

73

4 -32 -92 -16 3.7892 14 L GOi, BA 18

4 52 -12 -24 4.3006 12 R GTi, BA 20

4 -68 -32 8 4.4919 17 L GTs, BA 42

4 -32 -64 56 4.6935 42 L LPs, BA 7

4 8 -64 52 4.3502 13 R PCu, BA 7

4 36 -12 -8 4.7755 16 R Cl

4 -16 16 -20 4.0934 12 L GFi, BA 47

4 -56 -44 -8 -4.9660 32 L GTm, BA 21

4 0 -40 44 -6.4183 30 GC, BA 31

4 -8 36 56 -4.3438 17 L GFs, BA 8

4 -44 40 -24 -5.1050 105 L GFm, BA 11

4 -36 0 56 -4.5083 12 L GFm, BA 6

4 28 36 -20 -4.8231 11 R GFi, BA 47

4 32 -32 -20 -3.8373 25 R Gh, BA 36

4 20 -88 -28 -5.1512 12 R cerebellum (post)

5 -68 -32 8 7.8235 47 L GTs, BA 22/42

5 24 -72 52 4.6070 32 R PCu, BA 7

5 -48 -76 36 -5.1583 18 L Ga, BA 39

5 44 -4 -40 -4.7507 16 R GTi, BA 20

5 12 -40 48 -4.3657 50 R PCu, BA 7

74

5 -8 -76 28 -4.2362 33 L PCu, BA 31

5 24 -24 80 -4.5527 15 R GPrC, BA 4

5 -16 -32 72 -3.8635 25 L GPrC, BA 4

5 -44 48 0 -3.7210 20 L GFi, BA 10

5 -36 36 -20 -5.1333 28 R GFi, BA 47

5 -8 32 52 -3.4512 17 L GFs, BA 8

Note: Lag is the period in TRs (TR = 2 s) after the onset of the second stimulus during which the peak occurred. X, Y, and Z refer to MNI voxel coordinates. BSR is the bootstrap ratio which represents Task ST-PLS parameter estimate divided by standard error. The cluster size is the number of contiguous voxels included in the cluster. Region shows the location and its corresponding Brodmann Area as referenced by Talairach and Tournoux (1988).

75

Figures

Figure 4.1. Correlation profiles for the auditory cue.

Auditory cue tasks (AC-VT_C, AC-VT_IC) are negatively correlated with z-score transformed reaction time. Auditory target tasks were not considered in this analysis. C and IC represent compatible and incompatible conditions, respectively. The correlation coefficient (r) is plotted on the y-axis and the conditions are plotted on the x-axis.

76

Figure 4.1

77

Figure 4.2. Brain scores plotted by participants for the auditory cue.

The scatter plots show brain scores as a dot product of voxel saliences for an LV that expresses the cue-target contrast and reaction time data for each subject. Auditory cue conditions (AC-VT_C, AC-VT_IC) are compared to auditory target conditions (AT- VC_C, AT-VC_IC). C and IC correspond to compatible and incompatible conditions, respectively. Brain scores are shown on the y-axis and z-score transformed reaction times are represented on the x-axis. The conditions that the brain scores correspond to are shown with their correlation coefficients above each plot. Each shape/colour represents a participant. The slope of the line is the correlation coefficient whose magnitude is stated above each plot.

78

Figure 4.2

79

Figure 4.3. Singular images of brain regions that facilitate reaction time for the auditory cue.

The brain images displayed in this figure correspond to areas that are correlated with reaction time. Dominant positive weights, represented by yellow-red areas, are negatively correlated with behaviour (facilitate reaction time) for cue conditions (AC- VT_C, AC-VT_IC) while dominant negative weights, indicated by blue areas are negatively correlated with behaviour for target conditions (AT-VC_C, AT-VC_IC). C and IC refer to compatible and incompatible conditions, respectively. The singular brain images are superimposed on a T1-weighted MRI template image. A bootstrap ratio of 2.8 (99% confidence interval) was used to threshold images. The brain areas displayed in the figure are brainstem and cerebellum, medial visual areas, inferior parietal cortex, medial and lateral frontal areas and inferior frontal gyrus. Refer to Table 4.1 for exact MNI coordinates and cluster sizes for these brain regions.

80

Figure 4.3

81

Figure 4.4. Correlation profiles for the visual cue.

Visual cue tasks (VC-AT_C, VC-AT_IC) are negatively correlated with z-score transformed reaction time. Visual target tasks were not considered in this analysis. C and IC represent compatible and incompatible conditions, respectively. The correlation coefficient (r) is plotted on the y-axis and the conditions are plotted on the x-axis.

82

Figure 4.4

83

Figure 4.5. Brain scores plotted by participants for the visual cue.

The scatter plots show brain scores as a dot product of voxel saliences for an LV that expresses the cue-target contrast and reaction time data for each subject. Visual cue conditions (VC-AT_C, VC-AT_IC) are compared to visual target conditions (VT-AC_C, VT-AC_IC). C and IC correspond to compatible and incompatible conditions, respectively. Brain scores are shown on the y-axis and z-score transformed reaction times are represented on the x-axis. The conditions that the brain scores correspond to are shown with their correlation coefficients above each plot. Each shape/colour represents a participant. The slope of the line is the correlation coefficient whose magnitude is stated above each plot.

84

Figure 4.5

85

Figure 4.6. Singular images of brain regions that facilitate reaction time for the visual cue.

The brain images displayed in this figure correspond to areas that are correlated with reaction time. Dominant positive weights, represented by yellow-red areas, are negatively correlated with behaviour (facilitate reaction time) for cue conditions (VC- AT_C, VC-AT_IC) while dominant negative weights, indicated by blue areas are negatively correlated with behaviour for target conditions (VT-AC_C, VT-AC_IC). C and IC refer to compatible and incompatible conditions, respectively. The singular brain images are superimposed on a T1-weighted MRI template image. A bootstrap ratio of 2.8 (99% confidence interval) was used to threshold images. The brain areas displayed in the figure are cerebellum, brainstem, hippocampal region, lateral occipital area and claustrum. Refer to Table 4.2 for exact MNI coordinates and cluster sizes for these brain regions.

86

Figure 4.6

87

Chapter 5: General Discussion

I conducted an fMRI experiment to examine how crossmodal facilitation of response time is manifested in the brain. A modified version of the stimulus-response compatibility paradigm (Fitts & Seeger, 1953) was used where a cue indicated the response rule to a lateralized target. The cue signalled either a compatible response rule where participants responded with a button press on the same side as the presented target or an incompatible response rule where participants responded on the opposite side of the target’s location. In my experiment, cues and targets were presented auditorally and visually resulting in four task types. The first pair of tasks involved an auditory cue which preceded (task 1: AC-VT) or followed (task 2: VT-AC) a visual target. The pitch of the auditory cue signalled the response rule to the lateralized visual target (checkerboard pattern) that appeared on the left or right of fixation. The second pair of tasks were composed of a visual cue that came before (task 1: VC-AT) or after (task 2: AT-VC) the auditory target. The orientation of a checkerboard pattern indicated the response rule to a lateralized tone presented to the right or left ear. Cues and targets were separated by a four- second inter-stimulus interval. The cue-target order manipulation was designed to dissociate cue from target processing in the brain during a substantial delay interval.

The behavioural results, described in Chapter 3, can be summarized into three significant effects. Participants responded faster in compatible conditions compared to incompatible conditions. This stimulus response compatibility effect has been reported previously by Fitts and Seeger (1953) and Simon (1969). Speeded reaction times were found when cue tasks came first compared to target tasks. This effect has also been well documented in the sense that cues that precede targets are capable of preparing the brain to orient to different locations thereby, improving performance (Posner, Inhoff, Friedrich, & Cohen, 1987). The third effect showed an interaction between cue modality and the order of presentation. This interaction effect revealed a large difference between reaction times for the auditory cue conditions compared to target conditions and a very small insignificant difference between visual cues with respect to visual targets. My findings provide evidence for auditory superiority in crossmodal facilitation of reaction time. This asymmetrical facilitation has been replicated in previous work by Spence and Driver (1997) and others (Bertelson & Tisseyre, 1969; Buchtel & Butter, 1988; Davis & Green,

88

1969; Farah, Wong, Monheit, and Morrow, 1989). My behavioural results did not replicate Ward’s (1994) findings.

The fMRI findings reported in Chapters 3 and 4 can be summarized as follows. Task ST- PLS, which identifies task-dependent changes in brain activity across spatio-temporal domains, found a modulation of the BOLD response in both auditory and visual sensory cortices. In a contrast that compared auditory cue conditions with auditory target conditions, there was an increase in percent signal change for the BOLD response in cue conditions. I also found a similar enhancement in BOLD in visual cue conditions compared to visual target conditions. Since cue and target tasks were split into compatible and incompatible conditions, I was able to see if there were any stimulus-response compatibility differences in the BOLD signal for cue and or target conditions. In the case of the auditory cue, compatible responses in the cue conditions showed the most BOLD percent signal change followed by incompatible responses. There was no difference in compatibility in target conditions. BOLD responses for the visual cue indicated no stimulus-response compatibility differences in cue or target tasks.

Next, I investigated the spatio-temporal profiles of the auditory and the visual cues. Brain areas that were active in auditory cue conditions included auditory (BA 41/42) and visual cortices (BA 17/18). The co-activation in both these areas suggested that the auditory cue was able to prime the visual cortex for subsequent target processing. The visual cue conditions showed a different pattern of brain areas with increased activity in visual areas (BA 17/18) and the medial frontal gyrus (BA 10). There was no robust activation in the auditory cortices. The results indicated that auditory and visual cues tapped into different neural resources for cue processing.

I, then used behavioural ST-PLS to delineate neural patterns that capture the relationship between brain activity and normalized (z-score transformed) reaction time. Similar to task ST- PLS, I found distinct brain-behaviour patterns for the auditory and the visual cues. Brain regions that were negatively correlated with reaction time (facilitated reaction times) in auditory cue conditions included medial visual areas (BA 18/19), brainstem, posterior parts of the cerebellum, posterior cingulate (BA 30), inferior parietal cortex (BA 40) and frontal areas. Middle frontal areas (BA 9/10) were negatively correlated with behaviour early in stimulus processing while inferior frontal areas (BA 45) showed negative correlations with behaviour late in stimulus

89 processing. The overall pattern of brain-behaviour correlations for the auditory cue implicated the involvement of a fronto-parietal network and medial visual sites in facilitation of reaction time. Brain-behaviour correlations for the visual cue identified lateral visual areas (BA 19/37), claustrum, and anterior and posterior parts of the cerebellum and the hippocampal region as mediating faster reaction time relative to visual target conditions. This pattern of correlations implied that the visual cue recruited a posterior network in producing fast behavioural responses even though there was no significant effect of visual cue facilitation in the behavioural results.

The findings from task and behaviour ST-PLS ascertain that cue processing is different from the production of behavioural responses in terms of brain activity. The auditory cue recruits both auditory and visual cortices to process the cue, as shown by task PLS. The activation in lateral visual areas is followed by activation in more medial areas before the onset of the target stimulus (see Figure 3.6). These same medial areas show facilitation of reaction times in behaviour ST-PLS in conjunction with a fronto-parietal network and subcortical structures. The combination of task and behavioural ST-PLS results for the auditory cue show a tight coupling between cue processing and the production of a fast motor response. In contrast, the visual cue activates the visual cortex and a cluster in the fronto-polar cortex to process cue information. Faster reaction times for visual cues relative to targets recruit lateral visual areas such as the LOC and subcortical structures. There are no frontal sites that mediate faster responses in the visual cue conditions even though a frontal cluster was involved in processing cue salience. One potential explanation for these findings is that fronto-polar cluster found in task ST-PLS for the visual cue is not part of the same frontal network that is recruited by the auditory cue to facilitate reaction time. Therefore, the fronto-polar cortex may be capable of cue processing but does not contribute to behavioural performance. The discordance between the areas recruited in cue processing and those responsible for producing faster behavioural outcomes for the visual cue may be one explanation for why there are no significant differences in behaviour for visual cues compared to targets.

The results from my experiment are in line with ideas of neural context (Bressler & McIntosh, 2007; McIntosh, 1999; McIntosh, 2004). Subcortical sites such as the cerebellum and brainstem and cortical sites such as the sensory cortices are differentially involved in auditory and visual cue processing. The neural environment within which these areas are active changes with respect to the cue modality leading to different behavioural outcomes – facilitation of

90 response for auditory cues but not visual cues. In this way, I can reiterate the idea that neural context can be elicited from changes in situational context (represented by cue modality in my study).

The next few sections will attempt explain my results in light of previous findings reported in the literature.

5.1 A Convergent Model of Audio-Visual Interactions

Macaluso and Driver (2005) conducted a positron emission tomography (PET) study where they were interested in multisensory integration across visual and tactile modalities.

Participants were presented with either visual or tactile stimuli. The right or left hand received a tactile stimulus. The visual stimuli were shown in the right or the left hemifield in close proximity to the right or left hand. The experiment had four conditions: touch-right, touch-left, vision-right, vision-left. Brain activity was detected in both modality-specific (unimodal) and multisensory (multimodal) areas. The authors found that when each condition was considered separately – there were activations in areas that were not traditionally found. For example, the tactile-right conditions showed activations in contra-lateral occipital areas in the absence of visual stimulation suggesting that areas that are associated with unimodal processing can also be influenced by processing in other modalities. Moreover, when left and right conditions collapsed across input modalities were considered, there was increased activity in multisensory areas such as the and temporo-parietal junction. Findings from this experiment were summarized in a model of visuo-haptic interactions. The model explains that in addition to unimodal areas sending input to multimodal sites for integration purposes, connections between unimodal areas may provide sensory-specific cortices with the ability to integrate some information.

91

The model proposed by Macaluso and others (2005, 2006) explained multisensory

interactions between vision and touch. While my experiment examined crossmodal facilitation

effects, it also revealed how auditory and visual modalities interacted. I was able to separate

task-related processing (using task ST-PLS) from areas that may be involved in producing

behavioural outcomes (see behavioural ST-PLS results) in space and in time. I also demonstrated that multiple pathways in the brain can be used to integrate sensory information across different modalities. My findings can be summarized in a convergent model of audio-visual interactions.

This model posits that audio-visual interactions can occur within sensory-specific cortices as well as multisensory regions in the brain. The exact spatio-temporal profiles of how the interactions occur for a given modality is determined by neural interactions present at that

moment in the brain (idea of neural context) and how these interactions change as a result of cue

modality and task type (situational context). In summary, my model of audio-visual interactions

has at its core the idea that the brain is a dynamical system that produces behavioural outcomes

as a result of the interaction between neural environments and changing external context.

5.2 Dynamic Processing in Sensory-specific Cortices

In my literature review (see Chapter 1), I proposed that dynamic processing may not just be exclusive to cognitive processes like attention but may also be seen at the level of sensory- specific cortices. Anatomical studies conducted on primates have shown that sensory-specific cortices have the capacity to interact with areas that are part of other sensory or cognitive systems (Falchier, Clavagnier, Barone, & Kennedy, 2002; Rockland & Ojima, 2003). For instance, it is generally believed that inputs to V1 and V2 (primary visual areas) originate from areas that are visually-related only, such as the retina and the lateral geniculate nucleus (Ungerleider & Mishkin, 1992). However, Rockland and Ojima (2003) and Falchier and others (2002) have shown that V1 and V2 can also receive inputs from auditory and parietal cortices. Multiple lateral, feed forward and feedback connections from sensory areas such as V1 and V2 (Felleman & Van Essen, 1991) provide support for the idea that sensory-specific cortices can

92 contribute to many different functional networks depending on how they interact with other areas (neural context) and as a result of changing external demands (situational context).

My experimental findings show that sensory-specific cortices can co-activate (auditory cue activates auditory and visual cortices) and that these interactions can change depending on cue processing or production of behavioural responses (different spatio-temporal profiles and brain-behaviour patterns for auditory and visual cues). While I am unable to establish causal links between sensory-specific cortices in our data, I can suggest that these regions are correlated and may contribute to different neural networks depending on task type. My study shows potential for exploring causal relationships using metrics such as structural equation modelling, to establish further that sensory-specific processing can be dynamic rather than strictly modular.

In conclusion, I have confirmed all three of the hypotheses mentioned in Chapter 2. I have reported an asymmetrical facilitation of reaction times for auditory compared to visual cues and that this asymmetry is represented by different brain activation patterns. Also, brain- behaviour analyses have shown that brain areas that respond to cue processing (input) may not be the same areas that coordinate behaviour (output) and that both input and output processing is influenced by cue modality.

5.3 Limitations

The use of the fMRI technique puts some constraints on the type of brain activity that can be studied. The time course of an fMRI BOLD response is usually measured in seconds. Therefore, small temporal changes (on the order of milliseconds) in brain areas may not be picked up by fMRI. In my study, I looked at BOLD responses following the onset of the second stimulus. The substantial length of the inter-stimulus interval (ISI; four seconds) provided me with results that were separated in time however, I was unable to distinguish precisely when the ISI ended and the target was presented. Nevertheless, robust activation patterns in sensory areas confirmed the timing of stimulus presentation. In the future, combining fMRI with techniques that have superior temporal resolution, such as EEG, can offer information about activation in space and time.

93

5.4 Future Directions

The complex experimental design of the present study has the potential for investigating a number of different questions only a small subset of which have been addressed in this dissertation.

In analyzing my fMRI data with respect to cue and target conditions, I found neural changes that corresponded strictly to target processing. I have yet to examine these brain activation patterns. It can be inferred that lateralized presentation of the target may recruit brain areas that are involved in spatial mapping of stimuli. In addition to spatial mapping, I can also delineate changes in activity that correspond to speeded reaction times for compatible versus incompatible conditions. The stimulus response compatibility effect (Fitts & Seeger, 1953) has been established for decades but the neural correlates of this process have not been determined to date. Spatial mapping and stimulus-response compatibility effects will be examined in the near future.

After exploring how targets are processed in the brain, I will try to establish functional and causal links between areas that support cue versus target processing. Functional connectivity is a term that is used to describe the extent of the correlation of activity between brain regions (Friston, 1994; Horwitz, 2003; McIntosh & Gonzalez-Lima, 1998; Sporns, 2004). For example, in my data I found co-activation in the auditory and visual cortices in response to the auditory cue. If the auditory and visual cortices are functionally connected, their brain activity will be positively correlated. However, if activity in the auditory cortex is not correlated with visual cortex activity, other cortical regions may be driving activity in the visual cortex.

Once an anatomical network or regions has been shaped by functional connections, the direction and magnitude of causal influences between nodes of this network can be ascertained. The causal influence of one region on another has been described by the term effective connectivity (Friston, 1994). There are numerous cognitive studies that have looked at effective connections between brain areas using structural equation modelling (SEM; Cabeza et al., 1997; McIntosh et al., 1994). I can use SEM to investigate the causal relationships between different brain areas involved in a particular task.

94

Functional and effective connectivity can also show how neural context manifests in terms of brain activity. In my tasks, I find subcortical regions that are similarly active in both auditory and visual cue conditions. By mapping the functional and effective connections of these regions with areas that are uniquely active with respect to one modality, I will be able to illustrate that neural context in inherent in the brain. This line of data exploration will highlight further the dynamical systems’ perspective of brain function.

Lastly, I did not investigate strategic differences in performing the four tasks in our participants and how these differences could manifest at the neural level. While accuracy across all four tasks was at ceiling (0.9976), there may be underlying differences in covert strategies used to perform the tasks. A brief analysis of how much a participant contributed to a particular behaviour result (fast reaction times for auditory cue conditions) showed that there were some individual differences in processing cues. While the majority of participants showed a strong effect of auditory cue facilitation, there were a few participants that contributed to this effect more weakly. These differences did not reach significance at the behavioural level but may show different cortical routes of processing cue information. Since most fMRI studies use small sample sizes, individual differences at the neural level do not reach significance. The large sample size of my study can work in my favour as I try to determine how covert strategies can change brain-behaviour relationships across participants.

95

References

Alain, C., Arnott, S. R., Hevenor, S., Graham, S., & Grady, C. L. (2001). "What" and "where" in the human auditory system. Proceedings of the National Academy of Sciences USA, 98(21), 12301-12306.

Allen, G., Buxton, R. B., Wong, E. C., & Courchesne, E. (1997). Attentional activation of the cerebellum independent of motor involvement. Science, 275, 1940-1943.

Amedi, A., Malach, R., Hendler, T., Peled, S., & Zohary, E. (2001). Visuo-haptic object-related activation in the ventral visual pathway. Nature Neuroscience, 4, 324-330.

Andersen, R. A. (1995). Encoding of intention and spatial location in the posterior parietal cortex. Reviews, 5, 457-469.

Andersen, R. A., Snyder, L.H., Bradley, D.C., & Xing, J. (1997). Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annual Reviews Neuroscience, 20, 303-330.

Asaad, W. F., Rainer, G., & Miller, E. K. (1998). Neural activity in the primate prefrontal cortex during associative learning. Neuron, 21, 1399-1407.

Baker, S. C., Rogers, R. D., Owen, A. M., Frith, C. D., Dolan, R. J., Frackowiak, R. S., & Robbins, T. W. (1996). Neural systems engaged by planning: a PET study of the Tower of London task. Neuropsychologia, 34(6), 515-526.

Bast, T., & Feldon, J. (2003). Hippocampal modulation of sensorimotor processes. Progress in Neurobiology, 70(4), 319-345.

Beckmann, C. F. & Smith, S. M. (2004). Probabilistic independent component analysis for functional magnetic resonance imaging. IEEE Transactions on Medical Imaging, 23(2), 137-152

Bedard, M. A., El Massioui, F., Pillon, B., & Nandrino, J. L. (1993). Time for reorienting of attention: A premotor hypothesis of underlying mechanism. Neuropsychologia, 31, 241- 249.

96

Belin, P., & Zatorre, R. J. (2000). 'What', 'where' and 'how' in auditory cortex. Nature Neuroscience, 3(10), 965-966.

Belliveau, J. W., Kennedy, D. N. Jr., McKinstry, R. C., Buchbinder, B. R., Weisskoff, R. M., Cohen, M. S., Vevea, J. M., Brady, T. J., & Rosen, B. R. (1991). Functional mapping of the human visual cortex by magnetic resonance imaging. Science, 254, 716-719.

Bense, S., Stephan, T., Yousry, T. A., Brandt, T., & Dieterich, M. (2001). Multisensory cortical signal increases and decreases during vestibular galvanic stimulation (fMRI). The Journal of Neurophysiology, 85(2), 886-899.

Ben-Shakhar, G. (1985). Standardization within individuals: A simple method to neutralize individual differences in skin conductance. Psychophysiology, 22, 292-299.

Berman, K. F., Ostrem, J. L., Randolph, C., Gold, J., Goldberg, T. E., Coppola, R., Carson, R. E., Herscovitch, P., & Weinberger, D. R. (1995). Physiological activation of a cortical network during performance of the Wisconsin Card Sorting Test: a positron emission tomography study. Neuropsychologia, 33(8), 1027-1046.

Bertelson, P. & Tisseyre, F. (1969). The time-course of preparation: confirmatory results with visual and auditory warning signals. Acta Psychologica, 30. In Attention and Performance II (ed W.G. Koster), 145-154.

Bland, B. H., & Oddie, S. D. (2001). Theta band oscillation and synchrony in the and associated structures: the case for its role in sensorimotor integration. Behavioural Brain Research, 127, 119-136

Bremmer, F., Schlack, A., Shah, N. J., Zafiris, O., Kubischik, M., Hoffmann, K.-K., Zilles, K., & Fink, G. R. (2001). Polymodal motion processing in posterior parietal and : A human fMRI study strongly implies equivalences between humans and monkeys. Neuron, 29(1), 287-296.

Bressler, S. L. (1995). Large-scale cortical networks and cognition. Brain Research Reviews, 20, 288-904.

97

Bressler, S. L. (2002). Understanding cognition through large-scale cortical networks. Current Directions in Psychological Science, 11, 58-61.

Bressler, S. L., & McIntosh, A. R. (2007). The role of neural context in large-scale neurocognitive network operations. In V. Jirsa & A. R. McIntosh (Eds.), Handbook of Brain Connectivity. Springer, pp. 403-420.

Bressler, S. L., & Tognoli, E. (2006). Operational principles of neurocognitive networks. International Journal of Psychophysiology, 60(2), 139-148.

Brodal, A. (1981). The auditory system. In Neurological Anatomy in Relation to Clinical Medicine, 3rd Edition. New York, NY: Oxford University Press.

Brodmann K. (1909). Vergleichende Lokalisationslehre der Grosshirnrinde in ihren Prinzipien dargestellt auf Grund des Zellenbaues. Leipzig: J. A. Barth.

Brown, C. H., & May, B. (1989). Sound localization and binaural processes. In M. Berkeley & W. Stebbins (Eds.), Comparative perception. New York, NY: Wiley, pp. 247-284.

Brown, M. A., & Semelka, R. C. (2003). MRI: basic principles and applications – 3rd Edition. NJ: Wiley.

Buchtel, H.A., & Butter, C. M. (1988). Spatial attention shifts: implications for the role of polysensory mechanisms. Neuropsychologia, 26, 499-509.

Bushara, K. O., Grafman, J., & Hallett, M. (2001). Neural correlates of auditory-visual stimulus onset asynchrony detection. Journal of Neuroscience, 21, 300-304.

Bushara, K. O., Weeks, R. A., Ishii, K., Catalan, M. J., Tian, B., Rauschecker, J. P., & Hallett, M. (1999). Modality-specific frontal and parietal areas for auditory and visual spatial localization in humans. Nature Neuroscience, 2(8), 759-766.

Buxton, R. B., Wong, E. C., & Frank, L. R. (1998). Dynamics of blood flow and oxygenation changes during brain activation: the balloon model. Magnetic Resonance in Medicine, 39, 855-864.

98

Cabeza, R., Kapur, S., Craik, F. I. M., McIntosh, A. R., Houle, S., & Tulving, E. (1997). Functional neuroanatomy of recall and recognition: A PET study of episodic memory. Journal of Cognitive Neuroscience, 9(2), 254-265.

Cabeza, R., McIntosh, A. R., Tulving, E., Nyberg, L., & Grady, C. L. (1997). Age-related differences in effective neural connectivity during encoding and recall. Neuroreport, 8(16), 3479-3483.

Calvert, G. (2001). Crossmodal processing in human brain: insights from functional neuroimaging studies. Cerebral Cortex, 11, 1110-1123.

Calvert, G. A., Brammer, M. J., Bullmore, E., Campbell, R., Iversen, S. D., & David, A. (1999). Response amplication in sensory-specific cortices during crossmodal binding. Neuroreport, 10, 2619-2623.

Calvert, G. A., Bullmore, E., Brammer, M. J., Campbell, R., Williams, S. C., McGuire, P. K., Woodruff, P. W., Iversen, S. D., & David, A. S. (1997). Activation of auditory cortex during silent lip reading. Science, 276, 593-596.

Calvert, G. A., Campbell, R.., & Brammer, M. J. (2000). Evidence from functional magnetic resonance imaging of crossmodal binding in human heteromodal cortex. Current Biology, 10, 649-657.

Chica, A. B., Sanabria, D., Lupianez, J., & Spence, C. (2007). Comparing intramodal and crossmodal cuing in the endogenous orienting of spatial attention. Experimental Brain Research, 179, 353-364.

Christoff, K., & Gabrieli, J. D. E. (2000). The fronto-polar cortex and human cognition: Evidence for a rostrocaudal hierarchical organization within the human prefrontal cortex. Psychobiology, 28(2), 168-186.

Clarey, J. C., Barone, P., & Imig, T. J. (1992). Physiology of thalamus and cortex. In R. Fay & A. Popper (Eds), The mammalian auditory pathways: neurophysiology. New York, NY: Springer, pp 232-334.

99

Cohen, Y. E., Cohen, I. S., & Gifford, G. W. (2004). Modulation of LIP activity by predictive auditory and visual cues. Cerebral Cortex, 14(12), 1287-1301.

Corbetta, M., & Shulman, G. L. (2002). Controls of goal-directed and stimulus-driven attention in the brain. Nature Reviews Neuroscience, 3, 201-215.

Dassonville, P., Lewis, S. M., Zhu, X. H., Ugurbil, K., Kim, S. G., & Ashe, J. (2001). The effect of stimulus-response compatibility on cortical motor activation. NeuroImage, 13, 1-14.

Davis, R. (1959). The role of ‘attention’ in the psychological refractory period. Quarterly Journal of Experimental Psychology, 11, 211-220.

Davis, R., & Green, F. A. (1969). Intersensory differences in the effect of warning signals on reaction time. Acta Psychologica, 30, 155-167. de Monasterio, F. M. (1978a). Prooperties of concentrically organized X and Y ganglion cells of macaque retina. Journal of Neurophysiology, 41, 1394–1417. de Monasterio, F. M. (1978b). Properties of ganglion cells with atypical receptive-field organization in retina of macaques. Journal of Neurophysiology, 41, 1435-1449.

Deiber, M. P., Ibanez, V., Honda, M., Sadato, N., Raman, R., & Hallett, M. (1998). Cerebral processes related to visuomotor imagery and generation of simple finger movements studied with positron emission tomography. NeuroImage, 7, 73-85.

Desimone, R., & Ungerleider, L. G. (1989). Neural mechanisms of visual processing in monkeys. In H. Goodglass & A. R. Damasio (Eds.), Handbook of Neuropsychology (Vol. 2). Amsterdam: Elsevier, pp. 267-300.

Downer, J., Crawley, A. P., Mikulis, D. J., & Davis, K. D. (2000). A multimodal cortical network for the detection of changes in the sensory environment. Nature Neuroscience, 3, 277-283.

Dringenberg, H. C., Dennis, K. E. B., Tomaszek, S., & Martin, J. (2003). Orienting and defensive behaviors elicited by superior colliculus stimulation in rats; effects of 5-HT

100

depletion, uptake inhibition, and direct midbrain or frontal cortex application. Behavioural Brain Research, 144(1-2), 95-103.

Driver, J., & Spence, C. (1998). Attention and the crossmodal construction of space. Trends in Cognitive Sciences, 2, 254-262.

DuBois, R. M., & Cohen, M. S. (2000). Spatiotopic organization in human superior colliculus observed with fMRI. NeuroImage, 12, 63–70.

Dypvik, A. T., & Bland, B. H. (2004). Functional connectivity between the red nucleus and the hippocampus supports the role of hippocampal formation in sensorimotor integration. Journal of Neurophysiology, 92, 2040-2050.

Efron, B. & Tibshirani, R. (1985). The bootstrap method for assessing statistical accuracy. Behaviormetrika, 17, 1-35.

Eichenbaum, H. (2004). Hippocampus: cognitive processes and neural representations that underlie declarative memory. Neuron, 44, 109-120.

Eimer, M. (2000). The time course of spatial orienting elicited by central and peripheral cues: evidence from event-related brain potentials. Biological Psychology, 53, 253-258.

Eimer, M., & Driver, J. (2001). Crossmodal links in endogenous and exogenous spatial attention: evidence from event-related potential studies. Neuroscience and Biobehavioural Reviews, 25, 497-511.

Ettlinger, G., & Wilson, W. A. (1990). Cross-modal performance: behavioural processes, phylogenetic considerations and neural mechanisms. Behavioural Brain Research, 40(3), 169-192.

Falchier, A., Clavagnier, S., Barone, P., & Kennedy, H. (2002). Anatomical evidence of multimodal integration in primate striate cortex. Journal of Neuroscience, 22(13), 5749- 5759.

Farah, M. J., Wong, A. B., Monheit, M. A., & Morrow, L. A. (1989). Parietal lobe mechanisms or spatial attention: Modality-specific or supramodal? Neuropsychologia, 27(4), 461-470.

101

Felleman, D. J., & Van Essen, D. C. (1991). Distributed hierarchical processing in the primate cerebral cortex. Cerebral Cortex, 1(1), 1-47.

Fitts, P. M., & Seeger, C. M. (1953). S-R compatibility: spatial characteristics of stimulus and response codes. Journal of Experimental Psychology, 46(3), 199-210.

Foxe, J. J., & Simpson, G. V. (2002). Flow of activation from V1 to frontal cortex in humans: a framework for defining “early” visual processing. Experimental Brain Research, 142(1), 139-150.

Frackowiak, R. S. J., Friston, K. J., Frith, C. D., Dolan, R. J., & Mazziotta, J. C. (1997). Human brain function. San Diego, CA: Academic Press USA. Friston, K. (1994). Functional and effective connectivity: a synthesis. Human Brain Mapping, 2(1&2), 56-78.

Friston, K. J., Frith, C. D. & Frackowiak, R. S. J. (1993). Principal component analysis learning algorithms – a neurobiological analysis. Proceedings of the Royal Society of London Series B – Biological Sciences, 254, 47-54.

Fuster, J. M. (1997). Network memory. Trends in Neurosciences, 20(10), 451-459.

Gazzaniga, M., Ivry, R., & Mangun, G. (2002). Cognitive Neuroscience: The Biology of the Mind (2nd Ed). New York: W. W. Norton & Company, Inc., pp 247-252.

Giesbrecht, B., Kingstone, A., Handy, T. C., Hopfinger, J. B., & Mangun, G. R. (2006). Functional neuroimaging of attention. In R. B. Cabeza & A. Kingstone (Eds.), Handbook of functional neuroimaging of cognition (2nd edition). Cambridge, MA: MIT Press, pp. 85-111.

Goldberg, T. E., Berman, K. F., Fleming, K., Ostrem, J., Van Horn, J. D., Esposito, G., Mattay, V. S., Gold, J. M., & Weinberger, D. R. (1998). Uncoupling cognitive workload and prefrontal cortical physiology: a PET rCBF study. NeuroImage, 7(4 Pt 1), 296-303.

Goldman-Rakic, P. S. (1988). Topography of cognition: parallel distributed networks in primate association cortex. Annual Review of Neuroscience, 11, 137-156.

102

Goodale, M. A., & Milner, A. D. (1992). Separate visual pathways for perception and action. Trends in Neurosciences, 15, 20-25.

Grieve, K. L., Acuna, C., & Cudeiro, J. (2000). The primate pulvinar nuclei: vision and action. Trends in Neurosciences, 23(1), 35-39.

Grill-Spector, K., Kushnir, T., Edelman, S., Itzchak, Y., & Malach, R. (1998). Cue-invariant activation in object-related areas of the human . Neuron, 21(1), 191-202.

Guillery, R. W. (1982). The optic chiasm of the vertebrate brain. Contributions to Sensory Physiology, 7, 39-73.

Guttman, S. E., Gilroy, L. A., & Blake, R. (2005). Hearing what the eyes see: auditory encoding of visual temporal sequences. Psychological Science, 16(3), 228-235.

Hadjikhani, N., & Roland, P. E. (1998). Cross-modal transfer of information between the tactile and the visual representations in the human brain: a positron emission tomography study. Journal of Neuroscience, 18, 1072-1084.

Hallworth, N. E., & Bland, B. H. (2004). Basal ganglia-hippocampal interactions support the role of the hippocampal formation in sensorimotor integration. Experimental Neurology, 188, 430-443.

Harrison, J. M., & Irving, R. (1966). Visual and nonvisual auditory systems in mammals. Science, 154, 738-743.

Heeger, D. J., & Rees, D. (2002). What does fMRI tell us about neuronal activity? Nature Reviews. Neuroscience, 3, 142-151.

Hommel, B. (2005). Perception in action: multiple roles of sensory information in action control. Cognitive Processes, 6, 3-14.

Hopfinger, J. B., Buonocore, M. H., & Mangun, G. R. (2000). The neural mechanisms of top- down attentional control. Nature Neuroscience, 3, 284-291.

Horwitz, B. (2003). The elusive concept of brain connectivity. NeuroImage, 19, 466-470.

103

Horwitz, B., Grady, C. L., Haxby, J. V., Schapiro, M. B., Rapoport, S. I., Ungerleider, L. G., & Mishkin, M. (1992). Functional associations among human posterior extrastriate brain regions during object and spatial vision. Journal of Cognitive Neuroscience, 4(4), 311- 322.

Howard, I. P., & Templeton, W. B. (1966). Human spatial orientation. New York, NY: Wiley.

Hubel, D. H., & Wiesel, T. N. (1959). Receptive fields of single neurones in the cat’s striate cortex. Journal of Physiology (London), 148, 574-591.

Hudspeth, A. J. (2000). Hearing. In E. R. Kandel, J. H. Schwartz, & T. M. Jessell (Eds.), Principles of neural science – Fourth Edition. New York, NY: McGraw-Hill, pp. 590- 613.

Huettel, S. A., Song, A. W., & McCarthy, G. (2004). Functional magnetic resonance imaging. Sunderland, MA: Sinauer Associates, Inc.

Iacoboni, M., Woods, R. P., & Mazziotta, J. C. (1998). Bimodal (auditory and visual) left fronto- parietal circuitry for sensorimotor integration and sensorimotor learning. Brain, 121, 2135-2143.

Itoh, K., Mizuno, N., & Kudo, M. (1983). Direct retinal projections to the lateroposterior and pulvinar nuclear complex (LP-Pul) in the cat, as revealed by the anterograde HRF method. Brain Research, 276(2), 325-328.

James, T. W., Humphrey, G. K., Gati, J. S., Servos, P., Menon, R., & Goodale, M. A. (2002). Haptic study of three-dimensional objects activates extrastriate visual areas. Neuropsychologia, 40, 1706-1714.

Kaas, J. H., Guillery, R. W., & Allman, J. M. (1972). Some principles of organization of the lateral geniculate nucleus. Brain, Behaviour, & Evolution, 6, 253-299.

Kastner, S., Pinsk, M. A., De Weerd, P., Desimone, R., & Ungerleider, L. G. (1999). Increased activity in human visual cortex during directed attention in the absence of visual stimulation. Neuron, 22(4), 751-761.

104

Kim, Y. H., Gitelman, D. R., Nobre, A. C., Parrish, T. B., La Bar, K. S., & Mesulam, M.-M. (1999). The large-scale neural network for spatial attention displays multifunctional overlap but differential asymmetry. NeuroImage, 9, 269-277.

King, A. J. (1993). A map of auditory space in the mammalian brain: Neural computation and development. Experimental Physiology, 78, 559-590.

King, A. J., & Carlisle, S. (1993). Changes induced in the representation of auditory space in the superior colliculus by rearing ferrets with binocular eyelid suture. Experimental Brain Research, 94, 444-455.

King, A. J., Hutchings, M. E., Moore, D. R., & Blakemore, C. (1988). Developmental plasticity in the visual and auditory representations in the mammalian superior colliculus. Nature, 332, 73-76.

Knudsen, E. I., Esterly, S. D., & Knudsen, P. F. (1984). Monaural occlusion alters sound localization during a sensitive period in the barn owl. Journal of Neuroscience, 4, 1001- 1011.

Komura, Y., Tamura, R., Uwano, T., Nishijo, H., & Ono, T. (2005) Auditory thalamus integrates visual inputs into behavioral gains. Nature Neuroscience, 8, 1203–1209

Kurata, K., Tsuji, T., Naraki, S., Seino, M., & Abe, Y. (2000). Activation of the dorsal premotor cortex and pre- of humans during an auditory conditional motor task. Journal of Neurophysiology, 84(3), 1667-72.

Kwong, K. K., Belliveau, J. W., Chesler, D. A., Goldberg, I. E., Weiskoff, R. M., Poncelet, B. P., Kennedy, D. N., Hoppel, B. E., Cohen, M. S., Turner, R., Hong-Ming, C., Brady, T. J., & Rosen, B. R. (1992). Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. Proceedings of the National Academy of Sciences USA, 89, 5675-5679.

Lashley, K. S. (1929). Brain Mechanisms and Intelligence. New York: Hafner Publishing Co., Inc.

105

Laurens, K. R., Kiehl, K. A., & Liddle, P. F. (2005). A supramodal limbic-paralimbic- neocortical network supports goal-directed stimulus processing. Human Brain Mapping, 24, 35-49.

Laurienti, P. J., Wallace, M. T., Maldjian, J. A., Susi, C. M., Stein, B. E., & Burdette, J. H. (2003). Cross-modal sensory processing in the anterior cingulate and medial prefrontal cortices. Human Brain Mapping, 19, 213-223.

Legatt, A. D., Arezzo, J., & Vaughan, H. G. Jr. (1980). Averaged multiple unit activity as an estimate of phasic changes in local neuronal activity: effects of volume conducted potentials. Journal of Neuroscience Methods, 2, 203-217.

Lenartowicz, A., & McIntosh, A. R. (2005). The role of anterior in working memory is shaped by functional connectivity. Journal of Cognitive Neuroscience, 17(7), 1026-1042. Leventhal, A. G., Rodieck, R. W., & Drehkr, B. (1985). Central projections of cat retinal ganglion cells. The Journal of Comparative Neurology, 237(2), 216-226.

Lewkowicz, D. J. (1988a). Sensory dominance in infants: I. Six-month-old infants’ response to auditory-visual compounds. Developmental Psychology, 24, 155-171.

Lewkowicz, D. J. (1988b). Sensory dominance in infants: II. Ten-month-old infants’ response to auditory-visual compounds. Developmental Psychology, 24, 172-182.

Lewkowicz, D. J., & Kraebel, K. S. (2004). The value of multisensory redundancy in the development of intersensory perception. In G. A. Calvert, C. Spence, & B. E. Stein (Eds.), The handbook of multisensory processes. Cambridge, MA: MIT Press, pp. 655- 678.

Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., & Oeltermann, A. (2001). Neurophysiological investigation of the basis of the fMRI signal. Nature, 412(6843), 150-157.

Macaluso, E. & Driver, J. (2005). Multisensory spatial interactions: a window onto functional integration in the human brain. Trends in Neurosciences, 28(5), 264-271.

106

Macaluso, E. (2006). Multisensory processing in sensory-specific cortical areas. The Neuroscientist, 12(4), 327-338. Magalhaes-Castro, H. H., Murata, L. A., & Magalhaes-Castro, B. (1976). Cat retinal ganglion cells projecting to the superior colliculus as shown by the horseradish peroxidase method. Experimental Brain Research, 25(5), 541-549.

Malach, R., Reppas, J. B., Benson, R. R., Kwong, K. K., Jiang, H., Kennedy, W. A., Ledden, P. J., Brady, T. J., Rosen, B. R., & Tootell, R. B. (1995). Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Sciences USA, 92(18), 8135-8139.

McDonald, J. J., & Ward, L. M. (1999). Spatial relevance determines facilitatory and inhibitory effects of auditory covert spatial orienting. Journal of Experimental Psychology: Human Perception and Performance, 25, 1234-1252.

McDonald, J. J., & Ward, L. M. (2003, April). Crossmodal consequences of involuntary spatial attention and inhibition of return. Poster presented at the annual meeting of the Cognitive Neuroscience Society, New York.

McDonald, J. J., Teder-Salejarvi, W. A., & Hillyard, S. A. (2000). Involuntary orienting to sound improves visual perception. Nature, 407, 906-908.

McGurk, H., & MacDonald, J. W. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.

McIntosh, A. R. (1999). Mapping cognition to the brain through neural interactions. Memory, 7(5/6), 523-548.

McIntosh, A. R. (2004). Contexts and catalysts: a resolution of the localization and integration of function in the brain. Neuroinformatics, 2, 175–182.

McIntosh, A. R., Bookstein, F. L., Haxby, J. V. & Grady, C. L. (1996). Spatial pattern analysis of functional brain images using partial least squares. NeuroImage, 3, 143-157.

McIntosh, A. R., Chau, W. & Protzner, A. B. (2004). Spatiotemporal analysis of event-related fMRI data using partial least squares. NeuroImage, 23, 764-775.

107

McIntosh, A. R. & Gonzalez-Lima, F. (1994). Structural equation modeling and its application to network analysis in functional brain imaging. Human Brain Mapping, 2, 2-22.

McIntosh, A. R., & Gonzalez-Lima, F. (1998). Large-scale functional connectivity in associative learning: Interrelations of the rat auditory, visual and limbic systems. Journal of Neurophysiology, 80, 3148-3162.

McIntosh, A. R., Grady, C. L., Ungerleider, L. G., Haxby, J. V., Rapoport, S. I., & Horwitz, B. (1994). Network analysis of cortical visual pathways mapped with PET. Journal of Neuroscience, 14, 655-666.

McIntosh, A. R., & Lobaugh, N. J. (2004). Partial least squares analysis of neuroimaging data: Applications and advances. NeuroImage, 23, 250-263.

Mesulam M-M. (1981). A cortical network for directed attention and unilateral neglect. Annals of Neurology, 10, 309-325.

Mesulam, M. M. (1990). Large-scale neurocognitive networks and distributed processing for attention, language, and memory. Annals of Neurology, 28(5), 597-613.

Mesulam, M.-M. (1998). From sensation to cognition. Brain, 121, 1013-1052.

Miller, J. M., & Towe, A. L. (1979). Audition: structural and acoustical properties. In T. Ruch., & H. D. Patton (Eds.), Physiology and biophysics, Vol. 1. The brain and neural function, 20th Ed. Philadelphia, PA: Saunders, pp. 339-375.

Mitzdorf, U. (1987). Properties of the evoked potential generators: current source-density analysis of visually evoked potentials in the cat cortex. International Journal of Neuroscience, 33, 33-59. Mizuno, N., Itoh, K., Uchida, K., Uemura-Sumi, M., & Matsushima, R. (1982). A retino- pulvinar projection in the macaque monkey as visualized by the use of anterograde transport of horseradish peroxidase. Neuroscience Letters, 30(3), 199-203. Mondor, T. A. & Amirault, K. J. (1998). Effect of same- and different-modality spatial cues on auditory and visual target identification. Journal of Experimental Psychology: Human Perception and Performance, 24(3), 745-755.

108

Mountcastle, V. B. (1979). An organizing principle for cerebral function: The unit module and the distributed system. In M. V.B. & G. M. Edelman (Eds.), The Mindful Brain. Boston, MA: MIT Press, pp. 7-50.

Mukamel, R., Gelbard, H., Arieli, A., Hasson, U., Fried, I., & Malach, R. (2005). Coupling between neuronal firing, field potentials, and fMRI in human auditory cortex. Science, 309, 951-954.

Muller, H. J., & Findlay, J. M. (1988). The effect of visual attention on peripheral discrimination thresholds in single and multiple element displays. Acta Psychologica, 69, 129-155.

Nagahama, Y., Fukuyama, H., Yamauchi, H., Matsuzaki, S., Konishi, J., Shibasaki, H., et al. (1996). Cerebral activation during performance of a card sorting test. Brain, 119 (5 Pt 1), 1667-1675.

Nagy, A., Eordegh, G., Paroczy, Z., Markus, Z., & Benedek, G. (2006). Multisensory integration in the basal ganglia. European Journal of Neuroscience, 24(3), 917-924.

Nakagawa, S., & Tanaka, S. (1984). Retinal projections to the pulvinar nucleus of the macaque monkey: a re-investigation using audioradiography. Experimental Brain Research, 57(1), 151-157.

Neumann, O., Van der Heijden, A. H. C., & Allport, D. A. (1986). Visual selective attention: Introductory remarks. Psychological Review, 48, 185-188.

Noback, C. R. (1967). The human nervous system: basic elements of structure and function. New York, NY: McGraw-Hill.

Nyberg, L., Cabeza, R., & Tulving, E. (1996). PET studies of encoding and retrieval: the HERA model. Psychonomic Bulletin & Review, 3, 135-148.

Ogawa, S., & Lee, T. M. (1990). Magnetic resonance imaging of blood vessels at high fields: in vivo and in vitro measurements and image simulation. Magnetic Resonance in Medicine, 16(1), 9-18.

109

Ogawa, S., Lee, T. M., Nayak, A. S., & Glynn, P. (1990). Oxygenation-sensitive contrast in magnetic resonance image of rodent brain at high magnetic fields. Magnetic Resonance in Medicine, 14(1), 68-78.

Ogawa, S., Tank, D. W., Menon, R., Ellermann, J. M., Kim, S. G., Merkle, H., & Ugurbil, K. (1992). Intrinsic signal changes accompanying sensory stimulation: functional brain mapping with magnetic resonance imaging. Proceedings of the National Academy of Sciences USA, 89(13), 5951-5955. Pandya, D. N. (1995). Anatomy of the auditory cortex. Revue Neurologique (Paris), 151(8-9), 486-494.

Perry, V. H., & Cowey, A. (1984). Retinal ganglion cells that project to the superior colliculus and pretectum in the macaque monkey. Neuroscience, 12, 1125-1137.

Porter, K. K., Metzger, R. R., & Groh, J. M. (2007). Visual- and saccade-related signals in the primate inferior colliculus. Proceedings of the National Academy of Sciences USA, 104(45), 17855-17860.

Posner, M. I. (1980). Orienting of attention. Quarterly Journal of Experimental Psychology, 32, 3-25.

Posner, M. I. (1992). Attention as a cognitive and neural system. Current Directions in Psychological Science, 1, 11-14.

Posner, M. I., Inhoff, A. W., Friedrich, F. J., & Cohen, A. (1987). Isolating attentional systems: a cognitive-anatomical analysis. Psychobiology, 15, 107-121.

Posner, M. I. & Petersen, S. E. (1990). The attention system of the human brain. Annual Reviews of Neuroscience, 13, 25-42.

Posner, M. I., Sandson, J., Dhawan, M., & Shulman, G. L. (1989). Is word recognition automatic? Journal of Cognitive Neuroscience, 1, 50-60.

Posner, M. I., Walker, J. A., Friedrich, F. J., & Rafal, R. D. (1984). Effects of parietal injury on covert orienting of attention. The Journal of Neuroscience, 4, 1863-1874.

110

Price, C. J. & Friston, K. J. (2002). Degeneracy and cognitive anatomy. Trends in Cognitive Sciences, 6(10), 416-421.

Protzner, A. B., & McIntosh, A. R. (2007). The interplay of stimulus modality and response latency in neural network organization for simple working memory tasks. Journal of Neuroscience, 27(12), 3187-3197.

Rafal, R. D., & Posner, M. I. (1987). Deficits in human visual spatial attention following thalamic lesions. Proceedings of the National Academy of Sciences USA, 84, 7349-7353.

Rafal, R. D., Posner, M. I., Friedman, J. H., Inhoff, A. W., & Bernstein, E. (1988). Orienting of visual attention in progressive supranuclear palsy. Brain, 111, 267-280.

Rockland, K. S., & Ojima, H. (2003). Multisensory convergence in calcarine visual areas in macaque monkey. International Journal of Psychophysiologyl, 50(1-2), 19-26. Rodieck, R. W., & Watanabe, M. (1993). Survey of the morphology of macaque retinal ganglion cells that project to the pretectum, superior colliculus, and parvicellular laminae of the lateral geniculate nucleus. The Journal of Comparative Neurology, 338(2), 289-303.

Rodway, P. (2005). The modality shift effect and the effectiveness of warning signals in different modalities. Acta Psychologica, 120(2), 196-226.

Rosen, A. C., Rao, S. M., Caffarra, P., Scaglioni, A., Bobholz, J. A., Woodley, S. J., Hammeke, T. A., Cunningham, J. M., Prieto, T. E., & Binder, J. R. (1999). Neural basis of endogenous and exogenous spatial orienting: A functional MRI study. Journal of Cognitive Neuroscience, 11, 135-152.

Rosenzweig, M. R., Breedlove, S. M. & Leiman, A. L. (Eds.). (2002). Biological Psychology: 3rd Edition. Sunderland: Sinauer Associates, Inc. Publishers.

Ruz, M., & Lupianez, J. (2002). A review of attentional capture: on its automaticity and sensitivity to endogenous control. Psicologica, 23, 283-309.

Sahibzada, N., Dean, P., & Redgrave, P. (1986). Movements resembling orientation or avoidance elicited by electrical stimulation of the superior colliculus in rats. Journal of Neuroscience, 6, 723–733.

111

Sampson, P. D., Streissguth, A. P. Barr, H. M. & Bookstein, F. L. (1989). Neurobehavioral effects of prenatal alcohol: II. Partial least squares analysis. Neurotoxicology and Teratology, 11, 477-491.

Scannell, J. W., & Young, M. P. (1999). Neuronal population activity and functional imaging. Proceedings of the Royal Society B: Biological Sciences, 266, 875-881.

Schiller, P. H., & Malpeli, J. G. (1977). Properties and tectal projections of monkey retinal ganglion cells. Journal of Neurophysiology, 40, 428-445.

Schmitt, M., Postma, A., & de Haan, E. (2000). Interactions between exogenous auditory and visual spatial attention. Quarterly Journal of Experimental Psychology, 53A, 105-130.

Sherman, S. M. (1988). Functional organization of the cat’s lateral geniculate nucleus. In M. Bentivoglio, & R. Spreafico (Eds.), Cellular Thalamic Mechanisms. New York, NY: Excerpta Medica, pp. 163-183.

Shulman, G. L., Tansy, A. P., Kincade, M., Petersen, S. E., McAvoy, M. P., & Corbetta, M. (2002). Reactivation of networks involved in preparatory states. Cerebral Cortex, 12, 590-600.

Simon, J. R. (1969). Reactions toward the source of stimulation. Journal of Experimental Psychology, 81(1), 174-176.

Small, D. M., Gitelman, D. R., Gregory, M. D., Nobre, A. C., Parrish, T. B., & Mesulam, M.-M. (2003). The posterior cingulate and medial prefrontal cortex mediate the anticipatory allocation of spatial attention. NeuroImage, 18(3), 633-641.

Snyder, L. H., Batista, A. P., & Andersen, R. A. (1997). Coding of intention in the posterior parietal cortex. Nature, 386, 167-170.

Soto-Faraco, S., & Spence, C. (2002). Modality-specific auditory and visual temporal processing deficits. The Quarterly Journal of Experimental Psychology, 55A(1), 23-40.

Sparks, D. L. (1999). Conceptual issues related to the role of the superior colliculus in the control of gaze. Current Opinion in Neurobiology, 9, 698-707.

112

Sparks, D. L., & Hartwich-Young, R. (1989). The deep layers of the superior colliculus. In R. H. Wurtz, & M. Goldberg (Eds.), The neurobiology of saccadic eye movements. Amsterdam: Elsevier, pp. 213-255.

Spence, C., & Driver, J. (1994). Covert spatial orienting in audition: exogenous and endogenous mechanisms facilitate sound localization. Journal of Experimental Psychology: Human Perception and Performance, 20, 555-574.

Spence, C., & Driver, J. (1997). Audiovisual links in exogenous covert spatial orienting. Perception & Psychophysics, 59(1), 1-22.

Spoendlin, H. (1974). Neuroanatomy of the cochlea. In E. Zwicker, & E. Terhardt (Eds.), Facts and models in hearing. New York, NY: Springer-Verlag, pp. 18-32.

Sporns, O. (2004). Complex neural dynamics. In V. K. Jirsa, & J. A. S. Kelso (Eds.), Coordination dynamics: issues and trends. Berlin: Springer, Heidelberg.

Stein, B. E. (1998). Neural mechanisms for synthesizing sensory information and producing adaptive behaviors. Experimental Brain Research, 123, 124-135.

Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge, MA: MIT Press.

Stein, B. E., Wallace, M. T., & Meredith, M. A. (1995). Neural mechanisms mediating attention and orientation to multisensory cues. In M. S. Gazzaniga (Ed.), The cognitive neurosciences. Cambridge, MA: MIT Press, pp. 683-702. Stepniewski, I., Qi, H. X., & Kaas, J. H. (2000). Projections of the superior colliculus to subdivisions of the inferior pulvinar in New World and Old World monkeys. Visual Neuroscience, 17(4), 529-549.

Talairach, J., & Tournoux, P. (1988). Co-Planar Stereotaxic Atlas of the Human Brain (M. Rayport, Trans.). NY, New York: Thieme Medical Publishers, Inc.

Tanji, J., & Hoshi, E. (2001). Behavioural planning in the prefrontal cortex. Current Opinion in Neurobiology, 11, 164-170.

113

Tanji, J., & Shima, K. (1994). Role for the supplementary motor area cells in planning several movements ahead. Nature, 371, 413-416.

Taylor, S., Kornblum, S., Minoshima, S., Oliver, L., & Koeppe, R. (1994). Changes in medial cortical blood flow with a stimulus-response compatibility task. Neuropsychologia, 14, 1048-1057.

Tesche, C. D., & Karhu, J. (1999). Interactive processing of sensory input and motor output in the human hippocampus. Journal of Cognitive Neuroscience, 11(4), 424-436.

Thompson-Schill, S. L., D’Esposito, M., & Kan, I. P. (1999). Effects of repetition and competition on activity of left prefrontal cortex during word generation. Neuron, 23(3), 513–522

Thulborn, K., Waterton, J. C., Matthews, P. M., & Radda, G. K. (1982). Oxygenation dependence of the transverse relaxation time of water protons in whole blood at high field. Biochimica et Biophysica Acta, 714(2), 265-270.

Ungerleider, L. G., & Mishkin, M. (1982). Two cortical visual systems. In D. J. Ingle, M. A. Goodale, & R. J. W. Mansfield (Eds.), Analysis of visual behavior. Cambridge, MA: MIT Press, pp. 549-586.

Vargas, L. C., Marques, T. D. A., & Schenberg, L. C. (2000). Micturition and defensive behaviors are controlled by distinct neural networks within the dorsal periaqueductal gray and deep gray layers of the superior colliculus of the rat. Neuroscience Letters, 280, 45- 48.

Vroomen, J., & de Gelder, B. (2000). Sound enhances visual perception: cross-modal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance, 26(5), 1583-1590.

Ward, L. M. (1994). Supramodal and modality-specific mechanisms for stimulus-driven shifts of auditory and visual attention. Canadian Journal of Experimental Psychology, 48, 242- 259.

114

Ward, L. M., McDonald, J. A., & Golestani, N. (1998). Cross-modal control of attention shifts. In R. Wright (Ed.), Visual Attention. NY, New York: Oxford University Press, pp. 232- 268.

Ward, L. M., McDonald, J. A., & Lin, D. (2000). On asymmetries in cross-modal spatial attention orienting. Perception & Psychophysics, 62, 1258-1264. Wassle, H., & Illing, R.-B. (1980). The retinal projection to the superior colliculus in the cat: a quantitative study with HRP. The Journal of Comparative Neurology, 190(2), 333-356.

Wickelgren, B. G. (1971). Superior colliculus: some receptive field properties of bimodally responsive cells. Science, 173(3991), 69-72.

Withington, D. J. (1992). The effect of binocular lid suture on auditory responses in the guinea- pig superior colliculus. Neuroscience Letters, 136, 153-156.

Withington-Wray, D. J., Binns, K. E., Dhanjal, S. S., Brickley, S. G., & Keating, M. J. (1990). The maturation of the superior collicular map of auditory space in the guinea pig is disrupted by developmental auditory deprivation. European Journal of Neuroscience, 2, 693-703.

Wurtz, R. H., & Kandel, E. R. (2000a). Central Visual Pathways. In E. R. Kandel, J. H. Schwartz, & T. M. Jessell (Eds.), Principles of neural science – Fourth Edition. New York, NY: McGraw-Hill, pp. 523-547.

Wurtz, R. H., & Kandel, E. R. (2000b). Perception of Motion, Depth, and Form. In E. R. Kandel, J. H. Schwartz, & T. M. Jessell (Eds.), Principles of neural science – Fourth Edition. New York, NY: McGraw-Hill, pp. 548-571.

115

Appendix A – fMRI Screening Form

Completed by: ______Date:______

Name:______

Sex: M or F Age:______DOB: ______

Address:______- ______

Email Address:______

Telephone: H:______B: ______

Primary Language: English or Other:______

Highest Level of Education: ______Total Yrs of Education:______

Current Main Occupation & for how long?______

Handedness? Left or Right?

Visual Health: Excellent Good Fair Poor

Optical Correction: Distance only Near only Both None

Do you suffer from astigmatism? Y or N

What is your prescription strength:______

Eye Disease: Cataract Glaucoma Age Related Maculopathy

Other:______

Last eye exam?______

Last new eyeglass/contact prescription? ______

116

MRI – Specific Questions 1. Have you ever had a stroke or heart attack? If yes, when? Recovery?

2. Are you claustrophobic?

3. Are you on antidepressants?

4. Do you have any metallic implants? Pace-maker etc.? Shrapnel/bullets?

5. Can you lie on your back comfortably for 2 hrs?

6. Do you suffer from stiff joints, arthritis, osteoporosis?

Medical History 1. Are you taking any medications currently? If yes, list prescription & duration. (For women in contraceptive pills.)

2. Are you currently seeing a doctor regularly for any medical problems? If yes, why & for how long?

3. In the past did you take any medications regularly? If yes, list prescriptions & duration.

4. In the past did you see a doctor regularly for any medical problems? If yes, what for & how long?

117

5. Have you ever been hospitalized for a serious illness or surgery? If yes, when, why & duration.

6. In the last 3 yrs have you undergone any medical diagnostic procedures?

7. For females---Is there a chance that you may be pregnant? Have you had your period within the last 6 to 8 weeks? If there is a chance that you are pregnant, you should not undergo the scan.

It is against hospital policy to conduct an fMRI scan on pregnant or potentially pregnant females.

I understand that the scans may pose a potential risk if I am currently pregnant and I confirm that I am not pregnant at this time.

Subject’s Name:______Date:______

Subject’s Signature:______

Witness’s Name:______Date:______

Witness’s Signature:______

118

Neurological History 1. Have you ever had a seizure/fits/epilepsy? If yes, when, how many, any medications, for how long?

2. Have you ever had a serious head injury? Did you lose consciousness or visit the hospital for it? If yes, how long did you lose consciousness for & what was the outcome?

3. Have you ever had an MRI or PET scan before? Why? What did they find?

Psychological History 1. Have you ever sought counseling for psychological help? If yes, under what circumstances? When? How long? Where?

Substance Use 1. Have you ever taken any medications to help you with energy, sleep, nervousness or mood?

2. Have you ever used any other substances such as marijuana, hash, cocaine, LSD, PCP or any other uppers or downers in the past 3 years? (when, max usage, times per month).

3. What are you drinking habits like? Frequency, type, total # per week?

4. Was there ever a period in your life when you drank too much? Has drinking ever caused problems for you (work-related, familial)? Has anyone objected to your drinking? Has anyone else thought you had an alcohol problem?

119

Appendix B: MRI Screening Form

120

Appendix C: Information and Consent Form

3.0T MRI INFORMATION SHEET The Rotman Research Institute 3560 Bathurst Street Toronto, Ontario, M6A 2E1 Phone: (416) 785-2500 ext.3550 Fax: (416) 785-2862

Title of the study: Investigating Sensorimotor Integration in Cross-modal Cueing Tasks using Event-Related fMRI

Investigators: Zainab Fatima (416) 785-2500 x 3112

Randy McIntosh, Ph.D. (416) 785-2500 ×3522

Rotman Research Institute of Baycrest

Additional laboratory staff:

Maria Tassopoulos: (416) 785 2500 x 3384

Purpose of Research: This study will investigate parts of the brain that are important for integrating sensory and motor information in normal healthy young adults. During the study, you will learn to use cues to respond to targets. The cues and targets will be crossmodal – auditory and visual. During the performance of these tasks, your brain activity will be monitored using a method known as functional Magnetic Resonance Imaging (fMRI).

Description of Research: This study will involve testing in two sessions on the same day. You will be tested on one half of the experiment followed by a short break and then tested on the other half of the experiment. The experiments will take place in the MRI testing facility at Baycrest. The MRI technique uses magnets and radio waves to construct a picture of your brain on a computer. Before the scan begins, you will be asked to remove any magnetic metals that you may be wearing. For the MRI scan of your brain, you will be asked to lie on a padded bed that will move into a tunnel-like machine. While you are inside the machine, a screen placed behind you will be available for viewing stimuli. Therefore, you may not be able to see the investigators or the technicians operating the machine. However, there is an intercom system that will allow

121

you to talk to them at any time. There is also a bulb that you will have with you in the scanner that you may squeeze if you feel uncomfortable and you wish to discontinue the procedure.

Setup and scanning for the first session takes about two hours to complete. After the first session, you will be taken out of the MRI scanner for a thirty minute break during which point, you can stretch and move about. The second session should take about an hour and thirty minutes to complete. During both sessions you will be presented with a variety of tasks to complete. You should try to remain as still as possible during the scan. Movement will not be dangerous to you in any way but will blur the picture of your brain. You will hear moderately loud knocking or beeping while the MRI machine is operating. You may also feel the padded bed shift at times during the MRI scan. Although you may find this to be unsettling, the machine cannot hurt you.

Potential Harm (Injury, Discomfort or Inconvenience): There is no known harm associated with participation in this study. An MRI does not involve any form of radiation or injections. However, some people may feel uncomfortable lying still in the confined space of the MRI scanner. You will be in constant contact with the technician who operates the MRI. At any time, you may ask to be taken out of the scanner without any penalty or consequences to you.

Potential Benefits: You will not benefit directly from participating in this study. The information gained from this research may be used in the future to help people with diseases or damage to the brain.

Confidentiality: No subject names will be recorded on MRI scanner image databases, data sheets, or computer files. Any data resulting from your participation that will be published in scientific journals, texts, or other media will not reveal your identity. Neither your identity nor any personal information will be available to anyone other than the investigators. Your decision whether or not to participate will not prejudice you or your future interactions with researchers performing the study, nor will it affect care provided to you or your family members at Baycrest. If you decide to participate, you are free to withdraw your consent and to discontinue your participation at any time. A copy of the consent form will be given to you and the other copy will be retained by the principal investigators – Zainab Fatima and Randy McIntosh.

Reimbursement: You will be reimbursed $50 for each of the two sessions. Total is $100.

122

CONSENT

Title of the study: Investigating Sensorimotor Integration in Cross-modal Cueing Tasks using Event-Related fMRI

Investigators: Zainab Fatima (416) 785-2500 x 3112

Randy McIntosh, Ph.D. (416) 785-2500 ×3522

Rotman Research Institute of Baycrest

I acknowledge that the research procedures described in the Information Sheet, and of which I have a copy, have been explained to me and that any questions that I have asked have been answered to my satisfaction. I have been informed of the alternatives to participation in this study, including the right not to participate and the right to withdraw without compromising the quality of medical care at Baycrest for myself and for other members of my family. As well, the potential harms and discomforts have been explained to me and I also understand the benefits (if any) of participating in the research study. I know that I may ask now, or in the future, any questions that I have about the study or the research procedures. I have been assured that my records will be kept confidential and that no information will be released or printed that would disclose personal identity without my permission unless required by law. I understand that the brain images that will be taken are not for medical use and will not typically be reviewed by a medical doctor. However, if any information is collected during course of the study that suggests an abnormality to a non-medically trained investigator, I understand that a radiologist may review my records and I may be referred to a physician. I hereby give my approval to participate in the study. ______Name of Volunteer Name of Investigator

______Signature of Volunteer Signature of Investigator

______Date Date

You do not waive any legal rights by signing this form. Your signature indicates that you have read and understood the above information, and that you have decided to participate based on the information provided. A copy of this form will be made available to you on request.