1

Supplementary Material

Talairach Onset Contacts Patient Anatomical region Side coordinates latency name X Y Z Patient 5 Medial Transverse 14 L H'2 -35 -24 7 Patient 10 Medial Transverse Gyrus 17 R H10 43 -19 7 Patient 4 Medial Transverse Gyrus 19 L H'6-7 -33 -28 10 Patient 7 Transverse Gyrus/ 23 L H'6 -48 -22 9 Patient 9 Medial Transverse Gyrus 23 L H'6-8 -35 -22 6 Patient 6 Medial Transverse Gyrus 23 R H4 39 -20 5 Patient 10 Medial Transverse Gyrus 23 R H6 30 -19 7 Patient 10 Medial Transverse Gyrus 23 R H8 36 -19 7 Patient 4 Transverse Gyrus/Planum Temporale 25 L H'8-9 -41 -28 10 Patient 8 Lateral Transverse Gyrus 25 R T3-5 42 -11 7 Patient 6 Medial Transverse Gyrus 27 R H3 36 -20 5 Patient 8 Medial Transverse Gyrus 27 L H'8 -39 -23 7 Patient 7 Transverse Gyrus 29 L H'2-3 -36 -22 9

Supplementary Table 1: Early responses (before 30 ms latency) to the auditory syllable. Side indicates the implanted hemisphere (Left or Right). Contact’s name refers to the name of the electrode as indicated in Figure 1 and the number of the contacts on this electrode: contacts are numbered medial to lateral. 2

Talairach Response Onset Offset Contacts Patient Explored Region Side coordinates type latency latency Name X Y Z Patient 9 Posterior MTG -100 160 L V'12-14 -46 -63 12 Patient 8 Posterior STS -80 400 L E'6-8 -42 -53 1 Patient 6 ITG/Posterior ITS/ -80 140 R L11 55 -55 0 Patient 10 Occipito-Temporal Junction -40 600+ L V'10-12 -35 -65 1 1 Patient 10 Occipito-Temporal Junction -40 600+ R V9-12 34 -64 1 Ventral Occipito-temporal Gyrus/ Calcarine Patient 10 -40 600+ R V7-9 26 -63 2 Fissure Patient 10 Ventral Occipito-Temporal Gyrus -20 350 L W'9-10 -34 -63 11 Patient 6 Ventral Occipito-Temporal Gyrus 60 350 R L3-4 32 -55 0 Patient 3 Anterior STS -120 600+ R T7-9 57 -7 1 Patient 8 STS/STG -60 500 L T'8-9 -62 -10 -3 Patient 3 STS Fundus -40 600+ R C8 46 -16 -11 Patient 3 Lateral MTG -40 600+ R C13 57 -15 -12 Medial Posterior Transverse Gyrus/ Planum Patient 1 -20 600+ R H7-10 42 -29 7 Temporale 2 Patient 7 Planum Temporale/ -20 450 L G'11-14 -56 -40 21 Patient 5 Planum Temporale/STS 0 600+ L G'3-5 -57 -39 17 Patient 8 Planum Temporale 0 550 L H'10-15 -47 -24 6 Patient 8 Insula/ Planum Polare 0 600+ L T'3 -40 -10 -2 Patient 1 Lateral Anterior Transverse Gyrus 40 600+ R T7-10 54 -9 -4 Patient 3 STG/ Lateral Anterior Transverse Gyrus 100 600+ L T'6-9 -57 -4 4 Patient 8 Insula/Medial Transverse Gyrus 160 550 R T2 34 -12 6 Patient 2 Medial Anterior Transverse Gyrus -120 450 R H6-7 33 -25 11 Patient 2 Planum Temporale -120 450 R H8-12 44 -27 10 Patient 10 Planum Temporale/Transverse Gyrus -20 600+ R H11-14 57 -19 6 3 Patient 10 Medial Anterior Transverse Gyrus -20 600+ R H6-9 33 -19 7 Patient 3 Planum Temporale /STS 60 600+ L H'6-13 -46 -30 5 Patient 7 Planum Polare 140 300 L T'2-7 43 -10 7 Patient 9 STS 40 600+ L H'11-13 -60 -24 6 Patient 6 Inferior bank of the STS 60 600+ R B10 49 -16 0 Patient 5 STG 220 600+ L H'8-10 -61 -24 7 4 Patient 4 STS Fundus/ ITS * 140 300 L B'7-9 -46 -24 -9 Patient 4 STG/Anterior STS 240 600+ L T'6-9 -54 -15 -2 Patient 4 Superior bank of the STS 260 600+ L H'11-15 -58 -28 10 Patient 4 MTG/STS 300 600+ L B'10-12 -56 -24 -9 Patient 7 Anterior STS 60 600+ L A'10-11 -49 -8 -16 Patient 3 Lateral MTG 60 600+ L B'11 -62 -20 -14 5 Patient 3 STG/STS 160 600+ R A11 57 -3 -9 Patient 7 STS * 160 600+ R B7-9 49 -23 -9 Patient 8 MTG/ITS 220 600+ L A'9-10 -56 0 -19 Patient 10 Supramarginal Gyrus/ -20 600+ R E13 52 -26 33 6 Patient 8 Supramarginal Gyrus 80 450 L G'11-15 50 -33 33 Patient 8 Supramarginal Gyrus 160 600+ R G15 57 -31 32 Patient 4 Posterior STS (lesion...) 100 400 R X11 47 -53 16 Patient 5 Posterior STS fundus/Supramarginal Gyrus 100 500 L Y'8-10 -35 -50 23 Supramarginal Gyrus/ Postcentral Patient 10 100 200 R E8-10 31 -26 33 Gyrus/Postcentral * 7 Patient 1 /Supramarginal Gyrus * 220 350 R V11-12 47 -43 34 Supramarginal Gyrus/Postcentral Patient 10 240 400 R E11 46 -25 33 Gyrus/Postcentral Sulcus Patient 10 Angular Gyrus/Supramarginal Gyrus * 300 600+ R G11-12 44 -43 26 Patient 5 Supramarginal Gyrus 300 600+ L Y'13-15 -53 -49 24 8 Patient 3 MTG 140 600+ L L'13-14 -64 -43 -3 Patient 1 Posterior STS 160 450 R R2 43 -39 3 3

Patient 7 Postcentral Gyrus 100 260 L R'1 -47 -18 27 Patient 8 Postcentral Sulcus * 160 600+ L G'8 -31 -36 30 Patient 2 Postcentral 160 600+ R N2 33 -14 12 9 Patient 7 Anterior Insula/Postcentral Gyrus 160 450 L P'2-3 -37 -3 3 Patient 5 Postcentral Operculum 220 600+ L N'8-10 -58 -14 20 Patient 7 Insula/Postcentral Operculum 240 600+ L N'2-3 -38 -10 20 Patient 5 Insula/Postcentral Operculum* 400 600+ L N'3-4 -38 -14 20 Patient 4 Insula* 140 400 R T4-5 38 -15 2 10 Patient 8 Insula/Planum Temporale 160 550 R T2 34 -12 6 Insula/Planum Polare/Lateral Anterior Transverse Patient 4 220 400 L T'4-5 -42 -15 -2 Gyrus * Patient 8 Posterior Cingular Gyrus/ * 160 600+ L G'3-4 -14 -37 30 11 Patient 10 Posterior Cingular Gyrus* 200 600+ R G3-4 12 -45 26 Patient 10 Posterior Cingular Gyrus 220 600+ L G'3 -12 -40 28 Patient 6 Precentral Operculum -60 550 R N4-7 43 -7 10 Patient 1 Precentral Operculum 140 600+ R N7-9 52 12 21 12 Patient 7 Precentral Operculum 240 600+ L P'8 -58 -1 18 Patient 7 Inferior Posterior Frontal Gyrus 240 600+ L E'5-8 -51 11 7 Patient 9 Precentral Operculum 240 400 L P'4-10 -48 6 5 Patient 7 Hippocampus 80 400 L B'2-5 -35 -23 -5 Patient 8 Hippocampus 180 600+ R B'1-2 -30 -20 -10 Patient 8 Hippocampus 220 600+ R B2-3 30 -22 -7 13 Patient 10 / 220 600+ R D3 22 -3 -33 Patient 10 Hippocampus/Insula/Medial Transverse Gyrus 260 600+ L B'2-4 -34 -15 -10 Patient 9 Parahippocampal Gyrus/ 300 600+ L L'2 -18 -48 5 Patient 6 Supramarginal Gyrus/ -60 600+ R R7-11 30 -48 35 Patient 10 Lingual Gyrus * 0 120 R L5 32 -31 -15 Patient 5 /Occipito-Parietal Sulcus * 0 600+ L W'3-5 -14 -68 36 Patient 5 / Occipito-Parietal Sulcus* 0 600+ L W'6-7 -26 -66 36 Patient 2 Inferior Postcentral Operculum/Transverse Gyrus 80 550 R N4-5 43 -15 12 Patient 8 Colateral Sulcus 80 600+ L D'5-7 -44 -5 -30 Patient 8 Anterior Ventral Temporal gyrus* 120 350 L A'4-7 -39 -1 -19 Patient 5 Intraparietal Sulcus* 160 600+ L Q'6 -23 -47 50 14 Patient 10 160 300 R K9-11 33 37 24 Patient 10 Lingual Gyrus 180 600+ R L2-5 28 -31 -14 Patient 10 Lingual Gyrus 200 600+ L L'4-5 -30 -30 -13 Patient 10 Calcarine Fissure * 280 400 R W7 25 -63 9 Patient 9 Colateral Sulcus 300 600+ R E'5-6 -42 -30 -9 Patient 6 * 400 600+ R K10-11 37 26 21 Patient 10 Posterior MTG * 450 600+ R W12 41 -63 7

Supplementary Table 2: Localization and latency of statistically significant visual ERPs. All activations are significant at p<10-5 both in monopolar and bipolar data, except stated otherwise. * indicates responses significant only in bipolar data; 600+ indicates that the response continues after 600 ms; Talairach coordinates are in the normalized Talairach space. Types of visual responses are broadly numbered according to their latency; inside a response type, individual patient responses are ordered by onset latency: 1. Visual-specific activity around the posterior MTG; 2. Activity recorded in the superior with a spatial profile resembling that of a transient auditory response beginning at 50 ms; 3. Activity recorded in the superior temporal lobe with a spatial profile resembling that of a transient auditory ERP beginning at 100 ms; 4. Visual-specific activity in the superior temporal lobe; 5. Activity around the anterior STS, 4

common to visual and auditory conditions; 6. Visual-specific response around the supramarginal gyrus; 7. Response around the supramarginal gyrus common to visual and auditory conditions; 8. Activity around the posterior STS, common to visual and auditory conditions; 9. Response around the post-central operculum and the insula, common to auditory and visual conditions; 10. Visual- specific response in the insula; 11. Visual-specific response around the cingular gyrus; 12. Visual- specific response around the precentral orpeculum and the ; 13. Visual-specific response around the hippocampus and hippocampal gyrus; 14. Unclassified responses. ITG: ; ITS: ; MTG: Medial Temporal Gyrus; STS: ; STG: . 5

Talarairach Type of Onset Offset Contacts Patient Explored Region Side coordinates interaction latency latency Name X Y Z Patient 6 Medial Anterior Transverse Gyrus 30 600+ R H3-5 39 -20 5 Patient 2 Anterior Planum Temporale 40 110 R T8-10 61 -13 -2 Patient 7 STG / Supramarginal Gyrus 70 600+ L G'9-14 -54 -39 21 Patient 8 Planum Temporale 70 500 L H'10-15 -58 -23 6 Patient 3 Superior STG 80 450 L T'8 -59 -4 4 Patient 10 Medial Transverse Gyrus 90 600+ R H7-10 38 -19 7 Patient 3 STG White Matter 100 250 L H'12-15 -61 -28 5 Planum Temporale/ Posterior Transverse Patient 1 110 250 R H8-10 44 -28 7 Gyrus Patient 3 Superior bank of the STS 120 600+ R T7-8 55 -7 2 Planum Polare/Lateral Anterior Transverse Patient 8 120 250 R T6-7 51 -11 8 1 Gyrus Patient 8 STG 120 180 R T9-10 62 -11 9 Patient 9 MTG/Inferior bank of the STS * 120 250 L L'12-14 -56 -47 5 Planum Temporale/ Lateral Transverse Patient 10 130 600+ R H11-13 57 -19 6 Gyrus Patient 7 130 500 L N'6-7 -52 -10 20 Patient 5 Planum Temporale/Supramarginal Gyrus 130 250 L G'4 -57 -39 17 Patient 8 Superior bank of the STS/STG 130 200 L T'8-9 -61 -9 -3 Patient 8 Supramarginal Gyrus 130 300 L G'13 -50 -35 29 Patient 7 Planum Polare/Superior bank of the STS 140 400 L T'4-9 -51 -4 2 Patient 8 Insula/Planum Polare 160 450 R T2-3 37 -11 7 Patient 6 Posterior Lateral Transverse Gyrus ** 40 120 R H7-9 53 -20 5 Patient 8 Superior bank of the STS ** p<0.005 50 90 L T'7-8 -58 -9 -3 Patient 3 STG ** p<0.01 50 100 L H'12-15 -61 -28 5 Patient 8 Planum Temporale 60 120 L H'11 -50 -23 7 2 Patient 3 Superior bank of the STS** p<0.005 60 120 R T7-8 55 -7 2 Patient 7 Superior bank of the STS* p<0.05 60 100 L T'7-8 -57 -4 2 Patient 10 Medial Transverse Gyrus 80 160 R H7-10 38 -19 7 Patient 8 Planum Temporale 80 160 L H'13-15 -62 -23 7 Patient 7 Superior bank of the STS/Planum Polare 120 200 L T'5-7 -57 -4 2 Patient 1 Posterior Cingular Gyrus * 20 60 R W4 13 -50 16 Patient 10 Calcarine Fissure 30 80 R V2 8 -68 5 Patient 2 Superior bank of the STS/STG 40 110 R C9-13 61 -25 -4 Patient 2 STG/Planum Temporale 50 100 R H15 63 -26 10 Patient 6 Cuneus 60 130 R G5-6 19 -57 16 Patient 2 Planum Temporale* 70 120 R H12 52 -26 10 Patient 9 Inferior bank of the STS 100 120 L B'11 -59 -17 -13 Patient 7 Lateral Posterior Transverse Gyrus 120 250 L H'8-9 -57 -22 9 3 Patient 8 Posterior Cingular Gyrus/Precuneus 120 300 L G'4 -15 -36 30 Patient 9 STS/STG * 120 250 L H'13 -62 -22 6 Patient 6 Insula 120 200 R T3 38 -10 0 Patient 6 Ventral Posterior ITG 130 200 R L6-7 39 -55 -8 Patient 3 MTG 140 550 L B'12 -66 -19 -14 Patient 6 Precentral Gyrus 140 200 R N7 50 -7 10 Patient 6 ITS/ITG 140 250 R L10 51 -54 -8 Patient 9 MTG/Inferior bank of the STS * 160 250 L V'14-15 -51 -61 12

Supplementary Table 3 : Localization and latency of statistically significant crossmodal interactions. All activations are significant at p<10-5;both in monopolar and bipolar data, except stated otherwise. * indicates responses significant only in bipolar data, ** indicates interactions 6

significant only in monopolar data; 600+ indicates that the response continues after 600 ms; Talairach coordinates are in the normalized Talairach space. Type of interactions: 1. Response in the superior temporal cortex having the same spatiotemporal profile as the visual-only response, with opposite polarity; 2. Response in the superior temporal cortex having the same spatiotemporal profile as a transient auditory-only response, with opposite polarity; 3. Miscellaneous interaction responses. ITG: Inferior Temporal Gyrus; ITS: Inferior Temporal Sulcus; MTG: Medial Temporal Gyrus; STS: Superior Temporal Sulcus; STG: Superior Temporal Gyrus.