Unicentre CH-1015 Lausanne http://serval.unil.ch

RYear : 2017

BRAIN DYNAMICS OF ILLUSORY CONTOUR PERCEPTION AND PERCEPTION VIA SENSORY SUBSTITUTION IN HEALTHY SUBJECTS.

Anken Jacques

Anken Jacques, 2017, BRAIN DYNAMICS OF ILLUSORY CONTOUR PERCEPTION AND PERCEPTION VIA SENSORY SUBSTITUTION IN HEALTHY SUBJECTS.

Originally published at : Thesis, University of Lausanne

Posted at the University of Lausanne Open Archive http://serval.unil.ch Document URN : urn:nbn:ch:serval-BIB_8A0C22C02A965

Droits d’auteur L'Université de Lausanne attire expressément l'attention des utilisateurs sur le fait que tous les documents publiés dans l'Archive SERVAL sont protégés par le droit d'auteur, conformément à la loi fédérale sur le droit d'auteur et les droits voisins (LDA). A ce titre, il est indispensable d'obtenir le consentement préalable de l'auteur et/ou de l’éditeur avant toute utilisation d'une oeuvre ou d'une partie d'une oeuvre ne relevant pas d'une utilisation à des fins personnelles au sens de la LDA (art. 19, al. 1 lettre a). A défaut, tout contrevenant s'expose aux sanctions prévues par cette loi. Nous déclinons toute responsabilité en la matière.

Copyright The University of Lausanne expressly draws the attention of users to the fact that all documents published in the SERVAL Archive are protected by copyright in accordance with federal law on copyright and similar rights (LDA). Accordingly it is indispensable to obtain prior consent from the author and/or publisher before any use of a work or part of a work for purposes other than personal use within the meaning of LDA (art. 19, para. 1 letter a). Failure to do so will expose offenders to the sanctions laid down by this law. We accept no liability in this respect.

Département des Neurosciences Cliniques

BRAIN DYNAMICS OF ILLUSORY CONTOUR PERCEPTION AND PERCEPTION VIA SENSORY SUBSTITUTION IN HEALTHY SUBJECTS.

Thèse de doctorat en Neurosciences

présentée à la

Faculté de Biologie et de Médecine de l’Université de Lausanne

par

Jacques Anken

Biologiste diplômé de l’Université de Lausanne, Suisse

Jury

Prof. Aki Kawazaki, Présidente Prof. Micah M. Murray, Directeur Prof. Olivier Collignon, Expert Prof. Daniel C. Kiper, Expert

Thèse n° 202

Lausanne 2017

Programme doctoral interuniversitaire en Neurosciences des Universités de Lausanne et Genève

Département des Neurosciences Cliniques

BRAIN DYNAMICS OF ILLUSORY CONTOUR PERCEPTION AND PERCEPTION VIA SENSORY SUBSTITUTION IN HEALTHY SUBJECTS.

Thèse de doctorat en Neurosciences

présentée à la

Faculté de Biologie et de Médecine de l’Université de Lausanne

par

Jacques Anken

Biologiste diplômé de l’Université de Lausanne, Suisse

Jury

Prof. Aki Kawazaki, Présidente Prof. Micah M. Murray, Directeur Prof. Olivier Collignon, Expert Prof. Daniel C. Kiper, Expert

Thèse n° 202

Lausanne 2017

Programme doctoral interuniversitaire en Neurosciences des Universités de Lausanne et Genève

Anken Jacques Département des Neurosciences Cliniques March 17

ACKNOWLEDGEMENTS

Before jumping into the thesis, I would like to acknowledge all the members of the Laboratory for Investigative Neurophysiology at the CHUV in Lausanne for their advice and shared ideas during coffee breaks. I would like to thank particularly some of them. First, Rosanna, for the ideas and discussions for developing the third paper presented in this thesis and for her friendship. Sandra for her friendship and the particular gift she has to tease me and others. The Foodies, they will recognize themselves, for their friendship, helping and cheering me up throughout my PhD thesis and improbable stories. David for his friendship, the multiples dinner, whiskey discoveries, and cycling experience up the “hill” for going to work in the early morning. Micky for friendship and all the hours spend redoing the world drinking a beer or two or vine/amaretto.

Two persons remain that I would like to especially thank. First, my dear office mate, Jeff, for his patience because I know that sometimes it is not easy to cohabitate with me in an office. I would also thank him for all the help he provided during my thesis and the programming skills he taught me. Finally, I would like to thank him for the discovery of board games, because killing zombies alongside a dear friend is priceless!

Second, I would like to thank my thesis director, Micah Murray for giving me the chance to do a PhD, his patience, knowledge transmission and finding the right words at the right time to cheer me up when I was in the writing process of the thesis and feeling overwhelmed.

And last but not least, I would like to thank my parents and family who believed in me, gave me the chance to perform higher studies and their unconditional support throughout this thesis. I would like to conclude these acknowledgements by thanking a very special person for me, Anne-Laure for supporting since the first day we met to the final point of this thesis. I would like to thank her for the cheering words, the laughs and finally for the nights she has to bear that I work very late or early in the morning during the last days of my thesis.

If I forget someone, I apologize!

I Brain dynamics of illusory contour perception and perception via sensory substitution in healthy subjects

ABSTRACT (ENGLISH)

This thesis is based three scientific articles. It focuses on the visual system in healthy subjects as well as on the perception of auditory soundscapes coming from a visual-to-auditory sensory substitution device. The first two articles deal with the perception of Kanizsa-type illusory contours (IC) and the underlying brain mechanisms using electroencephalography (EEG). The first paper investigated the brain processes of illusory contours biased towards the magnocellular pathway or the parvocellular pathway of vision. It has shown that illusory contours are perceived independently of the stimulus cue. In addition, this study allowed to demonstrate a phase shift on the order of ~30ms for stimuli biased toward the magnocellular pathway of vision. Finally, the distributed source estimation localized the IC sensitivity within the Lateral occipital cortex (LOC) in both condition. All these data expand current models and demonstrate the existence of cue-dependent circuits in the brain for the generation of IC perception. The second paper investigated the perception of illusory contour forming lines in humans. This study demonstrated that sensitivity to IC forming lines starts around 200 ms post-stimulation and is localized within the LOC. These findings are coherent with a model of IC sensitivity starting within higher-level visual cortices. Finally, the third study focuses on orientation training with a visual-to- auditory sensory substitution device in healthy participants. By training subjects with simple stimuli in central frequency range, this study showed that participants can discriminate the orientation of more complex stimuli independently of their frequencies range (high, central or low) of presentation.

The following introduction presents the essential concepts for understanding the 3 scientific articles that compose this thesis. The first part presents the structure of the eye and the visual tract as well as the cellular aspect, the organization of the visual regions in the cortex and their functional organization. A second part introduces illusory contours and the hypothesized processing models. A third part briefly present lines detection and their processing. Finally, sensory substitution devices are introduced.

The results of the three articles are briefly presented in a chapter with the same name as well as unpublished results.

The last part of this work is the discussion that will be divided along three axes: the discussion of the unpublished results, the discussion of the 2 neurophysiological articles which will propose new experiments and hypothesis to improve our understanding of IC sensitivity. Finally, the discussion of the article on sensory substitution which proposes a new experiment in healthy subjects to link the perception of ICs and sensory substitution devices to investigate whether ICs can be perceived when the stimuli are formed by auditory soundscapes.

Anken Jacques Département des Neurosciences Cliniques March 17

ABSTRACT (FRENCH)

Cette thèse est composée de trois articles scientifiques. Elle porte sur le système visuel chez les sujets en bonne santé ainsi que sur la perception de « paysage sonore » provenant d'un dispositif de substitution sensorielle de la vision par l’audition. Les deux premiers articles traitent de la perception des contours illusoires de type Kanizsa (CI) et des mécanismes cérébraux sous-jacents en utilisant l’électroencéphalographie (EEG). Le premier article a étudié les processus cérébraux de contours illusoires biaisés en faveur de la voie magnocellulaire ou la voie parvocellulaire de la vision. Il a démontré que les contours illusoires sont perçus indépendamment du type de stimulus. En plus de cela, cette étude a permis de montrer un retard de réponse l'ordre de ~ 30ms pour les stimuli biaisés pour la voie magnocellulaire de la vision. Enfin, l'estimation de la source localise la sensibilité au CI dans le cortex occipital latéral (LOC). Toutes ces données développent les modèles actuels et démontrent l'existence de circuits dépendant du stimulus dans le cerveau pour la génération de la perception de l'IC. Le deuxième article a étudié la perception de contours illusoires formant des lignes chez l'homme. Cette étude a démontré que la sensibilité aux lignées formant un CI commence environ 200 ms après la stimulation et est localisée dans le LOC. Ces résultats sont cohérents avec un modèle qui propose que la sensibilité au CI commence dans des cortex de niveau supérieurs. La troisième étude porte sur l’entraînement d’orientation avec un système de substitution sensorielle de la vision par les sons chez les sujets sains. En entrainant les sujets avec des stimulus simple dans des gammes de fréquence centrale, cette étude a permis de montrer que les participants sont capable de discriminer l’orientation de stimuli plus complexes indépendamment de la gamme de fréquence (haute, centrale et basse) à laquelle le stimulus est présenté.

L'introduction suivante présente les concepts essentiels pour la compréhension des 3 articles scientifiques qui composent cette thèse. La première partie présente la structure de l'œil et du tractus visuel ainsi que l'aspect cellulaire, l'organisation des régions visuelles dans le cortex et leur organisation fonctionnelle. Une deuxième partie introduit les contours illusoires et leur modèles de traitement hypothétiques de ces derniers. Une troisième partie présente brièvement la détection des lignes et leur traitement. Enfin, les dispositifs de substitution sensorielle sont introduits.

Les résultats des trois articles sont brièvement présentés dans un chapitre portant le même nom ainsi que des résultats non publiés.

La dernière partie de ce travail est composée par la discussion qui sera divisée en trois axes: la discussion des résultats non publiés, la discussion des 2 articles neurophysiologiques qui proposeront de nouvelles expérience et des hypothèses pour améliorer notre compréhension de la sensibilité aux CI. Enfin, la discussion de l'article sur la substitution sensorielle qui propose de nouvelles expériences chez les sujets en bonne santé pour lier la perception des CI et les dispositifs de substitution sensorielle pour enquêter si la perception de CI est possible lorsque les stimuli son formé par des paysages sonores.

III Brain dynamics of illusory contour perception and perception via sensory substitution in healthy subjects

TABLE OF CONTENTS

ACKNOWLEDGEMENTS I

ABSTRACT (ENGLISH) II

ABSTRACT (FRENCH) III

TABLE OF CONTENTS IV

LIST OF FIGURES VII

LIST OF EQUATIONS VIII

CHAPTER 1 INTRODUCTION 9

1.1 Structure of the Eye and the visual tract 10 1.1.1 Cellular aspect of the visual system 11 1.1.2 Organization of Visual regions within the cortex 11 1.1.3 Functional organization of extra striate visual areas 13 1.1.4 Retinotopy (retinal mapping) in the visual cortex. 14

1.2 Illusory Contours 15 1.2.1 Hypothesized model of Illusory contour processing 16

1.3 Sensitivity to Line orientation 17

1.4 Sensory substitution devices (SSD) 18 1.4.1 Major findings with visual to auditory sensory substitution leading towards a model of Task-specific sensory-independent brain organization. 18

CHAPTER 2 METHODS 21

2.1 Electroencephalography (EEG) 21 2.1.1 Reference problem 21 2.1.2 EEG data Analysis (ERPs, Topographies, and source estimations) 22 2.1.3 Visual EEG components (C1, P1, N1). 25

2.2 The EyeMusic sensory substitution device. 26

Anken Jacques Département des Neurosciences Cliniques March 17

CHAPTER 3 RESULTS 28

3.1 Cue-dependent circuits for illusory contours in humans 28 3.1.1 Abstract 28

3.2 Brain mechanisms for perceiving illusory lines in humans. 28 3.2.1 Abstract 29

3.3 Training Orientation Discrimination Using Visual-to-auditory Sensory Substitution 29 3.3.1 Abstract 29

3.4 Unpublished Results for experiment 3.2 Brain mechanisms for perceiving illusory lines in humans 30 3.4.1 Lateralization of stimuli within the parafovea. 30 3.4.2 Orientation detection of Illusory lines. 33

CHAPTER 4 GENERAL DISCUSSION 35

4.1 Discussion of unpublished results. 35 4.1.1 Lateralization of stimuli within the parafovea. 35 4.1.2 Orientation detection of Illusory lines. 36

4.2 Neurophysiological Studies 37 4.2.1 Cue-dependent circuits for illusory contours in humans 37 4.2.1.a Summary of general conclusion 37 4.2.1.b General comments and Directions 38 4.2.2 Brain mechanisms for perceiving illusory lines in humans 42 4.2.2.a Summary of general conclusion 42 4.2.2.b General comments and future Directions 43

4.3 Sensory substitution 47 4.3.1 Training Orientation Discrimination Using Visual-to-auditory Sensory Substitution 47 4.3.1.a Summary of general conclusion 47 4.3.1.b General comments and future Direction 48

CHAPTER 5 CONCLUSION 51

5.1 Conclusion on neurophysiological papers 51

5.2 Bridging the gap between Sensory substitution, perceptual completion and the use of sensory substitution in every day life situations. 51

V Brain dynamics of illusory contour perception and perception via sensory substitution in healthy subjects

CHAPTER 6 REFERENCES 52

CHAPTER 7 ARTICLES 60

7.1 Cue-dependent circuits for illusory contours in humans 60 7.1.1 Abstract 61 7.1.2 Introduction 62 7.1.3 Material and Methods 63 7.1.4 Results 69 7.1.5 Discussion 73 7.1.6 Conclusions 76 7.1.7 Acknowledgements 77 7.1.8 References 78

7.2 Brain mechanisms for perceiving illusory lines in humans 82 7.2.1 Abstract 83 7.2.2 Introduction 84 7.2.3 Materials and Methods 85 7.2.4 Results 89 7.2.5 Discussion 90 7.2.6 Acknowledgements 92 7.2.7 References 93

7.3 Training Orientation Discrimination Using Visual-to-auditory Sensory Substitution 96 7.3.1 Abstract 97 7.3.2 Introduction 98 7.3.3 Material and Methods 99 7.3.4 Results 103 7.3.5 Discussion 105 7.3.6 Conclusion 107 7.3.7 References 108

Anken Jacques Département des Neurosciences Cliniques Decembermars 17

LIST OF FIGURES

Figure 1: Strucuture of the Eye and visual tract...... 10 Figure 2: Anatomical the location of visual cortical areas...... 12 Figure 3:Visual system of the macaque monkey...... 13 Figure 4: Scheme of the Ventral and dorsal streams of vison...... 14 Figure 5: Retinotopic organization within primary visual cortex of macaque monkey...... 15 Figure 6: Kanizsa-type illusory contour forming a square shape...... 16 Figure 7. The three hypothesized model of Illusory contour processing in Humans...... 17 Figure 8. Schematic representation of the link between Dipoles and Topographies ...... 24 Figure 9: VEP correlates of Localization effect as revealed by the electrical neuroimaging analysis framework...... 31 Figure 10: Distributed source estimation of the localization effect over the 200-250ms period...... 32 Figure 11: VEP correlates of IC Line orientation detection as revealed by the electrical neuroimaging analysis framework ...... 33 Figure 12: Schematic of the "Fat" IC shape, "Thin"IC shape and the classical kanizsa illusory square...... 37 Figure 13: Schematic of an array of pacmen forming an upward (a.) and leftward (b.) IC of equilateral triangle...... 44 Figure 14: Schematic of a verticl array of pacmen inducers shifted toward the left / right part of the parafovea (5° of visual angles)...... 45 Figure 15: Schematic of a bisected line surrounded by (a.) Salient Region inducers and (b.)Kanisza-type Illusory Contours ...... 47 Figure 16: Schematic of stimuli for sensory substitution experiment ...... 49

VII Brain dynamics of illusory contour perception and perception via sensory substitution in healthy subjects

LIST OF EQUATIONS

Equation 1. The Global Field Power (GFP)...... 22 Equation 2. The Global Dissimilarity...... 24

VI Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

CHAPTER 1 INTRODUCTION

This thesis focuses on two distinct topics. First, illusory contour perception which is an example of completion. A biological explanation for the evolution of completion might come from the prey and predator’s system. On the one hand, the prey needs to recognize predators to avoid them or alternatively try to camouflage not to be detected and stay alive. The principle of camouflage is to leave the least visual cues so that the detection of body’s contour is difficult or even impossible. Some animals became masters of camouflage, for example the peppered moth. On the other hand, while preys were evolving their camouflage, the visual system of predators (e.g. birds for peppered moth) started to evolve to become masters of breaking camouflage. To break the camouflage, predators started to rely on objects boundaries and luminance contrast. By grouping the objects boundaries that are incomplete or absent the predators can recreate the shape of the prey and detect it. This biological explanation of contours perception has been made in 1987 by Ramachandran. Completion and illusory contour perception happens in everyday life situations with objects in the natural world that are often partially visible (as in the example of prey and predators). Objects edges might be absent due to differences in surface or absence in contrast gradients. To investigate the neural basis of illusory contour perception in human, the Kanizsa-type illusory contour model was used in the two first scientific papers presented in this thesis (see chapter 1.2).

The first paper investigated the perception of illusory contours presented in isoluminant chromatic contrast (biased toward the parvocellular pathway) or achromatic low luminance contrast (biased toward magnocellular pathway). The question addressed by this paper was to identify whether illusory contours can be perceived when the stimulus is biased toward one specific pathway of vision. The second paper investigated the perception of illusory contours forming line presented within the parafovea (5° visual angle). The question addressed in this study was to identify the brain mechanism underlying the perception of illusory contours forming line in human.

The Second topic are the Sensory substitution devices. As the number of blind individuals continues to rise worldwide, it appears important to develop apparatus that can helps them retrieve vison or at least a sense of vision. Among sensory substitution devices, the visual-to-auditory sensory substitution devices are using sounds to convey information within visual areas (see chapter 1.4). These apparatuses appear promising for the rehabilitation of blind individuals and are the subject of third paper presented in this thesis.

The aim of the third paper was to determine if sighted subjects can understand the basis of transformation applied by a visual-to-auditory sensory substitution device called the EyeMusic after a specific training on the vertical orientation with visual feedback.

The following introduction aims at giving a sufficient background to understand the three scientific papers comprised within this thesis. This introduction is composed of four major parts: The first part is an overview of the visual system and its complexity. The second part introduces illusory contours and their hypothesized processing models. The third part briefly present lines detection in animal and human

9 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. models and their processing. Finally, the sensory substitution device and the state of the research in the case of visual-to-auditory sensory substitution.

1.1 Structure of the Eye and the visual tract

The visual system is one of the most efficient and powerful senses in humans. The human eye is able to see tiny objects such as ants as well as big objects such as buildings. The visual system can also distinguish details and colors when there is enough light. In opposite, in presence of dim light, for example a clear night with a full moon, the human eye can distinguish the rough form of objects. These abilities are based on the reflection of light off an object. This reflection enters through the pupil and hits a light sensitive tissue in the rear of the eye: the retina. The light is detected by photoreceptors placed on the retina, namely the cones and rods. On the one hand, the S-cones, the M-cones and the L-cones are dedicated to color vision in daylight conditions (photopic vision). These photoreceptors differ by the wavelength they detect, respectively short (400 to 540 nm, blue), medium (440 to 670 nm, green) and large (from 410 to 690 nm, red). The cones are mainly concentrated in the fovea (center of the retina). On the other hand, rods allow the vision under dim light condition (scotopic vision). This vision in mostly black and white. Rods outnumber cones in the retina. They are mostly localized in the periphery of the retina.

When photoreceptors at the retina receive a light signal, they transform it into an electrical signal. This electrical signal is further transmitted to the retinal ganglion cells (RGC) via bipolar and amacrine cells. The RGC transfer the electric signal to the lateral geniculate nucleus (LGN) and the superior colliculus. Finally, from the LGN, the signal is transmitted to the primary visual cortex. The visual field of each eyes is split into right and left visual field. The left part of the visual field of each eye projects to the visual cortex of the right hemisphere (i.e. contralateral) whereas the right part of the visual field projects to the visual cortex of the left hemisphere (i.e. contralateral), this will be further developed in chapter 1.1.5.

Figure 1: Strucuture of the Eye and visual tract. a. displays an eye and the cellular composition of the retina with Cones, rods and retinal ganglion cells. b. representation of the visual tract from the retina to the primary visual cortex with the contralateral activation in the primary visual cortex (downloaded from https://www.sciencenews.org/article/how-rewire-eye and http://www.edoctoronline.com).

10 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

1.1.1 Cellular aspect of the visual system

For this thesis, the focus will be done on the midget RGC and Parasol RGC. On the one hand, midget RGCs receive inputs from one cone and multiple rods. They represent approximately 80% of the RGCs. Midget RGCs project to the parvocellular layers of the lateral geniculate nucleus (LGN) and further to the ventral visual stream (see chapter 1.1.3). Therefore the midget RGCs are also called the parvocellular cells (P-cells). These cells were anatomically characterized as being rather small. Their conduction velocity is slow. They have been demonstrated to be sensitive to high contrast stimuli (>8%) and also respond to chromatic contrast (Kaplan, 1991). They are also responsive to high spatial frequencies (Dacey and Petersen, 1992; Jindra and Zemon, 1989). On the other hand, the Parasol RGCs were anatomically characterized as big cells. They receive inputs from many cones and rods. These ganglion cells project to the magnocellular layers of the LGN. further project to the dorsal visual stream of vision (see chapter 1.1.3). This is why they are also referred to as the magnocellular cells (M-Cells). They are sensitive to low contrast stimuli (<8%) and respond to achromatic stimuli (Kaplan, 1991). They are sensitive to low special frequencies (Dacey and Petersen, 1992; Jindra and Zemon, 1989). Both the P- cells and M-cells are characterized by center-surround receptive fields. A receptive field is defined, in the case of vision, as the region of the visual field in which the stimulus elicits the triggering of a response in a given cell or region from which we record. Receptive fields are present at the level of photoreceptors throughout the visual system. Generally speaking, the further on the visual pathway, the bigger the receptive field. At the level of the visual cortex, receptive fields are larger, and more complex stimuli are required than at the level of RGCs to stimulate them. Nevertheless, receptive fields in the visual cortex have been classified into three types: simple cell, complex cell and hyper complex cell (Hubel, 1963). Simple cell receptive fields are sensitive to orientation and position. They are localized in Brodmann area 17 (primary visual cortex). Complex cell receptive fields are sensitive to orientation, motion, and direction. They are localized within Brodmann area 17,18 (primary visual cortex, cuneus, lingual gyrus, and lateral occipital gyrus). Finally, the hyper complex cell receptive fields are sensitive to orientation, motion, direction and length. These receptive fields are localized within Brodmann area 18,19 (cuneus, lingual gyrus, lateral occipital gyrus, and superior occipital gyrus).

1.1.2 Organization of Visual regions within the cortex

The first step in understanding how the brain processes visual information is to have knowledge on the visual cortex’s organization. The following chapter introduces a selected number of visual cortical areas, their localizations, and their connections. It is important for the reader to note that many other visual regions exist but are not introduced here to keep things simple. The primary visual cortex (V1), also called the striate cortex, is the gateway for inputs coming from the lateral geniculate nucleus. It is located in the most occipital part of the brain around the calcarine sulcus (Brodmann area 17; see Figure 2, left hand side). Anatomically speaking the primary visual cortex is surrounded by the secondary visual cortex, also called peristriate cortex or V2 (Brodmann area 18; see Figure 2, left hand side). V2 is the regions in which V1 projects the majority of its neurons (Felleman and Van Essen, 1991). Properties of V1 and V2 are very similar. They have been identified as responsible for the extraction of low-level features as well as extracting physical information of visual stimuli in macaque monkey (Hubel and

11 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

Wiesel, 1977). V1 and V2 neurons also present early sensitivity to luminance (Romani et al., 2003; Tootell et al., 1998) and sensitivity to line orientation (von der Heydt et al., 1984; von der Heydt and Peterhans, 1989a). The sensitivity to line orientation is discussed in the paper presented in 7.2. V2 is already considered as part of the extra-striate cortex. Alongside with V2, the extrastriate cortex is composed of the third visual complex (V3), V4 and middle temporal visual area (V5) in human (Figure 2). These regions are localized in the periphery of V2 except V5 that is localized in the middle temporal part of the brain (Figure 2, right hand side). Each of these visual areas have specific roles: V3 is believed to treat information of dynamic and static shapes (Tootell et al., 1997), V3A is an additional sub division of V3. V3A has been linked to contrast sensitivity and motion perception to a lesser extent (Tootell et al., 1997). V4 respond to forms and colors (Tootell et al., 2004; Zeki et al., 1991), finally V5 is sensitive to motion (Tootell et al., 1995; Zeki et al., 1991).

Figure 2: Anatomical localization of visual cortical areas. The left image displays a sagittal cut of the brain with the posterior part on the left and the anterior part on the right. The right image displays a left-brain hemisphere with the anterior part on the left and the posterior part on the right. Areas V3 (dark blue), V3A (violet), V4 (red) and V5 (pink), and those of face and object recognition (blue), receive their input largely from V1 (yellow) and V2 (green) (Adapted from (Zeki, 2003)).

The face and object recognition area in figure 2 is composed of multiple sub regions including the fusiform face area (FFA), the extrastriate body-selective area (EBA), the visual word form area (VWFA), the visual number form area (VNFA) and the Lateral occipital cortex (LOC). Here, only the LOC is introduced. The VWFA, VNFA and EBA are discussed further for their role in chapter 1.4.1. The LOC extends ventrally and dorsally from the lateral bank of the fusiform gyrus. It is a high-level cortical area. It was described for the first time by Malach and colleagues in 1995 in an fMRI study that compared the response of subjects that were looking passively at photographs of everyday objects and photographs of textures that did not contain interpretable shapes. The LOC has been demonstrated to play a major role in object recognition (for a complete review see (Grill-Spector et al., 2001)). Here it is important to notice that the LOC has been demonstrated to be sensitive to illusory contours in humans with fMRI (Hirsch et al., 1995; Mendola et al., 1999; Seghier et al., 2000), magnetoencephalography (Halgren et al., 2003) and Electroencephalography (Murray et al., 2002b; Murray et al., 2004; Murray et al., 2006). This specific role of the LOC is further developed in Chapter 1.2.1 and is a central point in papers presented in 7.1 and 7.2.

An analysis of the visual cortex was performed in the macaque monkey by Felleman and Van Essen in 1991. In this paper, they divided the extra striate cortex into 32 different areas. They also investigated

12 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. the nature of cortico-cortical connections between these 32 areas. Figure 3 displays these 32 areas and the inter-connections between them. An important point to mention is that connections in Figure 3 have no arrows to indicate their sense, because cortico-cortical connections are bi-directional. The following text introduce the “bottom-up” and the “top-down” processes that rely on these cortico-cortical connections. On the one-hand, the “bottom-up” processes rely on feedforward architectures that run from low-level visual areas to higher level visual areas to modulate the response in these latter regions. On the other hand, “top-down” processes rely on 2 different architectures. First, feedback architectures that run from high-level cortices to low-level visual areas to modulate their responses. Second, lateral architectures that run from two cortical areas that are considered to have the same hierarchical level. For example, in macaque monkey, several dorsal areas are considered to be at the same hierarchical level as the inferotemporal (IT) regions (Felleman and Van Essen, 1991; C E Schroeder et al., 1998). In summary, bottom-up processes are referred to as feedforward modulations whereas top-down processes are also referred to as feedback modulations. Feedforward and feedback modulations are discussed for their implications in the different models of illusory contour processing in Chapter 1.1.2.

Figure 3:Visual system of the macaque monkey. Organization of the cortico-cortical connection between the 32 visual areas of the macaque monkey (from (Felleman and Van Essen, 1991).

1.1.3 Functional organization of extra striate visual areas

As mentioned previously, the paper presented in 7.1 focuses on P-cells and M-cell. The P-cells project to the parvocellular layers of the LGN and progress further to the extrastriate cortex and the ventral visual stream. This stream of vision runs from the primary visual cortex to the inferior occipital and lateral occipito-temporal cortices (Ungerleider and Mishkin, 1982) (Figure 4). The ventral stream is thought to perform object recognition and was demonstrated to respond selectively to colors. Because of its role in object recognition, the ventral stream is also called the “What” pathway.

M-cells project mainly to the magnocellular layers of the LGN and progress further to the dorsal subdivision of the extra-striate cortex. The dorsal stream runs from the striate cortex (V1) to the parietal

13 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. lobe (Ungerleider and Mishkin, 1982) (Figure 4). This stream of vision is thought to play a major role in the spatial aspect of vision. More precisely, it is involved in the localization of objects and analysis of motion. Therefore, it is also called the “where” pathway. Previous studies supposed that the detection of information begins within the primary visual cortex and is processed afterward in parallel within the ventral and dorsal pathways (Ungerleider and Mishkin, 1982). This parallel processing of information implies that object recognition can be performed without their localization and the inverse is also true. In the case of agnosia (i.e inability to process sensory information) for example, it has been shown that the special localization (dorsal) remains untouched while the object recognition (ventral) is impaired. The inverse was also reported (Ungerleider and Mishkin, 1982).

Another important finding in the functional organization of ventral and dorsal streams comes from monkey studies. Schroeder and colleagues demonstrated latency differences between ventral and dorsal streams of vision (C. E. Schroeder et al., 1998). They also demonstrated that first inputs into the ventral stream present responses that correspond either to lateral connection profile or top-down profile rather than a bottom-up profile. These findings suggest that the dorsal visual stream of vision could influence the treatment of information in the ventral stream of vision. Although evidence of sensitivity of the dorsal pathway to magnocellular inputs and sensitivity of the ventral pathway for the parvocellular inputs exists, these sensitivity preferences are not completely segregated (Merigan and Maunsell, 1993). This would suggest that ventral and dorsal pathway are connected by both M-cells and P-cells. The processing of information within parvocellular and magnocellular pathway in the particular case of illusory contour perception is discussed in the paper presented 7.1.

Figure 4: Scheme of the Ventral and dorsal streams of vison. (adapted from Wikipedia).

1.1.4 Retinotopy (retinal mapping) in the visual cortex.

Retinotopy refers to the representation of visual inputs from the retina up to the neurons of the visual cortex. In other words, the neurons in the visual cortex from V1 to V5 are specifically organized, so that an image present onto the retina is structurally represented in neurons of the visual cortex. Moreover, neighbor points onto the retina are represented in neighboring areas in the visual cortex. The fovea is represented in the primary visual cortex, and its surrounding is represented around the primary visual cortex from V2 to V5. It is important to note that the retinotopic representation is reversed compared the image on the retina (Figure 5). Several studies also reported a possible retinotopic organization that is maintained within the LO regions (Large et al., 2008; Sasaki et al., 2001), this feature of the LOC will be discussed further in chapter 4.1.1.

14 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

Figure 5: Schematic of the retinotopic transformation from stimulus (A) to striate cortex (B) in macaque monkey. 1,2,3 are selected regions from the fovea (1) to the periphery (2). VI are the vertical inferior rays, VS are the vertical superior rays. H is the horizontal meridian and OI/OS the oblique inferior rays and oblique superior rays, respectively (adapted from (Tootell et al., 1988)).

Now that the visual system has been introduced from the retina up to the retinotopic organization within the primary visual cortex, the next chapter will introduce the illusory contour and their hypothesized models of processing.

1.2 Illusory Contours

Humans are able detect contour even if the borders are absent or incomplete. This ability is known as Illusory contour perception and happens in everyday life situations. A good model to investigate this phenomenon is the kanizsa-type illusory contours (Kanizsa, 1976). In this model, ‘pacmen’ inducers mouths can be oriented to face each other to create an illusory shape or oriented in opposite directions to form a non-contour equivalent. For example, Figure 6 displays a kanizsa-type illusory contour forming a square. In this figure, the square is illusory because it is not physically present on the retina. The white background in the square and out of the square are the same luminance although they seem different. In this example, to recreate the shape, the visual system groups the ‘pacmen’ that are physically present on the retina and consider the image as a whole. By doing this, the visual system completes the missing borders to give the impression of a square. This phenomenon is called binding or completion. In neurophysiology, binding refers to processes in which different information are coded to refer to the same object. In the case of illusory contours, the ‘pacmen’ inducers that are spatially separated are coded together to create the illusion. This binding process remains controversial. Although several studies used kanizsa-type illusory contour to investigate this process, the question of top-down vs bottom-up processes remains (Seghier and Vuilleumier, 2006). Some reported the perception of illusory contour to be associated with activity in early visual areas (V2) and concluded that occipital cortex can contribute to the perception of IC (Ffytche and Zeki, 1996; Ohtani et al., 2002) without higher order cognitive function. This would support a pre-attentive model (i.e bottom-up) of Illusory contour sensitivity. Other studies reported sensitivity to IC within higher order brain regions such as the LOC (Mendola et al., 1999, p. 199; Murray et al., 2004; Murray et al., 2006). This would support a model of IC processing that requires attention (i.e top-down).

15 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

Now that illusory contours have been introduced, the next chapter will present the principal findings of illusory contour processing in animals and humans and the hypothesized models of processing ensuing.

Figure 6: Kanizsa-type illusory contour forming a square shape.

1.2.1 Hypothesized model of Illusory contour processing

Kanizsa-type Illusory contours have been widely used in animal research as well as human research. On the one hand, studies in macaque monkey reported activation in the primary and secondary visual cortex in response to the presentation of illusory contours with intracranial recordings (Grosof et al., 1993; von der Heydt et al., 1984). Similar findings were reported in cats (Redies et al., 1986) and owls (Nieder and Wagner, 1999). From these studies, the first model of illusory contour processing was suggested. This model states that sensitivity to illusory contour takes place in primary visual areas (V1/V2) and mediate IC sensitivity in ”bottom-up” manner (Figure 7, model 1).

Later intracranial studies in monkey with electrodes implanted in the inferior temporal cortex (IT) reported activation to IC (Sáry et al., 2008, 2007). Likewise, studies investigating the perception of IC in humans reported activity within the lateral occipital cortex; the human equivalent of monkey IT. Activation within the LOC in human have been made with fMRI (Mendola et al., 1999), MEG (Ohtani et al., 2002), and EEG (Anken et al., 2016a; Knebel et al., 2011a; Murray et al., 2002b; Murray et al., 2004; Murray et al., 2006; Yoshino et al., 2006). This IC sensitivity within the LOC also finds support in lesion studies (Vuilleumier et al., 2001a; Vuilleumier and Landis, 1998) and schizophrenia studies in human (Foxe et al., 2005; Knebel and Murray, 2012). An electroencephalographic correlate of IC sensitivity has been identified: the IC effect. The IC effect is defined as the subtraction between the presence of IC and its counterpart part (IC-NC). It onsets at ~90ms post-stimulus and peaks at~150ms post-stimulus. It is localized in the bilateral LOC (Murray et al., 2002a). Moreover, The IC effect appears to be robust across a wide range of stimulus parameters like shape (Murray et al., 2002, exp 1-3), localization in the visual field (Murray et al., 2002, exp 5), luminance (Anken et al., 2016b), retinal eccentricity, ratio of real to illusory contour, and inducer diameter (Altschuler et al., 2012). From these findings, a second model propose that IC sensitivity occurs within the LOC (within the ventral visual stream (Ungerleider and Mishkin, 1982)) which modulates responses in V1/V2 via feedback (Figure 7, model 2).

Finally, other studies reported activation in the LOC in response to salient regions (Stanley and Rubin, 2005, 2003). Salient regions come from the observation that in real world scenes, bounding contours are hard to delineate due to light condition, surface similarity or blurring effects. This led to a third model that postulates sensitivity to salient regions within the LOC, and IC sensitivity within V1/V2 after feedback modulations from the LOC. (Figure 7, model 3).

16 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

The next subchapter will present the sensitivity to line in animal models and human models. This is important for the paper presented in chapter 7.2 and the unpublished results presented in chapter 3.4.2 that used illusory contour forming lines in human.

Figure 7. The three hypothesized model of Illusory contour processing in Humans. The first model supports a “bottom-up” modulation from V1/V2 to LOC and parietal cortex. The second model supports a Illusory contour differentiation that starts in LOC and modulates activities in other regions via “top-Down” communication. The third model supports salient region differentiation in LOC and a differentiation of Illusory contour in V1/V2 “top-down” modulated by LOC. (adapted From Knebel JF et al., 2012).

1.3 Sensitivity to Line orientation

An important feature of the visual cortex is orientation tuning. One of the first evidence of neurons’ orientation tuning in primary visual cortex came from intracranial recordings in cat visual cortex (Hubel and Wiesel, 1962). This orientation specificity has since been extensively studied. Animals and human studies allowed to discover a specific pattern of response called the “oblique effect” (Appelle, 1972). This oblique effect is characterized by a better visual resolution when stimuli are oriented vertically or horizontally in comparison to oblique stimuli. This effect has been confirmed in many studies in human (Furmanski and Engel, 2000; Li et al., 2003) and animals ((De Valois et al., 1982; Payne and Berman, 1983) where a larger number of cells were responding to horizontal and vertical orientations compared to oblique orientations. Many studies tried to localize which brain areas were sensitive to the orientation of line. Among these studies, An important intracranial study in monkey investigated the response of V1 and V2 to real bars and anomalous contour (i.e illusory lines that were created by misaligning two sets of horizontal bars) reported V1 neurons activations in response to real bars of the preferred orientation whereas V2 neurons were responding to anomalous contour of preferred orientation (von der Heydt and Peterhans, 1989b). Similar responses were reported in other species (Nieder and Wagner, 1999; Redies et al., 1986; Sheth et al., 1996; Song and Baker, 2006). Likewise, human visual cortices present preference for vertical and horizontal lines (Aspell et al., 2010; Mannion et al., 2010; Ohtani et al., 2002; Sasaki et al., 2001).

17 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

This line orientation tuning has also been investigated in animal with visual stimuli similar to Kanizsa- type illusory contours. Intracranial recordings in monkey were performed and presented responses to illusory contour forming line within V2 (von der Heydt et al., 1984). Later studies in macaque monkey also found responses of V1 neurons to illusory contour forming lines (Grosof et al., 1993). In summary, V1 has been demonstrated to be sensitive to real bars as well as Kaniza type illusory line and V2 to be sensitive to illusory line formed by misaligned gratings and Kanizsa-type illusory lines. Although the sensitivity to Kanizsa-type illusory contour forming line appears to be localized within V1/V2 in monkey, it remains unknown to which extent it is true for humans. This open question of the sensitivity to illusory contour forming line in human is the subject of the paper presented in 7.2. In more, the sensitivity to orientation detection of illusory lines in human is investigated in chapter 3.4.2.

Now that Illusory contour processing and the detection of lines have been introduced, the last part of this introduction will jump into a different topic which are the sensory substitution devices that were used in the paper presented in 7.3.

1.4 Sensory substitution devices (SSD)

Over the last decades, researchers have developed many approaches to reconstruct vision or at least a sense of vision. Among these approaches, surgical implantation of retinal prosthesis (SecondSight Argus® II: (Luo and da Cruz, 2016)) or gene therapy (Busskamp et al., 2010) can be cited. Shortcomings of these approaches are their expensive costs as well as their final sight restoration that remains low- resolution.

Several other systems have been developed to help restoring vision, like sensory substitution devices. The basis of sensory substitution is to compensate the deficit of a sense by conveying information through a non-impaired modality. In 1969, Paul Bach-y-Rita presented the first tactile to visual sensory substitution device (TVSS). In a following paper using this TVSS, the first evidences that blind individuals can perceive the world with another organ than their eyes was presented (Bach-y-Rita et al., 1969a). Other types of sensory substitution have been developed since then. The most developed systems are the visual to auditory sensory substitution device in which blind individuals receive sound stimuli to convey the information to the visual system. The main visual-to-auditory sensory substitution devices are the vOICe (Meijer, 1992), the Prosthesis Substituting Vision for Audition abbreviated PSVA (Capelle et al., 1998) and the EyeMusic (Abboud et al., 2014a). Chapter 2.2 presents the functioning of the EyeMusic which is the sensory substitution device that was used in this thesis.

1.4.1 Major findings with visual to auditory sensory substitution leading towards a model of Task-specific sensory-independent brain organization.

This part will be organized as follow, first findings made in specific brain regions in sighted individuals are presented and directly followed by evidences of activation of the same brain regions in blind individuals using sensory substitution (tactile or visual). This part will only focus on four specific

18 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. regions, the visual word form area (WVFA), the visual number form area (VNFA), The extrastriate body-selective area (EBA) and the Lateral occipital cortex (LOC).

The Visual word form area (VWFA) is a specific functional brain region localized in the left lateral occipitotemporal sulcus. The WVFA has been shown to be activated when sighted individuals are reading. This region is specialized in the processing of letter strings (Cohen et al., 2000; Dehaene et al., 2010; McCandliss et al., 2003). This role in reading was confirmed by patients with pure alexia condition (i.e patients with severe reading problems but with intact naming and writing) who showed lesion in the visual word form area site (Cohen et al., 2003). A particular study, reported greater activations of the WVFA to written words in comparison to objects (Szwed et al., 2011). In blind individuals, the visual word form area has first been identified with braille reading, considered as tactile-to-visual sensory substitution (Reich et al., 2011). With visual-to-auditory substitution device, it has been shown that blind individuals can discriminate the orientation of Snellen’s E (Striem-Amit et al., 2012b), moreover their accuracy was even better for the same task when color was added to the side bar of the letter “E” (Levy-Tzedek et al., 2014). Neural correlate of blind individuals identifying letters delivered with sounds from the vOICe sensory substitution device also reported activation within the VWFA (Striem- Amit et al., 2012a).

The visual number form area (VNFA) is localized in the right inferior temporal gyrus (rITG). This region has recently been correlated with a preference for the vision of numbers. The existence of such a region was hypothesized on the basis of a case study of a patient with deficit for numbers (Cohen and Dehaene, 1991). The existence of this region was confirmed with an electrocorticography study in which the response in this area was greater when subjects were looking at numbers compared to letters (Shum et al., 2013). It is important to note here that the separation of the VWFA and the VNFA is not always completely clear. A TMS study reported interruption of early visual number perception when pulse was applied on the right VNFA, but this TMS pulse also interfered with the letter perception (Grotheer et al., 2016). This latter study nevertheless indicated a preference of the VNFA for numbers and familiar symbols. This finding revealed two different role for this region. In Congenitally blind individuals (i.e Individuals that never saw from birth), the VNFA has been reported in an fMRI study using the EyeMusic sensory substitution device (Abboud et al., 2015). In this study, blind individuals were listening to sounds that could either be letters or numbers of different colors. Greater activation in the rITG were reported when congenitally blind individuals perform a Numeral detection task in comparison with other tasks (Abboud et al., 2015).

The Extrastriate body-selective area (EBA), is localized in the posterior inferior temporal sulcus and middle temporal gyrus. This region appears to be specialized in the perception of body part and human body (Downing et al., 2001). Other studies confirmed this specific role of EBA, when sighted subjects were looking at body vs. other objects (Striem-Amit and Amedi, 2014). Apparently congenitally blind individuals also show an EBA preference for full-body silhouettes (Vs. other objects) when receiving an auditory stimulation from the vOICe sensory substitution device (Striem-Amit and Amedi, 2014).

The Lateral occipital cortex has already been introduced for his role in the detection of objects and illusory contour sensitivity with visual stimuli. Here its role for the recognition of object presented in a haptic or auditory manner will be addressed. It has been demonstrated that a specific part of the LOC, is activated when sighted individuals touch an object in comparison to a random texture (Amedi et al.,

19 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

2001). At the time the author postulated that the LOC might constitute a “multimodal object-related network”. More recent studies also support these activation of the LOC in the tactile recognition of objects (Amedi et al., 2010; Lacey et al., 2010; Stilla and Sathian, 2008). Similar activation of the LOC were reported in sighted individuals and blind individuals who extracted the shape information from auditory stimuli coming from the vOICe sensory substitution device after training (Amedi et al., 2007).

Taken together, these studies show the potential of SSD for rehabilitation, but not only; these findings also tend to operate a paradigm shift in the functional perception that we have of the brain. The traditional neuroscience view is that the human brain is divided into “specific” cortex, depending on the sensory modality that elicits it. So to say the visual cortex for vision, the auditory cortex for audition, and the somato-sensory cortex for touch, etc. In this model, higher-order multisensory areas integrate the information from the previously mentioned unimodal areas. It is the principle of sensory division- of-labor (Zeki, 1978). However, the recent finding made with sensory substitution devices are making sort of a paradigm shift toward a model of Task-specific sensory-independent (TSSI) brain organization. In this model, the brain is thought to be a flexible task machine that will recruit the best area(s) to extract the maximum information from a stimulus, irrespective of the type of stimulus (auditory, tactile, visual) and the region to which it should be bound based on the division-of-labor model (Heimler et al., 2015). This pattern of TSSI brain organization is also supported by studies in healthy participants who are deprived of vision or sound (Collignon et al., 2011; Ricciardi et al., 2014).

20 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. CHAPTER 2 METHODS

2.1 Electroencephalography (EEG)

Electroencephalography is a non-invasive technique that allows recording the brain’s electrical activity at the level of the scalp. It records the electrical field generated by post-synaptic potentials. However, only the measurement of the sum of synchronous post-synaptic potentials of large groups of neurons is possible at the level of the scalp. It is important to understand here that an electrode at the scalp not only records the brain response of the underlying region but records the whole activity of the brain (e.g. for example, a frontal electrode does not only record frontal activity, but also activity from other parts of the brain). This neuroimaging technique can identify differences in response strength, latency modulations (precision on the order of the millisecond), topographical modulation at the scalp and allow to estimate the brain sources. Electroencephalography is physically measured as a difference of potential at the level of the scalp and is typically represented as a trace across time for each electrode. This potential difference measurement requires a reference in order to be calculated. However, the choice of the reference can be problematic. The issues in the choice of references are introduced in the next chapter.

2.1.1 Reference problem

Classically, recording references are taken on the nose or the mastoids in EEG recordings. Some recording systems use “two” electrodes to create a loop that simulate a zero potential like the “ActiveTwo” system of Biosemi (http://www.biosemi.com). The earth reference is generally avoided due to its poor signal. As mentioned above, these multiple choices of reference is problematic in EEG recording, because depending on the localization of the reference the waveforms (i.e. the brain responses measured at each electrodes) can be dramatically changed in their local maxima, minima and amplitude (Cf. figure 1 in (Murray et al., 2008))). To address this problem in a didactic way, a practical parallel with the height of mountains can be made. For example, if the sea level (0m) is taken as the reference, the Matterhorn mountain will peak at 4478m high. However, if the Leman Lake (372) is taken as the reference instead of the sea level, the Matterhorn will peak at only 4106m high (4478m-372m). In this example, the Matterhorn mountain height has changed in function of the reference. This height modification is the same with local peaks of electrical waveforms recorded in an EEG setup. Nevertheless, it is required to have a reference for the recordings of electroencephalographic traces. To bypass this reference problem, the average reference is usually applied offline (after the recording). The average reference is defined as the mean across electrodes (Michel et al., 2004; Pascual-Marqui and Lehmann, 1993). It is the best approximation of the electrical potential at the level of the scalp. This recalculation of the data against an average reference is also important for the calculation of EEG source imaging (Grave de Peralta Menendez et al., 2004a).

Now that the problem of the reference has been addressed, the next chapter will introduce the analysis of EEG as it was performed for this thesis. It is important to note that classical EEG analysis are based

21 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. on single electrodes analysis. This kind of analysis are limited in their neurophysiological interpretability. Therefore, methods that have a better neurophysiological interpretability, like global field power, scalp topography analysis and source estimations were developed and are the subject of the following chapter.

2.1.2 EEG data Analysis (ERPs, Topographies, and source estimations)

High density EEG rests on 64 up to 256 channels and is classically used in research. The scientific papers presented in chapter 7.1 and 7.2 used 128 channels during recording. Paradigms were based on event-related potentials methods. Event-related potentials (ERP) are time locked to the presentation of stimulus to the participants. The brain response to event-related potential is much smaller than environmental noise (e.g. electromagnetic noise of the screen used for stimuli presentation, muscle artefacts, etc). To solve this problem, averaging method is applied to increases the signal-to-noise ratio and allow a better visibility of the ERP (Grave de Peralta Menendez et al., 2004a). Practically, a period from -100 ms up to 800 ms is taken over each trial; this is an epoch. All epochs of a given stimulus are then averaged together at each channel to obtain an ERP for each stimulus. It is also during this processing step that the data are re-calculated against the average reference. Because data are recalculated against the average reference, it is important to interpolate the “bad” channels (due to broken electrodes, poor impedance or an excessive thickness of the skull under an electrode). The interpolation is a process during which a normal behavior of the “bad” channel is recalculated based on the behavior of neighboring channels. The interpolation is applied for two main reasons (Perrin et al., 1987). First, if these channels are not interpolated, they can introduce noise into the data. Secondly, it is important to interpolate them and not removing them to keep the maximal information.

Now that ERPs have been calculated, analysis can be performed at three different levels: 1. Global Field Power (GFP), where the measure is the strength of the signal; 2. Topographic analyses (Microstate), where the measure are the scalp electrical maps; 3. Source estimations; where the measure is the estimated brain activity. GFP and topographic analysis are independent of each other. Source estimations are linked either to the GFP or the topographies. These three levels of analysis are presented separately in the following text.

The first Level of analysis is the Global Field Power (GFP). The GFP has been introduced in the 80’s by Dietrich Lehman and Wolfgang Skrandies. It represents the standard deviation across all electrodes of the montage at a given time point (Lehmann and Skrandies, 1980) (Equation 1).

Equation 1. The Global Field Power (GFP). n is the number of electrodes, ui is one electrode of one condition, u is the average across electrodes

22 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

The GFP is a non-linear (the GFP of the mean is not equal to the mean of GFPs) and reference free measurement. To address the independency of the GPF from the reference, a didactic parallel with geographic topography maps used to hike in the mountain is presented. On geographic topography maps, one can find lines that indicate how flat or how steep a given region is. If the Matterhorn is taken as a mountain example, the relative peak’s height depends on the chosen reference as introduced in Chapter 2.1.1. However, the mean steepness from the top to the bottom of the Matterhorn will remain the same no matter the chosen reference. In EEG data, The GFP is equivalent to the mean steepness of a region. This explains why the GFP is independent of the reference. The GFP represents a measure of the signal’s strength and is represented with strictly positive values. The greater the GFP, the stronger the underlying cerebral activity is. One caveat is that the GFP is blind to spatial information. In terms of statistical analysis, T-test and ANOVA at every time point of the GFP can be performed. To take account of temporal autocorrelation, only statistical differences that are present for a certain number of time point are kept and considered as reliable.

The second level of analysis is the topographic analysis. As introduced previously, each electrode at the scalp records the whole brain activity and not only the underlying sources. If enough electrodes are distributed over the whole scalp, it is possible to recreate the scalp’s electric field. This recreation of electric field is called scalp potential maps (or topography). These scalp potential maps change every millisecond. It was defined by a physical law that changes of topographic maps at the scalp reflect the orientation of active dipoles within the brain (Lehmann et al., 1987). A dipole is created by the sum of synchronous post-synaptic potentials. In more, the topography depends on the location and orientation of the dipoles (i.e. sources), this will be developed later. The topography is orthogonal to the GFP (i.e. independent). Again, a parallel with topographical maps can be done. If one takes Lausanne and Evian for example, the geographical north of Lausanne corresponds to the higher point of the city and the geographical south corresponds to the lowest point of the city. However, the highest point of Evian corresponds to its geographical south whereas its lowest point corresponds to its geographical north. Now, imagine that the highest point of Lausanne (geographical north) is located at the same height than the highest point of Evian (geographical south) and that the lowest point of Lausanne (geographical south) is located at the same height as the lowest point of Evian (geographical north). Both cities will present the same steepness resulting in equivalent GFP values but both opposite topographies.

Now that a simple illustration of the independence of the topography and GFP has been presented with a geographic parallel, the following text tries to present in a simplified manner the biological bases of topographic differences. As mentioned previously, topographies are the reflects of dipoles created by synchronous post-synaptic potentials within the brain. To explain how different brain generators (i.e dipoles) can give similar and different topographies, Figure 7 represents three possibility of dipoles repartition (black arrows) within a head (black circle). The localizations of the dipoles in each example is important. On the one hand, if one compares figure 7a and 7b, the red arrow and green arrow represent a vector that corresponds to the sum of dipoles. These two vectors have the same strength but of opposite direction. Therefore, these two configurations will be reflected at the scalp by two opposite topographies. On the other hand, if one compares Figure 7a and Figure 7c, the sum of dipoles creates two equivalent vectors that go in the same directions (red arrows). In this case, Although the

23 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. configuration of the dipoles is different, the resulting scalp topographies are similar. Therefore, in term of interpretation only two different maps can ensure the presence of two different configurations of generators. A different configuration in the generators indicate different brain networks.

Figure 8. Schematic representation of the link between Dipoles and Topographies

Although topographies are changing every millisecond, a specific electrical field configuration can remain quasi-stable for milliseconds, this particular observation was termed “microstate”(Lehmann and Skrandies, 1980; Michel et al., 1999). The quasi-stable state is defined by a fixed topography over a certain period of time, but the strength of the electrical field may vary a little across this period (Van de Ville et al., 2010). Usually a microstate is separated from another by transient period where different maps are present until the recurrence of a new quasi-stable state. A microstate reflects the activation of specific brain regions/networks that remains for a certain time period.

To identify these microstate, topographic analysis time frame by time frame or cluster analysis can be used. The Topographic analysis time frame by time is based on a metric measure called the global dissimilarity (DISS)(Lehmann and Skrandies, 1980) (Equation 2). The DISS values are comprised between 0 and 2. This metric indicates if average maps across condition and time are spatially different (Koenig et al., 2011).

Equation 2. The Global Dissimilarity. n is the number of electrodes ui and vi are the electrodes of the different conditions, GFPu and GFPv are the GFP of the conditions.

Another technique for topographic analysis is based on clustering methods. For this thesis, only the basis of the hierarchical clustering is introduced, because it is the method applied in the scientific papers presented in chapter 7.1 and 7.2. The hierarchical clustering or Atomize and Agglomerate Hierarchical Clustering (“AAHC”) is a method of clustering that is based on the grand-average ERP (i.e group- averaged ERP across experimental conditions and experimental groups). At start, each time point corresponds to a potential topographic map. The AAHC will progressively reduce the number of maps based on a least correlation method until the data is reduced to a defined number of consecutive maps that represent microstates ( Murray et al., 2008). A shortcoming of this method is that the remaining maps are “artificial”, because they are based on the grand-average data. These maps are considered as

24 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. template maps. If two different template maps are identified over a certain time period, they are back- fitted into the individual subjects of ERPs, to find which map is more present at each time point for each subject. The output measures of the fitting at single subject’s level can be multiple; such as frequency of map presence, first onset of a map, last onset of a map, global explained variance of a given map, etc. Statistical analysis is performed on these outputs to identify which template map represents which condition.

The last level of analysis is the estimation of brain sources. This step is a mathematical calculation that allows to estimate the localization of generators within the brain. To compute these estimations, four components are required:

• A Brain MRI image

• A scalp coverage with a certain number of Electrodes

• A head model

• A inverse calculation matrix

The Brain MRI image that was used for this work is the Montreal Neurological Institute (MNI) average brain. The number of Electrodes at the scalp is an important parameter for the calculation inverses, the more the better. The head model was generated with a the Spherical Model with Anatomical Constrains (SMAC, (Spinelli et al., 2000). Finally applying Local Auto-regressive average (LAURA) regularization approach (inverse calculation matrix), that is composed of a matrix with 2 nodes distributed equally in the grey matter of the MNI brain (Grave de Peralta Menendez et al., 2004b, 2001).

Practically, the first step consists in averaging single subject’s ERPs over a time period of interest selected upon the above-mentioned techniques (Electrodes, GFP, Topographies). The intracranial sources are then computed with the application of Local Auto-regressive average (LAURA) regularization approach using single subject’s average ERP as inputs. More specifically for this thesis, the LAURA inverse solution matrix was composed of 3005 nodes and the brain activity was based on the response of 128 scalp channels. The output of source estimations gives current density values (mA/mm3) for each of the 3005 nodes. In terms of analysis, statistical test (T-test and ANOVA) can be performed over the nodes. To partially correct for multiple testing an alpha threshold of 5% or less as well as a spatial criterion of contiguous voxels is usually applied.

Finally, to conclude with EEG analysis, early visual components of ERPs are presented in the following chapter. It will only focus on three specific components, namely the C1, the N1 and the P1 components.

2.1.3 Visual EEG components (C1, P1, N1).

The C1 component was the first component identified in 1972 in a study from (Jeffreys and Axford, 1972). This component usually peaks around 50-90ms. The scalp topography of this component is dependent on the localization of the stimulus. If stimuli are localized in the upper visual field, the ERP waveform presents a negative peak. Usually, occipital and parietal electrodes that are contralateral to

25 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. the presentation of stimulus present positivity. This particular feature of the C1 components suggest a retinotopic dependence (Jeffreys and Axford, 1972). The C1 component has been linked to modulation within the primary visual cortex (Di Russo et al., 2003).

The P1 component has historically been attributed to (Spehlmann, 1965). P1 stands for Positive 1, because it was the first to be identified with a positive peak when EEG was recorded with a mastoid reference (Mangun, 1995). The P1 peak was characterized by a posterior bilateral positivity and fronto- central negativity in the VEP topography. It usually peaks ~80-130ms post-stimulus. Brain generators localized within the dorsal and ventral pathway of vision have been identified as possible sources of the P1 component with ERPs (Murray et al., 2001) and ERP/PET (Woldorff et al., 1997).

The N1 component stands for negative 1 because it was the first component identified with a negative peak ~150-200ms when EEG was recorded with mastoid reference. This GFP peak was characterized by a posterior bilateral negativity and fronto-central positivity in the VEP topography. N1 component was linked with modulations in the ventral pathway of vision in intracranial recording (Allison et al., 1999) as well as EEG recording (Doniger et al., 2001; Murray et al., 2002a).

The definition of the P1 component was used in the scientific paper presented in 7.1 for realigning data that were phase-shifted between two conditions (full detail on the technique can be found in the paper in chapter 7.1)

2.2 The EyeMusic sensory substitution device.

The EyeMusic is the visual-to-auditory sensory substitution device that was used in paper 7.3. Visual- to-auditory sensory substitution devices transform images into auditory soundscapes (see chapter 1.4). The following text introduces the basis of transformation applied to obtain an auditory soundscape when the image is presented in the EyeMusic algorithm.

The EyeMusic is based on images that are 30 pixels high by 50 pixels wide. The transformation of images into auditory “soundscapes” follows three basic principles. The first principle is the sweeping of an image from left to right. The sweeping process happens column after column. It is equivalent to a time component that is represented on the X-axis. Practically, a pixel on the left of the image will sound before a pixel on the right. The second principle is the fragmentation of each column in pitch, this fragmentation is represented in the Y-axis. According to the Y-axis coordinate of a given pixel, a specific pitch is attributed. Practically, the higher a pixel, the higher in pitch of the sound will be. The third principle is the volume of the sound. The volume is based on the brightness of a given pixel. The brighter the pixel is, the louder the sound will be. To recreate “soundscape”, all rows and columns are combined in a single auditory stimulus (Abboud et al., 2014a). To render the auditory experiment more pleasant, sounds are based on a pentatonic scale. The pentatonic scale also ensure that no dissonance can be perceived when two pixels one over the other are played. The EyeMusic algorithm also implements colors, which is a novelty compared to other visual-to-auditory sensory substitution devices that exists (Capelle et al., 1998; Meijer, 1992). Each color is substituted by a specific instrument. For Example, the white color is represented by a choir. The only exception to this color rule is the black color which is

26 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. silent. Here only black and white are presented because they are the colors that were used in paper 7.3. Further information about the EyeMusic and color implementation can be found in (Abboud et al., 2014b).

27 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

CHAPTER 3 RESULTS

3.1 Cue-dependent circuits for illusory contours in humans

Jacques Anken, Jean-François Knebel, Sonia Crottaz-Herbette, Pawel J. Matusz, Jérémie Lefebvre and Micah M. Murray

This paper made the cover of the issue of Neuroimage. 2016 Apr 1;129:335-44. doi: 10.1016/j.neuroimage.2016.01.052

Contribution: The candidate contributed to the elaboration of the experimental design, data acquisition, analysis and writing of the paper.

3.1.1 Abstract

Objects’ borders are readily perceived despite absent contrast gradients, e.g. due to poor lighting or occlusion. In humans, a visual evoked potential (VEP) correlate of illusory contour (IC) sensitivity, the “IC effect”, has been identified with an onset at ~90ms and generators within bilateral lateral occipital cortices (LOC). The IC effect is observed across a wide range of stimulus parameters, though until now it always involved high-contrast achromatic stimuli. Whether IC perception and its brain mechanisms differ as a function of the type of stimulus cue remains unknown. Resolving such will provide insights on whether there is a unique or multiple solutions to how the brain binds together spatially fractionated information into a cohesive perception. Here, participants discriminated IC from no-contour (NC) control stimuli that were either comprised of low-contrast achromatic stimuli or instead isoluminant chromatic contrast stimuli (presumably biasing processing to the magnocellular and parvocellular pathways, respectively) on separate blocks of trials. Behavioral analyses revealed that ICs were readily perceived independently of the stimulus cue – i.e. when defined by either chromatic or luminance contrast. VEPs were analyzed within an electrical neuroimaging framework and revealed a generally similar timing of IC effects across both stimulus contrasts (i.e. at ~90ms). Additionally, an overall phase shift of the VEP on the order of ~30ms was consistently observed in response to chromatic vs. luminance contrast independently of the presence/absence of ICs. Critically, topographic differences in the IC effect were observed over the ~110-160ms period; different configurations of intracranial sources contributed to IC sensitivity as a function of stimulus contrast. Distributed source estimations localized these differences to LOC as well as V1/V2. The present data expand current models by demonstrating the existence of multiple, cue-dependent circuits in the brain for generating perceptions of illusory contours.

3.2 Brain mechanisms for perceiving illusory lines in humans.

Jacques Anken, Jean-François Knebel and Micah M. Murray

Under review in NeuroImage

28 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

Contribution: The candidate contributed to the elaboration of the experimental design, data acquisition, analysis and writing of the paper.

3.2.1 Abstract

Illusory contours (ICs) are perceptions of visual borders despite absent contrast gradients. The psychophysical and neurobiological mechanisms of IC processes have been studied across species and diverse brain imaging/mapping techniques. Nonetheless, debate continues regarding whether IC sensitivity results from a (presumably) feedforward process within low-level visual cortices (V1/V2) or instead are processed first within higher-order brain regions, such as lateral occipital cortices (LOC). Studies in animal models, which generally favor a feedforward mechanism within V1/V2, have typically involved stimuli inducing IC lines. By contrast, studies in humans generally favor a model where IC sensitivity is mediated by LOC and typically involved stimuli inducing IC forms or shapes. Thus, the particular stimulus features used may strongly contribute to the model of IC sensitivity the data support. To address this, we recorded visual evoked potentials (VEPs) while presenting human observers with an array of 10 inducers within the central 5°, two of which could be oriented to induce an IC line on a given trial. VEPs were analyzed using an electrical neuroimaging framework. Sensitivity to the presence vs. absence of centrally-presented IC lines was first apparent at ~200ms post-stimulus onset, followed from modulations in the topographic distribution of the VEP, and was localized to LOC. The timing and localization of these effects are consistent with a model of IC sensitivity commencing within higher- level visual cortices. We propose that prior observations of effects within lower-tier cortices (V1/V2) are the result of feedback from IC sensitivity that originates instead within higher-tier cortices (LOC).

3.3 Training Orientation Discrimination Using Visual-to-auditory

Sensory Substitution

Jacques Anken, Jean-François Knebel, Rosanna De Meo, Francine Behar-Cohen, Amir Amedi, and Micah M. Murray

Contribution: The candidate contributed to the elaboration of the experimental design, data acquisition, analysis and writing of the paper.

3.3.1 Abstract

Vision impairments and blindness are a worldwide burden. Non-invasive devices constitute one approach to help restore visual functions. In particular, visual-to-auditory sensory substitution devices (SSDs) are promising. Individuals (both sighted and blind) trained with these SSDs can achieve a high level of discrimination and identification. Moreover, their use results in specific brain activity in what are otherwise nominally visual brain regions, though the qualia of such brain activity remains unresolved. Nonetheless, there is little empirical evidence concerning how to effectively and efficiently train individuals in using SSDs. The present study addressed this issue by training sighted individuals to use the EyeMusic SSD to discriminate between 2 letters (T and U) and their vertical orientation.

29 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

Immediately prior to and following training, participants were tested in their ability to identify each of the 4 cardinal orientations of a different letter (E). Moreover, the stimuli were either presented at the same location (and by extension auditory frequency) or a new location as that used during the training session. First, we found that training for 15 minutes significantly improved T/U vertical orientation discrimination, independent of whether the SSD algorithm was explained. Second, performance on E orientation discrimination improved for the trained (vertical) orientation and generalized to the untrained (horizontal) orientation. Third, effects of training also generalized to orientation discrimination when stimuli were presented at untrained locations, albeit to a greater extent for the trained than untrained orientations. Collectively, these results provide insights into effective training protocols for the use of SSDs that may be completed autonomously.

Key words: Visual-to-auditory sensory substitution, EyeMusic, Training, Psychophysics, sighted individuals.

Contribution: The candidate contributed to the elaboration of the experimental design, data acquisition, analysis and writing of the paper.

3.4 Unpublished Results for experiment 3.2 Brain mechanisms for perceiving illusory lines in humans

In this sub-section, we present unpublished preliminary results regarding the lateralization of stimuli within the parafovea as well as the orientation detection of Illusory lines. As a reminder, in this experiment, participants had to discriminate the localization of vertical line and horizontal line presented in an array of ‘pacmen inducers’.

3.4.1 Lateralization of stimuli within the parafovea.

For this supplementary analysis, conditions IC right and IC left were created by collapsing the Horizontal illusion and the vertical illusion at each localization as it was performed in the paper for ICC condition. Then the “ICeffect” was computed at each localization by subtracting the Non-contour condition to the IC to create ICeffectLeft, ICeffectCenter and ICeffectRight. The following results are calculated on these ICeffect data.

30 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

Figure 9: VEP correlates of Localization effect as revealed by the electrical neuroimaging analysis framework.

Figure 9 represents the results of Analysis performed on the Global field power (GFP), Electrodes and DISS (Figure 9a, 9b and 9c, respectively). Figure 9a displays the results of a one way Anova at the level of GFP (response strength), p-value ≤0.05 for at least 15 continuous data points, with localization of

Illusion as factor revealed no statistical differences between ICeffectL, ICeffectC and ICeffectR .Figure 9b displays the percentage significant electrodes subsequent to a one way Anova with localization of Illusion as factor, p-value ≤0.01 for at least 10 continuous data points with a chosen an artificial threshold of 20% electrodes (i.e. > 25 electrodes). This analysis revealed that more than 20% of electrodes were statistically different from 200 ms and onward. A Tanova analysis was also performed with a temporal criterion of 10 contiguous time points and revealed different topographies from 63-128ms and from 153ms onward (Figure 9c). These results indicate collectively that distinct brain networks contribute to ICeffect during the 200ms and onward post-stimulus period when stimuli are defined by localization. A period from 200-250 ms was chosen to compute the source estimation. This period corresponds to the onset of the first peak to its end at the level of electrodes. Source estimation results are displayed in Figure 10.

31 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

Figure 10: Distributed source estimation of the localization effect over the 200-250ms period.

Figure 10 a displays the statistical differences between ICeffectL, ICeffectC and ICeffectR condition at the level of source estimations. We considered as reliable those clusters wherein each node yielded a significant test (p<0.05) and was moreover located within a cluster of at least 15 significant nodes

(kE>15nodes). The results showed significant differences between the generators of ICeffectL, ICeffectC and ICeffectR located within bilateral lateral occipital cortices and extending to the Inferotemporal cortex. Figure 10b and 10c displays the results of a manual separation of the left and right Lateral occipital cortices. This separation was performed to calculate the activation for each condition within each LOC separately. Figure 10b, left panel displays the Mean source estimation localized in the Left LOC within which the activations were calculated. The right panel displays the Mean±SEM activity for each condition within which the Left LOC. A Wilcoxon signed rank test revealed that the activity of ICeffectC is different from ICeffectL (t(10)= 3.20, p<0.01) and ICeffectR(t(10)= 2.44, p<0.05). Furthermore, ICeffectL is statistically different from ICeffectR(t(10)= -2.2934 , p<0.05). These results show a gradient of activity within the Left LOC : ICeffectC > ICeffectR> ICeffectL. Figure 10c, left panel displays the Mean source estimation localized in the right LOC within which the following activations were calculated. The right panel displays the Mean±SEM activity for each condition within the right LOC. T-tests revealed that the activity of ICeffectC is different from ICeffectR (t(10)= 3.77 , p<0.01). Furthermore, ICeffectL is statistically different from ICeffectR (t(10)= 3.28 , p<0.01). These results show a gradient of activity within the right LOC : ICeffectC = ICeffectL > ICeffectR.

32 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

3.4.2 Orientation detection of Illusory lines.

To investigate the brain mechanism underlying the detection of illusory lines’ orientation we analyzed the differences between illusory contour of line oriented horizontally in the center position (ICHC) and the illusory contour of line oriented vertically in the center position. The results of this analysis are displayed in figure 11.

Figure 11: VEP correlates of IC Line orientation detection as revealed by the electrical neuroimaging analysis framework

Figure 11a displays the VEPs in response to the ICHC and ICVC conditions at an exemplar midline occipital scalp site (Oz). Both conditions elicited robust VEPs with characteristic P1-N1 components of indistinguishable magnitude. In order to identify the timing of differential VEP responses, we first performed a mass univariate analysis as a function of time across the full 128-channel electrode montage (Figure 11b). To (partially) account for both temporal and spatial correlation, differences were considered reliable if significant for at least 15ms consecutively (i.e. >15 time samples) as well as across at least 20% of the electrode montage (i.e. >25 electrodes). Reliable differences were first present over the 317-493ms post-stimulus onset. Next, analyses of GFP revealed significant differences over the 373- 455ms post-stimulus period (Figure 11c). Responses were significantly stronger to the ICVC than ICHC condition. Analysis of the VEP topography, using global dissimilarity, indicated that responses to ICHC and ICVC conditions differed topographically over the 314-397ms post-stimulus interval (Figure 11d).

33 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

Group-averaged data were then submitted to a topographic cluster analysis, which identified two template maps characterizing responses over the 350-389ms post-stimulus period. These template maps were then fitted to the single-subject ICHC and ICVC responses from individual participants in order to determine the presence of each template map over the 350-389 ms period. These values were then submitted to a Wilcoxon signed-ranks test, which indicated a significant interaction Z=0, p <0.05. One map predominated responses to the ICHC condition, whereas another map predominated responses to ICVC (Figure 11e). Finally, analyses were performed on distributed source estimations over the 350- 389 ms period (i.e. the period identified in the above topographic clustering analysis). Significant differences were observed within the precuneus and cuneus (Figure 11b).

34 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. CHAPTER 4 GENERAL DISCUSSION

This chapter is divided in three parts. The first part discusses the unpublished results that are presented in chapter 3.4. The second part addresses the neurophysiological papers that were both investigating the perception of Illusory contour presented within the parafovea, albeit with different stimuli types (paper 7.1 and 7.1). Finally, the third part addresses the results of training with sensory substitution devices (paper 6.3). Each of the three subparts present a summary of the general discussion of each paper and further directions. It is important to note that each study presented in chapter 7 owns a specific discussion. Consequently, the following discussion will address issues other than these discussed in the papers in chapter 7.

4.1 Discussion of unpublished results.

4.1.1 Lateralization of stimuli within the parafovea.

The unpublished results of paper 7.2 presented in chapter 3.4.1 Lateralization of stimuli within the parafovea show no evidence for reliable differences in response strength (GFP) between the ICeffectL,

ICeffectC and ICeffectR. By contrast, these IC effects differed in their percentage significant electrodes as well as in their topography (DISS) over the 200ms and onward post-stimulus period. Moreover, the source estimations over the 200-250 ms period revealed a differential activity within the bilateral LOC for the detection of lateralization. These results demonstrate that lateralizing stimuli within the parafovea yield a lag in neurophysiological response equivalent to lateralized stimuli within a visual hemifield (Murray et al., 2002; Exp 5). This could indicate that similar brain networks are involved for the detection of lateralization from the parafovea to the periphery.

As our stimuli presented responses similar to lateralization within one hemifield of vision, to address a potential visual field preference of the LOC with stimuli presented within the parafovea, the right and left LOC have been separated manually and the relative activity of each condition (i.e ICeffectC, ICeffectL,

ICeffectR) was measured. Results within the left LOC show a gradient of activity in which the activity for ICeffectC is significantly greater than activity of ICeffectR. In more, the activity for ICeffectR and

ICeffectC significantly differ from CeffectL (figure 10b). A similar, but opposed gradient of activity was found within the right LOC for which the activity of ICeffectC was not statically different from ICeffectL.

However, they were both significantly different form the activity yielded ICeffectR (Figure 10c). These preferences of right and left lateral occipital cortex for the central localization and contralateral presentation of stimuli supports a visual field preference within the LOC when stimuli are lateralized within the parafovea.

Many studies have investigated the extent to which lateral occipital (LO) neurons receive information from retinal inputs. Intracranial recordings in monkey have reported evidence that sensitivity to retinal position is present within inferotemporal (IT) region (Tanaka, 1996). In humans, A study demonstrated the relative preference of LO regions (IT equivalent in human) for the contralateral hemifield by presenting black and white outlines of animal forms and scrambled images outside the parafovea. They

35 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. claimed that the differential LO preference that they observe is due to their stimuli that only covered a visual half field (Niemeier et al., 2005). Other studies reported similar lateralization effects with fMRI and demonstrated a bigger activation in LO regions to stimuli presented in the contralateral hemifield of vision (Grill-Spector et al., 1998; Hemond et al., 2007). However, it would be an oversight not to mention that this contralateral preference of the LO remains debated. Indeed, The LO area has also been reported to be equally activated by stimuli presented in the ipsi and contralateral hemifield (Grill-Spector et al., 1998). Nevertheless, all these studies demonstrated a specific response of LO regions to contralateral and ipsilateral stimuli. Some propose that several retinotopic areas might exist beyond V4 and partially overlap with lateral occipital cortex (Brewer et al., 2005; Large et al., 2008). However, retinotopic organization of the lateral occipital areas remains as controversial as the contralateral preference. We believe that Our results on the lateralization of stimuli within the parafovea might reflect a low retinotopic organization of the LOC. However, to ascertain that our results are retinotopic it would be important to assess the upper and lower visual fields responses when stimuli are presented in the parafovea. This aspect is developed in subchapter 4.2.2.b.3 lateral occipital cortex and retinotopic mapping.

4.1.2 Orientation detection of Illusory lines.

In the unpublished results of paper 7.2 presented in chapter 3.4.2 Orientation detection of Illusory line, participants had to detect the presence/absence of an illusory line, and then localize the IC within the array. The perception of line’s orientation was implicit to the detection of Illusory contour. This analysis show first significant difference between ICHC Vs. ICVC starting ~317-493ms post stimulus onset with mass univariate analysis as a function of time across the full 128-channel electrode montage. Analyses of GFP revealed significant differences over the 373-455ms post-stimulus period and analysis of the VEP topography, using global dissimilarity, indicated different topography between ICHC and ICVC at ~314-397ms. Finally, Hierarchical clustering identified two different map between 350-389ms post- stimulus onset and this period was chosen to compute source localization that revealed significant difference in the precuneus and cuneus.

First, the timing of 350-389ms post-stimulus occurs ~100ms after the sensitivity of IC line within the LOC (presented in chapter 7.2) and is consistent with a potential feedback coming from the LOC. Previous work from Halgren and colleagues in 2003 with magnetoencephalography reported a peak activation within the LOC in response to IC at ~ 155 ms followed by later activations within the occipital cortex at~ 235ms. The authors claimed that this occipital activation might reflect feedback from LOC. If this is the case, the difference of timing is on the order of ~80 ms between the LOC and the occipital cortex. This timing approximately corresponds to the difference of timing from LOC to primary visual cortex in our experiment (~100ms). Second, our results on distributed source estimation localized significant differences within the precuneus and cuneus. On the one hand, these results are in line with intracranial recording in macaques monkeys that localized the sensitivity to real and illusory lines within V1/V2 (von der Heydt et al., 1984; von der Heydt and Peterhans, 1989a). On the other hand, these results derive from human studies with the “thin/fat” model (Ringach and Shapley, 1996) of IC. This model could be considered as functionally synonymous to orientation detection. Indeed, the “Fat” shape is

36 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. equivalent to a horizontal orientation of IC and the “thin” shape is equivalent to a vertical orientation of IC (Figure12). An EEG study from Murray and colleagues in 2006 investigated the neurophysiological response of sighted individuals exposed to these particular illusory shapes with EEG. Their results show a detection of IC at ~ 124-186 ms post-stimulus and a discrimination of “Thin/Fat” condition at later periods over ~ 330-406 ms period. Distributed source estimations over this latter period yielded activity within bilateral lateral-occipital regions (Murray et al., 2006).. Here our timing of orientation detection at ~350-389ms is consistent with the discrimination of “thin/fat” condition, but the source localizations differ (i.e. cuneus/precuneus vs. LOC). This might indicate two different mechanism of illusory contour detection that depend on the shape. Chapter 4.2.2.b.2 will address this issue and propose an experiment to try solving these discrepancies.

Figure 12: Schematic of the "Fat" IC shape, "Thin"IC shape and the classical kanizsa illusory square. (Adapted from Ringach and Shapley, 1996)

4.2 Neurophysiological Studies

The two papers presented in chapter 7.1 and 7.2 were investigating the perception of illusory contours in humans. The paper 7.1 entitled “Cue-dependent circuits for illusory contours in humans” investigated the brain mechanism of illusory contour forming shapes presented within the parafovea and biased toward the magnocellular and parvocellular pathway of vision. The paper 7.2 entitled “Brain mechanisms for perceiving illusory lines in humans” investigated the brain mechanism of detection of Illusory line in an array of pacemen presented within the parafovea. Although these papers were using different stimuli, they both confirmed that Illusory contour sensitivity is achieved within the ventral pathway of vision, more precisely within the lateral occipital cortex. This specific role of the LOC was already demonstrated in many other studies (Knebel et al., 2011a; Murray et al., 2002a, 2006).

As stimuli and other conclusions of these studies differ, summary of general conclusions and further directions will be treated separately in the following subparts 4.2.1 and 4.2.2.

4.2.1 Cue-dependent circuits for illusory contours in humans

4.2.1.a Summary of general conclusion

This paper was designed to assess the contributions of magnocellular and parvocellular pathways to the detection of illusory contours forming shapes in human. It revealed that isoluminant chromatic and low luminance contrast stimuli (biased toward the P and M pathways of vision) both elicit a neurophysiological response to IC but with a stronger efficiency when stimuli are chromatic (P-biased). Both stimulus type displayed different topographies over 108-159ms post-stimulus indicating different

37 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. brain mechanisms. Finally, distributed source estimation demonstrated a greater activity in the LOC and

V1/V2 to the ICeffect when stimuli are displayed in chromatic Vs. luminance contrast. These results support a model whereby the IC sensitivity occurs within the LOC. Furthermore, they provide the first evidence that different brain circuits contribute to the perception of Illusory contours and that the sensitivity to IC occurs within the ventral visual stream independently from the stimulus types.

By contrast, this paper failed to demonstrate a sensitivity to illusory contour eccentricity when stimuli are biased toward the magnocellular and parvocellular condition. Indeed, illusory contours in this paper were forming either a circular shape or a square shape. The circular IC had a 1.4° centre-to-centre eccentricity whereas the square IC had a 2.0° centre-to-centre eccentricity. This eccentricity condition was designed to investigate the dorsal visual stream. Previous study presented sensitivity to eccentricity localised within dorsal visual stream with high contrast stimuli (Knebel et al., 2011a).

4.2.1.b General comments and Directions

4.2.1.b.1 Eccentricity detection in magnocellular and parvocellular condition.

One shortcoming of the paper presented in 7.1 is that even though both conditions demonstrated sensitivity to illusory contours, the experiment failed to provide a difference in eccentricity detection between magnocellular and parvocellular conditions. The eccentricity condition was designed to investigate the relationship between the dorsal visual stream and magnocellular/parvocellular pathways. This lack of differences in the eccentric condition, might be due to a difference of eccentricity that was not big enough in our stimuli (i.e. the difference of eccentricity between the illusory circle and the illusory square was on the order of 0.6°). Perhaps the magnocellular and parvocellular pathways need a larger difference of eccentricity between two shapes to be able to detect the change in eccentricity. We know from previous study performed in high contrast that differences of eccentricity of illusory contours yield differences in neurophysiological responses (Altschuler et al., 2012; Knebel et al., 2011a). Knebel et al., 2011 used high contrast circle and square IC and revealed that healthy subjects can detect illusory contour’s eccentricity. In this experiment, the sensitivity to eccentricity was localized within the left precuneus, medial inferior parietal cortex and left post central gyrus at early latencies ~54-108 ms. These localizations are supporting a detection of eccentricity within the dorsal visual stream. Nevertheless, the contribution of magnocellular and parvocellular pathway in the detection of eccentricity remains unknown. In a theoretical point of view, is no reason to believe that it should not occur when stimuli are biased toward magnocellular pathway of vision because this pathway project mainly to the dorsal visual stream. For parvocellular biased stimuli, the detection of eccentricity might be more complicated, because it projects mainly to the ventral visual stream. To try assessing the M and P contribution to the detection eccentricity, we propose a new experiment with isoluminant chromatic and achromatic low luminance conditions inducing an illusory contour forming a square for which eccentricity vary from 10° down to 2° in steps of 2° of visual angles. This should allow to investigate if a certain difference of eccentricity is required to be detected when stimuli are biased toward the magnocellular and parvocellular pathway of vision in healthy individuals.

38 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

For this experiment, different outcomes can be hypothesized. First, none of the 2 pathways of vision is sufficient to detect the eccentricity of illusory contour. In this case, we expect a 2x5 ANOVA with factors condition (Magnocellular/Parvocellular) and eccentricities (2°/4°/6°/8°/10°) to yield no differences. Second, only the Magnocellular pathway detects differences in eccentricity but the parvocellular pathway does not. In this case, we expect we expect a 2x5 ANOVA with factors condition (Magnocellular/Parvocellular) and eccentricities (2°/4°/6°/8°/10°) to yield a main effect of condition as well as a main effect of eccentricity. Third, only the Parvocellular pathway detects differences in eccentricity but the magnocellular pathway does not. In this case, we expect a 2x5 ANOVA with factors condition (Magnocellular/Parvocellular) and eccentricities (2°/4°/6°/8°/10°) to yield a main effect of condition as well as a main effect of eccentricity. Fourth, both the magnocellular and parvocellular condition detect differences in eccentricity. In this configuration, we expect a 2x5 ANOVA with factors condition (Magnocellular/Parvocellular) and eccentricities (2°/4°/6°/8°/10°) to yield only a main effect of eccentricity.

Finally, in all the cases where the detection of eccentricity occurs, we expect to find differences within the dorsal visual stream as it was previously demonstrated (Knebel et al., 2011a). However, we expect the detection of eccentricity to happen at later latencies than those that were reported in Knebel at al. 2011. Indeed, we believe that the detection of eccentricity requires more effort when stimuli are biased toward M and P pathway of vision. Therefore, it should occur after the IC sensitivity. This hypothesis is also supported by a model of two stage object recognition in which a first phase, the perceptual phase (i.e. sensitivity to IC) occurs rapidly and a second stage, the conceptual phase, require more effort to be achieved (Altschuler et al., 2012). Moreover, based on the latency shift observed in paper 7.1 for the M- biased stimuli, if the M-biased condition present a sensitivity to eccentricity, we expect its neurophysiological response to be shifted toward later latencies in comparison with high contrast stimuli and P-biased stimuli.

4.2.1.b.2 Using magnocellular and parvocellular biased stimuli to investigate Schizophrenia impairments.

The first part of this subchapter will briefly introduce schizophrenia and the potential impairments that have been discovered in this clinical population. The second part will propose an experiment to investigate these impairments with stimuli that differ in eccentricity and could either be biased toward the Magnocellular or the parvocellular pathway of vision.

Schizophrenia is a mental disorder which diagnosis is based on DSM IV published by the American Psychiatric Association. To be diagnosed with schizophrenia, two or more symptoms should be present for at least six months. This six-month period should include at least one month of characteristic symptoms from the following list: Delusion, hallucination, disorganized speech, severely disorganized behavior (e.g dressing inappropriately, crying frequently) or negative symptoms (e.g Reduction of emotional expression, Inability to perform planned actions, and great difficulty in making decisions). Previous neurophysiological researches have identified possible cognitive deficits (high level: memory skills, working memory) and early-visual deficits (low-level), that could be linked with schizophrenia.

39 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

However, these deficits are still debated. An EEG study investigating illusory contours in schizophrenia reported a reduce P1 component in patients. However, source estimations over the P1 latency were localized within the LOC for both the patients and control subjects. Consequently, the authors attributed this reduction of the P1 component to dorsal stream dysfunctions with no impact upon subsequent early IC processing within ventral stream areas (Foxe et al., 2005). This hypothesis of dorsal stream dysfunction is supported by earlier studies based on topography (Foxe et al., 2001) and relative sensitivity to luminance contrast (Butler et al., 2001). More recent fMRI studies showed reduced activations to low spatial frequencies (biased toward magnocellular pathway) in multiple regions of the occipital, parietal, and temporal lobes in schizophrenia patients but no deficit in high spatial frequencies (biased toward parvocellular pathway) (Butler et al., 2007; Calderone et al., 2013; Martínez et al., 2008). These studies support a dysfunction of the magnocellular pathway of vision in schizophrenia (see Javitt, 2009 for a review of early visual deficit in the magnocellular pathway). An EEG study confirmed the abovementioned findings by investigating the response to illusory contours of two different eccentricities (Square and circle IC shape discussed earlier). This study reported that patients exhibited a reduce P1 in response to eccentricity whereas the control group exhibited distinct P1 generation. Source estimation localized the differences within parietal structures. The authors claimed that these results are consistent with a magnocellular and/or dorsal visual stream impairments in schizophrenia (Knebel et al., 2011b).

Other studies used visual masking experiment to investigate potential impairments in schizophrenia. In visual masking experiment, a target is followed, after its presentation, by a “mask” that can overlap completely the target or only surround it. This type of masking is called visual backward masking. Masking can also happen if the mask is presented before the target; this is known as visual forward masking. fMRI studies of masking reported that patient and control show an enhanced activation of lateral occipital regions when the mask becomes weaker. However, the response of patients is overall weaker than to those of controls in these regions. This led to the hypothesis of a potential ventral pathway deficit in schizophrenia (Wynn et al., 2008). For a review of visual masking in schizophrenia see (Green et al., 2011).

Finally, some studies used perceptual closure to investigate impairments in schizophrenia. The perceptual closure refers to the brain ability to form objects on the basis of fragmented visual information. An ERP studies from Doniger and colleagues 2000 used fragmented images in control subjects and revealed an ERP component that starts ~ 230ms and peaks ~290ms post-stimulus, the closure negativity (Ncl component). The amplitude of the Ncl component increases as images are presented in less fragmented form. They likewise showed bilateral scalp foci consistent with brain generators localized within bilateral LOC. With this type of stimulation, Doniger et al., 2002 revealed that schizophrenia patients showed impaired performance in the recognition of objects. This impaired capacity was linked with a diminution of P1 that is consistent with prior finding reporting a magnocellular And/or dorsal impairment in schizophrenia. This P1 diminution was accompanied by an impaired Ncl generation. However, the early stage of ventral visual processing (N1 component) was not impaired. The conclusion of this paper was that impaired dorsal processing might lead to impairments in the late stage of ventral visual processing (i.e Ncl) in object recognition. Similarly, a study using Kanizsa type illusory contour reported similar results (Foxe et al., 2005).

40 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

In conclusion, it remains uncertain whether early visual deficits in schizophrenia are due to a magnocellular dysfunction, a dorsal stream impairment or a ventral stream dysfunction. Assuming that the experiment proposed in 4.2.1.b.1 demonstrates the sensitivity to changes in eccentricity of IC is localized within the dorsal visual pathway independently of stimulus type in healthy individuals, we could use the same stimuli to investigate further which deficit(s) is/are present in schizophrenia patients. For that, we propose to use two different eccentricity and stimuli biased toward the magnocellular or parvocellular pathway, respectively. The outputs of this experiment could be multiple and depend on the potential impairments present in patient with schizophrenia. First, if patients present a purely magnocellular deficit, we expect IC sensitivity and eccentricity detection to be absent in M-biased stimuli but not in P-biased stimuli. Second, if patients suffer from a purely dorsal deficit, we expect the eccentricity detection to be impaired in both M and P biased stimuli, but the sensitivity of illusory contour should be intact in both conditions. This hypothesis is based on the results of paper 7.1 which demonstrated that IC sensitivity occurs within the LOC (i.e.in the ventral pathway) with M and P biased stimuli. It is important to note here that a potential deficit in IC sensitivity in magnocellular condition cannot be completely ruled out, because M cells project mainly to the dorsal visual stream. Third, if schizophrenia patient suffers from a purely ventral dysfunction, we expect both IC and eccentricity to be impaired in the P and M biased stimuli. This supposition is also based on the results of paper 7.1. Therefore, if the ventral pathway is impaired, patients should not detect IC in any of the conditions and subsequently no eccentricity. Finally, if patient have a parvocellular impairment, which has never been demonstrated or hypothesized to our knowledge, we expect the IC sensitivity and the sensitivity to eccentricity changes to be impaired in P-biased stimuli but not for M-biased stimuli.

4.2.1.b.3 Taking advantage of specific eye disorder to investigate cones and rods relationship with magnocellular and parvocellular pathway and high contrast stimuli.

A remaining question concerning magnocellular and parvocellular pathways of vision is their respective contribution when stimuli are presented in high contrast. In an effort to try answering this question, we could take advantage of specific retinal disorders.

First Retinitis Pigmentosa (RP) which is a genetic degenerative eye disorder, could help understanding what happen when cones are specifically excited. Indeed, the initial phase of RP is characterized by the progressive loss of rods photoreceptors, which leads to night blindness and peripheral vision loss. In a second phase, the visual loss can extend into the central visual field, leading to tunnel like vision and finally blindness (Shintani et al., 2009). If we could have access to RP patient when their cones are still present but with most of their rods destructed, we could hypothetically have access the specific response of cones. Comparing the response of cones to parvocellular biased stimuli and high contrast stimuli would allow to investigate if the cones’ response to high contrast (HC) stimuli is only composed of the parvocellular(P) pathway activation (HC=P). If the later proposition is not true two explanations are possible. In first place, HC stimuli might activate specific brain networks that do not implicate the P pathway. In second place, the cones’ response to HC might activate P pathway + other visual pathway(s), such that our equation becomes HC=P+?. If this hypothesis turns out to be correct, one potential suspect could be the koniocellular (K) pathway of vision. Chapter 1.1.4 only described the magnocellular and

41 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. parvocellular pathways of vision originating from retinal afference to specific layers of the Lateral Geniculate Nucleus. However, there exist two other types of ganglion cells, called bistratified ganglion cells which respond to blue color and giant melanopsin cells which respond to yellow color (i.e. they are activated by the S-cones (Jeffries et al., 2014)). These ganglion cells project to the Koniocellular layers of the LGN. The number of koniocellular cells is equivalent to the number of magnocellular cells. To test this new hypothesis of HC = P+K, RP patients could be tested with an IC experiment and stimuli biased toward the parvocellular pathway (isoluminant Red/green) or koniocellular pathway (isoluminant Blue/Yellow) and compare the summation of these response to high contrast response. Finally, we would like to add a comment on achromatic stimuli, cones should normally respond to achromatic stimuli, but to a lesser extent that chromatic stimuli, we expect to be able to measure the response of magnocellular pathway in response to achromatic stimuli presented to cones as well. Therefore, the additive model that we propose might become in the case of cones HC=P+K+M.

Second, achromatopia which is a genetic eye disorder, could help us understanding what happens when rods are specifically excited. Achromatopia is characterized by the loss of cone cell functions (Atrophy). Patient that suffer from achromatopia see in black and white or grey scale (Remmer et al., 2015). Comparing the response of rods to magnocellular biased stimuli and high contrast stimuli in achromatopes would allow to investigate if the rods’ response to high contrast (HC) stimuli is only composed of the magnocellular (M) pathway activation (HC=M). If this is not the case, again 2 hypotheses are possible. First, Rods activation in High contrast uses distinct brain networks that do not involve the magnocellular pathway. Second, High contrast stimuli activate the M pathway plus other pathways so that our equation becomes HC=M+?. For this configuration, we do not have idea on potential systems to add in the equation. Here we would like to add a comment on the parvocellular biased stimuli; since achromatropes only see black and white as well as grey scale, we expect to detect no neurophysiological response to stimuli presented in isoluminant colors (i.e. isoluminant colors transformed in grey scale present only one uniform grey color).

Finally, it could hypothetically be possible to investigate the M, P and K implication in High contrast stimuli in healthy subjects at the conditions that the two-previous experiments allowed to show that P, M and K pathways are the only visual systems activated by cones and rods activation when high contrast visual stimuli are presented (i.e cones excitation: HC=P+K; rods excitation HC=M).

4.2.2 Brain mechanisms for perceiving illusory lines in humans

4.2.2.a Summary of general conclusion

This paper was designed to investigated the sensitivity to IC forming line presented within the parafovea (5°visual angle) in human. This was intended to address knowledge gap between animal studies and human studies. Indeed, studies in animals often used stimuli inducing IC line and are in favor of a feedforward processing within V1/V2. On the other hand, human studies used illusory contour of shapes and propose a feedback from LOC. The results of this paper demonstrated that sensitivity to illusory lines occurs within the LOC at ~200ms post-stimulus. The timing and localization of the IC effect

42 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. confirms the crucial role of the LOC for the IC sensitivity and talk in favor of the model in which LOC feedback within V1/V2.

4.2.2.b General comments and future Directions

4.2.2.b.1 Illusory contours forming line biased toward the magnocellular and parvocellular pathway.

Here we propose to perform an experiment to merge paper 7.1 and 7.2 to investigate the detection of illusory contour forming line in isoluminant chromatic and low luminance achromatic contrasts. Previous studies demonstrated that the IC sensitivity within the LOC is independent of shapes when presented in high contrast (Knebel et al., 2011b; Mendola et al., 1999; Murray et al., 2004; Murray et al., 2002a). The results of paper 7.1 that demonstrated that IC sensitivity within the LOC is independent of the contrast in which stimuli are presented (magnocellular biased and parvocellular biased). The results of paper 7.2 showed a sensitivity to IC forming line within the LOC in high contrast which confirmed, with a new shape, that the IC sensitivity of the LOC is independent of the shape. Taking all these results together, we propose to investigate the brain response to the presentation of IC forming lines biased either toward the magnocellular or parvocellular conditions to confirm first that IC sensitivity within the LOC is not dependent of stimulus contrast. Second, this study will also allow to investigate whether the orientation to IC forming line happens when stimuli are biased toward the parvocellular or magnocellular pathway of vision. For this experiment, we foresee multiple outcomes. First, we expect sensitivity to IC forming line biased toward the M and P pathway to occur within the LOC, albeit with a greater activity for line presented in isoluminant chromatic contrast (P-biased (as the results of paper 7.1)). In more, based on the temporal shift found in paper 7.1 for the magnocellular biased stimuli, we expect a similar delay of ~30ms for the M-biased stimuli in comparison with the P- biased stimuli. Second, for the detection of orientation of the IC forming line, 2 outcomes are possible. First, magnocellular, and parvocellular pathway are sufficient to detect the orientation. In this case, we would expect to find a sensitivity to orientation for M and P biased stimuli localized within the primary and/or secondary visual cortices (The orientation of Illusory line was demonstrated to occur within the primary visual cortex in the unpublished results of the paper 7.2 in chapter 3.4.2). Second, as paper 7.1 failed to demonstrate a sensitivity to eccentricity of IC biased toward the M and P pathways, it is questionable whether M and P pathways will detect the orientation of illusory line. Therefore, we cannot exclude the non-detection of orientation of illusory line for this experiment.

4.2.2.b.2 Illusory contour processing: orientation detection of illusory contour forming shape.

Chapter 4.1.2 Orientation detection of illusory lines addressed the sensitivity to orientation of IC lines and introduced the discrepant results between intracranial recording in monkey that localized the orientation detection of illusory contour within V1/V2 (von der Heydt et al., 1984; von der Heydt and Peterhans, 1989a) and the detection of orientation of illusory shapes in human with the “thin /fat” model that localized the orientation sensitivity within LOC (Murray et al., 2006). An explanation for the discrepant results of chapter 4.1.2 and the study on “thin/fat” IC might come from the stimulus (line Vs. shape) as already proposed in chapter 4.1.2. We would like to stress that to the best of our knowledge no experiment has been conducted to investigate the perception of Illusory contour forming shape oriented in the four-cardinal point (Upward, Downward, Leftward, Rightward). Nonetheless, one study

43 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. that investigated the perception of IC forming square, triangle and diamond reported that an oblique effect might exist. This suggest that cardinal orientation of illusory shape lead to more efficient performance in comparison with oblique shape orientations (Norman, 2002). Based on these observations, we propose to investigate further the mechanism of illusory contour orientation sensitivity in human. To do that, we propose to use an equilateral triangle and rotate it in the four-cardinal orientation in an array of pacmen ‘inducers’ presented within the parafovea (5° visual angle). Figure 13a and 13b display the equilateral triangle in the upward orientation and leftward orientation. This proposition of an array within the parafovea is made to have a stimulus similar to those used in chapter 7.2. The use of an array would also ensure that participants do not attend at specific localization on the screen and render the detection of IC more challenging. The task would be a five-alternative force choice (5 AFC) with no contour and the 4 cardinal orientations as answers. For this experiment, we hypothesize different outcomes. First, based on previous results in macaque monkey and results of paper 7.2, we would expect the detection of orientation to be localized within V1/V2 after a feedback from the LOC at ~300-400 ms post-stimulus. If this hypothesis is verified, it would implicate that the orientation detection within the primary visual cortex is independent of the shape. Second, based on the “Thin/Fat” Illusory contours experiment, we propose that sensitivity to orientation might occur within the LOC at ~300-400 ms post-stimulus. If this latter hypothesis is verified, the conclusion would be that two different brain mechanisms exist for illusory contour orientation detection. One specific mechanism for the orientation of illusory lines, and one specific mechanism for the orientation of geometrical shapes.

Figure 13: Schematic of an array of pacmen forming an upward (a.) and leftward (b.) IC of equilateral triangle.

4.2.2.b.3 Lateral occipital cortex and retinotopic mapping.

As mentioned in 4.1.1, our results on the lateralization of IC contour within the parafovea reflect visual field preference. This preference might be linked with a low retinotopic organization within the LOC. Several studies investigated the retinotopic organization of the lateral occipital regions with retinotopic mapping methods and reported that LO region present a contralateral bias with two distinct visual field maps for the upper visual field (UVF) and lower visual field (LVF), named LO1 and LO2 (Larsson and Heeger, 2006). Others propose that LO regions show a contralateral bias with a combined representation of UFV and LVF (Large et al., 2008). Finally, an fMRI study that investigated the relation between the object-selective response and retinotopy within the human LOC reported that LOC present a bias for objects presented within the LVF, and most importantly that LO responses were stronger for stimuli presented within the fovea (Sayres and Grill-Spector, 2008). Taken together, these results demonstrated that a low retinotopic representation might exists within the LOC and not only within primary visual areas. Therefore, it appears important to assess the presentation of stimuli shifted toward the upper and lower visual regions of the parafovea to confirm that LOC activation that we demonstrated with

44 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. lateralized stimuli are the reflect of a low-retinotopic organization within the LOC rather than visual field preferences.

To achieve this, we propose to assess the presentation of IC forming line in the UVF and LVF within the parafovea (5° of visual angle). We propose to lateralize the presentation of a vertical array of pacmen toward the left or right part of the parafovea (Figure 14). We propose this lateralization of the array because the LOC demonstrated a preference for contralateral presentation in the experiment presented in 4.1.1. In more, previous studies showed that perceptual “filling-in” appears to be poorer when stimuli cross the vertical meridian when compared to stimuli that reside entirely within the left or right visual hemifield (Pillow and Rubin, 2002).

Now supposing that we perform an experiment with the left presentation of the array (Figure 14, left panel), if the LOC is reflecting a retinotopic organization, we expect to have differences in response to the ICeffect in the UVF compared to the ICeffect in the LFV localized within the bilateral LOC, but with a greater activation within the right LOC (following results of paper 7.2). We also expect a greater response for stimuli localized within the LVF in comparison to UVF (Sayres and Grill-Spector, 2008). On the contrary, If the array is localized within the right part of the parafovea (Figure 14, left panel), we also expect a difference within bilateral LOC; but with a greater activation in the left LOC (following our results in 7.2). and a greater activation for stimuli presented within the LVF. Finally, if the array is presented in the central location, we believe that the activity within bilateral LOC should be about the same; also with grater activity for the LVF.

Figure 14: Schematic of a vertical array of pacmen inducers shifted toward the left / right part of the parafovea (5° of visual angles).

4.2.2.b.4 Does the lateral occipital cortex detect salient regions or Illusory contours ?

Chapter 1.2.1 presented the three hypothetic models of Illusory contour processing. Although findings presented in paper 7.1 and 7.2 are supporting that IC sensitivity occurs within the LOC which in turn modulates responses in V1/V2 via feedback, the question regarding the sensitivity of the LOC for Illusory contour or salient regions remains. Indeed, the third model presented in chapter 1.2.1 is in favor of a sensitivity of the LOC to salient region (i.e crude-region based segmentation) which modulates IC sensitivity within V1 and V2 via feedback (Stanley and Rubin, 2003; Yoshino et al., 2006). An indirect possibility to investigate which of these two models is correct might come from investigation in healthy individuals and hemi-spatial neglect patients performing line bisection task (i.e. participants look at a line and bisect it within its center).

The following paragraph will introduce hemi-spatial neglect, and specific findings that were made with this particular impairment in conjunction with the use of illusory contour. This will be followed by a

45 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. presentation of a similar effect known as pseudoneglect in healthy population and a potential experiment to try to identify if the LOC can detect Salient region as well as Illusory contour.

Hemi-spatial neglect is a neurological condition that is subsequent to damage in one brain hemisphere such as a stroke, a clogged artery or a cranial trauma. Patients affected by hemi-spatial neglect fail to react or process stimuli (auditory, visual or tactile) presented in the contralateral side of the damaged hemisphere (for a review on Spatial hemineglect in humans, see (Kerkhoff, 2001). Therefore, when patients perform a line bisection task, they fail to place the center at the correct place. Specifically, they judge that the middle of the line is located approximately to the right quarter of the line because they neglect the left part of the line. Two notable studies investigated the implicit and explicit detections of illusory contour and their effect on line bisection task with patients affect by a right hemisphere damage and presenting left neglect. These studies proved that the grouping of illusory contour occurs without spatial attention and appeared to be independent from the severity of the neglect. When analyzing the results of the bisection task when the line was surrounded by kanizsa-type illusory contour (Figure 15a), patients tended to bisect closer to the true midpoint. The critical finding of these studies was that only patients with an intact lateral occipital cortex were able to perform the bisection task better when the line was surrounded by Illusory contours. (Vuilleumier et al., 2001b; Vuilleumier and Landis, 1998). This finding indirectly demonstrated the crucial role of the lateral occipital regions for the detection of illusory contours.

The line bisection task might also be feasible in healthy participants because an effect known as pseudoneglect exists in the population. It is described as a mild asymmetry in spatial attention, present in neurologically normal individuals. This asymmetry is characterized by a favoring of the left side of space which induce leftward errors in line bisection task (the bisection is shifted toward the left compared to the true midpoint (Brooks et al., 2014)). Pseudoneglect is not as severe as hemi-spatial neglect, but shows robust and consistent behavioral phenomenon in healthy subjects (Turnbull and McGeorge, 1998). It was demonstrated with line-length discrimination task (i.e the task is to judge if two line are the same length or not) that healthy participants are influenced by unconscious perception of features that induce grouping mechanism (Moore and Egeth, 1997) as well as attentional factors (Porac et al., 2006). Therefore, we can reasonably think that the response to line bisection task might as well be influenced by illusory contour in healthy participants. As far as we know, this has never been investigated.

Based on these observations in hemi-spatial neglect and healthy participants, we propose a line bisection task with salient regions and kanizsa-type Illusory contour (Figure 15 a and b) to investigate the potential of salient regions to induce the perception of Illusory contours. We hypothesized that Kanizsa-type illusory contour (IC) should induce a better judgment of the midpoint in bisection task in both healthy participants and hemi-spatial neglect compared to its no contour counterpart (NoIC). This is supported by previous study in hemi-spatial neglect patients (Vuilleumier et al., 2001b). For the line surrounded by salient region inducers (SR), we propose two different possibilities. The first possibility is that SR are not sufficient to induce the perception of illusory contours in both groups. In this case, we expect the results of the bisection to be equivalent in the SR condition compared to its no contour counterpart (NoSR). The second outcome supposes that salient regions are sufficient to induce the perception illusory contours in both groups. If this is the case, we expect the results of the bisections task to be

46 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. better in SR condition than in the no contour condition (SR pacmen turned facing outside). Additionally, we could address a potential question about the efficacy of SR compared to Kanizsa illusory contour. Indeed, behavioral differences between SR and kanizsa might appear. For example, if SR induce the perception of illusory contour; but with a weaker force than Illusory contours, the judgment of the midpoint in SR condition might be better than in the no contour condition but still be shifted toward the left in comparison to the true midpoint of the line.

In a first step, we propose to perform a behavioral experiment to first replicate prior results made with hemi-spatial neglect and investigate if pseudo neglect can be corrected by perceptual grouping and also investigate the potential effect of SR on bisection task. For this we propose to use stimuli composed of a line that is surrounded by IC, NoIC,SR, NoSR that participant will have to bisect with a pen. We propose to measure the deviation from the true midpoint in the four conditions to assess if participants perform the task better with IC and SR compared to NoIC and NoSR in which results should be equivalent.

In a second step, we propose to record electroencephalography with IC, NoIC ,SR and NoSR. To avoid movements artifacts, we propose a slightly different task for EEG sessions in which participants would have to judge whether an artificial bisecting line is localized at the midpoint or shifted (either to the left or right (Figure 15a and b)). We propose to shift the bisection line at five different positions toward each directions in increment of 0.6° as performed in (Foxe et al., 2003). In this experiment, we expect participants to be more accurate in IC/SR conditions for the judgement of the center. The electroencephalography will allow to investigate the time-course of brain processes to all stimuli and allow the detection of which brain regions are involved in the task. This will hopefully help proving that the LOC is or is not sensitive to salient regions.

Figure 15: Schematic of a bisected line surrounded by (a.) Salient Region inducers and (b.)Kanisza-type Illusory Contours

4.3 Sensory substitution 4.3.1 Training Orientation Discrimination Using Visual-to-auditory Sensory

Substitution

4.3.1.a Summary of general conclusion

This paper was designed to investigate the effect of training with a visual-to-auditory sensory substitution device. To the best of our knowledge, there exist no report of the extent to which training with one stimulus set extends to facilitate the discrimination of another, untrained stimulus set. The

47 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. present study addressed this issue by training sighted participants on the vertical orientation with a specific soundscapes stimulus set from the EyeMusic SSD to investigate if a potential transfer can operate to the horizontal orientation. This study demonstrated that participants augment their accuracy after training for both the trained and untrained stimuli, albeit with better results for the trained stimuli. Moreover, the trained orientation yielded better results independently of the frequency pitch in which the stimuli are presented. The conclusion was that participants can transfer the training of orientation across frequencies. This study finally proposes a new method to train participants with a visual to auditory sensory substitution device that differs slightly from the current training that is given to blind individuals. This study proposes to avoid one-on-one training to avoid a potential bias that can be introduce by the experimenter when he gives feedback during training. Indeed, a bias is inherent to the feedback because the experiment adapts his speech to the subjects who is trained.

4.3.1.b General comments and future Direction

4.3.1.b.1 Electroencephalography recording to assess the timing between auditory stimulation and activation within visual cortices.

As presented in Chapter 1.4.1, visual-to-auditory sensory substitution devices have been demonstrated to activate brain areas that were thought to be dedicated to vision when blind individuals as well as sighted individuals are tested with auditory soundscapes. All studies that demonstrated these activations used fMRI. Due to timing constraints in fMRI, it appears difficult to identify brain dynamics between the auditory cortex and the visual cortex. Therefore, we propose to perform an electroencephalography study with Snellen ‘E’s (Figure 16b, upper part) that have already been investigated with the vOIce and the EyeMusic sensory substitution devices (Levy-Tzedek et al., 2014; Striem-Amit et al., 2012b). In both cases, participants demonstrated activation within the visual word form area in fMRI when hearing letters Vs other categories (Abboud et al., 2015; Striem-Amit et al., 2012a). Therefore, we propose to train participants to discriminate the orientation of Snellen E with the EyeMusic and record electroencephalography when participants hear soundscapes of ‘E’s vs. other shapes to assess the temporal difference between auditory and visual areas. As source localization is blurred in EEG, we expect to localize the sensitivity to ‘E’ vs other shapes within the left lateral occipitotemporal sulcus.

4.3.1.b.2 Can healthy individuals and blind individuals perform visual completion with visual to auditory sensory substitution devices?

One shortcoming of all studies in sensory substitution published up to now, is that auditory soundscapes are invariably based on real objects or scrambled images. It was also the case in the paper presented in 7.3. Therefore, one question that remains is whether and how sensory substitution devices can engender perceptual completion with auditory soundscapes. To answer this question, Kanizsa-type illusory contours are a stimulus of choice, because they were extensively investigated in sighted individuals who demonstrated a sensitivity to IC localized within the LOC. This thesis provides similar finding on IC sensitivity in paper presented in 7.1 and 7.2.

48 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

To investigate further the question of completion with visual-to-auditory SSDs, we propose to create soundscapes of an illusory contour forming an isosceles triangle and the no contour counterpart (Figure 16a). This shape is proposed for several reasons: first, if the triangle is rotated, the tip of the isosceles triangle will either point upward, downward, leftward, or rightward (i.e it has a direction). Second, a triangle is formed by three ‘pacmen’ inducers that could be sounded one after the other when the triangle in facing upward or downward. This orientation condition would be used to ensure that participant really perceive the Illusory shape and did not only rely on the orientation of ‘pacmen’ mouths (i.e if the tree mouths are facing inside there is an illusory shape). The task will be a 5 alternative-forced choice with the 4 cardinal orientations and no contour as possible answers for each soundscape presentation.

Supposing that perceptual grouping occurs in blind individuals and sighted participants with auditory soundscapes generated with a visual to auditory SSD, we expect the sensitivity to IC to be localized within the lateral occipital cortex for both groups. The second question that we would like to address, if possible, is whether blind individuals and sighted participants can detect the orientation of the illusory shape with auditory soundscape. For this analysis, we first expect to have better results in accuracy when triangle is facing upward and downward. We make this assumption because when the triangle is facing rightward or leftward, the two ‘pacmen’ inducers that form the base of the triangle are superimposed; therefore, they are sounded at the same time which render the discrimination of the sounds more complex. Given this, if participants can discriminate the orientation, we make the hypothesis that orientation sensitivity would occur either in the LOC or in the primary visual cortex as it was hypothesized in 4.2.2.b.2 with a visual stimuli of triangle.

In order to start this experiment, we identified two possible problems. First blind individuals should learn the orientation of pacmen as they did in previous studies with the orientation of Snellen’s E (Striem-Amit et al., 2012b) (Figure15 b.). We propose to train them to discriminate the orientation of one pacman presented in the center of the image as proposed by results of paper 6.3. Second, which appears as the most critical point, is teaching blind individuals what are illusory contours. To overcome this issue, we propose to try explaining this to blind individuals with haptic stimuli. If this possibility does not work, we propose to take advantage of the color implementation present in the Eyemusic algorithm. The idea would be to present pacmen in one color and the illusory shape in another color (like in real image). By playing these two different sounds, we hope that blind individuals understand the basis of illusory contours. The second step would be to remove the illusory shape to leave just the pacmen and control with a small psychophysics task that blind participant understood what are illusory contours. Perhaps the best way to explain illusory contour to blind individuals will consist in the presentation of the two soundscapes (i.e. pacmen + Shape/ only pacmen) in conjunction with the haptic stimuli. All these hypotheses remain to be tested in further work.

Figure 16: Schematic of stimuli for sensory substitution experiment

49 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

a. displays an IC forming an Isosceles triangle and its non-contour counterpart. b. displays the similarity between Snellen ’E’ and ‘pacmen’

mouths’ orientations.

4.3.1.b.3 Can blind individuals learn the basis of visual-to-auditory sensory substitution device via haptic feedback?

One finding of paper 7.3 is that sighted individuals can learn the basis of transformation of the EyeMusic SSD with visual feedback only. Therefore, we ask the question of training blind participants with haptic feedback instead of a one-on-one training. As discussed in paper 7.3, this would allow to create a controlled training for all participants and avoid a potential bias due to experimenter feedback. If we can demonstrate that blind participants are able to learn the basis of the EyeMusic algorithm with haptic feedback, it would open a door towards a possible training of participants on their own which appears a critical point towards the use of sensory substitution devices in rehabilitation. Moreover, if participants can learn the algorithm on their own, they would not have to go in a lab to be trained; which would be a huge time gain for them and experimenter. We believe that blind participants might be able to learn the Eyemusic SSD with haptic feedback because previous studies demonstrated that blind individuals performed better than sighted controlled blindfolded in tactile acuity task (i.e tactile grating orientation discrimination). However only the early and late blind showed better results in a 3D haptic shape discrimination task (Norman and Bartholomew, 2011). Other studies recording fMRI at the same time as object recognition by touch in blind and sighted individuals reported that visual imagery is not necessary for activation in visual cortices (Amedi et al., 2010). Taken together, these observations are supporting the idea that blind individuals would be able to recognize shapes by tactile stimulation. Therefore, we propose to perform the same experiment as paper 7.3 with blind participants that are naïve to SSDs with 3D haptic feedback instead of experimenter feedback to try replicate results made in healthy subjects with visual feedback. We expect early and late blind that had a sense of vision, to be able to perform the task as good as sighted participants. Concerning the congenitally blind, the task might be more complicated (Norman and Bartholomew, 2011). Nevertheless, we believe that congenitally blind individuals might be able to perform the task because the objects that are presented in the training sessions are letters that share similarity with points in braille alphabets (W and R in braille form a T rightward and leftward respectively, and Yi in Braille is equivalent to a U rightward). Finally, if blind participants can indeed learn the basis of transformation of the EyeMusic with haptic feedback, we expect participants to perform better on complex stimuli presented in the trained orientation independently of the frequency of presentation.

50 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

CHAPTER 5 CONCLUSION

5.1 Conclusion on neurophysiological papers

The two neurophysiological papers presented in this thesis demonstrated that the lateral occipital cortex plays a crucial role in the sensitivity to illusory contours. The first paper demonstrated for the first time the existence of multiple brain circuits that projects in the ventral visual pathway (LOC) where the IC sensitivity occurs. The second paper demonstrated for the first time that the sensitivity to IC forming line in human occurs in the LOC. Taken together, these two papers supports a model whereby the IC sensitivity occurs within the LOC which then modulate response within V1/V2 via feedback. Despite these new findings, many questions remain in illusory contour processing. The most challenging is to confirm this model. The ideal solution would be to prove that feedback indeed occurs by showing first activations within the LOC subsequently followed by activations within primary visual cortices. Few studies were able to measure and report actual proof of feedback from the LOC. Nevertheless, a MEG study using illusory contour reported a timing ~80ms from LOC to V1/V2 (Halgren et al., 2003; Knebel et al., 2011b). To prove that IC sensitivity really occurs within the LOC, the optimal solution would be to perform Electrocorticography (ECoG) in human with electrodes placed in V1/V2 as well as within the LOC to record where the first response occurs. As ECoG in human is rather complicated, perhaps recording EEG from patients that suffer from a homonymous hemianopia might be the best solution. Homonymous Hemianopia affects the right halves or the left halves of the visual fields of both eyes due to lesion in the left or right occipital lobes, respectively. If these patients present a lesion of V1/V2 and can perceive Illusory contours that cover both hemifields of vision, that might prove that the LOC is the responsible region for IC sensitivity.

5.2 Bridging the gap between Sensory substitution, perceptual

completion and the use of sensory substitution in everyday life

situations.

Multiple evidences demonstrating a possible use of SSDs for the rehabilitation of visually impaired individuals have accumulated over the last two decades. As mentioned in the introduction, blind individuals trained on visual-to auditory sensory substitution devices can discriminate body positions, letters, numbers, and objects when hearing an auditory soundscape. As discussed in chapter 4.3.1.b.2, one critical question that remains open and of interest for basic research and rehabilitation is whether blind individuals hearing auditory soundscape can perceive illusory contour as it is common in sighted individual that see a natural scene. Proving this, would, from the basic research side, provide precious information on the brain plasticity and dynamics in visually impaired individuals. From a clinical point of view, this might lead to improvements of training procedures as well as possible technical modification that could open sensory substitution devices range of use. Another critical point that remain concerns the perception of depth that rely on 3-dimensional vision. A particular study from Renier and

51 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. colleagues (2005) demonstrated that normally sighted individuals trained with a visual-to-auditory sensory substitution devices recruit occipito-parietal areas and the ventral visual pathway to a lesser extent. This demonstrated that some “visual” areas of the brain might be multimodal and recruited for depth perception with a sense other than vision. Similar finding were also demonstrated later in early blind patients (i.e individuals that had a sense of vision but lost it early in life (Renier and De Volder, 2010). Despite these findings, it remains unknown if congenitally blind individuals can achieve depth perception via sensory substitution. On the one hand, if congenitally blind individuals can learn depth; this might confirm the multimodality of “visual” brain regions dedicated to depth perception as it was the case with early blind individuals. On the other hand, if congenitally blind individuals cannot achieve depth perception via sensory substitution, this might indicate that depth perception can only be learned with vision. Finally, if future studies can demonstrate that blind individuals can achieve either perceptual completion or depth perception with sensory substitution device delivering auditory information to replace vision, one of the major problem that would remain is the processing of real world images. Indeed, most of the pictures that were used in previous studies were rather simple images. These observations implicate that sensory substitution devices need to be further improved and that synergies with computer science are required to develop methods to extract important features (which remain to be defined) of real world images to finally implement a possible everyday use of these devices in blind users.

CHAPTER 6 REFERENCES

Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., Amedi, A., 2014a. EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution. Restor. Neurol. Neurosci. 32, 247–257. doi:10.3233/RNN-130338 Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., Amedi, A., 2014b. EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution. Restor. Neurol. Neurosci. 32, 247–257. doi:10.3233/RNN-130338 Abboud, S., Maidenbaum, S., Dehaene, S., Amedi, A., 2015. A number-form area in the blind. Nat Commun 6, 6026. doi:10.1038/ncomms7026 Allison, T., Puce, A., Spencer, D.D., McCarthy, G., 1999. Electrophysiological studies of human face perception. I: Potentials generated in occipitotemporal cortex by face and non-face stimuli. Cereb. Cortex 9, 415–430. Altschuler, T.S., Molholm, S., Russo, N.N., Snyder, A.C., Brandwein, A.B., Blanco, D., Foxe, J.J., 2012. Early electrophysiological indices of illusory contour processing within the lateral occipital complex are virtually impervious to manipulations of illusion strength. Neuroimage 59, 4074–4085. doi:10.1016/j.neuroimage.2011.10.051 Amedi, A., Malach, R., Hendler, T., Peled, S., Zohary, E., 2001. Visuo-haptic object-related activation in the ventral visual pathway. Nat Neurosci 4, 324–330. doi:10.1038/85201 Amedi, A., Raz, N., Azulay, H., Malach, R., Zohary, E., 2010. Cortical activity during tactile exploration of objects in blind and sighted humans. Restor. Neurol. Neurosci. 28, 143–156. doi:10.3233/RNN-2010-0503 Amedi, A., Stern, W.M., Camprodon, J.A., Bermpohl, F., Merabet, L., Rotman, S., Hemond, C., Meijer, P., Pascual-Leone, A., 2007. Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nat. Neurosci. 10, 687–689. doi:10.1038/nn1912 Anken, J., Knebel, J.-F., Crottaz-Herbette, S., Matusz, P.J., Lefebvre, J., Murray, M.M., 2016a. Cue- dependent circuits for illusory contours in humans. Neuroimage 129, 335–344. doi:10.1016/j.neuroimage.2016.01.052

52 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. Anken, J., Knebel, J.-F., Crottaz-Herbette, S., Matusz, P.J., Lefebvre, J., Murray, M.M., 2016b. Cue- dependent circuits for illusory contours in humans. Neuroimage 129, 335–344. doi:10.1016/j.neuroimage.2016.01.052 Appelle, S., 1972. Perception and discrimination as a function of stimulus orientation: the “oblique effect” in man and animals. Psychol Bull 78, 266–278. Aspell, J.E., Wattam-Bell, J., Atkinson, J., Braddick, O.J., 2010. Differential human brain activation by vertical and horizontal global visual textures. Exp Brain Res 202, 669–679. doi:10.1007/s00221-010-2173-y Bach-y-Rita, P., Collins, C.C., Saunders, F.A., White, B., Scadden, L., 1969a. Vision substitution by tactile image projection. Nature 221, 963–964. Bach-y-Rita, P., Collins, C.C., White, B., Saunders, F.A., Scadden, L., Blomberg, R., 1969b. A tactile vision substitution system. Am J Optom Arch Am Acad Optom 46, 109–111. Brewer, A.A., Liu, J., Wade, A.R., Wandell, B.A., 2005. Visual field maps and stimulus selectivity in human ventral occipital cortex. Nat. Neurosci. 8, 1102–1109. doi:10.1038/nn1507 Brooks, J.L., Della Sala, S., Darling, S., 2014. Representational pseudoneglect: a review. Neuropsychol Rev 24, 148–165. doi:10.1007/s11065-013-9245-2 Busskamp, V., Duebel, J., Balya, D., Fradot, M., Viney, T.J., Siegert, S., Groner, A.C., Cabuy, E., Forster, V., Seeliger, M., Biel, M., Humphries, P., Paques, M., Mohand-Said, S., Trono, D., Deisseroth, K., Sahel, J.A., Picaud, S., Roska, B., 2010. Genetic Reactivation of Cone Photoreceptors Restores Visual Responses in Retinitis Pigmentosa. Science 329, 413–417. doi:10.1126/science.1190897 Butler, P.D., Martinez, A., Foxe, J.J., Kim, D., Zemon, V., Silipo, G., Mahoney, J., Shpaner, M., Jalbrzikowski, M., Javitt, D.C., 2007. Subcortical visual dysfunction in schizophrenia drives secondary cortical impairments. Brain 130, 417–430. doi:10.1093/brain/awl233 Butler, P.D., Schechter, I., Zemon, V., Schwartz, S.G., Greenstein, V.C., Gordon, J., Schroeder, C.E., Javitt, D.C., 2001. Dysfunction of early-stage visual processing in schizophrenia. Am J Psychiatry 158, 1126–1133. Calderone, D.J., Martinez, A., Zemon, V., Hoptman, M.J., Hu, G., Watkins, J.E., Javitt, D.C., Butler, P.D., 2013. Comparison of psychophysical, electrophysiological, and fMRI assessment of visual contrast responses in patients with schizophrenia. Neuroimage 67, 153–162. doi:10.1016/j.neuroimage.2012.11.019 Capelle, C., Trullemans, C., Arno, P., Veraart, C., 1998. A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Transactions on Biomedical Engineering 45, 1279–1293. doi:10.1109/10.720206 Cohen, L., Dehaene, S., 1991. Neglect Dyslexia for Numbers? A Case Report. Cognitive Neuropsychology 8, 39–58. doi:10.1080/02643299108253366 Cohen, L., Dehaene, S., Naccache, L., Lehéricy, S., Dehaene-Lambertz, G., Hénaff, M.A., Michel, F., 2000. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain 123 ( Pt 2), 291– 307. Cohen, L., Martinaud, O., Lemer, C., Lehéricy, S., Samson, Y., Obadia, M., Slachevsky, A., Dehaene, S., 2003. Visual word recognition in the left and right hemispheres: anatomical and functional correlates of peripheral alexias. Cereb. Cortex 13, 1313–1333. Collignon, O., Vandewalle, G., Voss, P., Albouy, G., Charbonneau, G., Lassonde, M., Lepore, F., 2011. Functional specialization for auditory–spatial processing in the occipital cortex of congenitally blind humans. PNAS 108, 4435–4440. doi:10.1073/pnas.1013928108 Dacey, D.M., Petersen, M.R., 1992. Dendritic field size and morphology of midget and parasol ganglion cells of the human retina. Proc. Natl. Acad. Sci. U.S.A 89, 9666–9670. De Valois, R.L., Yund, E.W., Hepler, N., 1982. The orientation and direction selectivity of cells in macaque visual cortex. Vision Res. 22, 531–544. Dehaene, S., Pegado, F., Braga, L.W., Ventura, P., Nunes Filho, G., Jobert, A., Dehaene-Lambertz, G., Kolinsky, R., Morais, J., Cohen, L., 2010. How learning to read changes the cortical networks for vision and language. Science 330, 1359–1364. doi:10.1126/science.1194140

53 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

Di Russo, F., Martínez, A., Hillyard, S.A., 2003. Source analysis of event-related cortical activity during visuo-spatial attention. Cereb. Cortex 13, 486–499. Doniger, G.M., Foxe, J.J., Murray, M.M., Higgins, B.A., Javitt, D.C., 2002. Impaired visual object recognition and dorsal/ventral stream interaction in schizophrenia. Arch. Gen. Psychiatry 59, 1011–1020. Doniger, G.M., Foxe, J.J., Murray, M.M., Higgins, B.A., Snodgrass, J.G., Schroeder, C.E., Javitt, D.C., 2000. Activation timecourse of ventral visual stream object-recognition areas: high density electrical mapping of perceptual closure processes. J Cogn Neurosci 12, 615–621. Doniger, G.M., Silipo, G., Rabinowicz, E.F., Snodgrass, J.G., Javitt, D.C., 2001. Impaired sensory processing as a basis for object-recognition deficits in schizophrenia. Am J Psychiatry 158, 1818–1826. Downing, P.E., Jiang, Y., Shuman, M., Kanwisher, N., 2001. A cortical area selective for visual processing of the human body. Science 293, 2470–2473. doi:10.1126/science.1063414 Felleman, D.J., Van Essen, D.C., 1991. Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47. Ffytche, D.H., Zeki, S., 1996. Brain activity related to the perception of illusory contours. Neuroimage 3, 104–108. doi:10.1006/nimg.1996.0012 Foxe, J.J., Doniger, G.M., Javitt, D.C., 2001. Early visual processing deficits in schizophrenia: impaired P1 generation revealed by high-density electrical mapping. Neuroreport 12, 3815–3820. Foxe, J.J., McCourt, M.E., Javitt, D.C., 2003. Right hemisphere control of visuospatial attention: line- bisection judgments evaluated with high-density electrical mapping and source analysis. Neuroimage 19, 710–726. Foxe, J.J., Murray, M.M., Javitt, D.C., 2005. Filling-in in Schizophrenia: a High-density Electrical Mapping and Source-analysis Investigation of Illusory Contour Processing. Cereb. Cortex 15, 1914–1927. doi:10.1093/cercor/bhi069 Furmanski, C.S., Engel, S.A., 2000. An oblique effect in human primary visual cortex. Nat. Neurosci. 3, 535–536. doi:10.1038/75702 Grave de Peralta Menendez, R., Gonzalez Andino, S., Lantz, G., Michel, C.M., Landis, T., 2001. Noninvasive localization of electromagnetic epileptic activity. I. Method descriptions and simulations. Brain Topogr 14, 131–137. Grave de Peralta Menendez, R., Murray, M.M., Michel, C.M., Martuzzi, R., Gonzalez Andino, S.L., 2004a. Electrical neuroimaging based on biophysical constraints. Neuroimage 21, 527– 539. doi:10.1016/j.neuroimage.2003.09.051 Grave de Peralta Menendez, R., Murray, M.M., Michel, C.M., Martuzzi, R., Gonzalez Andino, S.L., 2004b. Electrical neuroimaging based on biophysical constraints. Neuroimage 21, 527– 539. doi:10.1016/j.neuroimage.2003.09.051 Green, M.F., Lee, J., Wynn, J.K., Mathis, K.I., 2011. Visual masking in schizophrenia: overview and theoretical implications. Schizophr Bull 37, 700–708. doi:10.1093/schbul/sbr051 Grill-Spector, K., Kourtzi, Z., Kanwisher, N., 2001. The lateral occipital complex and its role in object recognition. Vision Res 41, 1409–1422. Grill-Spector, K., Kushnir, T., Hendler, T., Edelman, S., Itzchak, Y., Malach, R., 1998. A sequence of object-processing stages revealed by fMRI in the human occipital lobe. Hum Brain Mapp 6, 316–328. Grosof, D.H., Shapley, R.M., Hawken, M.J., 1993. Macaque V1 neurons can signal “illusory” contours. Nature 365, 550–552. doi:10.1038/365550a0 Grotheer, M., Ambrus, G.G., Kovács, G., 2016. Causal evidence of the involvement of the number form area in the visual detection of numbers and letters. NeuroImage 132, 314–319. doi:10.1016/j.neuroimage.2016.02.069 Halgren, E., Mendola, J., Chong, C.D.R., Dale, A.M., 2003. Cortical activation to illusory shapes as measured with magnetoencephalography. Neuroimage 18, 1001–1009. Heimler, B., Striem-Amit, E., Amedi, A., 2015. Origins of task-specific sensory-independent organization in the visual and auditory brain: neuroscience evidence, open questions and clinical implications. Curr. Opin. Neurobiol. 35, 169–177. doi:10.1016/j.conb.2015.09.001

54 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. Hemond, C.C., Kanwisher, N.G., Op de Beeck, H.P., 2007. A preference for contralateral stimuli in human object- and face-selective cortex. PLoS ONE 2, e574. doi:10.1371/journal.pone.0000574 Hirsch, J., DeLaPaz, R.L., Relkin, N.R., Victor, J., Kim, K., Li, T., Borden, P., Rubin, N., Shapley, R., 1995. Illusory contours activate specific regions in human visual cortex: evidence from functional magnetic resonance imaging. Proceedings of the National Academy of Sciences of the United States of America 92, 6469–6473. Hubel, D.H., 1963. THE VISUAL CORTEX OF THE BRAIN. Sci. Am. 209, 54–62. Hubel, D.H., Wiesel, T.N., 1977. Ferrier lecture. Functional architecture of macaque monkey visual cortex. Proc. R. Soc. Lond., B, Biol. Sci 198, 1–59. Hubel, D.H., Wiesel, T.N., 1962. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol. (Lond.) 160, 106–154. Javitt, D.C., 2009. When doors of perception close: bottom-up models of disrupted cognition in schizophrenia. Annu Rev Clin Psychol 5, 249–275. doi:10.1146/annurev.clinpsy.032408.153502 Jeffreys, D.A., Axford, J.G., 1972. Source locations of pattern-specific components of human visual evoked potentials. II. Component of extrastriate cortical origin. Exp Brain Res 16, 22–40. Jeffries, A.M., Killian, N.J., Pezaris, J.S., 2014. Mapping the primate lateral geniculate nucleus: A review of experiments and methods. Journal of Physiology-Paris, Special issue from “AREADNE 2012: Research in Encoding and Decoding of Neural Ensembles” held in Santorini, 21-25 June 2012 108, 3–10. doi:10.1016/j.jphysparis.2013.10.001 Jindra, L.F., Zemon, V., 1989. Contrast sensitivity testing: a more complete assessment of vision. J Cataract Refract Surg 15, 141–148. Kanizsa, G., 1976. Subjective contours. Sci. Am 234, 48–52. Kaplan, E., 1991. The receptive field structure of retinal ganglion celles in cat and monkey, in: Vision and Visual Dysfunction. leventhal AG, pp. 10–40. Kerkhoff, G., 2001. Spatial hemineglect in humans. Prog. Neurobiol. 63, 1–27. Knebel, J.-F., Javitt, D.C., Murray, M.M., 2011a. Impaired early visual response modulations to spatial information in chronic schizophrenia. Psychiatry Res 193, 168–176. doi:10.1016/j.pscychresns.2011.02.006 Knebel, J.-F., Javitt, D.C., Murray, M.M., 2011b. Impaired early visual response modulations to spatial information in chronic schizophrenia. Psychiatry Res 193, 168–176. doi:10.1016/j.pscychresns.2011.02.006 Knebel, J.-F., Murray, M.M., 2012. Towards a resolution of conflicting models of illusory contour processing in humans. Neuroimage 59, 2808–2817. doi:10.1016/j.neuroimage.2011.09.031 Koenig, T., Kottlow, M., Stein, M., Melie-García, L., 2011. Ragu: a free tool for the analysis of EEG and MEG event-related scalp field data using global randomization statistics. Comput Intell Neurosci 2011, 938925. doi:10.1155/2011/938925 Lacey, S., Flueckiger, P., Stilla, R., Lava, M., Sathian, K., 2010. Object familiarity modulates the relationship between visual object imagery and haptic shape perception. Neuroimage 49, 1977–1990. doi:10.1016/j.neuroimage.2009.10.081 Large, M.-E., Culham, J., Kuchinad, A., Aldcroft, A., Vilis, T., 2008. fMRI reveals greater within- than between-hemifield integration in the human lateral occipital cortex. Eur. J. Neurosci. 27, 3299–3309. doi:10.1111/j.1460-9568.2008.06270.x Larsson, J., Heeger, D.J., 2006. Two retinotopic visual areas in human lateral occipital cortex. J. Neurosci. 26, 13128–13142. doi:10.1523/JNEUROSCI.1657-06.2006 Lehmann, D., Ozaki, H., Pal, I., 1987. EEG alpha map series: brain micro-states by space-oriented adaptive segmentation. Electroencephalogr Clin Neurophysiol 67, 271–288. Lehmann, D., Skrandies, W., 1980. Reference-free identification of components of checkerboard- evoked multichannel potential fields. Electroencephalogr Clin Neurophysiol 48, 609–621. Levy-Tzedek, S., Riemer, D., Amedi, A., 2014. Color improves “visual” acuity via sound. Front Neurosci 8, 358. doi:10.3389/fnins.2014.00358

55 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

Li, B., Peterson, M.R., Freeman, R.D., 2003. Oblique Effect: A Neural Basis in the Visual Cortex. Journal of Neurophysiology 90, 204–217. doi:10.1152/jn.00954.2002 Luo, Y.H.-L., da Cruz, L., 2016. The Argus(®) II Retinal Prosthesis System. Prog Retin Eye Res 50, 89–107. doi:10.1016/j.preteyeres.2015.09.003 Malach, R., Reppas, J.B., Benson, R.R., Kwong, K.K., Jiang, H., Kennedy, W.A., Ledden, P.J., Brady, T.J., Rosen, B.R., Tootell, R.B.H., 1995. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Sciences of the United States of America 92, 8135–8139. doi:10.1073/pnas.92.18.8135 Mangun, G.R., 1995. Neural mechanisms of visual selective attention. Psychophysiology 32, 4–18. Mannion, D.J., McDonald, J.S., Clifford, C.W.G., 2010. The influence of global form on local orientation anisotropies in human visual cortex. Neuroimage 52, 600–605. doi:10.1016/j.neuroimage.2010.04.248 Martínez, A., Hillyard, S.A., Dias, E.C., Hagler, D.J., Butler, P.D., Guilfoyle, D.N., Jalbrzikowski, M., Silipo, G., Javitt, D.C., 2008. Magnocellular pathway impairment in schizophrenia: evidence from functional magnetic resonance imaging. J. Neurosci. 28, 7492–7500. doi:10.1523/JNEUROSCI.1852-08.2008 McCandliss, B.D., Cohen, L., Dehaene, S., 2003. The visual word form area: expertise for reading in the fusiform gyrus. Trends Cogn. Sci. (Regul. Ed.) 7, 293–299. Meijer, P.B., 1992. An experimental system for auditory image representations. IEEE Trans Biomed Eng 39, 112–121. doi:10.1109/10.121642 Mendola, J.D., Dale, A.M., Fischl, B., Liu, A.K., Tootell, R.B., 1999. The representation of illusory and real contours in human cortical visual areas revealed by functional magnetic resonance imaging. J. Neurosci 19, 8560–8572. Merigan, W.H., Maunsell, J.H., 1993. How parallel are the primate visual pathways? Annu. Rev. Neurosci. 16, 369–402. doi:10.1146/annurev.ne.16.030193.002101 Michel, C.M., Murray, M.M., Lantz, G., Gonzalez, S., Spinelli, L., Grave de Peralta, R., 2004. EEG source imaging. Clin Neurophysiol 115, 2195–2222. doi:10.1016/j.clinph.2004.06.001 Michel, C.M., Seeck, M., Landis, T., 1999. Spatiotemporal Dynamics of Human Cognition. News Physiol. Sci. 14, 206–214. Moore, C.M., Egeth, H., 1997. Perception without attention: evidence of grouping under conditions of inattention. J Exp Psychol Hum Percept Perform 23, 339–352. Murray, M.M., Brunet, D., Michel, C.M., 2008. Topographic ERP analyses: a step-by-step tutorial review. Brain Topogr 20, 249–264. doi:10.1007/s10548-008-0054-5 Murray, M.M., Brunet, D., Michel, C.M., 2008. Topographic ERP analyses: a step-by-step tutorial review. Brain Topogr 20, 249–264. doi:10.1007/s10548-008-0054-5 Murray, M.M., Foxe, D.M., Javitt, D.C., Foxe, J.J., 2004. Setting boundaries: brain dynamics of modal and amodal illusory shape completion in humans. J. Neurosci 24, 6898–6903. doi:10.1523/JNEUROSCI.1996-04.2004 Murray, M.M., Foxe, D.M., Javitt, D.C., Foxe, J.J., 2004. Setting boundaries: brain dynamics of modal and amodal illusory shape completion in humans. J. Neurosci. 24, 6898–6903. doi:10.1523/JNEUROSCI.1996-04.2004 Murray, M.M., Foxe, J.J., Higgins, B.A., Javitt, D.C., Schroeder, C.E., 2001. Visuo-spatial neural response interactions in early cortical processing during a simple reaction time task: a high-density electrical mapping study. Neuropsychologia 39, 828–844. Murray, M.M., Imber, M.L., Javitt, D.C., Foxe, J.J., 2006. Boundary completion is automatic and dissociable from shape discrimination. J. Neurosci 26, 12043–12054. doi:10.1523/JNEUROSCI.3225-06.2006 Murray, M.M., Wylie, G.R., Higgins, B. a, Javitt, D.C., Schroeder, C.E., Foxe, J.J., 2002a. The spatiotemporal dynamics of illusory contour processing: combined high-density electrical mapping, source analysis, and functional magnetic resonance imaging. The Journal of –73. Murray, M.M., Wylie, G.R., Higgins, B.A., Javitt, D.C., Schroeder, C.E., Foxe, J.J., 2002b. The neuroscience :spatiotemporal the dynamics official ofjournal illusory of thecontour Society processing: for Neuroscience combined 22, high 5055-density electrical

56 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. mapping, source analysis, and functional magnetic resonance imaging. J. Neurosci 22, 5055–5073. Nieder, A., Wagner, H., 1999. Perception and neuronal coding of subjective contours in the owl. Nat. Neurosci. 2, 660–663. doi:10.1038/10217 Niemeier, M., Goltz, H.C., Kuchinad, A., Tweed, D.B., Vilis, T., 2005. A contralateral preference in the lateral occipital area: sensory and attentional mechanisms. Cereb. Cortex 15, 325–331. doi:10.1093/cercor/bhh134 Norman, J., 2002. Two visual systems and two theories of perception: An attempt to reconcile the constructivist and ecological approaches. Behav Brain Sci 25, 73-96-144. Norman, J.F., Bartholomew, A.N., 2011. Blindness enhances tactile acuity and haptic 3-D shape discrimination. Atten Percept Psychophys 73, 2323–2331. doi:10.3758/s13414-011- 0160-4 Ohtani, Y., Okamura, S., Shibasaki, T., Arakawa, A., Yoshida, Y., Toyama, K., Ejima, Y., 2002. Magnetic responses of human visual cortex to illusory contours. Neurosci. Lett 321, 173–176. Pascual-Marqui, R.D., Lehmann, D., 1993. Comparison of topographic maps and the reference electrode: comments on two papers by Desmedt and collaborators. Electroencephalogr Clin Neurophysiol 88, 530–531, 534–536. Payne, B.R., Berman, N., 1983. Functional organization of neurons in cat striate cortex: variations in preferred orientation and orientation selectivity with receptive-field type, ocular dominance, and location in visual-field map. J. Neurophysiol. 49, 1051–1072. Peelen, M.V., He, C., Han, Z., Caramazza, A., Bi, Y., 2014. Nonvisual and visual object shape representations in occipitotemporal cortex: evidence from congenitally blind and sighted adults. J. Neurosci. 34, 163–170. doi:10.1523/JNEUROSCI.1114-13.2014 Perrin, F., Bertrand, O., Pernier, J., 1987. Scalp current density mapping: value and estimation from potential data. IEEE Trans Biomed Eng 34, 283–288. Pillow, J., Rubin, N., 2002. Perceptual completion across the vertical meridian and the role of early visual cortex. Neuron 33, 805–813. Porac, C., Searleman, A., Karagiannakis, K., 2006. Pseudoneglect: Evidence for both perceptual and attentional factors. Brain and Cognition 61, 305–311. doi:10.1016/j.bandc.2006.01.003 Ramachandran, V.S., 1987. Visual Perception of Surfaces: A Biological Theory, in: Petry, S., Meyer, G.E. (Eds.), The Perception of Illusory Contours. Springer New York, pp. 93–108. doi:10.1007/978-1-4612-4760-9_10 Redies, C., Crook, J.M., Creutzfeldt, O.D., 1986. Neuronal responses to borders with and without luminance gradients in cat visual cortex and dorsal lateral geniculate nucleus. Exp Brain Res 61, 469–481. Reich, L., Szwed, M., Cohen, L., Amedi, A., 2011. A ventral visual stream reading center independent of visual experience. Curr. Biol. 21, 363–368. doi:10.1016/j.cub.2011.01.040 Remmer, M.H., Rastogi, N., Ranka, M.P., Ceisler, E.J., 2015. Achromatopsia: a review. Curr Opin Ophthalmol 26, 333–340. doi:10.1097/ICU.0000000000000189 Renier, L., De Volder, A.G., 2010. Vision substitution and depth perception: early blind subjects experience visual perspective through their ears. Disabil Rehabil Assist Technol 5, 175– 183. doi:10.3109/17483100903253936 Renier, L., Laloyaux, C., Collignon, O., Tranduy, D., Vanlierde, A., Bruyer, R., De Volder, A.G., 2005. The Ponzo illusion with auditory substitution of vision in sighted and early-blind subjects. Perception 34, 857–867. Ricciardi, E., Handjaras, G., Pietrini, P., 2014. The blind brain: how (lack of) vision shapes the morphological and functional architecture of the human brain. Exp. Biol. Med. (Maywood) 239, 1414–1420. doi:10.1177/1535370214538740 Ringach, D.L., Shapley, R., 1996. Spatial and temporal properties of illusory contours and amodal boundary completion. Vision Res. 36, 3037–3050. Romani, A., Callieco, R., Tavazzi, E., Cosi, V., 2003. The effects of collinearity and orientation on texture visual evoked potentials. Clin Neurophysiol 114, 1021–1026.

57 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

Sáry, G., Chadaide, Z., Tompa, T., Köteles, K., Kovács, G., Benedek, G., 2007. Illusory shape representation in the monkey inferior temporal cortex. Eur. J. Neurosci 25, 2558–2564. doi:10.1111/j.1460-9568.2007.05494.x Sáry, G., Köteles, K., Kaposvári, P., Lenti, L., Csifcsák, G., Frankó, E., Benedek, G., Tompa, T., 2008. The representation of Kanizsa illusory contours in the monkey inferior temporal cortex. Eur. J. Neurosci 28, 2137–2146. doi:10.1111/j.1460-9568.2008.06499.x Sasaki, Y., Hadjikhani, N., Fischl, B., Liu, A.K., Marrett, S., Dale, A.M., Tootell, R.B., Marret, S., 2001. Local and global attention are mapped retinotopically in human occipital cortex. Proc. Natl. Acad. Sci. U.S.A. 98, 2077–2082. doi:10.1073/pnas.98.4.2077 Sayres, R., Grill-Spector, K., 2008. Relating Retinotopic and Object-Selective Responses in Human Lateral Occipital Cortex. Journal of Neurophysiology 100, 249–267. doi:10.1152/jn.01383.2007 Schroeder, C.E., Mehta, A.D., Givre, S.J., 1998. A spatiotemporal profile of visual system activation revealed by current source density analysis in the awake macaque. Cereb. Cortex 8, 575– 592. Schroeder, C.E., Mehta, A.D., Givre, S.J., 1998. A spatiotemporal profile of visual system activation revealed by current source density analysis in the awake macaque. Cereb. Cortex 8, 575– 592. Seghier, M., Dojat, M., Delon-Martin, C., Rubin, C., Warnking, J., Segebarth, C., Bullier, J., 2000. Moving illusory contours activate primary visual cortex: an fMRI study. Cereb. Cortex 10, 663–670. Seghier, M.L., Vuilleumier, P., 2006. Functional neuroimaging findings on the human perception of illusory contours. Neurosci Biobehav Rev 30, 595–612. doi:10.1016/j.neubiorev.2005.11.002 Sheth, B.R., Sharma, J., Rao, S.C., Sur, M., 1996. Orientation maps of subjective contours in visual cortex. Science 274, 2110–2115. Shintani, K., Shechtman, D.L., Gurwood, A.S., 2009. Review and update: current treatment trends for patients with retinitis pigmentosa. Optometry 80, 384–401. doi:10.1016/j.optm.2008.01.026 Shum, J., Hermes, D., Foster, B.L., Dastjerdi, M., Rangarajan, V., Winawer, J., Miller, K.J., Parvizi, J., 2013. A brain area for visual numerals. J. Neurosci. 33, 6709–6715. doi:10.1523/JNEUROSCI.4558-12.2013 Song, Y., Baker, C.L., 2006. Neural mechanisms mediating responses to abutting gratings: luminance edges vs. illusory contours. Vis. Neurosci. 23, 181–199. doi:10.1017/S0952523806232036 Spehlmann, R., 1965. The averaged electrical responses to diffuse and to patterned light in the human. Electroencephalogr Clin Neurophysiol 19, 560–569. Spinelli, L., Andino, S.G., Lantz, G., Seeck, M., Michel, C.M., 2000. Electromagnetic inverse solutions in anatomically constrained spherical head models. Brain Topogr 13, 115–125. Stanley, D.A., Rubin, N., 2005. Rapid detection of salient regions: evidence from apparent motion. J Vis 5, 690–701. doi:10:1167/5.9.4 Stanley, D.A., Rubin, N., 2003. fMRI activation in response to illusory contours and salient regions in the human lateral occipital complex. Neuron 37, 323–331. Stilla, R., Sathian, K., 2008. Selective visuo-haptic processing of shape and texture. Hum Brain Mapp 29, 1123–1138. doi:10.1002/hbm.20456 Striem-Amit, E., Amedi, A., 2014. Visual cortex extrastriate body-selective area activation in congenitally blind people “seeing” by using sounds. Curr. Biol. 24, 687–692. doi:10.1016/j.cub.2014.02.010 Striem-Amit, E., Cohen, L., Dehaene, S., Amedi, A., 2012a. Reading with sounds: sensory substitution selectively activates the visual word form area in the blind. Neuron 76, 640– 652. doi:10.1016/j.neuron.2012.08.026 Striem-Amit, E., Guendelman, M., Amedi, A., 2012b. “Visual” acuity of the congenitally blind using visual-to-auditory sensory substitution. PLoS ONE 7, e33136. doi:10.1371/journal.pone.0033136

58 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. Szwed, M., Dehaene, S., Kleinschmidt, A., Eger, E., Valabrègue, R., Amadon, A., Cohen, L., 2011. Specialization for written words over objects in the visual cortex. Neuroimage 56, 330– 344. doi:10.1016/j.neuroimage.2011.01.073 Tanaka, K., 1996. Inferotemporal cortex and object vision. Annu. Rev. Neurosci. 19, 109–139. doi:10.1146/annurev.ne.19.030196.000545 Tootell, R.B., Hadjikhani, N.K., Vanduffel, W., Liu, A.K., Mendola, J.D., Sereno, M.I., Dale, A.M., 1998. Functional analysis of primary visual cortex (V1) in humans. Proc. Natl. Acad. Sci. U.S.A 95, 811–817. Tootell, R.B., Mendola, J.D., Hadjikhani, N.K., Ledden, P.J., Liu, A.K., Reppas, J.B., Sereno, M.I., Dale, A.M., 1997. Functional analysis of V3A and related areas in human visual cortex. J. Neurosci 17, 7060–7078. Tootell, R.B., Reppas, J.B., Dale, A.M., Look, R.B., Sereno, M.I., Malach, R., Brady, T.J., Rosen, B.R., 1995. Visual motion aftereffect in human cortical area MT revealed by functional magnetic resonance imaging. Nature 375, 139–141. doi:10.1038/375139a0 Tootell, R.B., Switkes, E., Silverman, M.S., Hamilton, S.L., 1988. Functional anatomy of macaque striate cortex. II. Retinotopic organization. J. Neurosci. 8, 1531–1568. Tootell, R.B.H., Nelissen, K., Vanduffel, W., Orban, G.A., 2004. Search for color “center(s)” in macaque visual cortex. Cereb. Cortex 14, 353–363. Turnbull, O.W., McGeorge, P., 1998. Lateral bumping: A normal-subject analog to the behaviour of patients with hemispatial neglect? ResearchGate 37, 10–78. Ungerleider, L.G., Mishkin, M., 1982. Two cortical visual systems, in: Visual Behavior. D.J. Ingle, M.A. Goodale & R.J.W. Mansfield, pp. 549–586. Van de Ville, D., Britz, J., Michel, C.M., 2010. EEG microstate sequences in healthy humans at rest reveal scale-free dynamics. Proc. Natl. Acad. Sci. U.S.A. 107, 18179–18184. doi:10.1073/pnas.1007841107 von der Heydt, R., Peterhans, E., 1989a. Mechanisms of contour perception in monkey visual cortex. I. Lines of pattern discontinuity. J. Neurosci. 9, 1731–1748. von der Heydt, R., Peterhans, E., 1989b. Mechanisms of contour perception in monkey visual cortex. I. Lines of pattern discontinuity. J. Neurosci 9, 1731–1748. von der Heydt, R., Peterhans, E., Baumgartner, G., 1984. Illusory contours and cortical neuron responses. Science 224, 1260–1262. Vuilleumier, P., Landis, T., 1998. Illusory contours and spatial neglect. Neuroreport 9, 2481–2484. Vuilleumier, P., Schwartz, S., Husain, M., Clarke, K., Driver, J., 2001a. Implicit processing and learning of visual stimuli in parietal extinction and neglect. Cortex 37, 741–744. Vuilleumier, P., Valenza, N., Landis, T., 2001b. Explicit and implicit perception of illusory contours in unilateral spatial neglect: behavioural and anatomical correlates of preattentive grouping mechanisms. Neuropsychologia 39, 597–610. Woldorff, M.G., Fox, P.T., Matzke, M., Lancaster, J.L., Veeraswamy, S., Zamarripa, F., Seabolt, M., Glass, T., Gao, J.H., Martin, C.C., Jerabek, P., 1997. Retinotopic organization of early visual spatial attention effects as revealed by PET and ERPs. Hum Brain Mapp 5, 280–286. doi:10.1002/(SICI)1097-0193(1997)5:4<280::AID-HBM13>3.0.CO;2-I Wynn, J.K., Green, M.F., Engel, S., Korb, A., Lee, J., Glahn, D., Nuechterlein, K.H., Cohen, M.S., 2008. Increased extent of object-selective cortex in schizophrenia. Psychiatry Res 164, 97–105. doi:10.1016/j.pscychresns.2008.01.005 Yoshino, A., Kawamoto, M., Yoshida, T., Kobayashi, N., Shigemura, J., Takahashi, Y., Nomura, S., 2006. Activation time course of responses to illusory contours and salient region: a high- density electrical mapping comparison. Brain Res 1071, 137–144. doi:10.1016/j.brainres.2005.11.089 Zeki, S., 2003. Improbable areas in the visual brain. Trends Neurosci. 26, 23–26. Zeki, S., Watson, J.D., Lueck, C.J., Friston, K.J., Kennard, C., Frackowiak, R.S., 1991. A direct demonstration of functional specialization in human visual cortex. J. Neurosci. 11, 641– 649. Zeki, S.M., 1978. Uniformity and diversity of structure and function in rhesus monkey prestriate visual cortex. J. Physiol. (Lond.) 277, 273–290.

59 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

CHAPTER 7 ARTICLES

7.1 Cue-dependent circuits for illusory contours in humans

Jacques Anken1, Jean-François Knebel2,3, Sonia Crottaz-Herbette1, Pawel J. Matusz1,4, Jérémie Lefebvre2,5, Micah M. Murray1-3,6,7,* The Laboratory for Investigative Neurophysiology (The LINE), 1Department of Clinical Neurosciences and 2Department of Radiology, University Hospital Center and University of Lausanne, Lausanne, Switzerland 3Electroencephalography Brain Mapping Core, Center for Biomedical Imaging (CIBM) of Lausanne and Geneva, Switzerland 4Attention, Brain and Cognition Group, Department of Experimental Psychology, Oxford, UK 5Toronto Western Research Institute, Toronto, Canada 6Department of Ophthalmology, University of Lausanne, Jules-Gonin Eye Hospital, Lausanne, Switzerland 7Department of Hearing and Speech Sciences, Vanderbilt University, Nashville, TN, USA

Address correspondence to: Prof. Micah Murray CHUV-UNIL Radiologie, BH08.078 Rue du Bugnon 46 1011 Lausanne [email protected]

60 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

7.1.1 Abstract

Objects’ borders are readily perceived despite absent contrast gradients, e.g. due to poor lighting or occlusion. In humans, a visual evoked potential (VEP) correlate of illusory contour (IC) sensitivity, the “IC effect”, has been identified with an onset at ~90ms and generators within bilateral lateral occipital cortices (LOC). The IC effect is observed across a wide range of stimulus parameters, though until now it always involved high-contrast achromatic stimuli. Whether IC perception and its brain mechanisms differ as a function of the type of stimulus cue remains unknown. Resolving such will provide insights on whether there is a unique or multiple solutions to how the brain binds together spatially fractionated information into a cohesive perception. Here, participants discriminated IC from no-contour (NC) control stimuli that were either comprised of low-contrast achromatic stimuli or instead isoluminant chromatic contrast stimuli (presumably biasing processing to the magnocellular and parvocellular pathways, respectively) on separate blocks of trials. Behavioural analyses revealed that ICs were readily perceived independently of the stimulus cue – i.e. when defined by either chromatic or luminance contrast. VEPs were analysed within an electrical neuroimaging framework and revealed a generally similar timing of IC effects across both stimulus contrasts (i.e. at ~90ms). Additionally, an overall phase shift of the VEP on the order of ~30ms was consistently observed in response to chromatic vs. luminance contrast independently of the presence/absence of ICs. Critically, topographic differences in the IC effect were observed over the ~110-160ms period; different configurations of intracranial sources contributed to IC sensitivity as a function of stimulus contrast. Distributed source estimations localized these differences to LOC as well as V1/V2. The present data expand current models by demonstrating the existence of multiple, cue-dependent circuits in the brain for generating perceptions of illusory contours.

Key words: illusory contour, Kanizsa, event-related potential (ERP), visual evoked potential (VEP), object recognition, magnocellular; parvocellular

61 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

7.1.2 Introduction

The visual system can create perceptions of boundaries despite the visual input to the retina being discontinuous or incomplete; resulting from poor lighting, occlusion, or myriad other everyday situations. These perceptions, including ICs, have been the subject of extensive theoretical debate and experimental research across species (Murray and Herrmann, 2013). A commonly-used stimulus was popularized by Kanizsa (1976) and includes an array of circular sectors (pacmen), whose mouths are oriented so to induce ICs or, alternatively, rotated as to prevent such perceptions (hereafter no-contour; NC) (Figure 1).

Several competing models have been proposed regarding how the brain produces ICs (Murray and Herrmann, 2013). Some propose that low-level areas V1/V2 mediate IC sensitivity in a feed-forward manner (Grosof et al., 1993; Nieder and Wagner, 1999; Redies et al., 1986; von der Heydt et al., 1984). Others propose that lateral occipital cortices (LOC) within the ventral visual pathway (Ungerleider and Mishkin, 1982) mediate it, and that any effects in V1/V2 reflect feedback modulations subsequent to IC sensitivity itself (Lee and Nguyen, 2001; Mendola et al., 1999; Murray et al., 2006, 2004, 2002; Sáry et al., 2008, 2007). Still others propose that the LOC detects salient regions defined by the pacmen inducers, but that IC sensitivity is itself performed within V1/V2 albeit under the control of feedback modulations (from the LOC and elsewhere) (Hochstein and Ahissar, 2002; Stanley and Rubin, 2003; Yoshino et al., 2006).

A VEP correlate of IC sensitivity has been identified – the IC effect – that onsets at ~90ms post-stimulus (i.e. during the P1/N1 components of the VEP) and is localised to the bilateral LOC (Murray et al., 2002). This IC effect has been observed across various manipulations of low-level stimulus features inducing ICs, including contrast polarity, eccentricity, types of inducers, and modal/amodal completion (Murray and Herrmann, 2013). However, a major shortcoming of all prior neurophysiologic studies of IC sensitivity in animals and humans is that the employed stimuli were invariably high in contrast (black and white, in fact). The extent to which the spatio-temporal brain dynamics of the IC sensitivity are cue- dependent and impacted by stimulus features, such as luminance and chromaticity, remains unknown. Resolving the role of these stimulus features in IC sensitivity would provide much-needed insights into potentially differing contributions of magnocellular versus parvocellular subdivisions of the visual system to IC sensitivity (Ejima and Takahashi, 1988; Gregory, 1977; Li and Guo, 1995; Soriano et al., 1996) as well as potentially reconcile the discrepant findings and the resultant models of IC sensitivity.

It has been suggested that mechanisms based on luminance and those based on chromaticity might both contribute to IC sensitivity, with the two mechanisms operating concurrently (Takahashi et al., 1992). At present, direct neurophysiologic support for this proposal is largely lacking. Indirect supporting evidence for has been provided by the results demonstrating that ICs can be induced with both static and moving inducers (e.g., Seghier et al., 2000) as well as with inducers that oscillate (Masuda et al., 2015). Such data would suggest that both dorsal and ventral visual pathways (which are thought to receive a preponderance of magnocellular and parvocellular inputs, respectively) likely contribute to IC

62 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. sensitivity processes. In line with this suggestion, parietal sources have been identified as contributing to the IC effect (cf. Figure 6a in Murray et al., 2002; reviewed in Murray and Herrmann, 2013). However, parietal structures do not appear to forcibly be requisite for IC perception. Studies of brain- lesioned patients have shown that IC perception critically depends on the integrity of the LOC, but persists despite damage to parietal cortices (cf. Figure 5 in Vuilleumier et al., 2001). Similarly, studies of patients with schizophrenia would indicate that the IC effect triggered by high-contrast achromatic stimuli is indistinguishable from that recorded from healthy controls, despite severely impaired P1 component responses in the former group (Foxe et al., 2005; Knebel et al., 2011). More generally, evidence is accumulating to support the idea of impaired magnocellular system function in schizophrenia (Butler et al., 2007; Javitt, 2009). One implication for IC sensitivity is that this process might operate largely independently of the magnocellular and/or dorsal pathway, relying instead on the integrity of the parvocellular system and the ventral stream structures. However, and because extant studies have used high-contrast achromatic stimuli, it is not clear if IC processes operate in a cue- invariant manner. Data from recordings within lower-level visual cortices (V1/V2) in animals would suggest that orientation and contour sensitivity may operate in a largely cue-invariant manner (e.g. (Song and Baker, 2007; Gharat and Baker, 2012)).

In light of such, we reasoned that cue-invariant IC effects would be consistent with IC processing being mediated in a (largely) feed-forward manner by regions such as V1/V2. By contrast, cue-dependent IC effects would instead support LOC-centred models of IC processing. The current study thus determined whether the IC effect is limited to the specific type of stimulus contrast used to elicit it; the evidence for such limitation would undermine the emerging consensus on the (uniform) brain underpinnings of IC sensitivity. By analysing VEPs within an electrical neuroimaging framework, we differentiated effects arising due to changes in the brain response timing, strength, and topography (Michel and Murray, 2012; Murray et al., 2008). If differences were found merely in the strength of responses of a statistically indistinguishable network across the two IC contrasts, this would suggest that a single, uniform brain network/mechanism mediates IC sensitivity. If, instead, early differences in the topography and underlying sources were found between the two types of contrast, such a result - depending on how strong / early were the differences observed - could suggest a 1) certain flexibility within the already identified network or, alternatively, 2) separate and distinct brain circuits activated by different types of stimulus contrast. The latter would necessitate revision of the emerging consensus about the brain mechanisms giving rise to IC sensitivity.

7.1.3 Material and Methods

7.1.3.a Participants

Analyses presented in this study are based on data from 12 participants (4 male, all right-handed; aged 23-33 years, mean 25.8 years). All were post-graduate university students at the time of testing. No subject had history of or current neurological or psychiatric illness. All participants had normal or corrected-to normal vision and no problems with color vision or color-blindness were reported. The integrity of color vision was based on participants’ self-reports according to their prior experiences with

63 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. the Ishihara color test (Ishihara, 1972), as routinely performed in Swiss primary schools as well as in military recruiting centres. Data from an additional 8 subjects were excluded due to either excessive muscle and/or alpha frequency EEG artefacts (N=7) or technical issues with behavioural response recording during data acquisition (N=1).

7.1.3.b Stimuli and task

Stimuli were comprised of a set of 4 circular Kanizsa-type (Kanizsa, 1976) ‘pacmen’ inducers that were arranged to either form an illusory contour or not (IC and NC conditions, respectively) (Figure 1). Each inducer subtended 1.26° in diameter of visual angle at a distance of 150cm. On a given trial, the four pacmen were either positioned along the horizontal and vertical axes to form a circular IC (and its NC equivalent; 1.4° centre-to-centre eccentricity) or, alternatively, were positioned along the 45° diagonals to form a square IC (and its NC equivalent; 2.0° centre-to-centre eccentricity). These variations in how ICs were created were included to prevent participants from selectively attending to particular regions of space as a strategy to successfully complete the task. The employed forms have been used in prior IC studies by our group and are known to result in similar IC sensitivity (Knebel and Murray, 2012; Knebel et al., 2011; Murray and Herrmann, 2013).

The IC stimulus features that were the main focus of our study were the chromatic versus low-luminance contrast of the pacmen vs. the background display (hereafter referred to as C and L, respectively). C- and L-ICs were separately adjusted, as to generate stimuli biasing the processing predominantly towards the parvocellular and magnocellular visual pathways, respectively. For the measure of luminance, the following definition was used: Y= 0.2126R + 0.71524G + 0.0722B. (i.e. the luminance Y is composed in 21.26% of Red, in 71.524% of Green and in 7.22% of Blue) (see Knebel et al., 2008). For the chromatic contrast condition, there were either green ‘pacmen’ against a pink background or the vice versa. The RGB values were for green: R:74, G:146, B:109 (Y=128) and for pink: R:236, G:92, B:166 (Y=128). The Weber contrast (with the measurement on the screen) for the pink stimuli on a green background was 28%, while for green stimuli on a pink background this value was -22%. The Weber contrast for the achromatic condition was -0.11%. The measurements were performed with a digital light meter TES-1332A. The luminance of the ‘pacmen’ and the background on the screen was equivalent and controlled with photographic techniques using the diaphragm aperture (Minolta Auto Meter IV F). The aperture was identical for the green, pink and the grey background values. The photographic aperture ƒ/N was on average ƒ/5.62 on the screen for each of the 3 colors. This value equals the ratio of the lens’s focal length to the diameter of the entrance pupil. It is a dimensionless value indicative of lens speed. If the aperture is the same for two colors, the luminance of the colors is equivalent. This was the case here for the green and pink colors. For the low-luminance contrast condition, the RGB values of the inducers were R:125, G:125,B:125 (Y=125) and those of the background were R:128,G:128,B:128 (Y=128) (i.e. the ‘pacmen’ were slightly darker than the background). Stimuli were displayed on an LCD computer monitor (20″ active TFT, 1600 x 1200 at 60Hz, 16ms pixel response time). We opted here for an optical method for achieving isoluminance, rather than a method based, for example, on minimum-motion or flicker, to minimize subjective influences on stimulus intensity (e.g. Chaudhuri and Albright, 1990).

64 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

Fig 1. Illustration of stimulus conditions. a. Isoluminant chromatic contrast stimuli either appeared pink on green or vice versa. b. Low-luminance achromatic contrast stimuli used pacmen inducers that were slightly darker than the background. Note that the contrast in the figure was modified for ease of visibility. The illusory contour (IC) condition involved inducers whose mouths all faced inward (top row), while the no contour (NC) condition involved inducers whose mouths all faced outward (bottom row). The central fixation point has been enlarged for illustration purposes.

Chromatic and low-luminance contrast stimuli were presented in separate blocks of trials. Each block contained 200 stimuli with equal probability of IC and NC conditions. Each participant completed four chromatic contrast blocks (two for pink on green and two for green on pink) and four luminance contrast blocks. Block order was counterbalanced across participants. Stimuli were presented for 500ms with an inter-stimulus interval ranging between 800 and 1200ms. A central fixation dot of the same chromaticity or luminance as the ‘pacmen’ was displayed on the computer screen throughout the entire block of trials. The participant’s task was a two-alternative forced choice that required the discrimination between IC and NC presence on each trial via a right-handed button press. Stimulus delivery and behavioural response collection were controlled by E-prime 2 software (Psychology Software Tools Inc., Pittsburgh, Pennsylvania, USA; www.pstnet.com/eprime). During the experiment, participants took regular breaks between blocks of trials to maintain high concentration and prevent fatigue.

7.1.3.c EEG acquisition and pre-processing

Continuous EEG was acquired at 1024Hz through a 128-channel Biosemi ActiveTwo AD-box (http://www.biosemi.com) referenced to the common mode sense (CMS; active electrode) and grounded to the driven right leg (DRL; passive electrode), which functions as a feedback loop driving the average potential across the electrode montage to the amplifier zero (full details, including a diagram of this circuitry, can be found at http://www.biosemi.com/faq/cms&drl.htm). Prior to epoching, the continuous EEG was filtered (second-order Butterworth with -12db/octave roll-off; 0.1Hz high-pass; 60Hz low- pass; 50Hz notch). The filters were computed linearly in both forward and backward directions to eliminate phase shifts.

EEG epochs were time-locked to the presentation of visual stimuli and spanned 100ms pre-stimulus and 500ms post-stimulus. Epochs with amplitude deviations in excess of ±60μV at any channel, with the

65 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. exception of those labelled as ‘bad’ due to poor electrode–skin contact or damage, were considered artefacts and were excluded. Likewise, trials with blinks or other transients were excluded off-line based on vertical and horizontal electro-oculograms. Data from ‘bad’ channels were interpolated using 3D splines (Perrin et al., 1987). VEP data were also re-calculated against an average reference and baseline- corrected using the pre-stimulus interval. For each participant, four VEPs were calculated: ICC, NCC, ICL, and NCL. The mean (±s.e.m.) number of accepted EEG epochs for each of these conditions was 365±8, 365±7, 368±7, and 374±5, respectively. These values did not significantly differ (F(3,9)=2.1; p>0.15). Differences were then calculated between the IC and NC conditions for each type of contrast, separately, in order to isolate brain activity associated with IC sensitivity (Murray et al., 2002). Hereafter, we refer to these differences for the chromatic contrast and luminance contrast conditions as the CICeffect and LICeffect, respectively.

7.1.3.d VEP analyses

VEP analyses were performed using the Cartool freeware (http://sites.google.com/site/fbmlab/cartool/cartooldownload; Brunet et al., 2011) as well as STEN utilities (http://unil.ch/line/home/menuinst/about-the-line/software--analysis-tools.html). Effects were identified using an analysis framework referred to as electrical neuroimaging (Koenig et al., 2014; Michel and Murray, 2012; Michel et al., 2004; Murray et al., 2008; Tzovara et al., 2012). These electrical neuroimaging analyses allowed us to differentiate between effects following from modulations in the strength of responses of statistically indistinguishable brain generators, alterations in the configuration of the active generators (inferred from the topography of the electric field at the scalp), as well as the latency shifts in brain processes across experimental conditions. We conducted two sets of VEP analyses. The first set of analyses focused on identifying the timing of the IC effect for each type of stimulus contrast. The second set of analyses focused on identifying mechanistic differences between the IC effects for each type of stimulus contrast after first removing any confounding effects of differences in the general VEP response latency (see Results).

To ascertain the timing of the IC effect, we first analyzed the VEP voltage waveform data from all electrodes as a function of time post-stimulus onset in a series of paired comparisons (t-tests) between IC and NC responses to the chromatic contrast and luminance contrast conditions, separately. Temporal auto-correlation at individual electrodes was corrected through the application of a 20 contiguous data- point temporal criterion (~20ms at 1024Hz sampling) for the persistence of differential effects (Guthrie and Buchwald, 1991). Similarly, spatial correlation was addressed by considering as reliable only those effects that entailed >10% of the electrodes from the 128-channel montage (i.e. ≥12 electrodes). These combined criteria were applied to (partially) correct for multiple comparisons. Our use of an average reference receives support from biophysical laws as well as from the implicit re-centring of VEP data to it when performing source estimations (discussed in Brunet et al., 2011). Analyses of the VEP voltage waveform data (vs. the average reference) are presented here to provide a clearer link between the canonical VEP analysis approaches and the electrical neuroimaging framework (Figure 2). While the former give a visual impression of specific effects within the dataset, our conclusions are principally based on reference-independent global measures of the electric field at the scalp. The measure we used for assessing the timing of the IC effect was the Global Field Power (GFP), which quantifies the electric

66 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. field strength (Lehmann and Skrandies, 1980). This measure is equivalent to the standard deviation of the voltage potential values across the entire electrode montage at a given time point and represents a reference-independent measure of the VEP strength (Koenig and Melie-García, 2010; Koenig et al.,

2014; Murray et al., 2008). The GFP was used to determine when the CICeffect and LICeffect responses were themselves each significantly different from baseline (i.e. when the IC effect itself was reliable). This was achieved using the so-called Topographic Consistency Test (TCT) (Koenig and Melie-García, 2010), which tests the observed GFP against a distribution based on permutations of electrode values in space across the montage (i.e. a pure noise response that preserves the statistical properties of the original dataset). In the case of the IC effect, the TCT provides a statistical determination of when a reliable response begins for each type of stimulus contrast.

The second set of analyses focused on ascertaining whether there are mechanistic differences in the IC effect across stimulus contrasts. As expected from the findings of Foxe et al. (2008; cf. their Figure 3), and confirmed by visual inspection of the VEPs, VEPs were generally shifted ~30ms later for luminance vs. chromatic contrast conditions, independently of the presence/absence of ICs (Figure 2). Thus, we re- aligned each participant’s VEPs from each condition to the peak of the P1 component (Supplementary Figure 1). The P1 component was defined as the first peak in the GFP that was characterized by a posterior bilateral positivity and fronto-central negativity in the VEP topography (see also Murray et al.(2001)). The re-alignment procedure was semi-automated using a customized Python script (Python Software Foundation; Python Language Reference, version 2.6. Available at http://www.python.org) in the following way: First, the maximal GFP value over the 0-100ms period was identified. All values smaller than this maximum were set to zero. Next, the maximal GFP value and its latency over the 105- 182ms period were identified. To ensure that this GFP maximum was indeed indexing a P1 response, the polarity of nine posterior and nine frontal electrodes at the latency of this GFP peak amplitude was also queried. If at least 7 posterior electrodes measured positive potentials and at least 7 frontal electrodes measured negative potentials, then the GFP maximum was indeed considered as a P1, and the VEP was temporally re-aligned (i.e. the peak was labelled as 100ms post-stimulus onset) as a consequence. We likewise assessed whether or not differences in peak P1 latency were correlated with reaction times. There was no evidence for reliable correlations (all p’s>0.35), despite reaction times in the chromatic condition being ~30ms faster than those in the luminance condition (488±13ms vs. 522±15ms; t(11)=5.44;p<0.01).

The re-aligned VEP data from the ICC and NCC conditions, as well as the ICL and NCL conditions were then subtracted separately to obtain the CICeffect and the LICeffect (Figure 3a). The remainder of the analyses compared these differential responses. First, modulations in the VEP strength (quantified by GFP) were tested using a millisecond-by-millisecond paired t-test in conjunction with the abovementioned 20 contiguous data-point temporal criterion (~20ms at 1024Hz sampling) for significant effects to correct for multiple contrasts. While GFP provides an assay of the VEP strength, it is inherently insensitive to spatial (i.e. topographic) differences in the VEPs.

In order to test the VEP topography independently of its strength, we used Global Dissimilarity (DISS)(Lehmann and Skrandies, 1980). DISS is equivalent to the square root of the mean of the squared difference between the potentials measured at each electrode for different conditions, normalized by the

67 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. instantaneous GFP. It is also directly related to the (spatial) correlation between two normalized vectors (cf. Appendix in Murray et al., 2008). We also performed a non-parametric randomization test (colloquially termed “TANOVA”; Murray et al. (2008)). The DISS value at each time point was compared to an empirical distribution derived from permuting the condition label of the data from each subject. Because changes in topography forcibly follow from changes in the configuration of the underlying active sources (Lehmann et al., 1987), this analysis reveals whether, and if so, when illusory contour sensitivity defined by chromatic vs. luminance contrast activated distinct brain networks. Next, the collective post-stimulus group-average VEPs were subjected to a topographic cluster analysis based on a hierarchical clustering algorithm (Murray et al., 2008). This clustering identifies stable electric field topographies (hereafter template maps). The clustering is sensitive exclusively to topographic modulations, because the data are first normalized by their instantaneous GFP. The optimal number of temporally-stable VEP clusters (i.e. the minimal number of maps that accounts for the greatest variance in the data) was determined using a modified Krzanowski-Lai criterion (Murray et al., 2008). The clustering makes no assumption regarding the orthogonality of the derived template maps (De Lucia et al., 2010; Koenig et al., 2014; Pourtois et al., 2008). Template maps identified in the group-average VEP were then submitted to a fitting procedure wherein each time point of each single-subject VEP was labelled according to the template map with which it best correlated spatially (Murray et al., 2008), so as to statistically test the relative presence of each template map in the moment-by-moment scalp topography of the VEPs and the differences in such across conditions. These values can be expressed as the probability of a given template map yielding a higher spatial correlation in the single-subject data from each condition. A statistical analysis of these values was performed using ANOVA, with experimental condition (CICeffect and LICeffect) and map as within-subject factors.

Finally, we estimated the intracranial sources of the VEPs as a function of time using a distributed linear inverse solution (ELECTRA) and applying the local autoregressive average (LAURA) regularization approach to address the non-uniqueness of the inverse problem (Grave de Peralta Menendez et al., 2004, 2001; Michel et al., 2004). The inverse solution algorithm is based on biophysical principles derived from the quasi-static Maxwell's equations, most notably the fact that independently of the volume conductor model used to describe the head, only irrotational and not solenoidal currents contribute to the EEG (Grave de Peralta Menendez et al., 2004, 2001). As part of the regularization strategy, homogenous regression coefficients in all directions and within the whole solution space were used. LAURA uses a realistic head model, and the solution space included 3005 nodes, selected from a grid equally distributed within the gray matter of the Montreal Neurological Institute's average brain (grey matter segmentation courtesy of Grave de Peralta Menendez and Gonzalez Andino; http://www.electrical-neuroimaging.ch/). The head model and lead field matrix were generated with the Spherical Model with Anatomical Constraints (SMAC; Spinelli et al. (2000) as implemented in Cartool. As an output, LAURA provides current density measures; their scalar values were evaluated at each node. Prior basic and clinical research has documented and discussed in detail the spatial accuracy of this inverse solution (Gonzalez Andino et al., 2005; Grave de Peralta Menendez et al., 2004; Martuzzi et al., 2009). The source estimations were calculated after first averaging across time for each subject and condition. The relevant time interval was determined based on the above topographic clustering analysis used to identify periods of stable VEP topography. These data matrices were then contrasted using a paired t-test. To partially correct for multiple testing, we applied a significance threshold of

68 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. p<0.05 and a spatial-extent criterion (kE) of >25 contiguous solution points (Bourquin et al., 2013; De Lucia et al., 2010; Knebel and Murray, 2012; Matusz et al., 2015; Toepel et al., 2015).

7.1.4 Results

7.1.4.a Behaviour

Accuracy rates for all conditions were between 92% and 100%, with a mean of 97%, providing no evidence that IC perception varied with the type of stimulus contrast. These values also provide a clear indication that inducers from all conditions were visible and readily perceived. Behavioural data were analyzed using inverse efficiency scores (Townsend and Ashby 1978, 1983), which are mean reaction times divided by percent correct responses and therefore account for any potential differences due to a speed-accuracy trade-off across conditions. Inverse efficiency was analyzed using a 2x2 repeated measures analysis of variance (ANOVA) with factors illusory contour presence (IC/NC) and contrast type (C/L). There was a main effect of contour presence, with generally greater efficiency for IC than NC stimuli (508±13 vs. 535±13ms; F(1,11)= 36.49; p<0.01; ηp2=0.77). Paired t-tests confirmed this was the case for both chromatic (t(11)=4.05; p<0.0002) and luminance contrast (t(11)=8.20; p<0.0001) conditions. There was also a main effect of contrast type, with generally greater efficiency for chromatic than luminance contrast (503±11 vs. 540±15ms; F(1,11)= 24.83; p<0.01; ηp2=0.69), which was confirmed by paired t-tests (ICC vs. ICL: t(11)=4.38; p<0.0012 and NCC vs. NCL: t(11)=5.32; p<0.0003). Finally, there was a significant interaction between the factors of illusory contour presence and contrast type (F(1,11)= 7.12; p<0.01; ηp2=0.39). This interaction was due to a larger IC vs. NC difference for stimuli defined by luminance than chromatic contrast (31±4ms vs. 22±6ms; t(11)=3.79; p<0.005). Future research will likely need to parametrically vary either the perceived strength of or accuracy in discriminating ICs defined by chromatic versus luminance contrast and their consequences on the corresponding brain mechanisms.

7.1.4.b Timing of the IC effect as a function of type of stimulus contrast

We first provide the results of the analysis carried out on the non-subtracted data (as well as prior to the temporal re-alignment), separately for responses to luminance contrast and chromatic contrast conditions, as to provide an indication of the timing of the IC effect for each type of contrast (see Materials and Methods for details). Figure 2a displays VEPs from an exemplar parieto-occipital scalp site (PO7) from luminance contrast and chromatic contrast conditions in response to IC and NC stimuli. Both luminance and chromatic contrast stimuli elicited robust VEPs of similar magnitude, though generally the VEPs were delayed in response to luminance vs. chromatic contrast stimuli. Figure 2b displays the IC-NC difference waveform (i.e. the IC effect) at the same scalp site. Both types of contrast elicited robust IC effects with what appears to be similar morphologies, and again with what appears to be a general delay for luminance vs. chromatic contrast. Figure 2c displays the corresponding p-value of the IC vs. NC paired contrast as a function of time. Albeit focused on a single electrode, this analysis indicates that the IC effect for stimuli defined by luminance contrast onsets at 83ms, while for stimuli defined by chromatic contrast it does so at 95ms. Applying the TCT analysis to the IC effect for each type of stimulus contrast revealed that the responses onset at 85ms for stimuli defined by luminance

69 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. contrast and at 93ms for stimuli defined by chromatic contrast (Figure 2d). In general, the timing of the IC effects for both types of stimulus contrast are consistent with prior findings using high-contrast luminance stimuli (Knebel and Murray, 2012; Murray and Herrmann, 2013; Murray et al., 2002).

Fig 2. Comparative timing of IC effects. a. Group-averaged (N=12) VEP waveforms from a left parieto-occipital scalp location (PO7) for all 4 stimulus conditions of the study are displayed. Comparable and robust responses were obtained for all conditions, although shifted later in time for the low-luminance achromatic conditions. b. The group-averaged VEP difference waveforms (IC effect) are shown for the isoluminant chromatic contrast and low-luminance achromatic contrast conditions (red and black traces, respectively, with shading indicating the s.e.m.). c. Paired t-tests were performed between IC and NC responses as a function of time for isoluminant chromatic and low-luminance achromatic contrast, separately (red and black area plots, respectively). Significant IC effects were observed starting at ~90-95ms post-stimulus onset (p<0.05 for a minimum of 20 consecutive data points). d. The topographic consistency test was performed using the difference waveforms across the entire electrode montage for isoluminant chromatic and low-luminance achromatic contrast, separately (red and black area plots, respectively). This was done to empirically

70 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. test for onset of the IC effect using a global and reference-independent VEP measure. Significant IC effects were observed starting at ~90-

95ms post-stimulus onset (same criterion as above).

7.1.4.c Mechanistic comparison of IC effects across types of stimulus contrast

Our primary interest was the mechanistic comparison of IC processes for stimuli defined by chromatic versus luminance contrast. We therefore calculated the IC effect for each contrast type, i.e. CICeffect and

LICeffect, and performed all subsequent analyses by statistically comparing these VEP differences. To prevent any general temporal differences in the VEP responses to luminance and chromatic contrast from confounding the identification of mechanistic differences between the LICeffect and CICeffect, we first temporally aligned the VEPs to the peak of the P100 component. Figure 3a displays the LICeffect and

CICeffect at electrode PO7 after alignment to the P100 component. While the morphology of the IC effect for both types of stimulus contrast (as detailed above) were generally similar, differences appeared to onset already after the initial ~100ms post-stimulus. We examined this statistically using GFP and DISS (Figure 3b and 3c, respectively). There was no evidence for reliable differences in response strength

(GFP) between the LICeffect and CICeffect. There were no periods of time when the t-test result yielded a p- value ≤0.05 for at least 11 continuous data points. By contrast, these IC effects significantly differed in their topography (DISS) over the 109-130ms post-stimulus period. In order to better understand the nature of these topographic differences, the group-averaged LICeffect and CICeffect data were submitted to a topographic cluster analysis. Two template maps were identified to be present over the 108-159ms post-stimulus period, consistent with the above analyses. These template maps were then fitted to the single-subject LICeffect and CICeffect VEPs, so as to determine the amount of time over the 108-159ms period when each template map better correlated spatially with the single-subject data. These values were then submitted to a 2×2 ANOVA (condition × template map), which revealed a significant interaction (F(1,11)=9.97; p=0.009; ηp2=0.48). One map predominated LICeffect responses, whereas another map predominated CICeffect responses (Figure 3d). These results indicate collectively that distinct brain networks contribute to IC sensitivity during the 108-159ms post-stimulus period when stimuli are defined by luminance vs. chromatic contrast.

71 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

Fig 3. Comparative mechanisms of IC effects.

a. The group-averaged (N=12) VEP difference waveforms (IC effect) are shown for the isoluminant chromatic contrast and low-luminance achromatic contrast conditions after first re-aligning the un-subtracted VEPs (see Figure 2a) to the peak of the P100 component (red and black traces, respectively, with shading indicating the s.e.m.). b. The group-averaged global field power (GFP) waveforms of the CICeffect and

LICeffect are shown (red and black traces, respectively, with shading indicating the s.e.m.). No statistically reliable differences were observed. c. The global dissimilarity (blue trace) is shown as a function of time. Significant topographic differences between CICeffect and LICeffect were observed over the 109-130ms period (pale green area). d. The topographic cluster analysis identified two maps over the 108-159ms period.

The bar graph shows that one map better account for the CICeffect and another for the LICeffect, which was quantified as the amount of time over the 108-159ms period when each map best correlated spatially with the single-subject data (mean±s.e.m. shown).

7.1.4.d Source Estimations

Finally, distributed source estimations were performed on the single-subject data after first averaging the CICeffect and LICeffect VEPs over the 108-159ms period (i.e. the period identified in the above topographic clustering analysis). Figure 4a displays the mean source estimations for the CICeffect and

LICeffect (top and bottom, respectively). For the CICeffect, there were prominent sources localised within the bilateral lateral occipital cortex and the calcarine sulcus. For the LICeffect, prominent sources were localised within the bilateral calcarine sulcus and the left lateral occipital cortex. For each node within the solution space, a paired t-test was performed. We considered as reliable those clusters wherein each node yielded a significant test (p<0.05) and was moreover located within a cluster of at least 25 significant nodes (kE>25nodes). The results showed significant differences in the generators underlying the CICeffect and LICeffect , located principally within right lateral occipital cortices and extending posteriorly to the occipital pole and the calcarine sulcus bilaterally as well as superiorly to inferior parietal cortices bilaterally (Figure 4b). In all the significant voxels, sources were stronger for the CICeffect

72 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. than the LICeffect. The maximal difference (t(11)=-4.87; p<0.0005) was located within the right lateral occipital cortex (43, -79, 2mm according to the Talairach and Tournoux 1988 coordinate system). This pattern of differences in source estimations is consistent with the topographic modulations described above insofar as a subset of commonly active regions was differentially active in the CICeffect than in the

LICeffect

Fig 4. Comparative distributed source estimations for IC effects.

a. Group-averaged distributed source estimations over the 108-159ms period are shown for the CICeffect and LICeffect (top and bottom, respectively). b. Statistical contrast of these source estimations are displayed and revealed reliable differences within the right lateral occipital cortices as well as calcarine sulcus bilaterally.

7.1.5 Discussion

This study revealed that brain mechanisms underlying IC sensitivity vary according with whether the eliciting stimuli are defined by chromatic versus luminance contrast. Robust perceptions of ICs were achieved with both types of stimulus contrast, but with greater perceptual efficiency when ICs were

73 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. defined by chromatic than luminance contrast. The VEP data revealed highly similar IC effects in terms of their timing across the two types of stimulus contrast, despite a general latency advantage for responses to the isoluminant chromatic contrast than for the low-luminance achromatic contrast. Critically, the topography, but not magnitude, of the IC effect differed as a function of the type of stimulus contrast. Partially distinct circuits were found to contribute to IC sensitivity across contrast types starting as early as 110ms post-stimulus. Distributed source estimations further indicated there to be a greater involvement of the LOC as well as V1/V2 to the CICeffect than to the LICeffect. Collectively, these data provide the first evidence for multiple brain circuits contributing to IC perception.

7.1.5.a Brain mechanisms of IC sensitivity across types of contrast

The timing of both the CICeffect as well as the LICeffect is highly consistent with prior findings that used high-contrast achromatic stimuli. For example, Murray et al. (2002) identified the onset of the IC effect at ~90ms and localized it to the LOC. This ‘brain signature’ of IC sensitivity is robust across a range of stimulus/perceptual variations: contrast polarity (Murray et al., 2002; see also Dresp et al., 1996 for psychophysical findings), modal vs. amodal completion (i.e. whether or not there was a perceived brightness enhancement; Murray et al., 2004), accuracy in discriminating convex versus concave shapes created by the ICs (Murray et al., 2006), or the stimulus type eliciting the ICs (Kanizsa-type stimuli or misaligned gratings; Knebel and Murray, 2012;Mendola et al., 1999). The present CICeffect and LICeffect have yet again corroborated this brain response pattern. Across both conditions, the IC effect onset again at ~90ms and originated within the LOC (albeit significantly more strongly for the CICeffect than for the LICeffect). These results contradict some prior behavioural studies that had suggested that ICs are not perceived (or are at best poorly perceived) under isoluminant chromatic contrast conditions (Li and Guo, 1995; Soriano et al., 1996). However, the task used by Li and Guo (1995) required observers to judge line lengths within a Kanizsa form (i.e. a variant of the Ponzo illusion), rather than the presence/absence of an illusory contour per se. Their paradigm thus did not provide a direct metric of IC sensitivity. By contrast, Ejima and Takahashi (1988) showed that ICs can be reliably perceived when defined by chromatic contrast. Our current results provide direct evidence for the ability of ICs based on both luminance and chromaticity differences to elicit IC perception.

The timing alone of these IC effects, similar to those reported before (Murray and Herrmann, 2013; Murray et al., 2002), is inconsistent with a purely feedforward nature of the underlying mechanism. As the IC effect follows the VEP onset by several tens of milliseconds, there is ample time for both response propagation across brain regions as well as integration of activity across brain areas containing distinct retinotopic representations (Murray et al., 2001). This temporal lag is particularly important given that the inducers of ICs were spatially distributed across quadrants of the visual field and thus necessitated interhemispheric and intrahemispheric integration and/or convergence of responses in brain regions consisting of neurons with large, bilateral receptive fields. Higher-order visual cortex substrates for IC sensitivity are thus also supported by the progressive increase in neural receptive field size and interhemispheric connectivity across visual cortices from V1 through to the LOC (Smith et al., 2001; see also Box 3 in Murray and Herrmann, 2013). While prior work would indicate that placing all of the inducer elements within one visual hemifield delays the IC effect by approximately 120ms (Brandeis and Lehmann, 1989; Murray et al., 2002), it may be possible to further dissociate circuits for IC

74 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. sensitivity by capitalising on knowledge concerning photoreceptor distribution across the peripheral retina where it may be more readily possible to target rod-driven IC effects (Ahnelt, 1998).

More generally, the current findings enrich the extant literature in several important ways. First, we demonstrated that the previously reported LOC activity is accompanied by prominent activity within the calcarine sulcus, across both types of stimulus contrast. Second, we provide the first evidence for topographic differences as a function of the type of contrast of the IC-eliciting stimuli. The initial time period of the previously identified IC effect differed in its topography across the CICeffect and LICeffect. Importantly, we were able to establish that this topographic difference followed from enhanced activity within the right LOC as well as within the bilateral calcarine sulcus (most likely V1/V2; Figure 4) in the case of the CICeffect. In these ways, the present study provided evidence that different brain circuits are activated during the initial stages of IC sensitivity according to the type of stimulus contrast used (see also Bushnell et al., 2011, for similar evidence in the macaque V4).

7.1.5.b The role of V1/V2 in IC sensitivity: New insights

There are several possible accounts for the currently observed activity within V1/V2 that starkly differ between each other in how important a role they attribute to this activity in mediating IC sensitivity. On the one hand, it may be contended that the varieties of contrast used here resulted in greater demands on, and thus stronger activation of, top-down attentional control processes for the purpose of distinguishing between ICs and NCs (e.g. (Ding et al., 2014). Studies in humans that reported V1/V2 activity have similarly used highly demanding tasks (Larsson et al., 1999; Maertens and Pollmann, 2005), making it difficult to establish whether the activity within V1/V2 contributes directly to IC or is instead a result of task demands. In line with the latter, sensitivity within V1 to ICs was found when the IC forms were perceived as moving vertically across the visual field, but substantially less so when the ICs were static (Seghier et al., 2000). Likewise, effects within V1/V2 were observed in studies where the task necessitated discriminating fine features of the IC shape (i.e., task demands were high; e.g., curvature, as in Maertens and Shapley(2008) or crispness, as in Stanley and Rubin (2003). However, the present data speak against the proposition of the role of V1/V2 activation being linked to task demands alone. First, the near-ceiling performance (>92% for all conditions) demonstrates that the task was not particularly demanding for the participants. In line with this, several studies in humans have suggested that IC perception occurs pre-attentively (Davis and Driver, 1994; Mattingley et al., 1997; Vuilleumier et al., 2001) and matures by 8 months of age (Csibra et al., 2000; though see Altschuler et al. (2012). Second, we demonstrated stronger V1/V2 responses in what was an ostensibly easier stimulus condition, i.e. when ICs were induced by the isoluminant chromatic contrast. If predominantly task difficulty was mediating these effects, then stronger responses should have been observed for the low-luminance achromatic contrast condition, whose processing was less efficient.

We propose that under certain conditions the LOC and V1/V2 give rise to IC sensitivity by operating in concert. Notably, this cooperation does not seem to occur with high-contrast achromatic stimuli. Because LOC and V1/V2 sources were both identified over the same time period, there is no evidence to suggest that one region is mediating responsiveness of the other. This notwithstanding, lesion data in humans and monkeys would indeed suggest that perception of (high-contrast achromatic) ICs relies

75 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. predominantly on the integrity of the LOC (Sáry et al., 2007; Vuilleumier et al., 2001). Further research in brain-lesioned patients is necessary to determine the exact role of V1/V2 in IC sensitivity when defined by isoluminant chromatic contrast or low-luminance achromatic contrast.

7.1.5.c Timing of magnocellular- vs. parvocellular- biased VEPs

Based on the present evidence, one could contend that the isoluminant chromatic and low- luminance achromatic contrast stimuli used here successfully biased brain responses towards the parvocellular and magnocellular visual pathways, respectively. It is therefore noteworthy that there was a ~30ms latency advantage for responses to chromatic than luminance contrast (Figure 2). This result could be considered contrary to the evidence from studies using single-unit recordings that suggest a latency advantage for magnocellular- vs. parvocellular-biased stimuli both in the lateral geniculate nucleus as well as the primary visual cortex (Maunsell and Gibson, 1992; Nowak et al., 1995; Schmolesky et al., 1998; Schroeder et al., 1998). However, single-unit activity does not forcibly reflect the net output response of the entire neuronal population. Likewise, computational modelling suggests that the greater convergence of parvocellular than magnocellular fibres onto primary visual cortex could actually result in an overall faster response to parvocellular inputs (Maunsell et al., 1999). In accord, human VEP and fMRI studies have reported precisely such a latency advantage for chromatic vs. luminance stimuli (Foxe et al., 2008; Liu et al., 2006). For example, a clear latency difference is apparent in Figure 3 of Foxe et al. (2008), although their focus was instead on the putative contributions of the two pathways to the C1 component of the VEPs and the potential implications thereof for models of attention. While not quantified, the latency shift would appear to be on the order of approximately 30ms, similarly to our findings. Our study thus provides the first direct evidence for temporally facilitated responses to chromatic and, thus, putatively parvocellular-biased, stimuli. While the neurophysiologic bases for this latency difference as well as for the discrepancies between single-unit and population- based levels of quantification remain to be fully resolved in future research, the present results lay grounds for further investigations of the visual system dynamics and of the ‘division of labour’ between the two pathways and, thus, for a better understanding of their contribution to visual processing deficits in clinical populations (e.g., Javitt, 2009). That said, it is important to also note that peak latency and amplitude of VEP components, such as P1 and N1, are strongly influenced by low-level stimulus features, including contrast (e.g. Butler et al., 2007), the number and distribution of pixels within an image (Doniger et al., 2002; Foxe et al., 2001), and stimulus location within the visual field (Clark et al., 1995; Murray et al., 2001).

7.1.6 Conclusions

This is the first study that compared the brain mechanisms underlying IC sensitivity to isoluminant chromatic and low-luminance achromatic contrast stimuli. Our results substantially enrich the existing models by revealing that there are partly distinct parallel circuits that can mediate IC sensitivity. It will be important for future research to determine when and how these circuits cooperate or compete during natural vision.

76 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

7.1.7 Acknowledgements

This work has been supported by the Swiss National Science Foundation (Grant 320030-149982 to MMM, Marie-Heim-Vögtlin fellowship PMPDP3_129028 to SCH, as well as the National Centre of Competence in Research project ‘‘SYNAPSY: The Synaptic Bases of Mental Disease” [project 51AU40-125759 to MMM]) as well as the Swiss Brain League (2014 Research Prize to MMM). The authors declare no competing financial interests.

Supplementary Fig 1. a. Exemplar group average VEP from occipital electrode PO8 before realignment. In this plot, the black line represents the Response to IC in luminance contrast and the red line represents the response to IC in chromatic contrast. b. Exemplar group average VEP from occipital electrode PO8 after realignment. In this plot, the black line represents the response to IC in luminance contrast and the red line represents the response to IC in chromatic contrast

77 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

7.1.8 References

Ahnelt, P.K., 1998. The photoreceptor mosaic. Eye (Lond). 12 ( Pt 3b, 531–40. doi:10.1038/eye.1998.142 Altschuler, T.S., Molholm, S., Russo, N.N., Snyder, A.C., Brandwein, A.B., Blanco, D., Foxe, J.J., 2012. Early electrophysiological indices of illusory contour processing within the lateral occipital complex are virtually impervious to manipulations of illusion strength. Neuroimage 59, 4074–85. doi:10.1016/j.neuroimage.2011.10.051 Bourquin, N.M.-P., Murray, M.M., Clarke, S., 2013. Location-independent and location-linked representations of sound objects. Neuroimage 73, 40–9. doi:10.1016/j.neuroimage.2013.01.026 Brandeis, D., Lehmann, D., 1989. Segments of event-related potential map series reveal landscape changes with visual attention and subjective contours. Electroencephalogr. Clin. Neurophysiol. 73, 507–19. Brunet, D., Murray, M.M., Michel, C.M., 2011. Spatiotemporal analysis of multichannel EEG: CARTOOL. Comput. Intell. Neurosci. 2011, 813870. doi:10.1155/2011/813870 Bushnell, B.N., Harding, P.J., Kosai, Y., Bair, W., Pasupathy, A., 2011. Equiluminance cells in visual cortical area v4. J. Neurosci. Off. J. Soc. Neurosci. 31, 12398–12412. doi:10.1523/JNEUROSCI.1890-11.2011 Butler, P.D., Martinez, A., Foxe, J.J., Kim, D., Zemon, V., Silipo, G., Mahoney, J., Shpaner, M., Jalbrzikowski, M., Javitt, D.C., 2007. Subcortical visual dysfunction in schizophrenia drives secondary cortical impairments. Brain 130, 417–30. doi:10.1093/brain/awl233 Chaudhuri, A., Albright, T.D., 1990. A new technique for estimating chromatic isoluminance in humans and monkeys. Vis. Neurosci. 5, 605–608. Clark, V.P., Fan, S., Hillyard, S.A., 1995. Identification of Early Visual Evoked Potential Generators by Retinotopic and Topographic Analyses 187. Csibra, G., Davis, G., Spratling, M.W., Johnson, M.H., 2000. Gamma oscillations and object processing in the infant brain. Science 290, 1582–5. Davis, G., Driver, J., 1994. Parallel detection of Kanizsa subjective figures in the human visual system. Nature 371, 791–793. doi:10.1038/371791a0 De Lucia, M., Michel, C.M., Murray, M.M., 2010. Comparing ICA-based and single-trial topographic ERP analyses. Brain Topogr. 23, 119–27. doi:10.1007/s10548-010-0145-y Ding, Y., Martinez, A., Qu, Z., Hillyard, S.A., 2014. Earliest stages of visual cortical processing are not modified by attentional load. Hum. Brain Mapp. 35, 3008–3024. Doniger, G.M., Foxe, J.J., Murray, M.M., Higgins, B. a, Javitt, D.C., 2002. Impaired visual object recognition and dorsal/ventral stream interaction in schizophrenia. Arch. Gen. Psychiatry 59, 1011–20. Dresp, B., Salvano-Pardieu, V., Bonnet, C., 1996. Illusory form with inducers of opposite contrast polarity: evidence for multistage integration. Percept. Psychophys. 58, 111–124. Ejima, Y., Takahashi, S., 1988. Illusory contours induced by isoluminant chromatic patterns. Vision Res. 28, 1367–1377. Foxe, J.J., Doniger, G.M., Javitt, D.C., 2001. Early visual processing deficits in schizophrenia: impaired P1 generation revealed by high-density electrical mapping. Neuroreport 12, 3815–20. Foxe, J.J., Strugstad, E.C., Sehatpour, P., Molholm, S., Pasieka, W., Schroeder, C.E., McCourt, M.E., 2008. Parvocellular and magnocellular contributions to the initial generators of the visual evoked potential: high-density electrical mapping of the “C1” component. Brain Topogr. 21, 11–21. doi:10.1007/s10548-008-0063-4 Gharat, A., Baker, C.L., 2012. Motion-defined contour processing in the early visual cortex. J. Neurophysiol. 108, 1228–1243. doi:10.1152/jn.00840.2011 Gonzalez Andino, S.L., Michel, C.M., Thut, G., Landis, T., Grave de Peralta, R., 2005. Prediction of response speed by anticipatory high-frequency (gamma band) oscillations in the human brain. Hum. Brain Mapp. 24, 50–8. doi:10.1002/hbm.20056 Grave de Peralta Menendez, R., Gonzalez Andino, S., Lantz, G., Michel, C.M., Landis, T., 2001. Noninvasive localization of electromagnetic epileptic activity. I. Method descriptions and

78 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. simulations. Brain Topogr. 14, 131–7. Grave de Peralta Menendez, R., Murray, M.M., Michel, C.M., Martuzzi, R., Gonzalez Andino, S.L., 2004. Electrical neuroimaging based on biophysical constraints. Neuroimage 21, 527–39. doi:10.1016/j.neuroimage.2003.09.051 Gregory, R.L., 1977. Vision with isoluminant colour contrast: 1.A projection technique and observations. Perception 6, 113–119. Grosof, D.H., Shapley, R.M., Hawken, M.J., 1993. Macaque V1 neurons can signal “illusory” contours. Nature 365, 550–552. doi:10.1038/365550a0 Guthrie, D., Buchwald, J.S., 1991. Significance testing of difference potentials. Psychophysiology 28, 240–244. Hochstein, S., Ahissar, M., 2002. View from the top: hierarchies and reverse hierarchies in the visual system. Neuron 36, 791–804. Ishihara, S., 1972. Tests for color-blindness. Kanehara Shuppan Co., Ltd., Tokyo. Javitt, D.C., 2009. When doors of perception close: bottom-up models of disrupted cognition in schizophrenia. Annu. Rev. Clin. Psychol. 5, 249–75. doi:10.1146/annurev.clinpsy.032408.153502 Kanizsa, G., 1976. Subjective contours. Sci. Am. 234, 48–52. Knebel, J.-F., Javitt, D.C., Murray, M.M., 2011. Impaired early visual response modulations to spatial information in chronic schizophrenia. Psychiatry Res. 193, 168–76. doi:10.1016/j.pscychresns.2011.02.006 Knebel, J.-F., Murray, M.M., 2012. Towards a resolution of conflicting models of illusory contour processing in humans. Neuroimage 59, 2808–17. doi:10.1016/j.neuroimage.2011.09.031 Knebel, J.-F., Toepel, U., Hudry, J., le Coutre, J., Murray, M.M., 2008. Generating controlled image sets in cognitive neuroscience research. Brain Topogr. 20, 284–9. doi:10.1007/s10548-008-0046- 5 Koenig, T., Melie-García, L., 2010. A method to determine the presence of averaged event-related fields using randomization tests. Brain Topogr. 23, 233–42. doi:10.1007/s10548-010-0142-1 Koenig, T., Stein, M., Grieder, M., Kottlow, M., 2014. A tutorial on data-driven methods for statistically assessing ERP topographies. Brain Topogr. 27, 72–83. doi:10.1007/s10548-013-0310-1 Larsson, J., Amunts, K., Gulyás, B., Malikovic, a, Zilles, K., Roland, P.E., 1999. Neuronal correlates of real and illusory contour perception: functional anatomy with PET. Eur. J. Neurosci. 11, 4024– 36. Lee, T.S., Nguyen, M., 2001. Dynamics of subjective contour formation in the early visual cortex. Proc. Natl. Acad. Sci. U. S. A. 98, 1907 –1911. Lehmann, D., Ozaki, H., Pal, I., 1987. EEG alpha map series: brain micro-states by space-oriented adaptive segmentation. Electroencephalogr. Clin. Neurophysiol. 67, 271–288. Lehmann, D., Skrandies, W., 1980. Reference-free identification of components of checkerboard- evoked multichannel potential fields. Electroencephalogr. Clin. Neurophysiol. 48, 609–621. Li, C.Y., Guo, K., 1995. Measurements of geometric illusions, illusory contours and stereo-depth at luminance and colour contrast. Vision Res. 35, 1713–20. Liu, C.-S.J., Bryan, R.N., Miki, A., Woo, J.H., Liu, G.T., Elliott, M.A., 2006. Magnocellular and parvocellular visual pathways have different blood oxygen level-dependent signal time courses in human primary visual cortex. AJNR. Am. J. Neuroradiol. 27, 1628–1634. Maertens, M., Pollmann, S., 2005. fMRI reveals a common neural substrate of illusory and real contours in V1 after perceptual learning. J. Cogn. Neurosci. 17, 1553–64. doi:10.1162/089892905774597209 Maertens, M., Shapley, R., 2008. Local determinants of contour interpolation. J. Vis. 8, 3.1–11. doi:10.1167/8.7.3 Martuzzi, R., Murray, M.M., Meuli, R. a, Thiran, J.-P., Maeder, P.P., Michel, C.M., Grave de Peralta Menendez, R., Gonzalez Andino, S.L., 2009. Methods for determining frequency- and region- dependent relationships between estimated LFPs and BOLD responses in humans. J. Neurophysiol. 101, 491–502. doi:10.1152/jn.90335.2008 Masuda, T., Matsubara, K., Utsumi, K., Wada, Y., 2015. Material perception of a kinetic illusory object with amplitude and frequency changes in oscillated inducer motion. Vision Res., Perception of Material Properties (Part I) 109, Part , 201–208. doi:10.1016/j.visres.2014.11.019 Mattingley, J.B., Davis, G., Driver, J., 1997. Preattentive filling-in of visual surfaces in parietal

79 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

extinction. Science 275, 671–4. Matusz, P.J., Thelen, A., Amrein, S., Geiser, E., Anken, J., Murray, M.M., 2015. The role of auditory cortices in the retrieval of single-trial auditory-visual object memories. Eur. J. Neurosci. doi:10.1111/ejn.12804 Maunsell, J.H., Ghose, G.M., Assad, J.A., McAdams, C.J., Boudreau, C.E., Noerager, B.D., 1999. Visual response latencies of magnocellular and parvocellular LGN neurons in macaque monkeys. Vis. Neurosci. 16, 1–14. Maunsell, J.H., Gibson, J.R., 1992. Visual response latencies in striate cortex of the macaque monkey. J. Neurophysiol. 68, 1332–1344. Mendola, J.D., Dale, a M., Fischl, B., Liu, a K., Tootell, R.B., 1999. The representation of illusory and real contours in human cortical visual areas revealed by functional magnetic resonance imaging. J. Neurosci. 19, 8560–72. Michel, C.M., Murray, M.M., 2012. Towards the utilization of EEG as a brain imaging tool. Neuroimage 61, 371–85. doi:10.1016/j.neuroimage.2011.12.039 Michel, C.M., Murray, M.M., Lantz, G., Gonzalez, S., Spinelli, L., Grave de Peralta, R., 2004. EEG source imaging. Clin. Neurophysiol. Off. J. Int. Fed. Clin. Neurophysiol. 115, 2195–2222. doi:10.1016/j.clinph.2004.06.001 Murray, M.M., Brunet, D., Michel, C.M., 2008. Topographic ERP analyses: a step-by-step tutorial review. Brain Topogr. 20, 249–64. doi:10.1007/s10548-008-0054-5 Murray, M.M., Foxe, D.M., Javitt, D.C., Foxe, J.J., 2004. Setting boundaries: brain dynamics of modal and amodal illusory shape completion in humans. J. Neurosci. 24, 6898–903. doi:10.1523/JNEUROSCI.1996-04.2004 Murray, M.M., Foxe, J.J., Higgins, B. a, Javitt, D.C., Schroeder, C.E., 2001. Visuo-spatial neural response interactions in early cortical processing during a simple reaction time task: a high-density electrical mapping study. Neuropsychologia 39, 828–44. Murray, M.M., Herrmann, C.S., 2013. Illusory contours: a window onto the neurophysiology of constructing perception. Trends Cogn. Sci. 17, 471–481. doi:10.1016/j.tics.2013.07.004 Murray, M.M., Imber, M.L., Javitt, D.C., Foxe, J.J., 2006. Boundary completion is automatic and dissociable from shape discrimination. J. Neurosci. 26, 12043–54. doi:10.1523/JNEUROSCI.3225-06.2006 Murray, M.M., Wylie, G.R., Higgins, B. a, Javitt, D.C., Schroeder, C.E., Foxe, J.J., 2002. The spatiotemporal dynamics of illusory contour processing: combined high-density electrical mapping, source analysis, and functional magnetic resonance imaging. J. Neurosci. 22, 5055–73. Nieder, a, Wagner, H., 1999. Perception and neuronal coding of subjective contours in the owl. Nat. Neurosci. 2, 660–3. doi:10.1038/10217 Nowak, L.G., Munk, M.H., Girard, P., Bullier, J., 1995. Visual latencies in areas V1 and V2 of the macaque monkey. Vis. Neurosci. 12, 371–384. Pourtois, G., Delplanque, S., Michel, C., Vuilleumier, P., 2008. Beyond conventional event-related brain potential (ERP): exploring the time-course of visual emotion processing using topographic and principal component analyses. Brain Topogr. 20, 265–277. doi:10.1007/s10548-008-0053-6 Redies, C., Crook, J.M., Creutzfeldt, O.D., 1986. Neuronal responses to borders with and without luminance gradients in cat visual cortex and dorsal lateral geniculate nucleus. Exp. Brain Res. 61, 469–81. Sáry, G., Chadaide, Z., Tompa, T., Köteles, K., Kovács, G., Benedek, G., 2007. Illusory shape representation in the monkey inferior temporal cortex. Eur. J. Neurosci. 25, 2558–64. doi:10.1111/j.1460-9568.2007.05494.x Sáry, G., Köteles, K., Kaposvári, P., Lenti, L., Csifcsák, G., Frankó, E., Benedek, G., Tompa, T., 2008. The representation of Kanizsa illusory contours in the monkey inferior temporal cortex. Eur. J. Neurosci. 28, 2137–46. doi:10.1111/j.1460-9568.2008.06499.x Schmolesky, M.T., Wang, Y., Hanes, D.P., Thompson, K.G., Leutgeb, S., Schall, J.D., Leventhal, a G., 1998. Signal timing across the macaque visual system. J. Neurophysiol. 79, 3272–8. Schroeder, C.E., Mehta, a D., Givre, S.J., 1998. A spatiotemporal profile of visual system activation revealed by current source density analysis in the awake macaque. Cereb. Cortex 8, 575–92. Seghier, M., Dojat, M., Delon-Martin, C., Rubin, C., Warnking, J., Segebarth, C., Bullier, J., 2000. Moving illusory contours activate primary visual cortex: an fMRI study. Cereb. Cortex 10, 663–

80 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. 70. Smith, A.T., Singh, K.D., Williams, A.L., Greenlee, M.W., 2001. Estimating receptive field size from fMRI data in human striate and extrastriate visual cortex. Cereb. Cortex 11, 1182–90. Song, Y., Baker, C.L., 2007. Neuronal response to texture- and contrast-defined boundaries in early visual cortex. Vis. Neurosci. 24, 65–77. doi:10.1017/S0952523807070113 Soriano, M., Spillmann, L., Bach, M., 1996. The abutting grating illusion. Vision Res. 36, 109–16. Spinelli, L., Andino, S.G., Lantz, G., Seeck, M., Michel, C.M., 2000. Electromagnetic inverse solutions in anatomically constrained spherical head models. Brain Topogr. 13, 115–25. Stanley, D. a, Rubin, N., 2003. fMRI activation in response to illusory contours and salient regions in the human lateral occipital complex. Neuron 37, 323–31. Takahashi, S., Kaihara, T., Takemoto, A., Ido, K., Ejima, Y., 1992. Spectral sensitivities for illusory contour perception: a manifold linkage of chromatic and achromatic cues with the generation of contours. Vision Res. 32, 1709–1718. Toepel, U., Bielser, M.-L., Forde, C., Martin, N., Voirin, A., le Coutre, J., Murray, M.M., Hudry, J., 2015. Brain dynamics of meal size selection in humans. Neuroimage 113, 133–142. doi:10.1016/j.neuroimage.2015.03.041 Tzovara, A., Murray, M.M., Michel, C.M., De Lucia, M., 2012. A tutorial review of electrical neuroimaging from group-average to single-trial event-related potentials. Dev. Neuropsychol. 37, 518–44. doi:10.1080/87565641.2011.636851 Ungerleider, L.G., Mishkin, M., 1982. Two cortical visual systems, in: Visual Behavior. D.J. Ingle, M.A. Goodale & R.J.W. Mansfield, pp. 549–586. von der Heydt, R., Peterhans, E., Baumgartner, G., 1984. Illusory contours and cortical neuron responses. Science 224, 1260–1262. Vuilleumier, P., Valenza, N., Landis, T., 2001. Explicit and implicit perception of illusory contours in unilateral spatial neglect: behavioural and anatomical correlates of preattentive grouping mechanisms. Neuropsychologia 39, 597–610. Yoshino, A., Kawamoto, M., Yoshida, T., Kobayashi, N., Shigemura, J., Takahashi, Y., Nomura, S., 2006. Activation time course of responses to illusory contours and salient region: a high-density electrical mapping comparison. Brain Res. 1071, 137–44. doi:10.1016/j.brainres.2005.11.089

81 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

7.2 Brain mechanisms for perceiving illusory lines in humans

Jacques Anken1, Jean-François Knebel1,2, Micah M. Murray1-4,* 1The LINE (Laboratory for Investigative Neurophysiology), Department of Radiology and Department of Clinical Neurosciences, University Hospital Center and University of Lausanne, Lausanne, Switzerland 2Electroencephalography Brain Mapping Core, Center for Biomedical Imaging (CIBM) of Lausanne and Geneva, Switzerland 3Department of Ophthalmology, University of Lausanne, Jules-Gonin Eye Hospital, Lausanne, Switzerland 4Department of Hearing and Speech Sciences, Vanderbilt University, Nashville, TN, USA

Address correspondence to: Prof. Micah Murray CHUV-UNIL Radiologie, BH08.078 Rue du Bugnon 46 1011 Lausanne [email protected]

82 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

7.2.1 Abstract

Illusory contours (ICs) are perceptions of visual borders despite absent contrast gradients. The psychophysical and neurobiological mechanisms of IC processes have been studied across species and diverse brain imaging/mapping techniques. Nonetheless, debate continues regarding whether IC sensitivity results from a (presumably) feedforward process within low-level visual cortices (V1/V2) or instead are processed first within higher-order brain regions, such as lateral occipital cortices (LOC). Studies in animal models, which generally favour a feedforward mechanism within V1/V2, have typically involved stimuli inducing IC lines. By contrast, studies in humans generally favour a model where IC sensitivity is mediated by LOC and typically involved stimuli inducing IC forms or shapes. Thus, the particular stimulus features used may strongly contribute to the model of IC sensitivity the data support. To address this, we recorded visual evoked potentials (VEPs) while presenting human observers with an array of 10 inducers within the central 5°, two of which could be oriented to induce an IC line on a given trial. VEPs were analysed using an electrical neuroimaging framework. Sensitivity to the presence vs. absence of centrally-presented IC lines was first apparent at ~200ms post-stimulus onset, followed from modulations in the topographic distribution of the VEP, and was localized to LOC. The timing and localization of these effects are consistent with a model of IC sensitivity commencing within higher-level visual cortices. We propose that prior observations of effects within lower-tier cortices (V1/V2) are the result of feedback from IC sensitivity that originates instead within higher-tier cortices (LOC).

Key words: illusory contour, Kanizsa, event-related potential (ERP), visual evoked potential (VEP), Line.

83 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

7.2.2 Introduction

Visual boundaries are perceived even when input to the retina is discontinuous or incomplete, such as under low luminance or low contrast. These perceptions of boundaries have been investigated extensively with illusory contours (IC) (Micah M. Murray and Herrmann, 2013). A particular type of IC was popularized by (Kanizsa, 1976)and is still widely used to investigate both the psychophysical and neurobiological bases of IC perception (Kanizsa, 1976). Kanizsa-type ICs are based on an array of circles, each of which has a sector removed (hereafter referred to as pacmen inducers). These pacmen inducers can be oriented to form an IC or rotated to block such perceptions (i.e. forming non-contours; NCs) (Figure 1). Typical Kanizsa-type ICs result in the perception of geometric shapes (triangles, squares, circles, pentagons, etc.; cf. Figure 1 in(Murray et al., 2002).

Several neurophysiologic models of IC processing have been hypothesized that differ principally in terms of where (and when) sensitivity to ICs first manifests (Micah M. Murray and Herrmann, 2013). One model proposes that low-level visual areas V1/V2 are sensitive to ICs in a bottom-up and feed- forward manner. Support for this model derives principally from microelectrode recordings in animals, where the IC was typically induced by phase-shifted line gratings, rather than Kanizsa-type stimuli (Grosof et al., 1993; Nieder and Wagner, 1999; Redies et al., 1986; von der Heydt et al., 1984). Moreover, the ICs were typically line segments extending just beyond the limits of the classical receptive field of the recorded neuron.

A second model proposes that IC sensitivity is instead first achieved within lateral occipital cortices (LOC) in the ventral visual pathway (Ungerleider and Mishkin, 1982). Any effects in V1/V2 result from feedback modulations from the LOC (Anken et al., 2016; Lee and Nguyen, 2001; Mendola et al., 1999; Murray et al., 2006, 2004, 2002, Sáry et al., 2008, 2007). Support for this model comes largely from studies in humans that involved IC shapes (cf. Figure 3 in Murray and Herrmann, 2013). In particular, our laboratory has previously identified a visual evoked potential (VEP) correlate of illusory contour sensitivity. This so-called IC effect involves stronger VEP responses to IC presence than absence, with an onset as early as ~90ms post-stimulus and sources within bilateral lateral occipital cortices (Murray et al, 2002; reviewed in Murray and Herrmann, 2013). The IC effect is robust to myriad differences in the kinds of stimuli used to induce perceptions of illusory contours (i.e. the particular shape induced, the contrast polarity of the stimuli/background, the types of inducers, whether or not modal or amodal completion is induced, the chromaticity of the inducers, and the parafoveal spatial eccentricity of the inducers) as well as whether or not participants perform a task or even correctly perceive the IC shape (Micah M Murray and Herrmann, 2013).

A further model contends that LOC are sensitive to salient regions defined by inducers and that IC sensitivity happens in V1/V2 only after feedback modulation from the LOC (Stanley and Rubin, 2003; Yoshino et al., 2006). However, positive evidence of modulated responses within V1/V2 is critically missing in the results reported by Stanley and Rubin (2003). It thus remains unknown to what extent regions V1/V2 exhibit illusory contour sensitivity in humans. Evidence supporting the necessary, albeit perhaps pre-attentive, role of LOC in illusory contour sensitivity comes from neuropsychological reports in brain-lesions individuals. Perceptual benefits of illusory contours on a line bisection task were

84 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. observed only when LOC were intact, but not when lesioned (Vuilleumier et al., 2001).In all cases, areas V1/V2 were intact; there was no evidence of lesions or anopia.

Nonetheless, embracing any of these models is complicated, in part, by the variability in the stimuli employed as well as by any contributions of inter-species differences in brain mechanisms of IC sensitivity. Demonstrations of the IC effect in humans typically used inducers that spanned across several degrees of visual angle as well as across either/both the vertical or horizontal meridians of the visual field. Consequently, it can reasonably be argued that feedforward processing of ICs in brain areas such as V1/V2 would be favoured when the stimuli induce illusory lines rather than geometric forms and also when the induced contour spans relatively short distances. Some have partially addressed these points by positioning inducing stimuli within a single visual hemifield (Brandeis and Lehmann, 1989; Murray et al., 2002; Senkowski et al., 2005) or by parametrically varying the eccentricity of the inducers or the support ratio of the bound IC form (and by extension the distance to be perceptually completed;(Altschuler et al., 2012). Such manipulations resulted in delayed IC effects, which is contrary to expectations if IC sensitivity is a strictly feedforward process in V1/V2.

In light of these collective discrepancies, here we presented participants with illusory contour lines while measuring VEPs and the IC effect. Our objective was to emulate stimulation conditions similar to those used in the seminal work in non-human primates by (von der Heydt et al., 1984), thereby facilitating the reconciliation of discrepant findings across species and stimulus parameters. We reasoned that centrally presented and small illusory contour lines would favour visual completion processes within lower-tier visual cortices (V1/V2), if indeed V1/V2 are required for IC sensitivity in humans. Specifically, we hypothesized that if IC sensitivity is indeed mediated by feedforward processes in V1/V2 then the IC effect in response to illusory contour lines would be earlier than that characterized in our (and others’) prior works that presented relatively large IC shapes. Likewise an IC effect mediated by feedforward processes within V1/V2 would not be predicted to be affected by other surrounding inducer stimuli that fail to form illusory contours. For this reason, stimulus arrays consisted of multiple inducer stimuli; although only a pair of which could result in an illusory contour on any trial. By contrast, IC sensitivity at latencies >90ms post-stimulus and with localization within LOC would be consistent with a potentially size-invariant mechanism within higher-level visual cortices (Dura-Bernal et al., 2011; Mendola et al., 1999; Murray et al., 2002). Moreover, the inclusion of additional inducers aside from those producing IC lines would reveal the extent to which visual clutter and/or distributed spatial attention impact IC sensitivity, which would likewise support models of IC sensitivity occurring first in high-level visual cortices with potential feedback to low-level regions.

7.2.3 Materials and Methods

7.2.3.a Participants

Analyses presented in this study are based on data from 11 participants (6 male, all right-handed; aged 23-36years, mean 27.7 years). No subject had a history of or current neurological or psychiatric illnesses. All participants had normal or corrected-to normal vision. Data from an additional 3 subjects, beyond

85 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. the 11 reported here, were excluded due to either technical issues with behavioural response recording during data acquisition (N = 1) or excessive muscle and/or alpha frequency EEG artefacts (N = 2).

7.2.3.b Stimuli and task

Stimuli were comprised of a set of 10 circular Kanizsa-type (Kanizsa, 1976) ‘pacmen’ inducers that were arranged in an array (Figure 1). The size of the array was 4.43° wide x 1.86° high, each inducer subtended 0.57° in diameter, and the induced illusory contour line was 1.50° in length from a distance of 80cm. Illusory contour lines could be oriented, when present, either horizontally or vertically with equal likelihood. Stimuli were displayed on a LCD computer monitor (20″ active TFT, 1600 x 1200 @ 60Hz, 16ms pixel response time). On a given trial, two of the ten inducers were positioned with their mouths facing each other to create an illusory contour line that was presented either centrally or laterally (left, right). Alternatively and with equal likelihood, no inducers were facing each other to create a no contour (NC) equivalent. These variations in the presentation of ICs were included to prevent participants from selectively focusing on a particular region of space or on any single inducer as a strategy to successfully complete the task. Each participant completed 10 blocks of trials. Each block contained 160 stimuli with equivalent probability of apparition of vertical and horizontal IC in one location and NC conditions. Stimuli were presented for 500ms with an inter-stimulus interval ranging between 800 and 1200ms with a uniform distribution. A white central fixation cross was displayed on the computer screen during the inter-stimulus interval. During the experiment, participants took regular breaks between blocks to maintain high concentration and prevent fatigue.

Participants performed a four-alternative forced choice that required indicating the presence vs. absence of an illusory contour and, if judged present, whether the IC was positioned in the left, center, or right. All participants answered with their right hand. Accuracy and reaction time were measured with a serial response box (Psychology Software Tools; https://www.pstnet.com/hardware.cfm?ID=102). Stimulus delivery and behavioural response collection were controlled with PsychoPy (Peirce, 2007; Peirce and Peirce, 2009).

Fig 1. Exemplar inducer stimulus array and its spacing.

The array consisted of 10 circular inducers presented within the parafovea (5° of visual angle). Spacing between inducers was as indicated. In this example, 2 inducers were oriented to result in the perception of a central, horizontal line. Full details are provided in Materials and Methods.

7.2.3.c EEG acquisition and pre-processing

Continuous EEG was acquired at 1024Hz through a 128-channel Biosemi ActiveTwo AD-box (http://www.biosemi.com) referenced to the common mode sense (CMS; active electrode) and grounded to the driven right leg (DRL; passive electrode), which functions as a feedback loop driving the average

86 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. potential across the electrode montage to the amplifier zero (full details, including a diagram of this circuitry, can be found at http://www.biosemi.com/faq/cms&drl.htm). Prior to epoching, the continuous EEG was filtered (0.1Hz high-pass; 60Hz low-pass; 50Hz notch). The filters were computed linearly in both forward and backward directions to eliminate phase shifts.

EEG epochs were time-locked to the visual presentation of stimuli and spanned 100ms pre-stimulus to 800ms post-stimulus. Epochs with amplitude deviations over ±80μV at any channel, with the exception of electrodes with poor electrode-skin contact or damage labelled as ‘bad’, were considered artefacts and were excluded. Eye blinks were also excluded off-line based on vertical and horizontal electro- oculograms. After averaging, channels labelled as ‘bad’ were interpolated using 3D splines (Perrin et al., 1987). Data from the visual evoked potential (VEP) were baseline-corrected using the pre-stimulus interval and re-calculated against an average reference. For each participant, 2 VEPs were calculated: IC in the Center (ICC) and No contour (NCC). The mean number of accepted EEG epochs (±s.e.m.) for each of these conditions was 189±3 and 190±3.

7.2.3.d VEP analyses

VEP analyses were performed with both the Cartool freeware (http://sites.google.com/site/fbmlab/cartool/cartooldownload; (Brunet et al., 2011) and statistical as well as STEN utilities (http://www.unil.ch/line/home/menuinst/about-the-line/software--analysis- tools.html). An electrical neuroimaging analysis framework identified effects (Koenig et al., 2014; Michel et al., 2004; Michel and Murray, 2012; Murray et al., 2008; Tzovara et al., 2012). These analyses differentiate between effects due to modulations in VEP response strength, latency, or topography. Our VEP analyses focused on identifying brain mechanisms underlying the detection of centrally-presented illusory contour lines and contrasted ICC vs. NCC.

We first performed a mass univariate test (each electrode as a function of peri-stimulus time). This was included here primarily for illustrative purposes, given the well-known effect of the choice of the reference on statistical analyses of voltage waveforms (c.f. (Murray et al., 2008)for discussion). It also is included here to facilitate comparison with works displaying voltage waveforms and to facilitate comprehension of the results for readers less familiar with measures such as global field power or global dissimilarity.

In terms of the electrical neuroimaging framework, we first statistically compared the Global Field Power (GFP), which quantifies the electric field at the scalp level (Lehmann and Skrandies, 1980). This measure of the VEP strength is reference-independent and is equivalent to the standard deviation of the voltage potential values of electrodes at a given time point (Koenig et al., 2014; Koenig and Melie- García, 2010; Murray et al., 2008). Global Dissimilarity (DISS) was then analysed in order to test for changes in the VEP topography independently of its strength (Lehmann and Skrandies, 1980). DISS is equivalent to the square root of the mean of the squared difference between the potentials measured at each electrode for different conditions, normalized by the instantaneous GFP. This measure is directly related to the (spatial) correlation between two normalized vectors (cf. Appendix in (Murray et al., 2008))). The DISS was used in an analysis called “TANOVA” (Murray et al., 2008)). In the “TANOVA”, The DISS value at each time point is compared to an empirical distribution derived from permuting the condition label of the data from each subject. As changes in VEP topographies at the

87 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. scalp forcibly reflect changes in the configuration of active generators in the brain (Lehmann et al., 1987), this analysis indicates if distinct brain networks are involved in IC sensitivity and/or during brain discrimination of the orientation of ICs. Additionally, a topographic cluster analysis based on a hierarchical clustering algorithm (Murray et al., 2008) was performed on the group-average VEPs. This clustering analysis identifies stable VEP topographies across time in the group-averaged data after first strength-normalizing the data (hereafter template maps). In this way, this clustering is sensitive exclusively to topographic modulations within and between conditions. The optimal number of template maps (i.e. the minimal number of maps that accounts for the greatest variance of the dataset) was determined using a modified Krzanowski-Lai criterion (Murray et al., 2008). Template maps identified in the group-averaged VEP were then submitted to a fitting procedure in which each time point of each single-subject VEP from each condition was labelled according to the template map with which it best correlated spatially (Murray et al., 2008).This procedure allows for statistically testing the relative presence of each template map across time of the single-subject VEPs; and therefore the differences across conditions. These values can be expressed as the probability of a given template map yielding a higher spatial correlation in the single-subject data from each condition. Statistical analysis of these values was performed with the Wilcoxon signed-rank test.

7.2.3.e Source Estimations

Finally, we estimated the intracranial sources using a distributed linear inverse solution (ELECTRA) together with the local autoregressive average (LAURA) regularization approach to address the non- uniqueness of the inverse problem (Grave de Peralta Menendez et al., 2004, 2001; Michel et al., 2004). This inverse solution algorithm is based on biophysical principles derived from the quasi-stationary Maxwell's equations; most notably the fact that independent of the volume conductor model used to describe the head, only irrotational and not solenoidal currents contribute to the EEG (Grave de Peralta Menendez et al., 2004, 2001). As part of the LAURA regularization strategy, homogenous regression coefficients in all directions and within the whole solution space were used. The solution space includes 3005 nodes, distributed within the grey matter of the Montreal Neurological Institute's average brain (courtesy of Dr. Rolando Grave de Peralta Menendez and Dr. Sara Gonzalez Andino). The head model and lead field matrix were generated with the Spherical Model with Anatomical Constraints (SMAC;(Spinelli et al., 2000) as implemented in Cartool. As an output, LAURA provides current density measures; the scalar values of which were evaluated at each node. Prior basic and clinical research has documented and discussed in detail the spatial accuracy of this inverse solution (Gonzalez Andino et al., 2005; Grave de Peralta Menendez et al., 2004, 2004; Martuzzi et al., 2009). The relevant time interval for source estimation was determined based on the above topographic clustering analysis to identify periods of temporally-stable VEP topography. Data were first averaged across time for each subject and condition, and then source estimations were calculated based on these time-averaged data. These data matrices were then contrasted using a paired t-test. To partially correct for multiple testing we applied a significance threshold of p<0.05 at each solution point as well as a spatial-extent criterion (kE) of >15 contiguous solution points (Bourquin et al., 2013; De Lucia et al., 2010; Knebel and Murray, 2012, p. 201; Matusz et al., 2015; Toepel et al., 2015).

88 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

7.2.4 Results

Accuracy rates for all conditions were between 96-98%, mean 97%. These values provide a clear indication that all conditions were visible and readily perceived. There were no reliable differences between ICC and NCC (t(10) = 0.52; p>0.05). Analyses of reaction times indicated faster responses for the presence than absence of illusory contours (ICC vs. NCC: 589±17ms vs. 688±20ms; t(10) = 8.95,p<0.01).

Figure 2. VEP correlates of sensitivity to IC lines as revealed by the electrical neuroimaging analysis framework. a. Group-averaged (N=11) VEPs in response to the ICC and NCC conditions at an exemplar midline occipital scalp site (Oz) are displayed. b. The results of a mass univariate analysis as a function of time across the full 128-channel electrode montage show differences starting at 189ms post-stimulus onset. c. The group-averaged global field power (GFP) waveforms from the ICC and NCC conditions are displayed. Significant differences were observed over the 416-456ms post-stimulus period (shaded pale green area). d. The analysis of VEP topography based on global dissimilarity showed that ICC and NCC first differed topographically over the 181-334ms post-stimulus with additional subsequent effects over the majority of the post-stimulus period (indicated by green areas). e. The group-averaged ICC and NCC responses were submitted to a topographic cluster analysis. Two template maps were identified over the 212-319ms post-stimulus time period. These maps were fitted to the single-subject data from each condition using spatial correlation. The amount of time each map better correlated spatially with each condition is shown in the bar graph (mean±s.e.m. displayed). Different maps significantly better characterised responses to each condition. f. Statistical differences between group-averaged distributed source estimations in response to ICC and NCC conditions were observed within the left LOC and extending to the ventral occipito-temporal cortex.

Figure 2a displays VEPs in response to the ICC and NCC conditions at an exemplar midline occipital scalp site (Oz). Both ICC and NC elicited robust VEPs with characteristic P1-N1 components of indistinguishable magnitude. In order to identify the timing of differential VEP responses, we first performed a mass univariate analysis as a function of time across the full 128-channel electrode montage (Figure 2b). To (partially) account for both temporal and spatial correlation, differences were considered

89 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. reliable if significant for at least 15ms consecutively (i.e. >15 time samples) as well as across at least 20% of the electrode montage (i.e. >25 electrodes). Reliable differences began at 189ms post-stimulus onset. Next, analyses were performed using reference-independent and global measures of the brain’s electric field at the scalp. The first of these was the global field power (GFP), which exhibited significant differences between responses to ICC and NCC over the 416-456ms post-stimulus period (Figure 2c). Responses were significantly stronger to the ICC than NCC condition. Analysis of the VEP topography, using global dissimilarity, indicated that responses to ICC and NCC first differed topographically over the 181-334ms post-stimulus interval with additional differences during subsequent post-stimulus intervals (Figure 2d).

In order to better understand the nature of these topographic differences, the group-averaged ICC and NCC data were submitted to a topographic cluster analysis. Two template maps were identified to be present over the 212-319 ms post-stimulus period in the group-averaged data. These template maps were then fitted to the single-subject ICC and NCC VEPs to determine the relative presence of each map over the 212-319ms period in each subject’s data. These values were then submitted to a Wilcoxon Signed- Ranks Test, which indicated a significant interaction Z=21, p <0.05. One map predominated responses to ICC, whereas another map predominated responses to NCC (Figure 2e). Finally, analyses were performed using distributed source estimations from the 212-319ms period (i.e. the period identified in the above topographic clustering analysis). Figure 2f displays the statistical differences between group- averaged source estimations in response to ICC and NCC conditions. Significantly stronger source estimations were observed in response to the ICC condition within the left LOC and extending to the ventral occipito-temporal cortex.

7.2.5 Discussion

This study addressed a major discrepancy across prior investigations of the neural mechanisms subserving illusory contour sensitivity in humans and animals; namely whether or not this sensitivity unfolds in a strictly feedforward manner. This discrepancy was hitherto exacerbated by the fact that while many studies in animals presented small and central stimuli resulting in illusory contour lines, all prior studies in humans had presented comparatively large, bilateral stimuli resulting in illusory contour forms or shapes. By presenting illusory contour lines to humans and recording VEPs we addressed this discrepancy and showed that a strictly feedforward model of IC sensitivity is untenable. Rather, IC sensitivity appears to rely first on processes within LOC.

Contrary to what would have been predicted by a strictly feedforward mechanism, the timing of the IC effect was delayed and onset at ~200ms when presenting participants with IC lines, in contrast to the onset at ~90ms in prior works involving presentation of IC shapes (Micah M Murray and Herrmann, 2013). Two temporally distinct stages of perceptual completion have been distinguished. The first, considered to be largely automatic or at least invariant to task demands and performance outcome, onsets at ~90ms and peaks at ~150ms. This is the IC effect that our laboratory and many others have repeatedly characterised, which has been reliably observed independently of multiple variations in the low-level features and types of stimuli presented (reviewed in Micah M Murray and Herrmann, 2013; see also Anken et al., 2016). The second stage of perceptual completion is considered to reflect a shift to a more

90 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. effortful or conceptual mode of perceptual completion (e.g. (Altschuler et al., 2012; Doniger et al., 2001; Humphreys et al., 2000; Murray et al., 2002; Ritter et al., 1982; Tulving and Schacter, 1990). This stage has been characterized by a VEP modulation referred to as the closure negativity or Ncl (Doniger et al., 2000). The Ncl has been shown to track the perceptual completion of fragmented images, with an onset at ~220ms post-stimulus onset, peak at ~300ms, and bilateral LOC localisation (Sehatpour et al., 2008, 2006). These temporal stages have moreover been dissociated in patients with schizophrenia, in whom the IC effect appears to be intact (Foxe et al., 2005, p. 205; Knebel et al., 2011) whereas the Ncl is severely impaired (Doniger et al., 2002).

The relatively protracted timing of IC sensitivity we observed here was also accompanied by significant topographic VEP modulations between responses to ICC and NCC over the 212-319ms post-stimulus period (Figures 2d and 2e). IC sensitivity occurring during the period of the IC effect consistently manifests as a modulation in VEP strength in the absence of modulations in VEP topography (Knebel and Murray, 2012; Murray et al., 2006, 2004; Pegna et al., 2002). However, prior studies reporting IC sensitivity during the Ncl stage have instead reliably observed topographic VEP modulations between contour present and absent conditions (Brandeis and Lehmann, 1989; Shpaner et al., 2009; Yoshino et al., 2006). The present results are thus in strong agreement with this pattern. Topographic modulations forcibly reflect differences in the configuration of the underlying sources active in the brain (Lehmann et al., 1987). Statistical analyses of source estimations localised effects within the left LOC and ventral occipito-temporal cortices, which are regions repeatedly shown to be involved both in IC sensitivity as well as perceptual completion.

It could perhaps be contended that the timing of the IC sensitivity we observed here was a consequence of the number of inducers presented on a given trial, many of which did not result in the formation or perception of an IC line. That is, the complexity of the visual context in which illusory contours appear may be a (partial) determinant of the timing of IC sensitivity. Two studies from separate laboratories argue against this possibility. On the one hand, Senkowski et al. (2005) showed that visual search performance was unaffected by increasing stimulus set sizes; i.e. RTs increased with a slope below 10ms per item in the stimulus set. On the other hand, Halgren et al. (Halgren et al., 2003) observed an IC effect with a peak latency of 155ms within LOC despite presenting a large array of 56 inducers on each trial in the absence of task requirements beyond the maintenance of central fixation (cf. their Figure 1). If the number of inducers were strongly influencing the timing of IC sensitivity, then a delayed effect relative to that observed with substantially fewer inducers would have been expected. This was not the case.

It might also be argued that the uncertainty of the spatial location of the illusory contours is the determinant of the latency of IC sensitivity. That is, in our study the IC lines, when present, could appear with equal probability at any of three locations (centrally, left, and right) and two orientations (horizontal and vertical). Participants were thus required to divide their attention accordingly; albeit limited spatially to the central 5° for the whole array of 10 inducers. While this might perhaps account for differences between an IC effect at ~90ms vs. ~200ms, and by extension the shift from a perceptual to conceptual mode of visual completion, it would nonetheless fail to explain why the IC effect did not shift earlier in time if indeed IC sensitivity were under the control of a strictly feedforward mechanism. Precisely this shift to a later IC effect has been reported in Experiment 5 of Murray et al. (2002), where stimulus arrays were limited to one or the other visual hemifield on any trial, as well as in Experiment 2 of Senkowski

91 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. et al. (2005), where stimulus arrays consisted of 23 bilaterally-distributed inducers and IC shapes were presented on 67% of trials to either the left or right visual hemifield (see also Brandeis and Lehmann, 1989).

More generally, the present data contribute to the accumulation of evidence favouring a model of IC sensitivity that relies first on processes within LOC (Micah M Murray and Herrmann, 2013). The principal contribution here is to have addressed a prior gap between how IC sensitivity was studied in humans versus in animal models.

7.2.6 Acknowledgements

The Swiss National Science Foundation (grants 320030-149982 and 320030-169206), the Swiss Brain League (2014 Research Prize) and Carigest SA provided financial support to MMM.

92 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

7.2.7 References

Altschuler, T.S., Molholm, S., Russo, N.N., Snyder, A.C., Brandwein, A.B., Blanco, D., Foxe, J.J., 2012. Early electrophysiological indices of illusory contour processing within the lateral occipital complex are virtually impervious to manipulations of illusion strength. NeuroImage 59, 4074–4085. doi:10.1016/j.neuroimage.2011.10.051 Anken, J., Knebel, J.-F., Crottaz-Herbette, S., Matusz, P.J., Lefebvre, J., Murray, M.M., 2016. Cue- dependent circuits for illusory contours in humans. NeuroImage 129, 335–344. doi:10.1016/j.neuroimage.2016.01.052 Bourquin, N.M.-P., Murray, M.M., Clarke, S., 2013. Location-independent and location-linked representations of sound objects. NeuroImage 73, 40–49. doi:10.1016/j.neuroimage.2013.01.026 Brandeis, D., Lehmann, D., 1989. Segments of event-related potential map series reveal landscape changes with visual attention and subjective contours. Electroencephalogr. Clin. Neurophysiol. 73, 507–519. Brunet, D., Murray, M.M., Michel, C.M., 2011. Spatiotemporal analysis of multichannel EEG: CARTOOL. Comput. Intell. Neurosci. 2011, 813870. doi:10.1155/2011/813870 De Lucia, M., Michel, C.M., Murray, M.M., 2010. Comparing ICA-based and single-trial topographic ERP analyses. Brain Topogr. 23, 119–127. doi:10.1007/s10548-010-0145-y Doniger, G.M., Foxe, J.J., Murray, M.M., Higgins, B.A., Javitt, D.C., 2002. Impaired Visual Object Recognition and Dorsal/Ventral Stream Interaction in Schizophrenia. Arch Gen Psychiatry 59, 1011–1020. doi:10.1001/archpsyc.59.11.1011 Doniger, G.M., Foxe, J.J., Murray, M.M., Higgins, B.A., Snodgrass, J.G., Schroeder, C.E., Javitt, D.C., 2000. Activation timecourse of ventral visual stream object-recognition areas: high density electrical mapping of perceptual closure processes. J. Cogn. Neurosci. 12, 615–621. Doniger, G.M., Foxe, J.J., Schroeder, C.E., Murray, M.M., Higgins, B.A., Javitt, D.C., 2001. Visual perceptual learning in human object recognition areas: a repetition priming study using high-density electrical mapping. NeuroImage 13, 305–313. doi:10.1006/nimg.2000.0684 Dura-Bernal, S., Wennekers, T., Denham, S.L., 2011. The role of feedback in a hierarchical model of object perception. Adv. Exp. Med. Biol. 718, 165–179. doi:10.1007/978-1-4614-0164- 3_14 Foxe, J.J., Murray, M.M., Javitt, D.C., 2005. Filling-in in Schizophrenia: a High-density Electrical Mapping and Source-analysis Investigation of Illusory Contour Processing. Cereb Cortex 15, 1914–1927. doi:10.1093/cercor/bhi069 Gonzalez Andino, S.L., Michel, C.M., Thut, G., Landis, T., Grave de Peralta, R., 2005. Prediction of response speed by anticipatory high-frequency (gamma band) oscillations in the human brain. Hum. Brain Mapp. 24, 50–58. doi:10.1002/hbm.20056 Grave de Peralta Menendez, R., Gonzalez Andino, S., Lantz, G., Michel, C.M., Landis, T., 2001. Noninvasive localization of electromagnetic epileptic activity. I. Method descriptions and simulations. Brain Topogr. 14, 131–137. Grave de Peralta Menendez, R., Murray, M.M., Michel, C.M., Martuzzi, R., Gonzalez Andino, S.L., 2004. Electrical neuroimaging based on biophysical constraints. NeuroImage 21, 527– 539. doi:10.1016/j.neuroimage.2003.09.051 Grosof, D.H., Shapley, R.M., Hawken, M.J., 1993. Macaque V1 neurons can signal “illusory” contours. Nature 365, 550–552. doi:10.1038/365550a0 Halgren, E., Mendola, J., Chong, C.D.R., Dale, A.M., 2003. Cortical activation to illusory shapes as measured with magnetoencephalography. NeuroImage 18, 1001–1009. Humphreys, G.W., Cinel, C., Wolfe, J., Olson, A., Klempen, N., 2000. Fractionating the binding process: neuropsychological evidence distinguishing binding of form from binding of surface features. Vision Res. 40, 1569–1596. Kanizsa, G., 1976. Subjective contours. Sci. Am. 234, 48–52.

93 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

Knebel, J.-F., Javitt, D.C., Murray, M.M., 2011. Impaired early visual response modulations to spatial information in chronic schizophrenia. Psychiatry Res. 193, 168–176. doi:10.1016/j.pscychresns.2011.02.006 Knebel, J.-F., Murray, M.M., 2012. Towards a resolution of conflicting models of illusory contour processing in humans. NeuroImage 59, 2808–2817. doi:10.1016/j.neuroimage.2011.09.031 Koenig, T., Melie-García, L., 2010. A method to determine the presence of averaged event-related fields using randomization tests. Brain Topogr. 23, 233–242. doi:10.1007/s10548-010- 0142-1 Koenig, T., Stein, M., Grieder, M., Kottlow, M., 2014. A tutorial on data-driven methods for statistically assessing ERP topographies. Brain Topogr. 27, 72–83. doi:10.1007/s10548- 013-0310-1 Lee, T.S., Nguyen, M., 2001. Dynamics of subjective contour formation in the early visual cortex. Proc. Natl. Acad. Sci. U. S. A. 98, 1907–1911. Lehmann, D., Ozaki, H., Pal, I., 1987. EEG alpha map series: brain micro-states by space-oriented adaptive segmentation. Electroencephalogr. Clin. Neurophysiol. 67, 271–288. Lehmann, D., Skrandies, W., 1980. Reference-free identification of components of checkerboard- evoked multichannel potential fields. Electroencephalogr. Clin. Neurophysiol. 48, 609– 621. Martuzzi, R., Murray, M.M., Meuli, R.A., Thiran, J.-P., Maeder, P.P., Michel, C.M., Grave de Peralta Menendez, R., Gonzalez Andino, S.L., 2009. Methods for determining frequency- and region-dependent relationships between estimated LFPs and BOLD responses in humans. J. Neurophysiol. 101, 491–502. doi:10.1152/jn.90335.2008 Matusz, P.J., Thelen, A., Amrein, S., Geiser, E., Anken, J., Murray, M.M., 2015. The role of auditory cortices in the retrieval of single-trial auditory-visual object memories. Eur. J. Neurosci. 41, 699–708. doi:10.1111/ejn.12804 Mendola, J.D., Dale, A.M., Fischl, B., Liu, A.K., Tootell, R.B., 1999. The representation of illusory and real contours in human cortical visual areas revealed by functional magnetic resonance imaging. J. Neurosci. Off. J. Soc. Neurosci. 19, 8560–8572. Michel, C.M., Murray, M.M., 2012. Towards the utilization of EEG as a brain imaging tool. NeuroImage 61, 371–385. doi:10.1016/j.neuroimage.2011.12.039 Michel, C.M., Murray, M.M., Lantz, G., Gonzalez, S., Spinelli, L., Grave de Peralta, R., 2004. EEG source imaging. Clin. Neurophysiol. Off. J. Int. Fed. Clin. Neurophysiol. 115, 2195–2222. doi:10.1016/j.clinph.2004.06.001 Murray, M.M., Brunet, D., Michel, C.M., 2008. Topographic ERP analyses: a step-by-step tutorial review. Brain Topogr. 20, 249–264. doi:10.1007/s10548-008-0054-5 Murray, M.M., Foxe, D.M., Javitt, D.C., Foxe, J.J., 2004. Setting boundaries: brain dynamics of modal and amodal illusory shape completion in humans. J. Neurosci. Off. J. Soc. Neurosci. 24, 6898–6903. doi:10.1523/JNEUROSCI.1996-04.2004 Murray, M.M., Herrmann, C.S., 2013. Illusory contours: a window onto the neurophysiology of constructing perception. Trends Cogn. Sci. 17, 471–481. doi:10.1016/j.tics.2013.07.004 Murray, M.M., Herrmann, C.S., 2013. Illusory contours: a window onto the neurophysiology of constructing perception. Trends Cogn. Sci. 17, 471–481. doi:10.1016/j.tics.2013.07.004 Murray, M.M., Imber, M.L., Javitt, D.C., Foxe, J.J., 2006. Boundary completion is automatic and dissociable from shape discrimination. J. Neurosci. Off. J. Soc. Neurosci. 26, 12043–12054. doi:10.1523/JNEUROSCI.3225-06.2006 Murray, M.M., Wylie, G.R., Higgins, B.A., Javitt, D.C., Schroeder, C.E., Foxe, J.J., 2002. The spatiotemporal dynamics of illusory contour processing: combined high-density electrical mapping, source analysis, and functional magnetic resonance imaging. J. Neurosci. Off. J. Soc. Neurosci. 22, 5055–5073. Nieder, A., Wagner, H., 1999. Perception and neuronal coding of subjective contours in the owl. Nat. Neurosci. 2, 660–663. doi:10.1038/10217 Pegna, A.J., Khateb, A., Murray, M.M., Landis, T., Michel, C.M., 2002. Neural processing of illusory and real contours revealed by high-density ERP mapping. Neuroreport 13, 965–968.

94 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. Peirce, J.W., 2007. PsychoPy—Psychophysics software in Python. J. Neurosci. Methods 162, 8–13. doi:10.1016/j.jneumeth.2006.11.017 Peirce, J.W., Peirce, J.W., 2009. Generating stimuli for neuroscience using PsychoPy. Front. Neuroinformatics 2, 10. doi:10.3389/neuro.11.010.2008 Redies, C., Crook, J.M., Creutzfeldt, O.D., 1986. Neuronal responses to borders with and without luminance gradients in cat visual cortex and dorsal lateral geniculate nucleus. Exp. Brain Res. 61, 469–481. Ritter, W., Simson, R., Vaughan, H.G., Macht, M., 1982. Manipulation of event-related potential manifestations of information processing stages. Science 218, 909–911. Sáry, G., Chadaide, Z., Tompa, T., Köteles, K., Kovács, G., Benedek, G., 2007. Illusory shape representation in the monkey inferior temporal cortex. Eur. J. Neurosci. 25, 2558–2564. doi:10.1111/j.1460-9568.2007.05494.x Sáry, G., Köteles, K., Kaposvári, P., Lenti, L., Csifcsák, G., Frankó, E., Benedek, G., Tompa, T., 2008. The representation of Kanizsa illusory contours in the monkey inferior temporal cortex. Eur. J. Neurosci. 28, 2137–2146. doi:10.1111/j.1460-9568.2008.06499.x Sehatpour, P., Molholm, S., Javitt, D.C., Foxe, J.J., 2006. Spatiotemporal dynamics of human object recognition processing: an integrated high-density electrical mapping and functional imaging study of “closure” processes. NeuroImage 29, 605–618. doi:10.1016/j.neuroimage.2005.07.049 Sehatpour, P., Molholm, S., Schwartz, T.H., Mahoney, J.R., Mehta, A.D., Javitt, D.C., Stanton, P.K., Foxe, J.J., 2008. A human intracranial study of long-range oscillatory coherence across a frontal– occipital–hippocampal brain network during visual object processing. Proc. Natl. Acad. Sci. 105, 4399–4404. doi:10.1073/pnas.0708418105 Senkowski, D., Röttger, S., Grimm, S., Foxe, J.J., Herrmann, C.S., 2005. Kanizsa subjective figures capture visual spatial attention: evidence from electrophysiological and behavioral data. Neuropsychologia 43, 872–886. doi:10.1016/j.neuropsychologia.2004.09.010 Shpaner, M., Murray, M.M., Foxe, J.J., 2009. Early processing in the human lateral occipital complex is highly responsive to illusory contours but not to salient regions. Eur. J. Neurosci. 30, 2018–2028. doi:10.1111/j.1460-9568.2009.06981.x Spinelli, L., Andino, S.G., Lantz, G., Seeck, M., Michel, C.M., 2000. Electromagnetic inverse solutions in anatomically constrained spherical head models. Brain Topogr. 13, 115–125. Stanley, D.A., Rubin, N., 2003. fMRI activation in response to illusory contours and salient regions in the human lateral occipital complex. Neuron 37, 323–331. Toepel, U., Bielser, M.-L., Forde, C., Martin, N., Voirin, A., le Coutre, J., Murray, M.M., Hudry, J., 2015. Brain dynamics of meal size selection in humans. NeuroImage 113, 133–142. doi:10.1016/j.neuroimage.2015.03.041 Tulving, E., Schacter, D.L., 1990. Priming and human memory systems. Science 247, 301–306. Tzovara, A., Murray, M.M., Michel, C.M., De Lucia, M., 2012. A tutorial review of electrical neuroimaging from group-average to single-trial event-related potentials. Dev. Neuropsychol. 37, 518–544. doi:10.1080/87565641.2011.636851 Ungerleider, L.G., Mishkin, M., 1982. Two cortical visual systems, in: Visual Behavior. D.J. Ingle, M.A. Goodale & R.J.W. Mansfield, pp. 549–586. von der Heydt, R., Peterhans, E., Baumgartner, G., 1984. Illusory contours and cortical neuron responses. Science 224, 1260–1262. Vuilleumier, P., Valenza, N., Landis, T., 2001. Explicit and implicit perception of illusory contours in unilateral spatial neglect: behavioural and anatomical correlates of preattentive grouping mechanisms. Neuropsychologia 39, 597–610. Yoshino, A., Kawamoto, M., Yoshida, T., Kobayashi, N., Shigemura, J., Takahashi, Y., Nomura, S., 2006. Activation time course of responses to illusory contours and salient region: a high- density electrical mapping comparison. Brain Res. 1071, 137–144. doi:10.1016/j.brainres.2005.11.089

95 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

7.3 Training Orientation Discrimination Using Visual-to-auditory

Sensory Substitution

Jacques Anken1, Jean-François Knebel2,3, Rosanna De Meo4, Francine Behar-Cohen5, Amir Amedi6-7, and Micah M. Murray1-4,6,8,*

The Laboratory for Investigative Neurophysiology (The LINE), 1Department of Clinical Neurosciences and2Department of Radiology, University Hospital Middle and University of Lausanne, Lausanne, Switzerland.

3Electroencephalography Brain Mapping Core, Centre for Biomedical Imaging (CIBM) of Lausanne and Geneva, Switzerland.

4Center for Mind and Brain, University of California, Davis

5Department of Ophthalmology, University of Lausanne, Jules-Gonin Eye Hospital, Lausanne,Switzerland.

6Department of Medical Neurobiology, Institute for Medical Research Israel-Canada (IMRIC), Hadassah Medical School, Hebrew University of Jerusalem, Jerusalem, Israel.

7The Edmond & Lily Safra Centre for Brain Sciences (ELSC), and Cognitive Science Program Hebrew University of Jerusalem, Jerusalem, Israel.

8 Department of Hearing and Speech Sciences, Vanderbilt University, Nashville, TN, USA

Address correspondence to:

96 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

7.3.1 Abstract

Vision impairments and blindness are a worldwide burden. Non-invasive devices constitute one approach to help restore visual functions. In particular, visual-to-auditory sensory substitution devices (SSDs) are promising. Individuals (both sighted and blind) trained with these SSDs can achieve a high level of discrimination and identification. Moreover, their use results in specific brain activity in what are otherwise nominally visual brain regions, though the qualia of such brain activity remain unresolved. Nonetheless, there is little empirical evidence concerning how to effectively and efficiently train individuals in using SSDs. The present study addressed this issue by training sighted individuals to use the EyeMusic SSD to discriminate between 2 letters (T and U) and their vertical orientation. Immediately prior to and following training, participants were tested in their ability to identify each of the 4 cardinal orientations of a different letter (E). Moreover, the stimuli were either presented at the same location (and by extension auditory frequency) or a new location as that used during the training session. First, we found that training for 15 minutes significantly improved T/U vertical orientation discrimination, independent of whether the SSD algorithm was explained. Second, performance on E orientation discrimination improved for the trained (vertical) orientation and generalized to the untrained (horizontal) orientation. Third, effects of training also generalized to orientation discrimination when stimuli were presented at untrained locations, albeit to a greater extent for the trained than untrained orientations. Collectively, these results provide insights into effective training protocols for the use of SSDs that may be completed autonomously.

Key words: Visual-to-auditory sensory substitution, EyeMusic, Training, Psychophysics, sighted individuals.

97 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

7.3.2 Introduction

In 2016, the World Health Organization reported 37 million blind individuals and 124 million individuals with visual impairments, and these numbers will increase in the upcoming years. Therefore, it is important to develop strategies to help restore vision or at least provide a sense of vision. Invasive techniques, such as artificial retinal prostheses (Luo and da Cruz, 2016) and gene therapies (Busskamp et al., 2010), are promising but nonetheless confront technical and accessibility issues and are further limited by their expense and currently low-resolution of sight restoration (Humayun et al., 2012). SSDs constitute a non-invasive, low-cost, and accessible complementary approach. The core tactic of SSDs is to compensate for the deficit of one sense by conveying information via another unimpaired sensory modality. The first evidence of the potential of SSDs for rehabilitation of blind individuals came from the seminal work of Paul Bach-y-Rita, which used tactile stimuli to deliver visual information (Bach-y- Rita et al., 1969). SSDs were later extended to visual-to-auditory sensory substitution in which pictures are transformed into auditory “soundscapes”(e.g. the vOICe (Meijer, 1992), the PSVA Prosthesis for Substitution of Vision by Audition(PSVA), (Capelle et al., 1998) and the EyeMusic (Abboud et al., 2014)).

Extensive research using visual-to-auditory SSDs has reported activation in specific brain areas that were thought to be dedicated exclusively to the treatment of visual information (Maidenbaum et al., 2014). Among these regions, the visual word form area has been reported for its role in the detection of letters (Levy-Tzedek et al., 2014a; Striem-Amit et al., 2012a). Similar findings were made for the detection of numbers in the visual number forma area (Abboud et al., 2015), the extrastriate body- selective area for the detection of bodies (Striem-Amit and Amedi, 2014) and the lateral occipital cortex for the detection of objects (Amedi et al., 2007). All these results provide evidences on the potential of SSD for visual rehabilitation of visually impaired individuals.

Despite the insights such studies have provided regarding the capacity for high-level visual functions to be achieved using SSDs, it remains unclear how to best train an individual to use the SSD itself. Moreover, for SSDs to be widely used, particularly in developing countries, it will be essential to devise training approaches that can be completed autonomously by the user (or at least with minimal assistance or tele-assistance). In this context, finding effective and intuitive way of training participants on auditory soundscape appears important. In more, training methods are crucial because if participants do not understand a concept correctly, it might ruin the global learning process. Prior studies reported their training methods. First, participants are introduced with the basis of functioning of the algorithm. Second, horizontal, and vertical lines are presented to the participants. These presentations of lines differ in their numbers, thickness and localization. Third, horizontal and vertical lines are grouped step by step to form angles. This is followed by the augmentation of the number of line and connections, so that participants start to hear letters. Finally, the training is complexified with the creation of shapes like triangles, squares, circles and so on with more and more complicated shapes. During the whole training process, the experimenter provides auditory feedbacks to the subjects and eventually answer the questions. (Abboud et al., 2014; Levy-Tzedek et al., 2014a).

Nevertheless, to the best of our knowledge, there exist no evidence concerning the extent to which training with one stimulus set extends to facilitate the discrimination of another, untrained stimulus set. This is a key aspect of perceptual learning, and is critical for the optimization of training protocols in

98 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. demonstrating the efficacy of SSDs for everyday use. The present study addressed this issue by training naive sighted individuals in the use of the EyeMusic SSD with specific soundscape stimulus sets designed to train participants on the vertical orientation. The training was performed with a visual feedback to avoid a potential bias present in one on one trainings when the experimenter gives auditory feedback. All participants were tested in their ability to discriminate a broader set of soundscapes that consist in the letter ‘E’ in the four cardinal orientations, prior to and immediately following training.

7.3.3 Material and Methods

7.3.3.a Participants

Analyses presented in this study are based on data from 22 participants (9 males, three left-handed female participants; aged 20-32 years, 24.09 years). All participants were graduate or post-graduate university students at the time of testing. No subject had history of or current neurological or psychiatric illness. All participants had normal or corrected-to normal vision and were tested for hearing loss with a Colson K20A audiometer.

7.3.3.b The EyeMusic SSD

The transformation algorithm is based on images that are 30 pixels high by 50 pixels wide. The conversion of images into auditory “soundscapes” follows three basic principles. The first principle is the sweeping of an image from left to right. The sweeping process occurs column after column. It is assimilated to a time component that is represented on the X-axis. Practically, a pixel on the left of the image will sound before a pixel on the right. The second principle is the fragmentation of each column into acoustic frequency. This fragmentation is represented in the Y-axis, such that the higher a pixel is located on the Y-axis, the higher in frequency the sound will be. The third principle is the volume of the sound. The volume is based on the brightness of a given pixel. The brighter the pixel is, the louder the sound will be. To recreate “soundscape”, all rows and columns are combined in a single auditory stimulus (Abboud et al., 2014a). In the case of the present experiment, we sub-divided the 30 vertical pixels into 3 equal sets of 10 pixels each, so that the stimuli were placed within High, Central and Low frequency ranges (Figure 1b).

Figure 1: depiction of stimuli. a. Displays an example of letters used to generate the auditory soundscapes. The upper left quadrant displays the E leftward and E rightward. The upper right quadrant displays and the E downward and E upward. The lower part of the scheme displays T and U letters. The other stimulus is displayed alone on the right-hand side of figure 1. b. Displays the grid’s

99 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

separation in the three-frequency range (Y-axis, in red) that were used for the creation of stimuli. The X-axis (green) represent the “time” component of the algorithm. Only the central time range between the two green bars was used for this experiment.

7.3.3.c Stimuli

The auditory stimuli were created based on four different shapes (i.e. ‘E’, ’T’, ’U’ and a meaningless shape) as well as horizontal and vertical lines. The shapes were designed in a 30x50 pixels black template in Microsoft® Paint software as follows:

I. The ‘E’ shape: formed by 2px thick white lines. The ‘E’ shape is 10px wide on 10px high, rotated in 4 cardinal orientations (leftward, rightward, upward and downward (Figure 1a)) and located at 3 different frequency range (high, central, low (Figure 1b)). In total 12 auditory ‘soundscapes’ of ‘E’ were created.

II. The ‘T’ shape: formed by 2px thick white lines. The ‘T’ shape is 10px wide on 10px high (i.e. equivalent to the ‘E’ shape without the two external lines), rotated in 2 cardinal orientations (upward, and downward (Figure 1a)) and located in the central frequency range. In total, 2 auditory ‘soundscapes’ of ‘T’ were created.

III. The ‘U’ shape: is composed of 2px thick white lines. The ‘U’ shape is 10px wide on 10px high (i.e. equivalent to the ‘E’ shape without the central line), rotated in 2 cardinal orientations (upward, and downward (Figure 1a)) and located in the central frequency range. In total 2 auditory ‘soundscapes’ of ‘U’ were created.

IV. The Lines: they are 2px thick and 10px long in white color. The line presentation was ‘simple line’ or ‘two parallel lines’ in the horizontal or vertical orientation. The lines were located in the central frequency range. In total 4 auditory ‘soundscapes’ of lines were created.

V. The other stimulus is an abstract shape that was designed to sound like the vertical ‘E’ shape (Figure 1a)). This shape was located at 3 different frequency range (high, central, low). In total 3 auditory ‘soundscapes’ were created.

All shapes were placed within the 10 central pixels of the horizontal axis (Figure 1b, green lines). Auditory ‘soundscape’ (16 bit mono; 44100 Hz digitization) were generated with the free online EyeMusic application (available at: http://amedilab.com/). The auditory lengths of soundscapes were composed as follow: All E, T, U letters and the meaningless stimulus had a total playing time of 500ms no matter their orientation. The ‘single horizontal’ line and 2 ‘horizontal parallel’ lines had a total playing time of 500ms. The ‘single’ vertical line had a total playing time of 100ms. The 2 ‘vertical parallel lines’ had a total playing time of 300 ms separated in 2 blocks of 100ms (heard) with a silence of 100ms in between.

100 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

7.3.3.d Task

The experiment was divided into three stages: A Pre-session (I.) in which participants were naïve to the EyeMusic SSD to ensure that auditory soundscapes were meaningless at the start of the experiment. A rapid training with visual feedback (II.) and a Post-session (III.) identical to the Pre-session except that participants were trained with the algorithm in the meantime (Figure2).

I.Pre-session: in this session, naïve participants to the EyeMusic heard auditory ‘soundscape’ corresponding to ‘E’ letters and the meaningless stimulus. The sequence of one trial was composed as follow: the participant was fixating a central cross displayed on an LCD computer monitor (20″ active TFT, 1600 x 1200 @ 60Hz, 16ms pixel response time). The disappearing of the fixation cross correspond to the start of a sound. After hearing a random soundscape from one of the 12 ‘E’ or one of the ‘3’ meaningless stimulus, participants had to make a five-alternative forced choice (5 AFC), with no limit of time. Buttons 1 to 4 corresponded to each ‘E’ shapes in one of the four cardinal orientations and button 5 was the other stimulus. A visual depiction of stimuli appeared to remind the participant with the 5 possibilities after each trial which remained as long as the participant did not answer. Stimulus delivery and behavioural responses collection were controlled by Psychopy free open-source software (Peirce and Peirce, 2009) and a serial response box™(Psychology Software Tools Inc., Pittsburgh, Pennsylvania, USA; http://www.pstnet.com/srbox). Participants were instructed to keep their eyes open through the entire session.

One block in the Pre-session was composed of 150 trials. Among the 150 trials, 120 trials were composed of ‘E’ letters (i.e. 10 ‘E’ shapes in each cardinal orientations (Figure 1a, upper part) presented at each of the three frequency range (Figure 1b)) and 30 trials were composed of the meaningless stimulus (i.e 10 at each frequency range (high, central, Low)). In total, each participant performed 3 blocks in the Pre-session.

II.Training session: For the training session, participants were randomly separated into two equal groups (n=11). A. The Control groups that had no explanation of the algorithm before the training. B. The Explained group that was instructed on the basics of transformations from the algorithm before the training. The explanation of the algorithm were that «the vertical axis represents the frequency of the sound, so that the higher a pixel is, the higher the frequency of the sound will be” and that « the algorithm reads a picture form left to right so that a pixel on the left will sound before a pixel on the right ». This explanation was performed with a small drawing of both axis and the presentation of three pixels within the axis. The experimenter did not answer participants’ questions in any of the groups. For both groups (Control and Explained) the training was as follow:

1. First a habituation block composed of horizontal and vertical lines to fix the basis of the algorithm transformation. Participants were hearing soundscape of lines that were presented in the central frequency. After hearing a soundscape, participants had to perform a 4-button forced choice with no limit of time. The four possibilities of answer were ‘simple horizontal line’, ’simple vertical line’,’2 horizontal parallel lines’ or ‘2 vertical parallel lines’. After each sound presentation, A visual depiction of stimuli appeared to remind the participant with the 4 possibilities. This panel remained as long as the participant did not answer. When the answer

101 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

was correct a green square was displayed, otherwise the correct shape was displayed. Each line stimuli were presented 5 times per block. All participants performed 1 block of line habituation.

2. After the habituation, participants were trained on the vertical orientations with auditory soundscapes of ‘T’ and ‘U’ shapes. Participants were hearing soundscape of ‘T’ and ‘U’ that were presented in the central frequency. After hearing a soundscape, participants had to perform 4 buttons forced choice with no limit of time. The four possibilities of answer were ‘T’ upward, ‘t’ downward, ‘U’ upward and ‘U’ downward (Figure 1b, lower part). After each sound presentation, A visual depiction of stimuli appeared to remind the participant with the 4 possibilities. This panel remained as long as the participant did not answer. When the answer was correct a green square was displayed, otherwise the correct shape was displayed. One block of vertical training was composed of 20 trials (i.e. Each stimulus was presented a total number of 5 times/block). All participants performed 5 blocks of training on vertical orientations.

The total training time was 13±2 minutes (mean± SEM) which consists of a fast training compared to the two to tens of hours usually reported with blind and sighted participants learning visual-to-auditory sensory substitution devices (Amedi et al., 2001; Levy-Tzedek et al., 2014b; Striem-Amit et al., 2012a, 2012b).

III. Post-Session: The post-session was identical to the pre-session except that participants were trained with the algorithm in the meantime. Again, at the start of this session, participants were reminded to keep their eyes open at all time.

As participants were trained with vertical ‘T’ and ‘U’ shapes and their responses for ‘E’ upward and ‘E’ downward were collapsed into the trained orientation compared to the ’E’ leftward and ‘E’ rightward that are the untrained orientation. Likewise, the frequencies were divided in two groups. The Central frequency corresponds to the central presentation of the ‘E’s (trained) and the peripheral frequencies correspond to the presentation of the ‘E’s in high and low frequencies (untrained) merged together. This terminology will remain for following analysis on the experimental sessions.

Figure 2. Schematic of the experimental process and the related tasks.

102 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

7.3.4 Results

7.3.4.a Training sessions

During the training session participants improved their performance in discriminating ‘T’ vs. ‘U’ soundscapes as well as their upward vs. downward orientation (Figure 3). This improvement was tested with a 2x5 mixed model repeated measures ANOVA with one between-subjects factor (Group: Control vs. Explained) and 1 within-subjects factor (training session 1-5). There was a significant main effect of

Training Session (F(1,20) = 43.79; p < 0.01, p2 =0.68). Neither the main effect of Group (F(1,20)= 2,234,p>0.05) nor the interaction was statistically reliable (F(4,80) = 1.08; p > 0.05). Consequently, data are displayed collapsed across groups in Figure 3. Post-hoc T-tests on the data across groups revealed a significant improvement between the first 2 training sessions (t(21)= -2.5357, p<0.05) as well as between training sessions 3 and 4 4 (t(21)= -2.65, p<0.05). These results suggest that explanation of the SSD algorithm was not a critical factor in participants improving in their ability to discriminate between two forms and their upward vs. downward orientation.

Figure 3. Results of training sessions. This figure displays the percentage correct response through the training sessions for Explained group and control group were

merged together. Post-hoc T-test revealed significant augmentation from training session 1 to training session 2 (t(21)= -2.54,

p<0.05) as wee as a second significant augmentation from training session 3 to training session 4 (t(21)= -2.65, p<0.05).

7.3.4.b Experimental Sessions

Before any analysis, we would like to remind the reader that participants were trained to discriminate vertical orientations of the letters ‘T’ and ‘U’. Therefore, their responses for ‘E’ upward and ‘E’ downward were collapsed into the trained orientation, whereas the ’E’ leftward and ‘E’ rightward were the untrained orientation. Likewise, the frequencies were divided in two groups. The Central frequency corresponds to the central presentation of the ‘E’s (trained) and the peripheral frequencies correspond to the presentation of the ‘E’s in high and low frequencies (untrained) merged together. This terminology will remain for following analysis on the experimental sessions. A four-way, mixed model repeated measure ANOVA with three within subjects factor 2 sessions (Pre/Post) x 2 frequencies (Central/Peripheral) x 2 orientations (Trained/Untrained) and one between subject factor 2 groups (Explained/Control) was performed. This analysis revealed a main effect of 2 Session (F(1,20)=14.58,p<0.01, ηp =0.42) that was explained by generally better performance post-

103 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

training than pre-training. There was also a significant Session × Orientation interaction (F(1,20)=6.18 2 2 ;p<0.05, ηp =0.24) and a significant Frequency × Orientation interaction (F(1,20)=14.90 ; p<0.01, ηp =0.43). A series of post-hoc paired t-tests was performed to better understand the bases for these interactions. First, tests were performed on the average data of the orientation conditions from the pre-session and post-session. Accuracy significantly improved from the pre-session to the post-session for the trained orientation (t(21)= -3.96,p<0.01) as well as for the untrained orientation (t(21)= - 2.78,p<0.05). There was no reliable accuracy difference between the trained and untrained orientations during the pre-training session (t(21)= 0.4023,p>0.05) , and accuracy was moreover near chance levels. By contrast, accuracy was significantly higher for the trained vs. untrained orientation during the post-training session (t(21)=2.35,p<0.05; Figure4b). Likewise, a series of post-hoc paired t- tests was performed on averaged data for the orientation (trained/untrained) and frequencies (central/peripheral). These tests show a significant difference between the trained orientation in central frequency Vs. peripheral frequencies (t(21)=-2.92;p<0.01). A significant difference between the trained Vs. untrained orientation in peripheral frequencies (t(21)=3.97,p<0.01) was also present (Figure 4b). This result show that participants are overall better for trained orientation independently from the frequency of presentation. Taken together, these results demonstrate a potential transfer across frequencies. Finally, A significant difference between untrained orientations in central frequency Vs. peripheral frequencies was also present (t(21)=3.53,p<0.01).

Figure 4. Results of the experimental sessions. a. displays the Mean±s.e.m of the percentage correct responses for the central and peripheral frequencies in Pre-session and Post-session. In Pre-session, both conditions were at chance level and did not statistically differ. T-tests revealed a significant difference between the trained orientation in Pre Vs. Post-session (t(21)= -3.96,p<0.01), a significant difference for the untrained orientation in Pre Vs. Post-session(t(21)= -2.78,p<0.05) and a significant difference between the trained orientation in Pre- session Vs. the untrained orientation in Post-session (t(21)=2.35,p>0.05). b. displays Mean±s.e.m of correct responses for the trained and untrained orientations in central and peripheral frequencies. T-tests show a significant difference for the trained orientation in central frequency Vs. trained orientations in peripheral frequencies (t(21)=-2.92;p<0.01), a significant difference for untrained orientations at central frequency Vs. untrained orientations at peripheral frequencies( t(21)=3.53,p<0.01) and a significant difference for the trained and untrained orientation at peripheral frequencies (t(21)=3.97,p<0.01).

104 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here.

7.3.5 Discussion

The present study provides insights on the training of visual-to-auditory sensory substitution in sighted individuals receiving a visual feedback during training. This study demonstrates the relative intuitiveness of the algorithm’s transformation. Second, it demonstrates the capacity of participants to augment their accuracy in discriminating the orientation of auditory soundscape of both trained and untrained orientations after training. In more, the trained orientation yielded better results independently of the pitch frequency in which stimuli were presented.

First, our participants successfully learned to discriminate soundscapes of T and U shapes presented at the central frequency with visual feedback. This was the case for both groups; i.e. those who had the EyeMusic algorithm explained to them and those who did not. The implication is that the transformation algorithm is relatively intuitive or otherwise readily learned; both of which bode well in terms of autonomous training. Second, during the training, participants improved significantly their performances in associating the auditory soundscape to the correct image. After training session 1 participants were about 59±4% correct, which shows the intuitiveness of the algorithm when participants hear relatively simple shapes (T and U) for the first time, albeit after rapid exposure to horizontal and vertical lines. The improvement in performances was punctuated by the presence of two plateau from training session 2 to training session 3 and from training session 4 to training session 5 in which participants finally reached 80±3% (16/20) of correct responses. These results indicate that participants seem to need a certain number of stimuli repetition to improve their percentages of responses. The plateau might reflect that after a fast understanding of the algorithm, the ability to extract more precise information requires more effort. The good performances reported above are reached after a rapid training of ~15 minutes compared to previously reported duration of training around 2 to 10 hours with sensory substitution devices in blind and sighted subjects (Levy-Tzedek et al., 2014a; Renier et al., 2005; Striem-Amit et al., 2012a). This is promising for the reduction of training time in futur development of training protocols. Finally, we would like to emphasis the difference between our training methods and previously reported methods. The main difference in the training method in this study is that participants only hear stimuli that are present in one central spatial location during training. In other words, the stimuli were not moved in different spatial location as it was previously reported (Abboud et al., 2014a). Another big difference is that participants were trained with ‘T’ and ‘U’ letters that were not constructed step by step but were presented directly constructed. Finally, after the training on orientation it the central frequency, participants had to discriminate the orientation of more complex shape that were presented in different spatial location in the post-session but never had feedback on their responses on these stimuli.

First, results of Pre Vs Post sessions demonstrate that learning the orientation of simple letters (i.e ‘T’ and ‘U’) in central frequency improved the categorization of more complex stimuli for both the trained and untrained orientations. A 2X2X2X2 repeated measure ANOVA with groups, orientation, session, and frequencies was performed and revealed a main effect of session. Both trained and untrained orientations were at chance level in the pre-session and augmented in the post session, albeit with a bigger augmentation for the trained orientation (Figure 4a). The augmentation of correct responses in the untrained orientation demonstrate that participants might operate a transfer of what they learned during training to more complex stimuli. Because participants yielded better results for the trained

105 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects. orientation over the untrained orientation in post-sessions, this suggests that the training of visual-to- auditory SSD should be focused on specific stimulus. However, the specific stimulus should present similarities with the object that the experimenter wants the participants to be able to discriminate at the end of the experiment (e.g ‘T’ and ‘U’ letters merged together create an ‘E’ letter).

A second important finding is that participants were overall better for the trained orientation irrespective of the frequency. As a reminder participants were trained on the vertical orientation with stimuli presented in the central frequency range. Our results demonstrate that participants have similar results for the trained and untrained orientation when the letter ‘E’ is presented in central frequency range, but when the letter ‘E’ is presented in peripheral frequencies, participants achieved a better discrimination for the trained orientation (Figure 4b). This indicates that participants can transfer across frequencies. We explain this finding by the fact that participants were indirectly trained on the vertical shifting of stimuli during the training. Indeed, the vertical shifting of stimuli rely on the principle of frequency pitch. As a reminder, the principle of frequency pitch states that “the higher a pixel is, the higher the sound will be”. We suggest that participants were trained on this principle during training, because the only way to discriminate between the upward and downward orientations of the letters T and U was to understand if there was a preponderance of high or low frequency in the respective auditory soundscapes (Figure 1, lower part). This interpretation is supported by results demonstrating that blind individuals who were trained on the orientation discrimination of letter ‘E’ yielded better results when the vertical bar was colored in red VS. white (the red color sounds differently from the white color) (Levy-Tzedek et al., 2014). This demonstrate the importance of localizing the longest bar of the letter ‘E’ to determine the orientation of the letter which is similar with the horizontal bars of ‘T’ and ‘U’ letters in our training. Importantly, this transfer in pitch demonstrate that it is not necessary to present stimuli over a wide array of locations during the training, which is in turn promising for reducing training time in future development of training protocols.

Based on these results, we would like to propose a different method to train participant with visual-to- auditory sensory substitution devices. To make the understanding of this process easier, we will make a parallel with the teaching of word pronunciation in children. The starting point of learning a new complicated word such as ‘Theoretical’ to a child is to decompose the word into syllables. In a second step, the syllables are grouped together to pronounce the word. Similarly, with the algorithm, individuals should learn what corresponds to horizontal and vertical lines (i.e. syllables). The second step is to learn how to discriminate the connections between them and what sound corresponds to parallel lines (i.e. the pronunciation of the word). These two steps were achieved during the training with first the presentation of lines and second, the discrimination of ‘T’ and ‘U’ soundscapes. Here the difference with what is classically done in SSD training, is that we propose to directly present letters instead of building them systematically by adding one line after the other to create the letter (alternatively a shape). On the basis of these letters, the participants learned the vertical orientation. Now that vertical orientation has been learned, to better understand the parallel between the orientation of letters ‘T’ and ‘U’ and orientation of the letter ‘E’, we will rely on the child who learned the word ‘Theoretical’ who now face a more complex word such as ‘Theoretically’ for the first time. To pronounce ‘Theoretically’ in a correct manner, he will rely on the pronunciation of the word ‘Theoretical’ that he previously learned because the two words share similarities (I.e the first syllables). By doing this, he has a great chance to pronounce the word ‘Theoretically’ well at his first attempt. It is identical with the algorithm. When participants

106 Anken Jacques Département des Neurosciences CliniquesError! Use the Home tab to apply Titre 1 to the text that you want to appear here. are hearing a complex shape such as an ‘E’ in the upward position (I.e equivalent to ‘Theoretically’) for the first time, they will rely on the sounds learned during the training (equivalent to ‘Theoretical’) because they share similarities. More precisely, the global sound that an ‘U’ upward makes is similar to the ‘E’ in upward position). Therefore, participants deduct that the orientation of the ‘E’ soundscape is either the trained orientation or another orientation. The very last step is relative to the frequency in which a stimulus is presented that appears intuitive. Again, if we rely on the pronunciation of the word ‘theoretical’, it remains the same independently of the voice pitch from the person who says it (Adult Vs. children). It appears to be similar with the algorithm in which the articulation of the ‘letter ‘E’ (i.e pronunciation) remains no matter the frequency range in which it is sounded.

We would like to add that even though our results support a training with specific stimuli, participants appear to be able to somehow acquire the capacity to discriminate ‘auditory objects’ that were not trained, reflected by the augmentation of correct response for the untrained stimuli in Post-session. This augmentation might just be here because participants understood what horizontal stimuli sound like by elimination. Therefore, they augment their results just by pressing at random between leftward and rightward propositions. Nevertheless, these augmentations in concert with the idea of indirect learning of the frequency pitch principle during training support a hypothetic feasibility of implicit learning that should be further investigated with more complex stimuli.

Finally, as our results demonstrate that participants can learn the algorithm with a visual feedback instead of an auditory feedback from the experimenter; we would to suggest training blind individuals with haptic stimuli in future studies for several reasons. First, previous studies already reported that the knowledge of an object shape is independent of the modality through which it is learned (Peelen et al., 2014). Therefore blind individuals might be able to learn shapes and their relative orientation with haptic stimuli. Second, haptic feedback would also be a means of controlling training and ensure that all subjects benefit from the same information during the training sessions. In more, controlled training (with haptic feedback) might help avoiding potential bias introduced by experimenter feedback in one- on-one training. Indeed, potential bias can come from the experimenter who adapts involuntarily his speech to the subjects and thus do not explain the stimuli similarly to all participants. In more the experimenter sometimes answers questions that participants have. This is also a source of bias.

7.3.6 Conclusion

To sum up, the present study proposes to train participants with stimuli presented in a chosen frequency range rather than on the whole frequencies and teach them the basis of transformation with vertical and horizontal line as it is usually performed. In a second step, we propose to train participants directly on simple shapes in the chosen frequency range. For this step, it is important to keep in mind to use shapes that share similarities with the final shape that participants should learn at the end of the training session (i.e try to decompose the final shape in multiple simpler shapes). Finally, after this training, participant can be presented with the final stimuli in different frequency ranges and should normally understand rapidly that the localization of stimuli in the grid is changing (indirect learning of the frequency pitch principle during training).

107 Brain dynamics of Illusory contour perception and perception via sensory substitution in healthy subjects.

7.3.7 References

Abboud, S., Hanassy, S., Levy-Tzedek, S., Maidenbaum, S., Amedi, A., 2014. EyeMusic: Introducing a “visual” colorful experience for the blind using auditory sensory substitution. Restor. Neurol. Neurosci. 32, 247–257. doi:10.3233/RNN-130338 Amedi, A., Malach, R., Hendler, T., Peled, S., Zohary, E., 2001. Visuo-haptic object-related activation in the ventral visual pathway. Nat. Neurosci. 4, 324–330. doi:10.1038/85201 Amedi, A., Stern, W.M., Camprodon, J.A., Bermpohl, F., Merabet, L., Rotman, S., Hemond, C., Meijer, P., Pascual-Leone, A., 2007. Shape conveyed by visual-to-auditory sensory substitution activates the lateral occipital complex. Nat. Neurosci. 10, 687–689. doi:10.1038/nn1912 Bach-y-Rita, P., Collins, C.C., Saunders, F.A., White, B., Scadden, L., 1969. Vision substitution by tactile image projection. Nature 221, 963–964. Busskamp, V., Duebel, J., Balya, D., Fradot, M., Viney, T.J., Siegert, S., Groner, A.C., Cabuy, E., Forster, V., Seeliger, M., Biel, M., Humphries, P., Paques, M., Mohand-Said, S., Trono, D., Deisseroth, K., Sahel, J.A., Picaud, S., Roska, B., 2010. Genetic Reactivation of Cone Photoreceptors Restores Visual Responses in Retinitis Pigmentosa. Science 329, 413–417. doi:10.1126/science.1190897 Capelle, C., Trullemans, C., Arno, P., Veraart, C., 1998. A real-time experimental prototype for enhancement of vision rehabilitation using auditory substitution. IEEE Trans. Biomed. Eng. 45, 1279–1293. doi:10.1109/10.720206 Humayun, M.S., Dorn, J.D., da Cruz, L., Dagnelie, G., Sahel, J.-A., Stanga, P.E., Cideciyan, A.V., Duncan, J.L., Eliott, D., Filley, E., Ho, A.C., Santos, A., Safran, A.B., Arditi, A., Del Priore, L.V., Greenberg, R.J., Argus II Study Group, 2012. Interim results from the international trial of Second Sight’s visual prosthesis. Ophthalmology 119, 779–788. doi:10.1016/j.ophtha.2011.09.028 Levy-Tzedek, S., Riemer, D., Amedi, A., 2014a. Color improves “visual” acuity via sound. Front. Neurosci. 8, 358. doi:10.3389/fnins.2014.00358 Levy-Tzedek, S., Riemer, D., Amedi, A., 2014b. Color improves “visual” acuity via sound. Front. Neurosci. 8, 358. doi:10.3389/fnins.2014.00358 Luo, Y.H.-L., da Cruz, L., 2016. The Argus(®) II Retinal Prosthesis System. Prog. Retin. Eye Res. 50, 89– 107. doi:10.1016/j.preteyeres.2015.09.003 Maidenbaum, S., Abboud, S., Amedi, A., 2014. Sensory substitution: Closing the gap between basic research and widespread practical visual rehabilitation. Neurosci. Biobehav. Rev., Multisensory integration, sensory substitution and visual rehabilitation 41, 3–15. doi:10.1016/j.neubiorev.2013.11.007 Meijer, P.B., 1992. An experimental system for auditory image representations. IEEE Trans. Biomed. Eng. 39, 112–121. doi:10.1109/10.121642 Peelen, M.V., He, C., Han, Z., Caramazza, A., Bi, Y., 2014. Nonvisual and visual object shape representations in occipitotemporal cortex: evidence from congenitally blind and sighted adults. J. Neurosci. Off. J. Soc. Neurosci. 34, 163–170. doi:10.1523/JNEUROSCI.1114-13.2014 Peirce, J.W., Peirce, J.W., 2009. Generating stimuli for neuroscience using PsychoPy. Front. Neuroinformatics 2, 10. doi:10.3389/neuro.11.010.2008 Striem-Amit, E., Amedi, A., 2014. Visual cortex extrastriate body-selective area activation in congenitally blind people “seeing” by using sounds. Curr. Biol. CB 24, 687–692. doi:10.1016/j.cub.2014.02.010 Striem-Amit, E., Cohen, L., Dehaene, S., Amedi, A., 2012a. Reading with sounds: sensory substitution selectively activates the visual word form area in the blind. Neuron 76, 640–652. doi:10.1016/j.neuron.2012.08.026 Striem-Amit, E., Guendelman, M., Amedi, A., 2012b. “Visual” acuity of the congenitally blind using visual-to-auditory sensory substitution. PloS One 7, e33136. doi:10.1371/journal.pone.0033136

108