Overt Selection and Processing of Visual Stimuli
Total Page:16
File Type:pdf, Size:1020Kb
Overt Selection and Processing of Visual Stimuli Dissertation zur Erlangung des Grades “Ph.D. in Cognitive Science” im Fachbereich Humanwissenschaften der Universitat¨ Osnabruck¨ vorgelegt von Jose´ P. Ossandon´ D. Osnabruck,¨ im November 2015 Supervisor: Prof. Dr. Peter Konig¨ University of Osnabruck,¨ Osnabruck,¨ Germany Additional reviewers: Prof. Dr. Frank Jakel¨ University of Osnabruck,¨ Osnabruck,¨ Germany Dr. rer. nat. Tobias Heed University of Hamburg, Hamburg, Germany The human understanding when it has once adopted an opinion ... draws all things else to support and agree with it. And though there be a greater number and weight of instances to be found on the other side, yet these it either neglects and despises, or else by some distinction sets aside and rejects, in order that by this great and pernicious predetermination the authority of its former conclusions may remain inviolate. Novum Organum FRANCIS BACON Going beyond accepted data is not very different from filling gaps, except that interpola- tions are limited to the next accepted data point; extrapolations, however, have no end point in what is known or assumed, so extrapolations may be infinitely daring, and so may be dramatically wrong. Perception as hypothesis RICHARD L. GREGORY Abstract To see is to act. Most obviously, we are continuously scanning different regions of the world with the movement of our eyes, head, and body. These overt movements are intrinsically linked to two other, more subtle, actions: (1) the prior process of deciding where to look; and (2) the prediction of the sensory consequences of overt movements. In this thesis I describe a series of experiments that investigate different mechanisms underlying the first process and that evaluate the existence of the second one. The aiming of eye movements, or spatial visual selection, has traditionally been explained with either goal-oriented or stimulus-driven mechanisms. Our experiments deal with the tension of this dichotomy and present further evidence in favor of two other type of mechanisms, not usually considered: global orienting based on non-visual cues and viewing biases that are independent of stimulus and task. Firstly, we investigate whether stimulus-driven selection based on low-level features can operate independently of top-down constraints. If this is the case, the inhibition of areas higher in the hierarchy of visual processing and motor control should result in an increased influence of low-level features saliency. The results presented in Chapter 2 show that inhibition of the posterior parietal cortex in humans, by a permanent lesion or by transient inhibition, result in similar effects: an increased selection of locations that are characterized by higher contrast of low-level features. These results thus support a selection system in which stimulus-driven decisions are usually masked by top-down processes but can nevertheless operate independently of them. Secondly, we investigate how free-viewing selection can be guided by non-visual content. The work in Chapter 3 indicates that touch is not only an effective local spatial cue, but that, during free viewing, it can also be a powerful global orienting signal. This effect occurs always in an external frame of reference, that is, to the side where the stimulation occurred in the external world instead of being anchored to the side of the body that was stimulated. Thirdly, we investigate whether selection can operate even without reference to any sensory stimulus or goal. Results from our experiments presented in Chapters 2 to 5, demonstrate normal and pathological biases during free-viewing. First, patients with neglect syndrome show a strong bias to explore only the right side of images (ch. 2). In contrast, healthy subjects present a strong leftward bias, but only during the early phase of exploration (ch. 3 & 4). Finally, patients with Parkinson’s disease show a subtle overall bias to the right and no initial leftward bias (ch. 5). The results described so far indicate that visual selection operates based on diverse mechanisms which are not re- stricted to the evaluation of visual inputs according to top-down constraints. Instead, selection can be solely guided by the stimulus, which can be of a multimodal nature and result in global rather than local orienting, and by strong biases that are also independent of both stimuli and goals. The second part of this thesis studies the possibility that eye movements result in predictions of the inputs they are about to bring into sight. To investigate this with electroencephalography (EEG) we had to first learn how to deal with the strong electrical artifacts produced by eye movements. In Chapter 6, a taxonomy of such artifacts and the best way of removing them is described. On this basis, we studied trans-saccadic predictions of visual content, presented in Chapter 7. The results were compatible with the production of error signals after a saccade-contingent change of a target stimulus. These errors signals, coding the mismatch between trans-saccadic predictions and sensory inputs, depend on the reliability of pre-saccadic input. The violation of predictions about veridical input (presented outside the blind spot) results in stronger error signals than when the pre-saccadic stimulus is only inferred (presented inside the blind spot). Thus, these v results support the idea of active predictive coding, in which perception consists in the integration of predictions of future input with incoming sensory information. In conclusion, to see is to act: We actively explore the visual environment. We actively select which area to explore based on various competing factors. And we make predictions about the sensory consequences of our actions. vi Contents Abstract v 1 General Introduction 1 2 Unmasking the contribution of low-level features to the guidance of attention 23 3 Irrelevant tactile stimulation biases visual exploration in external coordinates 37 4 Spatial biases in viewing behavior 49 5 The influence of DBS in Parkinson’s patients free-viewing behavior 71 6 Combining EEG and eye tracking 85 7 Predictions of visual content across eye movements and their modulation by inferred information 113 8 General Discussion 127 Publication List 153 Contents (detailed) 156 Bibliography 157 vii Chapter 1 General Introduction By transducing light to neural activity, the eyes provide extensive information about the reflectance properties of surfaces and their spatial relations, information that is highly useful for multiple tasks like translation, foraging, avoidance of potential threats, and object manipulation. Besides their ecological relevance, eye movements are also the most frequent action performed by animals that is not by default an automatic process, thus arguably representing also the most frequent decision process carried out by the nervous system.Yet the underlying cognitive mechanisms and their perceptual impact are not well understood. Within the framework of this dissertation I will present and discuss a series of experiments that were driven by two main questions: (1) How do we decide where to look? and (2) What are the effects of ocular movements on visual processing? The first question originates in how visual information is normally sampled in animals with functional eyes. In most mammals, eyes are highly movable with respect to the head, through eye movements, and with respect to the body, through head movements. Humans perform various types of eye movements (see Fig. 1.1a-b), but for the purpose of the research discussed and presented here I will manly focus on saccades, which consist of fast ballistic rotations of the eyes and on fixations, which comprise the periods between saccades. During fixations, the eyes remain in a relative stable position, even though a fair amount of smaller movements, both passive and active, can still occur. Human saccades are executed via six small muscles within each eye orbit, which allow the re-orientation of the eyes that is necessary for evaluating areas of the surrounding environment not yet in the field of view, and the re-alignment of the area of the retina with the highest resolution, the central fovea, with the locations of the field of view that are important at a given moment. Since the seminal works of G. Buswell (111) and A. Yarbus (949), two of the earliest researchers that were able to measure ocular movements and gaze direction, we known that both the latency and the destination of eye movements are highly variable. As an example of this variability, figure 1.1c-d illustrates the viewing behavior of a large group of human subjects that were asked to look freely at static photographs on a screen. In these examples it is readily apparent what is common knowledge since the beginning of eye movements research, namely that exploration is not homogenous but clustered around regions of interest. Nevertheless there is a high variability among observers while they explore the same image (fig. 1.1c), within single observers in successive views of the same image, and depending on the specific instructions of what to look at (949). This variability arises from many different factors, such as the characteristics of the visual input, the subjects’ goals, the spatial and temporal context, the influence of other sensory modalities, and from systematic biases. These different factors have been intensely studied during the last century, and multiple functional mechanisms have been proposed for integrating them in a decision framework for visual selection. In the first section of this introduction, I will present a summarized view of some of these mechanisms, focusing specially on the ones that are the motivation of the empirical work presented in Chapters 2 to 5. The second question that guides this thesis is closely related to the first one, since it focuses on what are the conse- quences of eye movements on visual processing. Once light is transformed to neuronal activity in the retina, it continues to be processed in different unimodal visual brain areas, starting at the subcortical structures of the visual thalamus and superior colliculus, to then diverge, in successive stages, to a multitude of cortical areas.