Things and Places The Jean Nicod Lectures Franc¸ois Recanati, editor The Elm and the Expert: Mentalese and Its Semantics, Jerry A. Fodor (1994) Naturalizing the Mind, Fred Dretske (1995) Strong Feelings: Emotion, Addiction, and Human Behavior, Jon Elster (1999) Knowledge, Possibility, and Consciousness, John Perry (2001) Rationality in Action, John R. Searle (2001) Varieties of Meaning, Ruth Garrett Millikan (2004) Sweet Dreams: Philosophical Obstacles to a Science of Consciousness, Daniel C. Dennett (2005) Reliable Reasoning: Induction and Statistical Learning Theory, Gilbert Harman and Sanjeev Kulkarni (2007) Things and Places: How the Mind Connects with the World, Zenon W. Pylyshyn (2007) Things and Places How the Mind Connects with the World Zenon W. Pylyshyn A Bradford Book The MIT Press Cambridge, Massachusetts London, England ( 2007 Massachusetts Institute of Technology All rights reserved. No part of this book may be reproduced in any form by any elec- tronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher. MIT Press books may be purchased at special quantity discounts for business or sales promotional use. For information, please e-mail [email protected] or write to Special Sales Department, The MIT Press, 55 Hayward Street, Cambridge, MA 02142. This book was set in Stone Serif and Stone Sans on 3B2 by Asco Typesetters, Hong Kong, and was printed and bound in the United States of America. Library of Congress Cataloging-in-Publication Data Pylyshyn, Zenon W. Things and places : how the mind connects with the world / by Zenon W. Pylyshyn. p. cm.—(The Jean Nicod lectures) ‘‘A Bradford book.’’ Includes bibliographical references and indexes. ISBN 978-0-262-16245-6 (hardcover : alk. paper) 1. Visual perception. 2. Cognition. 3. Mental representation. I. Title. BF241.P953 2007 152.14—dc22 2006035507 10987654321 Contents Series Foreword vii Preface and Acknowledgments ix 1 Introduction to the Problem: Connecting Perception and the World 1 1.1 Background 1 1.2 What’s the Problem of Connecting the Mind with the World? Doesn’t Every Computational Theory of Vision Do That? 2 1.3 The Need for a Direct Way of Referring to Certain Individual Tokens in a Scene 10 1.4 Some Empirical Phenomena Illustrating the Role of Indexes 22 1.5 What Are We to Make of Such Empirical Demonstrations? 29 2 Indexing and Tracking Individuals 31 2.1 Individuating and Tracking 31 2.2 Indexes and Primitive Tracking 34 2.3 What Goes On in MOT? 37 2.4 Other Empirical and Theoretical Issues Surrounding MOT 40 2.5 The Infant’s Capacity for Individuating and Tracking Objects 49 2.6 Summary and Implications for the Foundations of Cognitive Science 52 3 Selection: The Key to Linking Representations and Things 59 3.1 Selection: The Role of Focal Attention 59 3.2 Selection and Demonstrative Reference: The Role of FINSTs 67 3.3 Problems with Selection by Location 79 3.4 Feature Placing and Sentience 91 3.5 What Do FINSTs Select? Some Consequences of the Present View 94 4 Conscious Contents and Nonconceptual Representation 99 4.1 Nonconceptual Representation and Perceptual Beliefs 99 4.2 The Role of Conscious Experience in the Study of Perception and Cognition 101 4.3 What Subjective Experience Reveals about Psychological Processes 112 vi Contents 4.4 The Phenomenal Experience of Seeing 119 4.5 The Phenomenal Experience of Mental Imagery 125 4.6 Does Phenomenal Appearance Correspond to a Level of Representation? 143 5 How We Represent Space: Internal versus External Constraints 147 5.1 What Does It Mean to Represent Space? 148 5.2 Internalizing General Spatial Constraints 151 5.3 Internalizing Spatial Properties by Mapping Them Onto an Inner Space 156 5.4 What Is Special about Representing Space? 169 5.5 Externalizing Spatial Properties: The Index Projection Hypothesis 179 5.6 Index Projection in Nonvisual Modalities 191 Conclusions 205 References 211 Index 235 Series Foreword The Jean Nicod Lectures are delivered annually in Paris by a leading philos- opher of mind or philosophically oriented cognitive scientist. The 1993 inaugural lectures marked the centenary of the birth of the French philoso- pher and logician Jean Nicod (1893–1931). The lectures are sponsored by the Centre National de la Recherche Scientifique (CNRS), in cooperation with the Ecole des Hautes Etudes en Sciences Sociales (EHESS) and the Ecole Normale Superieure (ENS). The series hosts the texts of the lectures or the monographs they inspire. Jean Nicod Committee Jacques Bouveresse, President Je´roˆme Dokic and Elisabeth Pacherie, Secretary Franc¸ois Recanati, Editor of the Series Daniel Andler Jean-Pierre Changeux Stanislas Dehaene Emmanuel Dupoux Jean-Gabriel Ganascia Pierre Jacob Philippe de Rouilhan Dan Sperber Preface and Acknowledgments This book is based in part on the Jean Nicod Lectures that I delivered in Paris in May–June 2004. The temporal gap between the lectures and the publication is not entirely due to my slow typing, but arose from the need to assimilate the rather wide range of publications that are relevant to the thesis I am presenting. The thesis, it turns out, is one that I have been gestating over many years, and hints of it occur in fragmentary form in a number of my publications. Many of these are reports of experimental work carried out with graduate students over the years, whose contribution is much appreciated. The thesis rests on a growing appreciation of an idea I first learned from David Marr, who refers to it as the principle of natural constraints. The mind has been tuned over its evolutionary history so that it carries out certain functions in a modular fashion, without regard for what an organism knows or believes or desires, but because it is in its nature, or as I more often put it, because of its architecture. So far this is an innocent enough idea that fits many different schools of psychology (and in fact is a familiar part of J. J. Gibson’s direct realism theory, though used there to very differ- ent ends). The particular constraint I am interested in here takes the form of a mechanism that allows the modular perceptual system to do things that many philosophers have said (correctly) can only be done by using the sophisticated machinery of concepts and the logical machinery of in- duction, deduction, and what Charles Sanders Peirce called abduction. The mechanism includes the capacity to select individual things in one’s field of view, to reidentify each of them under certain conditions as the same in- dividual thing that was seen before, and to keep track of their enduring individuality despite radical changes in their properties. I claim that so long as we are in the kind of world we live in there are mechanisms that allow the visual system to do these things without using the heavy equip- ment of concepts, identity, and tenses (which are needed for other tasks). x Preface and Acknowledgments For example, this is a world in which most surfaces that we see are sur- faces of physical objects, so that most of the texture elements we see move coherently as the object moves; almost all elements nearby on the proxi- mal image are at the same distance from the viewer; and, when objects disappear, they frequently reappear nearby, and often with a particular pat- tern of occlusion and disocclusion at the edges of the occluding opaque surfaces, and so on. Identifying things as ones we have seen before and keeping track of them as being the same individual objects over time is at the heart of my research, which has shown that we are very good at doing this in a way that does not use encoded properties (or the conceptual cate- gory) of the things that are tracked and reidentified. This mechanism is im- portant to us because if it were not for the existence of such nonconceptual processes, our concepts would not be grounded in experience and thus would not have the meaning that they do. I have proposed that the capacity to individuate and track several inde- pendently moving things is accomplished by a mechanism in the early vision module that I have called FINSTs (I call them ‘‘Fingers of INSTantia- tion’’ because they were initially viewed as a mechanism for instantiating or binding the arguments of visual predicates to objects in the world). This primitive nonconceptual mechanism functions to identify, reidentify, and track distal objects. It is an ability that we exercise every waking minute, and it has also been understood to be fundamental to the way we see and understand the world. I came upon these ideas in quite a different context, initially when I (along with my colleague Edward Elcock) attempted to develop a computer system for reasoning about diagrams, and later when I was carrying out experimental research on vision, visual attention, and mental imagery. This may seem like a circuitous route, but it has turned out that all these endeavors involve the same puzzles, which I later discovered were also the puzzles that preoccupy many philosophers: how concepts are grounded in experience; how we manage to encode and represent properties of the world when there are so many of them; why we feel that we are conscious of seeing an enormous number of things but are unable to report most of them, while at the same time a great deal of information of which we are not conscious can affect our behavior. These puzzles appear in their most striking form in discussions of two related problems: What are the proper- ties of mental images that allows them to function in thought, and how do certain kinds of thoughts—thoughts about spatial layouts—manage to dis- play properties very similar to those of perceived space? Preface and Acknowledgments xi In this book I examine a number of critical functions of early vision (the part of vision that is informationally encapsulated from the rest of the mind) in the light of the FINST mechanism.