THE BEHAVIORAL AND BRAIN SCIENCES (1979)2,535-581 Printed in the United States of America Stephen M. KossSyn Department of Psychology and Social Relations, Harvard University, Cambridge, Mass. 02138 Stewen Pinker Department of Psychology and Social Relations, Harvard University, Cambridge, Mass. 02138 George E_ Smith Department of Philosophy, Tufts University, Medford, Mass. 02155 Stewen P- Shwartz Department of Psychology, The Johns Hopkins University, Baltimore, Md. 21218 Abstract; What might a theory of mental imagery look like, and how might one begin formulating such a theory? These are the central questions addressed in the present paper. The first section outlines the general research direction taken here and provides an overview of the empirical foundations of our theory of image representation and processing. Four issues are considered in succession, and the relevant results of experiments are presented and discussed. The second section begins with a discussion of the proper form for a cognitive theory, and the distinction between a theory and a model is developed. Following this, the present theory and computer simulation model are introduced. This theory specifies the nature of the internal representations (data structures) and the processes that operate on them when one generates, inspects, or transforms mental images. In the third, concluding, section we consider three very different kinds of objections to the present research program, one hinging on the possibility of experimental artifacts in the data, and the others turning on metatheoretical commitments about the form of a cognitive theory. Finally, we discuss how one ought best to evaluate theories and models of the sort developed here. Keywords: computer simulation; imagery; memory; mental representation; perception; visual information processing A history of mental imagery would almost require a complete history study and a suitable explanatory construct in psychology. Rather, our of the idea of mental representation, so intimate is the relationship argument will consist of a demonstration that progress can in fact be between the two concepts. The objections to mental imagery have made in studying imagery scientifically. The second thrust of these traditionally been of two forms. First, it has been argued that objections against the use of imagery as an explanatory construct imagery cannot serve the functions that have been attributed to it. focuses on the claim that imagery is not a well-formed domain in its Most notably, it has been pointed out (at least since Berkeley's time) own right, but is merely one special aspect of a more general that an image cannot represent an object or scene uniquely without processing system (see Pylyshyn 1973). Again, if this were the case, some interpretive function that picks out certain characteristics of one would not expect to see much progress in attempts to develop a the image as being important and others as being incidental. That is, special theory of imagery. However, if a coherent theory that treats an image of John sitting could represent John, John's head, bent imagery as a distinct "mental organ," a theory having explanatory knees, and so forth, depending on what one pays attention to in the power and predictive utility, can be developed, this alone should image. And the "stage directions" indicating what is important in an make us hesitate to abandon the construct. In the course of describing image cannot themselves be images - if they were, the problem the theory and its development we will raise questions about how would only be pushed back a step. This class of objections is to the imagery - or any other mental structure or process - ought to be point: images cannot be the sole form of internal representation that studied and how theories of mental phenomena ought to be evalu- exists in human memory. But this does not mean that images cannot ated. be one form of representation in memory. This paper has three main sections. In the first we outline some The second class of objections historically leveled against the use of particulars of the approach to theory construction that is adopted mental imagery as an explanatory construct in psychology has two here and that has guided the research program since its inception. In thrusts: first, it has been claimed that there are incoherencies and addition, we present an overview of the empirical foundations of the inconsistencies inherent in the concept. Pylyshyn (1973) has recently theory, briefly describing four issues that we attempted to resolve summarized and developed these claims, and Kosslyn and Pomerantz empirically before beginning to construct a detailed model. In the (1977) have provided counterarguments. Not surprisingly, neither second section, we present the core theory itself and describe how it the arguments nor the counterarguments have been definitive, and has been instantiated in a computer simulation model. We will neither seems to have had enough force to sway most people from discuss not only the model itself, but the rationale for using a whatever position they found most congenial in the first place. In the computer simulation model per se. Finally, we conclude by consider- present paper we will not attempt to argue from purely rational ing a number of possible issues, problems, and objections surrounding grounds that mental imagery is a suitable topic for psychological the present program. © 1979 Cambridge University Press 0140-525X/79/KOPSS 029/$04.00 535 Kosslyn et al.: Demystifying imagery 1.0 Empirical foundations of the present theory information in a discursive way. Presumably, information implicit in long-term memory becomes explicit in an image (e.g., people claim The present research program had two phases. In the first, we that when asked which is higher off the ground, a horse's knees or the attempted to delimit empirically the class of acceptable models. We tip of its tail, the information becomes apparent only when they form began with a simple conception of how the imagery representation an image of the beast). Alternatively, images could be nonfunctional, system might operate. This conception hinged on the notion that epiphenomenal concomitants of more abstract unconscious process- visual images might be like displays produced on a cathode ray tube ing. On this view, images could simply be like the lights that flash on (CRT) by a computer program operating on stored data. That is, we the outside of a computer while it is adding; although they systemati- hypothesized that images are temporary spatial displays in active cally vary with the functioning of an information-processing mecha- memory that are generated from more abstract representations in nism, they take no part in the processing (see Kosslyn and Pomerantz long-term memory. Interpretive mechanisms (a "mind's eye") work 1977). None of the models of imagery based on artificial intelligence over ("look at") these internal displays and classify them in terms of research treats the images that people report experiencing as func- semantic categories (as would be involved in realizing that a particu- tional representations (see Baylor 1971; Farley 1974; Moran 1973; lar spatial configuration corresponds to a dog's ear, for example). Pylyshyn 1973; Simon 1972). Thus, this issue must be resolved before This simple "protomodel" was used as a heuristic to help construct a we can even begin to understand imagery. "decision tree" in which the nodes represented issues and the Four classes of experiments were performed to address the image- branches stood for alternative positions on the issues. Sets of experi- as-epiphenomenon view. These experiments were motivated ments were conducted to eliminate branches (as far as possible), primarily by the claim that experienced images "depict" informa- allowing us to descend to the next issue. The decision tree we tion in a spatial medium (in relation to the interpretive processes that ultimately formed is illustrated in Figure 1. operate on the image). If a representation depicts an object, then any Our CRT protomodel directed our attention to the following four part of that representation is a representation of the corresponding key issues: first, it suggested that the "quasi-pictorial" image we part of the object. For example, the rear portion of my image of a car experience is not an epiphenomenal concomitant of more abstract, is a representation of the rear portion of the car. This property is not nonpictorial processing; second, it led us to ask whether such images true of nondepictive representations. For example, "my" is part of are simply retrieved or can be generated; third, if images are "my car," but "my" is not part of the car itself (Ned Block, personal generated, we could then ask whether generation is simply a piece- communication, 1979). Because a quasi-pictorial image depicts, it meal retrieval of stored information, or whether it involves retrieving also has the following property: size and orientation of an object must organized units; last, we were faced with the question of whether be represented whenever a shape is represented; these properties are images are composed solely by retrieving encodings of how some- inextricably linked in the quasi-pictorial format. Thus, if images are thing appeared (the products of "seeing as"), or whether "descrip- in fact functional, then factors like spatial extent - which is inherent tive" information (such as the products of "seeing that") is also used. in the way visual images depict information - should affect informa- At the end of Phase I, then, we had a set of constraints on the viable tion processing when images are used. In contrast, if our spatial, data structures and processes of a theory of imagery. Let us now quasi-pictorial images are not functional, then their spatial properties briefly review the progress of empirical work in Phase I (see Kosslyn, (which do not characterize listlike linguistically-based representa- in press, for more detail).