Perceptions of Perceptual Symbols Science Evidence Implicates Sensory-Motor Regions of the Brain in Knowledge Representation (Sect
Total Page:16
File Type:pdf, Size:1020Kb
BEHAVIORAL AND BRAIN SCIENCES (1999) 22, 577–660 Printed in the United States of America Perceptual symbol systems Lawrence W. Barsalou Department of Psychology, Emory University, Atlanta, GA 30322 [email protected] userwww.service.emory.edu/~barsalou/ Abstract: Prior to the twentieth century, theories of knowledge were inherently perceptual. Since then, developments in logic, statis- tics, and programming languages have inspired amodal theories that rest on principles fundamentally different from those underlying perception. In addition, perceptual approaches have become widely viewed as untenable because they are assumed to implement record- ing systems, not conceptual systems. A perceptual theory of knowledge is developed here in the context of current cognitive science and neuroscience. During perceptual experience, association areas in the brain capture bottom-up patterns of activation in sensory-motor areas. Later, in a top-down manner, association areas partially reactivate sensory-motor areas to implement perceptual symbols. The stor- age and reactivation of perceptual symbols operates at the level of perceptual components – not at the level of holistic perceptual expe- riences. Through the use of selective attention, schematic representations of perceptual components are extracted from experience and stored in memory (e.g., individual memories of green, purr, hot). As memories of the same component become organized around a com- mon frame, they implement a simulator that produces limitless simulations of the component (e.g., simulations of purr). Not only do such simulators develop for aspects of sensory experience, they also develop for aspects of proprioception (e.g., lift, run) and introspec- tion (e.g., compare, memory, happy, hungry). Once established, these simulators implement a basic conceptual system that represents types, supports categorization, and produces categorical inferences. These simulators further support productivity, propositions, and ab- stract concepts, thereby implementing a fully functional conceptual system. Productivity results from integrating simulators combinato- rially and recursively to produce complex simulations. Propositions result from binding simulators to perceived individuals to represent type-token relations. Abstract concepts are grounded in complex simulations of combined physical and introspective events. Thus, a per- ceptual theory of knowledge can implement a fully functional conceptual system while avoiding problems associated with amodal sym- bol systems. Implications for cognition, neuroscience, evolution, development, and artificial intelligence are explored. Keywords: analogue processing; categories; concepts; frames; imagery; images; knowledge; perception; representation; sensory-motor representations; simulation; symbol grounding; symbol systems The habit of abstract pursuits makes learned men much infe- As Figure 1 illustrates, this view begins by assuming that rior to the average in the power of visualization, and much more perceptual states arise in sensory-motor systems. As dis- exclusively occupied with words in their “thinking.” cussed in more detail later (sect. 2.1), a perceptual state can Bertrand Russell (1919b) contain two components: an unconscious neural represen- 1. Introduction tation of physical input, and an optional conscious experi- ence. Once a perceptual state arises, a subset of it is ex- For the last several decades, the fields of cognition and per- tracted via selective attention and stored permanently in ception have diverged. Researchers in these two areas know ever less about each other’s work, and their discoveries have had diminishing influence on each other. In many universi- Lawrence Barsalou is Professor ties, researchers in these two areas are in different programs, of Psychology at Emory University, and sometimes in different departments, buildings, and uni- having previously held positions at versity divisions. One might conclude from this lack of con- the Georgia Institute of Technology tact that perception and cognition reflect independent or and the University of Chicago. He is the author of over 50 scientific pub- modular systems in the brain. Perceptual systems pick up in- lications in cognitive psychology and formation from the environment and pass it on to separate cognitive science. His research ad- systems that support the various cognitive functions, such as dresses the nature of human knowl- language, memory, and thought. I will argue that this view is edge and its roles in language, memory, and thought, fundamentally wrong. Instead, cognition is inherently per- focusing specifically on the perceptual bases of knowl- ceptual, sharing systems with perception at both the cogni- edge, the structural properties of concepts, the dynamic tive and the neural levels. I will further suggest that the di- representation of concepts, the construction of cate- vergence between cognition and perception reflects the gories to achieve goals, and the situated character of widespread assumption that cognitive representations are in- conceptualizations. He is a Fellow of the American Psy- herently nonperceptual, or what I will call amodal. chological Society and of the American Psychological Association, he served as Associate Editor for the Jour- 1.1. Grounding cognition in perception nal of Experimental Psychology: Learning, Memory, and Cognition, and he is currently on the governing In contrast to modern views, it is relatively straightforward board of the Cognitive Science Society. to imagine how cognition could be inherently perceptual. © 1999 Cambridge University Press 0140-525X/99 $12.50 577 Barsalou: Perceptual symbol systems of images. The goal of these attacks was not to exclude im- ages from mentalism, however, but to eliminate mentalism altogether. As a result, image-based theories of cognition disappeared with theories of cognition. 1.2. Amodal symbol systems Following the cognitive revolution in the mid-twentieth century, theorists developed radically new approaches to representation. In contrast to pre-twentieth century think- ing, modern cognitive scientists began working with repre- sentational schemes that were inherently nonperceptual. To a large extent, this shift reflected major developments Figure 1. The basic assumption underlying perceptual symbol outside cognitive science in logic, statistics, and computer systems: Subsets of perceptual states in sensory-motor systems are science. Formalisms such as predicate calculus, probability extracted and stored in long-term memory to function as symbols. theory, and programming languages became widely known As a result, the internal structure of these symbols is modal, and and inspired technical developments everywhere. In cog- they are analogically related to the perceptual states that produced nitive science, they inspired many new representational them. languages, most of which are still in widespread use today (e.g., feature lists, frames, schemata, semantic nets, proce- dural semantics, production systems, connectionism). long-term memory. On later retrievals, this perceptual These new representational schemes differed from ear- memory can function symbolically, standing for referents in lier ones in their relation to perception. Whereas earlier the world, and entering into symbol manipulation. As col- schemes assumed that cognitive representations utilize lections of perceptual symbols develop, they constitute the perceptual representations (Fig. 1), the newer schemes as- representations that underlie cognition. sumed that cognitive and perceptual representations con- Perceptual symbols are modal and analogical. They are stitute separate systems that work according to different modal because they are represented in the same systems as principles. Figure 2 illustrates this assumption. As in the the perceptual states that produced them. The neural sys- framework for perceptual symbol systems in Figure 1, per- tems that represent color in perception, for example, also ceptual states arise in sensory-motor systems. However, the represent the colors of objects in perceptual symbols, at next step differs critically. Rather than extracting a subset least to a significant extent. On this view, a common repre- of a perceptual state and storing it for later use as a symbol, sentational system underlies perception and cognition, not an amodal symbol system transduces a subset of a percep- independent systems. Because perceptual symbols are tual state into a completely new representation language modal, they are also analogical. The structure of a percep- that is inherently nonperceptual. tual symbol corresponds, at least somewhat, to the percep- As amodal symbols become transduced from perceptual tual state that produced it.1 states, they enter into larger representational structures, Given how reasonable this perceptually based view of such as feature lists, frames, schemata, semantic networks, cognition might seem, why has it not enjoyed widespread and production systems. These structures in turn constitute acceptance? Why is it not in serious contention as a theory a fully functional symbolic system with a combinatorial syn- of representation? Actually, this view dominated theories tax and semantics, which supports all of the higher cogni- of mind for most of recorded history. For more than tive functions, including memory, knowledge, language, 2,000 years, theorists viewed higher cognition as inherently and thought. For general treatments of this approach, see perceptual.