Psychological Review Copyright 1985 by Ihe American Psychological Association, Inc. 1985, Vol. 92, No. 3, 350-371 0033-295X/85/S00.75

So You Think Gestures Are Nonverbal?

David McNeiil University of Chicago

In this article I argue that gestures and speech are parts of the same psychological structure and share a computational stage. The argument is based on the very close temporal, semantic, pragmatic, pathological, and developmental parallels between speech and referential and discourse-oriented gestures. Most of the article consists of a description of these parallels. A concept that unites outer speech and gesture is the hypothesis of inner speech.

Many cognitive psychologists hold that pany acts of speaking. In such gestures the overt acts of linguistic production are the hands function as symbols that are closely result of internal "computations." My aim in connected to the speech channel in terms of this article is to make the following point both time and semantic and pragmatic func- concerning gestures: Gestures share with tion. In the idiom of my title, such gestures speech a computational stage; they are, ac- are verbal. They are the overt products of the cordingly, parts of the same psychological same internal processes that produce the structure. The metaphor of a shared com- other overt product, speech. putational stage captures the processing as- Because there are such close connections pects of speech: that sentences and gestures between gesture and overt speech, gestures develop internally together as psychological offer themselves as a second channel of ob- performances. The metaphor of a common servation of the psychological activities that psychological structure captures the idea that take place during speech production—the speech and gesture respond to the same forces first channel being overt speech itself. The at the same times. channels of gesture and speech are close, yet Taking into account concurrent gestures different. Combining a spoken sentence and suggests that in the dynamic situation under- its concurrent gesture into a single observation lying sentence generation two opposite kinds gives two simultaneous views of the same of thinking, imagistic and syntactic, are co- process, an effect comparable to triangulation ordinated. The types of gestures that provide in vision. In this article I consider some of this insight are the referential and discourse- the implications that can be drawn from the oriented gestures that spontaneously accom- systematic comparison of speech and gesture. The statement that gestures and speech are

This article was written while I was a fellow at the parts of the same psychological structure is Netherlands Institute for Advanced Study, Wassenaar, contrary to the idea of body language, that The Netherlands. Financial support for my research was is, a separate system of body movement and provided by the National Science Foundation (linguistics postural signals that is thought to obey its section), the Spencer Foundation, and the University of own laws and convey its own typically affective Chicago. The gesture research was carried out with the help of the sharp eyes of Catherine Greeno, Mitsuko and unconscious meanings. It is also contrary Iriye, Elena Levy, Laura Fedelty, and Debra Stephens. to the assumptions of many linguistic analyses I have greatly benefited from comments at various that hold that language structures should be times and in various forms from friends and colleagues, analyzed only in terms of speech sounds plus among them E. Bates, J. Bybee, J. Cosnier, G. B. Flores d'Arcais, S. Frey, M. Johnson, D. Kelter, A. Kendon, J. grammar. We tend to consider linguistic what Lawler, W. J. M. Levelt, L. Mackenzie, N. B. McNeiil, we can write down, and nonlinguistic, every- N. Quinn, M. Silverstein, and W. Washabaugh. The title thing else; but this division is a cultural owes something to the late Glenn Gould. artifact, an arbitrary limitation derived from Requests for reprints should be sent to David McNeiil, a particular historical evolution. Both body Committee on Cognition and Communication, Depart- ment of Behavioral Sciences, University of Chicago, 5848 language and customary linguistic analyses South University Avenue, Chicago, Illinois 60637. follow a narrow approach, in which connected 350 SO YOU THINK GESTURES ARE NONVERBAL? 351 parts of a single psychological structure are is described here. Emblems are gestures that studied separately. Both begin by radically have a specific social code of their own, have separating gestures from speech, and from conventional paraphrases or names (Kendon, this starting point they go in opposite direc- 1983), are learned as separate symbols, and tions. That gesture and speech are parts of a can be used if they were spoken words; in single psychological structure has been a view many uses they are, in fact, unspoken words. held, however, by a number of psychologists An example is the okay sign or the psycho- and psychiatrists (Argyle, 1975; Barroso, logically peculiar sign (the first finger pointing Freedman, Grand, & van Meel (1978); Con- at the temple while the hand moves in a don & Ogston, 1971; Cosnier, 1982; Ekman small circle). These kinds of gesture are in- & Friesen, 1969;Kendon, 1972, 1980, 1983— terpretable in the absence of speech, and this to mention a few), but not by linguists (for a is one of their chief functions. The types of brief but quite definite opinion, see Chomsky, gesture described in this article, in contrast, in Rieber, 1983). In this article 1 present new are not interpretable in the absence of speech, evidence that supports the broader view, that are individual and spontaneous, are outside the whole of gesture and speech can be of any special social code that regulates them, encompassed in a unified conception, with and show interesting cross-cultural similarities gesture as part of the psychology of speaking, (again, in contrast to emblems, which, as along with, and not fundamentally different Morris et al. show, are highly specific to from, speech itself. particular linguistic and national groups and Acts of speaking are often accompanied in even subgroups). our culture by movements of the arms and In this article I first explain and present hands that are termed gestures (Kendon, arguments for the statement that gestures are 1972, uses the term gesticulation). These are manual symbols. Second, I exemplify the movements that (with a class of exceptions similarities that point to gesture and speech to be described) occur only during speech, sharing a computational stage. Third, I con- are synchronized with linguistic units, are sider and reply to counterarguments; and last, parallel in semantic and pragmatic function I present conclusions regarding the psycho- to the synchronized linguistic units, perform logical process of speech production based text functions like speech, dissolve like speech on considerations of gesture and speech data. in aphasia, and develop together with speech in children. Because of these similarities, a Gestures Are Manual Symbols strong case can be made for regarding gestures and speech as parts of a common psycholog- To demonstrate the symbolic character of ical structure. gestures, I will present examples of gestures The similarities hold for the kinds of spon- produced by 5 adult female subjects who taneous and semiconscious gesture that can were narrating the same event from a cartoon be seen accompanying much conversational story (they had been shown the cartoon just and narrative speech. Such gestures play pic- previously and were now telling the story to torial and discourse-related roles (Efron, 1941) a listener—each subject separately). The point and have been variously called "pictograms," I wish to make is that, like conventional "ideograms," "illustrators," and "batons" liguistic symbols, there is great commonality (Ekman & Friesen, 1969). among subjects. Despite individual variation There is another better known, more ste- in the detailed manner of performance, every reotypic type of gesture that should not be gesture includes upward movement. This confused with these. The French and Italians commonality is produced, not by a shared are acknowledged as master gesticulators, but code or language that exists specifically for the Italianate gesture (a type called the em- gestures, but by each speaker separately cre- blem by Ekman & Friesen, 1969, and docu- ating her own manual symbol of upward mented throughout Europe by Morris, Collett, movement. The concurrent sentences also Marsh, & O'Shaughnessy, 1979—see Ken- conveyed the idea of upward movement, and don's, 1981, review of the latter) is not the this fact shows that semantic parallels existed referential and discourse-oriented gesture that between gestures and concurrent speech. As 352 DAVID McNEILL they performed their gestures the subjects cluding their intended meanings as well as were saying1 their memories. Whether these concepts are categories, propositions, prototypical exam- he tries going up inside the drainpipe ples, contextualized individual ideas, or something else, is left unresolved, and undis- ("hand rises and"! cussed. [points upwardj The gestures above are symbols in Saus- he tries climbing up the drainspout of the sure's (1916/1959) sense. Even though they building are not conventional or arbitrary, these ges- tures are analyzable as paired signifiers and fhand rises "I signifieds. The physical upward movement of |_and starts to point upwardj the gesture itself is the signifier, and the and he goes up through the pipe this time associated memory of the character's upward movement, the signified concept. Considering fhand rises quickly just the gesture's upward movement in iso- and fingers open to lation from the speaker's memory of the (_form a basket character's upward movement, would destroy the symbol. Taking into account both sides this time he tries to go up inside the raingutter of the gesture-symbol allows us to see that the gestural and linguistic channels exhibited fhand steady and! ("hand rises quick-"! the same meanings. [pointing J |_ly (st'M pointing)J Further evidence that gestures are symbols in their own right comes from a phenomenon he tries climbing up the rain barrel that can be seen when the same hand config- [hand flexes back and up] uration and/or movement occurs twice in immediate succession. It can happen under Although a quantitative comparison is not this circumstance that there are two gestures realistic, there seem to be at least as much that utilize the same configuration and motion variation among the sentences of these ex- of the hand, but convey different meanings. amples as among the gestures; for example, In such cases there is added a small extra among the words "pipe," "spout," "gutter," and otherwise superfluous movement that and "barrel," the variation is not less than in separates the two meanings and indicates the the manner of moving the hands upward, moment that the meaning changes. The fol- pointing or not pointing, in a basket shape lowing example illustrates this movement: or with a backward flexion. and as he's coming up and the bowling Saussure (1916/1959) denned a symbol as a semiotic entity in which there are two ball's coming down distinguishable but inseparable sides—insep- Heft hand descends (= the bowling ball)! arable in the sense that considering one side [while right hand rises (= he) J in isolation from the other destroys the symbol (Saussure used the term sign, but in this [both hands loop out and back (= the article I use symbol, with the same meaning). extra movement)] The two sides of the symbol, Saussure called the signifier and the signified. The signifier is 1 The reader is urged to act out the gesture examples. the form of the symbol; in the case of a word, The experience of motor-speech synchrony is part of what the example is meant to demonstrate. Although for example, the form is the sequence of two streams of behavior are produced, the experience sounds or letters, /k x, t/ or c-a-t. The other should be of a single coordinated action. The left-most side is the concept that the symbol signifies, bracket that sets off the gesture description indicates the concept of a cat or cathood. Clearly, approximately when to start the gesture. Thus, in the much can be said about verbal concepts. In first example, the hand should begin to rise as you are saying the word "inside." Both the spoken and gestured this discussion the term concept is used to parts of the example should be performed in a continuous refer to subjects' mental representations, in- and natural manner, without pauses. SO YOU THINK GESTURES ARE NONVERBAL? 353

he swallows it full cognitive representation that the speaker had in mind, both the sentence and the left hand descends again (= still the gesture must be taken in account. (Examples bowling ball) of cooperation between gesture and speech while right hand rises and engulfs left are not difficult to find; several are cited in [ hand (= now a mouth) Kendon, 1984.) Even though the posture and motion of the It is important to note in this example the hands is the same in the second gesture, the existence of sentence and gesture synchrony. meaning of the right hand has changed: At Evidently the basis for synchronization is not first it stood for the whole character, but then that the gesture and sentence are translations became just a part of this character (his of one another, but that they arise from a mouth). Where the meaning shifted, the hands common cognitive representation, which nei- briefly withdrew and reentered the gesture ther exhibits completely. space. The extra movement implies that the For several reasons, therefore, it is apparent hands and their configuration had symbolic that gestures of the kind demonstrated are value for the speaker, and this value changed referential symbols. I have shown that (a) at the moment of the extra movement. gestures are similar between individual Another indication that gestures are true speakers when the meanings conveyed are symbols occurs when the speaker divides a similar; the similarities are no less than among cognitive representation into two parts. One the concurrent sentences, (b) When the hands part is carried in the familiar way via the change meaning, small extra movements mark sentence, while the other is conveyed through the boundary between one meaning and the the gesture. Speech and gesture cooperate to next, (c) Complex meanings are divided be- present a single cognitive representation. The tween the speech and gesture channels; speech gestures in these examples are not repairs. and gesture cooperate in these examples to The sentences are well formed and adequately present a single complex meaning. convey meanings. What seems to happen is that there is no grammatically convenient Evidence That Gestures and Speech Share a way to combine some aspects of the common Computational Stage cognitive representation with other aspects. In the example below ("she chases him out The principal argument of this article is again") the gesture conveys the idea of the that gestures and speech share a computa- instrument of the act, whereas the act itself tional stage (are parts of the same psycholog- is described in the concurrent sentence. The ical structure); they are connected internally. speaker could have conveyed the instrument This section presents evidence in support of with a more complex sentence, but by pro- this conclusion: (a) Gestures occur only dur- ducing a gesture the speaker avoided this ing speech, (b) they have semantic and prag- extra complexity. The gesture in such a case matic functions that parallel those of speech, is necessarily regarded as a symbol on an (c) they are synchronized with linguistic units equal footing with the sentence. in speech, (d) they dissolve together with she chases him out again speech in aphasia, and (e) they develop to- gether with speech in children. ["hand, gripping an~| object, swings Lfrom left to right J Gestures Occur Only During Speech

The sentence conveys the concepts of pursuit Gestures and speech overwhelmingly occur ("chases") and recurrence ("again"), but not together. We can understand this on the the means of pursuit. The gesture shows this ground that they are parallel products of a method—swinging an umbrella. The sentence common computational stage. Gestures by is well formed and the gesture is not a repair listeners also occur, but they are extremely or transformation of the sentence. To get the rare. In approximately 100 hours of recording, 354 DAVID McNEILL

during which time there were thousands of this container forth. These gestures occurred gestures by speakers, there was a gesture by during silence, but this kind of silence is part a listener only once (Stephens, 1983). Speakers of speaking. Speaking was temporarily trans- and listeners are experiencing the same lin- ferred to the person's gestures, which provided guistic forms and content, but the passive metalinguistic commentary on the process of comprehender role does not evoke gestural speaking itself. activity to anything like the same degree as the active producer role. One can speculate about the cause of this huge difference; for Gestures and Speech Have Parallel Semantic example, gesture production is part of a and Pragmatic Functions signal system used to extend the speaker's claim over the speaking turn (Duncan & The goal of this section is to demonstrate Fiske, 1977), or the motor arousal needed to that gestures are symbols equivalent to various produce speech generalizes and induces activ- linguistic units in meaning and function. The ity in other parts of the body. The point interpretation of such parallels is obvious, remains, nonetheless, that gesture production given the argument that gestures and speech and speaking are virtually restricted to the share a computational stage. During this same situations. stage semantic and pragmatic functions are More narrowly, the majority of gestures decided on, and both speech and gesture also occur during the speaker's actual speech perform these functions in parallel. I start articulation. About 90% of all gestures before with referential gestures (iconix and various clauses in a sample of six narrations occurred kinds of metaphorix), and end with gestures during active speech output. Of the 10% of that have purely off-propositional, including gestures that took place during silence, all discourse-oriented, functions (beats). were immediately followed by further speech. Iconic gestures. Several examples of se- In their own way these gestures of silence mantic parallels between gestures and lin- also indicate that speech and gesture share a guistic units have already been described. computational stage. The silence gestures did McNeill and Levy (1982) called gestures of not exhibit the content of the word or clause this type iconic. An iconic gesture is one that that was uttered subsequently. The break in in form and manner of execution exhibits a the internal computations that led to speech meaning relevant to the simultaneously ex- interruption changed the character of the pressed linguistic meaning. Iconic gestures gesture. The gestures at these moments were have a formal relation to the semantic content of two kinds. One was a simple and rapid of the linguistic unit. The signifier part of the hand movement of a type that usually accom- symbol is formed so as to present an image panies words whose importance depends on of the signified part. The five versions of the multisentence text relations (called beats); the going-up-the-pipe gesture were all iconic. other a type of gesture that itself symbolizes They were iconic because, for instance, when concepts about language and communication the person's hand rose upward while she was (called conduit gestures). These two kinds of saying "he crawls up the pipe," the gesture gestures are related to the breakdown of exhibited, via its own upward motion, the speech itself. They are gestures that symbolize signified concept of upward movement. the functions of silence in this situation. The McNeill and Levy (1982) performed an beat gesture served as a sort of metalinguistic analysis of gesture form in terms of 44 dif- index of the point of the breakdown, and ferent movement features. These specify details perhaps also as an attempt to get the speech of the gesture movement, such as whether process going again (adding a new dimension the gesture was made with one hand or two to the term beat). The conduit gesture, pre- hands; whether the palm was open or closed; senting a conceptualization of language as whether the fingers were extended, spread, or consisting of objects and containers, appeared curled; whether movement was upward, left, where the language process was temporarily or right; and so forth. McNeill and Levy also unable to provide the next such container, analyzed the linguistically conveyed meanings and perhaps was also part of the effort to call in terms of 38 meaning features, specifically, SO YOU THINK GESTURES ARE NONVERBAL? 355 the meanings of the verbs in the clauses that recreate and manipulate a virtual object, for the gestures accompanied. These features example, specify details of meaning, for example, he takes a crate whether the verb described horizontal motion, closure, or reaching an end state (see Miller fboth hands held apart, "I & Johnson-Laird, 1976). For example, "catch |_as if holding onto an object J up to" as used in cartoon narrations, has the The gesture recreates and manipulates a dis- meaning features of pursuit and horizontal course crate, and the hands play the part of motion (at least); it also has the meaning hands. In iconix2, the hands function more feature of end state (because you don't catch abstractly to depict an entity, also mentioned up to something unless you achieve the end in speech, other than hands, for example, state of reaching or becoming adjacent to it; go into Sylvester see Vendler, 1967). In contrast, "chase" in cartoon narrations has the meaning features Pleft hand encircles right- ~| of horizontal motion and pursuit, but not of hand space, then right hand end state. |_slides down left J McNeill and Levy (1982) prepared a 38 X The gesture shows an object being swallowed 44 matrix, in which, along the rows, one saw but does not depict hands. One might expect the frequency (in the gestures produced by that with children, iconix! would develop six narrators) of each movement feature ac- earlier than iconix , and I show in the section companying the meaning feature at that row. 2 Down the columns one saw the frequency on gesture development that this proves to be the case. with which each meaning feature co-occurred with the movement feature at that column. Not only are iconic gestures capable of referring, but they reveal discourse functions. Looking at this matrix (only part of which is In the following example from a male speaker reproduced in McNeill & Levy) it is obvious the gesture reveals the speaker's point of view that the patterns of peaks and valleys in each row and column are different. Every meaning toward the event he is narrating: feature has its own distinctive gesture profile and he bends it way back and, similarly, every gesture feature has its ["hand rises up, appears ~| own distinctive meaning profile. For example, when a verb lacked the meaning feature of to grasp something, and I end state (as in "chase") the gestures tended Lpull it back J to be performed with one hand, whereas This gesture iconically exhibits the event being when a verb possessed this meaning feature described in the clause, "he bends it way (as in "catch up to") the gestures tended to back." The object that was being bent back be two handed. In other words, the end state was a tree (as is known from the earlier meaning feature, in which there are logically narrative), and the gesture shows the hand of two component ideas—the idea of an end the character grasping this tree and bending state itself and that of something reaching it back. the end state—was exhibited in gestures that The gesture also reveals that the speaker also included two parts. This example illus- was "seeing" this event from the viewpoint trates the essence of gesture symbols that of the character who was performing the act. refer iconically: There is predictability of The speaker's hand symbolized the character's signifier form from the signified concept (in hand. If the viewpoint had been that of the part). tree (the other participant), we would have Stephens and Tuite (1983) proposed a sub- expected a gesture in which the arm moves division of iconix according to the degree of backward, but the hand—now representing alienation of the hands from their primary a tree rather than a hand—is not grasping function of object manipulation. Under this anything. proposal there would be a progression of The viewpoint of the character performing gesture types running from iconixi to iconix2 the action also appears in the concurrent to beats (defined later). In iconix,, the hands sentence. The agent of the act of bending 356 DAVID McNEILL back is functioning as the discourse reference phoric vehicle for the abstract concept of the point, or thematic topic, of the sentence cartoon. (Clancy, 1980; also see Kuno, 1976). It was In conversations between previously un- this agent's hand that the gesture depicted. If acquainted adult subjects recorded for a re- the viewpoint had been that of the tree search projectonface-to-faceinteraction(Dun- instead, one would have expected a passive can & Fiske, 1977), metaphoric gestures or other sequence form capable of indicating amounted to as much as half of all the that the tree is the discourse reference point gestures the participants produced. Often (e.g., "it got bent back"). there were no iconic gestures at all. In these Gestures that function specifically to em- conversations, that is, all of the gestures that phasize discourse relationships are described are referential could be metaphoric. under the heading Beats. The next three sections describe some of Metaphoric gestures. Another type of ref- the varieties of metaphoric gesture that can erential gesture, closely related to iconix, is be observed. termed metaphoric (McNeill & Levy, 1982). Mathematics gestures. Metaphoric ges- Gestures of this type demonstrate that refer- tures appear frequently in technical discus- ences with gestures are not limited to concrete sions where abstract content plays an impor- objects and events. Metaphorix are semanti- tant role. The following examples are taken cally parallel to sentences with abstract from a videotaped discussion between two meanings. The shared computational stage mathematicians. between speech and gesture, therefore, is one Verbal references to the concept of a math- that generally can be part of acts of linguistic ematical dual were accompanied by gestures production regardless of the level of concep- where the hands alternated between two po- tual abstraction. sitions. In a mathematical dual, one relation Metaphoric gestures are like iconic gestures is replaced by its converse relation. For ex- in that they exhibit a meaning relevant to the ample, the dual of "A is above B" is "A is concurrent linguistic meaning. However, the below B." The hand rotation thus exhibits a relation to the linguistic meaning is indirect. concrete image in which one position is re- Metaphoric gestures exhibit images of abstract placed by another: There are two hand posi- concepts. In form and manner of execution, tions and two relations, and there is replace- metaphoric gestures depict the vehicles of ment: metaphors (Richards, 1936). The metaphors are independently motivated on the basis of and this gives a complete duality cultural and linguistic knowledge. (Thus, the hand loops up and metaphoric gestures that accompany speech backwards through are a source of information about cultural a circle, ending beliefs and attitudes—in general, the world view of the speaker.) For example, a common _at a higher level metaphor in our culture (but not in others) The motion of the hand was synchronized is a bounded physical object for the concept with the verbal reference to the dual concept, of a mental or cultural product (e.g., human which itself of course also is a metaphor. knowledge, works of art, and memory; thus In other examples, gestures exhibited con- we speak of an artistic object), and there are cepts of limits. For instance, with the concept gestures that exhibit these metaphors. The of a direct limit there were gestures in which following is an example from a narration: the hand moved along a straight line followed it was one of the Tweetie Pie and Cat by end marking (a tensed stop). An example cartoons is the following:

fboth hands are lifted up 1 this gives a direct limit [_as if supporting an object J left index finger slides The speaker appeared to be holding onto an along right index finger object while he referred to a work of art. and comes to tensed stop This gesture-created object was the meta- _just beyond the tip SO YOU THINK GESTURES ARE NONVERBAL? 357

End marking and the barrier it implies pro- an inverse limit ... of... vides a metaphor for the concept of a limit, ["hand moves upward! and the straight line along which the hand and forward in moved, a metaphor for directness. This inter- pretation is confirmed by the gestures for |_straight line J inverse limits. Like direct limit gestures, in- (listener says: it's a direct limit) verse limit gestures also had end marking, but the movement, rather than along a straight I mean a direct limit line, followed a downward loop. That is, the ["hand makes ~| gesture showed the interchange of two points, same gesture an image of the concept of an inverse (in |_as before J which the start point and end point of a relation are exchanged). The following is an If gestures pass through a less complex example: transformation after the common computa- tional stage, it would not be by chance that it's an inverse limit the error here occurred in the speech channel and not in the gesture channel. With aphasics also, referential gestures exhibit contextually hand moves down then" appropriate meanings when there are para- upward in continuous phasias (see the Aphasia section). movement, and stops Conduit metaphors. This type of gesture _ at original level has been in use for at least 2,300 years. Montaigne (1958), explaining Zeno's attitudes I infer from metaphoric gestures that both toward the divisions of the soul, described a an abstract concept and a concrete image conduit metaphoric gesture—in fact, a graded (regarded by the speaker as similar to the series of them—in which Zeno conveyed the abstract concept) are part of the same com- concept of degrees of knowledge with degrees putation underlying the production of the of closure of the hand:2 sentence. Together with the abstract dual Zeno pictured in a gesture his conception of this division concept, in which one relation replaces an- of the faculties of the soul: the hand spread and open other, there was also an image of a spatial was appearance; the hand half shut and the fingers a position being replaced by another. Together little hooked, consent; the closed fist, comprehension; with the abstract limit concept there was also when with his left hand he closed his fist still tighter, an image of a moving object coming up to a knowledge, (p. 372) barrier. Whatever the internal computations Conduit gestures (including Zeno's) are are that underlie the production of sentences iconic depictions of abstract concepts of with abstract meanings, the evidence of met- meaning and language. Such metaphors are aphoric gestures suggests that they can be independently motivated by linguistic evi- highly similar to the computations that un- dence (Lakoff& Johnson, 1980;Reddy, 1979). derlie sentences whose meanings are concrete A conduit gesture exhibits a container or (Gildea & Glucksberg, 1983). substance (e.g., forming a cup shape), whereas Output errors sometimes appear in one simultaneously the linguistic channel refers channel but not in the other. This can occur to an abstract concept (an appearance, con- if the error is due to a disturbance that takes tent, comprehension, knowledge, language, place after the shared computational stage. meaning, art, the soul, etc.). The speech and In the mathematics discussion there was a context of speaking need not contain anything speech error of the semantic type (Garrett, about containers or substances. To describe 1976), in which the word inverse was used accurately a speaker's cognitive representation where direct was appropriate. The synchro- of an abstract concept, it is necessary to nized gesture was the gesture for a direct limit. This suggests that conceptual confusion was lacking, and the error occurred after the 11 am grateful to J. Stern of the University of Chicago, shared computational stage of gesture and Department of Philosophy, for bringing this wonderful speech. The following is the example: passage to my attention. 358 DAVID McNEILL include the concrete images implied by con- to this. First, this interpretation still takes the duit gestures. An example is the gesture gesture to be a metaphor for a concept that produced by the speaker cited earlier who is abstract. Second, this particular interpre- appeared to be holding onto an object while tation is also an instance of the conduit saying "it was a Tweetie Pie and Cat cartoon." metaphor. If the hand is ready to receive an The person's cognitive representation of the answer, then the answer is being regarded as work of art evidently included a container a substance. with the cartoon inside, or the cartoon itself Metaphoric gestures in other cultures. as a physical substance, in either case invoking Conduit gestures enable the speaker to act as the conduit metaphor. The computational if he could hold onto and manipulate abstract stage shared by speech and gesture con- concepts such as "knowledge," "art," "lan- tained—together with the abstract concept of guage," and "meaning." The manipulation of a work of art—the image of a concrete object, abstract concepts is not present in the gestures metaphorically interpreted as this abstract of some other cultures, even though there are concept. gestures in these cultures that present abstract In another example, a conduit metaphoric concepts in the form of concrete objects. gesture is part of a virtuosic image of the Specifically, conduit gestures are absent from concept of the past tense of the subjunctive a series of filmed conversations in Turkana, mood: a language spoken by nomadic herdsmen of the Turkana district of Kenya (see Dimmen- daal, 1983, for a description of this language). even though one might I examined more than 100 gestures recorded ("both hands form cups"! in an anthropological film about the Turkana [and spread wide apartj (MacDougall & MacDougall, 1977) and found no clear instances of gestures accompanying have supposed (no pause) verbal references to abstract concepts in which fcups vanish"! the hand appeared to more than briefly hold or manipulate a container or substance. The Labruptly J Turkana speakers produced numerous ges- tures in which abstract concepts were meta- The cup shapes could be images of the con- phorized as concrete entities or places. How- cept of meaning; the spreading apart, the ever, they did not seem to manipulate these idea of possibility (cf. an anything-is-possible entities in the way that North American gesture), whereas the abrupt closure of the speakers manipulate objects evoked by the cups of meaning, that what might have been, conduit gesture. Without far more knowledge in fact, did not obtain. Such is the essence of Turkana culture one cannot begin to in- of the past tense of the subjunctive mood terpret the gestures of these speakers. One (Curme, 1931). would at least have to know independently The hand shape seen in the above gesture something about their underlying cultural is also typical of conduit gestures for Wh models. It is an observable difference from questions—gestures that accompany, for ex- the behavior of North American speakers, ample, "what did you mean/say/think?" The nonetheless, that metaphoric gestures in palm is forward and held upright, while the which abstract concepts are "handled" are fingers extend and spread out, usually also essentially lacking. The following is an ex- curling: a metaphor for the question as an ample of what a Turkana speaker produces, object or container into which the listener is in a context where from a North American supposed to place an answer. One of the speaker there would very probably be a reviewers for this Journal suggested that there conduit gesture (cited by Stephens & Tuite, are other interpretations of the Wh-question 1983):3 gesture that do not rest on the conduit met- aphor. For instance, the opened hand could 5 Translations and transcriptions were kindly provided be interpreted as the questioner's readiness by R. Dyson-Hudson of the Cornell University, Depart- to receive the answer. There are several replies ment of Anthropology. SO YOU THINK GESTURES ARE NONVERBAL? 359

these Europeans want to extract all our knowledge—pft! toditarite ngitunga lu na kilna yoka —nith! (they-extract people) (this-here knowledge our-inclusive)

["hand moves to brow] [fingers create | object and appears to small object seems to [remove something J |_rising upward J fly away on its own

In this gesture an object was presented as word with its desired presence.) These situa- though it moves on its own, not requiring tions typify the circumstances in which beats manipulation. "Knowledge" was pulled from are likely to occur. The beat signals that what the speaker's brow (this much manipulation is important is an interitem relation. Such occurring), but then rose up and flew off on relations are, collectively, at off-propositional its own. Although there might be a universal levels (both supra- and subpropositional), in tendency of human thinking to embody ab- contrast to the propositional level of refer- stract concepts in images of concrete entities, ential (iconic and metaphoric) gestures. the metaphor of manipulation is by no means Lacking form, beats say that the material universal. they accompany is not part of the story line. Beats. In the discussion of speech-gesture Beats should occur at points of significant similarities so far (with the exception of the discontinuity in the discourse structure. "point-of-view" discussion), I have been de- McNeill and Levy (1982) found that sentences scribing parallels of prepositional content. classified as extranarrative were accompanied There are also gestures that demonstrate par- only by beats. Iconic gestures accompanied allels of pragmatic function. In the common only sentences classified as narrative. Beats computational stage of gestures and speech, also accompanied narrative sentences when decisions are made at several levels. Beats are the gestures performed the off-propositional gestures that emphasize off-propositional re- function of emphasizing individual lexical or lations. These gestures add another dimension sublexical items (syllables). Without excep- to the case I am making for gestures being tion, beats did not emphasize the proposi- part of the same psychological structures as tional content of sentences. An objective speech. method for identifying points at which dis- In contrast to referential gestures, beats (as course discontinuities occur has been devel- they were called by McNeill & Levy, 1982; oped by Levy (1984). batons by Ekman & Friesen, 1969) have no The definition of an iconic gesture (as prepositional content of their own. The ges- stated previously) is that it exhibits in its ture is an abstract visual indicator. Hence form and manner of production an aspect of beats are particularly appropriate for empha- meaning relevant to the meaning of the con- sizing discourse-oriented functions where the current linguistic item. This definition is se- importance of a linguistic item arises, not miotic in character, and refers to the mean- from its own prepositional content, but from ing-function of the gesture type. A beat is its relation to other linguistic items. Back- correspondingly denned as a gesture that does ground sentences in narrations are important not exhibit meaning iconically. There is, in because of their relations to other parts of addition to the semiotic definition, a formal the narration (the main story line), not for difference between iconix and beats. Iconic their own propositional content as such. Sim- gestures are typically large complex move- ilarly, lexical items that contrast with other ments that are performed relatively slowly lexical items (e.g., "it's Wednesday—no, I and carefully in the central gesture space. mean Thursday") are important because they Beats are typically small simple movements contrast with the other items. (Beats during that are performed more rapidly at or near silences implicitly contrast the absence of a the rest position of the hands. 360 DAVID McNEILL

These form differences themselves can be of a string of events like the one described. regarded as an iconic depiction of the dis- The prepositional content of the sentence tinction between the narrative and extranar- was that the character tries to catch the bird. rative functions. The display of prepositional However, the emphasized aspect of the sen- content in an iconic gesture takes place on tence was that it was providing background the central stage, whereas the extranarrative commentary. In keeping with this off-propo- commentary on this performance by a beat sitional role, the form of the gesture was emanates from the wings. unrelated to catching the bird or anything Moreover, referential gestures are prefer- else. The linguistic unit with which the gesture entially made with the dominant hand (Ste- was synchronized, "keeps" (on), coded the phens, 1983)—right or left, depending on the same discourse function. That the gesture handedness of the speaker—but beats are and this verb were synchronized further sup- made bimanually or with the subordinate ports the proposal that the gesture and word hand. Kimura (1973a, 1973b) observed that shared a computational stage. The following the right hands (of dextral subjects) were illustrates a beat that accompanied a linguistic engaged in gesture production during speech. repair—highlighting the successful completion In Stephen's study this was true of iconic and of a subpropositional element: metaphoric gestures, but not of beats. Alice li- Alice lives with her father who's a The connection of referential gesture pro- new uh runs a ... sort of a newsstand duction to the hand used for object manip- ulation is presumably not accidental because f fingers rise ~] these gestures often evoke images of objects and then and their manipulation. Beats do not evoke [_come down J this sort of imagery, and also are not prefer- entially performed with the hand used for The beat synchronized with "newsstand", indicating its contrast as a repair and the (at manipulating objects. Thus, beats occupy the end of the progression of gesture types that last) appropriate lexical choice. Summary. Gestures are semantically and are increasingly alienated from the primary pragmatically parallel to their concurrent lin- function of the hand for object manipulation guistic units in speech output. Metaphoric (iconix iconix , beats). 1( 2 gestures show that the parallels are not limited It is also possible that the choice of the to concrete references. Iconic and metaphoric hand in gesture production is influenced by gestures exhibit the prepositional content of the motor complexity of the gesture; the linguistic units. The form of the gesture is dominant hand is preferred for more complex determined by the form of the content being performances (Kimura & Archibald, 1974). presented. Beats emphasize elements that Motor complexity appears to be a minor code off-propositional functions and have an factor, however. Stephens (1983) observed abstract form that is unrelated to preposi- that the dominant hand was still preferred tional content. Together iconix-metaphorix for conduit metaphoric gestures in which (as and beats provide gesture coverage that par- in beats) there are not more than two move- allels linguistic functions ranging from the ment components. That is, the dominant referential uses of language to the discourse- hand was preferred for simple movements indexing uses. These parallels imply a shared when the gesture was referential. computational stage at which the semantic The following example is a typical beat and pragmatic values of sentences and ges- (right-handed speaker) performing a dis- tures are jointly worked out. course-oriented function:

he keeps trying to catch the bird Gestures Synchronize With Parallel left hand at rest position"! Linguistic Units rotates quickly outward [and back J Speakers tend to perform gestures at the same time they produce semantically and The statement comments on the structure of pragmatically parallel linguistic items. Iconic the story as a whole, namely, that it consisted and metaphoric gestures almost never cross SO YOU THINK GESTURES ARE NONVERBAL? 361 clause boundaries. Such synchronization sug- In the following, a gesture for the location of gests that the gesture reveals the moment at a character anticipated the verbal reference which the speaker's thinking process formu- to the location ("there"): lates the concept that the linguistic item they keep on flashing back to Alice just signifies. A particularly interesting form of synchronization is shown in the following Thand moves out and! example, in which there is artificial prolon- [points to location J gation of a gesture: sitting there and she dashes out of the house The speaker evidently formulated a mental dynamic ("static stroke:! representation of the location of Alice no stroke: hand keeps later than the moment of saying "flashing hand shows [position J back." This incident of thinking, however, "dashing anticipated by four words the verbal reference _ out" to the location. In general, two kinds of gestural anticipa- (no pauses) tion must be distinguished. On the one hand [retraction] there are gestures during silence in which no The gesture illustrated something "dashing computation of semantic or pragmatic func- out" (this was the dynamic stroke). The tion takes place. In this case, speech comes resulting posture—the hand extended out- to a halt and the gesture is either a beat or a ward and the fingers splayed (definitely a conduit metaphoric, which comments on the marked posture)—was held in midair until speech breakdown. On the other hand there the clause describing the same event came to is gestural anticipation that occurs during an end. uninterrupted speech output. In this case In this example the speech channel con- there is computation of a semantic and prag- trolled the timing of the gesture channel. A matic function, and the gesture exhibits the recent series of experiments by Levelt, Rich- product of this computation stage, but for ardson, and La Heij (in press) found evidence grammatical reasons the unpacking of the of the reverse direction of influence, deictic cognitive representation in the speech channel (pointing) gestures that control the timing of is delayed. The linguistic version of the se- speech. The speaker evidently can compare mantic-pragmatic representation necessarily the gesture and speech channels and confine occurs at the end of a grammatical constitu- symbols felt to be equivalent to the same ent, in this case, enforcing delay. span of time. The source of influence may To summarize, gestures synchronize with shift between the speech and gesture channels linguistic units that have the same semantic depending on the complexity of the speech and pragmatic functions. Synchronization transformation after the common computa- shows that the linguistic unit and gesture tional stage (the speech transformation in the belong to the same psychological moment experiment by Levelt et al. was relatively and could arise from a common psychological uncomplex). structure activated on-line during speech. That gestures reveal the moment at which the speaker formulates a concept is suggested Gestures Are Affected Like Speech in another way, in which—seemingly a par- in Aphasia adox—there is not synchronization. There exist anticipations where the concept revealed Gestures and speech are affected in parallel in the gesture becomes available before the ways by the neurological damage that pro- sentence can grammatically make use of the duces Broca's and Wernicke's types of linguistic item that signifies the concept. If aphasia. Broca's aphasia consists of a relatively we are to link the gesture and linguistic item, intact ability to use referring terms, together we must interpret the moment of the gesture with a radical disturbance of the ability to to be the moment of the shared computational combine these referring terms into larger stage, but the speech product of this stage grammatical wholes. The speech of these was stored and later retrieved from memory. patients is often characterized as telegraphic. 362 DAVID McNEILL

Broca's aphasics produce numerous and tion of pronoun compared to noun occur- sometimes quite elaborate iconic gestures, rences in their speech (Wepman, 1977). Delis, but few or no beats. This is our interpretation Foldi, Hamby, Gardner, and Zurif (1979) of the findings of Cicone, Wapner, Foldi, described for Wernicke-type aphasics hand Zurif, and Gardner (1979). (A review of movements at the start of new subordinate various studies that touch on the gesture clauses when the clauses were semantically production of aphasics can be found in Bates, discontinuous. These movements may have Bretherton, Shore, & McNew, 1983.) Pedelty been beats. Pedelty (1985) has observed with (1985), who has collected observations of the Wernicke-type aphasics gestures that in terms gesture production of Broca- and Wernicke- of form are classifiable as beats. There are type aphasic patients, confirmed that the also conduit metaphorix when the speaker Broca type produces iconix. Both iconix! and refers to his or her own speech problems. iconix2 occur. Beats also occur, but only These patients, in other words, produce ges- during silences when the speaker is attempting tures that do not relate to prepositional con- to retrieve words (a circumstance that in tent, and apparently these are the only gestures normal speakers is associated with subpro- they do produce. The dissolution of function positional emphasis). There are no beats that in Wernicke-type patients therefore also shows perform extranarrative functions. Thus, in parallels of gesture and speech ability. parallel with the ability to use referring lin- If gestures pass through a less complex guistic terms, Broca-type patients retain the transformation after the shared computational ability to create referential gestures, but have stage than does speech, it should be possible lost the ability to mark interrelations of items for the speaker's choice of words to be inap- in parallel with the dissolution of the ability propriate (a paraphasia), whereas a concurrent to combine linguistic symbols. Beats that gesture exhibits the appropriate meaning. Pe- contrast the absence of a referring term to delty (1985) has observed several such ex- its desired presence also occur, but these are amples. The reverse combination, a correct the only beat gestures. Because as movements word choice together with an inappropriate the two kinds of beat are identical and quite gesture, should be rare or nonexistent, and simple, their absence for performing one she has seen no examples of this type. The function (interrelating parts of text) and pres- occurrence of appropriate gestures implies ence for performing the other (word retrieval) that the paraphasias are not due to conceptual is quite striking. confusions, but arise from disturbances after In other words, the pattern of preservations the speaker has developed an appropriate and losses that follow brain injury in these cognitive representation. patients is quite similar in the speech and Thus, gestures and speech show an intimate gesture domains. For Broca aphasics, speech connection to one another in the special and gesture appear to share a computational situation where dissolution of linguistic ability stage. There is lacking from this stage, how- has taken place. There is simultaneous, and ever, the ability to interrelate symbols, and functionally parallel, dissolution of gesture this affects speech and gesture in parallel. ability. This linkage implies again that gesture A similar but complementary picture ap- and speech are parts of the same psychological pears with Wernicke-type aphasics. In this structure. The aphasic data add the further syndrome there appears to be disruption of suggestion that speech and gesture are me- the ability to form coherent semantic plans, diated by the same parts of the dominant but preservation of the ability to construct cerebral hemisphere for speech/gesture pro- sequences of words. The speech of these duction as would be implied by a common patients is often described as vacuous. In the computational stage. gestures of Wernicke-type patients there are few or no interpretable iconix, but there are Gestures Develop Parallel to Speech movements that seem to be beats. The theo- in Children retical possibility of beats is suggested by the Wernicke-type aphasic's ability to produce Although interpretations vary, a broad word sequences, and also by the high propor- characterization of the linguistic development SO YOU THINK GESTURES ARE NONVERBAL? 363 of children would show that it passes through words are used within the same scripts, that three overlapping stages: (a) an initial empha- is, interactive routines between the baby and sis on denoting concrete objects and situa- objects and people. So, for instance, most tions. This is possible with very simple babies have words for greeting and farewell expressions, single words and simple combi- like "hi" and "bye," and at the same time nations of words, provided there is a detailed have waving gestures that are used in carrying context of speaking shared by the child and out the same script. Babies who have verbal the others with whom the child interacts, as telephone routines ("hello," "speaking," etc.) invariably there is. This stage typically occu- also have telephone gestures (placing the pies the first 2 years and part of the third, phone to the ear etc.; see Bates et al., Table (b) Gains in flexibility, particularly in the 3.3 for numerous examples). construction of grammatically structured Pointing and reaching gestures are probably sentences for expressing relations (spatial, derived from movements in which real objects temporal, causal, interpersonal, etc.) between are manipulated (Carter, 1975). These and objects and objects and persons, make up the imitative performances seem to be the gestural second stage. At this stage there is the possi- counterparts of the initial stage of linguistic bility of linguistically sorting out the ways in development, where the emphasis is on de- which different semantic relations are coded noting concrete objects and situations. by the language, but not yet of using devices Early in the second stage, by 2'/z years, such as anaphora for coding the relations of iconic gestures are clearly present (McNeill, sentences to text. This stage occupies the 3rd, in press). In the following example from a 4th, and 5th years, approximately, (c) The 2'/2-year-old, an iconic gesture accompanies a emergence of text coding constitutes the third sentence (part of a cartoon narration) de- stage. This stage occupies the remainder of scribing a scene in which a character was primary language acquisition. shown rolling transversely across the visual The point of the above summary (which, field with a bowling ball inside him:

and it went away with that. in th-e-r-e arm moves right' turns completely in large sweep- around in chair ing movement and points to while hand _ room behind wobbles up and down

I hope, is broad and exceptionless enough to Notwithstanding the occurrence of iconic escape theoretical and factual disputes) is to gestures at an early age, the production of compare language development, that is, overt these gestures follows a developmental path speech development to gesture development. parallel to that of speech development. Werner That these pass through the same stages is and Kaplan (1963) described an increasing my final piece of evidence that speech and symbolicization of linguistic forms during gesture can be usefully regarded as parts of a language development that they term distanc- single psychological structure. ing. This refers to the relation between sig- The earliest referential gestures described nifiers and signifieds. Young children act as in the developmental literature are pointing if properties of the signifier must reproduce (deictic) gestures and various imitative per- properties of the signified. There is accordingly formances such as waving hello/goodbye (see little distance between them. In the speech Bates et al., 1983, for extensive review and realm, lack of distance can be shown with discussion). Bates et al. present a number of examples of onomatopoecism (Werner & convincing arguments that such gestures and Kaplan, 1963). For instance, a child learning the earliest spoken symbols develop together French said "boom-er" (with the French ver- as a single system. In particular, gestures and bal suffix -er) for the meaning of something 364 DAVID McNEILL falling down; a child learning German said feet, and iconic gestures incorporating the "whee-en" (with the German verbal suffix head and legs are far more common with little -en) for the meaning of going down a slide; children than with adults. Moreover, the ges- a child learning English said "wau-wau dog" ture space of young children is (in many cases, for dogs; and when children are asked to absolutely) larger than that of adults and, repeat a word such as chicken spoken by an more significantly, is centered on themselves. adult, they might say, rather, "peep-peep" or For adults the gesture space is a more or less some other such onomatopoecism. A signiner flattened disc in front of the body onto which has been formed that has a (phonetic) prop- images (gestures) are projected and gestures erty that reproduces a property of the signi- rarely cross the frontal plane of the body; but fied. The lack of distance between signifiers for children it is a sphere that includes the and signifieds shown in these examples also child and his or her own movements as the exists in children's early iconic gestures. center point. The above example again is an In the iconic gestures of adults the same illustration. The child totally turned around posture and movement of the hands can be in her chair to complete the gesture (the correlated with new references. As described gesture space was wrapped around her). earlier, when changes of meaning occur in Children's gestures, in short, appear to be otherwise identical gesture movements, the enactments. It is for this reason that iconix, gestures are accompanied by small extra precedes iconix?, developmentally. The entire movements that mark the point at which the body and all of its relevant parts reproduce meaning changes. A closed hand can, accord- the movements of the character whose acts ingly, one moment play the part of one are being described. Body parts of the char- character and the next moment, the part of acter tend to be played by the corresponding a different character. The gesture space can body parts of the child (and if played by the one moment be a windowsill, the next mo- hands, with movements added to simulate ment a street, and so forth. Within limits set the real thing), the gestures are large, like real by the need for symbols to remain iconic actions, and the gesture space, like the space (limits that are admittedly hard to state), the of real actions, centers on the child. From hands and gesture space of adults are free to this situation of minimal distance between have the meanings the speaker designates the gesture and the concept or memory that them to have. The iconic gestures of young it represents, the child develops increasingly children are less flexible. The limits on ac- symbolic gestures in which size, space, and cepting new meanings, however we state them, meaning are (comparatively) freely assigned. are less wide. The child's hands tend to play The development of symbolic gestures out of only the part of the character's hands. If they enactments thus parallels the increasing dis- have to play the part of something else, the tance in Werner and Kaplan's sense between child includes additional movements that ex- signifiers and signifieds that is appearing at hibit nonhand properties and differentiate the the same time in the linguistic domain. This gesture from a hand, for example, the wobbly parallel joins the others mentioned earlier to movements in the example that exhibited the imply that gestures and speech are parts of a rolling of the bowling ball and differentiated single psychological structure. Speech and this gesture depicting an entire character from gesture rise along the same developmental a gesture depicting a hand (adults perform curve. the corresponding gesture with an unper- Also showing the phenomenon of enact- turbed sweep). Children's gestures tend to be ment by children is the absence from their more detailed and less abstract, therefore, gestures of sharp boundaries corresponding than the corresponding gestures of adults for to meaning changes. A gesture exhibiting one the same references, and this effect follows meaning gradually turns into a gesture exhib- from the lesser distance children tolerate be- iting another meaning, just as one physical tween signiner and signified. motion leads by degrees to another physical Children prefer to make gestures showing motion. Adult gestures, in contrast, have dis- running, not with their hands, but with their tinct physical boundaries corresponding to SO YOU THINK GESTURES ARE NONVERBAL? 365 the changes of meaning between the gestures ture-speech parallel, therefore, which sup- (cf. the extra movements, described earlier, ports the argument that gestures and speech that adult speakers add to mark the division share a computational stage. Not until this between otherwise identical gesture move- stage includes decisions about off-proposi- ments). tional indexing, do beats arise from it. The phenomenon of enactment implies a Metaphoric gestures also appear for the quantitatively, if not qualitatively, different first time during the third ontogenetic stage. kind of thinking during the common com- The first examples are conduits. These can putational stage on the part of little children, be observed by 5 or 6 years. More original in which actions are mentally recreated. The metaphors do not appear until much later roots of this style of thinking in the sensori- (at the conclusion of the article I cite one of motor intelligence of early childhood can be the earliest, from a 9-year-old). Children's easily imagined (Piaget, 1954; for theoretical ability to interpret metaphoric usages is like- linkage of sensorimotor intelligence to gesture wise a late development (Winner, Rosenstiel, production, see McNeill, 1979). & Gardner, 1976). Thus, like other gesture Enactment appears not only in gesture types, the development of metaphoric gestures symbols, but also in the smaller signifier- parallels that of the corresponding linguistic signified distance of children's linguistic sym- ability in children. bols. Onomatopoecisms are acoustic enact- ments; "whee-en" acoustically enacts sliding Recapitulation down a slide, "wau-wau" enacts (to a child if not to a dog) what it is like to be a dog, To recapitulate: Speech and gesture appear "boom-er" enacts an object falling down, and together and perform a variety of semantic so forth. and pragmatic functions in parallel. Iconix The third stage of language development, emphasize the prepositional content-bearing text coding, is accompanied by the first oc- elements of speech, ranging from words to currences of beats. Although beats demand full clauses. Beats emphasize discourse and very little in the way of movement control, other interitem relations in which there is they are absent from the gesture performances discontinuity or contrast. There are parallels of very young children. Thus children's de- therefore at the prepositional and at several velopmental order is iconixi-iconix2-beats, off-propositional levels. These parallels sup- corresponding to the progression of increasing port the hypothesis that gestures and linguistic alienation of the hands from their primary items are parts of the same psychological function of object manipulation (Stephens & structure and share a computational stage. Tuite, 1983). The unity of speech and gesture is also dem- The earliest beat we have observed was onstrated by the synchronization of gestures produced by a 5-year-old and emphasized a with linguistic units, particularly when there subpropositional contrast in word meaning: is artificial prolongation of the gesture to execute is something else maintain synchrony, and by the ability of speakers to transfer linguistic functions be- fboth hands repeatedly ~| tween gesture and speech (a) during silences move up and down on I caused by the temporary failures in the com- putational stage that result in hesitations, and |_armrest of chair J (b) during fluent speech, when part of the This gesture (assuming it was one of the first) prepositional content of speech is exhibited occurred long after the first iconic gestures. gesturally and the rest conveyed verbally. Clearly such simple movements are possible Finally, the unity of gestures and speech is as motor performances in children much seen in aphasia, where speech and gesture younger than 5 years. The complexity of the dissolve together, and in children's develop- beat gesture is internal. That beats do not ment, where speech and gesture develop to- appear in speech development until there is gether through the same stages of increasing the beginning of text coding is another ges- symbolicization. 366 DAVID McNEILL

The next section of the article presents The gesture is a translation of the sentence counterarguments to the proposal that speech into a different medium. This counterargu- and gesture share a computational stage. The ment runs as follows. The visual-actional final section considers the shared computa- medium of the gesture has its own qualities, tional stage itself. What can be said about it but these do not characterize the psychological on the basis of observable gestural and lin- structure of the sentence. There are in fact guistic data? two psychological structures. The sentence medium has one characteristic set of qualities Counterarguments (and Answers) (linear and segmented), and the gesture me- dium has another set (global and synthetic In this section I discuss three counterar- or imagistic). When a gesture occurs, the guments that the reader might also have linear-segmented qualities of the sentence thought of. Replying to these will clarify the medium are translated into the imagistic concept of a computational stage shared by qualities of the gesture medium. This does speech and gesture. The counterarguments not mean that there is a shared computational are (a) gestures are like photographs held up stage in which both sets of qualities exist while speaking, (b) gestures are translations simultaneously. On the contrary, it implies of sentences into the manual-visual medium, that there is computation of the sentence, and (c) the shared computational stage of then a separate computation of the gestural sentence generation is a linear-segmented ver- translation. bal plan. One way of replying to this counterargu- Gestures are like photographs held up while ment is to show that the gesture medium speaking. Just as holding up a real photo- does not impose a characteristic structure of graph would have nothing to do with the its own, that instead it is a pliable medium process of sentence generation, so performing capable of assuming the same linear and a gesture could have nothing to do with it segmented qualities as the sentence medium. either. The first point in reply to this coun- If the gesture medium is free to code meanings terargument is that, unlike a photograph, a segmentally, the imagistic global-synthetic gesture is something the speaker him- or property of gestures cannot be explained as herself is creating on-line while speaking. a translation of the sentence medium into Second, the gesture is very closely connected the gesture medium. to the sentence temporally, semantically, and The following are two cases (among many pragmatically, all of which point to a coor- that could be cited) that demonstrate that the dination between the gesture and the sentence gesture medium is not inherently global and that is quite different from holding up a synthetic. In both, the medium normally (in photograph. Finally, when meaning is divided the first case) or spontaneously (in the second) between a gesture and sentence, it is a true provides a linear and segmented representa- division. It is not that the gesture is called tion. The first is American Sign Language out, as a photograph might be held up, to (ASL), a language in which segmentation and repair an otherwise interrupted communica- linearity are fundamental properties. Gestures tion. We do not observe, for example, in ASL divide events into idea segments, and so he ... sentences arrange these segments into linear successions (Klima & Bellugi, 1979). The [gesture to complete the idea] second case is the behavior of hearing subjects but rather, (naive to ASL), who are required to narrate stories without use of speech. Bloom (1979) so he chases her out again observed that under these conditions subjects spontaneously subdivide event meanings into [gesture to show the means] idea segments by making a different gesture These points reduce to one reply: Gesture for each segment. To convey the meaning of and speech are operations that have been an event as a whole, the subjects performed connected within. This is the sense in which gestures in linear, sentencelike successions. they share a computational stage. In view of such cases, one concludes that SO YOU THINK GESTURES ARE NONVERBAL? 367 the global and synthetic properties of gestures cedure in which speakers were repeating ver- are not inherent products of the manual- bal material from memory. Finally, if gestures visual medium. Rather, gestures with global undergo a less complex transformation than and synthetic properties can be interpreted does speech after the shared computational as evidence of an internal psychological struc- stage, the global-synthetic image can itself be ture that has imagistic (global-synthetic) regarded as the verbal plan at an early stage form, and this form co-exists in acts of of development. This is, in fact, how I describe speaking with a linear and segmented verbal the aspect of the internal psychological struc- representation. ture of sentences that synchronized gestures By global and synthetic I mean that the exhibit: the sentence in the early phase of its gesture's meaning is exhibited as a whole, internal development. not as a construction made out of separate meaningful components. A gesture that ex- Gestures and Inner Speech hibits the concept of upward motion does so in one indivisible symbol. The hand simul- Although it might be said by some that taneously depicts the character, its movement, gestures parallel linguistic output because they and the upward direction of movement, all are a response to language, the situation is, in one symbol. There are not separate signifier if anything, the reverse: Compared to the segments for the parts of the concept. (This concurrent spoken linguistic string, gestures more abstract concept of an image may avoid are more direct manifestations of the speaker's some, if not all, of the difficulties that attach ongoing thinking process in at least three to the metaphor of a picture or percept, which ways. First, gestures are immune to the errors in recent years has been the model idea for that affect speech, both errors of normal an image; e.g., Kosslyn, 1975—but see Pyly- speakers and of aphasic patients. Second, shyn, 1973, for a critique of the picture/ gestures exhibit properties of events that percept metaphor.) speech does not convey. Third, gestures an- The shared computational stage of sentence ticipate references expressed at later points generation is a linear-segmented verbal plan in speech because of grammatical constraints. from which the overt sentence is generated. In all of these cases the less complex trans- Spontaneous gestures, too, are generated from formation undergone by the gesture allows it the verbal plan: to reflect more faithfully the content of the shared computational stage (in the first two Speech cases) and the moment of occurrence of this stage (in the anticipation case). Covert Verbal Plan Utterances also exhibit a form of thinking, but in contrast to imagistic thinking, the Gesture thinking revealed in speech is linear and segmented. Thus the occurrence of gestures, There are four replies to this counterargu- along with speech, implies that during the ment. First, from this theory one cannot act of speaking two kinds of thinking, ima- explain the global-synthetic form of gestures. gistic and syntactic, are being coordinated. They should be linear-segmented if they are The concept of inner speech explains this produced from a covert linear-segmented ver- coordination. We assume that an image and bal plan. Second, from this theory we cannot inner speech symbol are generated at the explain the division of meaning between the same time. Inner speech symbols have syn- gesture and speech channels. If there is a tactic implications. These implications and meaning in the gesture channel, it must have the imagistic properties that the gesture ex- come from the covert verbal plan and should hibits are jointly present from the start, and also be present in the speech channel, and to can develop internally together. In the final the same degree. Third, where covert linear- outer speech and gesture streams, each form segmented verbal plans have been postulated of output should therefore display properties (e.g., Steinberg, Monsell, Knoll, & Wright, of the other, and this would occur because 1978) there has been an experimental pro- they have developed simultaneously from a 368 DAVID McNEILL

common base of inner speech. Examples If such words are interpreted as cognitive below demonstrate such an interpenetration tools, they illustrate what Vygotsky meant by of syntactic and imagistic forms of represen- the term psychological predication: thinking tation. in terms of the contextually novel. To judge The concept of inner speech refers to from the speaker's choice of referring terms, thinking that utilizes words and other lin- the gesture of the "finds" example was com- guistic symbols as cognitive tools (Vygotsky, parable. Reference to the agent was suppressed 1962, 1978). A cognitive tool crystallizes the and was made by the maximally presupposing mental operations that are connected with referring "term" ft, a form that altogether the symbol (Leont'ev, 1979), and inner speech rids inner speech of the psychological subject is thinking carried out by means of such (which in this case was also the grammatical mental operations. In the intended use of this subject). concept, the symbols of inner speech are The form of the speaker's thinking in this purely mental operations that do not neces- example can be inferred from the combina- sarily engage the movements of the speech tion of the mental operations attached to the articulators. cognitive tool "finds" and the image of the Because of imagery, the mental operations person leaning back and grasping an object crystallized in the word or larger item of with a round handle. Among the concepts inner speech can be carried out as if they associated with "finds" would be the presup- were about entities in space (cf. Pinker, 1984). position that the action is accidental. Thus, Words of inner speech are impregnated with the thinking expressed incompletely in the what Vygotsky called sense, a synthesis of clause, "and finds a big knife," included the several meanings into a single symbol and an idea of an accidental discovery taking place enrichment of the word's meaning. The sense while a particular spatial arrangement ob- of inner speech—this enrichment of the men- tained—the person groping around behind tal operations attached to the word—accord- her for a round-handled object (a referentially ing to the evidence of spontaneous gestures equivalent verb like grabs for would have consists, in part, of images. been a comment of a quite different kind). An example that comes close to the syntax Thus "finds," enriched with the image of of inner speech, as Vygotsky described it, is the complete event, was plausibly a mental the following: predication about the subject and an appro- priate cognitive tool for the speaker's remem- and finds a big knife bering and organizing his thoughts of the J U hand in grip shoots out event that had taken place. |_to rear and grasps "knife Can the form of thinking that we infer from gestures and synchronized linguistic The gesture showed (a) the act of picking up items be regarded as the sentence in the early the knife, (b) that it was picked up from the phases of its internal development? In this rear, and (c) that the cross-sectional shape of example a considerable part of the informa- the knife was round (the knife's handle). Yet tion required for the outer sentence was the gesture synchronized with the single word already present in what we reconstruct of finds, a word conventionally carrying none inner speech. Inner speech implied an agent of this information. Such gestures can be and object in the image of a hand reaching called holophrastic (McNeill, in press, dis- out to grasp something; an object, moreover, cusses holophrastic gestures of children), that small and round (a knife handle) in the shape is, a gesture that shows an association of a of the hand; and an agent and object related full clause meaning with a single word within transitively by the logic of the cognitive tool a grammatical clause. "finds." Thus the main structural and lexical In my study of children's holophrastic choices of "and 0 finds a knife" were already gestures, the words with which the gestures implied by the organization of the speaker's synchronized denoted the contextually novel ongoing thinking, when this utilized "finds" parts of situations as judged from the narra- as a symbol of inner speech with the sense tion and the child's choices of referring terms. exhibited in the gesture. SO YOU THINK GESTURES ARE NONVERBAL? 369

Not only are the main features of outer the fish, the moving object on the right is the speech forecast by inner speech, but the sons collectively setting out on the rescue surface form of outer speech might exhibit mission, and the lack of closure of the objects, the imagistic global-synthetic properties of the inaccessibility of Anansi inside the fish. inner speech. Global-synthetic properties do With this interpretation, the gesture can be not disappear when inner speech is converted seen as having metaphorized into a compact to outer, but reappear in a new linear-seg- concrete symbol all of the ideas that we mented form. My final example illustrates ponderously express with the words pursuit the preservation of global-synthetic properties and inaccessibility. in surface sentence structure (from a 9-year- The following are some of the global- old boy; but the point of the example does synthetic parallels that can be noted between not depend on the fact that the speaker is a the surface grammatical form of the sentence child): and the synchronized gesture: 1. Two hands appear in the gesture, and and they wanted to get where Anansi was two participants are mentioned in the sen- both hands held up and facing tence ("they" and "Anansi"). each other; the left hand is 2. The right hand (dominant for this motionless, while the right speaker) becomes the thematic reference point . is "fluttering" back and forth . of the sentence (correspondingly, the sons are referred to with a presupposing pronoun, (This sentence and gesture occurred during "they"). a narration of one of the Anansi stories— 3. The hands are held apart in the gesture, folktales of the Ashanti people of Ghana— and the references in the sentence to the two that the narrator had learned from a filmed participants are contained in different clauses presentation.) (Clause 1: "they wanted to get somewhere"; To reconstruct inner speech from this ges- Clause 2: "Anansi was somewhere"), syntactic ture and its synchronized speech, we would distance reproducing the spatial distance of say that the phrase "they wanted" is shown the gesture. (Mention of the participants by the gesture to have been the cognitive tool within a single noun phrase would have been on this occasion (the pronoun "they" refers possible; e.g., "Anansi and the sons couldn't to Anansi's six sons). The mental operations get together.") crystallized in this phrase, combined with the 4. The right hand is in motion, and the image that the gesture exhibits, provide a corresponding participants in the sentence form of thinking that could have retrieved are the subject of the verb "get," convey- and structured the boy's memory of the ing movement. (Alternatively, the fluttering event. The verb "wants" presupposes that the movement could represent the numerosity of action has not taken place. "Wants" also the participants, coded with a plural pro- permits the references of Anansi and the sons noun.) to be in separate clauses. Thus as a cognitive 5. The left hand is not in motion, and the tool "they wanted" provided mental opera- corresponding participant in the sentence is tions geared to the crux of the narrative contained in a locative stative phrase, "where situation, inaccessibility. These mental oper- Anansi was," emphasizing lack of movement. ations were utilized to think about inacces- 6. The two hands do not close on one sibility in the form of spatial separation. another, despite the movement of the right At the point in the narration where the hand, and the subject of the movement verb example occurred, the boy had already ex- is also the subject of "want", the use of which plained that Anansi had been swallowed by presupposes that the action denoted by "get" a fish and that the six sons were setting out has not taken place. The choice of this verb to rescue him. The gesture exhibited two also permitted the two-clause structure of the objects that stood apart, one of which was sentence. motionless, whereas the other fluttered back The surface form of the sentence, in other and forth ineffectually. If the nonmoving words, had many of the same properties as object on the left is Anansi imprisoned inside the global-synthetic representation of the ges- 370 DAVID McNEILL

ture. In addition to coding the intentions of chology: Human Perception and Performance, 4, 321- Anansi's sons, the surface sentence exhibited 329. Bates, E., Bretherton, I., Shore, C, & McNew, S. (1983). the meaning of pursuit plus inaccessibility. Names, gestures, and objects: Symbolization in infancy Rather than replacing the metaphoric image and aphasia. In K. E. Nelson (Ed.), Children's language of two nonclosing objects, the constituents of (Vol. 4, pp. 59-123). Hillsdale, NJ: Erlbaum. the sentence unpacked this image, the same Bloom, R. (1979). Language creation in the manual image presented in a new way. (Other exam- modality: A preliminary investigation. Unpublished undergraduate honors paper, University of Chicago, ples also show preservation of global-synthetic Chicago. properties in surface structure; e.g., an image Carter, A. L. (1975). The transformation of sensorimotor from the point of view of one who performs morphemes into words: A case study of the development the action was unpacked with a choice of of "more" and "mine." Journal of Child Language, 2, 233-250. active voice and pronominal agent in "and Cicone, N., Wapner, W., Foldi, N. S., Zurif, E., & he bends it way back.") Gardner, H. (1979). The relation between gesture and Thus an argument can be put forth that language in aphasic communication. Brain and Lan- gestures reveal a stage deep within the speak- guage, S, 324-349. ing process. At this level two kinds of thinking Clancy, P. M. (1980). Referential choice in English and Japanese narrative discourse. In W. L. Chafe (Ed.), are being coordinated: imagistic (thinking The pear stories: Cognitive, cultural, and linguistic that is global and synthetic) and syntactic aspects of narrative production (pp. 127-202). Norwood, (thinking that is linear and segmented). This NJ: Able*. dialectic is suggested by the concept of un- Condon, W. S., & Ogston, W. E. (1971). Speech and body motion synchrony in the speaker-hearer. In packing a global-synthetic representation with D. J- Horton & J. J. Jenkins (Eds.), Perception and a linear-segmented string that exhibits the language (pp. 150-173). Columbus, OH: Merrill. same global-synthetic sense. An expression Cosnier, J. (1982). Communications et langages gestuels of the dialectic is that there is no system [Communication and gesture language]. In J. Cosnier, break between thinking and speaking. Gram- A. Berrendonner, J. Coulon, & C. Orecchioni (Eds.), Les voies du langage. Communications verbales ges- matical features exist in thought (agentivity, tuelles et animates (pp. 255-304). Paris: Dunod. transitivity), and thought features exist in Curme, G. (1931). Syntax. New York: Heath. grammar (a global-synthetic image in the Delis, D., Foldi, N. S., Hamby, S., Gardner, H., & Zurif, surface form of the sentence). If this reveals E. (1979). A note on temporal relations between language and gestures. Brain and Language, 8, 350- a language of thought, this language is much 354. more imagistic than Fodor (1975), the author Dimmendaal, G. J. (1983). The Turkana language. Dor- of this evocative phrase, believed. Yet it is drecht, Holland: Foris. capable of explaining the impression of ease Duncan, S. D., & Fiske, D. W. (1977). Face-to-face and speed of action that one has of normal interaction: Research, methods, and theory. Hillsdale, NJ: Erlbaum. speech generation. For related and supporting Efron, D. (1941). Gesture and environment: A tentative arguments, see Kendon (1983; 1984). study of some of the spatio-temporal and "linguistic" I finally can say why gestures bear the aspects of the gestural behavior of Eastern Jews and same relation to the verbal process of speaking Southern Italians in New York City, living under similar us well as different environmental conditions. Morn- as does the speech output itself. Gestures and ingside Heights, NY: King's Crown Press. speech share a computational stage that in- Ekman, P., & Friesen, W. V. (1969). The repertoire of cludes the operations of inner speech—think- nonverbal behavioral categories—origins, usage, and ing utilizing verbal symbols as cognitive coding. Semiotica, I, 49-98. Fodor, J. A. (1975). The language of thought. Cambridge, tools—and this is also the initial phase of MA: Harvard University Press. sentence generation. Such is one answer to Garrett, M. F. (1976). Syntactic processes in sentence the hypothetical skeptic addressed in my title. production. In R. J. Wales & E. Walker (Eds.), New approaches to language mechanisms (pp. 231-256). Amsterdam: North-Holland. Gildea, P., & Glucksberg, S. (1983). On understanding References metaphor: The role of context. Journal of Verbal Learning and Verbal Behavior, 22, 577-590. Argyle, M. (1975). Bodily communication. New York: Kendon, A. (1972). Some relationships between body International Universities Press. motion and speech. In A. Siegman & B. Pope (Eds.), Barroso, F, Freedman, N., Grand, S., & van Meel, J. Studies in dyadic communication (pp. 177-210). New (1978). Evocation of two types of hand movements in York: Pergamon Press. information processing. Journal of Experimental Psy- Kendon, A. (1980). Gesticulation and speech: Two aspects SO YOU THINK GESTURES ARE NONVERBAL? 371

of the process of utterance. In M. R. Key (Ed.), (1979). Gestures, their origins and distribution. New Relationship of verbal and nonverbal communication York: Stein & Day. (pp. 207-227). The Hague: Mouton. Pedelty, L. (1985). Gestures in aphasia. Unpublished Kendon, A. (1981). Geography of gesture. Semiotica, '37, doctoral dissertation, University of Chicago, Chicago. 129-163. Piaget, J. (1954). The construction of reality in the child Kendon, A. (1983). Gesture and speech: How they interact. (M. Cook, Trans.). New York: Basic Books. In J. M. Wiemann & R. P. Harrison (Eds.), Nonverbal Pinker, S. (1984). Visual cognition: An introduction. interaction (pp. 13-45). Beverly Hills, CA: Sage Pub- Cognition, IS, 1-63. lications. Pylyshyn, Z. (1973). What the mind's eye tells the mind's Kendon, A. (1984). Some uses of gesture. In D. Tannen brain: A critique of mental imagery. Psychological & M. Saville-Troike (Eds.), Perspectives on silence (pp. Bulletin, 80. 1-24. 215-234). Norwood, NJ: Ablex. Reddy, M. (1979). The conduit metaphor—a case of Kimura, D. (1973a). Manual activity during speaking: I. frame conflict in our language about language. In A. Right handers. Neuropsychologia, 11, 45-50. Ortony (Ed.), Metaphor and thought (pp. 284-324). Kimura, D. (I973b). Manual activity during speaking: New York: Cambridge University Press. II. Left handers. Neuropsychologia, 11, 51-55. Richards, I. A. (1936). The philosophy of rhetoric. New Kimura, D., & Archibald, Y. (1974). Motor functions of York: Oxford University Press. the left hemisphere. Brain. 97. 337-350. Richer, R. W. (1983). Dialogues on the psychology of Klima, E., & Bellugi, U. (1979). The signs of language. language and thought. New York: Plenum Press. Cambridge, MA: Harvard University Press. Saussure, F. de (1959). Course in general linguistics (W. Kosslyn, S. M. (1975). Information representation in Baskin, Trans.). New York: Philosophical Library. visual images. Cognitive Psychology, 7, 341-370. (Original work published 1916). Kuno, S. (1976). Subject, theme, and the speaker's em- Stephens, D. (1983). Hemispheric language dominance pathy—A re-examination of relativization phenomena. and gesture hand preference. Unpublished doctoral In C. N. Li (Ed.), Subject and topic (pp. 417-444). dissertation, University of Chicago, Chicago. New York: Academic Press. Stephens, D., & Tuite, K. (1983, October). The herme- Lakoff, G., & Johnson, M. (1980). Metaphors we live by. neutics of gesture. Paper presented at the symposium Chicago: University of Chicago Press. on Gesture at the meeting of the American Anthro- Leont'ev, A. N. (1979). The problem of activity in pological Association, Chicago. psychology. In J. V. Wertsch (Ed. and Trans.), The Sternberg, S., Monsell, S., Knoll, R. L., & Wright, C. E. concept of activity in Soviet psychology (pp. 37-71). (1978). The latency and duration of rapid movement Armonk, NY: Sharpe. sequences: Comparisons of speech and typewriting. In Levelt, W. J. M., Richardson, G., & La Heij, W. (in G. E. Stelmach (Ed.), Information processing in motor press). Pointing and voicing in deictic expressions. control and learning (pp. 118-152). New York: Aca- Journal of Verbal Learning and Verbal Behavior. demic Press. Levy, E. T. (1984). Communicating thematic structure in Vendler, Z. (1967). Linguistics in philosophy. Ithaca, NY: narrative discourse: The use of referring terms and Cornell University Press. gestures. Unpublished doctoral dissertation, University Vygotsky, L. S. (1962). Thought and language (E. Hanf- of Chicago, Chicago. mann & G. Vakar, Trans.). Cambridge, MA: MIT MacDougall, D., & MacDougall, J. (Producers and Di- Press. rectors). (1977). Lorang's way [Film]. Berkeley: Uni- Vygotsky, L. S. (1978). Mind in society: The development versity of California Extension Media Center. of higher psychological processes (M. Cole, V. John- McNeill, D. (1979). The conceptual basis of language. Steiner, S. Scribner, & E. Souberman, Eds.) Cambridge, Hfflsdale, NJ: Erlbaum. MA: Harvard University Press. McNeill, D. (in press). Iconic gestures of children and Wepman, J. M. (1977). Aphasia: Language without adults. Semiotica. thought or thought without language? ASHA, 18, 131- McNeill, D., & Levy, E. (1982). Conceptual representa- 138. tions in language activity and gesture. In R. Jarvella Werner, H., & Kaplan, B. (1963). Symbol formation: An & W. Klein (Eds.), Speech, place, and action: Studies organismic-developmenlal approach to language and in deixis and related topics (pp. 271-295). Chichester, the expression of thought. New York: Wiley. England: Wiley. Winner, E., Rosenstiel, A. K., & Gardner, H. (1976). Miller, G. A., & Johnson-Laird, P. N. (1976). Perception The development of metaphoric understanding. Devel- and language. Cambridge, MA: Harvard University opmental Psychology, 12, 289-297. Press. Montaigne, M. de (1958). The complete essays of Mon- taigne (D. Frame, Trans.). Stanford, CA: Stanford University Press. Received May 30, 1984 Morris, D., Collett, P., Marsh, P., & O'Shaughnessy, M. Revision received December 27, 1984