Gathercole – Development of short-term memory Review

item length Q. J. Exp. Psychol. 44A, 47–67 contributions from processing speed, item identification, and short- 57 Case, R.D., Kurland, M. and Goldberg, J. (1982) Operational efficiency term memory J. Exp. Child Psychol. 65, 1–24 and the growth of short-term memory span J. Exp. Child Psychol. 33, 63 Bull, R., Johnston, R.S. and Roy, J.A. (1999) Exploring the roles of the 386–404 visuospatial sketchpad and central executive in children’s arithmetical 58 Towse, J.N., Hitch, G. and Hutton, U. (1998) A re-evaluation of working skills: views from cognition and developmental neuropsychology Dev. memory capacity in children J. Mem. Lang. 39, 195–217 Neuropsychol. 15, 421–442 59 Swanson, H.L. (1999) What develops in working memory? A life span 64 Leather, C.V. and Henry, L.A. (1994) Working memory span and perspective Dev. Psychol. 35, 986–1000 phonological awareness tasks as a predictors of early reading ability 60 Adams, J.W. and Hitch, G.J. (1997) Working memory and children’s J. Exp. Child Psychol. 58, 88–111 mental addition J. Exp. Child Psychol. 67, 21–38 65 Daneman, M. and Green, I. (1986) Individual differences in 61 Logie, R.H., Gilhooly, K.J. and Wynn, V. (1994) Counting on working comprehending and producing words in context J. Mem. Lang. 25, 1–18 memory in arithmetic problem solving Mem. Cognit. 22, 395–410 66 Gathercole, S.E. and Pickering, S.J. Working memory deficits in 62 Bull, R. and Johnston, R.S. (1997) Children’s arithmetic difficulties: children with special educational needs Br. J. Spec. Educ. (in press)

The role of gesture in communication and thinking Susan Goldin-Meadow

People move their hands as they talk – they gesture. Gesturing is a robust phenomenon, found across cultures, ages, and tasks. Gesture is even found in individuals blind from birth. But what purpose, if any, does gesture serve? In this review, I begin by examining gesture when it stands on its own, substituting for speech and clearly serving a communicative function. When called upon to carry the full burden of communication, gesture assumes a language-like form, with structure at word and sentence levels. However, when produced along with speech, gesture assumes a different form – it becomes imagistic and analog. Despite its form, the gesture that accompanies speech also communicates. Trained coders can glean substantive information from gesture – information that is not always identical to that gleaned from speech. Gesture can thus serve as a research tool, shedding light on speakers’ unspoken thoughts. The controversial question is whether gesture conveys information to listeners not trained to read them. Do spontaneous gestures communicate to ordinary listeners? Or might they be produced only for speakers themselves? I suggest these are not mutually exclusive functions – gesture serves as both a tool for communication for listeners, and a tool for thinking for speakers.

People gesture. This phenomenon has been remarked upon in other, more codified forms of communication. In this re- for at least 2000 years, across domains as diverse as philosophy, view, I examine both types of gestures – those that substitute Susan Goldin- Meadow is at the rhetoric, theater, divinity and language. The gestures that for speech, and those that accompany speech – with an eye Department of are most salient to speakers, and to listeners, are the codified towards understanding the role each plays in communication. Psychology, (or conventionalized) forms that can substitute for speech. University of There is, however, another type of gesture that people rou- Gestures that substitute for speech Chicago, 5730 South tinely produce – informal, non-codified hand movements, Gestures that have meaning independent of speech, and can Woodlawn Avenue, fleetingly generated during the course of speaking. The con- occur on their own without speech, are known as ‘emblems’1. Chicago, IL 60637, tent of these gestures is not typically the object of public Emblems have standards of form and can clearly be ‘mis- USA. scrutiny. As a result, these speech-accompanying gestures pronounced’. For example, imagine producing the North tel: +1 773 702 2585 have the potential to reflect thoughts that may themselves be American ‘okay’ gesture with the pinkie rather than the index fax: +1 773 702 0320 relatively unexamined by both speaker and listener. This type finger touching the thumb – the resulting handshape is not e-mail: sgsg@ccp. of gesture may thus reveal aspects of thought that are not seen recognizable as an ‘okay’. For the most part, emblems are uchicago.edu

1364-6613/99/$ – see front matter © 1999 Elsevier Science Ltd. All rights reserved. PII: S1364-6613(99)01397-2 419 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999 Review Goldin-Meadow – Gesture in communication and thinking

Box 1. Biological underpinnings of speech, sign and gesture Sign languages can be characterized by the same kinds of organizational princi- commonalties between sign and speech, this study underscores the differences ples as spoken languages. There are, however, surface differences between the between gestures that are organized around linguistic principles (e.g. ASL) and two types of languages, the most striking of which is that many linguistic de- gestures that are not so organized. vices in sign rely on spatial contrasts while speech is linear and non-spatial. In Thus far, only emblems and nonmeaningful movements have been tested speakers, damage to the left hemisphere typically results in impairments in lin- in these paradigms, leaving two interesting questions unanswered: (1) Do the guistic tasks, and damage to the right in impairments in spatial tasks. What non-codified and spontaneous gestures that accompany speech in hearing happens then when signers, whose language is grounded in spatial contrasts, persons show left hemisphere dominance? The answer is likely to be ‘no’ as experience brain damage? It turns out that, like speakers, signers with left these gestures do not exhibit the hierarchically segmented structures found in hemisphere lesions perform more poorly on language tasks than signers with spoken languages and codified sign languages. (2) Do the idiosyncratically right hemisphere lesions. Moreover, they do not show the deficits in spatial codified gestures that deaf children of hearing parents create to serve the func- tasks that signers with right hemisphere lesions do. These findings suggest that tions of a primary communication system show left hemisphere dominance? If sign is processed as linguistic information rather than spatial information, and truly linguistic, these home-made gesture systems are likely to be processed that the left hemisphere is specialized for processing that information, be it like natural language, signed or spoken, and thus are likely to be processed by transmitted by hand or mouth (Ref. a) (but see the recent discussion between the left hemisphere. Hickok, Bellugi and Klima, and Corina, Neville and Bavelier on the possibility that the right hemisphere also plays a role in sign language processing; Refs b,c). References The same question can be addressed in neurologically intact speakers and a Hickok, G., Bellugi, U. and Klima, E.S. (1996) The neurobiology of sign language and its implications for the neural basis of language Nature 381, 699–702 signers. Using behavioral tasks, Corina, Vaid and Bellugi (Ref. d) found left b Hickok, G., Bellugi, U. and Klima, E.S. (1998) What’s right about the neural hemisphere specialization, not only when hearing speakers processed speech, organization of sign language? A perspective on recent neuroimaging results but also when deaf signers processed sign. Interestingly, they found no evidence Trends Cognit. Sci. 2, 465–468 of hemispheric asymmetry when either group processed gesture – neither c Corina, D.P., Neville, H.J. and Bavelier, D. (1998) Response Trends Cognit. Sci 2, 468–430 emblems (e.g. waving good-bye, giving the thumbs-up) nor sequences of limb d Corina, D.P., Vaid, J. and Bellugi, U. (1992) The linguistic basis of left hemisphere movements that had no meaning. In addition to demonstrating once again the specialization Science 255, 1258–1260

used to insult, praise or regulate the behavior of a communi- languages (Box 2). In this sense, the gestures have syntactic cation partner2. Speakers who produce emblems are aware structure. The systems also have morphological structure, of having done so, and listeners are aware of having seen an with each gesture itself composed of smaller, meaningful emblem produced. In other words, emblems are consciously handshape and motion components17. Finally, the gestures communicative. are language-like in that they are used for many of the func- Emblems are not, however, combined into gesture strings tions of natural language – describing the non-present18, to make longer and more complex sentences – they do not ‘talking’ to oneself19 and commenting metalinguistically on form a linguistic system. The manual modality can, of course, one’s own or another’s gestures20. support a system of gestures that has linguistic structure, as The striking aspect of these gesture systems is that they sign languages of the deaf illustrate. Sign languages, such as are invented by deaf children who have access, not to con- American Sign Language (ASL), are autonomous systems ventional sign languages such as ASL, but only to the spon- not based on the spoken languages of the hearing cultures taneous gestures that their hearing parents use as they talk. that surround them3–5. Like spoken languages, sign languages While the deaf children’s gesture systems have language-like are structured at syntactic6,7 morphological8,9 and phonolog- structure, the hearing parents’ gestures – like the gestures of ical10–12 levels and exhibit left hemisphere dominance (Box 1). all hearing speakers21 – do not22. In addition to being primary communication systems, Is it possible to predict when the manual modality will sign languages are also systems that have histories and are assume language-like structure and when it will not? We23 passed down from one generation of users to the next13. They have suggested that the manual modality takes on gram- are codified linguistic systems that assume the full burden of matical properties only when it is required to carry the full communication. There are, however, situations in which burden of communication (as in conventional sign languages non-codified gesture is forced to take on the functions of a of the deaf, and unconventional gesture systems created by primary communication system; for example, deaf children deaf children lacking language models). When the manual whose hearing losses prevent them from acquiring spoken modality is used in conjunction with speech and does not language even with intensive oral instruction, and whose carry the full burden of communication, it does not assume hearing parents have not yet exposed them to sign language. language-like form – that is, it does not convey meaning by Gesture is the primary means by which children in this situ- rule-governed combinations of discrete units. Rather, these ation communicate14,15. The question is whether their gestures gestures convey meaning mimetically and idiosyncratically assume the language-like forms characteristic of a codified through continuously varying forms21. The question is whether communication system like ASL. The answer is that they do the gestures that accompany speech play a role in communi- – on all levels that have been examined thus far. cation despite the fact that they are not language-like in form. Deaf children of hearing parents invent gestures that serve as the ‘lexicon’ of their communication system; the form of Gestures that accompany speech the gestures is stable over several years16. These lexical items Nonverbal behaviors, including gestures that accompany are combined into gesture strings characterized by patterns speech, have traditionally been assumed to reflect speakers’ reminiscent of ergative structures found in many natural feelings and emotions24 and have been used by researchers as

420 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999 Goldin-Meadow – Gesture in communication and thinking Review

Box 2. Spontaneous gesture systems created by deaf children Unlike hearing children and adults who rarely concatenate their spontaneous gestures into strings (Refs a,b), deaf children who use gesture as their primary means of communication often convey their message via gesture sentences rather than single gestures. Deaf children of hearing parents in both American and Chinese cultures produce gesture sentences that conform closely to a structural analog of the ergative pattern predominating in some natural languages (Refs c,d); importantly though, neither English nor Mandarin (the grammatical structure trends in Cognitive Sciences

noted in the deaf children’s gesture systems is therefore not likely to have been Fig. II. A deaf child uses gesture to invite a listener to join him for derived from the structure of the spoken languages that surrounded them). pretzels. The child first indicates the action, ‘eat’, and then the actor, ‘you’ (the The hallmark of an ergative pattern is that the actor in an intransitive sentence third gesture is a repetition of the second, presumably for emphasis). In this (‘you’ in the proposition ‘you move there’) is treated differently (i.e. given a dif- instance, the child omitted a gesture for the patient, the pretzels. Note that a ferent syntactic or morphologic marking) from the actor in a transitive sentence typical pattern for English would be ‘you eat’ rather than ‘eat you’. (‘you’ in ‘you eat pretzels’), and instead is marked like the patient (‘pretzels’). By contrast, in an accusative language such as English, intransitive actors are treated verb (‘you move there’ and ‘you eat pretzels’) while patients follow the verb (‘you like transitive actors and not like patients; for example, both actors precede the eat pretzels’). The results of a study that examined how likely American and Chinese deaf A B children (and their hearing mothers) were to produce gestures for transitive 1.0 Abe 1.0 Fen actors, patients, and intransitive actors (Ref. e) are illustrated in Fig. I. In seven of the eight children, gestures were produced significantly more often for patients 0.5 0.5 (pretzels) than for transitive actors (you, as eater). The crucial question is whether the deaf children treated intransitive actors (you, as mover) like patients or like 0.0 0.0 transitive actors. In fact, gestures were produced significantly more often for intransitive actors than for transitive actors, and there were no significant differ- 1.0 Karen 1.0 Ling ences between intransitive actors and patients. This production probability conformed to an ergative pattern – gesture production was equal for intransitive 0.5 0.5 actors and patients, and distinct for transitive actors. Like their children, mothers tended to produce more gestures for patients than for transitive actors in the spontaneous gestures they produced when talking. However, unlike their chil- 0.0 0.0 dren, they showed no reliable patterning of intransitive actors, and thus did not display an ergative pattern. 1.0 David 1.0 Bao In addition to reliably producing some semantic elements at the expense of others, children were also consistent in where those elements were positioned in 0.5 0.5 Production probability Production probability two-gesture strings. Intransitive actors were produced in the first position of a two-gesture string (‘you move’), as were patients (‘pretzels eat’). The one child 0.0 0.0 who produced enough transitive actors for analysis showed an ergative pattern 1.0 1.0 here too (Ref. f ). He produced transitive actors in the second position of a two- Marvin Qing gesture string (‘eat you’; see Fig. II), thereby distinguishing this type of actor from both intransitive actors and patients. 0.5 0.5 The deaf children thus introduced into their gesture systems a pattern that can be found in natural languages, but that was not found in the spontaneous 0.0 0.0 gestures their hearing parents used with them. TA P IA TA P IA TA P IA TA P IA References trends in Cognitive Sciences a Petitto, L.A. (1988) ‘Language’ in the prelinguistic child, in The Development of Fig. I. Probability with which transitive actors (TA), patients (P), and Language and Language Researchers (Kessel, F., ed.), pp. 187–221, Erlbaum intransitive actors (IA) are gestured. Probabilities were calculated using b McNeill, D. (1992) Hand and Mind, University of Chicago Press sentences in which three semantic elements could be gestured but only two el- c Dixon, R.M.W. (1994) Ergativity, Cambridge University Press ements actually were gestured. Both American (A) and Chinese (B) deaf children d Silverstein, M. (1976) Hierarchy of features and ergativity, in Grammatical Categories (dark bars, also identified by name) showed significant differences in production in Australian Languages (Dixon, R.M.W., ed.), pp. 112–171, Australian Institute of patterns across the three elements: gestures were produced reliably more often Aboriginal Studies for intransitive actors than for transitive actors, but were equally likely for intran- e Goldin-Meadow, S. and Mylander, C. (1998) Spontaneous sign systems created by sitive actors and patients – a structural analog of the ergative pattern found in deaf children in two cultures Nature 391, 279–281 certain natural languages. Hearing mothers (white bars) were not consistent in f Goldin-Meadow, S. and Mylander, C. (1984) Gestural communication in deaf children: their treatment of intransitive actors, and thus did not display an ergative pattern. the effects and non-effects of parental input on early language development (Data redrawn from Ref. e.) Monogr. Soc. Res. Child Dev. 49

a route to speakers’ attitudes. But gesture has the potential to taneous gestures that accompany speech in conversations25, convey substantive information about a task (Box 3). Are re- narratives21, descriptions of objects and actions23, and expla- searchers able to take advantage of this source of information nations26. The gestures produced in these situations can be into the speaker’s thoughts? assigned meanings and, most importantly, independent ob- The first step in addressing this question is to assess servers reliably assign the same meaning to the same gesture. whether gesture can be interpreted reliably and consistently. The second step is to determine whether the meanings A substantial number of investigators have observed the spon- experimenters assign to gesture are the meanings the speaker

421 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999 Review Goldin-Meadow – Gesture in communication and thinking

Box 3. Spontaneously produced gestures McNeill (Ref. a) has identified a number of different types of gesture produced during a liquid conservation task. Meaning is gestures that speakers routinely use when they talk: assigned to each gesture on the basis of the shape and placement of the hand and form of the motion in relation to its accompa- (1)‘Iconic’ gestures transparently capture aspects of the semantic nying speech. A flat palm held horizontally without movement content of speech. For example, when describing how water at the water level of a container is the gesture that typically accom- was poured from a glass into a dish, a child arced her fist in panies height explanations in speech (‘it’s tall’); the meaning the air as though pouring from one container to another. ‘height’ is therefore assigned to this gesture form. The form– (2)‘Metaphoric’ gestures are like iconics in that they are picto- meaning pairings that result from this process are then used to rial; however, the pictorial content is abstract rather than con- code gestures produced by other children performing this same crete. For example, when announcing that what he is about task. to narrate is a cartoon, a speaker raised his hands as though To ensure independence of the gesture and speech codes, one offering the listener an object. Just as we speak metaphorically experimenter codes gesture without listening to the accompany- about ‘presenting’ an idea or argument, gesture makes an ing speech (i.e. with the sound turned off) and another codes abstract entity (the cartoon) concrete by treating it as a speech without watching gesture (i.e. with the picture turned off). bounded object supported by the hands and presented to the A response is considered a gesture–speech mismatch if, in sep- listener. arate passes through the data, the meaning assigned to gesture (3)‘Beat’ gestures look as though they are beating musical time. is different from the meaning assigned to speech. If, for example, The hand moves along with the rhythmical pulsation of gesture is assigned the meaning ‘height’ by one coder and the speech. Unlike iconics and metaphorics, beats tend to have accompanying speech is assigned the meaning ‘width’ by another, the same form regardless of the content (a simple flick of the the response as a whole is considered a mismatch. hand or fingers up and down, or back and forth). Reliability between two experimenters, each of whom tran- (4)‘Deictic’ or pointing gestures indicate entities in the conver- scribes the same videotape independently, is typically high for sational space, but they can also be used even when there is coding speech, gesture, and the relationship between the two nothing to point at. For example, a speaker asked ‘where did (e.g. on conservation tasks, agreement between two coders ranges you come from before?’ while pointing at a space in the room. from 85% to 94%) (Ref. b). The space was not, in fact, the listener’s former location but, over the course of the conversation, had come to represent that location. References a McNeill, D. (1992) Hand and Mind, University of Chicago Press It is relatively easy to develop a gestural ‘lexicon’ for a particu- b Goldin-Meadow, S. and Sandhofer, C.M. (1999) Gesture conveys lar task. The lexicon can then be used to code gesture and its substantive information about a child’s thoughts to ordinary relation to speech in subsequent data. For example, consider listeners Dev. Sci. 2, 67–74

intends. Consider a child asked to complete an ‘explanation permits the speaker to represent ideas that are compatible task’ in which she explains her solutions to a set of math prob- with its mimetic and analog format (e.g. shapes, sizes, spatial lems and, later, a ‘rating task’ in which she judges solutions to relationships) – ideas that may be less compatible with the a second set of problems. If gesture is a vehicle through which discrete and categorical format underlying speech. Thus, speakers express their knowledge – and if experimenters can when it accompanies speech, gesture allows speakers to convey correctly assess this knowledge when attributing problem- thoughts that may not easily fit into the categorical system solving procedures to children based on their gestures in the that their spoken language offers29. For example, the gestures explanation task – then, on the rating task, the child should accompanying a description of the East Coast of the United judge procedures she expressed on the explanation task in States can convey aspects of the coastline that would be diffi- gesture as more ‘acceptable’ than procedures she did not ex- cult, if not impossible, to convey in speech. Gesture then has press at all. Children were found to do just that – even when the potential to display thoughts that are not conveyed in the those procedures were expressed only in gesture and not speech it accompanies (gesture also provides prelinguistic anywhere in speech27. In addition to underscoring an im- children with a vehicle for expressing thoughts they do not portant methodological point (that experimenters’ interpre- yet have words for, see Box 4). tations of gesture are valid), these findings confirm an im- Consider a six-year-old child attempting to justify his (in- portant conceptual point – that gestures are not random correct) belief that number changes when one of two identical movements but rather reveal substantive beliefs about the rows of checkers is spread out. The child says that the number task at hand. of checkers in the spread-out row is now ‘different because you moved them’, thus making it clear, in speech, that he has Gesture can reflect thoughts not conveyed in speech focused on what was done to the checkers. However, in the Speakers use gesture to depict notions ranging from the con- gestures accompanying this utterance, the child indicates crete (e.g. the actions or attributes of cartoon characters) to some understanding of the one-to-one correspondence that the abstract (e.g. mathematical notions, such as quotients, can be established between the two rows of checkers – he factors, and even limits in calculus)28. Because gesture rests moves a pointing hand back-and-forth between the two rows, on different representational devices from speech, and is not pairing the first checker in row 1 with the first checker in row dictated by standards of form as is speech, it has the potential 2, and so on30. The child speaks about how the checkers were to offer a different view into the mind of the speaker. Gesture moved, but he has also noticed – not necessarily consciously

422 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999 Goldin-Meadow – Gesture in communication and thinking Review

Box 4. Gesture is a precursor to speech At a time when children are limited in what they can say, there is entially (i.e. they do not understand that the point is about a another avenue of expression open to them, one that can extend particular object); rather, they learn to respond to the gesture in the range of ideas they are able to express. Children can gesture. such a way that they can garner a food reward (Ref. m). Finally, The earliest gestures children use, typically beginning around 10 although chimps who have been taught a communication system months and prior to their first words, are ‘deictics’ (Ref. a). For do gesture, they rarely use those gestures (or, for that matter, example, a child holds up an object to draw an adult’s attention the system they have been taught) to comment (Ref. n). Thus, to that object or, later in development, points at the object. At gesture may serve as a way-station on the road to language, the same time, some children also use ‘iconic’ gestures. For ex- both over ontogenetic and evolutionary time.

ample, a child opens and closes her mouth to represent a fish, or References flaps her hands to represent a bird (Refs b,c). ‘Metaphoric’ and a Bates, E. et al. (1979) The Emergence of Symbols: Cognition and ‘beat’ gestures are not produced before speech, and do not appear Communication in Infancy, Academic Press until later in development (see also Box 3). b Acredolo, L.P. and Goodwyn, S.W. (1985) Symbolic gesture in language Combining gesture and speech within a single utterance can development: a case study Hum. Dev. 28, 40–49 also increase the communicative range available to children. Most c Iverson, J.M., Capirci, O. and Caselli, M.S. (1994) From communication of the gesture–speech combinations that young children produce to language in two modalities Cognit. Dev. 9, 23–43 contain gestures that convey information redundant to the infor- d Goldin-Meadow, S. and Morford, M. (1985) Gesture in early child language: studies of deaf and hearing children Merrill-Palmer mation conveyed in speech; for example, pointing at an object Quarterly 31, 145–176 while naming it. However, young children also produce gesture– e Greenfield, P. and Smith, J. (1976) The Structure of Communication in speech combinations in which gesture conveys information that Early Language Development, Academic Press is different from the information conveyed in speech; for exam- f Morford, M. and Goldin-Meadow, S. (1992) Comprehension and ple, gesturing at an object while describing the action to be done production of gesture in combination with speech in one-word on the object in speech (pointing to an apple and saying ‘give’), speakers J. Child Lang. 9, 559–580 or gesturing at an object and describing the owner of that object g Zinober, B. and Martlew, M. (1985) Developmental changes in four in speech (pointing at a toy and saying ‘mine’) (Refs d–g). types of gesture in relation to acts and vocalizations from 10 to 21 Children begin producing gesture–speech combinations months Br. J. Dev. Psychol. 3, 293–306 prior to their first two-word utterances. More impressive is the h Goldin-Meadow, S. and Butcher, C. Pointing toward two-word speech in young children, in Pointing: Where Language, Culture, and fact that the age at which children first produce combinations in Cognition Meet (Kita, S., ed.), Cambridge University Press (in press) which gesture and speech convey different information (e.g. ‘give’ i Hewes, G.W. (1973) Primate communication and the gestural + point at apple) predicts the age at which they produce their origin of language Curr. Anthropol. 14, 5–24 first word–word combinations (‘give apple’) (Ref. h). Thus, the j Donald, M. (1991) Origins of the Modern Mind: Three Stages in the ability to use gesture and speech together to convey different Evolution of Culture and Cognition, Harvard University Press components of a proposition is a harbinger of the ability to k Tomasello, M. and Camaioni, L. (1997) A comparison of the gestural convey those components solely within speech. communication of apes and human infants Hum. Dev. 40, 7–24 Given theories about the importance of gesture in the origins l Plooij, F.X. (1978) Some basic traits of language in wild chimpanzees?, of language (Refs i,j), it is interesting to note that chimpanzees in Action, Gesture and Symbol: The Emergence of Language (Lock, A., in the wild do not use gesture as human children do (Ref. k). ed.), pp. 111–131, Academic Press m Povinelli, D. et al. (1997) Exploitation of pointing as a referential Chimps use gesture to request here-and-now action of another gesture in young children, but not adolescent chimpanzees chimp (e.g. raising their arms over their heads to invite another Cognit. Dev. 2, 423–461 chimp to groom them) (Ref. l). In contrast, children use gesture, n Greenfield, P.M. and Savage-Rumbaugh, E.S. (1991) Imitation, not only to request, but also to comment on objects in their grammatical development and the invention of protogrammar by surrounds (e.g. pointing to comment on distant events). Even an ape, in Biological and Behavioral Determinants of Language chimps who are raised by humans do not interpret points refer- Development (Krasnegor, N.A. et al. eds), pp. 235–262, Earlbaum

– that the spread-out row can be aligned with the untouched knows the equal sign breaks the string into two parts. Mis- row. This insight is one that the child expresses only through matches have also been found in toddlers32, preschoolers33, his hands. adolescents34 and even adults35. Nor are mismatches re- This child has produced a gesture–speech mismatch – a stricted to number puzzles – they arise in spontaneous talk36, communicative act in which the information conveyed in narratives21, reasoning about physics problems37, moral gesture is different from the information conveyed in the dilemmas38 and many other contexts. accompanying speech. Gesture–speech mismatches are not In a mismatch, a speaker conveys two distinct ideas about unique to conservation or to six-year-olds, however. We find the very same problem. But the explanations in which gestures the same phenomenon in nine-year-old children asked to are produced often come after the fact. Do speakers who pro- explain their solutions to mathematical equivalence problems duce mismatches in their post hoc explanations of a problem such as 4ϩ5ϩ3ϩ__ϭϩ3 (Ref. 31). For example, a child says, also activate two ideas when solving that problem on-line? ‘I added 4 plus 5 plus 3 plus 3 and got 15’, demonstrating no The evidence suggests that they do (Box 5) – mismatchers ac- awareness of the fact that this is an equation bifurcated by an tivate more than one idea, not only when explaining a math equal sign. Her gestures, however, offer a different picture: problem, but also when solving that same type of math she sweeps her left palm under the left side of the equation, problem. The cognitive state that underlies mismatch thus pauses, then sweeps her right palm under the right side. The involves having, and activating, more than one idea on a child’s gestures clearly demonstrate that, at some level, she single task.

423 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999 Review Goldin-Meadow – Gesture in communication and thinking

Box 5. How do gesture–speech mismatchers solve problems on-line?

If explanations are an accurate reflection of the processes that take place in A Match one-word problem-solving, mismatching children, who produce two procedures on each 1.0 Match three-word explanation of their math problems (one in speech and a second in gesture), Mismatch one-word should also activate two procedures when solving each problem. In contrast, 0.8 Mismatch three-word matching children, who produce a single procedure per explanation, ought to activate only one procedure when solving each problem. If so, mismatchers 0.6 work harder to arrive at their incorrect answers than matchers, and should have less cognitive effort left over to tackle other tasks. 0.4

Goldin-Meadow et al. (Ref. a) tested this prediction by first asking children math problems 0.2

to solve and explain a series of mathematical equivalence problems. Their Proportion of correct explanations were used to divide children into matchers and mismatchers. Children then participated in another math task and a concurrent word recall 0.0 Easy Hard task. On each trial, children were given a list of words to remember, either a one- math problem math problem word list (which was not expected to tax cognitive capacity) or a three-word list (which was more likely to strain capacity). Children were also given a math B problem to solve while remembering the words – either a hard problem (e.g. 1.0 3ϩ6ϩ7ϩ__ϭϩ8) which tends to elicit two-procedure explanations from mismatchers, and one-procedure explanations from matchers; or an easy 0.8 problem (e.g. 4ϩ7ϩ3ϩ5ϭ__) which elicits one-procedure explanations from 0.6 all children. ‘Easy’ problems were indeed easy, and were solved correctly by all children 0.4 (Fig. I). ‘Hard’ problems were hard and solved by none, irrespective of whether lists word the problems were solved along with one- versus three-word lists. The key 0.2

finding involves the proportion of word lists the children recalled correctly as Proportion of correct a function of the type of math problem (easy or hard) that accompanied the 0.0 task (Fig. IB). Three points are noteworthy: Easy Hard (1)Both matching and mismatching children were expected to activate a single math problem math problem

procedure per problem when solving easy math problems. They should thus trends in Cognitive Sciences expend the same amount of effort on these problems and should recall the Fig. I. Performance on math problems and a concurrently administered same proportion of word lists after solving them – as indeed they did. word-recall task. (A) The proportion of easy (e.g. 4ϩ7ϩ3ϩ5ϭ__) and hard (e.g. (2)Matchers were also expected to activate a single procedure when solving 3ϩ6ϩ7ϩ__ϭϩ8) math problems that were solved correctly by matching and mis- hard math problems. Their performance on word recall should therefore matching children while simultaneously recalling one- versus three-word lists. not differ for easy versus hard problems – and it did not. (Solid lines, data referring to matching children; broken lines, data referring to (3)In contrast, mismatchers were expected to active two procedures when mismatching children; error bars indicate SE.) All children solved the ‘easy’ prob- lems correctly but none solved the ‘hard’ problems. (B) The proportion of one- solving hard math problems. They should therefore have less capacity left and three-word lists that were recalled correctly by matching and mismatching over to recall words after solving these problems and should perform poorly children while simultaneously solving easy versus hard math problems. (Symbols on this task, particularly when capacity is taxed on the three-word lists. As and lines the same as in A.) Mismatching children’s recall differed from match- predicted, mismatchers recalled significantly fewer three-word lists after ing children’s recall only on three-word lists and only when solving hard math solving hard math problems than did matchers. problems. (Modified from Ref. a.)

Thus, when solving the hard math problems, mismatchers carry the extra Reference burden of too many unintegrated ideas, a burden which appears to take cognitive a Goldin-Meadow, S. et al. (1993) Transitions in learning: evidence for simultaneously effort, leaving less effort available for other tasks. activated strategies J. Exp. Psychol. 19, 92–107

Gesture can index cognitive instability instruction if it is provided but vulnerable to regression if Gesture–speech mismatch is of particular interest to re- it is not. searchers because it is a characteristic of learners who are in If mismatch is indeed a transitional period, we would transition with respect to a task. Children who produce a rel- expect learners to proceed from a stable state, through mis- atively large proportion of gesture–speech mismatches when match, to another stable state but at a higher level. Micro- explaining their (incorrect) solutions to a task are particularly genetic methods were developed so that learners’ progress likely to benefit from instruction in that task – significantly could be systematically monitored around periods of greatest more than children who produce few mismatches. Mismatch change40. Alibali and Goldin-Meadow39 used the technique has been found to be a reliable index of readiness-to-learn in to chart the course of children’s development of mathematical conservation30 and mathematical equivalence31 tasks. Thus, equivalence over the short term. They instructed children when children produce mismatches, they are ‘at risk’ for learn- individually and observed each child’s step-by-step progress. ing. They are also, however, at risk for regression. The same Focusing on children who gestured, they found that almost proportion of children who progress to a stable correct state all of the children passed through two, or even three, of the when given instruction in the math task, regress to a stable following steps in order: (1) a stable state in which the child incorrect state when given no instruction39. Mismatchers produced gesture–speech matches conveying incorrect pro- are apparently in a state of cognitive instability, open to cedures; (2) an unstable state in which the child produced

424 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999 Goldin-Meadow – Gesture in communication and thinking Review

Table 1. Gesture–speech matches and mismatches used by children as they progress towards mastery in two tasks

Modality Matching explanations Mismatching explanations Matching explanations (incorrect) (both incorrect) (correct)

Sample responses to the mathematical equivalence problem 6+4+7=__+7

Speech ‘I added the 6, the 4 and the 7’ ‘I added the 6, the 4 and the 7’ ‘I made this side equal to that (add-to-equal-sign) (add-to-equal-sign) side’ (equalize-two-sides)

Gesture Points at 6, 4 and and left 7 Points at 6, 4, left 7 and right 7 Palm sweeps under left side (add-to-equal-sign) (add-all-numbers) then under right (equalize- two-sides)

Sample responses to a liquid-conservation taska

Speech ‘This one’s tall and this one’s ‘This one’s tall and this one’s short’ ‘This one’s tall but it’s skinny’ short’ (comparison of heights) (comparison of heights) (compensation)

Gesture Flat palm marks water level of Narrow C demarcates width of glass Flat palm marks water level of glass then water level of dish then wider C demarcates width glass then narrow C demarcates (comparison of heights) of dish (comparison of widths) its width (compensation)

aIn this task, water is poured from a tall, thin glass into a short, wide dish. gesture–speech mismatches; (3) a stable state in which the who produce few mismatches. Moreover, the ‘extra’ ideas child again produced gesture–speech matches, now conveying that mismatchers have are found only in gesture and not in correct procedures. speech (Box 6). Thus, the variability that may make mis- Thus, as predicted, children tended to proceed through matchers particularly vulnerable to instruction can only be mismatch in acquiring mathematical equivalence. Interest- detected by looking at the children’s hands, not by listening ingly, the few children who skipped the mismatching state and to their words. progressed directly from an incorrect to a correct matching state were less able to generalize their knowledge than the Do gestures communicate to listeners? children who passed through the mismatching state.Skippers We have seen that gesture can communicate unique infor- appear to learn the concept less thoroughly than those who mation about a learner’s state, and that researchers have pass through mismatch. begun to take advantage of this ‘window’ to the mind. The Is gesture in advance of speech when children are in a question is whether the information displayed in gesture is mismatching state? Mismatchers’ gestures often convey ideas accessible to ordinary listeners not trained in laboratory that are more developed than those they convey in speech41, settings. as illustrated in the math and conservation examples presented This is a controversial question. On the one hand, thus far. But speech can also convey the mature ideas while Kendon46 concludes that listeners do attend to gesture and gesture lags behind, as is typically found in moral judgment alter their understanding of utterances accordingly. On the tasks38. Finally, gesture and speech can both convey incorrect other hand, Krauss and his colleagues47 argue that gesture notions, albeit different ones, as illustrated in the mismatching has little communicative value. However, many of the rele- examples in Table 1. vant studies do not control the type of speech that accom- Why might mismatch be associated with transition and panies gesture; without such control it is often difficult to learning? Mismatch is an index of variability, and variability demonstrate definitively that gesture is influencing under- is considered by many theorists to be essential to develop- standing. Other studies narrow the field too much, exploring mental progress42,43. A number of studies have demonstrated only gestures that are selected by observers to be completely variability in children’s repertoires across problems (e.g. one redundant with the speech they accompany. Indeed, not procedure is used to solve a problem at time 1, and another enough attention has been paid to gesture that conveys dif- is used to solve the problem at time 2)44,45. Gesture–speech ferent information from speech. One might expect that it is mismatch is unique in that it reflects within-problem vari- precisely in situations of gesture–speech mismatch that gesture ability – the child produces two different procedures, one in can play its largest role in communication. speech and the other in gesture, on the problem at the same A number of recent studies have found that ordinary time. Activating two procedures on the same problem may be listeners can reliably ‘read’ gesture when it conveys different essential for eventually coordinating the two procedures, or information from speech48–52, even when gesture is unedited resolving the conflict between them. It turns out, however, and fleeting, as it is in natural communication53. Take, for that children who produce a large number of mismatches example, an untrained adult asked to assess the child described also tend to have across-problem variability – they have more earlier who, in speech, focused on how the checkers were different kinds of ideas in their repertoires than children moved but, in gesture, highlighted the one-to-one alignment

425 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999 Review Goldin-Meadow – Gesture in communication and thinking

Box 6. Gesture–speech mismatchers have variable problem-solving repertoires Recent research has shown that, at a single moment in time, chil- d Siegler, R.S. (1999) Strategic development Trends Cognit. Sci. 3, dren frequently have in their repertoires a variety of strategies or 430–435 approaches to a problem (Refs a–d). Children who tend to pro- e Goldin-Meadow, S., Alibali, M.W. and Church, R.B. (1993) Transitions in concept acquisition: using the hand to read the mind Psychol. duce gesture–speech mismatches produce more than one notion Rev. 100, 279–297 on a single problem and, in this sense, are variable in their re- sponses. However, do these children also have variability in their repertoires when taken as a whole? To address this question, 4 Goldin-Meadow, Alibali and Church (Ref. e) determined how many different problem-solving procedures a child produced across a set of six math problems. They also determined the modality (or modalities) in which each different procedure was produced: in speech and never in gesture; in gesture and never in 3 speech; or in both speech and gesture (although not necessarily on the same problem). The calculations were done separately for children who produced a large proportion of mismatches, and for children who produced few mismatches. Overall, mismatchers produced significantly more different kinds of procedures than 2 matchers (Fig. I). They thus had more variability in their rep- ertoires, which might well account for their ‘vulnerability’ to instruction. In terms of modalities in which these procedures appeared, it is interesting to note that neither group of children produced many procedures in speech alone. Almost all of the information 1 that the children possessed about this task was accessible to gesture, No. of different procedures in repertoire of different No. either with or without speech. The groups also did not differ in the number of different procedures they produced in both speech and gesture. Where the groups did differ, however, was in the number of different procedures they produced in gesture alone. 0 The mismatchers produced signifcantly more procedures in Mismatching Matching gesture (and not in speech) than the matchers. Thus, the ‘extra’ children children procedures that the mismatchers had in their repertoires were trends in Cognitive Sciences all unique to gesture. Fig. I. Number of different problem-solving procedures in References the repertoires of mismatching and matching children. A a Adolph, K.E. (1997) Learning in the development of infant procedure was classified as a function of the modality (or modali- locomotion Monogr. Soc. Res. Child Dev. 62 ties) in which it was produced by a child across six problems. Overall, b Siegler, R.S. (1996) Emerging Minds: The Process of Change in mismatchers produced significantly more types of procedures than Children’s Thinking, Oxford University Press matchers, although the distribution varied in the two groups, c Siegler, R.S. and McGilly, K. (1989) Strategy choices in children’s particularly in procedures produced in gesture but not speech time-telling, in Time and Human Cognition: A Life Span Perspective (dark grey). Procedures in speech and gesture, light grey; proce- (Levin, I. and Zakay, D., eds), pp. 185–218, Elsevier dures in speech but not gesture, white. (Adapted from Ref. e.)

between the checkers in the rows. The adult attributed to this conveys information to naive listeners. Listeners glean mean- child reasoning based not only on the fact that the checkers ing from gesture which, in turn, has an effect on how the had been moved, but also on one-to-one correspondence. meaning conveyed in the accompanying speech is interpreted. Ordinary listeners can thus take advantage of the unique in- Gesture thus has the potential to play a role in cognitive sight gesture offers into the thoughts speakers have, but do change. Gesture can signal to those who interact with a not express in words. learner that a particular notion is in the learner’s repertoire. Given that a listener can extract substantive information Listeners may then alter their behavior accordingly, perhaps from gesture, it is perhaps not surprising that speech can be giving explicit instruction in that notion if it is correct, or affected by the gestural company it keeps. Gesture can facili- providing input that encourages the learner to abandon the tate comprehension of a spoken message when it conveys that notion if it is misguided. If gesture can play this type of role same message (gesture–speech match). But gesture can also in spontaneous interaction, learners may be able to shape the impede comprehension of a spoken message when it conveys day-to-day input they receive just by moving their hands. a different message (gesture–speech mismatch). Gesture is clearly part of the communicative process, one that at times Do gestures function for speakers? can lead the listener astray (Box 7). We have been proceeding as though communication is ges- To summarize thus far, gesture not only conveys sub- ture’s only function – but is it? The fact that gestures commu- stantive information to well-trained coders who have the ad- nicate to listeners does not preclude the possibility that gesture vantage of time and instant replay on their side, but it also functions for speakers as well. After all, speakers gesture when

426 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999 Goldin-Meadow – Gesture in communication and thinking Review

Box 7. Teachers gesture in math tutorials and children pay attention Gesture is understood by listeners asked to observe, but not participate, in an Gesture was found both to help and to undermine the child’s compre- interaction. However, in order to argue that gesture plays an important role in hension of teacher speech (Fig. IA). Children were more likely to repeat a pro- communication, it is essential to demonstrate that listeners can ‘read’ gesture cedure the teacher produced in speech when that speech was accompanied by when they themselves are participants in the conversation. Goldin-Meadow, a matching gesture than when it was accompanied by no gesture at all. When Kim and Singer (Ref. a) asked teachers to instruct a series of children individ- gesture conveys the same message as speech, perhaps not surprisingly, it helps ually in mathematical equivalence and videotaped the tutorials. As the teachers the listener arrive at that message. Conversely, children were less likely to repeat did most of the talking – and gesturing – in the interactions, the experimenters a procedure the teacher produced in speech when that speech was accompanied explored how the children responded to the problem-solving procedures that by a mismatching gesture than when it was accompanied by no gesture at all. teachers produced in speech and gesture. A child was conservatively assumed When gesture conveys a different message from speech, it detracts from the to understand the teacher’s procedure when the child responded by repeating listener’s ability to arrive at the message in speech. or paraphrasing that procedure. Were children able to glean substantive information from the teacher’s ges- tures? When the teacher procedures were expressed uniquely in gesture (i.e. in A 0.6 B 0.6 the gesture component of a mismatch) children reiterated these procedures 20% of the time (Fig. IB). Although this might seem like a small proportion, 0.5 0.5 note that child reiterations followed teacher procedures produced uniquely in speech only 25% of the time (compare the dark grey bars in Fig. IA and B). 0.4 0.4 Moreover, children frequently ‘translated’ procedures that the teacher produced uniquely in gesture into their own speech, thus clearly demonstrating that they 0.3 0.3 fully understood their meaning. Children were able to glean information from 0.2 0.2 teachers’ gestures even when they were active participants in the interaction. Indeed, there is anecdotal evidence fom the videotaped teacher–child tu- Child reiterations Child reiterations 0.1 0.1 torials that children zero in on a teacher’s gestures, sometimes to their own dis- of teachers' procedures advantage. For example, while telling a child to make both sides of the equation 0 0 Speech Gesture 4+6+7+__=7 equal, a teacher inadvertently pointed at the four numbers in the problem – a series of gestures children typically produce when they add trends in Cognitive Sciences all of the numbers to get the solution. The child ignored the teacher’s speech Fig. I. Mean proportion of child reiterations of teachers’ problem-solving and focused on her gestures. In the next turn, much to the teacher’s surprise, procedures. (A) The proportion of child reiterations following the teacher’s the child gave 24 as his solution, the solution obtained by adding up all four speech when accompanied by matching gesture (light grey), no gesture (white), numbers. Gesture is a potentially powerful source of input for children and or mismatching gesture (dark grey). (Error bars indicate SE.) (B) The proportion of child reiterations following the teacher’s gesture when accompanied by adults alike. mismatching speech (i.e. gesture component of a mismatch). Child reiterations followed procedures produced in the gesture component of a mismatch 20% Reference of the time – approximately as often as they followed procedures produced in a Goldin-Meadow, S., Kim, S. and Singer, M. What the teacher’s hands tell the student’s the speech component. (Modified from Ref. a.) mind about math J. Educ. Psychol. (in press)

no one is watching54. Indeed, blind speakers gesture routinely, standing of the one-to-one correspondence between checkers even though they themselves have never seen gesture – and in his gestures, but seemed unable to articulate this notion in they do so even when they are knowingly speaking with speech. The ease with which the two rows of checkers can be listeners who are themselves blind55. paired in gesture may have facilitated the child’s expression Gesture plays a variety of roles for speakers. Gesture helps of this notion. Once having entered the child’s repertoire, this speakers retrieve words from memory56. Gesture reduces cog- new-found idea can begin to change the system. At some nitive burden, thereby freeing up effort that can be allocated point, the child will have to reconcile his belief that the to other tasks. For example, pointing improves young chil- number of checkers changed with the fact that the checkers dren’s performance on counting tasks particularly if the point- in the moved and unmoved rows can be put into one-to- ing is done by the children themselves57. As another example, one alignment. By offering an alternative route in which de- gesturing while explaining a math task improves performance veloping ideas can be tried out and expressed, gesture may on a simultaneously performed word recall task58. Gesturing itself facilitate the process of change. thus appears to increase resources available to the speaker, Gesture may also have an advantage over speech in that perhaps by shifting the burden from verbal to spatial memory. novel (and perhaps contradictory) information can be brought Gesture may also provide a route through which learners into a learner’s repertoire without disrupting the current sys- can access new thoughts. For example, children participating tem. Spontaneous gestures are not part of a culturally rec- in science lessons frequently use gesture to foreshadow the ognized system and thus rarely are subject to comment and ideas they themselves eventually articulate in speech59 perhaps criticism. As a result, gesture is an ideal modality within needing to express those ideas in a manual medium before which to consider for the first time notions that are not fully articulating them in words. Because the representational developed. Not only are the notions conveyed in gesture formats underlying gesture are mimetic and analog rather than likely to go unchallenged by others, but they are also likely to discrete21, gesture may permit the learner to represent ideas go unchallenged by oneself. A speaker can unknowingly that lend themselves to these formats and that are not yet de- ‘sneak in’ an idea in gesture that does not cohere well with veloped enough to be encoded in speech. Take, for example, the set of ideas the speaker routinely expresses in speech. the child described earlier who demonstrated a clear under- Gesture may be a perfect place to try out innovative ideas

427 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999 Review Goldin-Meadow – Gesture in communication and thinking

Language: Biological Constraints on Linguistic Form, Verlag Chemie Outstanding questions 4 Klima, E. and Bellugi, U. (1979) The Signs of Language, Harvard University Press • Deaf children of hearing parents can invent the rudiments of a language- 5 Lane, H. and Grosjean, F., eds (1980) Recent Perspectives on American like system without the benefit of a conventional language model, or Sign Language, Erlbaum even a willing communication partner (the children’s hearing parents are 6 Lidell, S. (1980) American Sign Language Syntax, Mouton committed to teaching them to speak). How far can an individual child 7 Lillo-Martin, D. (1991) Universal Grammar and American Sign Language: (as opposed to a society) go in fashioning a linguistic system out of Setting the Null Argument Parameters, Kluwer gesture? Are there certain types of constructions and uses that can only 8 Supalla, T. (1986) The classifier system in American Sign Language, in be introduced into a language system with group support? Noun Classes and Categorization (Craig, C., ed.), pp. 181–215, John • What are the forces that shape the particular linguistic patterns found in Benjamins the deaf children’s gesture systems? They are not shaped by conventional 9 Supalla, T. and Newport, E.L. (1978) How many seats in a chair? The language, nor are they modeled after their hearing parents’ spontaneous derivation of nouns and verbs in American Sign Language, in gestures. One possibility is that the patterns reflect the way children Understanding Language through Sign Language Research (Siple, P., structure events in general or, more specifically, the way they structure ed.), pp. 91–132, Academic Press events for the purposes of communication. 10 Corina, D. and Sandler, W. (1993) On the nature of phonological • The spontaneous gestures that speakers use along with their talk do not structure in sign language Phonology 10, 165–207 feel as though they are consciously produced – one is not typically aware 11 Perlmuttter, D.M. (1992) Sonority and syllable structure in American of having gestured. What would happen if speakers were forced to be Sign Language Linguist. Inq. 23, 407–442 aware of their gestures? Would those gestures no longer have privileged 12 Padden, C. and Perlmutter, D (1987) American Sign Language and the access to the speaker’s unspoken thoughts? architecture of phonological theory Nat. Lang. Linguist. Theory 5, • What are the contexts in which listeners are able to ‘read’ gesture? 335–375 Listeners typically have difficulty apprehending both of the messages 13 Frishberg, N. (1975) Arbitrariness and iconicity: historical change in conveyed in a mismatch (the message in speech, and the message in American Sign Language Language 51, 696–719 gesture). What determines which of the two messages will be read? Are 14 Tervoort, B.T. (1961) Esoteric symbolism in the communication behavior there conditions under which listeners can grasp both messages? of young deaf children Am. Ann. Deaf. 106, 436–480 • Does gesturing help speakers think? There is growing evidence that 15 Moores, D.F. (1974) Nonvocal systems of verbal behavior, in Language gesture can facilitate word recall and help conserve cognitive effort. Are Perspectives (Schiefelbusch, R.L. and Lloyd, L.L., eds), pp. 377–417, these effects limited to certain types of domains – the spatial domain, for University Park Press example, which the manual modality is particularly well-suited to portray? 16 Goldin-Meadow, S. et al. (1994) Nouns and verbs in a self-styled gesture system: what’s in a name? Cognit. Psychol. 27, 259–319 17 Goldin-Meadow, S., Mylander, C. and Butcher, C. (1995) The resilience simply because neither listener nor speaker is consciously aware of combinatorial structure at the word level: morphology in self-styled of the fact that those ideas do not fit. gesture systems Cognition 6, 195–262 18 Morford, J.P. and Goldin-Meadow, S. (1997) From here to there and now to then: the development of displaced reference in homesign and Conclusion English Child Dev. 68, 420–435 When gesture is called upon to fulfill the burdens of a pri- 19 Goldin-Meadow, S. (1997) The resilience of language in humans, in Social mary communication system – that is, when it is explicitly Influences on Vocal Development (Snowdon, C.T. and Hausberger, M., communicative – it takes on the forms of language, convey- eds), pp. 293–311, Cambridge University Press ing meaning by systematic combinations of discrete units. 20 Singleton, J.L., Morford, J.P. and Goldin-Meadow, S. (1993) Once is not enough: standards of well-formedness in manual communication However, when gesture is used spontaneously along with created over three different timespans Language 69, 683–715 speech – when it is not an acknowledged vehicle of commu- 21 McNeill, D. (1992) Hand and Mind, University of Chicago Press nication – it is not language-like in form, and assumes instead 22 Goldin-Meadow, S. and Mylander, C. (1983) Gestural communication a mimetic and analog format that allows it to capture ideas in deaf children: the non-effects of parental input on language development Science 221, 372–374 not easily expressed in speech. As such, the gestures that ac- 23 Goldin-Meadow, S., McNeill, D. and Singleton, J. (1996) Silence is company speech have the potential to display thoughts that liberating: removing the handcuffs on grammatical expression in the are not conveyed in speech. These speech-accompanying manual modality Psychol. Rev. 103, 34–55 gestures serve two, not mutually exclusive functions. Gesture 24 Friedman, H.S. (1979) The concept of skill in nonverbal communication: provides speakers with another representational format in implications for understanding social interaction, in Skill in Nonverbal Communication (Rosenthal, R., ed.), pp. 2–27, Oelgeschlager, Gunn & Hain addition to speech, one that can reduce cognitive effort and 25 Kendon, A. (1980) Gesticulation and speech: two aspects of the process serve as a tool for thinking. Gesture also provides listeners with of utterance, in Relationship of Verbal and Nonverbal Communication a second representational format, one that allows access to (Key, M.R., ed.), pp. 207–228, Mouton the unspoken thoughts of the speaker and thus enriches 26 Crowder, E.M. and Newman, D. (1993) Telling what they know: the communication. role of gesture and language in children’s science explanations Pragmat. Cognit. 1, 341–376 27 Garber, P., Alibali, M.W. and Goldin-Meadow, S. (1998) Knowledge Acknowledgements conveyed in gesture is not tied to the hands Child Dev. 69, 75–84 This research was supported by grants from the Spencer Foundation, the 28 McNeill, D. (1987) Psycholinguistics: A New Approach, Harper & Row March of Dimes, the National Institutes of Child Health and Human 29 Goldin-Meadow, S. and McNeill, D. (1999) The role of gesture and Development (R01 HD18617), the National Institute on Deafness and mimetic representation in making language the province of speech, in other Communication Disorders (RO1 DC00491), and the National Science The Descent of Mind (Corballis, M.C. and Lea, S., eds), pp. 155–172, Foundation (BNS 8810769). Oxford University Press 30 Church, R.B. and Goldin-Meadow, S. (1986) The mismatch between gesture and speech as an index of transitional knowledge Cognition References 23, 43–71 1 Ekman, P. and Friesen, W.V. (1969) The repertoire of non-verbal 31 Perry, M., Church, R.B. and Goldin-Meadow, S. (1988) Transitional behavior: categories, origins, usage and coding Semiotica 1, 49–98 knowledge in the acquisition of concepts Cognit. Dev. 3, 359–400 2 Kendon, A. (1981) Geography of gesture Semiotica 37, 129–163 32 Gershkoff-Stowe, L. and Smith, L.B. (1997) A curvilinear trend in naming 3 Bellugi, U. and Studdert-Kennedy, M., eds (1980) Signed and Spoken errors as a function of early vocabulary growth Cognit. Psychol. 34, 37–71

428 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999 Goldin-Meadow – Gesture in communication and thinking Review

33 Evans, M.A. and Rubin, K.H. (1979) Hand gestures as a communicative 47 Krauss, R.M., Morrel-Samuels, P. and Colasante, C. (1991) Do mode in school-aged children J. Gen. Psychol. 135, 189–196 conversational hand gestures communicate? J. Pers. Soc. Psychol. 61, 34 Stone, A., Webb, R. and Mahootian, S. (1992) The generality of gesture– 743–754 speech mismatch as an index of transitional knowledge: evidence from 48 Alibali, M., Flevares, L. and Goldin-Meadow, S. (1997) Assessing a control-of-variables task Cognit. Dev. 6, 301–313 knowledge conveyed in gesture: do teachers have the upper hand? 35 Schwartz, D.L. and Black, J.B. (1996) Shuttling between depictive models J. Educ. Psychol. 89, 183–193 and abstract rules: induction and fallback Cognit. Sci. 20, 457–497 49 Goldin-Meadow, S., Wein, D. and Chang, C. (1992) Assessing knowledge 36 Goldin-Meadow, S. (1998) The development of gesture and speech as an through gesture: using children’s hands to read their minds Cognit. integrated system, in The Nature and Functions of Gesture in Children’s Instruct. 9, 201–219 Communications in New Directions for Child Development (No. 79) 50 McNeill, D., Cassell, J. and McCullough, K-E. (1994) Communicative effects (Iverson, J.M. and Goldin-Meadow, S., eds), pp. 29–42, Jossey–Bass of speech-mismatched gestures Res. Lang. Soc. Interact. 27, 223–237 37 Alibali, M.W. et al. (1999) Illuminating mental representations through 51 Thompson, L.A. and Massaro, D.W. (1994) Children’s integration of speech and gesture Psychol. Sci. 10, 327–333 speech and pointing gestures J. Exp. Child Psychol. 57, 327–354 38 Church, R.B. et al. (1995) The role of gesture and speech communication as 52 Kelly, S. and Church, R.B. (1997) Can children detect conceptual reflections of cognitive understanding J. Contemp. Legal Iss. 6, 123–154 information conveyed through other children’s non-verbal behaviors? 39 Alibali, M.W. and Goldin-Meadow, S. (1993) Gesture–speech mismatch Cognit. Instruct. 15, 107–134 and mechanisms of learning: what the hands reveal about a child’s 53 Goldin-Meadow, S. and Sandhofer, C.M. (1999) Gesture conveys state of mind Cognit. Psychol. 25, 468–523 substantive information about a child’s thoughts to ordinary listeners 40 Siegler, R.S. and Crowley, K. (1991) The microgenetic method: a direct Dev. Sci. 2, 67–74 means for studying cognitive development Am. Psychol. 46, 606–620 54 Rime, B. (1982) The elimination of visible behavior from social 41 Goldin-Meadow, S., Alibali, M.W. and Church, R.B. (1993) Transitions interactions: effects on verbal, nonverbal and interpersonal variables in concept acquisition: using the hand to read the mind Psychol. Rev. Eur. J. Soc. Psychol. 12, 113–129 100, 279–297 55 Iverson, J.M. and Goldin-Meadow, S. (1998) Why people gesture as 42 Siegler, R.S. (1994) Cognitive variability: a key to understanding cognitive they speak Nature 396, 228 development Curr. Dir. Psychol. Sci. 3, 1–5 56 Rauscher, F.H., Krauss, R.M. and Chen, Y. (1996) Gesture, speech, and 43 Thelen, E. (1989) Self-organization in developmental processes: can lexical access: the role of lexical movements in speech production systems approaches work?, in Systems and Development: The Minnesota Psychol. Sci. 70, 226–231 Symposium on Child Psychology (Gunnar, M. and Thelen, E., eds), 57 Alibali, M.W. and DiRusso, A.A. The function of gesture in learning to pp. 77–117, Erlbaum count: more than keeping track Cognit. Dev. (in press) 44 Piaget, J. (1952) The Child’s Conception of Number, Humanities Press 58 Goldin-Meadow, S. Giving the mind a hand: the role of gesture in 45 Strauss, S. (1972) Inducing cognitive development and learning: a review cognitive change, in Mechanisms of Cognitive Development: Behavioral of short-term training experiments: I. The organismic developmental and Neural Perspectives (McClelland, J. and Siegler, R.S., eds), Erlbaum approach Cognition 1, 329–357 (in press) 46 Kendon, A. (1994) Do gestures communicate?: a review Res. Lang. Soc. 59 Crowder, E.M. (1996) Gestures at work in sense-making science talk Interact. 27, 175–200 J. Learn. Sci. 5, 173–208

TICS Editorial Policy

Update Monitor

Trends in Cognitive Sciences is indispensable reading for all interested in Summaries of recently published papers of interest to cognitive scientists. Readers who would like to contribute to this section, by identifying appropriate papers and writing short summaries, should contact the Editor. this diverse field. The journal carries a core of authoritative Reviews, Facing up (or down) Musical There is an active ongoing debate re- garding the role of the fusiform region brainwaves of the ventral occipitotemporal cortex. When people read or hear syntactically The recent increase in the use of fMRI faces, and two-tone Mooney images incongruous sentences, such as: ‘Yester- has provided substance for this debate, (high-contrast black–white images of day, he mow the lawn’ their averaged leading to the hypothesis that an area in faces). The results strongly support the brain activity recorded at the scalp written in an easily accessible style by leading authors, summarizing the fusiform gyrus (FG) is specialized for hypothesis of a ‘face area’ while demon- (event-related potentials, or ERPs) will the perception of human faces. Nancy strating that the FG responds more show a P600 waveform (a positive com- Kanwisher strongly to inverted grayscale faces than et al. ponent of the ERP) at the incongruous tional imaging datareported that seemedsome func- to to inverted two-tone Mooney images. word. It is still a matter of debate demonstrate that there might indeed be Although a face-specific area in the FG is whether this brainwave component is a ‘face area’ in the FG (Ref. 1). The au- demonstrated by these studies, it is likely restricted to language-related input or thors gave subjects four different per- that there are other face-responsive reflects more general cognitive pro- the exciting developments of the subjects you need to know about. ceptual tasks and recorded a signifi- areas yet to be discovered. Undoubtedly, cesses. Patel cantly greater response in a region of functional imaging studies will continue dence for the etlatter al.1 position. They had the FG when the subjects were shown to play an important role in the investi- provide some evi- 15 musically educated subjects listen face stimuli compared with stimuli of gation of the visual processes underlying to sequences of musical chords. Some common objects. The authors also found the recognition of human faces, and of these chords were moderately or a stronger response when subjects were these in turn could help to guide the de- highly incongruous in terms of har- shown full front-view face photos than velopment of the next generation of mony and key-relatedness given the Accompanying the Reviews are a range of other types of article. The when shown front-view photos of models of brain function, incorporating preceding ones. The same subjects also houses. Three-quarter-view face photos face-recognition mechanisms. listened to English sentences, some of elicited a greater response than photos References which included words that were either of human hands, and intact faces a 1 ungrammatical or hard to integrate greater response than scrambled two- Kanwisher, N. et al. given the preceding syntactic structure. tone faces. Recently, Kanwisher area: a module in human(1997) extrastriate The fusiform cortex face For both linguistic and musical stimuli, conducted an additional study specialized for face perception the incongruous targets elicited a P600 Comment section offers short updates and discussions of one or ing both upright and inverted grayscaleet al. 17, 4302–4311 J. Neurosci component in the ERP. Moreover, the 2 involv- 2 Kanwisher, N. . inversion on theet al.human(1998) fusiform The effect face of area face degree of congruency had similar ef- fects on the P600 amplitude for both Attentional suppressionCognition in human 68, B1–B11 domains. These results suggest that the extrastriate cortex P600 does not reflect language-specific processes, but rather structural inte- What structures and processes mediate gration processes that are not domain- two ground-breaking and important primary papers. The authors of the operation of selective visual atten- specific. The authors point out that tion in the human brain? One hypothesis there are cases of brain damage that suggests that object representations at was directed to one of the four periph- selectively impairs the perception of different locations compete for process- eral stimuli. The authors argue that these musical harmonic relationships with- ing capacity in the visual system, and findings are consistent with the notion out affecting syntactic knowledge, and that attention works by biasing this that attention is protecting a represen- the opposite is also true. However, competition in favour of the attended tation of the target item from competi- these deficits might be explained in the primary papers have the opportunity of replying to the location. Recordings of single cells in tion. It is also possible, though, that the terms of damage to a domain-specific awake monkeys have been consistent difference between simultaneous and knowledge base as opposed to do- with this idea successive stimulation might have arisen 1 main-unspecific structural integration to an optimal . stimulusThe response is reduced of a neuron sub- from other factors, such as the differ- processes. Identifying the neural corre- stantially when an irrelevant stimulus is ence in presentation rate between the lates of harmonic processing and syn- Books presented simultaneously at another lo- two conditions. However, in a control tactic processing independent of inte- etcetera cation in the receptive field. However, if experiment, the authors presented gration processes should allow a better Comments, thus extending the dialogue on their work. Monitor the animal directs its attention to one of stimuli at a constant rate and showed understanding of the treatment of the the competing stimuli in the receptive that the response to a single peripheral Talking Nets: An Oral Historydomain-specific disorders. field, the responses are as large as when item presented alone was lowered when the stimulus is presented alone. Now, it was presented simultaneously with of Neural Networks Reference Kastner and colleagues provide evidence three other items. This suggests that the difference between successive and simul- 1 Patel, A.D. that a similar mechanism might operate et al. (1998) Processing syntactic in humans edited by James A. Andersontaneous andstimulation Edward is Rosenfeld, not simply relatedMIT Press, 1998.relations in language and music: an event- 2. Using fMRI, they examined to different presentation rates. These related potential study pieces summarize, in 100–200 words, recently published cortical responses to four£31.95/$39.95 adjacent ob- (xi + 434 pages) ISBN 0 262 01167 0 jects presented either simultaneously, findings address directly the mechanisms 717–733 J. Cogn. Neurosci or one at a time in rapid succession. of directed attention. Further investi- . 10, gation will explore the intriguing par- Although the total amountWhat of is it about neural networks that and Bertrand Russell is worth the price Review allels stimulation (integrated overmakes time) people was so excitable?between single-cellof and the functional- book (for which you get several imaging results revealed here. the same in the two experimentalJamesretinal con- Anderson, a noted neural- versions). The classic McCulloch and ditions, cortical responses differed. . – Concepts and categorization network researcher,References and Edward Pitts papers were written when Pitts et al papers. Meeting reports convey the excitement Simultaneous presentationRosenfeld, evoked less a journalist, had the splen- was still in his teens and living with Solomon 1 Moran, J. and Desimone, R. (1985) Selective activity than successive presentation,did idea andto generate an oral history of McCulloch and his wife. PittsStudents subse- this difference was more pronounced in attention gates visual processing in the neural-net research by talking to most quently gravitated to a second father- higher cortical areas. Moreover, the re- extrastriate cortex of the pioneering figures in the field. figure, Norbert Wiener. Wiener and proven mathematically, that neural duction with simultaneous presentation 2 Kastner, S., De Weerd,Science P., Desimone, R. and The result, Talking Nets, is an extra- 229,McCulloch 782–784 had a falling out because networks cannot ever do anything in- was much less severe when attention Ungerleider, L.G. (1998) Mechanisms of Subscribe to ordinary document that is a page- Mrs. Wiener didn’t like McCulloch; Pitts teresting.’ Widrow, in contrast, says, ‘I directed attention in the human extrastriate TICS turner for cognitive scientists. was caught in the middle, had a ner- couldn’t understand what the point of of recent conferences or seminars, focussing on 90 cortex as revealed by functi at 50% discount usi Physics is a science. Entymology is a vous breakdown, and drank himself to [Perceptrons] was, why the hell they 282 108 1 1364 661 science. ‘Neural networks’ is a toolbox an early grave. And we haven’t even did it. I figured that they must have Concepts do more than – a collection of tools for the simula- gotten to Minsky and Papert yet. gotten inspired to write that book re- tion and analysis of complex systems. The tale of how Minsky and Papert ally early on to squelch the field, to do These tools were developed by people slew the neural-net dragon in the what they could to stick pins in the bal- categorize working in diverse fields ranging from 1960s is the most important legend in loon. But by the time the book came the latest developments and the discussions control theory and signal processing to neural-net culture. The testimony in out, the field was already gone. There neurobiology and psychology. They are Talking Nets suggests a more compli- was just about nobody doing it.’ being vigorously applied to an equally cated story, however. It is true that So what really happened? It seems Karen O. Solomon, Douglas L. Medin and Elizabeth Lynch broad range of problems ranging from Minsky and Papert come off badly in clear that people such as Widrow had understanding basic brain mechanisms this telling of the tale (of course, it isn’t become discouraged because they to forecasting the fluctuations in the their side that is being told). The MIT knew that multilayer nets were needed bond market. This book recapitulates people seem to have been bothered by but didn’t know how to train them. His surrounding them. Opinion articles are reviews Concepts underlie all higher-level cognitive processes. Until recently, the study of much of the convoluted history of publicity over neural-network research testimony on this point is poignant: neural-network research through the being conducted elsewhere (Widrow: concepts has largely been the study of categorization. But categorization is only one testimony of people who figured cen- ‘You know, we had a lot of controversy ‘We would have given our eye conceptual function among several. We argue that concepts cannot be understood trally in its development. It is a remark- in the early days. It was due to publicity teeth to come up with some- able story, replete with drama, that [Frank] Rosenblatt had in the thing like Backprop…Backprop sufficiently through the study of categorization, or any other function, in isolation, for tragedy, hubris, irony, humor, bitter in- news media and publicity that I had…I to me is almost miraculous. two important reasons. First, concepts serve multiple functions which interact to affect tellectual warfare, and a couple of found that this kind of publicity infuri- The first exposure I had to of subjects written from a personal slant, and corpses. It is told in the words of a bril- ates colleagues…’). One has to imagine Backprop was around 1985 at conceptual structure and processing. Second, studying a single function in isolation liant and eccentric collection of people, what transpired the day that a meeting at Snowbird, Utah… encourages one to see cognitive processes that are particular to each function, but including several certifiable geniuses Rosenblatt, the psychologist from Someone gave a paper in the and a couple who are merely certifi- Cornell, came down to MIT to tell an first morning session and during discourages the discovery of processes that are common to multiple functions. For able, as Groucho would have said. (The audience that included Minsky, the question period…someone got up and said, “You know, these two reasons, we suggest that concepts should instead be studied in the context interviewees comprise Bernard Widrow, Shannon and McCulloch the news Carver Mead, Stephen Grossberg, about perceptrons. (Cowan: ‘It was a something like that was done therefore highlight some of the controversies in of a system of interrelated functions. Michael Arbib, James Anderson, David terrible lecture. McCulloch didn’t say by Widrow back in the early life, and in the following section discuss interrelations Rumelhart, Geoff Hinton, Terry anything. Shannon said, ‘It’s worth ‘60s.” They began to have this among these functions. Sejnowski and nine others.) The ac- looking at’… But by and large it was big discussion about what Most researchers assume that conceptual represen- counts aren’t all consistent with each clear that the perceptron wasn’t doing Widrow did and Widrow did- other and can’t all be equally true. the things that Frank claimed it could n’t do, and I’m just sitting Concepts are the building blocks of thought. How con- tations include procedures for identifying whether an entity Yet in reading them, one gets a vivid do.’) Rosenblatt had the even greater there, listening to all this cepts are formed, used, and updated are therefore, central is a member of a category, a process often referred to as sense of where people came from, misfortune to die soon thereafter in a stuff…I was like a dead man. I was a man who’d died, who cognition. Books etcetera features stimulating questions in cognitive science. The literature on concepts is categorization. Categorization is not an end in itself, but what they thought they were doing, boating accident. and how their ideas developed. It’s also The ironies here are staggering. was sitting up on a cloud vast (see Refs 1–3 for reviews and Refs 4–7 for recent edited rather it serves to connect old to new: categorizing novel en- ‘toothbrush’ a lot of fun. Minsky’s reaction to the publicity over somewhere, looking down on volumes) and impossible to summarize in a short review. tities allows the cognitive system to bring relevant previous At one level, the history of neural- perceptrons has to be considered in the Earth, watching what hap- For this reason, we focus on a single issue – namely, the idea knowledge to bear in the service of understanding the novel network research can be read as a light of his subsequent willingness to pened after he died.’ (p. 61) that concepts have multiple functions which interact to affect entity. Recognizing some unusual shape as a long-running soap opera with a cast of serve the mass media as a kind of all- allows one to understand its parts and their functions. A characters that is a little strange even purpose, techno-egghead commenta- People also had to wait for com- essay-reviews of recent publications, whether conceptual structure and processing. Concepts serve mul- for a group of academics. The ap- tor on the issues of the day (e.g. ‘The puter power to catch up with the ideas, tiple functions, and, as we will see, these functions are not related function is inference: knowledge of category proach originated with some papers by future of money’, Discover, 1998; ‘Is a point that Arbib emphasizes. Widrow independent of one another; rather, they interact with and membership supports predictions about behavior. For Turing (on biological dynamic systems), the body obsolete?, Whole Earth and Rosenblatt were experimenting with influence each other. We believe that these interactions example, medical diagnostic categories allow physicians Wiener’s ‘cybernetics’, and the crucial Review, 1989). And if the perceptron physical networks created out of elec- to predict what sorts of treatments will be effective. articles by McCulloch and Pitts treating was not able to do all the things that tric motors and potentiometers (some undermine the popular strategy of studying categorization, neurons as computational devices, were being claimed, how would that good to three decimal places!). Com- or any other function, in isolation. Moreover, these inter- Concepts are also crucial in explanation and reasoning. which in turn greatly influenced von compare to the years of relentless hype puters were primitive, limiting the use books, software, CD-ROMs, films, etcetera. And a actions suggest that concepts should be studied in the con- Having categorized a young man as a ‘football fan’, one Neumann, who was about to invent about artificial intelligence? of simulation as a tool; luckily things text of a system of interrelated functions. might be able to explain why he is walking down the street the modern computer. McCulloch was On the other hand, did it matter? were to move ahead rapidly on that a neurophysiologist at the University of Interestingly, opinions vary. Robert front. In the meantime, some interest- In this article, we first outline some common functions of bare-chested with blue and yellow paint on his face bellow-ball 8. For example, our football fan K.O. Solomon,Chicago. Pitts was a prodigy and misfit Hecht-Nielsen thinks that Minsky and ing work was done and the people in ing out the Michigan rouser. Categories are also used to in- 1 concepts. Then, we turn to recent research that demonstrates D.L. Medinwho and hung around the university but Papert’s book Perceptrons mattered this book managed to survive quite stantiate goals in planningcategory of things to bring to a foot wasn’t actually enrolled. The story of because it created ‘a new conventional well, ending up with positions at places that these functions indeed do interact with one another. ad hoc E.Lynch are at the might create an how Pitts met McCulloch via Carnap wisdom that some MIT professors have like Berkeley, Stanford, and CalTech. list of books received for review will be published Finally, we conclude with an example of how an integrated Department of approach can encourage the discovery of cognitive processes game (e.g. beer, binoculars, seat-cushion and transistor Psychology, that extend across multiple functions. radio). Northwestern Not only are new entities understood in terms of old, 121 University, 2029 Trends in Cognitive Sciences – Vol. 3, No. 3, March 1999 Functions of concepts but new entities also modify and update concepts. That is, Sheridan Road, A concept can be very difficult to define. However, in this concepts support learning. Thus, encountering a member Evanston, IL 60208, regularly. Letters stimulated by any of the articles paper, we will refer to a concept as a mental representation of a category with a novel property (e.g. an electric tooth- USA. brush) can result in that novel property being incorporated that is used to meet a variety of cognitive functions. The most tel: +1 847 467 6513 commonly studied function has been categorization, a pro- into the conceptual representation. The learning function is fax: +1 847 491 7859 cess by which mental representations (or concepts) are used also associated with a number of questions, such as how e-mail: k-solomon@ nwu.edu to classify entities. Recently, there has been a trend toward much weight to give new information versus old, when to the study of two other conceptual functions, inference, and set up subcategories, and so on. Moreover, given that in- 99 published in TICS are welcome. Authors will be stances can belong to multiple overlapping categories there conceptual combination. But, as we shall see, concepts are used to perform many cognitive tasks, and the boundaries is the issue of whether new information should modify all between conceptual functions blur as functions interact and influence each other. In this section we will briefly illustrate a number of ways in which concepts are used in everyday offered the chance to reply to any points raised. 1364-6613/99/$ – see frontTrends matter © in 1999 Cognitive Elsevier Science. Sciences All rights – reserved. Vol. 3, PII:No. S1364-6613(99)01288-7 3, March 1999

429 Trends in Cognitive Sciences – Vol. 3, No. 11, November 1999