University of Pennsylvania ScholarlyCommons Center for Human Modeling and Simulation Department of Computer & Information Science October 1998 Gesticulation Behaviors for Virtual Humans Liwei Zhao University of Pennsylvania Norman I. Badler University of Pennsylvania, [email protected] Follow this and additional works at: https://repository.upenn.edu/hms Recommended Citation Zhao, L., & Badler, N. I. (1998). Gesticulation Behaviors for Virtual Humans. Retrieved from https://repository.upenn.edu/hms/21 Copyright 1998 IEEE. Reprinted from Sixth Pacific Conference on Computer Graphics and Applications, 1998, pages 161-168. Publisher URL: http://dx.doi.org/10.1109/PCCGA.1998.732100 This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of the University of Pennsylvania's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to [email protected]. By choosing to view this document, you agree to all provisions of the copyright laws protecting it. This paper is posted at ScholarlyCommons. https://repository.upenn.edu/hms/21 For more information, please contact [email protected]. Gesticulation Behaviors for Virtual Humans Abstract Gesture and speech are two very important behaviors for virtual humans. They are not isolated from each other but generally employed simultaneously in the service of the same intention. An underlying PaT-Net parallel finite-state machine may be used to coordinate them both. Gesture selection is not arbitrary. Typical movements correlated with specific textual elements are used to select and produce gesticulation online. This enhances the expressiveness of speaking virtual humans. Keywords virtual human, agent, avatar, gesture, posture, PaT-Nets Comments Copyright 1998 IEEE. Reprinted from Sixth Pacific Conference on Computer Graphics and Applications, 1998, pages 161-168. Publisher URL: http://dx.doi.org/10.1109/PCCGA.1998.732100 This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of the University of Pennsylvania's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to [email protected]. By choosing to view this document, you agree to all provisions of the copyright laws protecting it. This conference paper is available at ScholarlyCommons: https://repository.upenn.edu/hms/21 Gesticulation Behaviors for Virtual Humans Liwei Zhao and Norman I. Badler Center for Human Mo deling and Simulation Department of Computer and Information Science UniversityofPennsylvania Philadelphia PA 19104-638 9 USA [email protected] enn.edu, [email protected] enn.edu 1-215-898-5 862 phone; 1-215-573- 745 3 fax Abstract Beats are small formless waves of the hand that o ccur with heavily emphasized words, o ccasions Gesture and speech are two very important behaviors of turning over the o or to another sp eaker, and for virtual humans. They are not isolatedfrom each other kinds of sp ecial linguistic work. other but general ly employed simultaneously in the ser- While Cassell's system implemented instances of each vice of the same intention. An underlying PaT-Net typ e of gesture, the most prevalentwere iconics linked paral lel nite-state machine may be usedtocoordinate to mentions of sp eci c ob jects, metaphorics linked to them both. Gesture selection is not arbitrary. Typi- sp eci c actions, and b eats linked to sp eechintonation. cal movements correlated with speci c textual elements Following Cassell's lead, new problems in gesture are used to select and produce gesticulation online. generation were exp osed. This enhances the expressiveness of speaking virtual hu- mans. 1. Coarticulation: Generating a smo oth transition Keywords: Virtual Human, Agent, Avatar, Gesture, from one gesture to the next without returning Posture, PaT-Nets to a sp eci c rest p ose. 2. Expression: Mo difying the p erformance of a ges- ture to re ect the agent's manner or p ersonality. 1 Intro duction 3. Spatialization: Integrating a deictic gesture into the surrounding context. The past few years have seen several research e orts on human gestures i.e. [1,2, 5, 8, 27, 18, 16, 13]. Many 4. Selection: Generating a metaphoric that mightbe of these pro jects have fo cused on interpreting human asso ciated with an abstract concept. gestures for interactive control. Creating appropriate Problem 1, coarticulation, has b een addressed bya gestures in a virtual human has not b een as well studied numb er of computer graphics researchers [8,12, 28, b ecause the range of gestures p erformed during sp eech 26], although the issue has other asp ects suchas output is much larger than a symb olic selection set used preparatory actions which remain unsolved. Prob- for discrete inputs. For example, in [8] four gesture lem 2, expression, is b eing investigated at a number typ es are distinguished: of places [6, 33, 10]. In this pap er, weinvestigate problems 3 and 4. Of these two, spatialization is eas- Iconics represent some concrete feature of the ac- ier, since the desired gesture is combined or comp os- companying sp eech, such as an ob ject's shap e. ited with inverse kinematics to p oint or align the ges- turing b o dy part with the spatial referent. Selection Metaphorics represent an abstract feature concur- entails determining gestures that p eople would likely rently sp oken ab out. interpret and accept as \natural" and \representative." These concepts are orthogonal: a naturally p erformed Deictics indicate a p oint in space, and may refer to motion captured gesture might not b e appropriate to p ersons, places and other spatializeable discourse entities. the sp eech text, while a synthesized less natural arm motion might nevertheless b e representative of the ex- tures and sp eech. Kendon [20] o ers a distinction b e- pressed concepts. tween autonomous gestures gestures p erformed with- out accompanying sp eech and gesticulation gesture The selection problem itself splits into two: one is p erformed concurrently with phonological utterance. the creation of the gestural motion and the other is the Gestures and sp eech are closely asso ciated together. mapping from the textual content to the gesture. For They are generally employed simultaneously in the ser- example, to create a character waving hello during a vice of the same intention. Well-co ordinated gestures greeting, one has to create the waving motion as well and sp eech enhance the expressiveness and b elievabil- as know when to invoke it up on encountering a greet- ity of sp eaking virtual humans. In this pap er, we re- ing context. In this work we assume that the motions strict our investigation to gesticulation. themselves are generated byinverse kinematics, mo- tion capture, or otherwise pre-created e.g. key p ose sequences. Our contribution lies in prop osing a repre- 2.1 Gestures sentative mapping from concepts to gestures such that they are selected based on stylized rhetorical sp eaking. The study of gestures in dance and oratory may date To select and spatialize various gestures correlating back to the b eginning of seventeenth century [7]. More sp eech and language, we use an underlying co ordina- recently, semioticists from the elds of anthrop ology, tion scheme called PaT-Nets [3]. The virtual human neurophysiology, neuropsychology and psycholinguis- 1 animation is implemented as an extensions to Jack . tics Freedman [17]; Wiener, Devo e, Rubinow and The inputs see b elow to the system are in the form of Geller [34]; McNeill and Levy [24] have b een inter- sp eech texts with emb edded commands, most of which ested in the study of gestures. The Lexis dictionary are related to gestures. The gestures are controlled 1977 gives the most general de nition of gesture | by PaT-Nets to coincide with the utterance of the \movements of b o dy parts, particularly the arms, the sp eech. While the emb edded commands in our exam- hands or the head conveying, or not conveying, mean- ples are manually inserted for now, the idea is to detect ing." the presence of the corresp onding concepts in the raw While gestures are the \little" movements that are text stream and automatically insert the deictics and con ned to a part or parts of the b o dy, if just con- metaphorics based solely on the words used. sidered in isolation they havevery limited contribu- tion to make to non-verbal communication. Emblems warning welcome. Hello, ngest nhead front Currently, I can supp ort following basic arm gestures. and manual languages, such as American Sign Lan- Now let me intro duce you some simple ob jects I know: idxfftable.table.cornerg this is a table np oint guage, are exceptions b ecause the communication is np oint idxffdo or.do or.panelg this is a do or idxffchair1.chair.redg this is red chair np oint fully b orne bymovements. Gestures are rarely p er- np oint idxffchair0.chair.yellowg this is yellowchair slant right Let me showyou the basic arm gestures nhead formed outside a communicative context and only o cca- ngest arm reject arm reject gesture ngest arm unlikely arm unlikely gesture sionally transmit any depth of emotion or information, arm not arm not gesture ngest ngest arm improbable arm improbable gesture since, as so on as there is any complicated meaning, the ngest arm doubtful arm doubtful gesture arm probable arm probable gesture ngest gestures can only b e \read" in relation to the whole ngest arm tis arm it is gesture expressivemovement of the b o dy [14,9,22].