Social Behavior, Emotion and Learning in a Pack of Virtual Wolves

Bill Tomlinson and Bruce Blumberg

Synthetic Characters Group, The Media Lab, MIT 77 Massachusetts Avenue, NE18-5FL, Cambridge, MA 02139-4307 USA {badger | bruce}@media.mit.edu

Abstract We are creating a pack of virtual creatures who exhibit the One species of animal that is renowned for its social kinds of social interactions found in a natural species of organizations is the gray wolf (Canis lupus). (Mech et al. animal, the gray wolf (Canis lupus). To do this, we are 1998) Wolves exhibit an array of interesting social extending our synthetic character building toolkit to enable phenomena; for example, they live and hunt in packs, they our characters to have and express emotional states; to form display hierarchical social relationships that facilitate the context-specific emotional memories; and to learn to adapt allocation of scarce resources and they communicate pre-existing behaviors for use in novel social contexts. We through a variety of sensory channels (e.g., howling, scent describe how these elements combine to form the marking, body posture). In the Synthetic Characters Group underpinnings for our interactive installation entitled at the MIT Media Lab, we are in the process of creating a “AlphaWolf”, shown at SIGGRAPH 2001. The computational representations that allow social learning in pack of virtual wolves (see Figure 1) who exhibit similar our virtual wolves demonstrate the intimate connections social behavior to real wolves. among social behavior, emotion and learning. These We are giving our wolves a variety of social competences, representations are applicable to building a wider range of including the ability to learn to use behaviors in novel socially intelligent agents. We hope that our studies of virtual wolves will offer insight into the processes by which real wolves and other animals understand their environments.

Introduction “The submissive activity is, in its essence, an activity of the cub.”? Rudolf Schenkel, “Submission: its features and function in the wolf and dog.” American Zoologist. 1967, p. 325

Social behavior is often evolutionarily advantageous in the natural world. Many species of animals live part or all of their lives in social groups, from the transient annual assemblages of elephant seals at their breeding grounds to Figure 1: A wolf pup and his father howl together. the cities and towns of human cultures. Various researchers have studied synthetic social systems, including social contexts, to express their emotional states and to flocking in birds (Reynolds 1987), schooling in fish (Tu & respond to other wolves in socially appropriate ways. These Terzopoulos 1994), virtual gorillas (Allison et al. 1996), social abilities are made possible by our models of action- chimpanzees (teBoekhorst & Hogeweg 1994) and primate- selection, emotion, motor control, learning and context- like artificial agents (Hemelrijk 1999). Research in specific emotional memory formation. By taking an animal synthetic social agents allows us to harness the benefits of model and trying to replicate the social phenomena found in natural social behavior for use in computational systems, that species, we hope to shed light on computational and may help us understand how nature works. mechanisms that could enable social competence, learning and development in a wider range of socially intelligent Copyright © 2001, American Association for Artificial Intelligence agents (SIAs, from Dautenhahn 2000) and other (www.aaai.org). All rights reserved. computational systems. As Dautenhahn points out, “[a]rtificial social agents (robotic or software) which are supposed to interact with humans are most successfully the meat, which the pups excitedly consume. (Schenkel designed by imitating life.” (1999, cited in Dautenhahn 1967) 2000, p.37) In addition, our experiments in simulation may be of some use to the biological community in framing Wolf social behaviors appear to be derived from other their experiments with real animals. behavioral patterns exhibited by wolves. (Schenkel 1967, Fox 1971) For example, there are two main types of In order to ensure that our work accurately represents the submission that wolves exhibit – passive submission and biology and ethology of wild wolves, we begin this paper active submission. Passive submission involves a wolf with a description of wolf biology and social behavior. lying on his side or back, exposing the ventral side of his Next, we analyze the computational representations that we chest. The ears are held close to the head, and the tail is believe are necessary to enable social learning in our virtual tucked between the legs. These behavioral patterns bear a wolves, with consideration of the biological literature and resemblance to infantile behaviors involved in reflex other models, both theoretical and computational, that have urination (in which a pup urinates when his mother licks his been proposed. Following that, we discuss an interactive belly). Active submission involves a crouched posture with installation that is currently under development as a result backward directed ears, and licking or pecking the mouth of of this research project. We close with some ideas for the dominant wolf. This behavior is very similar to the future work and a consideration of how this work can food-begging behavior of pups described above. Similarly, impact the broader field of socially competent dominant behaviors appear to be a form of “ritualized computational systems. fighting” (Golani and Moran 1983).

The work described in this paper is made possible by a range We chose the gray wolf as the model for our simulation for of other research that is taking place simultaneously in the several reasons. First, they manifest distinct social Synthetic Characters Group. While there is not space to phenomena that are complex enough to be interesting, yet describe each project here, we nevertheless want to mention clear enough to provide direction for our simulation. research in behavior system design (Isla et al. 2001), motor Second, wolves are closely related to the domestic dog, for control (Downie 2001), and learning (Blumberg 2001), all which we have a strong conceptual and technical base as a of which are essential to our wolf social simulation. result of previous installations that we have done featuring virtual dogs. Finally, the social behaviors of wolves are similar enough to those of humans that some of the lessons The Gray Wolf we learn from wolves might be relevant to human social behavior and simulation. In their natural environment, gray wolves form complex social groups called packs. The core of most packs is a family – a breeding pair of adults, their puppies, and Computational Models sometimes a few adult offspring of the breeding pair (Murie 1944, Mech et al. 1998). The average pack size is There are a variety of computational elements that must be approximately 7-9 individuals, but some packs may contain in place for virtual wolves to interact socially in a way more than 25 wolves. Larger packs may contain more than resembling real wolves. Our virtual wolves must be able to one breeding pair. Most young wolves disperse from their choose different behaviors; to move around their world; to natal pack in their second or third year to find a mate and learn that certain interactions lead to positive results while begin their own pack. (Mech et al. 1998) others lead to negative repercussions. These components are already functional parts of our character-building Wolves communicate with each other in a variety of ways. toolkit, and have been described elsewhere. (Isla et al. 2001, They have a wide array of vocalizations, including Downie 2001, Burke et al. 2001, Blumberg 2001) “whimpering, wuffing, snarling, squealing and howling”. (Zimen 1981, p. 68) They express their intentions and The fundamental representation of an action in our system motivational and emotional states through body posture as is the ActionTuple. Each ActionTuple consists of four well – a mother wolf assumes different postures with her components: the action itself; a TriggerContext, which pups than she does with her mate. The sense of smell is determines when the action will take place; a also integral to wolf social behavior (e.g., scent marking). DoUntilContext that determines when the action will cease; In wolves, as in most social creatures, communication is and an object to which the action will happen. central to the social relationships that are formed. ActionTuples are arranged into ActionGroups that determine which actions are mutually exclusive and which When the pack has young pups, the adult wolves travel can be run simultaneously. In accord with Thorndike’s Law away from the den to hunt, and carry back meat in their of Effect (Thorndike 1911), ActionTuples that are stomachs to feed the pups. Upon their return, the pups correlated with positive results will be chosen more perform stereotypical food-begging behavior, in which they frequently in the future. Each action is executed in an crouch in front of an adult and lick or peck at the adult’s emotional style, which we discuss below. muzzle. This pup behavior incites the adult to regurgitate The main extensions to our system for this project are: the a trigger that causes growling to occur, and (1 - D) will ability to learn to use behaviors in novel social contexts; the factor into the trigger that makes the wolf submit. ability to have and express emotional states; and the ability to form context-specific emotional memories. We will In addition, the emotional states of our wolves feed into our describe each of these in greater depth below. motor control system (Downie 2001) and affect the style in which they take their actions. This system is based on the Having and Expressing Emotional States “verbs and adverbs” system of Rose (1999), in which an In order to simulate emotional virtual characters, it is action (a “verb”) is taken in a certain style (an “adverb”). necessary to choose a computational representation that Because of our motor control system, which can blend captures the range of emotional phenomena we wish our between example animations, our animators need only characters to exhibit. Much research has already been done create a few extreme emotional styles of each action (for both in understanding emotions and in simulating them example, one high Arousal walk and one low Arousal walk) computationally. Darwin’s ideas about emotions (Darwin to get the full dynamic expressive range between those 1877) form the basis for much of modern research into examples. understanding emotions scientifically.

For the AlphaWolf project, we have considered two main emotional paradigms – a dimensional approach and a categorical approach. The dimensional approach (e.g., Schlosberg 1954, Plutchik 1991, Russell 1997, Smith 1989) maps a range of emotional phenomena onto explicitly dimensioned space. Various researchers have implemented versions of the dimensional approach; for example, Breazeal (2000) maps a 3-dimensional space (Arousal, Valence, Stance) onto a set of emotions that influence both the behavior system and the motor system.

The categorical approach separates emotional phenomena into a set of basic emotions – for example, fear, anger, Figure 2: The wolf pup play-bows at his father. sadness, happiness, disgust and surprise (Ekman 1992). In her 1998 paper, Dolores Cañamero asks: “What does this Ekman’s model provided the basis for an implementation by particular (type of) agent need emotions for in the concrete Velasquez (1998); others (e.g., Gadhano & Hallam 1998) environment it inhabits?” (p.54) In our case, the answer to have also implemented categorical models. For a far more this question is twofold: first, as we have just mentioned, comprehensive discussion of emotional models in the emotional state allows the virtual creature to express computational systems, the reader is directed to Rosalind itself to other individuals in its world. This has the Picard’s book, Affective Computing (1997). beneficial side effect that a person watching the creatures interact can also tell what emotional state is motivating the A dimensional approach better captured the range of creature’s actions. The second part of the answer will be emotional phenomena that we wanted our wolves to addressed more in the next section – emotion is central to display. Our emotion model is based most directly on the the mechanism by which our creatures remember their Pleasure-Arousal-Dominance model presented by relationships with other individuals. Thus, emotion is Mehrabian and Russell (1974). At each moment, a wolf has central to our creatures’ social interactions. three continuous variables describing a 3-dimensional emotional space. Emotional categories may also be mapped Context-Specific Emotional Memories onto these axes; for example, anger is equivalent to low A component of real wolf social behavior is that submissive Pleasure, high Arousal and high Dominance. wolves know to submit before the dominant wolf even

reaches them. (Schenkel 1967) Similarly, our virtual The value for each variable is affected by the wolf’s wolves need some way of triggering submissive behaviors previous emotional state, by some drift that each emotional before they are actually pinned to the ground by a dominant axis undergoes, and by attributes of his surrounding individual. How can our wolves remember previous environment. For example, imagine an example involving a interactions that they have had? In our simulation, we father wolf trying to fall asleep (see Figure 2). The drowsy utilize context-specific emotional memories (CSEMs) for father wolf’s Arousal might be fairly low, and drifting this purpose. CSEMs cause a wolf who is experiencing a lower as he falls asleep, but might be forced higher by the suite of environmental stimuli (for example, a dominant insistent attentions of his puppy. wolf) to return to an emotional state similar to the one he The emotional state of our virtual wolves affect the action experienced on previous encounters with those stimuli. In selection mechanism by feeding into the TriggerContexts. addition to storing the values of Pleasure, Arousal and For example, a wolf’s dominance value (D) will factor into Dominance that the wolf was feeling when he last rewarded and those in which he was not rewarded, and only experienced the relevant stimuli, the CSEM also features a propagated the value of the treat to those contexts in which Confidence value that reflects how reliable he believes the a reward occurred. This is the essence of the mechanism by CSEM’s values to be. Each continuously changing CSEM which our wolves will learn to perform certain behaviors in effectively reflects the interaction history between the wolf the correct context. and some bit of his context, without the need for specific memories of past interactions. Returning to the example above (see Figure 2), if the drowsy father wolf’s puppy starts yapping at him, the father Our CSEMs are based on the “somatic marker hypothesis” might get up and discipline the pup. The father’s presented by Damasio (1994), in which he proposes that disciplining will continue until he is satisfied, perhaps when people attach emotional significance to stimuli that they the pup submits. If submission behavior tends to cause the encounter in their environment, and then re-experience that father wolf to relent, the pup should rapidly learn to submit emotion when they encounter those stimuli on future in the context of being disciplined. occasions. Other researchers have implemented models of emotional learning or memory, for example “affective tags” A related behavioral component of our virtual wolves is (Yoon et al. 2000), “emotional memories” (Velasquez that dominant individuals need to be willing to relent once 1998), and the work of Kitano (1995), Gadhano & Hallam the submissive individual actually submits. Since the (1998) and Balkenius (2000). dominant individual provides the reward function from which the submissive learns, it is necessary for that Our wolves may form multiple CSEMs for one object, dominant individual to have some computational depending on the context in which the wolf has encountered mechanism that causes him to relent. Possibilities include a that object. For example, a wolf pup might form one hard-coded “evolutionary taboo” against hurting things that CSEM about his father when the father smells of meat (i.e., act like pups or a learned association that a submitting “He’ll probably regurgitate food for me if I harass him!”), individual will no longer perform whatever action caused and a very different one when he doesn’t smell of meat (i.e., the dominant’s distress. “He’ll probably pin me to the ground if I harass him!”) Both of these CSEMs would have high values for Arousal, The AlphaWolf project is changing the way we build but very different values for Pleasure. characters – rather than assembling finished adults, we are creating young pups with certain built-in behaviors and In order for CSEMs to work correctly, it is important to allowing them to grow up in a social context where they can have a model of emotion that continues to affect behavior re-use and modify these behaviors for other purposes. even when the creature’s emotion is at a fairly low level. A Additional suites of behaviors may be scheduled to “come low but nevertheless active level of emotion is necessary to online” at certain points in development. For example, at update the CSEMs created for all the mundane objects in sexual maturity, a whole new group of behaviors become the wolf’s world. For example, continued exposure to trees eligible for expression. In addition to changing behaviorally should create a CSEM towards trees that has a very high as they age, the wolves also change anatomically, their confidence, and very low arousal (e.g., trees are usually physical forms growing and morphing from pup to adult. present and rarely correlate to anything interesting). This will help the wolf focus on the more salient and novel Change on developmental time scale is finding its way into elements in his perceptual world. both academic research and commercial products. For example, the computer game “Creatures” (Grand et al. Using Behaviors in Novel Social Contexts 1997) and other virtual pets grow up as you interact with them. Cañamero’s Abbotts project (Cañamero 1997) also We mentioned above that wolf submission behaviors are features a developmental perspective. similar to infantile behaviors. How do wolves learn to use these behaviors to negotiate their social relationships? Over the next few paragraphs, we will describe some of the Future Work elements of our learning system that pertain to this co- opting of one behavior for another purpose. We have three main extensions to this system that we hope to pursue in the future. The first is the mechanism by which In our previous project, entitled “Clicker by Eire”, our our wolves perceive the emotional states of other virtual terrier Duncan had the ability to learn to exhibit a individuals. It would be interesting to enable wolves to behavior in response to an appropriate context. (Blumberg visually, acoustically and olfactorily discriminate among 2001) A human participant could train Duncan to do a dominant and submissive signals coming from other variety of tricks in response to voice commands. Duncan’s wolves. This would allow our wolves’ learning to be learning system back-propagated the value of a food treat to directed by the perceptual mechanisms of other wolves. For reinforce actions that he had been taking during a certain example, if an individual is less likely to act aggressive time window that preceded the treat. In addition, Duncan toward a larger wolf, then learning ways to appear big (e.g., was able to distinguish between contexts in which he was raising hackles, erecting ears and tail, standing up tall) social phenomena that occur in the more complex case of could be a way to inhibit aggression in others. humans.

The second major area of extension is in the area of alliance formation. In captive packs, wolves will sometimes appear Acknowledgements to work together to achieve dominance (Morss, pers. comm.) This could be a natural result of our context- We would like to thank all the members and friends of the specific emotional memory mechanism, if wolves form Synthetic Characters Group who have helped make our memories that relate to the simultaneous presence of wolves possible: Marc Downie, Matt Berlin, Jesse Gray, multiple wolves. This could be a very interesting extension Adolph Wong, Robert Burke, Scott Eaton, Damian Isla, in that it might shed some light on how alliances are formed Yuri Ivanov, Michael Patrick Johnson, Spencer Lynn, Ben among people as well. Resner, Derek Lyons, Jennie Cochran, Bryan Yong, Steve Curcuru, Geoffrey Beatty, Jed Wahl, Dan Stiehl, Rusmin A third experiment for our research project would be to Soetjipto, Dan Zaharopol, Aileen Kawabe, Madaline model wolves who exhibit the kinds of social behavior seen Tomlinson and Professor Irene Pepperberg. In addition, in the wild, and then confine several of those wolves in a many of the ideas in this paper were fleshed out through virtual pen to see if they develop the same behavioral discussions with Professors Rosalind Picard and Richard patterns found in captive wolves. While aggressive Wrangham. Finally we wish to thank our anonymous dominance conflicts are not uncommon in captive packs of reviewer, whose comments greatly improved this paper. wolves (Schenkel 1967, Zimen 1981), they appear to be a far less significant part of wolf social life in the wild, References (Mech 1999) where wolves may disperse (i.e. leave the pack) as a mechanism for conflict resolution. This might Allison, D., Wills, B., Hodges, L., and Wineman, J. 1996. suggest that dominance conflicts are a pathological result of Gorillas in the Bits, Technical Report, GIT-GVU-96-16, the close contact enforced by captivity. We could perform Georgia Institute of Technology. experiments in our simulation to determine what elements of captivity are most responsible for this pathological Balkenius, C. and Moren, J. 2000. A Computational Model behavior. We will have succeeded in our research if our of Context Processing, in SAB ’00, p. 256-265. experiments lead to more humane treatment of real captive wolves. Blumberg, B. 2001. Building characters that learn what they ought to learn. International Workshop on Story Telling. Avignon, France. Conclusion As this paper is going to press, we are showing the Breazeal, C. 2000. Sociable Machines: Expressive Social AlphaWolf installation at SIGGRAPH 2001. In the Exchange Between Robot and Human. Ph.D. Thesis, installation, each of three participants influences the social Artificial Intelligence Laboratory, MIT. relationships formed by one pup in a virtual litter. By howling, growling, whining or barking into microphones, Buck, R. 1984. The Communication of Emotion. New the participants affect whether their pups try to dominate, York, NY: Guilford Press. submit to or play with their littermates. While we are not collecting user data at SIGGRAPH, people appear to enjoy Burke, R., Isla, D., Downie, M., Ivanov, Y. and Blumberg, the interaction, in large part because of the social B. 2001 CreatureSmarts: The Art and Architecture of a competence of the virtual wolves. Virtual Brain, in Proceedings of the Game Developers Conference, pp. 147-166 The work presented here could have several significant intellectual results. First, it could inform the way in which Cañamero, D. 1997. Modeling Motivations and Emotions we design socially and emotionally intelligent agents, as a Basis for Intelligent Behavior. In Proceedings of 1st helping to flesh out the ways in which agents should Int'l. Conf. on Autonomous Agents (Agents ‘97). communicate, learn and develop with respect to each other. Second, it could be used as a tool for studying wild wolves, Cañamero, D. 1998. Issues in the Design of Emotional and perhaps make suggestions to biologists about Agents. In Emotional and Intelligent: The Tangled Knot experiments that they might perform, why certain of Cognition. Papers from the 1998 AAAI Fall misconceptions exist (such as the extreme dominance and Symposium. 49-54. Menlo Park, CA: AAAI Press. submission exhibited in captivity but not in the wild that we mentioned earlier), and how wolves might think about their Damasio, A. 1994. Descartes’ Error: Emotion, Reason, social environment. Finally, by considering the simulation and the Human Brain. New York, NY: G.P.Putnam’s of social phenomena in the simpler case of wolves, we Sons. might be able to come to a deeper understanding of the Dautenhahn, K. 2000. Socially Intelligent Agents and The Reynolds, C. 1987. Flocks, Herds and Schools: A Primate Social Brain – Towards a Science of Social Minds. Distributed Behavioral Model. Computer Graphics, 21(4). In Socially Intelligent Agents: The Human in the Loop. In Proceedings of the SIGGRAPH '87 Conference 25 - 34. Papers from the 2000 AAAI Fall Symposium. 35-51. Menlo Park, CA: AAAI Press. Rose, C. 1999. Verbs and Adverbs: Multidimensional Motion Interpolation Using Radial Basis Functions. Ph.D. Downie, M. 2001. Behavior, Animation and Music: The Dissertation, Dept. of Comp. Sci., Princeton University. Music and Movement of Synthetic Characters. M.S. Thesis, Media Lab, MIT. Russell, J. 1997. Reading emotions from and into faces: resurrecting a dimensional-contextual perspective. in Ekman, P. 1992. An Argument for Basic Emotions. Stein, Russell, J. and Fernandez-Dols, J. (eds.) The Psychology of N. and Oatley, K. eds. Basic Emotions. 169-200. Hove, Facial Expression. Cambridge: Cambridge University UK: Lawrence Erlbaum. Press.

Gadhano, S. and Hallam, J. 1998. Emotion-triggered Schenkel, R. 1967. Submission: its features and function in Learning for Autonomous Robots. in SAB ’98 Workshop the wolf and dog. American. Zoologist. 7:319-329. on Grounding Emotions in Adaptive Systems. Schlosberg, H. 1954. Three dimensions of emotions. Grand, S., Cliff, D., and Malhotra, A. Creatures: Artificial Psychological Review, 61(2):81-88. Life Autonomous Software Agents for Home Entertainment. in In Proceedings of 1st Int'l. Conf. on Smith, C. 1989. Dimensions of appraisal and physiological Autonomous Agents (Agents ‘97). response in emotion. Journal of personality and social psychology, 56: 339-353. Hemelrijk, C. 1999. An individual-orientated model of the emergence of despotic and egalitarian societies. Proc. R. teBoekhorst, I. and Hogeweg, P. 1994. Self-structuring in Soc. Lond. B 266: 361-369. artificial ‘CHIMPS’ offers new hypothesis for male grouping in chimpanzees. Behaviour 130:229-52. Isla, D., Burke, R., Downie, M. and Blumberg, B. 2001. A Layered Brain Architecture for Synthetic Characters. IJCAI Thorndike, E. 1911. Animal Intelligence. Darien, CT: 2001. Hafner.

Kitano, H. 1995. A Model for Hormonal Modulation of Tu, X. and Terzopoulos, D. 1994. Artificial Fishes: Physics, Learning. in IJCAI ’95, p. 532-538. Locomotion, Perception, Behavior. In Proceedings of ACM Computer Graphics, SIGGRAPH'94. 43-50. Mech, L. 1999. Alpha status, dominance, and division of labor in wolf packs. Canadian Journal of Zoology Velasquez, J. 1998. When Robots Weep: Emotional 77:1196-1203. Memories and Decision-Making. In Proceedings of the Fifteenth National Conference on Artificial Intelligence. Mech, L D., Adams, L. G., Meier, T. J., Burch, J. W., and Madison, Wisconsin: AAAI Press. Dale, B. W. 1998. The Wolves of Denali. Minneapolis, MN: University of Minnesota Press. Yoon, S., Blumberg, B. and Schneider, G. 2000. Motivation Driven Learning for Interactive Synthetic Characters. In Mehrabian, A. and Russell, J. 1974. An Approach to Proceedings of 4th Int'l. Conf. on Autonomous Agents Environmental Psychology. Cambridge, MA: MIT Press. (Agents 2000).

Morss, C. 2000. Seminar on wolf behavior. Wolf Hollow Zimen, E. 1981. The Wolf. New York, NY: Delacorte Ipswich, Fall 2000. Ipswich, MA. Press.

Murie, A. 1944. The Wolves of Mount McKinley. Fauna of the National Parks Series, No. 5. Washington, DC.: U.S. National Park Service.

Picard, R. 1998. Affective Computing. Cambridge, MA: MIT Press.

Plutchik, R. 1991. The Emotions. Lanham, MD: University Press of America.