Non-Linear Dynamical System Approach to Behavior Modeling

1 Introduction

The importance of game and simulation applications grows everyday, as does the need for animated agents Non-linear dynamical that operate autonomously in these environments. These agents must be able to exhibit certain low- and system approach to high-level behaviors and interact with the environ- behavior modeling ment. Various approaches for modeling behavior and movement decisions have been investigated, among them the pioneering work of Reynolds (Reynolds 1987) and that of others (Haumann and Parent 1988; Wilhelms 1990; Renault et al. 1990; Bates et al. Siome Goldenstein, 1992; Noser and Thalmann 1993; Reynolds 1993; Edward Large, Ko et al. 1994; Tu and Terzopoulos 1994; Noser et al. Dimitris Metaxas 1995). These include, kinematic, dynamic, learning, and artiﬁcial intelligence-based approaches. AI techniques for behavior generation, such as (Hau- VAST Lab., Center for Human Modeling and mann and Parent 1988; Lethebridge and Ware 1989), Simulation, Computer and Information Science Department, University of Pennsylvania, 200 South generally require complex inferencing mechanisms. 33rd Street, Philadelphia, PA, USA 19104-6389 This may require considerable computational re- e-mail: {siome,large,dnm}@graphics.cis.upenn.edu sources, raising the question of scalability of such systems, as the number of independent agents and behaviors grows, and each agent has a different set of goals and behavioral directives. In addition, agents We present a dynamic systems approach to must be able to interact with real-time moving ob- modeling and generating low-level behav- jects that might either contribute to or compromise iors for autonomous agents. Such behaviors the ﬁnal goal. Other approaches to this problem em- include real-time target tracking and obsta- ploy learning, perception, and dynamic techniques cle avoidance in time-varying environments. (Ridsdale 1990; Tu and Terzopouluos 1994; Noser The novelty of the method lies on the inte- et al. 1995; Grzeszczuk and Terzopouluos 1995), gration of distinct non-linear dynamic sys- while computational geometry techniques have been tems to model the agent’s interaction with employed in rather restricted environments (Wilfong the environment. An angular velocity con- 1988). trol dynamic system guides the agent’s di- Dynamic system approaches to this problem have rection angle, while another dynamic system mostly been explored at the level of control the- selects the environmental input that will be ory (Cohen 1992; Liu et al. 1994; Kokkevis et al. used in the control system. The agent inter- 1995). This methodology restricts the type of be- acts with the environment through its knowl- haviors that can be simulated since it does not cope edge of the position of stationary and mov- well with dynamically changing environments and ing objects. In our system agents automati- its computational complexity prohibits its use in cally avoid stationary and moving obstacles real-time simulations. Others discuss the useful- to reach the desired target(s). This approach ness of different layers of control and behaviors to allows us to prove the stability conditions achieve better results (Kurlander and Ling 1995; that result in a principled methodology for Magnenat-Thalmann and Thalman 1995; Blumberg the computation of the system’s dynamic pa- and Galyean 1995; Perlin and Goldberg 1996) rameters. We present a variety of real-time In this paper, we investigate and develop an alterna- simulations that illustrate the power of our tive methodology that has its roots in behavior-based approach. robotics (e.g., (Braitenberg 1989; Brooks 1991)) and is based on a novel way of combining differential Key words: Digital agents – Game anima- equations exhibiting particular behaviors (Steinhage tions – Low-level behaviors – Motion plan- and Schöner 1997; Large et al. 1999). Using this ning – Dynamical systems type of approach, Schöner and colleagues (Schöner and Dose 1992; Schöner et al. 1996) have devel-

The Visual Computer (1999) 15:349–364 c Springer-Verlag 1999 350 S. Goldenstein: Non-linear dynamical system approach to behavior modeling oped a dynamical system for robot path planning and porarily disregard a target if there is an unsurpassable control. In this system a set of behavioral variables, moving or stationary obstacle immediately between namely heading direction and velocity, define a state it and the target. Like a human, it will first focus on space in which the dynamics of robot behavior is avoiding the obstacle, and then refocus on the target. described. Path planning is governed by a nonlinear In this paper we also prove an important result on dynamical system that generates a time course of the properties of the parameters of the task constraint the behavioral variables. The system dynamics are system which determine the number of behaviors specified as a nonlinear vector field, while the task modeled. These properties are associated with the that the agent will execute depends upon the task stability2 of the task constraint system. We also pro- constraints. Task constraints, such as obstacle avoid- vide an inductive proof that extends the above result ance and target tracking, are modeled as component to any number of dimensions in the task constraint forces which define attractors and repellers for the system. This result allows the principled design of dynamical system. The individual constraint contri- complex systems with large number of behaviors. butions are weighted and then added together into Our system allows single and multiple target track- a single vector field, which determines the observed ing in the presence of multiple static and moving behavior. Next, a second dynamical system is used obstacles, and scales well with the number of behav- to compute these weights. This dynamical system, iors. Obstacles are processed in a local fashion based the task level system, operates at a faster time scale on their relative location to the agent and the tar- than the movement level (Large et al. 1999). Qualita- get. In our current implementation and without loss tively different behaviors are modeled as fixed points of generality the agents are memoryless and reactive of this dynamical system, and the environment deter- in nature. Depending on the situation (emergence of mines the values of the parameters. As the perceptual new obstacles and/or targets) their movement can be information changes, parameters change, causing bi- discontinuous. furcations in the task-level system. During the course This paper is organized as follows. In Sects. 2 and of the simulation, one fixed point loses stability and 3 we present the dynamic formulation of the sys- another becomes stable, modeling the decision to tem. Section 4 describes the task constraint system cease executing one behavior and to execute another responsible for the generation of the behaviors. We instead. devise an inductive proof for the restrictions on this Here, we adapt the above methodology to model and system’s parameters (stability analysis) that will re- simulate autonomous low-level dynamic behaviors sult in the desired behaviors. Finally, we present which can be used in applications ranging from vir- a series of real-time simulations involving complex tual environments to games. We use (see also our interactions between the agents, the environment and preliminary results, in (Goldenstein et al. 1998)) a set the targets. of time-varying differential equations that control the heading angle and forward speed of a given digital autonomous agent. Based on a principled com- 2 Movement dynamics and low-level bination of these equations we create a whole set of relatively complex low-level behaviors such as ob- behavior stacle avoidance and target tracking. To avoid unstable fixed points in the differential equations1 we add Our methodology consists of the combination of two a Gaussian noise term in each equation. Using this distinct dynamic systems to model the movement system, decisions are made online and do not require and behavior of each autonomous agent. The first any previous memory, training, or global planning. system controls the movement of the agent. The state The set of targets and obstacles can change during space of this system is two-dimensional, the first pa- the course of the simulation, since the agent is able rameter represents the heading direction, while the to make smart local decisions based on its current other specifies its velocity (In a three-dimensional global knowledge of the dynamic environment in space there would be two heading angles and one which it is situated. For example, an agent will tem- forward velocity). The second system controls the

1 In differential equation terminology a fixed point is a point 2 The stability is related to the local convergence properties of where the derivative of the vector field is zero the system around a fixed point S. Goldenstein: Non-linear dynamical system approach to behavior modeling 351 agent’s movement decision-making, i.e., its behav- from the sum of contributions from all obstacles. We ior. The state space of this system is the space of the then extend the formulations to account for multiple agent’s behaviors. The parameter values of the state groups. vector components determine which elements of the Using (1) an attractor is defined as environment (e.g., obstacles, targets) will be used in φ˙ = =− (φ − ψ) , the calculation of the agent’s movement and there- ftar a sin (2) fore behavior. ψ Each autonomous agent’s movement is described in where is the angle of the target’s location relative a polar coordinates. It consists of a heading direction φ to the agent’s location and is a constant parame- v ter. Figure 1 shows a plot of (2), assuming that the and a forward velocity . The heading angle is con- ψ = trolled by a one-dimensional non-linear dynamical target is in front of the agent ( 0). The negative slope around the fixed point (in this case, the origin) system, which consists of repellers placed in the sub- φ˙ tended angle of the obstacles, and attractors in the is what gives the attractor property, forcing to be subtended angles of targets (see Sect. 2.1). In our positive if the target is to the right and negative if it formulation, the heading speed is modified by the is to the left of the current viewing direction of the relative location of the obstacles (see Sect. 3). agent. Based on our formulation, an agent may ignore tar- We model different types of obstacles using repellers. A first simple approach would be to use gets or obstacles, depending on the scene geometry a sinusoidal function, just like we did for the target, around the agent at each time instance. The environ- but with a positive slope instead. This would result ment geometry is modeled based on another type of in a positive slope necessary for the repelling effect. nonlinear dynamical system running at a faster time However the repelling effect would not necessarily scale. This system outputs weights that linearly com- be preserved if we linearly sum the contributions of bine the different target and obstacle contributions as multiple obstacles. calculated by the first system. An important aspect Consider the example in Fig. 2 with two distinct ob- of our methodology is that it scales linearly with the stacles positioned symmetrically about the direction number of obstacles in the environment. of the agent by an angle ψ. They are far enough apart In the following sections we present the details of so that the agent can simply move between them by each of the two dynamical systems, the basic move- proceeding forward without the risk of colliding. The ment dynamics and the modeling of behaviors. obstacles’ contribution to the agent’s angle (through φ˙)is: 2.1 The basic movement dynamics ˙ φ = fobs = a sin(φ − ψ) + a sin(φ + ψ) The first dynamical system models the control of = a[sin φ cos(−ψ) + cos φ sin(−ψ) the basic movement of each autonomous agent. The + φ ψ + φ ψ movement is defined by a 2D vector representing the sin cos cos sin agent’s heading angle and forward speed. = 2a cos ψ sin φ = k sin φ. (3) The heading angle φ of a given agent is controlled by a dynamical system of the type: From (2) and (3) we observe that this simple sum results in a repeller in the forward direction φ (with the φ˙ = f(env), (1) amplitude controlled by ψ), certainly an undesirable effect. where env is the vector of variables which model the In order to create a repeller other aspects of the en- environment (e.g., the geometry and position of the vironment geometry, such as the distance from the obstacles and targets). agent, should also be taken in account. In addition, According to our dynamical system formulation we would like to control, based on parameters, prop- each element of the environment can attract or repel erties of the repellers such as their spatial extent. an agent. We will therefore use attractors to model Therefore, we use a model for a repeller which is targets and repellers to model objects that should a multiplication of three different functions, Ri , Di, be avoided. We begin our formulation by assuming Wi , each modeling a different property two groups of influences on our agent. The first con- φ˙ = = . sists of the contribution from the target, the second fobsi Ri Wi Di (4) 352 S. Goldenstein: Non-linear dynamical system approach to behavior modeling

ftar obs1 obs2

φ − ψ1

φ ∆ψ2 −π −π/2 π/2 π r1 ψ φ 1

fobs fobs fobs

φ φ φ −π −π/2 π/2 π −π −π/2 π/2 π −π −π/2 π/2 π

3a 3b 3c

Fig. 1. Target function for the angle dynamical system for ψ = 0. An attractor (stable ﬁxed point) at φ = 0 and a repeller (unstable ﬁxed point) at φ = π Fig. 2. Two obstacles in front of the agent Fig. 3a–c. Repellers for three different parameters settings

Function Ri models a generic repeller, and is con- and is modeled by structed as: h1 = 4/(cos(2∆ψ) − cos(2∆ψ + δ)) . (7) φ−ψ (φ − ψ ) i i 1− ∆ψ Ri = e i , (5) Here δ is a safety margin constant. ∆ψi The third and last function, Di , models the influence of the obstacle in the environment by using the dis- where ψi is the angle of obstacle i with respect to the tance of the obstacle from the agent. It is modeled agent’s direction, and ∆ψi is the subtended angle of the obstacle as seen by the agent. as The second function, Wi , is responsible for limiting − ri = d0 , the angular range of the repeller’s influence in the en- Di e (8) vironment and is defined as where ri is the relative distance between the obsta- 1 cle and the agent, and d0 controls the strength of this Wi = [tanh(h (cos(φ − ψi) − cos(2∆ψi + σ))) + 1] , 2 1 influence as the distance changes. (6) The parameters φi, ∆φi and ri are graphically illus- trated in Fig. 2. which models a window-shaped function and h1 is Figure 3 shows three repellers obtained by using dif- responsible for the inclination of the window’s sides ferent values for the parameters (for all examples the S. Goldenstein: Non-linear dynamical system approach to behavior modeling 353 obstacle is directly ahead of the agent, i.e., ψ = 0). is obtained as: In Fig. 3a ∆ψ = π/10, r = 3, d = 6andδ = 0.8. i 0 φ˙ = ( ) =|w | +|w | + . Figure 3b shows a situation where we used the same f env tar ftar obs fobs n (10) ∆ψ π/ parameters as in Fig. 3a, but changed to 20. w w In Fig. 3c we used the same parameters as in Fig. 3a, The weights tar and obs are intended to eliminate spurious attractors that can occur by a simple sum of but now a change was made to Wi (window-shaped function), by setting δ to 0.1. the nonlinear functions modeling the various obsta- The resulting effect on the agent’s heading direction cles and targets in the environment. These weights from all obstacles i = 1,... ,n, is the sum of the re- are obtained through a constraint competition dy- spective repellers namical system which will be described in detail in Sect. 2.2 (see also (Large et al. 1999)). The weights n are the essence of the low-level behavior modeling. = . fobs fobsi (9) Finally, as explained before, the Gaussian noise term i=1 n allows the system to escape from unstable fixed Based on the above design of each repeller (see (4)), points. we can use the simple sum of each individual ob- Let’s first ignore the influence of the weights (set- stacle repeller to combine multiple contributions (9). ting them to 1), and see the properties of our design Let’s assume again that the agent is facing two dif- in (10). In Fig. 5 we show the interaction between ferent obstacles located side by side (see Fig. 4). If a target (attractor) and an obstacle (repeller). It is in- the obstacles are too far apart, the agent should be tuitively clear that the agent will have to go around able to pass between them, otherwise it will should the obstacle in order to arrive at the target without hitting the obstacle. In this case, the modeling of go around them. This decision is taken automatically, φ˙ as shown in Fig. 4. Figure 4a depicts the case where the agent’s heading direction dynamics, , based on two obstacles are too close; Fig. 4b depicts the case (10) is shown in the lower-right graph of Fig. 5. It where the distance between the obstacles is exactly is the summation of the target (upper-right graphic) equal to the size of the agent, a critical condition; and and obstacle (middle-right graphic) functions. The Fig. 4c depicts the case when the obstacles are far presence of two final attractors, indicated by the two enough apart to allow the easy passage of the agent arrows in the lower-right graph, shows the two pos- between them. In this simple case (no target and sible ways to get to the target and avoid the obstacle two obstacles) we have plotted (9) at the bottom of (going either to the left or to the right of the obstacle). each figure as a function of φ. These functions show Unfortunately, the simple sum (setting the weights to that the agent exhibits the correct behavior. Next, (9) 1) will not always be sufficient. In Fig. 6 we extend the needs a slight modification. In Fig. 4a we see that for situation shown in Fig. 4a through the addition of an attractor located behind the two closely placed obsta- the case where the agent is facing directly between φ˙ = the obstacles, φ˙ = 0. This implies that there will be cles.Theresultingfunctionnowhas 0intheorigin no change in the heading direction and eventually and a negative slope around it. This means that the re- there will be a collision situation. If we look at the sult of this simple sum is an attractor in the forward value of φ˙ slightly off the origin, we notice that the direction. This implies that even with noise, the agent slope of the function is positive so the agent will re- will proceed straight ahead and will try to go through pelled. We call this case an unstable fixed point.Itis the obstacles as if they were not there. just like a ball situated in the crest of a hill – if placed The use of weights whose values are determined exactly on the top is does not move, but if any slight based on the second dynamical system described be- perturbation occurs it will fall. To solve this we must low will result in the correct action of the dynamic add to the system a noise term n. system. Based on the above we modeled the environment as a sum of a set of repellers (obstacles) and attractors 2.2 Constraint competition: modeling the (targets). The dynamic system that controls the head- weights ing angle φ of our agent is the weighted sum of these set of attractors and repellers with the addition of Individually, the attractors and repellers defined in a noise term. Therefore, the final definition of the dy- Sect. 2.1 work in the desired way. Due to the non- namical system controlling the heading angle in (1) linearity in their definitions their simple sum might 354 S. Goldenstein: Non-linear dynamical system approach to behavior modeling

fobs fobs fobs 1 1 1

0.5 0.5 0.5 φ φ φ −π −π/2 π/2 π −π −π/2 π/2 π −π −π/2 π/2 π -0.5 -0.5 -0.5

-1 -1 -1 4a 4b 4c

Fig. 4a–c. Interaction of repellers of two different obstacles Fig. 5. Interaction between one attractor and one repeller

not always yield the expected result. In the ﬁrst ex- To avoid this kind of problem, the composition of ample (Fig. 5) the simple sum gave the correct result, the attractors and repellers functions is obtained not but in the example shown in Fig. 6, the sum of one at- by a simple sum, but through a weighted sum. The tractor with the two repellers results in an impossible weights, wi, are the result of a second dynamical sys- path in between the two obstacles – they are too close tem, which runs at a faster time scale with respect to to each other to allow the agent’s passage. the dynamical system in (10). This second constraint S. Goldenstein: Non-linear dynamical system approach to behavior modeling 355

0 −2 −π −π/ π/ π 2 0 2 2 0

− 2 2 −π −π/2 0 π/2 π

−2 −π −π/2 0 π/2 π

Fig. 6. Undesired attractor of the angle dynamical system

system is defined as each step. This simplification allows us to precisely study the stability conditions. w˙ = α w ( − w2) − γ w2w , w˙ = i i i 1 i i, j j i (11) The fixed points have the property that 0. Once j =i the system arrives to one it does not leave. But sometimes the fixed point might be unstable in the pres- where in the simple case with only one kind of ob- ence of small perturbations, just like a ball on the stacle and a single target, the state space (wi )is crest of a hill. It is important to study how the sys- bidimensional and consists of (wtar, wobs),asused tem behaves in regions close to the fixed point. The in (10). fixed point is stable if the eigenvalues of the Jacobian The parameters αi and γij are functions of the en- (Jij = ∂w˙i/∂w j ) are all negative (this is the multidi- vironment geometry at each time instant. They are mensional generalization of the negative slope seen called competitive advantage and competitive inter- in Sect. 2). action, respectively. Together with the initial value of The correct design of the parameter functions αi and the state space they completely define the dynamical γ ji will provide the desired low-level behavior of the system. agent as expressed in (10). Therefore, it is important If the constraint competition system runs at a much to understand the stability of this system (for more faster rate than the basic movement system, we can details see (Perko 1991), and incorporate the geom- assume that the environment and thus αi and γij are etry of the environment in the low-level behaviors fixed in (11). This is practically accomplished by let- through the αi and γij. ting the constraint system run until it stops (at a fixed The details of the stability analysis for the two- point) for each step of the movement system. The αi dimensional case used in (10) is in Appendix A and and γij are only calculated once at the beginning of follow closely the approach of (Large et al. 1999; 356 S. Goldenstein: Non-linear dynamical system approach to behavior modeling

Table 1. Stability analysis results. Obstacles might not be static, changing the w w conﬁgurations is a faster way than the heading dy- tar obs Stability namical system can react only by itself. Therefore

00α1,α2 < 0 adaptive speeds are necessary in order to achieve 0 ±1 γ12 >α1 a better behavior. α2 > 0 The Constant time to contact method (also used ±10γ21 >α2 by Neven and Schöner (Neven and Schöner 1996)) α1 > 0 seeks to solve this problem by making the speed ±A1 ±A2 α1,α2 > 0 change as a function of the distance between the α1 >γ12 agent and the closest visible obstacle. It is even pos- α2 >γ21 γ γ < γ ,γ > sible to make a retreat (negative forward speed) if 12 21 0or 12 21 0 this distance is too small. As the name suggest, this approach fixes the time that it would take the agent to reach the closest obstacle Schöner and Dose 1992). Table 1 summarizes these as if it were right in front of it: results. r − d The four distinct cases can be physically rein- v = min 1 , (12) terpreted as turning off both target and obstacle τ2c w =[ , ] contributions (when 0 0 , an obviously un- In (12) τ is the fixed time to contact, r is the desired solution), turning off either one of them 2c min w =[ , ± ] w =[± , ] distance between the agent and the closest target, ( 0 1 and 1 0 ),andanaveragingso- and d is a safety distance. With nearby obstacles lution, where both contributions are taken in account 1 w =[± , ± ] in the vicinity, the agent will chose a smaller ve- ( A1 A2 ). locity, and for situations where the obstacles are far The design of αi and γij depends on the desired away, it will go faster. The value of d1 also intro- properties of the final system. We want the obsta- duces a notion of personal space, in a sense that if cles contribution active in (10) unless there are no the obstacle is too close to the agent it will try to obstacles close enough to the agent. Target contri- retreat. butions should be deactivated when they are behind Based on our experiments we have found that con- a repeller. For ease of design we will use positive val- α γ stant time to contact method provides the most rea- ues for the parameter functions i and ij. sonable behaviors. In Appendix B, we describe the design of the parameter functions αi and γij following again the approach from (Large et al. 1999; Schöner and Dose 4 Expanding the constraint 1992). competition system by induction In Sect. 2.2 we have seen the constraint competition 3 Modeling the agent’s velocity system for a two-dimensional vector w of weights. In a general application, the classification of the contri- In the previous sections we modeled the change butions into only two groups (obstacles and targets) in the agent’s heading direction. In this section we might be a rather restrictive description. In some ap- model its forward movement. There is a series of plications it is conceivable to classify and treat sep- different possibilities to accomplish this, but in this arately different types of obstacles, or have different paper we will examine two of them. The first is the mutually exclusive targets. Constant velocity and the second is a Constant time We will use an inductive method to generalize our to contact method. approach so that we can design multidimensional The first option is obviously the easiest one, the systems. Of course the base case (two dimensions) agent always moves with a constant speed. This has already been examined in Sect. 2.2. Our induc- method can work well if the forward velocity is tion hypothesis is that a k-dimensional system will small enough so that the heading angle dynamics have at most 2k fixed points. has enough time to react and avoid obstacles. Un- Let’s suppose (by strong induction hypothesis) that fortunately in many situations it leads to disastrous we know how to solve/design all systems with di- S. Goldenstein: Non-linear dynamical system approach to behavior modeling 357 mensions ≤ k. We then want to solve/design a k + 1- We can write this as an linear system of w2 dimensional system. As in Sect. 2.2 we will first equations for i , that has the matrix form: find the fixed points, and then check their stability     2 conditions. α γ ··· γ ( + ) w 1 12 1 k 1  1   γ α ··· γ ( + )  w2   21 2 2 k 1   2   . . . .   .  4.1 The fixed points  . . .. .   .  2 γ(k+1)1 γ(k+1)2 ··· αk+1 w The fixed points of the k + 1-dimensional system k+1   can either have wk+1 = 0orwk+1 = 0. We analyze α w 1 separately the case where all other k are equal to  α   2  zero. =  .  (14) There are three cases:  .  α Case 1: w =[0, ··· , 0, 0]t and w =[0, ··· , 0, 1]t. k+1 It is clear by inspection, just as in the two-dimensional case, that for the vectors or in a more compact form: w =[0, ··· , 0, 0]t and w =[0, ··· , 0, 1]t, w˙ = 0, so these are two of the fixed points. Mw2 = α.¯ (15) Case 2: w =[A , ··· , A , 0]t,whereA can be zero. 1 k i w = It is easy to see that this case reduces to From (15) we conclude that the i 0fixed a k-dimensional case. By the induction hy- point is (when solvable), pothesis we know that have at most 2k fixed −1 points. wi = (M α)¯ i , (16) w =[ , ··· , , ]t = Case 3: A1 Ak Ak+1 and Ak+1 0. | | = Again, this will hold in at most 2k fixed but only if M 0 (for the solution of the −1α¯ points. If any of the A , i = k + 1 is equal linear system), and all components of M i are larger or equal to zero (because we are to zero, then it’s existence conditions are w known by induction (without loss of gener- only interested in real values of i ). These are the existence conditions. ality we could swap variables wi with wk+1, and then we would have the case 2). w = The fixed point in the case where all i 0, 4.2 Stability of the fixed points i ∈ 1 ···k + 1is   For the analysis of the stability of each of these can- w˙ = w α − α w2 − γ w2 . didate points we have to check the eigenvalues of the i i i i i i, j j (13) Jacobian matrix of (11) at each point. j =i The Jacobian is: (see (17))

  α ( − w2) − γ w2 − γ w w ··· − γ w w 1 1 3 1 ij j 2 12 1 2 2 1(k+1) 1 k+1  j =i   − γ w w α ( − w2) − γ w2 ··· − γ w w   2 21 2 1 2 1 3 2 ij j 2 2(k+1) 2 k+1  (w) =  j =i  . J  . . . .  (17)  . . .. .   . . .  − γ w w − γ w w ··· α ( − w2 ) − γ w2 2 (k+1)1 k+1 1 2 (k+1)1 k+1 1 k+1 1 3 k+1 ij j j =i 358 S. Goldenstein: Non-linear dynamical system approach to behavior modeling

If w = 0¯ the Jacobian will have the format: 5 Summary of the algorithm   α1 0 ··· 0 We start by defining the environment and it’s geom-  α ···  etry; which consists of agents, targets and obstacles.  0 2 0  J(w) = . . . . (18) For each agent, target and obstacle we define its at-  . .. .  tributes, like size location and initial orientation. ··· α 0 0 k+1 The following pseudocode describes the flow of the algorithm: Therefore the zero vector case is just like the one in two dimensions. To be a stable point we have to re- 1: for each instant t do quire that αi < 0, ∀i. 2: for each agent do t For the case where w =[0, ··· , 0, 1] , 3: Calculate ψtar and ∆ψtar (see Fig. 2); ∂ /∂φ   4: Use (2) to calculate ftar and also ftar ; α1 − γ1(k+1) 0 ··· 0 5: Wt = 0, fobs = 0, Dfobs = 0;  0 α − γ ( + ) ··· 0  6: for each Obstacle i do (w) =  2 2 k 1  . J  . . .  7: ψ ∆ψ . .. . Calculate obsi and obsi ; 8: Using (5), (6), (8) and (4) calculate Ri , 0 ··· 0 −2αk+1 Di, Wi , fobs and ∂ fobs /∂φ; (19) = + i =i + 9: Wt Wt Wi , fobs fobs fobsi and Df = Df + ∂ f /∂φ; + γ >α obs obs obsi This gives us k 1 stability conditions: i(k+1) i , 10: end for = ... α > for i 1 k,and k+1 0. 11: Calculate γ (32), α (33) and α (34) (γ w w 12 2 1 21 The case where k is equal to zero, but where i (for is a constant, here set to 0.05); = i k) can be anything, the Jacobian will be 12: Using initial value as w =[0, 0],simulate   (21) until it stops at a fixed point; 13: Use (10) to calculate φ˙ ;     14: Use one of the velocity methods to calcu-    0  late the forward velocity;   J(w) =   . 15: Find next position and orientation (φ)ofthe     agent through forward integration.  k  16: end for α − γ w2 0 k+1 (k+1) j j 17: end for 1 (20)

That means that the stability conditions will be ex- 6 Experimental results actly the ones for the sub-matrix (it has size k so by induction we know how to solve) plus the additional Our system was implemented in C, using lua (Ierusa- condition that limschy et al. 1996, 1997; WebSite a), as an extensible embedded language to describe both the scene k and the target(s)/agent(s) movement. α − γ w2 < . k+1 (k+1) j j 0 All the rendered animations we have obtained using 1 our system can be viewed at the following web site: “http://www.cis.upenn.edu/∼siome/research This same argument is valid if any other of the wi /nonlin/”. is zero with wk+1 = 0 (without loss of generality we The following experiments explore and demonstrate can rearrange the order of the variables). the power of the system described in (10), which in- The only case left is when all wi are different than volves agents, targets and obstacles. In all the exam- zero. This case is by far the hardest, and has to be ples the constant a in (2) was set to 1 and the safety solved on a case by case basis, only if it is needed. All margin δ in (7) was set to 0.8. The Euler integra- the eigenvalues of the Jacobian at the averaging point tion time step was 0.25 and all the simulations run in must be smaller than zero. faster than real time. S. Goldenstein: Non-linear dynamical system approach to behavior modeling 359

8a 7

8b 8c

Fig. 7. First example: static target and obstacles. The circle with an arrow represents the agent, the circle with the cross is the target and the empty circles are obstacles. Dotted lines show the past trajectory of the moving objects. The same representation applies to the other ﬁgures Fig. 8a–c. Second example, static target and moving obstacles

In the first experiment shown in Fig. 7 we used ent trajectories and speeds. The value of d0 has to a single static target and a series of static obstacles be smaller than the one used in the first experiment between its location and the target’s initial position. (Fig. 7), this way the agent will try to reach the target Note that in this case d0 was 3.0. The agent has to (target’s contribution is turned on on the task con- deactivate the target’s influence for most of the time straint system) even with more obstacles close by. in order to successfully avoid the obstacle barrier. In In the third experiment shown in Fig. 9, there is this case, the parameter d0 has a very simple geomet- a group of static obstacles and a moving target. The ric interpretation, the clearance from the obstacles agent successfully reaches the target and avoids the increases with its value. The value of d0 changes the moving obstacles. In this case d0 was set to 0.8 magnitude of the repeller in (4), but it is also used in and the final velocity was the result of the method the calculation of the parameters of α1 and α2 in (21). scaled by 0.8. This example is an illustration of this In the second experiment, shown in Fig. 8, the scene method’s target tracking ability. In this case the value is composed of one static target and multiple moving of d0 had to be even smaller, so that the agent would obstacles. The agent avoids collision by automati- be able to dodge closer around the static obstacles. cally changing direction and sometimes by a velocity In the last experiment (Fig. 10), we illustrate the flex- reduction or even a complete stop. In this simula- ibility of our method by showing multiple moving tion d0 was set to 2.0. In this example the velocity and static targets together with moving and static ob- control system plays an important role in obstacle stacles. In this case the value of d0 had to be small avoidance. The environment is composed of many enough to allow the agent to pass in the spaces be- moving obstacles close to each other, all with differ- tween the obstacle walls. In the snapshots taken here 360 S. Goldenstein: Non-linear dynamical system approach to behavior modeling

9a 9b 9c 9d

10a 10b 10c 10d

Fig. 9a–d. Third example, moving target and static obstacles Fig. 10a–d. Fourth example, multiple moving/static targets and moving/static obstacles

d0 was set to 1.0. Note in Fig. 10c that the tar- parameters and their geometrical meaning. Fur- get gets really close to the obstacle wall before it ther we have shown how our approach can be ex- turns right to avoid it. This turning moment is ex- tended to construct systems with a large number of actly the change in the behavior, the moment where behaviors. γ12 (see (32)) becomes larger than α1 (see (34)). Al- This approach is appropriate as a basic low-level re- though in this example there were multiple targets, active behavior layer for an animation or virtual en- only one (the closest) was taken in account at each vironment system. The increase of the number of ob- time, this was done independently of the dynamical stacles does not affect the efficiency of the method, systems. since for each extra obstacle only a new evaluation The animations on the web site corresponding to the of fobs is necessary, along with some other tests (like above experiments were rendered using the render- finding the closest obstacle for the forward velocity ing package Pov-Ray (WebSite b). control). This means that the number of operations for an agent grows linearly with the number of objects affecting it. 7 Conclusions The design of the constraint competition system parameter is not simple, but once it is done all that We have presented a technique to model autonomous is needed to follow the method is to perform func- agents exibiting low-level behaviors. Using a dy- tion evaluations, something that computers can do namical system approach we control the agent’s fast. We are currently working towards extending the heading direction and its velocity. We have demon- above methodology to model more complex high- strated natural low-level agent behavior in environ- level behaviors, allow a different treatment for static ments with multiple targets and stationary/moving obstacles in order to give the agent a more global obstacles that is computationally efficient – all the knowlege of the environment. We also plan to param- experiments presented run in real time. We have eterize the systems in order to achieve a family of paid particular attention to the setting of the few controled reactive behaviors. S. Goldenstein: Non-linear dynamical system approach to behavior modeling 361

Appendix A. Jacobian of (21) at a point w is: Stability analysis of the obstacle and target ∂w˙ system J = = (26) ∂w α ( − w2) − γ w2 − γ w w In this appendix we analyse the stability for the two- 1 1 3 1 12 2 2 12 1 2 −2γ w w α (1 − 3w2) − γ w2 dimensional case used in (10), where wtar = w1 and 21 2 1 2 2 21 1 w = w : obs 2 For w =[0, 0], w =[±1, 0] and w =[0, ±1] the Ja- w˙ = α w ( − w2) − γ w w2 cobian will be respectively 1 1 1 1 1 1,2 1 2 . 2 2 (21) w˙2 = α2w2(1 − w ) − γ2,1w2w α 2 1 1 0 −2α1 0 J(0,0) = , J(±1,0) = , 0 α 0 α2 − γ21 The candidate points to study are called equilibrium 2 α − γ or fixed points, which are the ones where w˙ 1 = 0and = 1 12 0 . J(0,±1) − α w˙ 2 = 0. Note that we can rewrite (21) as: 0 2 2 (27) w˙ = w (α − α w2 − γ w2) 1 1 1 1 1 1,2 2 2 2 (22) The stability analysis for w = 0 is more complex. w˙ = w (α − α w − γ , w ) i 2 2 2 2 2 2 1 1 The Jacobian will change with the change of sign (four solutions: [±A , ±A ]) because now it does and by simple inspection we can see that w =[0, 0], 1 2 not depend only on w2, but also on the product w w , w =[0, ±1] and w =[±1, 0] will result in w˙ = 0. i i j i i = j. As an extra algebraic complication, these Ja- From (22), for a fixed point that has both w different i cobians (26) are not diagonal matrices. than zero: By observing the system we can see that there [+ , + ] α w2 + γ w2 = α are only two cases; one where A1 A2 or 1 1 1,2 2 1 [− , − ] [− , + ] 2 2 (23) A1 A2 , and another where A1 A2 or α2w + γ2,1w = α2 2 1 [+A1, −A2] The eigenvalues for the Jacobian for the first case are or, in matrix notation, λ = 1 − ( + ) α γ w2 α 1 m d1 d2 1 12 1 = 1 . det 2 α (24) γ21 α2 w 2 2 2 2 − m (d1 + d2) + 4md1d2γ12γ21 (28) w2 1 Now we can just solve the linear system for i .This λ2 = − m(d1 + d2) system can be solved when the determinant is non- det α α = γ γ α >γ α >γ 2 2 zero ( 1 2 12 21)andwhen 1 12, 2 21, + m (d1 + d2) + 4md1d2γ12γ21) , because we are only interested in real fixed points. = α −γ > = α −γ > = The wi are: where d1 1 12 0, d2 2 21 0andm α α > = α α −γ γ > 1 2 0 and det 1 2 12 21 0 (from the conditions for the existence of A1 and A2). α2(α1 − γ12) w1 = A1 =± After some long and careful manipulation, the extra α1α2 − γ12γ21 γ γ < stability conditions is 12 21 0. In the second case, the eigenvalues of the Jacobian α1(α2 − γ21) w2 = A2 =± . (25) are (29) α1α2 − γ12γ21 Again, after some manipulations it becomes clear that these eigenvalues are always negative if γ12, Now that we already know the fixed points ( [0, 0], γ21 > 0. [±1, 0], [0, ±1], [±A1, ±A2] )wehavetocheck Remember that the case (0, 0) is not desired (turning their stability. According to dynamical system the- off of both contributions), so it should be unstable. ory, the point is considered stable when all the eigen- Table 1 summarizes the results of the stability analy- values of the Jacobian have a negative real part. The sis from equation (11). 362 S. Goldenstein: Non-linear dynamical system approach to behavior modeling

1 2 2 λ1 = −m(d1 + d2) − m (d1 + d2) − 4mdet(m + γ12γ21 − α2γ12 − α1γ21) (29) det 1 λ = −m(d + d ) + m2(d + d )2 − 4mdet(m + γ γ − α γ − α γ ) . 2 det 1 2 1 2 12 21 2 12 1 21

Appendix B. When γ12 >α1 (see Table 1) the deactivation of the Parameter functions for obstacle and target target contribution (w =[0, ±1]) is a stable solution. γ system This definition results in a 12 that is slightly larger than zero in most regions, but will have a narrow peak (c2 controls how narrow) around the region The above considerations lead to the following de- where targets and obstacles are aligned. α γ sign of the parameters functions i , ij.Firstwede- The relation between γ21 and α2 will switch between sign a fixed point detector for each of the target and the stability of w =[±1, 0] and w =[±A1, ±A2]. obstacle functions, ftar (see (2)) and fobs (see (4)) The only instant where we should turn off the ob- respectively. stacles contribution is when there are no obstacles A fixed point detector for ftar is defined as close to the agent. The condition for it to happen is γ21 >α2,sowefixγ21 = 0.05, a low value. α2 should be related to the distance of the obstacles ∂ ftar | | P = ec1 ftar . to the target. The closer the obstacles are from the tar sgn ∂φ (30) target, the larger α2 should become. For this effect we use the functions Di defined in (8), such The first multiplication term (sgn function) will that: distinguish the attractor fixed point from the repeller fixed point of f (cases when the target is tar just in front or is just behind the agent) by detect- α = . ing the sign of the slope. The exponential term is 2 tanh Di (33) i one for fixed points (i.e., ftar = 0) and is close to zero elsewhere (these values are controlled by the constant c1). The hyperbolic tangent will smoothly limit the val- For the obstacle case we can’t use the exact same ues of α2 between 0 and 1. expression, since fobs has a limited range of influ- We still have to select α1. Remember that the ence (due to the window function Wi (6)). There relation between γ12 and α1 will deactivate the are several intervals in fobs that have zero value target when γ12 >α1. To improve the collision but do not indicate the location of the obstacle (see avoidance, if there are close obstacles the target Fig. 3). The solution to this uses the same design as should be disregarded. To achieve this we de- in (6). crease α1 as α2 increases, and we use the following equation: ∂ fobs | | = c1 fobs . Pobs sgn ∂φ e Wi (31) i α1 = 0.4(1 − α2). (34)

Now, using Ptar and Pobs, we can design γ12.

Acknowledgements. The ﬁrst author was supported by a Ph.D fel- lowship from CNPq, Brazil. This research is supported by NSF − e c2 Ptar Pobs (IRI-97-01803,EIA98-09209), NASA (96-OLMSA-01-147), NSF- γ12 = . (32) Career Award (IRI-9624604), AFOSR and ONR-YIP (N00014-97-1- ec2 0817). S. Goldenstein: Non-linear dynamical system approach to behavior modeling 363

References vision, memory and learning. Computer and Graphics, Switzerland 21. Perko L (1991) Differential Equations and Dynamical Sys- 1. Bates J, Loyall AB, Reilly WS (1992) An architecture tems. In: Texts in Applied Mathematics. Springer, Berlin for action, emotion, and socialbehavior. In: Proceedings of Heidelberg New York the Fourth European Workshop on Modeling Autonomous 22. Perlin K, Goldberg A (1996) IMPROV: A system for script- Agents in a multi Agents World, S. Martino al Cimino, Italy ing interactive actors in virtual worlds. In: SIGGRAPH 96 2. Blumberg BM, Galyean TA (1995) Multi-level direction of Conference Proceedings, pp. 205–216. ACM SIGGRAPH autonomous creatures for real-time virtual environments. 23. Renault O, Thalmann NM, Thalmann D (1990) A vision- In: Proceedings of SIGGRAPH ’95, pp. 47–54 based approach to behavioural animation. The Journal of 3. Braitenberg V (1989) Vehicles. Experiments in Synthetic Visualization and Computer Animation, 1(1):18–21 Psychology. MIT Press 24. Reynolds C (1987) Flocks, herds, and schools: A distributed 4. Brooks RA (1991) Intelligence without reason. Technical behavioral model. In: Proc. SIGGRAPH ’87, Computer Report 1293, Massachussetts Institute of Technology Graphics, 21, pp. 25–34 5. Cohen MF (1992) Interactive spacetime control for ani- 25. Reynolds CW (1993) An evolved, vision-based behavioral mation. In: Computer Graphics (SIGGRAPH ’92 Proceed- model of coordinated group motion. In: Proc. 2nd Interna- ings), 26, pp. 293–302 tional Conf. on Simulation of Adaptive Behavior, (ed), MIT 6. Goldenstein SK, Large E, Metaxas D (1998) Dynamic au- Press tonomous agents: Game applications. In: Proceedings of 26. Ridsdale G (1990) Connectionist modelling of skill dy- Computer Animation 98 namics. Journal of Visualization and Computer Animation, 7. Grzeszczuk R, Terzopoulos D (1995) Automated learning 1(2):66–72 of Muscle-Actuated locomotion through control abstrac- 27. Schöner G, Dose M (1992) A dynamics systems approach tion. In: SIGGRAPH 95 Conference Proceedings, pp. 63– to task level systems integration used to plan and con- 70 trol autonomous vehicle motion. Robotics and Autonomous 8. Haumann DR, Parent RE (1988) The behavioral test-bed: Systems, 10:253–267 Obtaining complex behavior from simple rules. The Visual 28. Schöner G, Dose M, Engels C (1996) Dynamics of be- Computer, 4(6):332–347 haviour: theory and applications for autonomous robot 9. Ierusalimschy R, Figueiredo LH, Celes W (1996) Lua - an architectures. Robotics and Autonomous Systems, 16(2– extensible extension language. Software: Practice & Expe- 4):213–246 rience, 26(6):635–652 29. Steinhage A, Schöner G (1997) The dynamic approach to 10. Ierusalimschy R, Figueiredo LH, Celes W (1997) Reference autonomous robot navigation. In: Proceedings IEEE Inter- manual of the programming language Lua 3.0 national Symposium on Industrial Electronics 11. Ko H, Reich BD, Becket W, Badler NI (1994) Terrain 30. Tu X, Terzopoulos D (1994) Artificial fishes: Physics, lo- reasoning for human locomotion. In: Proc. Computer Ani- comotion, perception, behavior. In: Proceedings of SIG- mation ’94. IEEE Computer Society Press GRAPH ’94, pp. 43–50. ACM Press 12. Kokkevis E, Metaxas D, Badler NI (1995) Autonomous ani- 31. WebSite, a URL “http://www.tecgraf.puc-rio.br/lua” mation and control of four-legged animals. In: Proc. Graph- 32. WebSite, b URL “http://www.povray.org” ics Interface’95, Quebec City, Quebec, Canada 33. Wilfong G (1988) Motion planning in the presence of mov- 13. Kurlander D, Ling DT (1995) Planning-based control of ableobstacles. In: Proc. 4th ACM Symp. Computational interface animation. In: Proc CHI95 Conf., pp. 472–479. Geometry, pp. 179–288 ACM Press 34. Wilhelms J (1990) A “notion” for interactive behavioral 14. Large EW, Christensen HI, Bajcsy R (1999) Scaling the animation control. IEEE Computer Graphics and Applica- dynamic approach to path planning and control: Competi- tions, 10(3):14–22 tion among behavioral constraints. International Journal of Robotics Research, 18(1):37–58 15. Lethebridge TC, Ware C (1989) A simple heuristically- based method for expressive stimulus-response animation. Computers and Graphics, 13(3):297–303 16. Liu Z, Gortler SJ, Cohen MF (1994) Hierarchical spacetime control. In: Proceedings of SIGGRAPH ’94, pp. 35–42. ACM Press 17. Magnenat-Thalman N, Thalmann D (1995) Digital actors for interactive television. Proc. IEEE (Special Issue in Digi- tal Television, Part 2), 83(7):1022–1031 18. Neven H, Schöner G (1996) Dynamics parametrically controlled by image correlations organize robot navigation. Biological Cybernetics, pp. 293–307 19. Noser H, Thalmann D (1993) L-system-based behavioral animation. In: Proc. Pacific Graphics ’93, pp. 133–146 20. Noser H, Renault O, Thalmann D, Magnenat Thalmann N (1995) Navigation for digital actors based on synthetic 364 S. Goldenstein: Non-linear dynamical system approach to behavior modeling

SIOME KLEIN GOLDEN- DR.DIMITRIS METAXAS STEIN studied Electronic En- received his Diploma in Elec- gineering at UFRJ and then re- trical Engineering from the Na- ceived a MSc in Computer Sci- tional Technical University of ence at PUC, both located in Rio Athens in 1986, his MSc in de Janeiro – Brazil. Currently Computer Science from the Uni- he is pursuing his PhD in Com- versity of Maryland in 1988, puter and Information Science and his PhD degree in Computer Department at the University of Science from the University of Pennsylvania. Toronto in 1992. He joined the Department of Computer and In- formation Science, University of Pennsylvania as an Assistant Professor in September of 1992. Since January 1998 he has been an associate Professor in the same Department. Dr. Metaxas is EDWARD W. LARGE is an the director of the VAST Lab (Vision, Analysis and Simula- Assistant Professor at the Center tion Technologies Laboratory). He specializes in physics-based for Complex Systems at Florida modeling techniques with applications to computer vision, Atlantic University. He received graphics and medical image analysis and has published over his PhD from the Ohio State 90 research papers in refereed journals and conferences. He University in 1994. He has held has recently written a book titled “Physics-Based Deformable fellowships in Cognitive Science Models: Applications to Computer Vision, Graphics, and Med- and Psychology at University ical Imaging”, published by Kluwer Academic Publishers. He of Pennsylvania, and in Arti- organized the first IEEE Workshop on Physics-based Model- ficial Intelligence at Toshiba’s ing in Computer Vision, is an associate editor of GMIP and Research and Development Lab- PAMI, is on the editorial board of Medical Image Analysis, and oratory in Kawasaki, Japan. His is a coeditor of the Computer Vision and Image Understanding research focuses on the dynam- 1997 Special Issue on Physics-based Modeling and Reason- ics of human perception, action, ing. In 1996, he received the U.S. National Science Foundation and cognition and the design of Career Award, in 1997, an Office of Naval Research Young autonomous agents utilizing dynamical principles. His areas of Investigator Award. His research has received several best pa- specialization include music perception, auditory perception, per awards, including a 1998 Gold Medal in the Ninth World and robotics. Congress on Medical Informatics.