Automatic Expressive Deformations for Stylizing Motion

Paul Noble* and Wen Tang† School of Computing, University of Teesside

Figure 1: A run cycle. Above: before. Below: after expressive deformations have been applied.

Abstract

3D computer often struggles to compete with the flexibility and expressiveness commonly found in , particularly when rendered non-photorealistically. We present an animation tool that takes skeleton-driven 3D computer and generates expressive deformations to the character geometry. The technique is based upon the cartooning and animation concepts of ‘lines of action’ and ‘lines of motion’ and automatically infuses computer animations with some of the expressiveness displayed by traditional animation. Motion and pose-based expressive deformations are generated from the motion data and the character geometry is warped along each limb’s individual line of motion. The effect of this subtle, yet significant, warping is twofold: geometric inter-frame consistency is increased which helps create visually smoother animated sequences, and the warped geometry provides a novel solution to the problem of implied motion in non-photorealistic still images.

CR Categories: I.3.7 []: Animation; I.3.5 [Computer Graphics]: Curve, surface, solid, and object representations Figure 2: Examples of expressive limb deformations in Keywords: expressive deformations, cartoon animation, non- cartoons. © Hart (top) [Hart 1997]. (Used with permission.) photorealistic rendering, stylizing motion and joints can be broken [Williams 2001] if it makes for a more 1 Introduction appealing image or dynamic motion. As a result, the limbs of hand-crafted animated characters (both pencil and computer- Traditional have always had a rather flexible view of generated) are often distorted to accentuate a motion or imply an bone structure. An understanding of anatomy is crucial but of emotion (see figure 2). Our aim is to mimic these distortions in paramount importance are the fundamental principles of order to imbue 3D character animations with a degree of the traditional animation [Thomas and Johnston 1981; Lasseter 1987], expressiveness and fluidity found in traditional animation. which maintain that bones are there to be squashed and stretched, Commonly seen in 2D animation, and now becoming more *e-mail: [email protected] common in 3D, the primary cause of these distortions is the use of †e-mail: [email protected] what animators and cartoonists refer to as ‘Motion Lines’ and ‘Action Lines’ [Lee and Buscema 1978; Hart 1994; Hart 1997; Blair 1994; White 1986; Brooks and Pilcher 2001]. Action Lines are “the basis for rhythm, simplicity, and directness in animation.” [Blair 1994]. As can be seen in figure 3, these two types of line are closely related and are typically drawn as smooth curves or arcs. They are used by artists as a visual aid to add dynamism and maintain consistency of motion between frames. It is far easier to animate a few curves and have a character follow them than it is capture to cartoons themselves [Bregler et al. 2002]. The motion- capturing of cartoons allows the work of master animators to be retargeted to 3D models. The inherent drawback of this technique is that no new animation is actually created and there is a finite supply of suitable source material. The creation of a coherent piece of animation based upon the retargeted sequences of old films would be almost impossible.

The most recent work on CG cartoon animation involves applying an inverted Laplacian of a Gaussian filter to a motion signal [Wang et al. 2006] to automatically generate three of the principles of traditional animation; anticipation, exaggeration, and follow-through. This elegant technique produces excellent results for a range of animations but on more complex character animations can lead to excessive changes in the motion data. Exaggerating the motion of animations that contain interacting characters may be so fundamentally altered that they no longer work. For example, an animated punch could, after filtering, no longer land on its intended target. Figure 3: Lines of action and motion. Perhaps the biggest challenge facing many 3D computer © Hart [Hart 1997]. (Used with permission.) animators attempting to produce cartoon-style animations is the fact that most hand-drawn characters simply cannot be modelled to animate every joint. The action line is often thought of as an in 3D. Traditional animators distort their characters to maximize extension of the spine and, in traditional animation, indicates the their aesthetic appeal depending on the direction from which they overall pose and direction of a character. Closely related to action are being viewed but a single 3D model cannot encompass all of lines, motion lines indicate “the direction of the most accentuated the possible distortions required. To avoid the need for a new movement of the pose.” [Hart 1997]. When applied to the motion model for each new viewpoint, it is possible to deform a single of humanoid characters, these motion lines usually define the model depending on the current direction of view [Rademacher motion of the limbs and, whether consciously or subconsciously, 1999]. Several deformed models, each linked to a different key often lead to the distortion of these limbs in the final illustration viewpoint can be used to warp a base model. At each frame of an (see figures 2 and 3). animation the base geometry is distorted by interpolating these key deformations to produce geometry unique to a particular Whereas an artist can intuitively add deformations to a hand- viewpoint. This technique produces excellent expressive drawn character, the anatomy of most 3D computer-generated distortions of animated characters but the initial modelling work is characters is more rigid and, rather than a loose sketch of a highly labour-intensive and is also character-specific. Tools can skeleton, animators work with virtual joints and bones. These be provided to assist the with the task of creating the key skeletal structures incorporate transformation restrictions based on deformations directly from drawings [Li et al. 2003]. Again, this reality: joints have rotation limits, bones have fixed lengths, and technique relies heavily on the animator creating the distortions the associated geometry of the character must follow these rules. and, furthermore, alters the underlying skeletal animation. In this respect, skeleton-driven struggles to compete with the flexibility and expressiveness of traditional Another problem encountered when trying to produce cartoon- hand-drawn animation. style animations is how to imply motion in a non-photorealistic still image without using motion blur. Simulating motion blur in In this paper, we present an animation tool that dynamically photorealistic animations is a relatively simple matter [Potmesil deforms the limbs of computer-generated characters based on the and Chakravarty 1983] but these techniques can rarely be used pose and motion of their virtual bones. Our aim is to enable with Non-Photorealistic Renderings. Traditional animators and computer animators to quickly add a layer of expressiveness to an cartoonists use many visual cues and techniques to convey the animation and possible applications include use in the computer motion of objects [Blair 1994; White 1986; Brooks and Pilcher games industry or as a tool for creative animators. 2001]. Speed-lines, after-images, and jagged distortions have been applied successfully in computer graphics [Kawagishi et al. 2003; 2 Related Work Hsu and Lee 1994; Lake et al. 2000; Strothotte et al. 1994] but the deformation of the whole object can also imply motion. As clearly With non-photorealistic rendering (NPR) techniques maturing, illustrated by the tennis racket in the third frame of the lower interest in non-realistic and expressive computer graphics has sequence in figure 2, the exaggerated distortion of the racket increased in recent years as 3D computer animators look to the shows its velocity and implies its motion. This aspect of implied world of traditional animation for inspiration and understanding motion in traditional animation, which has been largely [Strothotte and Schlechtweg 2002; Chenney et al. 2002]]. The overlooked until now, is reflected in our expressive animation techniques used in traditional animation are just as relevant to system. computer animation [Lasseter 1987] but many of these principles and practices are so closely tied to their medium that transferring 3 Overview from pen-and-ink to computer graphics is not always a straight- forward matter. The following sections describe a tool that dynamically bends the limbs of computer-generated animated characters to create the A direct method for infusing computer animation with the appearance of a more stylized motion. Our algorithm first expressiveness commonly found in cartoons is to apply motion- determines prospective ‘motion-lines’ based on the key-framed e e

e e

Figure 4: A . Top: traditionally animated © White [White 1986]. (Used with permission.) Middle: reproduced Figure 6: Convex (left) and concave (right) distortions and using 3D animation software. Bottom: the same animation their relationship to vector e with subtle limb-bending applied creative process, but to act as an animation aid. We cannot hope to procedurally capture the skills used in the creation of a piece of animation, and nor do we hope to. This is, as always, the task of the artist, but we do aim to help emphasize and accentuate the

j2,t+1 animator‘s intentions. The tool has been implemented in Autodesk j1,t ® a Maya [MAYA 2006] and utilizes this proprietary software’s lattice and nonlinear bend deformer utilities. c e 4 Finding the lines of motion b j j 2,t 3,t θ In order to deform the limbs we must first find the lines of motion embedded within the animation of the character. The highly Figure 5: Joint vectors. Solid grey lines indicate joint and subjective nature of the limb-bending found in traditional bone locations at time t, dashed lines at time t+1 animation makes any attempts to define it completely, or capture it algorithmically, bound to failure. Instead, we have determined empirically, that the limb deformations typically depend on two animation and then accentuates this line by bending the geometry main factors: of the character’s limbs. These deformations are applied to the geometry associated with the eight bones connected to the knees • The bone’s velocity and elbows (for the sake of simplicity, the forearms and lower • The bone’s associated child joints legs are judged to consist of one bone rather than two). The system is automatic and can be applied to any skeleton-driven animated character. 4.1 Direction of Motion

The geometry of the character’s limb is transformed using simple The motion of the character is driven by a key-framed skeletal lattice deformers that act along the plane formed by the particular animation and the kinematic chains of the character are divided uniaxial (hinge) joint and its child joints, and perpendicular to the into five different parts representing the body and the limbs. The bone in question. A child joint is defined to be one that is one step primary determining factor for the line of motion is the bone’s further down the skeletal hierarchy. For example, the bending of direction of motion. This can be extracted from the animation by the lower leg would be in the plane formed by the knee, ankle, sampling the position of key joints in consecutive frames. Figure and toe joints and in a direction perpendicular to the shin. The 5 shows three virtual joints in two consecutive frames of deformation of each bone can, therefore, act in a concave or animation. When determining the bend of the bone joining j and convex manner. In this work we consider a concave bend to be 1 j2, three vectors are required, and calculated by equation 1: one that bends in the opposite direction to the front of the leg, which can be determined from the local rotation axis of the j2,t − j1,t j3,t − j2,t j2,t+1 − j2,t originating joint of the bone. a = , b = , c = (1) j − j j − j j − j 2,t 1,t 3,t 2,t 2,t+1 2,t The direction and magnitude of the deformation are dependent upon the aforementioned motion-line, which is itself defined by where t is the current frame of animation. The vector e is found by the existing animation or via several user-controlled attributes. projecting c on to the plane formed by a and b, and at an angle Consequently, the tool’s success depends upon the quality of the perpendicular to both. The dot product of a and e gives the angle initial animation or the skill of the animator. A good piece of θ. If e lies in either shaded quadrant, as illustrated in figure 6, then animation should possess inherent lines of action and motion and the bend is convex, otherwise it is concave. Hence, there should our task is to simply extract and help highlight them. The be minimal distortion if e is perpendicular or parallel to a. principle behind this work is not to attempt to replicate the ⎧0,or1 Minimal distortion a • e = ⎨ (2) ⎩otherwise convex or concave distortion

The magnitude of e gives us the velocity of the joint in the plane ab, which is used in conjunction with θ to determine the magnitude of the bend. The calculation of the direction and magnitude of this bend are covered in section 5.3.

4.2 Influence of Associated Child Joints

The second influence on the line-of-motion we considered was the role of child joints. A child joint is defined to be one that is one step further down the skeletal hierarchy. For example, the ankle would be the child of the thigh bone. Child joints affect the bend of the parent bone because motion-lines are generally drawn as Figure 7: Lattice and non-linear bend deformers smooth curves and, as a consequence, any distortion that helps the overall shape of the limb approximate a curve is beneficial. For instance, at the maximum planar flexion of the ankle when the complex geometry using relatively few control points. toes are pointed down, we have observed that the lower leg is Furthermore, we can deform the lattice itself using other more likely to be drawn with a concave bend. The thigh is, more deformers. In our system we apply lattice deformers to the often than not, drawn with a convex bend because the knee is associated geometry of each bone and then manipulate the lattice usually bent. using nonlinear bend deformers.

To mimic this effect in our system, we assigned each joint a 5.2 Nonlinear bend deformers threshold value based upon the rest state of the skeleton. If the angle between a particular joint and its associated child joint Bend deformers warp geometry along a circular arc. They are exceeds this threshold, then the joint axes are skewed to increase controlled via a handle which defines the radius of the deforming the likelihood of a particular distortion (see figure 9). Whether arc (see figure 7). In our system the orientation of the deformer is this distortion is concave or convex will depend on the orientation defined by the bone itself and the bend handle is controlled by a of the hinge joint in question. Some joints rotate forwards, (for vector, d, the calculation of which is covered in the following example, the ankle) and some rotate backwards (e.g. the knee). By section. By default the deformer bends around its centre, so an decreasing the range of the shaded quadrants, the joint becomes expression is used to translate the lattice in the opposite direction more conducive to deformations that will help the limb of d to ensure the character geometry maintains its continuity at approximate a curve to convey the expressiveness of the motion. the joints.

4.3 Hyperextensions 5.3 Direction and magnitude of bend

As discussed in section 3, the direction of the bend is always Hyperextension of the knee or elbow is undesirable in everyday perpendicular to the bone in question and in the plane of the life but is common in cartoon-style animations (see the four triangle formed by the joints j1, j2, and j3 (see figure 5). We can corner images in figure 4). This ‘breaking of the joints’ [Williams calculate the vector d, as: 2001] occurs when the limb is fully extended and is treated in our system as a special case. In order to hyperextend the joint we d = a ×(a ×e) (3) deform the limb as a whole rather than each individual bone. If the joint reaches its rotation limit then the two adjoining bones are treated as one and any deformations of the individual bones are The magnitude of the bend is dependent on the user-defined omitted. parameter bmax and the value of θ. bmax is the maximum possible bend value for the limb. As seen in section 4.1, the bend should be 5 Deforming the limbs minimal if e is perpendicular or parallel to a, and at its maximum halfway between these angles. A simple trigonometric function is The transformation of the geometry is dependent on the particular used to interpolate the bend between these points and we define character-rig and the software being used. Most 3D animation the new bend vector d’, as equation 4: software supports skeleton-driven deformation and a host of other tools to warp geometry. We have implemented our system in ® d Autodesk Maya and we utilize its lattice and nonlinear bend d′ = bmax ()sin 2θ (4) deformer utilities. d

5.1 Lattice deformers The dashed line in figure 8 shows this interpolation curve which defines the maximum concave and convex distortions. As can be A lattice deformer allows you to warp an object or the seen by equation 3, d will always point in the opposite direction to components of an object. The lattice surrounds its associated e. This causes the bend vector to invert if e moves behind the geometry and any transformations applied to the lattice are bone and allows equation 4 to account for all quadrants illustrated propagated to the geometry [Sederberg and Parry 1986; in figure 6. MacCracken and Joy 1996]. This allows us to manipulate bMAX j1 a

θ π π j2 2 b ϕα

j3 α−ϕ

Figure 8: Bend interpolation curves showing the standard curve (dashed) and the influence of child joints Figure 9: The influence of child joints

5.4 Influence of the child joint bmax As described in section 4.2, the role of child joints in influencing vmin 0.0 0.4 0.8 1.2 1.6 2.0 the expressive deformation must be considered. To calculate the influence of these joints we first find the angle between the two 0.0 bones,ϕ , by the dot product of vectors a and b, where b is the vector between the joint j2 and the child joint j3 as shown in figure 5. Each joint is assigned a threshold valueα based upon the rest position of the skeleton and, where ϕ is less thanα , equation 4 0.1 is modified to take the secondary deformation influence into account, and becomes equation 5:

d 0.2 d′ = bmax ()sin 2γ (5) d

⎧ π 3π θ + c()α − ϕ , if < (θ mod π ) < where ⎪ 0.3 γ = ⎨ 4 4 ⎩⎪θ − c()α −ϕ , otherwise Figure 10: the influence of vmin and bmax on an animated racket and c is a user-defined scalar constant. This has the effect of with current velocity approximately 0.26. The model consists of increasing the area of the concave peak and decreasing the area of a single bone and therefore no child joints so their effect is the convex peak in our interpolation curve (see solid line in figure negated. 8). As mentioned in section 4.2, the scaling of these curves is related to the orientation of the hinge joint and bone being By making the joint velocity inversely proportional to the bone considered. The influence of some child joints will cause the length we ensure consistent relative deformations between bones bones to be more inclined to convex, rather than concave, of differing length and also enable the character skeleton to be bending. scaled without affecting the generated bend vector. The velocity

threshold parameters, vmin and vmax, can be pre-computed by 5.5 Influence of velocity calculating v for each joint in every frame of the animated sequence and setting vmin to its minimum value and vmax to its As discussed, the primary factor on the deformation of limbs is maximum value. Alternatively, to maintain consistency between motion, so the magnitude of the velocity of the joint is also separate animated sequences, both threshold parameters can be influential. If the limb is stationary there should be no motion- user-defined. based bending and the faster it moves the greater the deformation should be. We add velocity threshold parameters, vmin and vmax to equation 5 to incorporate the velocity effect as equation 6: 5.6 Smoothing

d ⎛⎛ v − v ⎞ ⎞ The calculation of the direction of the bend is based primarily on d′ = ⎜⎜ min ⎟b sin 2γ ⎟ (6) d ⎜⎜ v − v ⎟ max ⎟ the existing animation. As a result, it is entirely possible that ⎝⎝ max min ⎠ ⎠ sudden changes in direction or pace between frames can result in a bone ‘popping’ from concave to convex in adjacent frames. e where v = and vmin < v < vmax Rather than complementing the animation and adding a layer of j2,t − j1,t expressiveness, this bending would be a distraction.

To resolve this problem a simple smoothing algorithm is used. The magnitude of the bend is calculated for each bone over multiple frames and then the average is taken. Although this is a References relatively crude solution, we have found that good character animations possess a high degree of continuity of motion between BLAIR, P. 1994. Cartoon Animation. Walter Foster Publishing. frames and motion-capture data only causes the bones to ‘pop’ between convex and concave at very low velocities. As can be BREGLER, C., LOEB, L., CHUANG, E., and DESHPANDE, H. seen from equation 6, this results in either very small, 2002. Turning to the masters: motion capturing cartoons. In imperceptible deformations or no deformations at all, which ACM Transactions on Graphics, 21, 3, 399-407. render the need for smoothing somewhat redundant. BROOKS, B., PILCHER, T., and EDGELL, S.(editor). 2001. The 6 Results Complete Cartooning Course. David & Charles Publishers.

The system has been successfully implemented in Maya and the CHENNEY, S., PINGEL, M., IVERSON, R., and SZYMANSKI, expressive shape deformations can be generated automatically M. 2002. Simulating Cartoon Style Animation, In Proceedings from any skeleton-driven animation (motion-capture data driven of NPAR 2002, ACM Press, Annecy, France, 133-138. or hand-animated) or manually through a series of sliders. The maximum bend, bmax, velocity threshold, vmin and vmax, and child- HART, C. 1994. Everything You Ever Wanted To Know About joint influence, c, are user-defined parameters and figure 10 Cartooning But Were Afraid To Draw. Watson-Guptill. shows differing degrees of bend generated for the same frame of an animation by varying bmax and vmin. HART, C. 1997. How to Draw Animation. Watson-Guptill.

7 Conclusions and future work HSU, S. C., and LEE, I. H. H. 1994. Drawing and animation using skeletal strokes, In Proceedings of ACM SIGGRAPH 94, ACM Press, New York, 109-118.

KAWAGISHI, I., HATSUYAMA, K., and KONDO, K. 2003. Cartoon Blur: Non-Photorealistic Motion Blur, In Proceedings of Computer Graphics International 2003, Tokyo, Japan, 276- 281.

LAKE, A., MARSHALL, C., HARRIS, M., and BLACKSTEIN, M. 2000. Stylized rendering techniques for scalable real-time 3d animation, In Proceedings of NPAR 2000, Annecy, France, Figure 11: A still from a motion-captured animation of a 13-20. boxer performing an uppercut. Left: original. Right: with our LASSETER, J. 1987 . Principles of Traditional Animation system applied Applied to 3D Computer Animation, In Computer Graphics (Proceedings of ACM SIGGRAPH 87), 21, 4, ACM, 35-44. This paper has presented an animation tool that automatically adds expressive shape deformations to the limbs of animated LEE, S. and BUSCEMA, J. 1978. How to Draw Comics the characters. The direction and magnitude of the bending is derived Marvel Way. Simon and Schuster. from the originating animation. As such, these pose and motion- based deformations quickly add a layer of expressiveness to the LI, Y., GLEICHER, M., XU, Y., and SHUM, H. 2003. Stylizing 3D animation and help accentuate the underlying animation. motion with drawings. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer The primary area of future work is the extension of the tool to Animation. Symposium on Computer Animation. Eurographics include what traditional animators refer to as the line of action. Association, Aire-la-Ville, Switzerland, 309-319 Closely related to lines of motion, they are often viewed as an extension of the spine of a character and so the extraction of this MACCRACKEN, R. and JOY, K. I. 1996. Free-form line from the animation and its analysis could greatly influence deformations with lattices of arbitrary topology. In the bending of a character’s limbs. Proceedings of ACM SIGGRAPH 96, ACM Press, New York, 181-188. A relatively simple enhancement to the system would be the use of more complex bending functions to allow asymmetric bends or MAYA. 2006. Autodesk, Inc. S-shaped deformations. This would enable adjoining bones to bend in opposite directions while still allowing the limb as a NOBLE, P. and TANG, W. 2004. Modelling and Animating whole to follow a smooth curve. The blending of adjacent bend Cartoon Hair with NURBS Surfaces, In Proceedings of deformers to ensure continuity at the joints would also improve Computer Graphics International 2004, IEEE Press, Crete, the system. Greece, 60-67.

A more sophisticated smoothing algorithm could also be easily POTMESIL, M. and CHAKRAVARTY, I. 1983. "Modeling developed and an investigation into bend interpolation functions motion blur in computer-generated images", In Computer could be carried out. The trigonometric function described in this Graphics (Proceedings of ACM SIGGRAPH 1983), 17, 3, paper yields acceptable results but the analysis of a variety of ACM, 389-399. functions could be carried out to determine which, if any, generates the most visually pleasing results.

RADEMACHER, P. 1999. View-dependent geometry. In Proceedings of ACM SIGGRAPH 99, ACM Press/Addison- THOMAS, F. and JOHNSTON, O. 1981. Disney Animation - The Wesley Publishing Co., New York,, 439-446. Illusion of Life. Abbeville Press.

SEDERBERG, T. W. and PARRY, S. R. 1986. Free-form VANZO, S. Interview: 3D Animation in Futurama, deformation of solid geometric models. In Computer Graphics http://www.gotfuturama.com (Proceedings ACM SIGGRAPH 86), 20, 4, ACM, 151-160. WANG, J., DRUCKER, S. M., AGRAWALA, M., and COHEN, STROTHOTTE, T. and SCHLECHTWEG, S. 2002. Non- M. F. 2006 The Cartoon Animation Filter. ACM Transactions Photorealistic Computer Graphics - Modeling, Rendering, and on Graphics. Animation. Morgan Kaufmann. WHITE, T. 1986. The Animator's Workbook. Phaidon. STROTHOTTE, T., PREIM, B., RAAB, A., SCHUMANN, J., and FORSEY, D. R. 1994. How to render frames and WILLIAMS, R.. 2001. The Animator's Survival Kit. Faber and influence people, Computer Graphics Forum, 13, 3, 455-466. Faber.

Figure 12: A ‘sneak’ walk cycle similar to the image shown at the top of figure 2. Top – the original cycle. Bottom – the same animation with expressive shape deformations applied.

Figure 13: A before (right) and after (left) of a motion-captured ballet dance.