Animation of Natural Virtual Characters Data-Driven Approach to Synthesizing Facial Animation Using Motion Capture Kerstin Ruhland, Mukta Prasad, and Rachel McDonnell ■ Trinity College Dublin n the entertainment industry, cartoon anima- of the refining stage, with easily editable curves for tors have a long tradition of creating expres- final polish by an animator. sive characters that are highly appealing to The proposed approach replaces the acting and Iaudiences. It is particularly important to get right blocking phases by recording the artists’ move- the movement of the face, eyes, and head because ments in front of a real-time video-based motion they are the main communicators of the charac- tracking and retargeting system (see Figure 1). ter’s internal state and emotions and an indication We use a commercially available and affordable of its level of engagement with real-time facial-retargeting system for this stage, others. However, creating these which creates the initial motion curves of the Producing cartoon animations highly appealing animations is character animation. These motion curves con- is a laborious task, and there a labor-intensive task that is not tain realistic human movements that are typically is a distinct lack of automatic supported by automatic tools. Re- dense in keyframes, making them difficult for an tools to help animators, ducing the animator’s effort and animator to edit. particularly with creating facial speeding up this process, while Our approach thus focuses on the next stage animation. To speed up and allowing for artistic style and of the pipeline. The initial motion curves of the ease this process, the proposed creativity, would be invaluable to character animation provide the input to our algo- method uses real-time video- production companies that cre- rithm. Our pattern-matching algorithm replaces based motion tracking to ate content for weekly cartoons the refining stage, by matching the motion curves generate facial motion as input and feature-length films. to a database of hand-animated motion curves. Animation production typi- This creates synthesized animations that reflect and then matches it to existing cally goes through a number of the artist’s keyframing and animation style. In hand-created animation creation stages: acting (where the final stage, the animator can focus on polish- curves. the animator acts out the ex- ing the animation for final production. We expect pressions in front of a mirror), that artists need to be acquainted with polishing blocking (where key body and face poses are cre- synthesized animations, as opposed to animations ated), refining pass (where overlapping actions are of their own work, before such a system can be in- added, movements are exaggerated, and the timing corporated into a production pipeline. Therefore, and spacing of motions is refined), and final pol- our solution will be mostly suitable for produc- ish (where animation curves are cleaned and small tions that require a lot of animations in a short details are added). We propose an alternate, more amount of time (such as weekly TV cartoons). automated solution for use when production speed This article demonstrates our approach’s ability is most important. Our approach aims to produce to dramatically reduce the number of keyframes of facial animation that closely resembles the quality the motion-capture data. We also describe a set of 30 July/August 2017 Published by the IEEE Computer Society 0272-1716/17/$33.00 © 2017 IEEE Motion capture Retargeting Virtual character with realistic motion Database of Our pattern-matching hand-created approach animations Final polish by animator Figure 1. Producing facial animation at the refining stage. A motion-capture system lets us predefine key facial poses and timings. Then, by utilizing a database of hand-created animations, our approach adjusts the motion-captured animations to match the artist’s style. The animator can easily edit the animation for final production in the final polish stage. user studies that show that our synthesized anima- content that we commissioned for the purpose of tions can achieve the same expressiveness and car- this project. The results show that our approach tooniness as hand-created animations and that an can effectively synthesize new animations (see animator can improve them with a small amount Figure 2). The large number of motion-capture of polish. (See the web extra video for an illustra- keyframes generated from the real-time retarget- tion of the proposed approach: https://youtu.be/ ing, usually one per frame, are replaced by well- PQP9965jcCk.) placed keyframes and in-betweens. To evaluate our method, we synthesized a series of emotional sentences, using some input motion Motion Capture and 3D Animation capture, a database of hand-animated motions, Motion capture is the favored method for applying and three different pattern-matching algorithms realistic motion to 3D characters, but the results (distance measurement, symbolic aggregate ap- can appear too realistic and lack expressiveness proximation, and a hidden Markov model). For when applied to highly stylized cartoon charac- this work, we were limited to a database consisting ters. Editing motion-capture data is difficult and of just one minute of high-quality hand-animated work intensive because keyframes are created for Motion capture Synthesized Motion capture Synthesized Motion capture Synthesized (a) (b) (c) Figure 2. Pattern-matching approach using example curves of professional cartoon animators to match to a motion-capture sequence creating a new synthesized animation. The example emotions are (a) happy, (b) angry, and (c) happy. IEEE Computer Graphics and Applications 31 Animation of Natural Virtual Characters each frame, which can limit the animator’s cre- generating a stylized 3D animation.6 Our proposed ativity. Various keyframe simplification methods method takes the opposite approach by using a da- have been implemented to make motion capture tabase of hand-created stylized motion curves to more accessible.1 Ideally, any animation tool us- replace realistic human motion curves. ing motion capture created for an animator should Although there has been an abundance of work simplify the keyframes. Our method automati- contributing to the automatic generation of real- cally reduces motion-capture keyframes using a istic facial animation, facial animation stylization pattern-matching approach, so it is well suited as has received less attention. Previous attempts have an animator tool. used a 2D approach, typically using video data as Professional cartoon animators draw on a input and producing low-resolution facial ani- wealth of established animation principles to cre- mation, where details in the face are minimized ate appealing characters. The literature has com- and the important features are therefore exagger- prehensively documented the application of these ated. Jung-Ju Choi and his colleagues added the traditional animation principles to 3D computer traditional principle of anticipation to motion- animation, explaining their meaning in 2D ani- captured facial expressions.7 They used principal mation and how they can best be translated into component analysis (PCA) to classify facial fea- 3D.2 Squash-and-stretch, for example, makes ani- tures into components with similar motions and mations more fluid, while exaggeration and antici- extract expressions from each component’s ani- pation help to translate the character’s emotions mation graph. An anticipation effect for a specific and thoughts to the audience. expression is found by searching the component’s animation graph and finding the best-matching inverted expression, which is blended into the Our objective is to detect the original expression. Unlike this approach, we present a pattern-matching method that matches animator patterns in a given similar patterns in a hand-created animation to a motion-capture curve. The newly synthesized motion-capture curve. animation can exhibit an animator’s unique style. Automatically modifying motion-capture curves to add the traditional animation principles is an Some previous research focused on synthesizing effective method of changing the style of the re- these principles for 3D objects, for example, by cre- alistic motion data and making it appear more ating squash-and-stretch or exaggerated motions. cartoon like. However, this approach lacks the The Cartoon Animation Filter3 applies an inverted input of a trained animator, resulting in some- Laplacian of a Gaussian filter to the motion signal what predictable results—for example, the same to enrich a variety of motion signals with antici- exaggeration applied to all motions can look re- pation, follow-through, exaggeration, and squash- petitive over time. Our aim is to incorporate input and-stretch. Relatively few papers concentrate on from a trained animator in order to create more the application of traditional animation principles varied results. We observe that a motion-capture to virtual characters. Ji-yong Kwon and In-Kwon curve retargeted to a virtual character is basically Lee transformed motion-capture data into rubber- a sequence of data points over a time interval. like exaggerated motion by breaking down the orig- Therefore, we draw inspiration from the study of inal skeleton into shorter segments and applying a time-series data analysis, the goal of which is to Bézier curve interpolation for the bending effect detect and estimate patterns and predict future and a mass-spring emulation creating a smooth trends. Our objective is similar in that we wish to movement.4 Jong-Hyuk Kim and his colleagues detect the animator patterns from a given motion- implemented an anticipation effect added to char- capture curve. acter animation.5 With the notion that a main The search for matching patterns in time-series movement is preceded by an opposite movement, data has been widely researched. The most com- they calculated the rotations and translations of monly used strategy to compare two sets of time- the center of mass and then calculated the antici- series data with each other is the evaluation of pation pose with a nonlinear optimization tech- Euclidean distance.