High Fidelity Facial Animation Capture and Retargeting with Contours
Total Page：16
File Type：pdf, Size：1020Kb
High Fidelity Facial Animation Capture and Retargeting With Contours Kiran S. Bhat∗ Rony Goldenthal† Yuting Ye‡ Ronald Mallet§ Michael Koperwas¶ Industrial Light & Magic Figure 1: Our performance-capture approach excels at capturing and retargeting mouth and eyelid motion accurately. Abstract can convey emotions and communicate with the audience. Percep- tion studies [Cunningham et al. 2005; Nusseck et al. 2008] show Human beings are naturally sensitive to subtle cues in facial expres- that humans rely heavily on facial expressions in communications. sions, especially in areas of the eyes and mouth. Current facial mo- For example, micro-expressions helps to decipher emotions; and tion capture methods fail to accurately reproduce motions in those lip reading is vital to understand dialogues. Even the slightest dis- areas due to multiple limitations. In this paper, we present a new sonance between these facial cues and viewers’ expectations may performance capture method that focuses on the perceptually im- result in false emotions [Ekman 1993]. To date, the art of animat- portant contour features on the face. Additionally, the output of our ing a realistic face remains in the hands of few expert artists. The two-step optimization scheme is also easily editable by an anima- quality of an animation depends on modelers’ ability to craft a large tor. To illustrate the strength of our system, we present a retargeting number of blendshapes that capture the most distinctive expressions application that incorporates primary contour lines to map a perfor- of the actor, and the animators’ ability to animate a complex set of mance with lip-sync from an actor to a creature. shapes. CR Categories: I.3.7 [Computer Graphics]: Three-Dimensional Performance capture techniques provide an efficient alternative to Graphics and Realism—Animation I.3.3 [Computer Graphics]: create compelling facial animations. Simultaneous capture of an Animation—Motion capture actor’s facial and body motion, where an actor is free to move on- set, is an ideal setting for production use. The most common so- Keywords: facial animation, performance capture, facial retarget- lution is marker-based video systems because they are flexible and ing, contours, curves, blendshape, correctives reliable. However, the sparse set of markers cannot provide suffi- cient details to reconstruct the nuances on a realistic face. Areas around the eyelids and the mouth are especially problematic due to 1 Introduction frequent self-occlusions seen from the camera. In practice, artists have to manually clean-up the animation, by both sculpting addi- Digital characters, both humans and creatures, are becoming preva- tional blendshapes and carefully adjusting the animation curves. lent in feature films and video games, oftentimes performing along The results are sometimes inconsistent across artists, and difficult side human actors on screen. With realistically rendered images, to evaluate objectively. Consequently, while background characters it is a grand challenge to deliver convincing facial animations that can be animated efficiently using performance capture, high quality hero characters are still labor intensive and expensive to produce. ∗email: [email protected] †email: [email protected] ‡ In this work, we present an artist-friendly system that is capable of email: [email protected] producing high fidelity facial animations. An important feature of §email: [email protected] our system is the incorporation of silhouette contours of the eye- ¶email: [email protected] lids and the inner mouth to reconstruct accurate animation. Given a set of blendshapes and a set of 2D features (markers and contours), Permission to make digital or hard copies of part or all of this work for personal or our system automatically solves for animation curves on the blend- classroom use is granted without fee provided that copies are not made or distributed shapes and produces an additional corrective at each frame of the for commercial advantage and that copies bear this notice and the full citation on the sequence. The combination of blendshapes and correctives accu- first page. Copyrights for components of this work owned by others than ACM must be rately match the movements of the skin, eyes and the inner mouth. honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. While it is relatively straightforward to match fixed features such as Request permissions from [email protected]. markers, it is a challenge to match the occluding contours because SCA 2013, July 19 – 21, 2013, Anaheim, California. Copyright © ACM 978-1-4503-2132-7/13/07 $15.00 7 the silhouette is constantly changing on the skin. We therefore em- tours to track these areas. In our work, we use contours to track the ploy a simple yet effective heuristic to dynamically associate the eyelid and the silhouette contour of the inner lip since it also cap- tracked curves with contours on the mesh at every frame based on tures lip motion orthogonal to the camera. We also enforce eyelid- visibility information. cornea contact to help determine the 3D location of the image-based curve. While a linear blendshape system with a fixed set of shapes can provide actor-specific spatial nuances such as wrinkles on the fore- Editable facial animation is obtained by fitting a facial rig [Orvalho head and fine-scale folds on the skin, it typically fails to capture et al. 2012] to the tracked data. Using a facial rig applies prior subtle and complex nonlinear interactions between different blend- knowledge that can be used to fill in missing information not cap- shapes. Moreover, it is difficult to match the inner mouth accu- tured from the video. For example, designated blendshape rigs are rately with a relatively small number of blendshapes. Instead of designed to capture the distinctive expressions of an actor [Pighin having artists manually sculpt correctives, we solve for these ad- et al. 1999; Liu et al. 2008; Li et al. 2010; Huang et al. 2011; Seo ditional corrective shapes on top of the blendshape animation to et al. 2011]. There has been research to apply anatomy-based mus- better match the features at every frame using a Laplacian defor- cle models [Sifakis et al. 2005] to simulate facial animation. The mation method [Botsch and Sorkine 2008]. This two-step approach results are physically plausible, but difficult to control and animate. not only adds the needed subtleties to the motion, but also ensures Data-driven models can be used to train facial shape priors or mo- that artists can work with a familiar blendshape system to edit the tion priors [Li et al. 2010; Weise et al. 2011; Ma et al. 2008]. How- animation. ever, the training sessions are intensive and it is difficult to obtain consistent training data. Although our work also builds on a blend- We apply our system to the performance of two professional ac- shape rig that artists are familiar with, we allow for out-of-model tors with subtle expressions and dialogues. The resulting animation animation that captures nuances without relying on an elaborate set delivers emotions, accurate eye blinks, and readable lips, all from a of shapes. In parallel to our work, Li et al.[2013] apply a similar sparse set of 2D features. The otherwise challenging lip rolling mo- approach to face tracking using per-frame correctives. To obtain tion is made possible in our system with dynamically selected oc- realtime performance, however, they use fixed contours around the cluding contours. In addition, we also show that an accurate perfor- outer mouth instead of the occluding contours of the inner mouth. mance of the actor can directly drive a believable talking creature. Recently, sketch-based interfaces [Lau et al. 2009; Miranda et al. With a simple heuristic to warp the actor’s lip shape to the talk- 2011] were applied to control facial expressions, especially of the ing portion of the creature’s mouth, a lip-synced creature is readily eye and mouth region. We are inspired by the ease of use of these achieved. Our work showcases the importance of incorporating oc- methods and similarly utilize the eyes and mouth contours to ani- cluding contours around the eyes and inner mouth, and presents a mate the face. production tested system for creating high quality facial animation. 2 Related work 3 Performance capture pipeline Highly detailed facial expressions and performance can be captured The input to our system are video streams of an actor’s facial perfor- by dedicated hardware in controlled environments. 3D scanning mance, a neutral mesh of the actor’s face, and a set of blendshapes techniques such as multi-view videos [Fyffe et al. 2011; Ghosh built for the actor. Our system then automatically reconstructs the et al. 2011; Beeler et al. 2011] and structural lights [Zhang et al. facial performance on the given mesh to match features captured 2004; Zhang and Huang 2004; Weise et al. 2009; Li et al. 2009; from the video. Mova ; Dimensional Imaging Ltd ] are used to capture geomet- ric details of the skin and photometric information. These tech- The input videos are recorded from two synchronized HD cameras niques can be used to create realistic digital doubles [Alexander attached to a helmet worn by the actor (Figure 1). The actor wears et al. 2010; Borshukov et al. 2005]. However, the constrained cap- stipple makeup to provide sparse dot patterns on the skin . This set- turing environment is not suitable for productions that require an ting is common in film production because it works with body mo- on-set capture of facial animation and body motion at the same cap, provides maximal coverage of the actor’s face, and the cameras time. Moreover, scanning techniques cannot acquire accurate in- are static relative to the head.