DECONSTRUCTING DIGITAL HUMANS © 2017 Alcon Entertainment, LLC Alcon 2017 ©

A stand-in was filmed during principal photography, with MPC implementing its CG Sean Young into the live action photography

ollywood has been making phenomenal RE-CREATING RACHAEL love with. This clone of Rachael was in fact a CG leaps and bounds in the creation of Denis Villeneuve’s 2049, a sequel creation, from the shoulders up, by MPC, who Hphotoreal digital humans in recent times. to the 1982 sci-fi noir film, features a scene in matched the digital face to what Young looked DE Notable standouts include ’s which the retired Blade Runner Deckard like in the 1982 movie. Benjamin Button work, ’s () meets a clone of Rachael (Sean Young’s performance in the original Blade resurrection of Paul Walker for and Young), the replicant he had previously fallen in Runner was an obvious source of reference, as CONS Industrial Light & Magic’s CG Grand Moff Tarkin were her roles in Dune (1984) and Stripes (1981). DIGITAL in . MPC then gathered a head scan of Young as she Three films released in 2017 also greatly We built a skull so appears today in a Light Stage, a specialised advanced the art, in different ways: Blade Runner lighting and capture rig built by University of TRUC 2049 with its digital Rachael; Logan in creating we could get key points Southern California Institute for Creative CG digi-doubles of the main ; and on the face to Technologies (USC ICT). The Light Stage is Guardians of the Galaxy Vol 2 in de-ageing Kurt synonymous with so many virtual actors in TING Russell. 3D Artist caught up with the studios understand the history. HUMANS behind those digital humans – MPC, Image “From that,” says MPC Ian Failes discovers how far the art and tech Engine and Lola VFX – to find out how they proportions… better supervisor Richard Clegg, “we built a skull so we created the lifelike performances while Richard Clegg, could get key points on the face to understand has come for photoreal CG humans in film managing to avoid the ‘ Valley’. Visual effects supervisor, MPC the proportions a little better.” This was married

38 39 v DECONSTRUCTING DIGITAL HUMANS

We’d trawl back and the skin, and all the flesh on the face moves HOW TO REPLICATE around as the actor talks and performs.” through the original MPC tested its approach on a couple of actual A REPLICANT movie, and find stuff shots from the original film. One of the major In crafting a digital Rachael forBlade Runner challenges, in those tests and the final shots for 2049, MPC had several facial feature where she has a little , was ensuring the Rachael’s challenges to overcome skin remained incredibly delicate. The studio flicker under her eye used ’s RenderMan for rendering, and took Richard Clegg, advantage of a new model that utilised path Visual effects supervisor, MPC traced subsurface scattering. GROWING HAIR LIGHT RIG COPY “It really helped with things like skin folds, and Although the on-set stand-in On set, a unique rotating disc of actor did wear a matching hair keeping detail on all the pores and all the minute light was used by style to Rachael, MPC replaced with ‘best guesses’ of camera focal lengths and details on the face,” says Clegg. “I think if you Roger Deakins the hair and generated eyelashes, for the scene. MPC rebuilt that rig lens information extrapolated from 1980s don’t get really nice photoreal eyes then that’s eyebrows and peach hair fuzz in its CG lighting setup. “We had footage of Young. when people start to say things like, ‘You lose completely digitally with its a scan of the light rig,” says senior A general head sculpt of digital Rachael her soul,’ or, ‘You don’t feel the emotion.’” proprietary hair and groom lighter Ian Cooke-Grimes. “We started to take shape, but more detail was Animating digital Rachael was all about toolset called Furtility. used this light rig to position all necessary. So stand-in actress Loren Peta, who studying the reference and replicating particular the lights, which was also tricky performed the role of Rachael during principal moments that evoked the character. “We’d trawl shot-by-shot because the lighting photography with some tracking dots on her back through the original movie, and find stuff was always moving.” face, was also scanned in the Light Stage. Here, where she has a little flicker under her eye, Peta and Young performed several FACS (Facial even,” notes Clegg. “Or where she looks a Action Coding System) poses, which would certain way, or she expresses a certain emotion. serve as the basis for blend shapes in . Then we’d try and build facial shapes, and build In addition, Peta was captured with into our facial rig enough diversity to be able to Dimensional Imaging’s videogrammetry rig. “It’s re-create, rebuild and reproduce what we were an array of around nine cameras with a very high seeing from the original.” frame rate,” explains Clegg. “It’s the same as photogrammetry, but you’re just capturing MORE THAN JUST multiple frames a second in video, and then you CG DOUBLES can get a mesh out of that. That gives us a 3D CG humans are perhaps most widely used as understanding of how the cheeks, and the lips, digi-doubles or for stunt or face-replacement

A wireframe view of MPC’s CG version of Rachael

THE LITTLE DETAILS EYE REALISM Rachael’s soft and subtle Often referred to as the ‘window features, including her dimples, into the soul,’ the eyes of digital were challenging components of actors are perhaps the most the face to replicate. A new important. MPC also paid close subsurface scattering model attention to digital Rachael’s implemented into Pixar’s eyelashes, noticing that widening RenderMan helped the team them ever so slightly could achieve the necessary detail. produce some distracting heavy-set shadows. © 2017 Alcon Entertainment, LLC Alcon 2017 ©

40 41 DECONSTRUCTING DIGITAL HUMANS

We could… really sit Jackman and Keen then went through Light Stage scans, and used the our CG humans into resulting data to build rigs for the characters of Logan, X-24 and Laura. “In order to rig the heads the action, whether it we have developed a new face system to be a face or full body accommodate a FACS-based approach,” says senior rigger Andrea Arghinenti. “For each facial replacement expression, split areas have been defined to give Martyn Culpitt, animators the level of control needed to Visual effects supervisor, Image Engine compose the performance. Layered corrective shapes and on-face free form deformers have then been used to refine the overall work. But these are typically fleeting shots and deformations created by the muscle shapes. The rarely for close-ups. In James Mangold’s Logan, face rig has been treated as a separate asset, the continuation and, possibly, ending of the loaded only when needed, and in order to story of X-Men’s Wolverine (Hugh Jackman), improve the playback performance we have stunt doubles got a major boost. been supporting three different resolutions.” Several scenes of the film made use of a Image Engine’s lookdev and lighting is done completely digital double Hugh Jackman face using its own proprietary toolset, and the for fast action moments, but there were also a studio adopted Solid Angle’s for number of fully-digital CG human shots for both rendering. “For shading,” details R&D lead Wolverine and his clone, X-24, that were seen in Andrew Kaufman, “we used a combination of Keen’s stunt double incredible close-ups. Image Engine handled the shaders from Solid Angle, alShaders and performed fighting scenes, with her face work, and the studio simultaneously delivered a in-house OSL shaders, all combined together in replaced with Image CG human for the Laura/X-23 character played Gaffer. At the time, both the subsurface model Engine’s CG one by Dafne Keen. and the caustics available with alSurface were The process began with the live action shoot definitely a game changer for us, both in terms of in which stunt actors (who would be replaced look/plausibility and performance. [Also], the with CG human faces for Jackman and Keen) hair shading performance offered by Arnold… would wear orange markers for tracking and impressed us.” rotomating. “There were also multiple witness The development of these hero CG faces for cameras for every take,” says Image Engine Logan was a big undertaking for Image Engine, visual effects supervisor Martyn Culpitt, “so we but the team were confident of achieving could also lock down the motion of the actors high-quality results. “We all know what Hugh and really sit our CG humans into the action, Jackman looks like and so there is really no whether it be a face or full body replacement.” wiggle room in getting it right,” notes Culpitt. “I Image Engine’s approach to modelling started remember [overall visual effects supervisor] with a series of photos of one of its own crew Chas Jarrett saying to me once that he didn’t members delivering FACS poses. “That gave us really believe in the Uncanny Valley and I agree a base to start working on our facial shapes with him; it either works or it doesn’t. We Image Engine’s library and also to kickstart the whole process of definitely went through phases where it was not Logan lookdev had Image Engine also ingesting scan data and adapting it to our Hugh and we had to figure out what was not scars, skin Corporation. Film Fox Century Twentieth 2017 © and TM Marvel. 2017 © duplication or sale for reserved.Not rights All crafted a CG blemishes and double for Dafne generic topology,” outlines lead character working, but we spent the time and the final trademark beard Keen as Laura modeller Marco Menco. heads are fantastic.”

THE LOOKDEV BEHIND LOGAN

A step-by-step look at how Image Engine approached the task of creating the lookdev for a digital Hugh Jackman, The process began with scanning Jackman in “Those images,” continues Flanquart, “give us “We also use the OLAT to build any kind of 3D Image Engine’s digi-double of Jackman came into Then the final shot, complete with motion blur, using USC ICT’s Light Stage system, not USC ICT’s Light Stage. “The OLAT (one light at a perfect insight of how light scatters through hair light rig based on those photos,” explains play in several different scenes, including one was intercut directly with other live action scenes only for the Logan character, but also time) from ICT was great reference,” notes Image and skin, how eyes react to light direction and Flanquart. “For example, combining a frontlit where a had carried out a driving of Jackman. Visual effects supervisor Martyn the X-24 clone Engine senior lookdev artist Adrien Flanquart. how specular highlights roll over the face of the photo and a backlit one results in a nice studio rig, stunt. Here, the ‘stuntie’ performs the action Culpitt points out that every single detail was of “The OLAT light rig triggers a flash of light from a actor.” This is just another reason why the Light or turning all of them from northern hemisphere wearing orange and black tracking dots. The importance, including the CG hair. “It is really like different position across a 360-degree sphere Stage is often sought out by visual effects studios of the rig ‘on’ but with low intensity results in an digi-double would be animated to match the a fingerprint, each person has their own hair around the actor. Each image represents a single to aid in the creation of photoreal digital doubles; overcast equivalent.” The combinations echo the action, and then rendered and composited to patterns,” he explains. “It’s pretty amazing getting light source and the combined result represents it’s had a role in the making of some of the most different kinds of indoor and outdoor lighting that match, as closely as possible, the lighting of the down to dialling little curly hairs that would push nearly all possible real world lighting of the actor.” well-known CG actors in VFX history. might be encountered on a normal film set. original photography. the believability.” 42 43 DECONSTRUCTING DIGITAL HUMANS

© 2017 Marvel

Lola de-aged Kurt Russell by referencing his look from films such as Used Cars (1980)

YOUNG AGAIN That was a first for to the adjustment of skin tone. Then more A different type of digital actor stars in James complicated effects begin, such as adjusting Gunn’s Guardians of the Galaxy Vol 2 in the form us, a de-aged kiss. We bone structure and re-creating skin (since skin of a young Kurt Russell. In the film, the actor tends to sag in the upper eyelids and other performs the role of Ego at his current age but weren’t really sure how places over time). This was done mostly in also in flashbacks that show him as a suave lover we were going to Autodesk Flame, but it also involved matte around 35 years younger. The resulting digital painting and CG elements. actor was made courtesy of the ‘youthification’ accomplish that Ego’s hair proved to be one of the trickiest expertise of Lola VFX, the studio responsible for Trent Claus, parts of this de-ageing project, as well as having Skinny Steve in : The First Visual effects supervisor, Lola VFX young Ego and his lover kiss. “That was a first for Avenger, a young in Ant-Man us, a de-aged kiss,” says Claus. “When going and a de-aged Robert Downey Jr in Captain into it, we weren’t really sure how we were going America: Civil War. “We don’t necessarily look for someone that to accomplish that at all, because of the For young Ego, Lola started with reference looks exactly like Kurt Russell,” says Claus. interaction between the two faces. It has to from Russell’s films, especially Escape From New “What we’re really looking for is someone who compress like real skin does, it has to react to York (1981) and Used Cars (1980). “Used Cars has the key features of that actor when they her, and has to look like Kurt kissing someone as was just about perfect,” recounts Lola visual were young, like a strong jawline, the same a youth. That was something new and exciting.” effects supervisor Trent Claus, “because not shape of chin with the little divot in the centre, only did Kurt have kind of a goofy, lighthearted laugh lines that flex and compress in the same THE AGE OF DIGITAL ACTORS 80s sort of look and demeanour in the film, he way or roughly the same way that Kurt’s do.” In addition to film work, some equally technically was the right age, he smiled a lot and was Schwartz was also photographed in Lola’s impressive work has been seen in games and charming, which were some of the same facial custom lighting rig where he re-created some of real-time projects. But right now it remains in the expressions we were looking for.” the lines and looks from Russell’s performance. movies where the most photoreal actors are on On set, Russell performed the role in make-up It gave the studio hi-res skin textures, with no show. And it looks set to continue, with more and with a wig, as well as a few tracking dots motion blur, to use in the de-ageing process. superhero films requiring digi-doubles, and a painted on his face. An age-correct actor double, Scans and CG models do form a part of Lola’s number of CG human-centred projects in the Aaron Schwartz, performed the scene for the approach to this work, but it is actually largely works like Alita: Battle Angel and Gemini Man. purposes of lighting reference. Lola was actually done via 2D compositing. Lola starts with typical One thing is certain: the age of the digital involved with production in casting that double. ‘beauty’ work on the skin, from wrinkle removal actor is definitely upon us. 44