news

Technology | DOI:10.1145/3403972 Don Monroe Digital Humans on the Big Screen Motion pictures are using new techniques in computer-generated imagery to create feature-length performances by convincingly “de-aged” actors.

RTIFICIAL IMAGES HAVE been around almost as long as movies. As com- puting power has grown and digital photography Ahas become commonplace, special ef- fects have increasingly been created digitally, and have become much more realistic as a result. ACM’s Turing Award for 2019 to Patrick M. Hanrahan and Edwin E. Cat- mull reflected in part their contribu- tions to computer-generated imagery (CGI), notably at the pioneering anima- tion company . CGI is best known in science fic- tion or other fantastic settings, where audiences presumably already have suspended their disbelief. Similarly, exotic creatures can be compelling In Ang Lee’s 2019 action movie Gemini Man, (right) is confronted by a younger when they display even primitive hu- clone of himself (left). This filmmakers went through a complicated process to “de-age” man facial expressions. Increasingly, Smith for the younger role. however, CGI is used to save money on time, extras, or sets in even mun- actors. also in- “In many ways it felt less about a visual dane scenes for dramatic movies. creasingly will augment the labor-inten- effect and more like we were creating a re- To represent principal characters, sive effects-generation process, allowing ality and aiming to work out exactly how however, filmmakers must contend filmmakers to tell new types of stories. the human face works, operates, but also with our fine-tuned sensitivity to facial how it ages over time,” added Stuart Ad- expressions. Falling short can leave Synthetic Performers cock, facial supervisor at Weta. viewers in the “ valley,” dis- “The idea of de-aging has been around Because of this wholesale replacement, tracted or even repulsed by a zombie- for a while, and companies like Lola Vi- “we were not limited to any photogra- like representation. “Trying to do real- sual Effects have been doing an amaz- phy,” Adcock said. “We had full freedom, istic humans is still the most difficult ing job of using 2D tools to basically so when we did need to take it some- aspect of visual effects,” said Craig Bar- track young patches onto old faces to where else, then we were able to do that.” ron, creative director at visual develop- make people look convincingly a differ- To build its model, the team first ran ment and experience company Magno- ent age,” said Guy Williams of Weta Smith through various facial exercises, pus. Barron shared the 2008 Academy Digital in Wellington, New Zealand. such as raising his upper lip. Such “ac- Award for Best Visual Effects for The Cu- Williams was video-effects supervisor tion units,” which have overlapping, rious Case of Benjamin Button, in which for Ang Lee’s 2019 action movie Gemini interacting effects in various regions the title character ages backward from Man, in which Will Smith played an as- of the face, constitute the elements of an old man to an infant. sassin confronted by a younger clone of the widely used facial action coding In the last decade, many films have himself. In that film, filmmakers took system (FACS), originally based for de- included short flashbacks with younger the process a step further, he said. “In- tailed understanding of the skull and versions of their characters. Within the stead of modifying an image to make a its muscles. After a year or so of tuning last year, however, some films have used performance looks younger, we erase a “puppet” to reproduce “current-age new techniques to create feature-length the image, create a synthetic young per- Will,” they turned their attention to his

performances by convincingly “de-aged” formance, and place it into the shot.” younger clone. OF @GEMINIMANMOVIE/TWITTER COURTESY IMAGE

12 COMMUNICATIONS OF THE ACM | AUGUST 2020 | VOL. 63 | NO. 8 news

Adcock likens this process to cre- ating a musical score derived from a ACM performance on one instrument. This Instead of using representation allows the same musi- animators to build a Member cal piece to be performed on another instrument, which imparts its own puppet with DeNiro’s characteristic sound. facial motions, the News To construct the “score,” the team HUMAN-COMPUTER built on its previous expertise animating FLUX software PARTNERSHIPS fantastic characters like Gollum in The morphed captured Wendy Mackay Hobbit, equipping Smith with facial is a research performance and director at Inria markers and a head-mounted cam- Paris-Saclay, the era rig that recorded his expressions. lighting data to create French national These rigs have been getting sleeker research the new image. institute for the and lighter all the time, Adcock said. digital sciences, at the Such “performance capture,” as con- Université Paris-Saclay in Paris, trasted with “motion capture” of bodily France. She heads up the ExSitu movements, provides extremely high- research group, a lab of 30 people that explores the limits quality facial data for the animators. pipeline to construct the de-aged im- of human-computer interaction. ages. In addition to FACS capture from “I have always been Supporting the Storytellers the current actors, the team acquired interested in how people For renowned director Martin Scorsese, huge numbers of images from their pre- interact with technology,” Mackay says, and human- however, headgear and facial markers vious performances over the decades, al- computer partnerships have (dots on the actors faces) were still un- lowing them to select a facial model for become a focus for her. People acceptably disruptive to the human in- each of the years in the film. Rather than react to technology, she points out, but they also appropriate it teractions he sought in the 2019 epic reproduce the exact look of De Niro from and do innovative things with it; . The film features Robert Taxi Driver (1976) or Goodfellas (1990), as a result, technology should De Niro, , and , all however, they strove to create a de-aged be designed to support user- now in their late 70s, in roles spanning version of his Irishman character. More- innovation and enhance users’ capabilities, she says. many ages across several decades, so a over, instead of using animators to build Mackay received a bachelor’s strategy for de-aging was critical. This a puppet with De Niro’s facial motions, degree in experimental resulted in the film being nominated their FLUX software morphed the cap- psychology from the University of California, San Diego; an for Best Special Effects and nine other tured performance and lighting data to M.S. in experimental psychology Academy Awards. create the new image. from , Beginning several years earlier, The result was a compelling de- and a Ph.D. in Management in Pablo Helman and his team from In- aged character that let the actors act— Technological Innovation from the Massachusetts Institute dustrial Light and Magic (ILM), the and interact. of Technology. special effects house founded by “If you’re going to get the top actors She started her professional George Lucas, worked with Scorsese of our time, you need to support their career at Digital Equipment to develop ways to create the desired acting process and not create a tech- Corporation (DEC), worked there in different capacities images much less intrusively. To cap- nology or impose a technology that over a period of 11 years. She ture the performances, they built a would somehow diminish that per- was working on something new bulky apparatus in which a normal formance. That’s what they were able at the time: human-computer interaction. “It was a new field, camera was flanked by two infrared to achieve,” said Barron, who said the combining psychology and cameras and associated infrared less-intrusive technology “allows more how we think about people and lighting that provided synchronized, adoption among a wider group of peo- technology, and putting them high-quality stereoscopic information ple into different kind of films.” together, with an emphasis on innovation,” she explains. needing neither visible facial markers After DEC, Mackay nor changes in the lighting desired by Under the Hood joined academia, teaching at the filmmakers. Unlike the latest superhero movie, a Denmark’s University of Aarhus, France’s University of Paris-Sud, The size and weight of this equipment film like The Irishman succeeds when and even as posed some challenges; for example, the audience forgets about its tech- a visiting professor. precluding the popular hand-held Stea- nical sophistication. Nonetheless, In the future, Mackay dicam shots. Moreover, although the achieving this realism requires an plans to explore the physics of interactions with digital infrared light did not interfere with the enormous, diverse team innovating in material. “We will use this to main camera, the infrared cameras were both hardware and software (and a rethink how we use intelligent sensitive to cigarette smoke, and could reported $160-million budget). systems— and not see through vintage car windshields. “There are people that are creative, other types of AI—to enhance users’ capabilities, to move In parallel with the hardware, the they’re thinking about design and light- people forward and do more.” ILM team developed a new software ing and composition and performance —John Delaney

AUGUST 2020 | VOL. 63 | NO. 8 | COMMUNICATIONS OF THE ACM 13 news

of storytelling, and there’s the people scheduled for release late this year. under the hood that have to make that all The filmmakers obtained legal per- work. It’s really a collaboration between “The power of AI missions to use the actors’ likenesses the technologist and the artist,” said Bar- being applied to in these films. Still, some commenters ron, whose current work at Magnopus are worried about the effect of recycling aims to extend storytelling to the realm visual effects is actors from previous eras will make it of virtual and . “It’s the relatively new and harder for current actors to find roles, collaboration that determines whether except as a blank slate onto which more the project is successful or not.” there are huge famous faces are mapped. For example, the Weta team worked potentials for that,” Ironically, one of De Niro’s break- together to improve the “facial solver” out roles was as a young Vito Corleone that represents the captured positions Barron said. in The Godfather Part II. With today’s of face markers and other features in technology, he might have been de- terms of underlying action units. Al- moted to a body double for a de-aged though there may be different ways Marlon Brando. to break down the various training movements, they used deep learning Further Reading to ensure their description matched the ability to put words in the mouths the way their animators think about of politicians, or to put celebrities into Seymour, M., those movements. With that training, pornographic scenes. Fortunately, such De-Aging the Irishman, fxguide, December 2019, https://bit.ly/2XZV1DE the solver could then decide, for each “” are unlikely to have the frame of the footage, which muscles Hollywood-level resources and actor co- Tonelli, B., The Lies of The Irishman, Slate, August were firing, Stuart said. “It gave us a operation used to make truly convincing 2019, https://bit.ly/3cATzwF good starting point.” fakes, although they are already good Pioneers of Modern “The power of AI being applied enough to cause trouble. (Some critics Recognized with ACM A.M. Turing Award, to visual effects is relatively new and noted that The Irishman repeated some ACM, March 18, 2020, https://awards.acm. there are huge potentials for that,” implausible claims from the book on org/about/2019-turing Barron said. “To harness the power of which it was based, but worries about The Hobbit: An Unexpected Journey the computer to teach it to simulate movies distorting historical facts are VFX | Breakdown - Gollum, Weta Digital, reality, whether it’s through creating a nothing new.) https://bit.ly/2UbKgg performance or a synthetic human or Another concern is reuse of actors Dr. Ekman explains FACS (Facial Action an environment, I think will have a lot who are unavailable, or even dead. The Coding System), https://bit.ly/3gRRakx of benefit to creating more and more 2016 “ story” , for How The Irishman’s Groundbreaking VFX credible illusions.” example, included a brief but contro- Took Anti-Aging To the Next Level, Netflix, versial appearance by Peter Cushing, https://bit.ly/2AJFU99 Bringing Back the Dead who had died in 1994. The most ambi- Don Monroe is a science and technology writer based in Creating lifelike representations of ac- tious re-animation so far is the reported Boston, MA, USA. tors in roles they never played does casting of James Dean, who died in raise challenging ethical issues. One is 1955, as a costar in the film Finding Jack, © 2020 ACM 0001-0782/20/8 $15.00

Milestones Balcan to Receive ACM Grace Murray Hopper Award

ACM named Maria Florina leader in the theory of how AI learning methods. clustering algorithms. She “Nina” Balcan of Carnegie systems can learn with limited She also made significant also devised novel clustering Mellon University to receive supervision. More broadly, contributions to active learning algorithms derived from these the ACM Grace Murray Hopper her work has realigned the by establishing performance theoretical foundations, and Award for her contributions to foundations of machine learning, guarantees for active learning showed applications of these minimally supervised learning. and consequently ushered in that hold even when “noise” algorithms in computational Balcan’s pioneering work many new applications that have is present in the data, and biology and Web search. in machine learning solved brought about leapfrog advances with colleagues she developed The ACM Grace Murray longstanding open problems, in this exciting area of artificial algorithms that can learn more Hopper Award is given to the enabled entire lines of research intelligence.” efficiently under more specialized outstanding young (under 35) crucial for modern AI systems, Balcan introduced the first forms of “label noise.” computer professional of the and set the agenda of the field for general theoretical framework Balcan proposed a year, selected on the basis of a years to come. for semi-supervised learning, theoretical foundation for single recent major technical or ACM President Cherri M. showing how to achieve provable understanding the general service contribution. The award Pancake said although Balkan guarantees on the performance kinds of structures that can is accompanied by a prize of is still in the early stages of of such techniques with concrete be detected by clustering, $35,000 (financial support her career, “she has already implications for many different as well as characterizing for this award is provided by established herself as the world types of semi-supervised the functionality of specific Microsoft).

14 COMMUNICATIONS OF THE ACM | AUGUST 2020 | VOL. 63 | NO. 8