Digital Humans on the Big Screen

news Technology | DOI:10.1145/3403972 Don Monroe Digital Humans on the Big Screen Motion pictures are using new techniques in computer-generated imagery to create feature-length performances by convincingly “de-aged” actors. RTIFICIAL IMAGES HAVE been around almost as long as movies. As com- puting power has grown and digital photography Ahas become commonplace, special effects have increasingly been created digitally, and have become much more realistic as a result. ACM’s Turing Award for 2019 to Patrick M. Hanrahan and Edwin E. Cat- mull reflected in part their contribu- tions to computer-generated imagery (CGI), notably at the pioneering animation company Pixar. CGI is best known in science fic- tion or other fantastic settings, where audiences presumably already have suspended their disbelief. Similarly, exotic creatures can be compelling In Ang Lee’s 2019 action movie Gemini Man, Will Smith (right) is confronted by a younger when they display even primitive hu- clone of himself (left). This filmmakers went through a complicated process to “de-age” man facial expressions. Increasingly, Smith for the younger role. however, CGI is used to save money on time, extras, or sets in even mun- actors. Artificial intelligence also in- “In many ways it felt less about a visual dane scenes for dramatic movies. creasingly will augment the labor-inten- effect and more like we were creating a re- To represent principal characters, sive effects-generation process, allowing ality and aiming to work out exactly how however, filmmakers must contend filmmakers to tell new types of stories. the human face works, operates, but also with our fine-tuned sensitivity to facial how it ages over time,” added Stuart Ad- expressions. Falling short can leave Synthetic Performers cock, facial animation supervisor at Weta. viewers in the “uncanny valley,” dis- “The idea of de-aging has been around Because of this wholesale replacement, tracted or even repulsed by a zombie- for a while, and companies like Lola Vi- “we were not limited to any photogra- like representation. “Trying to do real- sual Effects have been doing an amaz- phy,” Adcock said. “We had full freedom, istic humans is still the most difficult ing job of using 2D tools to basically so when we did need to take it some- aspect of visual effects,” said Craig Bar- track young patches onto old faces to where else, then we were able to do that.” ron, creative director at visual develop- make people look convincingly a differ- To build its model, the team first ran ment and experience company Magno- ent age,” said Guy Williams of Weta Smith through various facial exercises, pus. Barron shared the 2008 Academy Digital in Wellington, New Zealand. such as raising his upper lip. Such “ac- Award for Best Visual Effects for The Cu- Williams was video-effects supervisor tion units,” which have overlapping, rious Case of Benjamin Button, in which for Ang Lee’s 2019 action movie Gemini interacting effects in various regions the title character ages backward from Man, in which Will Smith played an as- of the face, constitute the elements of an old man to an infant. sassin confronted by a younger clone of the widely used facial action coding In the last decade, many films have himself. In that film, filmmakers took system (FACS), originally based for de- included short flashbacks with younger the process a step further, he said. “In- tailed understanding of the skull and versions of their characters. Within the stead of modifying an image to make a its muscles. After a year or so of tuning last year, however, some films have used performance looks younger, we erase a “puppet” to reproduce “current-age new techniques to create feature-length the image, create a synthetic young per- Will,” they turned their attention to his performances by convincingly “de-aged” formance, and place it into the shot.” younger clone. OF @GEMINIMANMOVIE/TWITTER COURTESY IMAGE 12 COMMUNICATIONS OF THE ACM | AUGUST 2020 | VOL. 63 | NO. 8 news Adcock likens this process to creating a musical score derived from a ACM performance on one instrument. This Instead of using representation allows the same musi- animators to build a Member cal piece to be performed on another instrument, which imparts its own puppet with DeNiro’s characteristic sound. facial motions, the News To construct the “score,” the team HUMAN-COMPUTER built on its previous expertise animating FLUX software PARTNERSHIPS fantastic characters like Gollum in The morphed captured Wendy Mackay Hobbit, equipping Smith with facial is a research performance and director at Inria markers and a head-mounted cam- Paris-Saclay, the era rig that recorded his expressions. lighting data to create French national These rigs have been getting sleeker research the new image. institute for the and lighter all the time, Adcock said. digital sciences, at the Such “performance capture,” as con- Université Paris-Saclay in Paris, trasted with “motion capture” of bodily France. She heads up the ExSitu movements, provides extremely high- research group, a lab of 30 people that explores the limits quality facial data for the animators. pipeline to construct the de-aged im- of human-computer interaction. ages. In addition to FACS capture from “I have always been Supporting the Storytellers the current actors, the team acquired interested in how people For renowned director Martin Scorsese, huge numbers of images from their pre- interact with technology,” Mackay says, and human- however, headgear and facial markers vious performances over the decades, al- computer partnerships have (dots on the actors faces) were still un- lowing them to select a facial model for become a focus for her. People acceptably disruptive to the human in- each of the years in the film. Rather than react to technology, she points out, but they also appropriate it teractions he sought in the 2019 epic reproduce the exact look of De Niro from and do innovative things with it; The Irishman. The film features Robert Taxi Driver (1976) or Goodfellas (1990), as a result, technology should De Niro, Al Pacino, and Joe Pesci, all however, they strove to create a de-aged be designed to support user- now in their late 70s, in roles spanning version of his Irishman character. More- innovation and enhance users’ capabilities, she says. many ages across several decades, so a over, instead of using animators to build Mackay received a bachelor’s strategy for de-aging was critical. This a puppet with De Niro’s facial motions, degree in experimental resulted in the film being nominated their FLUX software morphed the cap- psychology from the University of California, San Diego; an for Best Special Effects and nine other tured performance and lighting data to M.S. in experimental psychology Academy Awards. create the new image. from Northeastern University, Beginning several years earlier, The result was a compelling de- and a Ph.D. in Management in Pablo Helman and his team from In- aged character that let the actors act— Technological Innovation from the Massachusetts Institute dustrial Light and Magic (ILM), the and interact. of Technology. special effects house founded by “If you’re going to get the top actors She started her professional George Lucas, worked with Scorsese of our time, you need to support their career at Digital Equipment to develop ways to create the desired acting process and not create a tech- Corporation (DEC), worked there in different capacities images much less intrusively. To cap- nology or impose a technology that over a period of 11 years. She ture the performances, they built a would somehow diminish that per- was working on something new bulky apparatus in which a normal formance. That’s what they were able at the time: human-computer interaction. “It was a new field, camera was flanked by two infrared to achieve,” said Barron, who said the combining psychology and cameras and associated infrared less-intrusive technology “allows more how we think about people and lighting that provided synchronized, adoption among a wider group of peo- technology, and putting them high-quality stereoscopic information ple into different kind of films.” together, with an emphasis on innovation,” she explains. needing neither visible facial markers After DEC, Mackay nor changes in the lighting desired by Under the Hood joined academia, teaching at the filmmakers. Unlike the latest superhero movie, a Denmark’s University of Aarhus, France’s University of Paris-Sud, The size and weight of this equipment film like The Irishman succeeds when and even Stanford University as posed some challenges; for example, the audience forgets about its tech- a visiting professor. precluding the popular hand-held Stea- nical sophistication. Nonetheless, In the future, Mackay dicam shots. Moreover, although the achieving this realism requires an plans to explore the physics of interactions with digital infrared light did not interfere with the enormous, diverse team innovating in material. “We will use this to main camera, the infrared cameras were both hardware and software (and a rethink how we use intelligent sensitive to cigarette smoke, and could reported $160-million budget). systems—machine learning and not see through vintage car windshields. “There are people that are creative, other types of AI—to enhance users’ capabilities, to move In parallel with the hardware, the they’re thinking about design and light- people forward and do more.” ILM team developed a new software ing and composition and performance —John Delaney AUGUST 2020 | VOL. 63 | NO. 8 | COMMUNICATIONS OF THE ACM 13 news of storytelling, and there’s the people scheduled for release late this year. under the hood that have to make that all The filmmakers obtained legal per- work. It’s really a collaboration between “The power of AI missions to use the actors’ likenesses the technologist and the artist,” said Bar- being applied to in these films.

Load more