slightly off. Mori's theory gradually made ONWARD AND UPWARD WITH THE AR.TS its way beyond the robotics world and into digital-animation circles. Pixellated hu mans could fall into the Uncanny Valley just as easily as robots cottld. Stylized car toons the ''Incredibles'' family, the The scientist behi11d the digital cloning of acto1'"s. blocky old man in ''Up'' ofte11 fill us with empathy, but a pset1do-ht1man like the BY MAR.GAR.ET TALBOT trai11 conductor voiced by Tom Hanl
32 THE NEW YOR.K.ER., APR.IL 28, 2014 had been achieved. ''You do get to feel the ing my soul.''' He laughed, a little ner the problem often lies in the upper face: skin as this dynamic, scrunchy surface," vously. ''fm telling you, there's going to be ''the eyes, the eyelids, the forehead, which he said. Human skin was devilishly hard all kinds of psychological literature once we rely on, subconsciously and con to simulate: ''It's this crazy landscape of this kind of thing becomes common." sciously, for social communication." She fuzz and squish." added, ''If we don't see what we expect O'Brien was being scanned again that bout · years ago, the first com- to see if a frightened character in a #-- day for an t1pdated version of Digital puter-generated elements started video game doesn't have the wide eyes, Emily, so we walked down to the room tt1rning up in Hollywood films; in deep wrinkles in the forehead, arched that houses one of the ''light stages'' that ''Young Sherlock Holmes," for exam eyebrows, and vertical wrinkle be Debevec and his team have designed: a ple, fragments of stai11ed glass inside a tween tl1e eyes which signal fear to most latticework geodesic dome, eigl1t feet cl1t1rcl1 magically unite to for1n a11 ar ofus we're disoriented." Ii1 certain con in diameter, equipped with more than mored knight. By tl1e i1ineteen-nineties, texts horror-ge11fe video games, for ex twelve tl1ousand L.E.D.s and seven Debevec told me, special-effects techni ample generating that queasy feeling Canon high-definition sports-photogra cians could rnake ''realistic morphing can be usefltl. But usually the effect un phy cameras. A mt1ch larger stage is used metal men, dinosaurs, spaceships, and dermines efforts to achieve realism. for capturi11g a fUll body. U.S.C. owns the steamboats." But ''there was something Tinwell recently completed a study patents on the technology and charges about the human face that seemed t1nat in wl1ich she asked subjects to react to studios sixteen thousand dollars to cap tainable ." Not only does the face l1ave the 2009 version of Digital Emily, and ture twenty expressions on an actor's face. complex geometries; evolution l1as she found that even it ''fell short of true, Emily sat on a chair inside the dome, primed us to be attuned to the tiniest di authentic human believability." People and for about an hour she made an array vergences from reality. weren't upset by Digital Emily, exactly, offacial expressions: snarling, happy, dis Even ifyou get the surface appearance but they sensed a faint strangeness. (She gusted, sleepy. She recited some Shake ofa digital face right, the movements can hasn't tested Ira yet, but she would like speare and sang ''I'll Be Seeing You'' in a do you in. As efforts to build android ro to do so.) Indeed, Tinwell and her col warm alto. The data obtained from these bots have demonstrated, manufactured leagt1es Mark Grimshaw and Andrew scans will be used to make a fresh geo blinks often seem owlish, and artificial Williams have proposed the term ''Un metric map of her facial structure and a mouths can appear to snap open too wide canny Wall'' to describe an emerging texture map of its surfaces. The I.C.T. or too precisely. (Human lips stick to phenomenon: ever greater digital realism will then give those maps to a video-game gether for a fraction of a second at the is stuck in a race against ever greater or computer-graphics company Activ corners after we open them to speak.) viewer expectations. ''Our discernment of ision and NVIDIA each did versions of Mari's original paper on the Uncanny subtle deviations from the norm will keep Ira which will use its own facial-anima Valley was more of a thought experiment pace with technology," Tinwell predicts. tion software to render a digital puppet of than a formal study: he didn't test his Masahiro Mori, meanwhile, said in a re Emily. This is painstaking, computation hypothesis about virtual humans on any cent interview, ''I have no motivation to • intensive work: NVIDIA's animated ver actual humans. But the argument has been build a robot that resides on the other side sion of Digital Ira reql1ires two tera:flops borne out in several formal studies since of the Valley." Robots can be useful, he of processing power. then. It's not yet clear why not-quite said, but they ''should be different from At one point, the inspiration for humans provoke so much anxiety. May human beings.'' Digital Ira, a forty-three-year-old man be characters that fall into the Uncanny In 1988, animators at Pixar made a named Ari Shapiro, poked his head in. Valley remind us of corpses. Perhaps short movie called ''Tin Toy," in which a He and O'Brien shook hands, a bit awk they flood us with uneasy intimations of baby menaces, and then charms, an ani wardly. Shapiro is a research scientist at soullessness. Maybe they confound the mate toy. It was a successful test-run for the I.C.T., where he works on synchro brain's ability to distingu.ish what is alive ''Toy Story," except for one thing: the in nizing speech and body movements in and human from what is neither. Thalia fant was inadvertently unnerving. Since virtual humans. Although Shapiro has Wheatley, a neuroscientist at Dartmouth then, Pixar has steered away from highly short brown hair, his C.G. counterpart is who has studied how we recognize ani realistic human characters. (The video is bald. It takes tremendous computing macy in others, told me, ''Think of horror on YouTube, and many viewers have un power to render hair in real time, and be movies zombies, vampires, or even derscored their horror in all caps: ''HOLY cause, as Debevec put it, ''there do exist clowns, because they have faces painted on FUCKING SHITTHATTHING IS FREAKY.") valid humans with no hair,"· he had de that don't move. It looks like a person's face As Karl MacDorman, a roboticist who cided to make it easier on the prototype but it doesn't move like one. A conflict studies the Uncanny Valley, told me, team. arises in the brain, which is unsettling." ''All of the toys in that movie are won Shapiro told me that he was glad that Angela Tinwell, a technology re derfully animated. The baby was hor the lab hadn't called his avatar Digital Ari. searcher at the University of Bolton, in rible. Its drool was like Silly Putty. It He needed to maintain a little distance: England, studies the Uncanny Valley, inspired no empathy." Pixar's feature ''Paul's team would walk by my office and and she thinks that our discomfort stems films with their cute robots, cars, ani say, 'Oh, you should see what they're from our reliance on nonverbal commu mals, fluffy monsters, and humans ren doing to Digital Ari.' And at some point it nication to read the intentions of others. dered with goggle-eyed stylization have became, like, 'O.K., they're kind of steal- In an imperfect humanoid, she told me, nearly all been artistic and box-office
THE NEW YORKER, APR.IL 28, 2014 33
------~ successes. Audiences have similarly em could brew her up to look exactly like you geekery in a big-budget movie was thrill braced the blue Na'vi of''Avatar'' and the want her to look voice synthesized, ing, if not that profitable the lab is an Gollum character in the Tolkien block with a facial expression that you could academic institution, after all. He told me, busters, which felt emotionally human yet modify and tweak." ''A lot of what drives me still is just the . didn't pretend to be human. But movies immense cool factor ofbeing able to make that have attempted to vault over the Un hen Debevec was a kid, he was so something synthetic that looks real." canny Valley with photorealistic humans obsessed with prograinming that Another Debevec breakthrough in based on motion-capture technology he was staii:led to learn tl1at some people volved digital lighting. He realized that if rarely charm audiences; the recent "Mars used computers just to do word process you wanted to insert digital objects into Needs Moms'' was a disastrous flop, los ing or to play games. He grew up in Illi- an otherwise naturalistic scene, you had ing more tl1an a l1t111dred rnillion dollars. 11ois, the only child of a physics professor to light them as tl1ougl1 they 11ad been Debevec told me, ''Sometimes when you and a psychiatric social worker, and was a there in the first place. A s he put it, ''A try something too early, it becomes sort of fa11 of special-effects-filled movies lilze digital car will look a lot 1nore persuasive a warning sign to everybody else you ''Star Wars'' and ''Baclz to the Ft1ture''; 11e if the richness of the 1·eflection and the know, 'Here be dragons. Don't go off the often stayed up late making stop-motion shadow on it are telling us that it's really edge of tl1e eai~ jt1st yet.''' animation films witl1 an 8-1nm. camera. there." In 1998, l1e developed a technique But, thanks in no sn1all part to De As an undergraduate at the University of for extracting light measurements from bevec, we are 11ow l1overing close to that Michigan, Debevec amused himself by real-world settings and applying them to edge. His scanning tech11ology made it maki11g a 3-D rendering of his car, a computer-generated objects and archi possible for Brad Pitt to age persuasively crummy silver Chevette, and simulating it tecture. His short film ''Fiat Lux'' sl1owed in ''The Curious Case of Benjamin But in flight. He went to graduate school a digital model of St. Peter's Basilica illu ton," and has allowed spookily realistic at Berkeley, where he made a short minated by digital light that mimicked stunt doubles to appear in scenes that are computer-generated film, ''The Campa photographs taken in Rome. In 1999, too dangerous for human beings. De nile Movie," that looked as ifit had been Debevec debuted tl1e film at SIGGRAPH, bevec thinks that it won't be long before shot from a plane flying around the cam the annual conference for the special digital actors begin replacing human ac pus's iconic tower. Debevec's method was effects industry. A Ti111es reporter took tors for entire performances. He said, used in ''The Matrix'' (1999) to fill in note of the riotous applause that greeted ''Ten years from now, if you want to backgrounds during the famed ''bullet a sequence in which giant ball bearings make a commercial for Progressive insur time'' sequences, in which the action appeared to fall from the ceiling ofthe ba ance with a spunky young woman, you slows down for one character. Seeing his silica, reflecting ''the whole scene with spherical distortion as they bounced down the length of the cathedral." One person in the audience was Mark Sagar, a visual-effects designer who was showing his own short film, ''The Jester," at the conference. The film offered an early example of a realistic computer generated face, but Sagar thought that the lighting made the skin look too waxy. Over the next several months, he met - with Debevec, and, as Sagar recalled, ''I talked to him about all the problems with creating realistic skin. Then he basically went back and created the light stage." Capturing hard data about how skin reflected light, Debevec realized, would lead to more believable virtual humans. 7 To re-create a face digitally, you needed to know all about that face's reflectance properties: not just how light bounced off the surface of the skin, which is called specular shine, but also what it did after penetrating skin, which is called subsur face scattering. The translucency of skin is part of what make us look alive, not plastic. Debevec's light stages also helped him capture the darting glint of eyeballs ''If we become friends now, in twenty years we and the bone structure revealed by differ can say we've been fil"iends fa11" that long.'' ent angles oflight. ' The first light stage, which was Advertisement finished in 1999, was humble, employing a two-hundred-and-fifty-watt spotlight pulled around in a spiral by a rope. Nev ertheless, in the course of a minute it pro vided a data set of a face being lit from every possible direction tl1at light can come from. Debevec has been t1pgrading the design ever si11ce so far, the1·e have been ten models of the light stage...... • • • • • In 2002, Mark Sagar tool< a job at • .. . . Sony Pictures Imageworks, where he began working on ''Spider-Man 2." He and a colleague, Scott Stokdyk, decided to use Debevec's light stage to create a digital stl1nt double for Alfred Molina, who was playing the villain Doc Ock. Stolzdyk told me, 'With actual stunt dou bles, you have to hide the fact that the re sembla11ce may not be that great the doubles must turn their heads away from the camera, or whatever." After Debevec scanned Molina, Sony animators created a digital replica ofMolina's face so lifelike ''Good nrws I hear the paradigm is shifting.'' that it could be rendered i11 closeup as Doc Oclz sinks to the bottom of the East River. The one complication was that, before shooting began, Molina was told 1 to shave his impressive sideburns. Stok ' 1 dyk, who loved that Debevec's avatars ' were based entirely on ''hard data," had to 1'....., fill in the skin beneath the missing side \ burns, which ''defeated the purity of the Share your insights on a variety of topics with The New Yorker's process." He explained, ''I had to make all these little artistic decisions: how marketing department as a member of our online OpinioR ' many days' growth would he have, what Leaders panel. As one of our most influential readers, you'll gain .. would the stubble look like, the blood VIP access to exclusive New Yorke~ events and spe<::ial offers .. flow under the skin there. That might have been the most expensive haircut in history." ' l Hollywood judged the ''Spider-Man ' 2'' experiment a success, and Debevec and the I.C.T. began scanning many actors. For ''Avatar," Zqe Saldana, Sam Worthington, and others were scanned in the light stage, and the data helped an imators create N a'vi aliens whose facial characteristics echoed those ofthe actors. But the filmmakers also created a digital version of Jake Sully Worthington's character in his human form, allowing them to integrate Jake into scenes set in the fully digital realm of Pandora. To ward the end of the movie, Neytiri, the N a'vi played by Saldana, leans over to kiss the human Jake. Last fall, in a talk at the Smithsonian, Debevec said, ''There's a lot oflighting interaction between the faces. She casts a shadow on him. He's bouncing
THE NEW YORKER., APR.IL 28, 2014 35
-- .... ·.·.:..
some pink light onto her; she's bouncing a frame and saying, 'That's me.' Theyre virtual character." Peach fuzz would be some blue light onto him. He has various going to be saying, more and more, 'I particularly helpful, they agreed, with bioluminescent materials creeping up his don't remember doing that.' Well, that's children's faces, since there's so little tex neck and his ears that are lighting him because they didn't." ture to work with otherwise. from other directions. And if he's a digi ''Kids are a challenge," von der Pahlen tal asset, then you can just simulate this all ebevec is a tall man with dark wavy said. ''No wrinkles or enlarged pores." on the computer, tvveak it until it looks hair; he periodically shaves off and Debevec was giving me a tour of the just right, and get a shot that really looks regrows a goatee, as if to make it impos I.C.T., which is housed in a sleek new like that digital perso11 is part of this real sible for a digital double to supplant him. office park in Playa del Rey, on the world environment." He works all the time he likes to call grounds of the former Howard Hughes Now that Worthington's face as it himself a ''naked-mole-rat scientist'' airplane factory. The lab's current schol looked in his early thirties has been and does a lot of his socializi11g with col arly projects have names like Virtual banked in a computer, this ''digital asset'' leagues. But 11ow he is engaged to a Human Embodiment and the Co1npu could be used in a film decades from now, U.S.C. graduate student in animation tational Emotion Group. Mucl1 of this when Worthington loolThe Social Network," the eyeball is farther than anything I've ever in one ofthe conference rooms. The hall Winklevoss twins were both played by seen. His beak is significant. He just has way decor alternates between movie post Armie Hammer; in scenes featuring the this really anatomically notable face." ers and whiteboards covered in impene twins, Hammer performed alongside an We'd been standing around with his trable thickets of math. actor whose face was later replaced with a colleague Oleg Alexander, a computer In one corner of the office, Debevec digital version of his own. Fincher, who artist, and Javier von der Pahlen, the head had set up a radical demonstration for began his career working in visual effects, of research and development at Activi von der Pahlen and me. The light stage at Industrial Light & Magic, also made sion, which wants to make its video had first helped Debevec insert digital ''Benjamin Button," and he seems espe games more realistic. They both nodded humans into films. Now he wanted to in cially comfortable with the concept of at the mention ofCruise' s beak. Debevec sert them into the real world, in the man digital actors. In 2007, he directed a pop continued, ''In a movie star, you want one ner of hologr£lil?.s. corn commercial featuring a resurrected standard deviation away from regularity. Arranged in a semicircle around a Orville Redenbacher. More You want that notability to shimmery screen was a constellation of recently, when Fincher was create a space in your brain. tiny projectors. Collectively, they could filming ''The Girl with the It's like the hook of a song." project tvvo hundred and sixteen different Dragon Tattoo," in Sweden, ''Humphrey Bogart had vie'1vs of a person, based on footage taken itwas deemed too dangerous the most asymmetrical face," on a light stage equipped with thirty for Rooney Mara's stunt Alexander said. video cameras. At the moment, Debevec double to ride a motorcycle ''And Bradley Cooper!'' and his team were using the technique to on an icy, dark road without Debevec said. ''Dude, look create a projected replica of Morgan a helmet. The stunt double at him! One eye is half an Spurlock, the host ofthe CNN series ''In wore a helmet, but Fincher inch,, lower than the other side Man," who was planning to inter later had the footage tvveaked, one. vievv his double for an episode on efforts superimposing a digital version ofMara's 'Well, you did look pretty closely at to achieve immortality. face. In the shots, the fake Mara, who is his data," Alexander said. (Cooper was They were also working on a much chasing the movie's villain, looks f ero scanned for a film that was later shelved.) richer and stranger holographic project: ciously determined. Steve Preeg, an ani Debevec shook his head. ''I tried to for the U.S.C. Shoah Foundation, they mation supervisor at Digital Domain, a watch 'Silver Linings Playbook,' but I were creating holographic versions of visual-effects company, worked on both couldn't not see it." Holocaust survivors. They had made a ''Dragon Tattoo'' and ''Benjamin Button." Tl1ey talked for a bit about the impor prototype based on twenty hours ofinter He told me that digital effects were going tance of including peach fuzz on digital vie\vs with an eighty-one-year-old con to make it harder to judge a performance: faces. '''Tintin' had it," Alexander noted. centration-camp survivor named Pinchas ''Actors are going to have trouble :freezing ''That was the first time I ever saw it on a Gutter. In March, they filmed him on a
36 THE NEW YORKER, APRIL 28, 2014
,. .. !L.. light stage with fifty high-definition cam eras. Soon, Debevec said, the digital ver ,:,···· . .... :. .. ;.. .
':·:;.: : .,.. :·:.:,. ·~: ... ·'.'::•:;·:;·.~:~·::;::·::;;:~.;,: .';.~~·:·:;.: : y;·.;·\ ...... :'::.:;::·~·~:· ··.:~. ~::.:.:·:::: . ;;~::·:·:·:·:·:: , sion of Gutter could be projected into a ...... , ...... '
classroom and questioned by students. f::::.•.:. ·.:.:;· !_:::;: . .· ..· ... The holographic replica could draw on B the archive ofinterviews to answer almost any question relevant to Gutter's personal .. ·.::·.. :.:. ·.:.>::.: .... history; it could also deflect off-the-wall queries. As Debevec and his colleagues
.. ~ envisaged it, Gutter would be answering ...· questions about the Holocaust long after [~~:E~~-:~v?:~KJ [L?~~o:B I [PA~;§ l [~~!J0~J cf;K~~] he was dead. The technique improves on one that Digital Domain had used to make a vir tual version ofTupac Shakur perform at the 2012 Coachella festival. (He was killed in 1996.) But the digital Shakur had appeared on a darkened stage, at a distance. The Shoah Foundation and the i---.--- -.,.-.. - -1 - I.C.T. want the holographic doubles of Gutter and the other Holocaust survivors ...... ·...... to be approachable, conversable, and viewable up close. Debevec' s light-stage • • technology will help the doubles look ·'.:· j~ natural: wherever they are projected than books, or even film, will. Maybe an children? I don't think so. It could con f.~ classroom, synagogue, park they will alog artifacts and testimony will seem ceivably say ''I love you'' to my hypotliet appear to be illuminated by the light of insufficiently responsive and theref ore ical grandchildren, but rd know nothing ~J' that environment. less real to them. Yet creating a virtual about them. Even Debevec wasn't sure The project seemed to me to be tech Holocaust survivor also risks diluting a that he'd want something like that. When I ~: ~~~ nically wondrous and philosophically fundamental truth, which is that death is I asked him if he wished he had holo m~ i~ weird. Debevec and his colleagues talked perrr1anent, and that those deaths did ir grams of dead relatives, he recalled how_ &,:?, ~!f ;:.~ (;;;>; ''•'>'if~ about ''future-proofing'' Holocaust testi reparable damage to humanity. There is taciturn his grandfathers had been, and··\ ,.,,~:~ ~::~ ~~ "'J. ~:.~ mony and other historical evidence. They also an uncomfortable echo ofDisney thought maybe not. ''A correct simula- \_ ~!:- ~~~ ~.~~''· ~';; :<~ in ti.on ofthem would be that they'd lose in .~>: wanted to make documentaiy artifacts of treating histoiy as entertainment. Ste ::;[: terest and kind ofwalk off,'' he said. ~·~J·•'.i forms that subsequent generations might phen Smith, the Holocaust scholar who .~:~ ~.~ ~ come to expect, and design them so that directs the fotJ.Ildation, told me that hed There's been talk of creating a holo :~~ they could be upgraded to work with new proceeded ''very carefully." Museum pro graphic Marilyn Monroe or Michael technology. Oil paintings used to offer f essi_onals had sometimes expressed Jackson or Elvis Presley, each ofwhose es windows into the past; then it was pho doubts to him, he said, but the survivors tates takes in millions of dollars a year. [! tography, then film and now wed have he consulted had ''no barriers to using this Steve Preeg, at Digital Domain, told me :;' ·~ a 3-D person talking to us. David Traum, technology." He went on, ''These are that he'd been approached to work on a f•..! the head ofthe natural-language research people who can remember having a sin singing Marilyn; he declined, because that f,i ·.. :· .: project, unlike the Tupac one, did not . ' group at the I.C.T., has worked closely gle telephone in their houses, and who ..~ ·; ·. ·. ~. ~ have the blessing ofMonroes family. But : : on the survivors project, and he told me now use smartphones to send pictures to : . .. } j : : that holographic preservations of the their grandchildren. They understand resurrection is already an established spe . ' dead are ''going to be a pervasive part of that technology isn't stopping because cial effect in Hollywood. When Philip .. the future." He went on, ''It's a way for they're at the end of their natural lives. Seymour Hoffman died, earlier this year, relatives and others to do more than just They want to move forward with what's he had not completed shooting the final watch a video or listen to audio after peo going to be right for the next generation." ''Hunger Games'' films. Hoffman was ple are gone. You could actually ask ques The ethics of digital resurrection be never scanned on a light stage, but there ' ' tions, hear their stories again. You could comes particularly fraught in a project in are other ways to digitally fill in the gaps. ' ' have a conversation, or something like it, volving Holocaust survivors. But I wasn't Debevec explained, ''The 3-D shapes of : ·; with yolrr departed relatives. It won't be all that comfortable with some of the many actors in every expression they . ·,
could possibly make are contained in .. exactly the person, with new inputs and other scenarios that Debevec has imag ·.·:
i things to say, but you'll have that pattern ined becoming popular. Would I want a the frames of the movies themselves. :: of questions and responses. Or even that hologram of my beloved, long-dead par Theres a scene at the end of Bruce Lee's little thing of'I love you,' 'I love you, too."' ents? They would be eternally available 'Enter the Dragon' where Lee walks ...... ;. ..;. ·. : ~: Perhaps interactive holograms will but fundamentally out ofreach. Would I through a hall of mirrors, and there are ....:. .. .. make more sense to people of the future want to leave one ofmyselfbehind for my frames where you see his face from several '
THE NEW YORKER., APR.IL 28, 2014 37
··-·····-·····--· -····------········- viewpoints simultaneously. By comparing it comes to actors. And really indie partly animated science-fiction movie the points in the different views, you can filmmakers, using virtual sets, would be starring Robin Wright and directed by triangulate his facial geometry. There's a able to make what only huge Hollywood the Israeli filmmaker Ari F olman, is nice wardrobe test reel out there of a producers, flying actors to the Riviera or scheduled to be released. A key scene was young Warren Beatty, showing him from wherever, can do now." filmed on an I.C.T. light stage, but its vi all angles, and keeping pretty consistent Debevec's enthusiasm is generally win sion of Debevec's technology is a dark facial expressions and constant lighting. ning, but when he talked this way I grew one. Wright plays an aging actress who In
38 THE NEW YORKER., APR.IL 28, 2014
------·------