Digital Heroes and Computer-Generated Worlds

Home , Andy Serkis, The Abyss, Weta Digital

COMMENT BOOKS & ARTS

The computer-generated Gollum (left, seen in 2012’s The Hobbit: An Unexpected Journey) is based on performances by actor Andy Serkis.

COMPUTER ANIMATION Digital heroes and computer-generated worlds Joe Letteri, Oscar-winning co-creator of Gollum on screen, looks at the evolution of computer animation as the next instalment of The Hobbit trilogy reaches cinemas.

rom the beginning, computer as we do today, animators used a variety of was out of the question. Instead, we stud- animation has had the ability to engage reference techniques to capture the essence ied elephants to understand weight, and viewers by giving artists a way to mix of organic movement. Snow White’s dance lizards, other reptiles and birds to get some Freality with fantasy. Early film examples with the dwarves in Disney’s 1937 film was ideas about how dinosaurs of different sizes such as the stained-glass knight in Barry created by matching the movements of a live, might have moved. Digital animators did Levinson’s 1985 Young Sherlock Holmes and filmed dancer, using a technique called roto- motion studies, copying the movement of the ‘water tentacle’ in James Cameron’s 1989 scoping — in the most basic terms, tracing these animals frame by frame until they The Abyss offered a glimpse of the potential the motion from a film one frame at a time. could synthesize a convincing idea of dino- of this new art form to create memorable This technique, although now slightly more saur movement. characters. That became immediately appar- sophisticated in its application, is still in use. Two years later, computer animation took ent in Cameron’s later Terminator 2 (1991). Take dinosaurs, for example. I was for- another big step forward with the astound- He took an already great idea for a character tunate to begin my career at the US visual- ing success of Pixar’s Toy Story. Software was (a terminator robot) and turned it in a new effects company Industrial Light & Magic, becoming sophisticated enough to tackle and unexpected direction: the shape-shifting then in Marin County, California, as it was the creation of a character’s performance. In T-1000 liquid-metal terminator. gearing up to create the computer-generated traditional animation, a lead animator sets Computer animation is a natural exten- dinosaurs for Steven Spielberg’s 1993 Jurassic key frames or poses for a character and jun- sion of hand-drawn methods developed Park. Naturally, we all ior animators draw the ‘in-betweens’ from during the early-twentieth-century golden NATURE.COM thought that roto- pose to pose. Now, artists could use the age of ‘cel’ animation ushered in by Walt For a video on the scoping dinosaurs computer to do that. Pixar proved that three- Disney, in which a series of images is played making of Gollum: would be a great idea, dimensional computer animation could be back at speed to give the illusion of life. Just go.nature.com/7qtgxr but unfortunately that used to create an entire film.

In 2001, I joined Wellington-based matched the motion’s underlying dynamics. underneath. Then we built Kong so that he visual-effects company Weta Digital to work Andy Serkis, who performed Gollum, had the same facial-muscle layout as Andy, with Peter Jackson on The Lord of the Rings wore a special suit with reflective markers and used Andy’s muscle movements to drive trilogy, inspired in large part by the chance to show the key positions of his joints. From Kong’s facial performance. to create the character Gollum. Gollum was the multiple cameras, we could calculate This breakthrough meant that we could

NEW LINE PRODUCTIONS a special challenge, because the more realis- his skeleton’s position at every frame as he now capture an actor’s performance in its tically human a character is, the more com- performed. Those positions were then trans- entirety. This became important for Weta’s plex the animation gets. People are attuned ferred to Gollum’s digital skeleton, which next film, James Cameron’s Avatar (2009), to recognizing all aspects of human motion allowed us to make the character move the for which we made one important modifica- and behaviour, no matter how subtle. And way Andy did. Traditional key-frame ani- tion. Each actor wore a helmet that filmed because the characters we create are three- mation techniques still apply, however. For their facial movements; we then extracted dimensional, we have to understand how to example, the first time we see Gollum in The the performance data from each frame and pose them frame by frame to achieve realis- Lord of the Rings: The Two Towers (2002), he used a ‘facial action coding system’ solver to tic performances. is climbing down a vertical rock face, some- translate the movements into muscle activa- With Gollum, we used the then relatively thing no human can do. So that motion is tions. The knowledge of which muscles are new technique of performance capture — based on animators observing what a human activated in a facial expression informed our effectively, an extension of rotoscoping. But can do and using their imaginations to create activation of the corresponding muscles in instead of looking at an actor from a single a believable performance. And, in a direct the digital characters. In addition, this pro- point of view and matching the form of throwback to rotoscoping, we created his cess allowed the director to see the actors the motion, we looked at his performance facial performance and dialogue-related live through a virtual camera as they were

TWENTIETH CENTURY FOX/THE KOBAL COLLECTION KOBAL FOX/THE CENTURY TWENTIETH using dozens of cameras simultaneously and movements by hand, frame by frame, from instantly transformed into their Na’vi char- Andy’s filmed performance. acters moving through the world of Pandora. The problem with trying to capture facial Digital characters also have to appear real- motion is that there are no joints, apart istic in their surroundings, whether that is a from the jaw, that have movements you photographed environment or a complete can track. So for Jackson’s 2005 King Kong, digital creation such as the jungles of Pan- Weta came up with a different technique. dora. So we looked to understand how light Again with Serkis, we glued small reflective and materials interact in nature. One of the markers all over his face. By using these to best examples of this interaction is subsurface track the changes in skin position and scattering. We first developed a technique to tension as Andy performed, we could replicate this mechanism of light transport compute what his muscles were doing to create the translucency of Gollum’s skin, leveraging pioneering research by computer- The Na’vi in Avatar (2009) lived in an graphics specialist Henrik Wann Jensen and entirely digitally created world. his colleagues at Stanford University in Cali- fornia (see go.nature.com/lyzuh2). The

thick skin of a dinosaur can be simu- lated by bouncing light off the exterior. But human skin is softer and more trans- lucent, so light enters and bounces around dozens of times before exiting. These properties, which are easily observed by putting your hand in front of a bright light, are crucial to a realistic portrayal. MUSEUM CARISBROOKE CASTLE Realistic animation also depends on knowledge of how skin, muscles and hair move independently of a character’s performance. “Realistic These second- animation ary motions are depends on achieved through knowledge intensive simula- of how skin, tions that compute muscles and all of the mass, hair move dynamics, ten- independently sions and interac- of a character’s tion of each part performance.” of the body as a character moves. The simulations help to create the com- plex visual cues that the human brain processes when taking in an image. They also ensure that the physiology of crea- tures (real or fantastical) has a ground truth and is believable. Combining this new level of detail with motion-captured John Milne (left) with his wife Tone, seismologist Boris Galitzin and a lamp-post seismometer in 1910. performances of talented actors has ena- bled computer-animated characters to GEOLOGY become lead actors. Rupert Wyatt’s 2011 Rise of the Planet of the Apes featured a chimpanzee named Caesar who was not just the protagonist; The maverick founder of he was the emotional centre of the film. This entirely digital character is a great example of how the advances in animation modern seismology work together, from muscle simulation, fur and realistic lighting, to motion-cap- George Helffrich relishes a film on John Milne, whose tured body and facial performance. work in Japan put earthquake science on the map. The following year saw all of these developments come full circle when we were able to once again present Gollum in hat European other than Marco of earthquakes — The Man Who Peter Jackson’s The Hobbit: An Unexpected Polo, on taking up a job in the instrumentally. His Mapped the Journey. This new Gollum benefited from Far East, would travel overland? goal became to build a Shaking Earth DIRECTOR: WILLIAM a much more detailed digital model, new WJohn Milne. No posh Brit he: the Victorian standard seismometer TWYCROSS subsurface scattering techniques and all of geologist reached Japan mainly by train, and establish a seismic 2013. the advances we have made in the past ten foot and pack animal, ostensibly to avoid network across Japan. years. If you watch closely, you will see the seasickness. As I write this, flying to Japan That plan was the inception of modern muscles moving under his skin and the across eastern Siberia, I am amazed. I detect global seismology. light refracting in his eyes. And you will a whiff of the iconoclast. To mark this year’s centenary of Milne’s get a glimpse of the worlds we can create Milne’s motivation for that epic journey death, his great-nephew William Twycross from the mix of all this art and science. ■ was an invitation from Japanese officials of made a documentary, The Man Who the Meiji era (1868–1912) to establish an Mapped the Shaking Earth. Shown in July Joe Letteri is senior visual effects Imperial College of Engineering in Tokyo, at the assembly of the International Union supervisor at Weta Digital in Wellington, and to transfer Western knowledge to Japan. of Geodesy and Geophysics in Gothenburg, New Zealand. He has received four After arriving in 1875 to teach geology and Sweden, the film immerses us in Milne’s Academy Awards for Visual Effects and the mining, he became aware of the frequency life in science, including his photographs Academy’s Technical Achievement Award of earthquakes and their damaging effects. and sketches. Often shooting on location, for co-developing the subsurface-scattering Building designs transplanted from the West Twycross and his film crew trace Milne’s technique that brought Gollum to life. His performed particularly poorly. By 1878, travels in Iceland, the Canadian island of latest film is The Hobbit: The Desolation Milne determined that to study earthquake Newfoundland, Britain, Russia and the of Smaug, on general release this week. damage properly, he needed to quantify United States, as well as to and in Japan. e-mail: [email protected] seismicity — or the frequency and strength Twycross imposes a fine narrative continuity