[in the SPOTLIGHT] Chris Bregler Motion Capture Technology for Entertainment he year 2007 started with this momentous goal. One of the considered “cheating” in the anima- two big bangs for Lucas- secret ingredients that will get us tion world, where it was thought that Film’s Industrial Light and there may be motion capture (MoCap), everything should be conjured up by Magic (ILM), the motion defined as a technology that allows us the imagination. “Good animation” picture visual effects compa- to record human motion with sensors was considered to be a rare and hard- Tny started by George Lucas in 1975, in and to digitally map the motion to to-acquire art form that followed the two eagerly awaited Hollywood events. In computer-generated creatures. MoCap famous drawing and animation princi- January, ILM won the “Scientific and has been both hyped and criticized ples of cartoon physics invented by Engineering Award” of the Academy of since people started experimenting Disney. Rotoscoping was thought to be Motion Picture Arts and Sciences for with computerized motion-recording “cheap animation” that lacked expres- their image-based modeling work applied devices in the 1970s. Whether reviled siveness. If one of animation’s main to visual effects in the film industry. In by animation purists as a shortcut or principles is exaggeration—everything February, ILM took home the Oscar needs to be bigger than life, not for “Best Visual Effects” for Pirates just a copy of life—rotoscoping, as of the Caribbean: Dead Man’s Chest, THE ILM AWARDS ARE precursor of MoCap, was consid- a milestone achievement for ILM IN LINE WITH ONE SPECIFIC ered even less than life because after their last Oscar for Forrest GOAL—THE CREATION OF many important subtleties got lost Gump in 1994 and many previous ARTIFICIAL HUMANS in the process. Oscars in their 32-year history of AND OTHER CREATURES As in the 1920s, in the late pioneering computer-based visual THAT ARE SO REALISTIC 1980s and early 1990s when early effects. In this column, we comment ONE CAN SAY THAT THEY MoCap (or semiautomatic roto- on the significance of these awards PASSED THE TURING TEST scoping) effects appeared in feature from a technical standpoint and the OF COMPUTER GRAPHICS. films like Total Recall, the century- goal, challenges, and status of long controversy was revived on motion capture technology. different battlegrounds in the visu- exalted as the solution to traditional al effects and computer animation THE SIGNIFICANCE film and animation shortcomings, industry. At conferences like SIG- The ILM awards are in line with the MoCap’s controversial applications are GRAPH, there were heated discussions advancement of the entire visual effects on the brink of transforming contem- between the lead animators from Pixar, industry and, in particular, the progress porary entertainment. Disney, Dreamworks, and film and ani- towards one specific goal—the creation mation schools (who argued for the of artificial humans and other creatures THE CONTROVERSY necessity of creating engaging and that are so realistic one can say that they The controversy around MoCap can be expressive characters) and the effects passed the Turing test of computer traced back to related techniques in experts from ILM, Digital Domain, graphics. That is, the creation of com- the 1920s, so-called “rotoscoping.” Sony, and others (who faced the chal- puter graphics generated humans that This technique entailed projecting lenge of creating photo-realistic stunts cannot be distinguished from film live-action footage onto cell animation and argued that animators can never recordings. As old as the invention of drawing tables and then tracing the animate real people realistically, and film itself, this goal has always remained motion onto cartoon characters. Walt especially not in the fast-paced time tantalizingly out of reach. Disney never publicly admitted to this, frame of their production schedules). While special effects designers have but when the Disney archives were So what exactly is this technology that been chasing this fantasy for 30 years, recently made public, most of Snow has divided the industry so powerfully this might be the decade that we reach White’s and all of the Prince’s motions for so many years? were shown to be traced from actors’ Digital Object Identifier 10.1109/MSP.2007.906023 motions in film recordings. This was (continued on page 156) IEEE SIGNAL PROCESSING MAGAZINE [160] NOVEMBER 2007 [in the SPOTLIGHT]continued from page 160 THE TECHNOLOGY and the performers do not need to wear Sony’s new MoCap system. A recent ILM Digitally capturing the movement of special suits. This has been the subject innovation allows the film director to humans can be done in many ways. Most of many computer vision research proj- see a real-time previsualization of the techniques can be classified as either ects in the academic community and MoCap effects right on the stage. marker-based or marker-free capture. In new R&D projects in industry. In theory, In addition to movie production, one marker-based capture, reflections on this should lead to much better motions, of the biggest users of MoCap is the markers that are placed on a body suit gaming industry. For instance, are recorded and these points are Electronic Arts has several in-house automatically visualized. In marker- THE CONTROVERSY AROUND MoCap studios, including one of the free capture, thousands of natural MOCAP CAN BE TRACED BACK world’s largest studios in Vancouver, points are captured, as in (for TO RELATED TECHNIQUES Canada, which is booked solid for all instance), a real-time three-dimen- IN THE 1920S, SO-CALLED of Electronic Arts sports titles, like sional (3-D) scan. ”ROTOSCOPING.” FIFA Worldcup (their Soccer game), Both of these methods have their Madden NFL (their Football game), or pros and cons. Most MoCap systems that but unfortunately, it is much more diffi- NBA Live (their Basketball game). have been used over the past decade are cult to track pixels with arbitrary surface Sometimes entire soccer or football marker based. In such systems, usually texture than with retro-reflective balls. teams are suited up in MoCap attire, between 6 and 50 cameras are placed on and several stars have performed their the walls and ceilings of a recording studio. SYSTEMS AND APPLICATIONS signature moves for electronic record- Infrared or visible light sources are mount- The leading marker-based commercial ings. These projects have formed a large ed next to each camera. The projected light systems have been developed by Vicon database of motion snippets that can be is reflected with maximum energy by the Motion Systems and Motion Analysis re-ordered and concatenated in real small “retro-reflective” markers back into Corp. More recently, PhaseSpace caught time on game consoles to enact any the cameras, and therefore can be easily up with the big players in terms of speed motion during a game. detected by further processing of the and accuracy, providing an alternative camera output. Triangulating the tracked active marker system based on light- TOWARD MARKERLESS METHODS two-dimensional (2-D) marker locations emitting diodes (LEDs) attached to the Most of the motions recorded with these of each camera allows for very accurate body. All commercial systems run now in marker-based systems still have a slight 3-D reconstruction of the markers in the real time and export the data to many feeling of diminished expressiveness. The recording studio. standard computer graphics packages, characters usually seem to have the For body segment (such as upper and such as Maya or MotionBuilder. Ten wrong weight or body force, and they lower arms and legs) MoCap, each body years ago, the use of MoCap was limited move in a more robotic fashion than a segment has a few such markers (usually to experts in studios and some research living, breathing human. This was a total of 50 markers are used for the universities, but nowadays, there are acceptable in Total Recall or Star entire body) and, in the next pipeline many large MoCap databases available for Wars—The Phantom Menace, when the step, a digital human skeleton can be fit- free or purchase, and smaller studios and target characters were robots, or in Lord ted to the marker motions. schools can afford multicamera systems of the Rings, where animators had the For human face capture, a number of for production, teaching, and even low- time to touch up the motions in postpro- tinier markers are placed at important budget art projects. duction. But in features like Square- facial features—for instance, around the Big visual effects studios like ILM, Soft’s Final Fantasy or Sony’s Polar mouth, cheeks, and eyebrows. Although Digital Domain, Weta Digital, and Sony Express, for which there was no time to the marker tracking itself is very accu- Imageworks used MoCap in feature films refine the motions by hand, the charac- rate and fast, very often the recorded such as Titanic, Lord of the Rings, and ters were stilted and somewhat zombie- motions look a bit robotic because there Star Wars, either for animating back- like. This again gave added fuel to the are many subtle deformations between ground motions (such as crowd scenes century-old debate between the anima- the markers (at the spine, hips, muscle or wide-angle shots on Titanic) or for tors and the MoCap experts. bulges, and so on) that can not be cap- providing motions for nonhuman crea- More recent applications of MoCap tured by this technique. However, since tures and touching them up with the suffer less from this phenomenon due to it is perceptually very important to con- hands of traditionally trained animators.