TECHNIQUES IN LIP-SYNC AND FACIAL TRACKING USING MOTION CAPTURE: THE PRE-EVOLUTION TO MOCAP Jong Sze Joon Eddie Soon Eu Hui Faculty of Creative Multimedia LimKokWing University College of Multimedia University Creative Technology 63100 Cyberjaya, 63100 Cyberjaya, Selangor, Malaysia Selangor, Malaysia e-mail: [email protected] email: [email protected] Abstract 1. Introduction In terms of realistic animation, MoCap (Motion Capture) gains "There is no particular mystery in animation...it is really the upper hand over traditional animation techniques such as very simple, and like anything that is simple, it is about the keyframing and simulation due to the capability of real-time visualization and the accuracy of recorded data to produce a hardest thing in the world to do.” Bill Tytla (1937). distinctive quality (natural looking) of movements for animated characters. Motion capture is not without controversy, however. There are many distinctive MoCap (Motion Capture) based The goal of animation is not to create human like motion, but to animation from all over the world where hyper-real virtual CG impart unique personalities to animated characters, to give them characters will be an element of the future of digital storytelling. the "illusion of life". Mocap is used to create 3D character animation and natural simulations in a performance oriented way. Thus, some might There are still many unsolvable problems such as difficulty in hair better define the term, performance animation. However, our plug-in, for realistic flowing hair and follicles, cloth, skin current state in local production of MoCap technology is still texturing and proportion deformations. This research will focus on limited in the factors of resources, knowledge, technology as well the complexity of manipulating this MoCap data to further as the lack of experience and reference in the individuals of develop the controls of character animation to achieve most of the Malaysians. Therefore, it is hard to develop a local content of animation principles. This will then enable the animators to better motion-capture based animation without any guidelines and understand character motion manipulation to a new degree. One references. way of accomplishing this task is to understand and form a proper integration among the applications needed from capturing the data MoCap can be considered as the shortcut for photo realistic real- to manipulating the animation curves for desired motion of the life based animation. It also has some very useful applications for character. many types of users, not only does it contribute to the filmmaking industry but it also confers to edutainment purposes. Despite that, With a proper integration system establish, the animator can have the many usefulness of Mocap also contributes greatly to medical, full control of how the character can be manipulated to the simulation, engineering and ergonomic applications, for the extreme. The final output will be more effective with creation of generic and special purpose virtual reality character enhancement over realistic motion for character animation. As integration. It's also used in the entertainment industry for feature conclusion, this research will recommend the creative way of films, advertising, TV, and 3D computer games. Its sole purpose is using MoCap in 3D animation based on various approaches not simply to duplicate the movements of an actor or animator, but proposed. also as a process of taking a human's emotion and recording it in some fashion. Keywords: Motion-capture, key frame animation, real-time visualization, character motion manipulation. The use of motion capture for computer character animation is fairly new, having begun in the late 1970's, and only now beginning to become widespread. The idea of copying human motion for animated characters is, of course, been practiced for some time. A method called rotoscoping, has been commonly utilized for replicating live footage of actors playing out the scenes. This technique was invented in 1915 by Max Fleischer, a cartoonist, in an attempt to automate the production of animated Figure 1. Mocap data by actor playing a drum cartoons. The idea is to painstakingly trace the image of the converted to virtual character in 3D captured film frame by frame onto paper. It became contentious almost immediately because Fleischer was trying to get studios to mass-produce cartoon characters using this new technology, and that meant 2D animators would end up rather jobless. It took time for the industry to understand that rotoscoping would only be applied in cases when hyper-realistic human motion was intended. It remains controversial, and most studios do not like to admit using it. Later on, this copying technique is applied to 3D Figure 2. Example of Mocap data for the game Quake animation as well, whereby the animator overlays 3D models on a 3: Arena background 2D drawn images and even video sequence. Despite that, compositing and editing softwares were upgraded to Although the introduction to Mocap (motion capture) may date perform rotoscoping as well. back to as far as the 70s, it is now globally projected more into computer graphics character animation. Basically, the Mocap The goal of animation is not to create human like motion, but to system enables the animator to record the precise movement of a impart unique personalities to animated characters, to give them human subject in time and space for immediate or delayed the "illusion of life". Both the Rotoscope and motion capture analysis and playback, which can later be modified and applied to impose human motion on animated characters, which make them an existing 3D character model in any 3D platform. The recorded seem subtle and lifeless in comparison to those animated or hand data can be as general as the simple position of the body keyframed by skilled artists. This is also because actors cannot interacting with the geographical environment around, or as break the law of reality to fill in the principles of animation intricate as the movement of facial expressions comprise of applied in a fine animation. In the case of rotoscoping, artists muscles movements. However, no matter how high the trace human motion but interpret it with the model of the technology, the most important thing about Mocap is the ability of animated character. In the case of motion capture, human motion the actor to act. is copied and the data is directly applied to the animated character. The temptation to use this captured motion and call it "animation" has led computer animators practiced in the art of traditional animation to call it "Satan's Rotoscope" (a term attributed to animator Steph Greenberg). Figure 3. Camera capturing with a slow shuttle speed to show the path of the markers in space. 2. From times of yore to contemporary 2.1 A STEP BACK TO THE PAST “The big question is, why make digital humans at all? Sure, they look cool and they've got a certain kitsch value, but what purpose Figure 4. Polygon mask by done in Digital Fusion do they really serve in society?” - Laura Schiff, (2002.) to show rotoscoping in compositing software 2.1.1. What The Future Beholds? 2.1.3 Electromechanic Motion Capture Out of the many Mocap systems engineered, there is a set of favor only to a few classifications. The These systems are armatures that are worn by the performer Electromagnetic and Electromechanic Mocap systems were during capture. They consist of angular measurement: devices that measure the rotation of joints. Their output can generate motion previously massively used in the industry. Recently in the data in real-time, but their margin of error is large and their past few years, Optical Mocap was introduced. The capture frequency is low. Due to that, this system is very following are a brief description of the Mocap systems. economical. Because they only measure the rotation of joints, most electromechanic suits do not have a way of capturing the 2.1.2 Electromagnetic Motion Capture global position of the performance, so they need to substitute with other resources. One of the disadvantages of using a mechanical One of the types of motion-capture device is the electromagnetic suit is that it cannot collide with another actor. However, it can be tracker, which consists of a series of receivers or sensors, a used outdoors covering a wider capture area. transmitter and a control unit. The sensors are placed at all the joints of the actor and they are connected to the control unit and are placed on the main body of the performer. The transmitter 2.1.4 Optical Motion Capture generates a low-frequency magnetic field, and as the receivers move through it, the control unit is able to track their signal in There are mainly two types of optical capture systems order to calculate their position in space. commercially used today: Active Optical and Passive Optical systems. They both use the same underlying principals. A chain of These systems have been widely used for many years in television cameras placed around the capture area track the positions of broadcast and military applications. However, it has a higher markers attached to the body of the actors. Triangulation is used margin of error and smaller capture frequency than the optical to compute the 3D position of a marker at any given sample, from types, but they are less expensive and capable of real-time an array of 2D information from every camera. The Active optical feedback. These systems' biggest drawback is their susceptibility system uses illuminating elements as markers. On the other hand, to interference by metallic objects and confined within a small Passive optical systems however, use retro-reflective markers. capture area. Most systems are more flexible in the use of passive markers.