Pedro Tavares Barata Bastos EASY FACIAL RIGGING AND ANIMATION APPROACHES A dissertation in Computer Graphics and Human-Computer Interaction Presented to the Faculty of Engineering of the University of Porto in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Digital Media Supervisor: Prof. Verónica Costa Orvalho April 2015 ii This work is financially supported by Fundação para a Ciência e a Tecnologia (FCT) via grant SFRH/BD/69878/2010, by Fundo Social Europeu (FSE), by Ministério da Educação e Ciência (MEC), by Programa Operacional Potencial Humano (POPH), by the European Union (EU) and partially by the UT Austin | Portugal program. Abstract Digital artists working in character production pipelines need optimized facial animation solutions to more easily create appealing character facial expressions for off-line and real- time applications (e.g. films and videogames). But the complexity of facial animation has grown exponentially since it first emerged during the production of Toy Story (Pixar, 1995), due to the increasing demand of audiences for better quality character facial animation. Over the last 15 to 20 years, companies and artists developed various character facial animation techniques in terms of deformation and control, which represent a fragmented state of the art in character facial rigging. Facial rigging is the act of planning and building the mechanical and control structures to animate a character's face. These structures are the articulations built by riggers and used by animators to bring life to a character. Due to the increasing demand of audiences for better quality facial animation in films and videogames, rigging faces became a complex field of expertise within character production pipelines. The demand for more quality has contributed to increase the productivity in this field but also to disperse it and deprive it of a consensus on how to build and provide the mechanics and controls of a facial rig. Today, facial rigging can be considered fragmented because it is an extremely laborious, time-consuming, disparate and sometimes frustrating process for both riggers and animators. Aware of the problems described previously, the author presents in this thesis optimized rig approaches for facial animation that help facilitate and empower the artists workflow, by presenting a guide to place and configure the rig mechanical components and a flexible rig interface to control a character's face. The optimization is oriented to decrease the complexity of the facial rigging process for a human subject facial model as a basis for other facial styles and also for key frame facial animation, a technique which is in its peak of development iv ABSTRACT because it continues to be popular, despite the existence of more recent techniques like motion capture, therefore key frame is highly prone to be optimized. The research steps carried out by the author include an extensive study of the facial rigging techniques used by several artists worldwide in this field, through literary as well as approaches posted in the web, and then the definition of facial rigging and animation approaches to help ease the job of the rigger and the animator. The approaches presented are based on generic and validated guideline proposals to build facial rig mechanics and on a flexible facial rig control system based on a multimodal interface. The approaches developed were validated initially in a pilot study with three users, followed by four detailed user experiments, respectively involving five, fifteen, twenty and twenty two users. The facial rig approaches have been approved and considered by the users to (i) ease the construction process of the mechanisms that deform a facial model and (ii) ease the use of the control interfaces that allow animators to trigger facial expressions. The results have been published in a sequence of papers which provide a coherent storyline to this dissertation, a storyline which is meant to help bring a consensus to the state of the art in facial rigging and animation and, hopefully, decrease the fragmentation found in this research and production field. Key-words : Faces, Rigging, Animation, Interaction, Interfaces. Resumo Os artistas digitais que trabalham em desenvolvimento de personagens precisam de soluções otimizadas de animação facial para mais facilmente criar expressões faciais apelativas para aplicações em off-line e em tempo-real (i.e. filmes e jogos). Mas a complexidade da animação facial cresceu exponencialmente desde que surgiu na produção de Toy Story (Pixar, 1995), devido à maior exigência do público por animação facial com mais qualidade. Nos últimos 15 a 20 anos, empresas e artistas desenvolveram uma diversidade de técnicas de animação facial em termos de deformação e de controlo, que representam um estado da arte fragmentado em rigging facial de personagens. O rigging facial é o ato de construir as estruturas mecânicas e de controlo para animar a cara de uma personagem. Estas estruturas são as articulações construídas por riggers e usadas por animadores para dar vida às personagens. Devido a uma maior procura por animação facial de qualidade, o rigging de caras tornou-se gradualmente mais complexo nas produções profissionais de personagens. A procura por mais qualidade contribuiu para aumentar a produtividade nesta área mas, também, para a dispersar e privar de um consenso sobre como construir e disponibilizar as mecânicas e controlos de um rig facial. Atualmente, o processo de rigging facial pode-se considerar fragmentado porque é trabalhoso, demorado, disperso e, por vezes, frustrante para riggers e animadores. Consciente das questões descritas, o autor apresenta, nesta tese, abordagens otimizadas de rigs faciais que ajudam a facilitar e fortalecer o trabalho dos artistas, através de um guia para colocação e configuração das mecânicas de um rig facial e de uma interface flexível de rig para controlar a cara de uma personagem. Esta otimização pretende diminuir a complexidade do processo de rigging facial de um modelo 3D humano enquanto base para outros estilos faciais e está orientada à animação facial por key frames , uma técnica de animação que está no vi RESUMO seu auge de desenvolvimento por continuar a ser popular e muito recetiva a ser otimizada, apesar da existência de técnicas mais recentes como o motion capture . As etapas de investigação realizadas incluem um estudo extensivo das técnicas de rigging facial desenvolvidas por artistas de todo o mundo, através de referências literárias e de abordagens disponíveis na internet, e a definição das abordagens de rigging e animação facial facilitadoras do trabalho do rigger e do animador. As abordagens apresentadas baseiam-se em propostas de instruções genéricas e validadas para construir as mecânicas de um rig facial e ainda numa interface multimodal para controlo da animação facial. As abordagens desenvolvidas foram validadas inicialmente através de um estudo piloto com três utilizadores seguido por quatro experiências, respetivamente com cinco, quinze, vinte e vinte e dois utilizadores. Os utilizadores aprovaram e consideraram as abordagens como (i) facilitadoras do processo de construção das mecânicas que deformam um modelo facial e (ii) facilitadoras da construção e uso das interfaces de controlo para os animadores manipularem expressões faciais. Os resultados foram publicados numa sequência de artigos que proporcionam uma história coerente a esta dissertação, concebida para ajudar a providenciar um consenso ao estado da arte na área de rigging e animação facial e, esperançosamente, ajudar a diminuir a dispersão encontrada neste campo de investigação e produção. Palavras-chave : Caras, Articulação, Animação, Interação, Interfaces. Acknowledgments The course of this investigation was demanding but also gratifying. It was only thanks to the support of several people that I was able to produce this thesis, of which I am proud and also recognize that I owe it not just to me but to everyone mentioned following. To my supervisor, Professor Verónica Costa Orvalho, for accepting me as a doctoral student, for guiding me during my research and for always having a critical sense over my research progress. To the friend and research colleague Xenxo Alvarez Blanco, Technical Director at the research group Porto Interactive Center, for sharing his knowledge in rigging and for his moral support. To my other colleagues who I met in Porto Interactive Center for their fellowship. I thank the president and the juries of my research proposal defense, Professors Eurico Manuel Carrapatoso, Karin Gwinn Wilkins and Rui Pedro Rodrigues, for presenting pertinent questions which were fundamental for the evolution of the research. To Karen Gustafson for her support during my exploratory visit to Austin and to College Station in Texas in late 2011, where I met with Professors Frederic Parke and Tim McLaughlin, who contributed to help define the research pathway. To Pedro Vieira, of the Center of Events of Faculdade de Engenharia da Universidade do Porto, for his help in requesting and configuring the equipments used in the recording of my facial expressions in late 2012. To Professor Paulo Baeta Mendes for his vast experience in photography and for his equipment which he made available to photograph my face so that I could build and texture my 3D facial model. viii ACKNOWLEDGMENTS To Luís Álvaro Fernandes and Sharon Lee Strover for their collaboration in the final results of this research. To the Blender Foundation and