3 DSTORYBOARDINGFORMODERNANIMATION alexandros gouvatsos Doctor of Engineering in Digital Media Media School Bournemouth University Centre for Digital Entertainment University of Bath September 2017 This copy of the thesis has been supplied on condition that anyone who consults it is understood to recognise that its copyright rests with its author and due acknowledgement must always be made of the use of any material contained in, or derived from, this thesis. Alexandros Gouvatsos: 3D Storyboarding for Modern Animation © September 2017 supervisors: Dr. Zhidong Xiao Prof. Jian J. Zhang Jerry Hibbert Neil Marsden 2 ABSTRACT Animation is now a classic medium that has been practiced for over a cen- tury. While Disney arguably made it mainstream with some hand-drawn classics, today’s industry is focused on Three-Dimensional (3D) animation. In modern 3D animation productions, there have been significant leaps in terms of optimising, automating and removing manual tasks. This has al- lowed the artistic vision to be realised within time and budget and empow- ered artists to do things that in the past would be technically more difficult. However, most existing research is focused on specific tasks or processes rather than the pipeline itself. Moreover, it is mostly focused on elements of the animation production phase, such as modelling, animating and render- ing. As a result, pre-production parts like storyboarding are still done in the traditional way, often drawn by hand. Because of this disparity between the old and the new, the transition from storyboarding to 3D is prone to errors. 3D storyboarding is an attempt to adapt the pre-production phase of mod- ern animation productions. By allowing storyboard artists access to simple but scale-accurate 3D models early on, drawing times as well as transi- tion times between pre-production and production can be reduced. How- ever, 3D storyboarding comes with its own shortcomings. By analysing ex- isting pipelines, points of potential improvement are identified. Motivating research from these points, alternative workflows, automated methods and novel ideas that can be combined to make 3D animation pipelines more efficient are presented. The research detailed in this thesis focuses on the area between pre-production and production. A pipeline is presented that consists of a portfolio of projects that aim to: • Generate place-holder character assets from a drawn character line-up • Create project files with scene and shot breakdowns using screenplays • Empower non-experts to pose 3D characters using Microsoft Kinect • Pose 3D assets automatically by using 2D drawings as input 3 CONTENTS i research background 18 1 introduction 19 1.1 Hibbert Ralph Animation . 19 1.2 Research focus . 20 1.3 Impact . 21 1.4 Research contributions . 21 1.5 Thesis structure . 22 2 digital media in 3d animation pre-production 24 2.1 An analysis of animation as a medium . 24 2.2 Animation as media, animation in media and media in ani- mation . 27 2.3 Pipeline elements . 29 2.4 Breakdown of different media used in pre-production . 32 2.5 Animation producers as consumers of media . 36 2.6 Remediation of the pre-production phase . 39 3 storyboarding in 3d 42 3.1 Why storyboard in 3D........................ 42 3.2 3D storyboarding in theory . 43 3.3 3D storyboarding in practice . 44 3.3.1 Storyboarding . 47 3.3.2 Layout . 47 3.3.3 The HRA pipeline . 48 3.4 Moving forward . 48 ii projects portfolio 51 4 automatic creation of place-holder assets 53 4.1 Motivation . 53 4.2 Background . 54 4.3 Methodology . 55 4.4 Results . 58 4 4.5 Discussion & future work . 58 5 screenplay analysis system for automatic asset import 65 5.1 Motivation . 65 5.2 Background . 66 5.3 Methodology . 68 5.3.1 Parsing the screenplay . 68 5.3.2 Linking the information . 69 5.3.3 Generating Redboard project file . 70 5.4 Results . 71 5.5 Discussion & future work . 73 6 integration of placeholder creation and screenplay analysis 75 6.1 Motivation . 75 6.2 Background . 75 6.3 Methodology . 76 6.4 Results . 77 6.5 Discussion & future work . 77 7 cheapmotion-capture devices for humanoid posing in pre-production 80 7.1 Motivation . 80 7.2 Background . 80 7.3 Methodology . 83 7.3.1 Mapping of storyboard timeline to Maya timeline . 84 7.3.2 Mapping skeleton capture data to 3D rigged assets . 85 7.4 Results . 89 7.5 Discussion & future work . 91 8 automatic 3d posing from 2d hand-drawn sketches 96 8.1 Motivation . 96 8.2 Background . 97 8.2.1 Related work . 98 8.2.2 State-of-the-Art work . 104 8.2.3 Proposed Approach . 109 8.3 Methodology . 109 8.3.1 Overview . 110 8.3.2 PARAC-LOAPSO . 112 8.3.3 Comparing Drawings to Renders . 119 5 8.4 Results . 122 8.5 Discussion & future work . 129 iii concluding remarks 135 9 discussion & future work 136 10 conclusion 144 references 147 glossary 162 6 LISTOFFIGURES Figure 1 Example flow of a traditional pipeline with its distinct stages. 29 Figure 2 Example of pre-vis, with overlay of the corresponding storyboard panel at the bottom right. 31 Figure 3 Example screenplay from the movie ‘Wolf of Wall Street’ (Winter and Belfort 2013). 32 Figure 4 Example storyboard (Glebas 2009). 33 Figure 5 Screenshot of Redboard. 43 Figure 6 Example of 3D storyboarding, with 2D drawing on top of a 3D environment. The 3D models are unposed. 45 Figure 7 Efficient storyboarding workflow example from Tree Fu Tom (FremantleMedia, CBeebies, Blue-Zoo Pro- ductions 2012). (©BBC MMXVI) . 50 Figure 8 Example of input character drawings (© Hibbert Ralph Animation). 55 Figure 9 Automatically generated 3D pegs as seen in Autodesk Maya.............................. 55 Figure 10 Examples of automatically generated character ab- stractions. 60 Figure 11 Screenshot of the Peg Creator interface. 61 Figure 12 Step 1 — Segmenting the character from the line-up. Step 2 — Trimming it based on the feet. Step 3 — Av- eraging the colours into stripes to create a ‘mosaic’ ef- fect. Step 4 — Generating the rectangular cuboid and adding a plane on top to convey orientation. 61 Figure 13 The same character drawn differently can result in a different bounding box that may affect how the char- acter’s width is estimated. 62 7 Figure 14 Side by side comparison of the original character drawing (left-most) and different mosaic stripe thick- ness parameters: 10, 40 and 100 respectively from left to right. 62 Figure 15 Screenshot of Final Draft software. 67 Figure 16 Automatically populated Redboard project from ‘Q Pootle 5’ (Blue-Zoo Productions 2013) screenplay. 71 Figure 17 Automatic shot breakdown as seen in Redboard. 72 Figure 18 Structure of integration between the Peg Creation and the Screenplay Analysis software. 76 Figure 19 A screenshot of the Peg Creation software, where dropdown boxes allow the artist to choose the char- acter’s name. 79 Figure 20 The created system running within Maya, tracking the pose of an actor while viewing a series of storyboard panels. 83 Figure 21 The main User Interface (UI) of the system, as an actor is posing in an office environment, according to the selected storyboard panel. 85 Figure 22 For 25 frames per second, the first panel corresponds to the first frame in Maya’s timeline. 86 Figure 23 For 25 frames per second, the second panel corre- sponds to the 25th frame in Maya’s timeline. 86 Figure 24 The skeletal structure of the Kinect data. There are 20 joints on the human body tracked from the Kinect sensor (Microsoft 2017). 87 Figure 25 Kinect data needs to be transformed to be used in Maya. 88 Figure 26 The skeletal structure of an example 3D rigged T- posed asset in Maya. Note that there is not a one- to-one correspondence to the skeletal structure of the Kinect data shown in Figure 24............... 90 8 Figure 27 A screenshot during practice for the live demo se- quence. The top left window shows the actor acting out the pose in front of the Kinect, with the joints overlaid on top. The bottom left window shows the storyboard panel the actor is currently acting out. The Autodesk Maya window is updated whenever the ac- tor captures a new pose. 90 Figure 28 An example of an actor posing a 3D character in Maya, according to the selected storyboard panel. 92 Figure 29 An example of a character in a Maya scene posed us- ing the proposed pose capture system. The actor is in this case controlling the computer and thus does not appear in front of the camera. 93 Figure 30 An example of the forward-backward ambiguity issue in inferred 3D pose of a lamp when not using the internal lines for additional information. 104 Figure 31 Character poses created by sketching lines of action (Guay et al. 2013)....................... 105 Figure 32 Bone rotations are constrained to the viewing plane. The bones are parameterised as a single axis-angle component. The axis is the camera’s viewing direc- tion projected onto the floor plane. From left to right is the initial pose, a side view with the LOA, the re- sult after optimisation, and a frontal view of the final pose (Guay et al. 2013).................... 106 Figure 33 The pose descriptor consists of in-the-image plane an- gles for every limb segment (Jain et al. 2009). 108 Figure 34 Character poses that match the hand animation much more closely than the mocap animation. ‘Our result’ refers to the work by Jain et al. (2012). 108 Figure 35 Different descriptors for comparing poses in 2D.