PIPELINE FOR POPULATING GAMES WITH REALISTIC CROWDS Rachel McDonnell, Simon Dobbyn and Carol O’Sullivan Graphics, Vision and Visualisation Group Trinity College Dublin Ireland email: [email protected]

Figure 1: Realistic crowds in virtual Dublin.

KEYWORDS rently, the that occupy crowd systems Real-time Crowds, Impostors, Cloth simula- in games use skinned meshes for their clothing, tion, Perceptual Metrics often resulting in rigid and unnatural motion of the individuals in a simulated crowd. While ABSTRACT there have been advances in the area of cloth With the increase in realism of games based simulation, both offline and in real-time, inter- in stadiums and urban environments, real-time active cloth simulation for hundreds or thou- crowds are becoming essential in order to pro- sands of clothed characters would not be pos- vide a believable environment. However, re- sible with current methods. We addressed this alism is still largely lacking due to the com- problem in previous work by devising a system putation required. In this paper, we describe for animating large numbers of clothed charac- the pipeline of work needed in order to prepare ters using pre-simulated clothing. and export models from a 3D modelling package into a crowd system. For our crowd, we use a In Dobbyn et al. [Dobbyn et al. 2006], we added hybrid geometry/impostor approach which al- realism to our crowd simulations by dressing lows thousands of characters to be rendered of the individuals in realistically simulated cloth- high visual quality. We use pre-simulated se- ing, using an offline commercial cloth simulator, quences of cloth to further enhance the charac- and integrating this into our real-time hybrid ters appearance, along with perceptual metrics geometry/impostor rendering system ([Dobbyn to balance computation with visual fidelity. et al. 2005]). We have also improved the quality of the impostors in the system by using percep- INTRODUCTION tual metrics described in [Hamill et al. 2005; In the games industry, real-time crowds are in- McDonnell et al. 2005; McDonnell et al. 2006; creasing in popularity due to the of McDonnell et al. 2007a] in order to drive the new level of detail (LOD) techniques. Cur- impostor rendering.

1 In this paper, we discuss the necessary steps to geometrical models with lower resolution ones, advance from a single character in a modelling and were able to create complex scenes with package such as 3D Studio Max, to a perceptu- thousands of characters. De Heras Ciechom- ally guided crowd of thousands of clothed char- ski et al. [de Heras Ciechomski et al. 2004] sig- acters in a real-time system. This useful step by nificantly improved performance by using four step guide will enable others to incorporate pre- discrete level-of-detail meshes for the humans in simulated deformable clothing and perceptual their crowd. Yersin et al. [Yersin et al. 2005] in- metrics into their crowd systems. Unlike pre- crease the number of humans significantly while vious work, we discuss in detail the challenges still having varied by switching be- that we faced when incorporating clothing into tween dynamic and static meshes, depending on the system and the practical use of our percep- the distance from the camera. tual metrics. Our results show a system capa- ble of rendering large realistic clothed crowds in Another approach to reducing the render- real-time. ing costs of large crowds is to replace the entire geometry of the character with BACKGROUND two-dimensional representations (impostors). Real-time crowds pose a difficult problem as Aubel et al. [Aubel et al. 1998] were the first there may be limited resources available at to use this technique for crowds of virtual hu- run-time. As the number of characters in- mans. They describe their dynamic impostor as creases, traditional techniques rapidly become “a simple textured plane that rotates to contin- overwhelmed and are unable to sustain inter- uously face the viewer.” A snapshot of the vir- active frame-rates. Generally, it is the number tual is textured onto the plane. The tex- of polygons that the graphics engine can dis- ture is updated from time to time, depending play per frame that limits the number of charac- on the movement of the human or camera. The ters that can be displayed in a real-time crowd. snapshot is produced at run-time by rendering Therefore, character meshes with a high num- the virtual human to an off-screen buffer. The ber of polygons need to be simplified in order polygonal model is then replaced with a single to achieve real-time display rates. Instead of plane with this texture pasted onto it. Tecchia simplifying all of the characters to a low level et al. [Tecchia et al. 2002] were able to render a of detail, different levels of detail of the charac- virtual city with thousands of walking humans ters’ meshes can be used in the scene. Detailed on a standard PC by using pre-generated im- meshes are used when the object is close to the postors to represent their humans. They im- viewer, and coarser approximations are substi- proved on existing techniques by shading the tuted for distant objects (e.g., as in [Funkhouser impostors in real-time. and Squin 1993]). This idea of using differ- One of the main disadvantages of the previ- ent levels of detail for objects in a scene is ous approaches was that the impostors looked an established technique. As early as 1976, pixellated when viewed up close. Dobbyn et Clark [Clarke 1976] described the benefits of al. [Dobbyn et al. 2005] presented the first hy- representing objects within a scene at several brid crowd system to solve the problem of de- resolutions. graded quality of impostors at close distances. Recently, two main approaches to real-time They did so by building an impostor system crowds have emerged in the crowd rendering on top of a full, geometry-based human an- community. One is to use low resolution geom- imation system, and switching between the etry, and the other is to use image-based rep- two representations with minimal popping ar- resentations for the characters in the crowd. tifacts. They also improved on previous impos- Ulicny et al. [Ulicny et al. 2004] replaced full tor techniques using graphics hardware to en-

2 hance the variety and realism of the impostors. In this paper, we describe the techniques nec- Alternatively, Pettre et al. [Pettre et al. 2006] essary for creating a perceptually guided crowd used three levels of detail, thus combining the of clothed characters using pre-simulated cloth strengths of the quality of dynamic sequences and perceptual metrics, starting from meshes with the performance of static meshes just one character in a modelling package. and impostors. PIPELINE

The depiction of cloth in computer graph- We present the pipeline of work needed to ics has occupied researchers and movie-makers achieve real-time crowds of clothed characters. since the 1980’s. Today, the need for re- In [McDonnell et al. 2006] we performed a rig- alistic clothing has become more and more orous series of psychophysical experiments in important for computer generated sequences order to determine the most appropriate repre- in movies. In the CG research community, sentations to use for depicting the deformations three types of physically-based models have of cloth. We found that a hybrid combination of been used for cloth simulation: Elasticity-based impostor and detailed geometry meshes would methods (e.g., [Carignan et al. 1992; Yang be most suited to our purpose since participants and Magnenat-Thalmann 1993]), particle-based were unable to tell the difference between this methods (e.g., [Breen et al. 1994]), and the type of crowd and a crowd containing all high mass-spring model (e.g., [Provot 1995]). Im- resolution geometry. plementing realistic cloth dynamics in real-time The system is an extension of the crowd system game applications still represents a significant detailed in [Dobbyn et al. 2005], and this pa- challenge for game developers. Simulating cloth per describes all of the alterations necessary to deformation requires much computation and is convert that crowd of skinned characters, to a a complex process both in terms of the dy- crowd of characters with deformable clothing. namics simulation and the collision detection for the changing cloth shape. In the animation There are three main stages in the pipeline research community attempts have been made process (see Figure 2): to produce more reliable real-time clothing for • Stage 1: Model Prepping: The first characters (e.g., [Vassilev et al. 2001; Cordier stage involves modelling the character and and Magnenat-Thalmann 2005]). adding a suitable animation for use in a crowd. The character’s clothes are then While generating cloth dynamics in real-time constructed and fitted, and finally simu- is becoming more advanced, the simulation lated in response to the underlying motion. groups or crowds of characters with simulated Our novel technique detailed in [Dobbyn clothing would not be possible with these tech- et al. 2006] for constructing a cyclical cloth niques. All previously described crowd systems animation is also applied here. used skinned meshes to clothe the humans in • Stage 2: Exporting the Data: The second their crowds. In [Dobbyn et al. 2006] we de- stage is concerned with exporting the hu- scribe how pre-simulated cloth sequences can man mesh, , cloth mesh be used to create large numbers of clothed char- and simulation, and impostors into a suit- acters. Perceptual metrics are also of extreme able format for use in the system. importance in order to make the most out of a • crowd. In [Hamill et al. 2005; McDonnell et al. Stage 3: Crowd Rendering: The final 2005; McDonnell et al. 2006] we developed met- stage involves details on how the crowd is rics for impostors and low resolution geometry, rendered in real-time. in order to balance visual fidelity with perfor- The following three sections will describe mance. the stages in detail. We also present some

3 Figure 2: Pipeline stages.

images of the final crowd system integrated achievable in real-time. Our characters ranged a Virtual model of Dublin city [Hamill and in detail from roughly 2000 to 9000 polygons, O’Sullivan 2003]. depending on the number of characters needed STAGE 1: MODEL PREPPING for the crowd. However, we also found in [Mc- Donnell et al. 2005] that the resolution that we Stage 1 of the process is concerned with the were using for our highest characters was actu- processing that is necessary in order to prepare ally perceptually equivalent to lower resolution a virtual character to be exported into the sys- characters. Therefore, there is a tradeoff be- tem. tween the number of polygons necessary for the Preparing the Human virtual human to look good in appearance and Choosing a suitable model is the first thing that the number necessary for displaying subtle mo- needs to be done. Characters in a crowd should tion information. This should be remembered look different from each other but should have when choosing the resolution of the high reso- the same overall style (e.g., a cartoon style char- lution character. We will use the example of a acter and a photorealistic character will not female character model called Angela with 6500 look good together in a crowd). polygons for the remainder of our description of We discovered in [McDonnell et al. 2005] that the pipeline process. characters of lower resolution around the joints are not as capable as those with more detail of displaying subtle motion variations. There- fore, the character should be detailed enough for both realism and the accurate deformation of the mesh around the joints in order to pro- duce good animations. However, when choosing Figure 3: (a) material IDs assigned to mesh, how detailed the character will be, it should be (b) mesh with greyscaled textures and diffuse remembered that, for a hybrid system, a num- colours set to white, (c) mesh with skeleton fit- ber of the high level of detail characters will ted. be displayed on-screen at one time, as well as many low level of detail characters. Therefore, Once a suitable character is chosen, it can be the detail of the character is limited to what is altered in one of the packages for 3D mod-

4 elling. We used 3D Studio Max but any other in the crowd. The diffuse colour of each ma- modelling package with built-in scripting lan- terial is set to white, and each texture map is guage or custom designed system could be used. greyscaled to allow for colour modulation with- Angela was purchased as a correctly texture out the loss of detail (Figure 3(b)). mapped mesh. However, in order to prepare Specifying Regions in a Single Texture Map her for exporting, some alterations had to be The alternative method is to apply a single made. texture map to the character. Some models are purchased with a single texture map, but Depending on the method used for colour mod- if not, it can be custom designed in an ap- ulation for the impostors in the system, two plication like Photoshop to suit the character. different approaches can be used at this stage The basic idea is to combine all of the texture to prepare the character’s mesh. The first is maps into a single compact texture map (Fig- to group the different coloured regions of the ure 4(left)). The coloured areas with no de- mesh together and use different materials for tail are simply areas that have a single diffuse each region. This method was used by Dob- colour. The alpha channel of this texture is then byn et al. [Dobbyn et al. 2005] for the purpose manually encoded with different colour regions of easily specifying the different colour regions (Figure 4(right)). Since we are using a sin- when generating the impostor images and the gle pass to render the impostors in the system mesh can be rendered using the OpenGL fixed (see [Dobbyn et al. 2005]), as many regions as function pipeline. The second method is to ap- the user requires can be specified at this stage. ply a single texture map image to the character, Once this custom designed map has been cre- with different colour regions specified in the tex- ated, it can be applied to the character. The ture’s alpha channel. This technique was used texture coordinates of the character need then by [de Heras Ciechomski et al. 2005] for the to be matched up to the texture, which can be preparation stage of their characters for a crowd a time-consuming task. This matching can be of low resolution geometry. For our clothed accomplished in 3D Studio Max using the Un- crowds, we use this approach in order to add wrap UVW modifier. pattern variety (detailed in Exporting the Im- postor). We will describe both techniques for completeness. Grouping Triangles The character’s triangles are first organised into groups, where each group is a body part that can be coloured differently from its sur- rounding parts. For Angela we had 8 differ- ent groups: eyes, head and shoulders, arms, fingernails, legs, top, shorts, and shoes (Fig- ure 3(a)). Each of these groups was assigned different diffuse materials or texture maps. The reason for organising the triangles in this way is Figure 4: (left) Example of a single greyscaled that it improves rendering when exported into texture map for Angela, (right) Corresponding the real-time system, due to the minimization alpha channel. of OpenGL states. Also, it allows for colour modulation, as each different group can have a Skinning and Animating palette of colours assigned to it, in order to cre- The character now needs to be skinned, in or- ate colour variety when duplicating characters der for an animation to be applied. Typically,

5 a biped skeleton is first fitted to the character produce confusing or unsatisfactory results in (Figure 3(c)), and then the vertices of the char- general. Interestingly, synthetic neutral walks acter mesh are weighted and linked to the differ- (such as those generated using 3D Studio Max’s ent bones of the skeleton. This can be achieved footsteps modifier) are considered male when in 3D Studio Max using the Physique or Skin viewed on the man and female when viewed on modifier. We used both from the woman. This implies that neutral walks our Vicon system and keyframing in 3D Studio may be applied to male and female virtual char- Max to obtain the animation for our character. acters as the appearance of the character takes The choice of animation depends on the type precedence over the motion in determining the of crowd scene that is being produced. The sex of the character. amount of animation data also has to be re- Preparing the Deformable Clothing stricted due to the memory limitations for the display of large crowds. In our system, an an- The commercial software that we used ([Clo imation is typically one second long and then ]) provided tools for constructing garment seg- looped in the system to create long sequences. ments, and for changing the different material The animations should be cyclical in order that parameters, along with accurate collision de- flickering animation artifacts do not occur when tection. We simulated many different types looping the animation in the system. of clothing using this software, including silky skirts, stiff dresses, sweaters and shorts. Cloth The amount of floor space that can be captured simulation is still limited in what it can achieve, using a motion capture system is often quite and therefore the choice of clothing and materi- limited. Therefore, capturing long sequences als is often compromised depending on the an- of humans walking is difficult, unless a tread- imation of the virtual human. For example, a mill is used. We can typically capture about character performing a back-flip in a tight fit- 4 steps of a human walk, and do some post- ting long dress would not produce nice results - processing in 3D Studio Max in order to make cloth by its nature is resistant to stretching, but it suitable for use. A candidate cycle is first not to bending. We found that a lot of tuning found in the motion captured walk, e.g., where was necessary to produce nice cloth animations, the left foot touched the ground first to the next for the human animations that we wanted. time it touches the ground. Although this mo- tion is cyclic, the pose of the character at the Template creation first frame is likely to be a bit different to that In [Grob et al. 2003; Keckeisen et al. 2004; at the last frame, which would cause artifacts Volino et al. 2005] and in most commercial when looped. To avoid this, we paste the pose packages, 3D virtual clothes are constructed at the start frame to the end frame and try from planar garment segments (often referred to interpolate the motion curves to make it fit to as ‘patterns’, however we will not use this without loosing too much of the individuality term to avoid confusion with the ‘pattern’ or and naturalness of the captured walk. motif variety detailed later). As this is the way real clothes are manufactured, virtual garments Care must be taken when capturing motion can therefore be designed to match real cloth- from actors and applying it to virtual charac- ing. It also makes particularly ters. In [McDonnell et al. 2007b], we found easy as correct texture coordinates can be ob- that walks captured from male actors applied tained automatically at the planar rest state. to female virtual characters are rated as am- biguous, as are female walks on the man. This The shape of the segment is usually drawn by implies that applying motion captured from ac- the user using splines or imported from a soft- tors of the opposite sex to the character will ware package designed for cloth manufacturers

6 like Modaris1 or PAD2. Seams and seam con- by the particles are determined by the type of nections are then specified by the user. A seam spring that connects them. Forces for struc- is usually an edge on a piece of fabric and can be tural springs are very large, whereas bend and attached to another seam using virtual sewing shear forces are small, which can lead to contra- threads. The planar segments are then posi- dicting effects. In order to have a more general tioned around the virtual character at approx- system, the mass-spring system can be thought imately correct positions. A good initial posi- of as a network of interwoven threads, where a tion for the clothing is very important and can thread is a chain of structural springs. Different determine how well the clothing fits. In order threads can then interact at mass points. More to achieve this, the segments should first of all complex forces are necessary to model this kind not collide with anything (the virtual character of behaviour. or each other). They should also be as close The Kawabata Evaluation System (KES) for as possible to the virtual character’s body and fabric is a technique used in the garment in- the sewing between seams should not penetrate dustry for measuring fabric mechanical proper- the character. The start pose of the character ties through normalised procedures [Kawabata can be changed in order to satisfy these rules; 1980]. In this system, five experiments are con- usually a T-pose is best. ducted using different instruments, and from these experiments 15 curves are obtained, which allow 21 parameters of fabric to be determined. These experiments measure shearing, bending, friction, compression, and deformation. These energies can be modelled in the particle system using a rectangular grid, where each particle in- Figure 5: Fitting a garment using ClothFX. teracts with its four direct neighbours. Once the segments and sewing are lined up, the In commercial software, coefficients that de- garment can be joined along the seam lines. scribe these 21 parameters can be changed in In [Keckeisen et al. 2004] sewing is achieved order to produce cloths that behave differently. by merging the segments along the seam lines. For example, for a material like silk the resis- For every pair of vertices along the seams, the tance to bending would be set to low, whereas matching triangles are coupled and the seams it would be set to high for a stiffer material like are moved halfway between the two original burlap. points, as shown in Figure 5. The segment Good collision detection is also necessary when pieces are attached and attain a shape influ- modelling clothing. When clothing a virtual hu- enced by the form of the body. man with deformable garments, full and self col- Material Modelling lision detection need to be implemented in or- The mass-spring model is the most commonly der for the clothing not to penetrate the human used technique for deforming garments. It con- or itself. Commercial cloth simulation packages sists of a simple particle system, where the usually have full and self collision built-in. This particles correspond to the vertices on a cloth is the most time-consuming task of cloth sim- mesh. These particles are connected by dif- ulation, and is a major reason why real-time ferent springs: structural springs for tension, simulation of clothing is so difficult. diagonal springs for shearing, and interleav- Cyclical Cloth Sequences ing springs for bending. The forces exerted In order to prepare the clothing generated for 1www.lectra.com/en/pds/modaris fashion.html our characters to be exported into the system, 2www.padsystem.com/ the animation had to be cyclical. As previ-

7 STAGE 2: EXPORTING THE DATA At this point, we have pre-simulated the defor- mation of both the virtual human’s skin mesh using linear blend skinning and its cloth mesh using the physical simulator, based on the mo- tion of its underlying skeleton. The cloth and human motion sequences are cyclical and the character should also have its triangles organ- ised into the appropriate groups or has a single Figure 6: Example of cyclical cloth. The cloth texture with specific alpha mapped coloured re- at the first frame is the same as the cloth at the gions. The character is ready to be exported last frame. from 3D Studio Max into files that can be loaded into the crowd system. ously described, in a real-time crowd system, Exporting the Geometry the characters’ animations are often cyclical in The virtual human was exported in the same nature, so that they can be smoothly linked to way as in [Dobbyn 2006]. A new plug-in was allow them to move in a fluid manner. Cyclical written to export the keyframes of the cloth animations are commonly obtained by manu- mesh. This plug-in simply stored the world- ally altering the underlying skeletal motion so space position of each vertex with respect to the that they loop in a realistic looking manner. root object for each keyframe of animation in an However, creating looping animations for char- organised manner. By pre-calculating and stor- acters with pre-simulated clothing is a more dif- ing the deformation of the cloth mesh in poses, ficult task, as manual cleanup of the cloth to this avoids the cost of simulating the clothes make it cyclical is very time-consuming, partic- at run-time. However, this also means that all ularly for very deformable items of clothing like animations are pre-determined and cannot be skirts, and can result in unrealistic effects. changed at run-time, which limits the motion variation in the crowd. In [Dobbyn et al. 2006], we describe an auto- Exporting the Impostor matic method for generating cyclical sequences of cloth simulation. Firstly, a very long ani- Generating the impostor representation of our mated sequence was created by repeating the character involves capturing two types of im- animation of the human many times and sim- ages from a number of viewpoints around the ulating the cloth in response to the repeating model: a detail map image to capture the de- animation. The long cloth sequence needed to tail of the model’s diffuse texture, and a nor- be searched using a distance metric that took mal map image whereby the model’s surface into account all of the vertices on the cloth mesh normals are encoded as an RGB value. For between the two frames of animation, in order more information on how these images are gen- to find one correctly cyclical loop. This resulted erated and rendered in the system see [Dobbyn in a two-pass algorithm. The first stage used a et al. 2005]. This section will describe any im- distance metric based on mesh vertex compar- provements to the algorithms and the necessary isons to find candidate cycles, and the second changes that were made in order to incorporate used edge image comparisons on the candidate deformable cloth. cycles to produce the final cyclical cloth anima- Generating Images tion (see Figure 6). This method should be used The normal maps in [Dobbyn et al. 2005] took at this point in the pipeline before proceeding considerable time to generate, as per-pixel look to export the animations. ups and operations were needed, so we im-

8 ping if it was back-facing. The new technique for adding hardware assisted pattern variety to the impostors that we devel- oped in [Dobbyn et al. 2006] involves a slightly different process for generating impostor im- ages. The detail map image is replaced with a texture coordinate map or UV map. This is similar to a normal map just described. How- ever, this time it is the texture coordinates that Figure 7: Mesh after each vertex has been au- are converted into an RGB colour and then tomatically painted with an RGB value corre- painted onto the vertices of the mesh. Similar sponding to its normal. to the normal map images, these images were generated for each viewpoint. proved the algorithm using a less computation- ally intensive technique. A copy of the char- Reducing Texture Memory Consumption acter’s mesh and cloth at the current frame Tecchia et al. [Tecchia et al. 2002] saved on was first needed. Each vertex normal was first texture memory when generating impostors by converted into eye-space coordinates, to find exploiting the symmetric nature of the human the normal with respect to the camera, and body performing a walk animation (both the then converted into an RGB colour (using the animation and the body are symmetric). They equations described in [Dobbyn 2006]). Per- halved the memory needed by mirroring the an- vertex colouring was then used to paint the imation: 32 samples were acquired by sampling RGB colours onto the vertices of the copied the mesh at 16 different points along one side, meshes (3D Studio Max’s VertexPaint modifier and mirroring these images to produce the sam- was used to do this). These vertex colours were ples on the other side. This worked very well for interpolated over the polygons, creating a char- skinned humans, as the human and the cloth- acter mesh with normal map colours (Figure 7). ing could be made symmetric quite easily (i.e., The normal map image was then generated by by mirroring one side), and would remain so rendering an image of this mesh, from the cur- for the animation. This approach could not be rent viewpoint. Per-vertex colouring and in- taken for generating impostors of humans wear- terpolating are operations that are performed ing deformable clothing, as the folds in cloth very quickly, as they are supported by graphics are rarely symmetric in nature and mirroring hardware. This meant that the image could be them would produce artifacts at run-time. We produced almost immediately, without the need tried generating a mirrored cloth animation us- for slow per-pixel operations. ing tools in 3D Studio Max, but it involved too much manual tuning and the results were not Incorrect back-facing polygons was another is- convincing. Therefore, the human was sam- sue which arose when implementing cloth im- pled on both sides to create the cloth impostor postors. The previous impostor plug-in did not images, which increased texture memory con- account for back-facing polygons as it was not a sumption, but avoided artifacts. noticeable problem for skinned clothing, which clung tightly to the body and did not turn Ideally, impostors would be generated at very inside-out at any point. However, for flowing small intervals around a sphere, the same num- garments, this occurred quite regularly and re- ber of times that a polygonal model would be sulted in incorrect normals in the impostor im- updated, which would allow seamless transi- age. A simple approach was used to solve this, tions between the images. However, as we are by testing the direction of the normal and flip- using pre-generated textures, texture memory

9 consumption prevents choosing such a dense in the pipeline should be chosen based on these sampling, so there is a need to pick an op- metrics. timal number of viewpoint images to gener- The number of frames of animation that are ate. In [McDonnell et al. 2006], we performed generated is also a factor to consider when a psychophysical experiment that determined reducing memory consumption. For memory the optimal impostor update frequency, balanc- critical systems such as real-time crowds ing texture memory consumption with visual fi- using impostors, the fewer frames required to delity. make an animation appear smooth the better. In [McDonnell et al. 2007a] we found that for a range of different types of animation, no more than 40 frames (or poses) per second was necessary for participants to find the animation smooth. We also found that background impostors could be displayed at 16Hz, with foreground geometry at 30 or 20Hz without observers noticing the difference. This results in the ability to store double the number of Figure 8: (left) character with large width to characters in memory than would be possible depth ratio, (right) character with small width if the impostors were being displayed at 30Hz. to depth ratio. Therefore, for every second of animation, 16 impostor frames should be generated. It was found that the amount of deformation in the clothing did not affect the update rate, where characters with flowing cloth needing the STAGE 3: CROWD RENDERING same number of updates as those with stiff We are finally at the stage where all data is clothing. However, we found that the width ready to be imported into the crowd system and to depth ratio of the characters did affect the used to display crowds of clothed characters. A update rate, where characters with large width number of alterations needed to be made to the to depth ratios needed more updates than those crowd system in order to incorporate our per- with smaller ratios (Figure 8). ceptual metrics and the pattern variation tech- For normal pedestrian characters (characters nique. In this section, we will first give a broad with small width to depth ratios), a viewpoint overview of how the system was written, then ◦ update rate of 17 is necessary to guarantee we will look at how each of the levels of detail with high probability that users will not no- were rendered and how switching was achieved. tice viewpoint changes of the impostors. This This work was based on the approaches detailed corresponds to 21 images that need to be gener- in [Dobbyn 2006]. ated at equal spacing around the character. We suggest rounding to the nearest even number of Setup images (22) in order to include the direct front The framework was based on a perceptually and the direct back images, particularly in ap- driven LOD approach, whereby the viewer’s plications where a front-on view would be most perception is exploited to compute less accu- noticeable. For other characters whose width rate models when they would not be noticed. to depth ratios are large, a viewpoint update The highest level of detail was the mesh model rate of 9◦ is advised. This corresponds to 40 and the lowest was the impostor, as previously images around the character. The number of described. The system is a C++ application impostor images to be generated at this stage running on the Win32 API, where the OpenGL

10 rendering library is used as the core component ferent outfits can be specified for each different of the rendering subsystem. The number of hu- template human as described in [Dobbyn 2006]. mans in the crowd is first specified and they As mentioned before, there are two techniques are then given random initial positions in the for setting up the virtual humans’ colours. We scene. Algorithms can be implemented to make will now describe the two corresponding tech- sure that no human intersects, or that they are niques to colour the human at run-time. all oriented in a certain way. Each crowd in- The first technique grouped together the tri- dividual is then randomly allocated a human angles of the different regions of the mesh and template model, and variety is added by ran- then tagged them with IDs. At run-time, the domly choosing a specified outfit, which is a set OpenGL fixed function pipeline is used for ren- of colours for each of the different parts of the dering the mesh, where the diffuse colour of body. each mesh region is changed depending on its Rendering the Geometric Human and material ID and corresponding outfit colour. Cloth Models This was the technique used in [Dobbyn et al. 2005] and is suitable for meshes that will be The system is optimised by taking advantage used alongside detail mapped impostors. of the fact that crowd individuals perform a default walk animation. Previously, we used The second technique is needed to match the static meshes, where the deformation of the geometry with the UV mapped impostors. This mesh is pre-calculated. However, with advance- method should be used if texture variation is ments in graphics hardware we now calculate important to add variety, as in the case of the deformation of the mesh on the GPU us- adding different designs to clothing. This tech- ing linear blend skinning. The cloth anima- nique uses programmable graphics hardware to tion is stored as pre-baked poses directly from add variation. As described previously, a sin- 3D Studio Max, as described previously. The gle texture map was applied to the character, rendering speed is improved by using Vertex where the alpha channel of the texture speci- Buffer Objects (VBOs) to store the key-frames fied the different colour regions. The shading of animation for both human and cloth poses. sequence for the geometry is shown in Figure 9. The idea behind VBOs is to provide buffers (re- gions of VRAM) accessible through identifiers. A buffer is made active through binding (in a similar manner to display list binding). This al- lows graphics drivers to optimise internal mem- ory management and also to choose the most suitable type of memory to store the buffers. At run-time, the correct VBO pose is selected and rendered, depending on the current frame of animation of the virtual human. Figure 9: Geometric mesh shading sequence. Adding colour variation to the mesh involves first creating different outfits, which specify the Rendering the Impostor different colours to be used for each different In order to render the impostor, the correct region in the mesh. For example, an outfit for viewpoint image needs to be found. Using the Angela would consist of: eyes coloured blue, camera position and the virtual humans posi- head and shoulders coloured pale pink, arms tion and direction, the most suitable viewpoint pale pink, fingernails red, legs pale pink, top image can be calculated and retrieved from white, shorts blue, and shoes white. Many dif- the large texture containing the pre-generated

11 viewpoint images. This image is then mapped ited to the playing field in sports games, pre- onto a quadrilateral. To dynamically orien- generating viewpoints from behind the charac- tate the quadrilateral towards the viewer, the ter is not necessary, thus reducing the amount amount to rotate can be calculated using the of texture memory consumed by the impostors camera and virtual humans position. See [Dob- (Figure 10). byn et al. 2005] for detailed descriptions of the Switching Between Representations equations. Ulicny et al. [Ulicny et al. 2004] noted that im- postors were an efficient approach for render- ing crowds of far-away humans, but that their pixellated appearance when displayed close to the viewer prevented them from being used for detailed crowds. Dobbyn et al [Dobbyn et al. 2005] solved this problem by using a hybrid impostor/geometry system, similar to that de- scribed in this paper (Figure 11). In crowd scenes where the humans and the camera are moving, the distance from impostors to viewer will change throughout the viewing time. Thus, a switching mechanism was employed to allow impostors to switch to geometry and vice versa when they reach a certain distance. In Hamill et al. [Hamill et al. 2005], we found a perceptual metric for the optimal pixel to texel ratio at which to switch representations, which ensures that users will not notice the switch. This ratio was one-to-one, which means that geometry can be switched to impostor repre- sentation when one of its texels is displayed on one pixel on the screen. A seamless transition is achieved by matching the pose of the impos- Figure 10: Examples of our stadium crowd. tor to the geometry at the point of switching. This technique did not need to be altered for In the case of crowds that do not move within our clothed characters, as the poses of the cloth the virtual environment, such as those found in were inherently linked to the human poses. sports games, viewpoint selection and the ori- enting of the billboard for each character can In [McDonnell et al. 2006], we validated this be done on the vertex processor. Billboards of metric by performing a system experiment individuals can be batched together in a sin- which examined participant’s perception of gle vertex buffer object and rendered in a sin- level of detail representations in different sized gle draw call. For more details see [Mill´anand crowds. We found that participants did not no- Rudom´ın2006]. It should be noted that smaller tice the difference between a hybrid crowd of numbers of animation frames will result in big- impostor/geometry and a crowd of all high reso- ger batch sizes, since only individuals using the lution geometry, when the camera was zooming same impostor texture can be batched together. in and out of the crowd, which meant that pop- Additionally, since the camera is typically lim- ping between representations was not noticed.

12 allows the matching of texture and colour varia- tion between the cloth and skin. In Figure 1 we show our virtual model of Dublin city inhabited by crowds of pedestrian characters wearing de- formable clothing. All of the walk cycles were captured using a motion capture system, and pattern variety was added using our UV map- ping technique.

Figure 12: Sheep impostors

In [Skrba et al. 2006], we extended the tech- niques described in this paper to produce ani- mal impostors, by adding highly realistic pre- simulated fur to sheep and dogs. Figure 12 shows a screen shot from this system. The pipeline could also be used to create realistic characters with pre-simulated hair, by replacing the cloth creation stage with hair simulation. Figure 11: (top) Crowd Scene, (bottom) The main limitation at present is that all of Crowd scene displayed in wireframe, the impos- the animations are pre-baked and do not allow tors are displayed as green quadrilaterals. for animation variety. Currently our impostors DISCUSSION are only generated for a single ani- mation, and are also restricted to “pre-baked” In this paper, we provided a useful step by step poses at 10 frames per second. This is be- approach to realising crowds of clothed charac- cause an image must be created for each view- ters incorporating perceptual principles. This point of the human for each frame of the ani- pipeline should be of use to game developers, mation, which represents a significant memory to create and/or improve the quality of their overhead. As we wish to imperceptibly switch crowds. from geometry to impostor, and vice versa, we Our technique of adding pre-simulated cloth- are therefore currently restricted to the simple ing improved the realism of our crowd sys- walk cycle for the high-detail models in our tem. The visual quality of the clothed crowd is crowds also. Therefore, while high visual fi- maintained by creating cyclical cloth motions delity of appearance can be achieved, the va- to avoid discontinuous motion artifacts. Ad- riety of motion in our crowds is restricted. To ditionally, the use of the impostor’s UV map improve upon this, we would like to investigate complements the use of impostors with its mesh efficient formats for storing representation in a LOD crowd system, since it character motion.

13 BIOGRAPHY disias. The 5th International Symposium Rachel McDonnell on , Archaeology and Cultural is a postdoctoral Heirtage (VAST), 9–17. researcher at the de Heras Ciechomski, P., Schertenleib, Graphics, Vision and S., Ma¨ım, J., and Thalmann, D. 2005. Visualisation Group in Reviving the roman odeon of aphrodisias: Trinity College Dublin Dynamic animation and variety control of where she recently crowds in virtual heritage. VSMM , 601–610. finished her PhD Dobbyn, S., Hamill, J., O’Conor, K., entitled “Realistic and O’Sullivan, C. 2005. Geopostors: a Crowd Animation: A real-time geometry / impostor crowd render- Perceptual Approach”. Her research inter- ing system. In SI3D ’05: Proceedings of the ests include perceptually adaptive graphics, 2005 symposium on Interactive 3D graphics real-time crowd simulation, virtual human and games, 95–102. animation and cloth simulation. Dobbyn, S., McDonnell, R., Kavan, References L., Collins, S., and O’Sullivan, C. 2006. Clothing the masses: Real-time clothed Aubel, A., Boulic, R., and Thalmann, crowds with variation. In Eurographics Short D. 1998. Animated impostors for real-time Papers, 103–106. display of numerous virtual humans. In Pro- Dobbyn, S. 2006. Hybrid Representations and ceedings of the First International Confer- Perceptual Metrics for Scalable Human Sim- ence on Virtual Worlds, 14–28. ulation. PhD thesis, University of Dublin, Breen, D. E., House, D. H., and Wozny, Trinity College. M. J. 1994. Predicting the drape of woven cloth using interacting particles. In Proceed- Funkhouser, T. A., and Squin, C. H. ings of ACM SIGGRAPH, 365–372. 1993. Adaptive display algorithm for interac- tive frame rates during of com- Carignan, M., Yang, Y., Thalmann, plex virtual environments. In Proceedings of N. M., and Thalmann, D. 1992. Dressing ACM SIGGRAPH, 247–254. animated synthetic actors with complex de- formable clothes. 26, 2, Grob, C., Fuhrmann, A., and Luckas, V. 99–104. 2003. Automatic pre-positioning of virtual clothing. In SCCG ’03: Proceedings of the Clarke, J. H. 1976. Hierarchical geometric 19th spring conference on Computer graphics, models for visible surface algorithms. Com- 99–108. munications of the ACM 19, 10, 547–554. ClothFX, cloth simulation software. Hamill, J., and O’Sullivan, C. 2003. Vir- Size8Software, 2004. tual dublin - a framework for real-time urban simulation. Proc. of the Winter Conference Cordier, F., and Magnenat-Thalmann, on Computer Graphics 11, 1-3. N. 2005. A data-driven approach for real- time clothes simulation. Computer Graphics Hamill, J., McDonnell, R., Dobbyn, S., Forum (Eurographics 2005) 24, 2, 173–183. and O’Sullivan, C. 2005. Perceptual eval- de Heras Ciechomski, P., Ulicny, B., uation of impostor representations for virtual Cetre, R., and Thalmann, D. 2004. A humans and buildings. Computer Graphics case study of a virtual audience in a recon- Forum 24, 3, 623–633. struction of an ancient roman odeon in aphro- Kawabata, S. 1980. The standardization and

14 analysis of hand evaluation. The Textile Ma- World (CAVW) Journal - CASA 2006 spe- chinery Society of Japan. cial issue 17 , 445–455. Keckeisen, M., Feurer, M., and Provot, X. 1995. Deformation constraints in Wacker, M. 2004. Tailor tools for a mass-spring model to describe rigid cloth interactive design of clothing in virtual behavior. In Graphics Interface ’95, 147–154. environments. In VRST ’04: Proceedings Skrba, L., Dobbyn, S., McDonnell, R., of the ACM symposium on Virtual reality and O’Sullivan, C. 2006. Animating software and technology, 182–185. dolly: Real-time herding and rendering of sheep. In Proceedings of Eurographics Ireland McDonnell, R., Dobbyn, S., and Workshop, 7–12. O’Sullivan, C. 2005. LOD human rep- resentations: A comparative study. Proceed- Tecchia, F., Loscos, C., and Chrysan- ings of the First International Workshop on thou, Y. 2002. Visualizing crowds in real- Crowd Simulation, 101–115. time. Computer Graphics Forum (Eurograph- ics 2002) 21, 4, 753–765. McDonnell, R., Dobbyn, S., Collins, S., Ulicny, B., deHeras Ciechomski, P., and O’Sullivan, C. 2006. Perceptual and Thalmann, D. 2004. Crowd- evaluation of LOD clothing for virtual hu- brush: interactive authoring of real-time mans. In Proceedings of the 2006 ACM SIG- crowd scenes. In Proceedings of the 2004 GRAPH/Eurographics Symposium on Com- ACM SIGGRAPH/Eurographics Symposium puter Animation, 117–126. on , 243–252. McDonnell, R., Jorg,¨ S., Hodgins, J. K., Vassilev, T., Spanlang, B., and Newell, F., and O’Sullivan, C. 2007. Chrysanthou, Y. 2001. Fast cloth Virtual shapers & movers: Form and motion animation on walking avatars. Computer affect sex perception. In Proceedings of the Graphics Forum 20, 3, 260–267. ACM Siggraph/Eurographics Symposium on Volino, P., Cordier, F., and Magnenat- Applied Perception in Graphics and Visuali- Thalmann, N. 2005. From early virtual sation (to appear). garment simulation to interactive fashion de- McDonnell, R., Newell, F., and sign. Computer-Aided Design 37, 6, 598–608. O’Sullivan, C. 2007. Smooth movers: Yang, Y., and Magnenat-Thalmann, N. Perceptually guided human motion simula- 1993. An improved algorithm for collision tion. In Proceedings of the 2007 ACM SIG- detection in cloth animation with human GRAPH/Eurographics Symposium on Com- body. In Proceedings of the First Pacific Con- puter Animation (to appear). ference on Computer Graphics and Applica- Millan,´ E., and Rudom´ın, I. 2006. Im- tions, 237–251. postors and pseudo-instancing for gpu crowd Yersin, B., Maim, J., de Heras Ciechom- rendering. In Proceedings of the 4th inter- ski, P., Schertenleib, A., and Thal- national conference on Computer graphics mann, D. 2005. Steering a virtual crowd and interactive techniques in Australasia and based on a semantically augmented naviga- Southeast Asia, 49–55. tion graph. In Proceedings of the First In- ternational Workshop on Crowd Simulation, Pettre, J., de Heras Ciechomski, P., 169–178. Ma¨ım, J., Yersin, B., Laumond, J.-P., and Thalmann, D. 2006. Real-time navi- gating crowds: Scalable simulation and ren- dering. Computer Animation and Virtual

15