Crossing the Line: Moving from Film to Games and Possibly Back

Crossing The Line: Moving From Film to Games and Possibly Back

Story = Gameplay Evan Hirsch

Senior Art Director Microsoft Game Studios [email protected] | [email protected]

Course Agenda

Time Section Presenter

8:30 - 9:00 Introduction Evan Hirsch Story=Gameplay (Trading Visual Expectations for Frame Rate and Gameplay) 9:00 - 9:45 Team Structures and Workflows Rick Stringfellow

9:45 - 10:15 Real Time Character Pipelines Paul Amer

10:15 - 10:30 Break

10:30 - 11:15 Lighting/Post processing Brien Goodrich Jamie Marshall 11:15 - 12:00 Transitioning your Shader and FX skills between Film Cliff Brett and Real Time 12:00 - 12:15 Q&A All

Two Different Mediums, One Ultimate destination • The Living Room is the ultimate market • Television Sets are ultimate delivery medium • Shared set of expectations by consumers… – Language of Visual Structure – Antagonists/Protagonists – Escapism

Consumer Spending: Film, DVD and Games

Consumer/End-User Spending (US$ Millions) $70,000

$60,000

$50,000

$40,000 DVD/VHS Sales Box Office $30,000 Video Games (console HW not included)

$20,000

$10,000

$- 1999 2000 2001 2002 2003 2004p 2005 2006 2007 2008 2009

More Market Trends: Game Software

Translating Market Spend to Real Time

PS1/N64 PS2/Xbox Xbox 360 Feature Film

Typ. Poly Count/Frame 5-10K 80-100K 150-300K N/A

1024 x 1024 x16 Typical Texture Size 64 x 64 x 4bits 256 x 256 x8bit 2048 x 2048 x 32bit bit

Bones per Character 12-25 20-35 30-60 100+

Animations per game 500 800 1500-2000 -

320 x 240 @ 640 x 480 @ 1280 x 720 ~2048X 1152 @ Rendering Resolution** 15-20FPS 30-60 FPS 30-60FPS 24FPS

Typ. Team Size for a AAA title 12-15/10 15-20/40-50 17-30/50-70 N/A (Programmers/Artists)

Maturity of Games as a Medium?

• The more successful developers have stopped trying to make games as interactive movies • Growth of business looking to “casual games” segment or broadening trends • Film and licensed properties continue as successful segments too

Mythbusting: Game Graphics Are Really Simple

Really Simple: CPU Geometry Size

Really Simple: Draw- Draw Calls

Really Simple: Draw - Sectors

Really Simple: Draw - Wireframe

Really Simple: Draw – Filled Wires

Really Simple: Draw – Portal Culling

Really Simple: GPU – Color Pass

Really Simple: GPU – Simple Shaders

Really Simple: GPU – Light Count

Really Simple: Lighting – Debug View

Really Simple: Lighting – Attenuation Map 1

Really Simple: Lighting - Lambert

Really Simple: Lighting – Point 0

Really Simple: Lighting – Point 1

Really Simple: Lighting – Point 2

Really Simple: Lighting – All Points

Really Simple: Lighting – Spec Pass

Really Simple: Lighting – Spherical Harmonic

Really Simple: Lighting – Normal (repeat)

Really Simple: Complete w/Post Processing Passes On

Creating for Linear v. Interactive

Film Games • All hail the Director • All hail the Player • Direction is supposed to • Direction is far more be clear collaborative • Programmers serve the • Programmers drive the image image • Every pixel tells a story • Every click drives the story • Story leads all • Gameplay leads all – Does your work push the – Does your work add value to story? the gameplay experience? Evan Hirsch: Crossing the Line: Story=Gameplay © 2007 Microsoft. All Rights Reserved

Moving from Film to Games

Pros Cons • Excellent production values • Games is a “lesser” medium • Used to taking and sticking to • May not have any interest in a direction games • Used to long hours and • Expect a more established crunches environment • Very experienced in taking • Don’t actually want the direction freedom/looseness of games • Expect high levels of support • Require training for R/T (TD’s, tools, etc) It all comes down to core craft Evan Hirsch: Crossing the Line: Story=Gameplay © 2007 Microsoft. All Rights Reserved

Moving from Games to Film

Pros Cons • Used to working in fluid teams • Production values are artists dependant • Expects to have technology change in production** • Titles mean different thing in different places • Used to long hours and crunches • Likely not used to level of artists direction that comes • Very proficient at working lean established hierarchy and light • Reel may be harder for you to relate to It all comes down to core craft Evan Hirsch: Crossing the Line: Story=Gameplay © 2007 Microsoft. All Rights Reserved

Games to film: some examples

• LODS – Necessary for game performance and memory constraints – Turns out they cut render times for film • Crowd systems and large flocking architectures

Film to games: Borrowing is Flattery

• Visual Language – Camera – The “Designs”: Production, Costume, Lighting – Audio (Soundtrack, Foley, etc) • Structure (3 Act Structure) • CG backbone: Shaders, Lighting, etc.

Applying Bloch’s Principles to Games • The same elements apply, but in space, not time. – Sets = Levels – Plots = goals/challenges – Mechanics of story = mechanics of gameplay – Tempo = Tempo – Heroes = Heroes; Bad Guys = Bad Guys (still)

A troll goes to the store with…

• Shadowrun, a 16 player game, each character can spawn 2 more characters. • If a troll spawned a pair of minions and then went to the store with a minigun that fires 30 rounds per second, how many collisions and draw calculations do you need per frame when the troll encounters a team of 8 bad guys?

Crossing the Line: Moving From Film to Games (and Possibly Back)

Rick Stringfellow Senior Art Director, EA Canada.

1. Game Team Structure. 2. Content Process. 3. Technical Design. 4. Viewing Content. 5. Artistic Design. 6. So is game development for you? 7. If you make the leap can you return to film?

1. Team Structure:

Discovering that game companies are not film companies!

The most immediate difference between the two industries is the structure of teams and the definition of roles. This is a product of the age difference between the two industries – film has had a century to mature, the games industry is only entering it’s 3rd decade and has seen a dramatic evolution of technology and growth in those years.

One of the most common reactions when entering this industry from film is to look for the people and processes that you depended on in the film industry – in many cases the roles aren’t there and the processes are different. On entering this industry I spent weeks looking for familiar roles – only to come to the conclusion that the roles and skills are often amalgamated, not required or non-existent. Directors, DOP’s, Editors and the like just don’t exist – although occasionally you might find them hired to create ‘cinematic’ sequences but in general the familiar film roles are nowhere to be found.

When the game industry started there were no other role models that could help guide the team development and no tools from 3rd parties to make it easier. Most games came from very humble beginnings in a basement or garage. Many people within these small groups had to wear multiple hats in order to get a complete game made. When the successful companies emerged out of this most of the future growth occurred through scaling up the roles rather than creating specific duties. When I joined the industry, artists had very little specialization – modelling, lighting, texturing and animation where often all done by the same team or person. In the larger companies even the salary structures were organized by generalist grades of ‘artist’.

Obviously every game company is different and the level of experience and maturity that you find varies widely. The reality is that the games industry is still evolving and what often happens is that you will find a real mix of software, corporate and film company experience mingled together.

The most successful game companies are the ones that have realized that making games requires defining a way of working that is unique and right for the end product. When entering this industry you have to be open minded and ready to adapt to it’s ways – making a game is not like making a film – it’s a medium on its own, it has it’s own ways of working and is still evolving.

2. Content Process

So how do you find out what you are making?

The process of making a film is very mature in comparison to games – from initial story, through storyboard and screenplay it generally progresses through stages to completion. Shots can be finished out of sequence and dropped into the final timeline and the film measured in a percentage of completion. Quality levels can be established early and hopefully upheld through to the final shot.

Unfortunately looking at game development this way is much harder. Game development is essentially a ‘holistic’ process. You can measure the completion of parts but in general the whole game only comes together towards the end of production. Because of the hardware and delivery media restrictions you can never be guaranteed to have finished everything until the games most complicated component fits in RAM and the whole game fits on the disk or through target bandwidth. This is one of the biggest design challenges that face the Art Director, CG Supervisor and technical teams.

Generally the first reaction coming from film is to try to find the equivalent of the final script. The hope being that you could read this document and have a solid idea about the type and scale of content that you need to build. The nearest thing to a script that you might find is the ‘Design Doc’ – however this often pales in comparison to a film script because its almost impossible to convey ‘gameplay’ with words.

Frequently the most insightful game designing is done when you actually play or see somebody playing the game. What you realize at this point is unlike film, which ‘displays’ your content, in a game somebody is essentially ‘playing’ your content. Consequently major changes can happen when the game designer suddenly realizes that either something is just not working as designed or a new gameplay element is needed to make a level fun or complete. So generally ‘game scripts’ are hard to find and don’t always tell you what you need to know.

As vague as this sounds, gameplay can also add a new dimension to the role of an AD or CGS coming from film. Where films are very ridged in that the script is normally locked before entering production, the possibility of contributing to a game design is much greater during the process of creation. The design of a character, animation or environment can have a dramatic effect on the whole game. I have seen and experienced many cases where the solution to a games entertainment or technical design problem has been entirely solved by the art team redefining the content.

From experience the best way to find out quickly what it is that you are making is to have the Game Designer, Producer, Technical Director and Audio Director talk you through the game or a game similar to the one you might be working on. This should reduce the frustration that you feel when you can’t find the script.

3. Technical Design

Your art is shipping with the game!

In film all of the hard work in building 3D content is eventually rendered into a 2 dimensional image. In games all of the rendering is done in real-time, a compiled version of your source art ships with the game. This presents a whole set of it’s own challenges – the biggest one being creating content that can be rendered in less than 1/60th of a second (and fits in memory). Coming from film this is the most daunting thing to deal with. No longer can you; wait a bit longer for a render, or break something large into layers to be composited later or even paint some offending artifact out. Your art needs to be ‘final’ as it is delivered to the game. In-game solutions to visual problems are rarely possible unless the art is authored in the engine (cameras and shader attributes are commonly authored in context).

So once you realize that it all has to fit in memory, fit on the disk and you can’t fix it in post and you still want to work in this medium the next step is to get to know the technology.

Unlike the film and TV industry where the artists generally have hands on ownership of the content right through to the final image, creating game art entails converting the none-playable art through a pipeline to the game format. Currently in order to get this done the skills of an engineer are required – these are your new best friends – without them, getting things done is almost impossible. And being close to this process of translating source art to compiled game art is essential, as this is the place where most visual issues arise.

The rendering team is also a group that you should become very connected with – unlike film where you can choose from a selection of completed renderers – in games the renderer is constantly being developed to add new features or for optimization. Depending on the type of game you are making and on what platform the feature sets of any specific renderer can vary widely – common graphical differences to film are:

Surface Types: Although modelling in any technology is generally possible the most common technique for modelling and rendering is polygonal. The current crop of next generation platforms can render a huge amount of polys’ per frame – however it’s not normally the polycount that is the biggest physical limitation of the ‘resolution’ of an object. The shader complexity, weighting and texture assignment tends to have a dramatic effect on the composition of an asset.

Numbers of character bones In real-time rendering every single transformation costs valuable time, consequently the number of bones, deformers and weighting types is tightly restricted. It’s also not uncommon to have LOD skeletons and restrictions on the type of transformation allowed on the bones (scaling is frequently not allowed).

Shaders: Unlike most software renderers that allow complex creation of very procedural textures, real-time hardware shaders are still very limited in what they can do. Although different types of shaders are possible, such as pixel, vertex and geometry shaders, the challenge is to write very efficient shaders that do not hinder the frame rate or drawing methods. Well-constructed and efficient shaders can be the keys to great looking and playing games. Some of the multi-pass techniques use similar film layer compositing techniques.

Numbers and types of light source Although the next gen platforms are powerful you are still very limited in the amount and type of dynamic light sources that you can use in real-time. Most games use baked in lighting that is Vertex shaded, lightmaps or a combination of both. Any dynamic environmental lighting changes – such as time of day or sudden lighting effects – can present very difficult challenges.

Types of texture compression In order to get everything to fit in memory its very common that textures are compressed into a format that gives good size reduction and yet maintains fast access. The result can be quite lossy in both detail and colour accuracy. Unlike film where textures are rarely affected during the render process, you must pay attention to the final result in games as they are frequently degraded from the source quality.

Streaming data or not. It’s pretty much guaranteed that all of the art that you are making won’t fit into memory all at one time. Consequently you will find that it’s often broken up into convenient visual chunks that can be discreetly loaded when needed and when possible. Visible distance is often limited and as an Art Director you may be asked to re-compose environments to allow easier streaming.

Numbers of LOD’s In games where the is a great or rapid change in the proximity to an object render engines often use different level of detail (LOD) to switch out hi-res object for lower res to help balance the rendering load and maintain the frame rate. There is often a very visible pop as you transition from one LOD to the next.

This is just a small subset of the techniques used within the game renderers. Coming from film this list may seem like a very limiting set of features – however when you realize that all this renders in 1/60th of a second it is a massive achievement.

4. Viewing Content.

Why can’t I see anything?

In the early stages of game development is frequently hard to see what the game looks like. Often render features haven’t been implemented or asset loading is not complete. Sometimes what you see and the first impressions can be a bit scary – to get around this issue there’s a number of ways to emulate the game look.

Pre-Viz in DCC – using shaders written to emulate the in game renderer In absence of a working game to view art in context we often resort to viewing that art content inside the DCC (Maya, Max, XSI) that we are using to build the art. Although this provides some idea of the accuracy of the art it is no guarantee that the art will actually look the same once it’s exported and traveled through the pipeline. Naming conventions, tagging and resolution can all break the game if not correctly specified. Also the workstation hardware, and display color space is also very different to that of the target platform.

Pre-Viz on target using a Viewer As an intermediate step one of the techniques used is to create a viewer on target (this means one of the destination platforms). Frequently this is actually a cut down version of the game engine. The advantage of this over viewing in the DCC is this art has been processed through the pipeline and what you are seeing is normally close to what the final game art will look like on target. However supporting a viewer like this often ads extra burden on the engineering team – keeping the viewer in sync with the game code is always a challenge.

Review in game using debug tools. Reviewing art in the game is by far the most preferred method of viewing content – if you are lucky enough that your game and team can support this. In most cases this form of viewing offers debug states which allow you to switch on and off various drawing modes and objects to enable efficient assessment and debugging of the content. The great advantage of this is it’s pretty much WYSIWYG, most games now offer scripting within the game itself which will allow you to setup scenes such as character lineups and switching character dressing.

Future tools. Editing in context – some of the more advanced games being developed today allow real- time editing of parameters just as you would in a DCC – the most common of these is adjusting shader or lighting values. However in some cases development companies have created mini editors within the game itself – with future consoles this is probably going to become a more normal way of creating content.

5. Artistic Design

Restrictions, Restrictions, Restrictions……

This can be one of the most challenging transitions from Film to Games. Personally I’ve never Art Directed a film – however I have seen the loss in translation from film concept art to the final rendered piece. In the game industry this is even more of a challenge. Frequently the hardware limitations can seriously dampen the essence of a concept. Something that would have been possible in film – just might not be possible in games.

This presents a challenge for both Art Directors and game companies – all to often resulting in frustration on both sides. Teamwork is normally the way out of this – the combination of AD, CGS and Technical team can normally resolve any visual design issue. And as an Art Director working in this medium you have to accept that there are physical limitations and also unique opportunities that exist.

As an Art Director in games my first step in providing a visual solution is always to get to understand what is possible within the game that I’m helping build. Learning in depth what the technical possibilities are dramatically increases the chances that your design can be implemented, also knowing where the boundaries are helps you push them and work with the team that built them.

Frustrations aside, Art Directing in games can offer challenges that film can’t. The fact that your art is not baked out into a final render means that you can change it more frequently as the game is being developed. You can also design things to have a unique appearance every time the game is replayed (if you desire) and with the growth of the internet you can deliver different visual experiences once the game has already shipped.

Also as the game industry evolves there are new types of Art Directors emerging – these are the ones that have realized that visuals and gameplay work very well when designed together. They are creating a new visual language that assists both the visual and interactive experience.

One other factor to include is the development of the visual fidelity of CG within the two mediums.

Film CG developed rapidly and then slowed with the increased complexity and cost of the last 10% of photorealism.

Games CG fidelity has always been capped by technology and has developed in ‘generational’ stages. Each stage is marked by slow initial periods and rapid increase in quality towards the end of the generation cycle. The slow start is caused by having to work with new and relatively unknown and unstable hardware. The rapid increase towards the end is enabled by known technology and stable hardware.

6. So is game development for you?

The gaming industry is not for everybody – especially if you are looking for everything that film is. The games industry is not low fidelity real-time film, its different in almost everyway. However the visual goals of high aesthetic quality remain the same as they would for any visual medium.

To feel comfortable working in the games industry it takes somebody that is ready to adapt to the new challenges. As the games industry and technology continue to evolve the processes and workflows will adapt accordingly.

Film gives people an alternate view on how to build high fidelity art, and this knowledge is valuable in games. However you need to bring the knowledge and adapt it to be relevant to games.

The non-linear interactive real-time nature of games is a group of dynamics that film just doesn't have. Those dynamics affect every aspect of production and permeate everyone's job within the industry. The frequent technology changes also have a dramatic effect on everything that is done - there are generational leaps in computing power every 5 years that cause us to rethink all aspects of production. Many of the people that I talked to about this paper remarked that this is one of the biggest reasons to work in the game industry – it’s so dynamic. The other noticeable factor is that the average age of people working in games is maturing and the influx of people from film is helping make the pipelines and processes feel more stable.

7. If you make the leap can you return to film?

Moving to the games industry is no longer a one way transition – the aesthetic quality of game art and the techniques used are definitely comparable to film. In some respects many of the game techniques can be used to positive effect in the film industry. It takes considerable skill to create quality art that renders in 1/60th of a second and that knowledge can be very effectively applied to film.

XSI is a trademark of Softimage Inc. Maya and 3ds Max are registered trademarks of Autodesk Inc.

Transitioning to Real Time Character Pipelines

Paul Amer Character Supervisor, Microsoft Game Studios [email protected] | [email protected]

This section of the course will focus on the processes we used for developing human base characters for an action oriented real time game. As a foundation for this section, I will focus on the processes we employed during the production of Shadowrun, a first person shooter which was the launch title for Microsoft’s first cross platform gaming initiative (allows players to play Xbox 360 v their friends on Games For Windows Live and vice versa) Live Anywhere. With the requirements of developing a game which runs natively at Hi Def (1920 x 1080), it was essential to marry my experience from feature and long form television projects with the needs of a real time environment.

The Needs of a Real Time Character

The difference in developing a character setup for an interactive medium, as opposed to linear visual mediums, is quite stark. Viewers have become sophisticated and as such have high expectation as to the production values, believability and credibility of the characters. However, this is where things become complex. Take the example of a character walking quickly down a hall, then deciding to change his direction and walk back the way he came. In traditional animation, that would require anticipation, thinking, motivation, the actual change of direction, probably some tilting in his body to indicate physics, and then some recovery. Perhaps as many as 2‐3 seconds for a quick movement.

In most types of games, the user is playing the game because they want control. Control of the hero specifically. And they want control to be immediate and flawless, for control to provide gratification and feedback in visual terms, but immediately in terms of “feel” or “gameplay”. When the player wants the hero to turn around, they demand their character change direction at the touch of a button, not fall over, keep his aim and of course, not waste time on such luxuries as hesitation or anticipation. In games, this is expected to happen in a second and in twitch type games (shooters, sports games, action adventure) it is expected to happen in 4‐8 frames.

As for the models and textures consider this. In games, characters are not controlled by a director but by a player, and where they are not viewed by a single camera @ 24 FPS, but by 16 cameras @ at least 30f fps** (players on the PC can over clock their machines allowing the game to run at speeds at greater than 30FPS; regardless, the character s along with their dependant motion must not fall apart as a result). This means that arms and legs move incredibly fast stressing hips, knees, wrists and elbows. If a rig or model fails, we can’t fix it in post because in a game as complex as Shadowrun, there are 16 cameras that at any given time are viewing the character at anywhere up to extreme close‐up distances.

Super high resolution geometry and its influence on what is seen in game

Once we have the character concepts / designs, where do we begin?

Our starting point is at the highest quality available, with the intent to squeeze as much into our given space as possible, whether it be screen, memory or both. In film that memory budget may be defined by the vital statistics of boxes in the render farm, in games the memory issue is focused on the ability of the CPU to gather the data, interpolate it, calculate the physics of the behavior and then the surface characteristics in1/30 of a second or less. Now that HDTV resolutions are available to console users and greater resolutions than are typical for feature film are available on PC monitors we must try to create and imply as much detail as possible in our end product. This is why we start with the most detailed source possible.

In order to achieve our high resolution source models we used two established methods. This was necessary because we did not have available a large enough team of character modeling and character texturing artists. The first was to model the character from scratch using the concepts as a guide and a suite of digital tools (including Maya, XSI, ZBrush) to achieve the most detailed original possible and the second was to sculpt in clay a relatively large maquette which was then scanned and digitally shrink wrapped (using CySlice and UVLayout). The resulting model was then finessed and ZBrushed where necessary. In order to give a frame of reference these models often contained anywhere up to 11 million polygons. We consistently found that modeling at such high resolutions gave far better results than relying too heavily on products like ZBrush. The quality difference between baked and painted normal maps was too significant to ignore.

Scan (1 micron scan, 8 million polygons)

Wireframe

Shaded

Creating Textures that work in with next‐generation hardware (normal, parallax)

Using textures instead of geometry

If screen resolution is important, why make something so complex? Well in both domains the answer is the same. We want to guarantee we have the best starting point in order to ensure our destination medium is not limited. We literally may have to, not only, deliver our characters in game with associated levels of detail and for in game real‐time or pre processed cinematics, but also trailers, commercials, print advertising and future versions of the game with different performance metrics. Given this high quality original we can also create all the geometry resolutions for use in real time, in a way that is perfect as the target for normal and parallax mapped textures, which are baked down from the high resolution source model. This is the same baking technique which is often used to generate bump and displacement maps. Our intention is always to imply the greatest level of visual sophistication with the optimum balance of elements to ensure fluid interaction with our realm of suspended disbelief.

Game viewer (color only, minus character and scene lighting)

Creating hi quality characters with limited bones and weights

Building rigs that pass data in game engines

Modularity is the key here. In both disciplines it’s wise to create skeletons and control systems in a modular fashion, not only for maintenance and manageability reasons, but also for the efficient creation and processing of the final output.

For us this breaks down to:

Control System <‐> Guide Skeleton <‐> Deformation Skeleton <‐> Geometry

The Control System is the place where all our high level manipulation and control takes place and where the handles are available to the Animators. This is the place where hooks are provided into the systems of automation embedded within the Guide Skeleton.

The Guide Skeleton is not only the home of the various levels of automation (eg IK and FK systems), but is also the enabler of structural priority or emphasis, often by simple hierarchy alone. I often think of this Guide Skeleton as the lead of the dance partnership that exists with the Deformation Skeleton.

The Deformation Skeleton is the element, which in this case, is almost entirely responsible for manipulating and quite literally driving our character through the environments. It is the following partner in this digital waltz. Where this skeleton differs from the norm in film is that it contains various attributes and elements which are specific to games and are needed for the engine to quickly calculate the characters relationship to the world and events occurring in it. Sites for example are primitive bounding volumes which are essential when calculating collisions with other characters and the world in addition to detecting hits from other player’s weapons. The key vital statistic for, to some extent, the Guide Skeleton and to a much greater extent the Deformation Skeleton is the number of joints/bones. Because the transformation and rotational data from these joints multiplies the size of the motion library so quickly and thus can place a huge burden on the machine it is running on, given that in our case it is possible for up to 32 characters to be seen in camera at one time with a myriad of effects being triggered in a relatively complex environment with complex shader, lighting and camera effects, economy is important in every aspect the game. However it is a worthwhile exercise to examine some motion capture samples that can be found on the internet (http://mocap.cs.cmu.edu/search.php?subjectnumber=%&motion=%) to really start to understand where bones are well invested.

And lastly the Geometry, which is bound to the Deformation Skeleton, is quite literally along for the ride. The bind in our case has to incur as few calculations per vertex as possible. This limiting factor forces hard limits on the number of bones which can be considered when transforming each vertex. For Shadowrun this limit was three bones per vertex. As a consequence we endeavored to create our binds in a relatively relaxed A pose and then to optimize the weighting to make the most pleasing forms during the most common poses and movements.

It is the Deformation Skeleton, the Geometry and the Animation sampled on the joints of the Deformation Skeleton which is ultimately exported in a suitable format for the game engine.

Specific to game we also take a duplicate of our Deformation Skeleton and dress it with custom constraints, joint limits and volumes. This is our Havok Skeleton, This skeleton is utilized by this third party package and our custom code to manage the character during rigid body death animations that must interact with the world and it’s contents.

Building a convincing character face with only 6 blend shapes

So from a character stand point we are focused on numbers of polygons, numbers of pixels and compressed size of textures, shaders and their complexity, numbers of bones and the quality and quantity of joint data. The complexity of deformations is not typically an issue on a film project unless they become cumbersome for the animators who are trying to work in as real time as possible. However for games this is a concern especially when there are multiple

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 34 of 94 characters interacting in a cross platform and networked scenario. This is specifically because for everyone that moves, its transformation must be communicated through a server to all players and the transformation must be recalculated on the local player’s machine. All of this must be done in a near instantaneous timeframe to prevent any one user from having an advantage over another player. So while it may be artistically more desirable to have a more complex face, its similar to traditional cell animation from the perspective that its more important the key emotion is clearly expressed to convey the overall emotion, as opposed to have a very complex emotion that isn’t seen by a player because the machine is forced to drop frames.

In this situation skinning can be the only deformation available and as we saw earlier it is often limited to the number of bones which can influence any given vertex. This is done in order to maintain performance (both locally and networked). Given the combat focused nature of the project and the design specification that characters should be seen no closer than torso up mid shot, we decided, for Shadowrun, a limited form of blend‐shapes was our only deformation luxury. We utilized this for some of the Troll “hardening” effects and also developed a system of six expressive shapes in combination with a single jointed jaw to form, given the constraints, a relatively sophisticated facial animation system. This system can be driven by event triggered animations and player dialogue input via the headset microphone.

Tribal Troll in damage hardened state

These shapes are:

• Blink • Squint • Squeeze • Brows • Corner Up Right (Mouth) • Corner Up Left (Mouth)

Considering we could manipulate the jaw and overdrive as well as drive negatively the blendshapes, a remarkable range of facial movement was possible.

Animating move segments (game) v linear shots (film/feature animation)

From the perspective of Shadowrun and the parallels to film we are in very similar territory to that of digital extras and crowds in a feature production. We worked with camera distance driven LODs, relatively standardized humanoid biped rigs which were more or less identical for all characters and proportionately structured to allow for motion sharing.

In film the performance is governed by the camera and to some extent random events which a predefined to conform to a directed story segment. A sequence of still images is the result. In game we are totally at the mercy of every unique player who will expect an instant feedback immersive experience that is responsive to every twitch and jolt. This occurring in tandem with a

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 36 of 94 completely mobile camera which may never move through the same space in the same way twice.

Our method for creating this animation would be almost identical to that of creating a library of motion for a crowd system and could as easily be defined by motion capture, character animation or for that matter some blended combination. This is significantly more complex than animating a given shot for a hero creature or character in a feature production. The key is deciding on the base or root pose from which the motion tree is grown.

It is this form of animation creation that Character Animators find the most challenging, given that every clip must blend naturally in such a few frames to every other. This is a distinctly different discipline to animating to camera in a scanned plate or in an environment where the layout team and Director of Photography have already made key decisions and defined the cameras behavior.

Working with the motion library in Fasa’s proprietary tool, Pretty

Composing motion for a game engine

On Shadowrun we created cycles of motion or clips that could be called on at any moment in game by the player, very much in the same way crowd driven agents are triggered to respond to terrain, environment and distance defined events. Each character had a library of approximately 460 moves which could be smoothly blended and overlaid in order to create fluid natural motion. This work was done by a team of 2 Character Scripters, who were essentially the motion compositors for the Character team. In this way character motion could be refined and transitions created in real‐time in order to create the most natural player triggered motion possible. This becomes even more fascinating when we consider that not only is the player driving the character seen by other players (the third person character), but also the arms of the chosen character seen grasping the weapon (the first person character). Therefore two different forms of motion per character are being played simultaneously at all times, and both have to be scripted to ensure first and third person player feedback feels as natural as possible at all times.

Editing transitions in Pretty

Conclusions

As we’ve seen today there are many parallels between film and games characters, but that of course is dependent on the content of the final product. Most importantly skills can transpose well given the often common tool sets and approach. The fundamentals don’t change as every project brings it’s own puzzles to solve and it’s own creative challenges to play with.

For me they both offer great payoffs in the final product. A film offers that amazing quality and sense of achievement given the levels of abstraction we have operated on and pushed through to realize the final polished frames. Our limitations in creating a game are their own challenge and now more than ever we are closing in on a quality bar that removes visual artifacts of the medium and offers the opportunity to really interact with the beings we have digitally formed.

Crossing the Line: Moving From Film to Games…. (and Possibly Back)

Brien Goodrich Lighting Supervisor / Associate Art Director Microsoft Game Studios – FASA Studios

Crossing the Line: Moving From Film to Games (and Possibly Back)

Brien Goodrich Lighting Supervisor / Assoc. Art Director, FASA Studios / Microsoft Game Studios

1. Fundamental Changes 2. New Budgets and Limitations 3. Finding (and Creating) Common Ground 4. Returning

Fundamental Changes - Please Surrender the Controller

• Camera Control – Concrete expectation of Film Lighting – The consumer as the Cinematographer • New Challenges for the Lighter – Nothing is “out of frame” or “off camera” – Dealing with, and understanding the entire game space

Fundamental Changes – Please Surrender the Controller

Camera Control

When Lighting a shot in film production, one of the most fundamental elements we take for granted is camera position. By the time a shot passes all the way down the pipeline to Lighting, camera position is essentially concrete. When we enter into Lighting for games however, we take this fundamental expectation and literally, throw it out the window. The cinematographer is no longer a seasoned, trained professional of the film making world, but rather the consumer – an individual seated in front of a monitor, mouse or controller in hand. Now depending on what game you’re making, this issue can be more or less significant. In a multiplayer shooter like SHADOWRUN where you can virtually go anywhere in the level, this problem is considerable – the user can essentially put the camera wherever they choose.

What exactly does this mean to the lighter? First, nothing is outside of frame or “off camera.” You must deal with the entire game play space. Some areas of the map might be less traveled and thus we can make choices about which areas to focus our time and effort. However, because we’re dealing with the user’s positional camera choices we’ve got to essentially light everything.

Fundamental Changes - Please Surrender the Controller • Establishing direction and reference – The Key – Secondary Keys – Use of Shadow – Achieving Lighting Continuity

Now lighting everything isn’t to say that we need to over light everything, just like in a film shot, the absence of light can be as important as the presence of light – it defines form and creates mood. This points to one of the very first things we discovered when we began lighting real time environments - a sense of direction is a significant issue. While map design and texture work might play a major role in this, lighting can either greatly enhance or subtract from this experience. Thus, establishing a sense of directionality and point of reference through lighting was something we found extremely useful and

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 41 of 94 important. Essentially, we achieved this by generating a strong key source such as a moon or a sun, which would cast reasonably believable shadows. I use the term reasonably believable because shadow and lightmap resolution budgets play a large role in real time lighting. We’ll discuss more budget limitations later on, but it should be noted that you can expect to make some hard choices about where, when and how to spend this resolution. This points back to the issue of understanding the important areas of the game play environment and determining where you get the most visual bang for your buck. In general we would tend to double the lighting resolutions in high traffic areas, and cut back resolutions in low traffic or extremely small areas. Additionally, we would also create secondary key sources in some of the enclosed spaces to serve as additional reference points when the primary key wouldn’t always be visible. When comparing this notion to film work, think of it as trying to establish continuity from shot to shot. If the lighting from one shot isn’t reasonably matched to a shot in the same sequence, the viewer can become disconnected or confused. The same is true for a real time environment.

Fundamental Changes - Achieving Continuity

Fundamental Changes - Please Surrender the Controller

• The Problem of Framing (a shot) – Again, passing this responsibility to consumer • The Influence of Film – Mark Romanek’s Criminal Video – Giving the frame Focus and Depth

Last, how do we deal with framing a shot when we can’t control the camera? Now for someone lighting a shot at the end of a film pipeline, the issue of framing has most likely, already been resolved (although still perhaps a consideration). However, for an animator or someone doing layout, this is a critical part of their craft. Yet in gaming, this task has again been passed on to the end user. Therefore, we came up with a subtle way of lending some assistance. Somewhere in the process of revamping our post processing system, the Lighting team did a few days of immersive research. Translation: we watched a lot of movies and music videos and played some other games. When going through some director reels, we came across Mark Romanek’s video for Fiona Apple’s Criminal. In that piece, he uses what he calls a “security camera” effect. Essentially, it is a vignette effect which darkens the outer peripheral of the frame. We borrowed this notion and added blur to the edge of the frame as well. The result created an immediate sense of focus to wherever the player looked – it essentially helped frame the shot. Additionally, this also gave the image a greater perceived sense of depth acting as a depth of field cheat of sorts.

- Vignette edge effect OFF

- Vignette edge effect ON

New Budgets and Limitations

• Level Lighting – Portals and their impact • Sectors vs. Shots – More continuity issues – Restrictions

Dealing with New Budgets and Limitations

Level Lighting

Any work in computer graphics, be it Film or Gaming, comes with a set of limitations and budgets (computational limits, memory footprint, access or load times, number of lights, shadow resolution, etc). In this section, we’ll look at some of the specific budget issues relating to game level lighting and contrast them to shot lighting for film.

What is a portal? Wikipedia says “A Portal System is based on using the partitioning of space to form generalizations about the visibility of objects within those spaces. Regions of map space are divided into polygonal, generally convex, areas called Sectors. Adjacent Sectors are linked to one another via shared dividing polygons termed Portals.” Got that? The important thing to grasp here is that sectors are physical spaces within the game level which are separated by portals. Further, depending on the sizes, numbers and locations of the sectors involved, lighting will be impacted.

In film terms, try for a moment to think of a sector as a shot. Sectors can be adjacent to one another and like the lighting in adjacent film shots, they usually need to have some type of continuity in lighting. If done poorly, it can result in seams between sectors. Just like a film, this can cause a break in emersion and generate a visual confusion for the viewer – think bad film editing. Additionally, within each film shot there commonly exists a set of lighting and or rendering restrictions. For example, this may be based on rendering time constraints, or output resolutions. Although shot restrictions may change from sequence to sequence, they are a significant part of film lighting production and are never far from a good lighter’s mind. In lighting for games, a sector will also usually carry a set of restrictions. However, the set of restrictions becomes a little more rigid.

New Budgets and Limitations

• Attenuated / Dynamic Lights – Seeing spec and normal – How many can we afford?

Because we’re dealing with real time calculations, these restrictions are most fundamentally illustrated by both the number and types of lights available to the lighter. At the forefront of this notion is what we call Attenuated Lights or Dynamic Lights. These lights allow the viewer to appreciate the spec qualities of the shaders as well as the normal map (similar to bump mapping). If a surface can’t “see” one of these Attenuated lights, it will appear flat and may fall apart very quickly. To complicate matters, our game design requirements and the fact that we’re multiplayer meant that we could generally only afford 3 Attenuated lights per sector. Therefore, placement and efficiency of these lights is absolutely critical.

New Budgets and Limitations - Attenuated / Dynamic Lights

New Budgets and Limitations

• Negotiation vs. “you’re out” – Film Render budgets – Sector size and shape

For a very large or awkwardly shaped sector this can be a significant hurtle – and in some cases, a deal breaker. For example, in film lighting you may really need to raytrace a set of shadows to sell a CG character’s interaction or proximity. If raytraced shadows weren’t part of that character’s original render budget (meaning that everything was designed to be done with cheaper depthmap shadows), you might have to negotiate the importance and expense associated with using raytrace shadows. Likewise, I found myself evaluating sector sizes and shapes with the Dev and Design teams early on. It was critical to determine if the lighting team could effectively meet the visual lighting objectives in the designed spaces under the real time restrictions. In extreme cases, if a

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 49 of 94 sector was too large or had an awkward layout, we would try to negotiate with the Dev and/or Design groups for alternatives, which sometimes meant adding an additional sector.

New Budgets and Limitations - Negotiation

Finding (and Creating) Common Ground

• Character Lighting – Plate Integration – Suspension of disbelief • HDRI Capture vs. PRT Solution – Solving similar problems

Finding (and Creating) Common Ground

Character Lighting

As I mentioned earlier, there are some significant and fundamental changes when going from Film Lighting to Game Lighting. In this section, we’ll look at some of the specific issues relating to lighting characters in games - some issues here are surprisingly similar to film.

In a CG character heavy film production, integration of that character into the live-action plate is absolutely critical. When done well, the storytelling process is seamless. When executed poorly, a break in the suspension of disbelief occurs and it becomes clear that what you are watching is fake - it’s really the worst thing a visual effect can do. How then do we integrate a character into a shot? How do we make something which is absolutely fake, look real? One part of this process might be to capture the environment through a spherical HDRI capture and then use that image as a basis to light the character.

In Game Lighting, some very similar notions (at least in principal) are being implemented. The mention of the HDRI capture above tries to determine how a CG character would be illuminated at a particular point in space. We can try to accomplish the same thing by using PRT or Precomputed Radiance Transfer for real time games. Wikipedia defines PRT as “a technique used to render a scene in real time with complex light interactions precomputed.” Now take away the precomputed element and the problems are virtually the same - we’re simply trying to capture the existing environment and push it back onto the character.

Finding (and Creating) Common Ground

• Application to Games – Traveling character lighting rigs – Camera based rim and fill properties – Limitations

Yes, base level, character and environment integration was absolutely desirable for the project. However, given SHADOWRUN’s game play, we quickly saw the need to pop the characters off the screen. These sound like conflicting goals. However, if you think about this in film terms, it starts to make more sense. In film, a cinematographer or CG lighter will often use a rim light to separate the character from the background. Sometimes the rim light is motivated, sometimes it’s not. The end result however is that is the character, CG or otherwise is both integrated and slightly separated from the background. We were looking for the same solution in SHADOWRUN so we employed a very similar tactic. Each character has traveling lighting rig with a rim light and a fill light. At close range, we used a subtle fill and a stronger rim to define the character’s contour and form. However at distance, the rim proved to be noisy and distracting yet the fill seemed to get lost. Therefore, based on the character’s distance to camera, the rim light fades on or off and the fill light becomes more or less visible.

When developing this process, we did however run into some very different constraints than we might see in film work. For example, we struggled with the fact that the rim light simply could not afford to carry any shadow casting properties. Obviously, this created some issues with the apparent lack of ambient occlusion in certain areas of the characters at close proximities to camera. Given the extremely frenetic pace of SHADOWRUN’s game play however, it was a limitation we felt we could to live with. The inspiration and use of traditional hero-type rim lighting was absolutely derived from our film experiences. The implementation however was driven again by the fundamental changes discussed earlier – the presence of new budgets and limitations and the fact that the user now controls the camera.

Finding (and Creating) Common Ground - Close Range Implementation of Rim Lighting

Finding (and Creating) Common Ground - Distance Based Fill Lighting Tests

Finding (and Creating) Common Ground - Distance Based Implementation of Fill Lighting

If You Make the Move, Can You Return?

• Production is production… is production – Fundamental differences, but surprising similarities – Biggest hurtle of getting film work is film experience (and project needs). – High end games work is high end visual problem solving… and that’s applicable anywhere.

If You Make the Move, Can You Return?

Production is production is production

I hope that this document has illustrated that although there are some absolute fundamental differences between Film Lighting and Game Lighting, there are also considerable similarities. Some people would argue that the biggest hurtle in getting a film job is having prior film experience. Assuming you already have that, transitioning

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 57 of 94 back to film shouldn’t be too great an issue. I can say from personal experience, that while working in games, I’ve still had film recruiters contact me about my availability. Like anything else, it depends on a project’s needs at a particular time.

Some very interesting and technically advanced work is going on in the gaming world. Although perhaps answering to a different set of limitations, interactive graphics work is high-end visual problem solving that can be on par with any work being done for a major VFX film. If you’re up for a new challenge, it can be great adventure.

Identifying Friend from Foe

Tom Holmes Shadowrun Graphics Lead FASA Studio

In “Shadowrun” you have to be smart, making split second decisions as you play. One of those decisions is, “I see somebody, should I shoot them?” In “Shadowrun” that decision is even harder since there are lots of things you can do to the enemy besides shooting them. You might want to gust them, or stick an elemental on them, or a variety of other things, depending on the situation in the game. But before you can make those decisions you need to tell if they are your enemy. Ideally within about a second of spotting somebody we want you to be able to identify your friend from your foes.

Traditionally there have been many methods used in shooters to help with this. Red vs. Blue is a classic and effective way to accomplish this. However, we didn’t want to just blanket color the teams. Putting floating team icons above players is a sure, if obtrusive way, as well. Another method is to make one team’s silhouette drastically different from the others. Silhouette would have been great if we were interested in fixing certain races on the teams, like dwarves and trolls versus elves and humans. But both sides can play all the races; a player needs to be able to tell if that troll is on his team or not.

Since all four races can be on both teams, the character design was very important, and made team identification a big focus. The RNA Global troll looks very different from the Lineage troll. When a troll comes around the corner and has that clean cut corporate look to him, you know he is on the RNA team, and you can react accordingly. But realistically this only really helps to a certain point. When that troll is far away from you, maybe he is only ten pixels tall on the screen, character design isn’t going to be helping too much. There needs to be more. Complementing the character design is the palettes of the levels themselves. Care was taken to avoid level color choices which would hinder distant character identification. The last thing we wanted was levels where that troll in the distance was the same exact color palette as the wall behind him. In the distance he could hide in plain sight! But we also didn’t want to limit ourselves so much that we couldn’t make great looking levels. Therefore, the levels are designed to not hinder distant character identification too much, but also can’t be relied on to push the character out either.

Another huge component of how characters look is lighting. I happen to think that the lighting in “Shadowrun” is top notch – level lighting has a complexity to it that you don’t see often in computer games. Many times unified lighting, where all items in a game engine are lit the same way, is considered a

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 59 of 94 universally desired engine feature. In “Shadowrun” we don’t want unified lighting. We want the lighting on the characters to make them fit in, but also stand out.

That may sound like a contradiction, but it isn’t. What this means is that we want the general ambient environment to be on the character. If you are in a red location you should be, well, red. But we also want you to pop off the screen. To accomplish this every character carries a lighting rig around with him, which adds lighting effects like a rim light and a complementary fill light. It’s not dissimilar to shining a light on actor’s faces in a movie – you don’t care if those lights are unmotivated. In a movie you want to see the actors; in “Shadowrun” you want to see the characters.

Up close these lighting effects make the characters look great. The detail really shines through. However, in the distance they aren’t as effective; Once again when you only have ten pixels you don’t have details. But this also plays to our advantage. If we go over the top with our character lighting effects up close we can easily destroy the character look, flatten it and wash it out. But in the distance detail isn’t a concern, and we can go as over the top as we want. So the last piece is lighting. As characters move into the distance we smoothly adjust the lighting rig to highlight the characters more and more. This really helps give them the extra contrast we are looking for to pop them off the screen.

With all these components we feel that players will be able to tell friend from foe, near and far, pretty fast. Give or take. But one really nice thing about “Shadowrun” is the options that it gives you to make your style of play your own. If you find that you still need help (or want that extra advantage), the game can help you out. Perhaps you tend to shoot that guy who rounds the corner on you, even when he is your friend. Then Smartlink is what you want – it will prevent you from shooting your friends. If picking out enemies in a crowd during a big firefight causes you trouble, Enhanced Vision will help that out. It will clearly call out your friends and enemies, even through walls. There are tradeoffs to using these technologies; however, as they both inform your enemies that you are around.

Identifying friends from foes is important in “Shadowrun.” We’ve set-up the system to help out a good bit, and you can smartly use the technologies in game for an extra boost as well. So get out there, smartly outfit yourself with weapons and abilities you like, and engage

Post Processing in Games Jamie Marshall Technical Artist Fasa Studio | Microsoft Game Studios [email protected] | [email protected]

Specific Post Processes:

Auto Exposure Auto exposure is the most common type of post processing in games. Basically it means adjusting the brightness of the frame based on how bright the on-screen content is. It is equivalent to changing the aperture of the lens. Auto exposure serves four purposes: 1. When using an HDR render target, some form of exposure mapping is required to convert the image to a displayable LDR image without simply clipping out the brighter content. 2. It allows light to vary significantly from one area to the next. (by adapting the camera to the light) This allows the cave to look black from the outside, while still allowing the player to see when inside the cave. 3. It exaggerates drastic changes in light value over time by lagging behind what is should be. This causes the feeling in the player that their eyes are adapting to the light. When first entering a dark area, it feels dark. After a while, the dark area feels correct, but looking back out at the previous area now feels bright. 4. It prepares the histogram for further manipulation. (I will explain this later)

Some games, even ones released very recently, simply take the brightest point on the screen and scale the frame to make that pixel close to full bright. This addresses purposes 1 and 2 nicely, but since it is instant and uncontrolled, it fails to address number 4, and at times feels very unnatural as the camera spins around and the exposure values change very quickly.

Slowing down the allowed movement of the aperture so that it does not adjust instantly will add purpose 3, but it still shackles the image. Imagine putting an auto exposure system from a cheap camcorder onto your cinematographer’s camera. Why would you do that? Sometimes you want a bit of blow out, or you just don’t want any very bright values.

A complex solution The problem we face is that we want creative control, but the major factor contributing to the exposure value (the camera facing) is out of our control. We could somehow embed clues in our scene as to what exposure should be based on camera position/facing, but this seems a daunting task, and it doesn’t deal with actions on screen, like special effects, that can have a major impact on exposure.

So we settled on a system with an extensive number of parameters. The aperture in affect has a range it’s allowed to move within. A maximum f-stop, and a minimum f-stop. There are also values for how fast the aperture can close and how fast it can open. These numbers are separate because we need the screen to darken much faster than we need it to brighten. (usually – imagine how bad it looks when 50% of the screen is brighter than white.)

In addition there is an auto-exposure gain value that scales the entire effect. If it’s set to zero, then there is no auto exposure. If it’s set to 1, then auto exposure behaves just as explained above. Any other value gives us some amount of auto exposure. This allows darker areas to still be at least somewhat darker than bright areas. Just not as much as they would with a fixed exposure.

What we should have done Something that never occurred to us when we were first working out these ideas was that auto-exposure has some serious research behind it in the film photography field. Only recently I got a fairly modern SLR film camera with serious auto-exposure systems and have realized that we may have benefited from this type of thinking. We could at least have at least used a center weighted system for calculating the histogram for analysis. We could even have used something like the multi-segments systems that better cameras have. They concentrate on giving good exposure to things that are in focus, assuming that those areas of the frame are more important. We could have done that using both the center (which is where the player is aiming their gun) and also nearby important events, which are sure to draw the players eye.

It is a problem in all fields, but I think especially games that there is a tendency to re-invent the wheel. We should strive to look for examples in the past that might be used to solve our problems, even if those areas are not directly applicable. Where parallels exist, there are lessons to be learned.

Histogram reshaping An additional feature related to auto exposure is called histogram reshaping. This is simply adjusting the grey point in the image towards “average”. The effect is the same as doing an auto-levels in Photoshop. It makes sure that exactly half of all pixels are darker than 50% and half are brighter.

On the surface this isn’t very useful. It becomes a good trick, however, when you implement it correctly. The target grey point isn’t fixed. Instead it is defined as being half of the currently target white point. Not the current used white point, but the target (remember that the aperture speed is limited by content). Now we can make the grey point move even slower than the white point.

That means that when we move out of the dark into the light, while we are force to quickly adjust our white point to avoid blowing out, we can allow the grey point to lag, and create a brighter overall frame. In effect, the brighter parts of our image adjust faster to the newer conditions than the dark parts of our image. And when moving from the light to the dark, the inverse is true. The darks quickly increase brightness, making things at least visible (though murky), but the brights stay subdued for a while until the virtual “retina” has caught up.

None of this is particularly realistic, nor based on any real behavior of a camera. It’s just good because it emphasizes these transitions and therefore makes the intensity of our lights seem higher and more “real”.

Technical consideration While finding the histogram can be difficult (especially on older graphics systems), once you have done that all the heavy lifting is over. The movement of the grey and white points is complex, but easily calculated. And applying the results to the frame is just a multiply and a gamma operation per pixel. As a general rule, any post process operation that just looks at a single input pixel, applies some (pre- calculated) math, and outputs a single pixel is going to be fairly cheap.

Bloom Bloom, glow, shader-glow, whatever you call it, it’s one of the oldest tricks in the compositing book. You take some part of the image (perhaps a thresholded sample of the brightest parts), you blur it, and you add it back on-top of the original frame. This has been done in games for a little while, and it can be very effective.

Base technique Technically any effect that requires blurring is going to be expensive. This is because each pixel needs to look at multiple input pixels. This requires multiple texture reads, which can add up in cost. In order to mitigate that cost, we do a draw pass where we down sample the main frame to a much smaller (sixteenth size) frame. At this time we apply a parameterized contrast operation. Later, when we are drawing the

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 62 of 94 final output frame, we use this bloom buffer as a source. This process very much mirrors the composting workflow.

Incandescent Surfaces In order to make things feel more like sources of light, we have the ability to tag certain surfaces with a texture that draws to the main render targets alpha channel (which is otherwise unused). This is a way to sneak extra information from our main render to our post processing system. We could instead do a full “glow pass” that would render only incandescent surfaces with whatever values we want, but that would be prohibitively expensive. Adding this extra alpha information is practically free, thought it’s a precious resource because you only get one alpha channel.

Later, when we are down sampling into the bloom buffer, we add extra value for whenver the alpha channel is brighter. This makes things that are incandescent glow more.

Operations The actual application of the blurred bloom buffer to the frame is complex. We experimented with many different combinations before settling on a screen operation. This is close to additive, but it makes things less likely to “blow out”. Also, we found it was important to apply bloom before auto-exposure. Otherwise, when we are looking at a bright window from inside (when our aperture was being stopped down) our bloom was also being stopped down, and become almost invisible. And this is a case where we most want bloom.

Color timing Color timing is the most severe and obvious change we made to the frame, but it’s really the simplest as well. Controls are for Pedestal, Gamma, Gain, and Saturation. Each of the controls save saturation is a full color control, so it is possible, for example, to increase the gamma for just the red.

Order of operation is very important. You really want to apply gain after saturation, for example, so that you can remove some color and then replace it with false color. This is how we did the “you are getting shot and the world is turning red” affect. If we only did gain without the desaturation, then red things would get brighter, and blue and green objects would not turn red, they would just get dark. The only thing interesting to note about these controls is that having an HDR source frame makes them much more useful. We really can increase gain significantly without banding in the low ranges. Also, don’t rule out crazy numbers that stretch the math a bit. Negative gain numbers do work in our system. So if you use gain -1 and pedestal +1, you get a true negative image. Using negative saturation inverts all the hues but leaves values alone.

Obviously these kinds of affects are not useful for most things, but when the game has magic and weird tech events they can become useful. When the player teleports in Shadowrun, the saturation inversion happens. It’s very disconcerting, without the cliché of the flashed white frame.

Vingetting Late in development it was suggested that we try vingetting the frame by darkening and or blurring the edges of the screen in a circular pattern. This was intended to help add depth to the frame. Darkening was a simple task and added very little cost to the post process. In fact calculating the circular mask was probably more expensive than the darkening.

The blur, on the other hand, was much trickier. There was no way we could afford an additional blur operation. By the time we got to drawing the final frame, the bloom buffer was nice and blurry, but it also had had it’s contrast turned up. We tried fading into the bloom buffer onto the edges, but it added too much contrast when we wanted less contrast.

So the trick was to piggyback onto the existing blur operation that was happening for the bloom buffer. When we were down sampling the main frame into the buffer to be blurred, we also tucked an untouched black and white version of the frame into the alpha channel. Because pixel shaders operate on the whole RGBA color simultaneously, it didn’t add any cost except the extra memory to have the alpha channel.

Now when we loaded the bloom buffer, the alpha channel contained a black and white blurred version of our scene. By fading this in slightly on the edges, we created a slight out of focus effect. It was faded in so slight that the desaturation wasn’t really noticed, and may have even helped a bit. If somebody asks why the edges seem less colorful, we can say it’s chromatic aberration.

Hooks and interfaces

All of these processes and their associated parameters need to be hooked into the game somehow. The simplest method would just be to set them all once per game and be done with it. Alternately one could just have a set of parameters per level or mission. In Shadowrun we opted for a much more complex solution. The various techniques we implemented can be used to created very special case content, and the cost is the same either way. Anytime in games we can give the artist additional capabilities without impacting performance, it’s a win-win situation.

Settings per area Many post processes are used differently from place to place on a level. In addition, many techniques only work well in certain areas.

For example, in interior spaces with large contrast (say a huge dark warehouse with really bright but small lights) it may be desirable to have a glow around bright areas. Thus the spotlights seem even brighter and the sense of glare is increased. However, if you took those same settings to an outdoor scene you might have much more glow than you want. For these reasons it is a good idea to set parameters via some sort of volumetric area. We used conglomerations of oriented bounding boxes (OBBs) to define areas each which have a set of parameters.

As the player moves from area to area a blend happens between parameters in the exited area and parameters in the entered area. We found it necessary to allow the time of the blend to be adjustable differently for different areas. This was defined as part of the volumetric area where an parameter defined the default “blend into this area” time, and also had overrides to the affect of “if coming from a_specific_area then take x seconds”

Settings per state In some games (of which Shadowrun was one) it might be useful to be able to override this system based on some state of the player or the simulation. For example, if the player is using night vision goggles, we wouldn’t expect the bloom and such to be the same as without the goggles. So we added what we called “special tone states”. These states override all the parameters set for the area the player is in. Fading in and out is set per state. In effect it was as if the player had moved into a new area, the “goggle wearing” area. In a game that depended heavily on such display modes, it may actually be useful to instead define the “goggle wearing mode” per area in addition to the normal mode per area. Specifically we use this technique when the player dies.

Settings per events Commonly in games, there are events that happen to the player that we want to communicate to the player and over-emphasize. Simple techniques like flashing to a solid color for one frame can do quite a bit to make sure the player notices something important (Like they have just been shot). As discussed bellow, the

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 64 of 94 blending of different parameters together is incredibly cheap compared to the cost of applying all of the effects to the image. For that reason, we should leverage a post processing system as much as we can. On way to do that is to create additional sets of parameters for what we called “tone events”. These are simply lists of parameters identical to our “per area” settings, which are associated with specific game events.

They additional have duration and an “intensity curve”. The curve is used to control how the event fades in and out. It some cases it is desirable for the sim to control duration and intensity, and in others it is simply a triggered event that the artist will set duration and the curve. The significant difference with the event driven parameters is that they are applied as deltas to the other states. These settings don’t replace the area and state based effects, they just modify them. Thus when you are hit by a bullet we don’t have to decide what kind of fog we see.

In fact you can probably rule out many of the parameters from even being useful in an event, but if there is magic or other unreal elements in the game, don’t be too fast to limit the artists. I may assume that the artists never wants to make the fog roll in, but what about that “ice the sidewalk” spell?

Blending of data The key to being able to do all these types of post processing and still maintain a fixed cost is that we layer them all together into a single set of parameters before we get to the actual post processing of the frame (and the expensive shader operations). Regardless of how many states and events we are blending together, there is still a single set of parameters that get passed to the shaders at draw time.

This blending is purely math on a small number of values, so it resides on the CPU. The amount of code (and CPU time) required is pretty trivial, but we can say that adding addition events and states will have a per frame cost. This is importantly different than if we added a new post process, which would have a per pixel cost and would affect our GPU performance. If complex behavior and smooth blending is required, then it is key to use a smart method to blend parameters, and that all parameters can be blended linearly. For us, this meant adjusting how some parameters worked and putting intentional non-linearities into them. For example the user might set the fog distance as a linear value, but the game actually is using the square root of that number as the parameter. Then when the fog distance changes, it changes more rapidly at distance that up close.

An additional advantage to “linearizing” all of the parameters is that the controls tend to become more rational and intuitive to the user. If a value of 2 gives a certain effect, then a value of 4 will usually be twice that effect. I cannot stress enough how important this blending is. Implementing it early and iterating it until it is smooth is a key to success.

Actual data entry Up till now I’ve written about these various parameters in a pretty abstract manner. Obviously at some point the artist needs to be setting and adjusting the numbers. Fasa, like many game studios, uses a custom piece of software for creating and modifying data files. This allows us to create custom interfaces with common widgets for all parameters in our game engine.

Discussion of how that works is outside the scope of this document, but I would like to add a few quick notes: • The names of parameters can be very important. The ability for an artist to comprehend this fairly complex and abstracted system is limited when they have to figure out what each parameter name means. Embedded help can remedy this a little, but nothing is as good as looking at a the name and just knowing what it is. • Be consistent. If this parameter is called “bloom spread” then don’t call that one “blur radius”. I would argue radius is better because it is precise.

• Organize the parameters intentionally. Putting them in the order they will be most often used is good, but at least try to group them functionally. We didn’t do this, and it was a problem. • When using parameters that have RGB values, try to make the color they create actually mean something. For example, our tone-mapping gamma value was normalized around 0.5 instead of 1. (Meaning that when the user specified 0.5, this gamma had no affect. Entering 1.0 had the affect of gamma=2.). This allowed the pedestal, gamma and gain values to be viewed as three colors. Normally they were black, grey and white. If one of these values was different, it tended to mean that they blacks, grays or whites (respectively) were being pushed in that direction.

Crossing the Line: Moving from Film to Games (and Possibly Back)

Taking your Shader Skills from Film to Games

L. Cliff Brett CG Supervisor Microsoft Game Studios [email protected] | [email protected]

Are my shader skills transferable?

Yes!

HLSL (High Level Shader Language) syntax will look familiar to RenderMan shader language users. But not identical.

Writing HLSL shaders will exercise many of the same mental muscles. But some others as well…

First: you don’t control the camera. The player does. This has implications for all art production on a game, and shader development is no exception.

In film (like still photography), we don’t just control the camera. We use principles of composition (e.g., shape, size, color, etc.) to direct the viewer’s eye within the frame!

We don’t have that kind of control in games. The camera can go anywhere, looking at things from far away, close up, or anything in-between. Any shader you develop has to work under any game-possible circumstance.

All game artists have to juggle three balls…

Three Balls:

1. Perf. 2. Gameplay. 3. Aesthetics.

Perf = performance. Games are interactive. The game has to play at its target framerate. Players don’t want to feel a lag between the time they do something with their controllers and see that action take some effect in the game. Perf

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 68 of 94 always win in any showdown with aesthetics. It doesn’t matter how good something looks if it costs too much.

Three Balls:

1. Perf. 2. Gameplay. 3. Aesthetics.

Games are for playing. A visual that doesn’t convey the right information about something happening in the game isn’t doing its job. Something that looks great closeup when you’re examining it carefully might be utterly lost in the heat of game action. If that means the player can’t tell that he’s just been hurt, you’re going to have to go back to the drawing board. Game art serves gameplay, not the other way around.

Three Balls:

1. Perf. 2. Gameplay. 3. Aesthetics.

Aesthetics take third place. By all means make it look awesome. Everybody wants it to look awesome. It’s your job to make it look awesome. But never at the expense of perf or gameplay—those are your responsibilities, too.

The need to deal with perf in particular adds a layer of complexity to shader development in games.

What is Perf?

• Most FPS and racing games: 60 frames/sec • Shadowrun: 30 frames/sec

Perf = target framerate. Dev team targets this based on a variety of factors: type of game, platform, game engine, visual complexity, etc.

Q: What can you do in 33 msec?

A: Lots!

A framerate of 30fps means you have to redraw the framebuffer every 33 msec. That doesn’t seem like a lot of time, particularly for those of us who have waited hours for individual frames to render. But hardware rendering has come a long way. You can’t do everything—you’re constantly making compromises and looking for ways to make things go a little faster. But you can still do quite a bit.

Fairly complex environments.

BRDF implemented as a texture lookup ( u=eye vector; v=light vector). Made it easy for artists to quickly achieve different material properties. Also minimized expense. Mainly used for character clothing.

Subsurface scattering Implemented as a two-pass render (but not using HLSL’s “pass” mechanism). • Render uv-unwrapped face with lighting to an intermediate texture buffer. • Blur using a 7x7 Gaussian filter. • Apply resulting texture to character’s in-game facial geometry.

Parallax mapping. Lots of mileage out of simple gags like texture scrolling and distortion.

CPU vs. GPU

Game App Shaders

CPU GPU

You need to keep an eye on perf in two places: the CPU and the GPU.

The game application—the engine with all the simulation and AI and other stuff— lives on the CPU. It bundles up the stuff it wants to draw each frame and sends it to the GPU for processing. This bundling is the equivalent of emitting a RIB stream.

Shaders live on the GPU.

You have to pay attention to both: • How long the bundling takes on the CPU side, and • How long the shading and other processing happens on the GPU side.

Shader Taxonomy

RenderMan HLSL Surface Pixel Displacement Light Volume Imager Vertex Geometry

HLSL pixel shaders correspond to RenderMan surface shaders.

HLSL has no equivalent of the other RenderMan shader types. Functionality of lights imagers can be achieved in pixel shaders.

HLSL has vertex shaders. Equivalent RenderMan functionality happens in fixed pipeline.

DX10 adds geometry shaders to HLSL. Hardware still relatively rare and expensive.

On the GPU

Vertex Geometry Pixel Shader Processing Shader

Untransformed Transformed Interpolated Pixels Vertex Data Vertex Data Per-Pixel Data

This is the part of the GPU pipeline that deals with vertex and pixel shaders. The Geometry Processing step is literally a black box: fixed pipeline.

Interesting side-note: character skinning happens in the vertex shader rather than as a pre-render baking step.

The Hardware Jungle

There are lots of PC’s out there with lots of different graphics cards. There will be even more by the time your game comes out. Different cards support different features. HLSL functions that work well on the latest, greatest cards might not even exist on earlier cards.

Early in the project on a PC game, the team will determine its “min spec”—the minimum hardware configuration for running the game. But that’s still likely to leave you with several generations of hardware to support.

What do you do? And how can you manage this complexity? Develop for the lowest common denominator?

Shader Models

• Versioning system for hardware • By shader type; e.g., – vs_1_1, vs_2_0, …, vs_3_0 – ps_1_1, ps_2_0, …, ps_3_0

The HLSL compiler recognizes different hardware profiles called Shader Models. Each Shader Model corresponds to a particular set of hardware functionality, and provides a shorthand name for that functionality.

Xbox 360 is effectively vs_3_0 + ps_3_0 DX10 cards now with vs_4_0 + ps_4_0

But how do you use these profiles?

What is an Effect?

Effect

Technique …

Pass …

Vertex Shader Pixel Shader Effects States

HLSL shaders are part of what’s known as an effect. An effect organizes all the shaders and other code bits needed to render some 3D geometry.

An effect contains one (or more) techniques. As the name suggests, a technique describes a particular method for rendering something. You can have different techniques for different graphics cards.

Each technique contains one (or more) passes. HLSL supports multi-pass renders. A shader that does what it needs to do in a single pass on a relatively recent piece of hardware might need two passes on an older graphics card.

Each pass contains a vertex shader, a pixel shader, and (optionally) effects states that control various rendering options. You specify which shader model to use when you assign each shader to its enclosing pass.

Techniques and Passes (example)

// Vertex shader myVS1() and // pixel shader myPS1() defined above.

technique myTechnique_3_0 { pass myPass_3_0 { VertexShader = compile vs_3_0 myVS1(); PixelShader = compile ps_3_0 myPS1(); // Effects states go here… } }

// More techniques below this?

DirectX looks for a technique that’s valid to run on the current hardware.

Effects States

• Like RenderMan’s current graphics state • Control options for: – Rendering – Transforms – Lights – Textures/Samplers – Shaders – …

Effects States control graphics pipeline function. Closest analog in the RenderMan world is the current graphics state, which includes things like current color, opacity, shading, and other options. In RenderMan, these controls are issued through the RIB stream.

Your application can manage state here, too, but HLSL also lets you do this sort of thing in the effects file itself.

Render states (both Pixel and Vertex pipeline): • Transforms • Lights • Textures • Samplers • Shaders • Materials

Effects States (examples)

AlphaBlendEnable = TRUE;

ZEnable = TRUE; ZWriteEnable = TRUE;

CullMode = CCW;

This is just a tiny random sample. Consult the docs for more options and what they do.

Data Types

RenderMan HLSL float bool, int, half, float, double point, vector, normal, vector, booln, intn, color floatn, … matrix (4x4) matrix, boolnxm, intnxm, floatnxm, … string array (1D) array

No operators or functions take string arguments. They’re mainly used for annotations.

Arrays can hold any of the other data types.

Other Object Types

• Structures • Textures • Samplers • User-defined types • Shaders

Other Object Types: Structures

struct vertData { float3 pos; float3 norm; };

struct vertData myData = { { 0.0, 1.0, 2.0 }, { 1.0, 1.0, 1.0 } };

float3 tmpPos = myData.pos;

Other Object Types: Textures

texture myTex;

The texture declaration just gives you a pointer for your texture data, but you need a sampler to do anything with it.

Other Object Types: Samplers

texture myTex;

sampler mySampler;

sampler_state { Texture = (myTex); MipFilter = LINEAR; MagFilter = LINEAR; MinFilter = LINEAR; AddressU = WRAP; AddressV = CLAMP; BorderColor = 0x00000000; };

Samplers encompass both the texture data and the parameters you need to sample it (e.g., filtering, wrap mode). The sampler_state is another example of effects states at work. A little different from the RenderMan shading language’s way of doing things, which associates some sampling params with the texture file itself and lets you specify others as params to the texture access function.

Other Object Types: User Defined

typedef float FLOAT;

typedef vector VECTOR;

typedef matrix MATRIX;

typedef struct mystruct { float3 pos; float3 norm; } MYSTRUCT;

Other Object Types: Shaders

vertexshader myVS = asm { // Microcode assembly vert shader code. };

pixelshader myPS = compile vs_3_0 my_PS();

The vertexshader and pixelshader data types represent microcode assembly objects.

Operators

• Famililar but not identical; e.g., – dot() instead of . – cross() instead of ^ – mul() instead of * – Specific keywords/built-in functions. – Swizzling; e.g., float3 mine = {0.0, 1.0, 2.0, 3.0}; float3 yours = mine.yyyx;

Control Flow

RenderMan HLSL if-else if-else switch while while for for do break, continue break, continue return return

No big surprises here, either. HLSL adds switch statements and do loops.

Control Flow (cont’d)

• Be careful! – Static branching = ☺ – Dynamic branching =

Static branching and looping works off a Boolean constant. It switches blocks of code on/off at compile time. It’s fast—the unused code won’t be compiled into your shader.

Dynamic branching works off the value of some variable and happens at runtime. Can be slow on some graphics cards (particularly shader model 2) because both

branches will be evaluated. Dynamic looping isn’t even supported in shader model 2.

Preprocessor Directives

RenderMan HLSL #define, #undef #define, #undef #include #include #if, #ifdef, #ifndef #if, #ifdef, #ifndef #else #else, #elif #endif #endif #line #line #pragma #pragma #error

The RenderMan shading language and HLSL both recognize preprocessor directives. Differences are highlighted.

Built-in (Intrinsic) Functions

• Good selection • Beware: many not native on hardware – cross() – faceforward() – pow() – sin(), cos(), tan(), etc. – smoothstep() – …

User-defined Functions

• (Mostly) familiar syntax

float4 transformVert( float3 inPos : POSITION, float4x4 M ) : POSITION { return mul(inPos, M); }

The significant change here from what we’re all used to seeing in the RenderMan shader language is the inclusion of the POSITION semantics. These are optional.

Another difference (not shown): you can optionally specify a target which indicates the function’s target platform. Consult the docs.

Semantics

• Identify shader inputs/outputs • Tell HLSL how to bind the graphics data stream to particular variables • Tell the game app how to bind other relevant data

Vertex Shader Semantics

BINORMAL[n] BLENDINDICES[n] COLOR[n] BLENDWEIGHT[n] DEPTH[n] COLOR[n] FOG NORMAL[n] Vertex Shader POSITION POSITION[n] PSIZE PSIZE[n] TEXCOORD[n] TANGENT[n] . . . TESSFACTOR[n] TEXCOORD[n] . . .

The n is an optional integer between 0 and the supported number of each type.

The vertex shader must minimally output normalized screen-space POSITION data.but it can write other kinds of data as well. You typically bundle inputs into a structure that the shader takes as an argument. The shader returns another structure containing outputs.

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 86 of 94 struct VS_INPUT { float4 vertPos : POSITION; float3 vertNorm : NORMAL; float2 vertUV : TEXCOORD0; }; struct VS_OUTPUT { float4 vertPos : POSITION; float4 vertUV: TEXCOORD0; };

VS_OUTPUT myVertexShader( VS_INPUT vs_in ) { VS_OUTPUT vs_out; // Shader code here fills out vs_out. return vs_out; }

Pixel Shader Semantics

COLOR[n] DEPTH[n] FOG COLOR[n] Pixel Shader PSIZE DEPTH[n] TEXCOORD[n] . . .

Similar rules apply for the pixel shader. It returns color and/or depth. It can (but does not need to) use the same structure for input that the vertex shader used for output; e.g.,

// Assume same VS_OUTPUT definition as previous slide. float4 myPixelShader( VS_OUTPUT ps_input ) { float4 out_color;

// Shader code here determines out_color.

return out_color; }

Other Semantics

• Standard semantics for commonly useful things; e.g., – Time – Matrices – Bounding boxes/spheres – And more! • Add your own, too.

Semantics are just a string, so you can make your own. Your game app can be programmed to recognize and use these in addition to the standard semantics.

HLSL doesn’t support the notion of named coordinate or color spaces, but semantics allow an application to define and support these sorts of mechanisms as needed.

Annotations

• Strings • Not used by HLSL • Used (or ignored) by game/other apps

Applications can use annotations in a variety of ways; e.g., Display as help strings for user input Min/max values for user input Specify what kind of UI widget to associate with the param.

Some things to watch out for…

Watch out: Too many instructions

• Microcode instructions ~= cost • But they can still add up! • Beware intrinsics without native hardware support; e.g., – sin() -> 14 instructions under vs_1_1

The HLSL compiler tells you how many instruction slots your shader used.

On Shadowrun, we tried to keep environment pixel shaders to 150-180 instructions, although it could get up around 250 for a shader with BRDF and parallax. We tried to limit pixel shaders for effects to around 15 instructions.

Vertex shaders came in as follows: • Up to 100 instructions for characters. • 50-60 instructions for environments. • 20-30 instructions for effects.

But your mileage will vary.

Watch out: Texture access

• = ~20 other instructions • Actual mileage depends on factors you can’t control

Texture access is the exception to the rule that most microcode instructions take about the same amount of time. It can be really slow, but it depends on factors outside your control (like whether the texture data happens to be in memory when you call it, or does it need to swap in). We just assume it’s going to be slow and try to use it as little as possible.

Watch out: Overdraw

• Anytime you have to redraw pixels – Multi-pass renders – Blending • Severity affected by – Number of pixels – Number of layers – Shader complexity

This bit us repeatedly on Shadowrun. Redrawing pixels costs.

The severity of the hit is proportional to several factors: • Number of pixels. A bigger framebuffer suggests more pixels will need to be redrawn. The real estate covered within the framebuffer also matters. • Number of layers. You pay dearly for multiple layers of overdraw. Watch out for alpha-blended particle systems. • Shader complexity. The pixel shader runs every time you process the pixel.

Overdraw was the culprit when we found this version of the tree was too expensive. Lots of transparency in the canopy, but you still pay for redrawing the transparent pixels.

Analyzing Perf

• We use PIX

PIX = Performance Investigator for Xbox. It’s a Microsoft tool that comes with the DirectX SDK.

Captures detailed information from running game. Records every command sent to GPU. Tells what you’re paying for your pixel and vertex shaders; gives you

Siggraph 2007 Crossing the Line: Moving From Film to Games (and Possibly Back) © Microsoft Corporation – All Rights Reserved Page 92 of 94 texture bandwidth; lets you know if you’re getting stalls (e.g., waiting to fetch vertex data or read textures).

PC version doesn’t give quite as much information as the Xbox 360 version, but it’s still useful.

Conclusions

• Your shader knowledge is valuable. • You still have a lot to learn. • Games are fun!

Bibliography

Apodaca, Anthony and Gritz, Larry, Advanced RenderMan: Creating CGI for Motion Pictures. Morgan Kaufmann Publishers, 2000.

Block, Bruce, The Visual Story: Seeing the Structure of Film, TV, and New Media. Focal Press, 2001.

DirectX 9 Software Development Kit Update Document, Microsoft Corporation, October 2005.

Engel, Wolfgang, Programming Vertex and Pixel Shaders. Charles River Media, Inc., Hingham, MA, 2004.

Engel, Wolfgang (Editor), Shader X3: Advanced Rendering with DirectX and OpenGL. Charles River Media, Inc., Hingham, MA, 2005.

Engel, Wolfgang (Editor), Shader X4: Advanced Rendering Techniques. Charles River Media, Inc., Hingham, MA, 2006.

Fernando, Randima (Editor), GPU Gems: Programming Techniques, Tips, and Tricks for Real-Time Graphics. Addison-Wesley, 2004.

Kahrs, John (Organizer), Pixel Cinematography: A Lighting Approach for Computer Graphics. Siggraph ’96 Course #30.

Luna, Frank, Introduction to 3D Game Programming with DirectX 9.0c: A Shader Approach. Wordware Publishing, Plano, TX, 2006.

The RenderMan Interface Version 3.2.1, Pixar, November 2005.

St-Laurent, Sebastien, The Complete Effect and HLSL Guide. Paradoxal Press, Redmond, WA, 2005.

St-Laurent, Sebastien, Shaders for Game Programmers and Artists. Thomson Course Technology, Boston, MA, 2004.

Upstill, Steve, The RenderMan Companion: A Programmer’s Guide to Realistic Computer Graphics. Addison-Wesley, 1990.

Xbox 360 Development Kit Document, Microsoft Corporation, February 2007.