POSTMORTEM garrett young, mario rodriguez, and chris pickford

GARRETT YOUNG, MARIO RODRIGUEZ, AND CHRIS PICKFORD | Garrett’s eight-plus years span from testing NBA FULL COURT PRESS to exec. producing the PGR series with Bizarre. Mario began testing games at Microsoft as a 2002 University of Miami grad, recently working on PGR2 and RALLISPORT CHALLENGE. Chris has been designing and testing at Bizarre for four years, shipping the PGR series and . 46 mar ch 2004 | game developer GAME DATA

PUBLISHER: Microsoft Game Studios FULL-TIME CORE TEAM: 40 MAX TEAM SIZE (including test, licensing, localization, and other support resources): 102 LENGTH OF DEVELOPMENT: 2 years RELEASE DATE: Nov 18, 2003 PLATFORM: DEVELOPMENT HARDWARE: Pentium 600MHz–2.4GHz machines with 256–1024MB RAM, GeForce 2–4 series and ATI Radeon 9700 Pro video boards. DEVELOPMENT SOFTWARE: Microsoft Visual Studio .NET, Microsoft SourceSafe, Alienbrain, built-in 3D Editor, Softimage XSI, Araxis Merge 2001, SoundForge PROJECT SIZE: 37,174 files, 219,538 lines of code, 41GB of data

roject Gotham Racing 2 achieved our goals. is the sequel to the best- We owe that success to the strength of selling Xbox , our people and the clarity of our chal- developed by Bizarre lenge. Smart, effective, hard-working peo- P Creations (Liverpool, ple are critical to achieving any worth- England) with production and publishing while goal, and our two teams had that in support from Microsoft Game Studios. spades. Though there were some disagree- Our two teams rolled directly into produc- ments and late changes in tactical direc- tion after finishing international versions tion, everyone on the team was always of PGR in February 2002. We significantly working toward a clear overall strategic expanded the scope and quality of the vision for the project. A key ingredient to combined team, bringing on new artists, our ultimate success was the strong rela- programmers, and testers. Our ultimate tionship between developer and publisher: goal for this project was to create a AAA matching Bizarre’s design, technical, and PGR title for the 2003 holiday season artistic strengths with Microsoft’s built upon the fundamental strengths of strengths in testing, licensing, usability, the PGR franchise and innovate in our creative writing, and production manage- use of the Xbox’s online system, Xbox ment. Without an extremely high level of Live. Given our on-time delivery and the trust, we would not have been able to game’s 92.4 percent average score from maximize the efforts of each team, and over 70 reviews (referenced from PGR 2 would not have been as strong. www.gamerankings.com), we feel that we Our hope is that this postmortem can w w w.gdmag.com 47 POSTMORTEM

provide some insight into how we of cars (from 25 to 100) and cities (from PGR 2. We bet gamers would love racing worked together to build Project four to 11). Our artists invested signifi- online against their friends, and we Gotham Racing 2—the things that cant time in researching routes through decided to incorporate Xbox Live worked well, and the things we would do cities we felt were interesting, recogniza- Scoreboards for each race in our game. differently if we had to do it all over ble, and fun to drive. As we broadened These interactive high-score rankings again. With luck, we hope other teams the scope of the car list, we added new car allow every gamer with an Xbox Live will be able to apply these lessons to categories like classics, muscle cars, super- account to post a race result, and allow improve their processes and avoid some cars, and even SUVs. The designers also the top 10 racers to post their actual race of our pitfalls. expanded the Kudos system to increase ghost replay for anyone to download and the reward for skillful driving, adding watch (or race against). rewards for taking a “good line” through There were major challenges inherent What Went Right a corner, drafting, and navigating track in each decision. To build all the new sections cleanly (without hitting walls). cars and cities, we virtually doubled the Strong early vision for innova- But as a new PGR game, we knew we size of the original art team, which also 1.tion. In our efforts to build on had to continue to push that spirit of increased the challenges in team manage- the market success of PGR, we knew it innovation. Though Xbox Live was an ment and communication. Relying on was critical to stay true to and build on a unproven and unknown technology dur- unfinished technology from external formula gamers loved. We decided to ing our initial design planning, we com- teams created a large bottleneck in our greatly expand the number and diversity mitted to pushing the online frontier in production schedule, as we awaited their

48 mar ch 2004 | game developer ABOVE. A Porsche Carrera GT edges out a Saleen S7 on the Sydney waterfront. LEFT. A classic Porsche Carrera purrs loudly on a bridge in Yokohama. PREVIOUS PAGES. A Ferrari Challenge Stradale scares off tourists at the Duomo in Florence.

vendors in France. database, e-mail, and phone calls. The During production, members of the Liverpool art team worked closely with team visited locations all around the the Redmond licensing managers on world to gather research, reference approvals and change requests from material, and recordings. We shot thou- vehicle manufacturers and other external sands of photos and hundreds of hours licensors. All members of our interna- of video in each city. We recorded real tional localization teams interacted DJs in Moscow and Tokyo. We even directly with our UI developer. made a special trip to the Promised As most teams do, we also planned Land, visiting the Ferrari plant in goals and deliverables for each milestone Maranello, Italy to photograph and over the life of the production schedule. record engine audio of the Ferrari Enzo However, it was the people we had in deliverables. We chose to accept these before it was released to the public. To place and our open communication challenges head-on, given our vision to borrow an old British saying, the sun channel team-wide that were the greatest expand the scope of the game beyond the never set on the PGR 2 team. contributors to our ability to resolve original title. Managing such a global team created issues quickly and hit our aggressive hol- many problems in communication and iday release schedule. Reducing worldwide manage- schedule management. Our approach to 2.ment. The solving these problems was to actually Proving stable online game- team included all of our core developers, reduce, rather than expand the amount 3.play early. Online multiplayer is artists, game and sound designers, and of management. We built up strong com- frequently the highest risk area for any production staff. The Microsoft team in munication channels between all mem- game, since multiplayer features can be Redmond, Washington included many bers of the team, removing the communi- the hardest to implement and require sig- production and design support staff, cation bottleneck that can occur at the nificant optimization and tuning. We licensing managers, a full test team, and producer level on game projects. All addressed this problem early by imple- the marketing team. Given the impor- members of the production support staff menting the bulk of our network code, tance of our international release, we at Microsoft were empowered to interact physics optimizations for interpolation of had localization staff working full-time directly with all of their peers and other car speed and trajectory across all boxes, in Japan, Korea, Taiwan, and Ireland. members of the Bizarre team. The and support for all Xbox Live features We also employed 3D art vendors in Redmond testers and Liverpool develop- by early April, 2003, five months before Australia and England, and translation ers interacted directly through the bug release. Our overall multiplayer execu- w w w.gdmag.com 49 POSTMORTEM

LEFT. A photograph of a bridge in Sydney. MIDDLE. A wireframe version of that bridge. RIGHT. An in-game screenshot of the bridge, tion was relatively smooth, with only the spectrum of all end users. The theory ers are required to get contractual performance and voice issues to resolve “a user is only a ‘new user’ once” pre- approval with the owners of each logo during our final code optimization phase. sumes everyone in your audience will be and likeness before releasing it in a game. We were able to achieve positive willing to struggle through a confusing With 11 cities (including a real-world results by hiring a strong network pro- interface and unbalanced difficulty levels race track), over 100 cars, unique DJs for grammer, working closely with the Xbox to experience and enjoy your game vision. each of our 33 radio stations, and over Live team to fully understand the new We had success in addressing these 300 songs, this was a monstrous task. technologies as they were being built, challenges on PGR 2, but not without We employed a team of five licensing and synchronizing our implementation serious investment of people and itera- managers over the course of the entire and test processes. Implementing a major tion time over the last few months before project to own and execute on this task, set of Xbox Live features to the require- release. We spent literally hundreds of establishing contacts and maintaining ments of the Xbox certification team hours hand-tuning each car, to balance relationships with all appropriate parties, required educating our team immediately. realistic handling with ease of use. We facilitating review of all in-game assets, This investment paid off later by allow- used the Microsoft usability and con- and working closely with legal counsel to ing us to quickly implement new APIs as sumer playtest labs extensively to seek close down each contract. Xbox Live 2.0 features such as stats with out feedback from racing gamers, and see The licensing team’s contributions attachments (to support upload of ghost first-hand their initial experience and dif- were vital. Not only did they enable our replays) were completed late in our proj- ficulty navigating our interface, under- artists to fully realize the authenticity of ect schedule. standing the Kudos system, and achiev- each city, they also secured the appropri- Testing’s early involvement quickly ing the micro-goals of each race mode. ate rights, without which we would not identified lag and interpolation problems, Aside from finding and fixing all func- have been able to include Ferraris, and allowed us to troubleshoot new tionality and content bugs, our greatest Porsches, BMWs, or any of the other real Xbox Live features such as friends, voice, efforts during the endgame were put cars in PGR 2. and Scoreboards during initial implemen- towards balancing our gameplay difficul- tation, which allowed the development ty. Almost everyone across the teams in team to stabilize new features earlier. both Liverpool and Redmond—and many What Went Wrong As a result of proving multiplayer sta- others not already on the PGR 2 team— bility early on, we were able to eliminate spent time providing play-balance feed- Synchronizing production a major risk to our production schedule back on specific challenges, each of the 1.deliverables worldwide. As and focus our efforts on other gameplay, five difficulty levels, and the overall rate mentioned previously, managing the con- tuning, and polishing issues during the for unlocking cars. Considering the scope tributions of our worldwide team was a endgame. of the game, the size of the audience, and great challenge, and we feel that process the concerns we had about exposing went well overall. However, as with any Driving user feedback into “golden paths” (shortcut cheats to high huge challenge, there were major prob- 4.product design. Designing a scores) in the scores and ghosts posted to lems that surfaced in some core areas of game under tremendous time pressure our Live Scoreboards, we were very happy production. can often create a myopic approach to with the results achieved during our Our original plan for addressing the interface design and play balance. The intense and collaborative tuning period. problem of recording, processing, and team is so close to the game during implementing source material from our development that objective evaluation of Effective licensing manage- worldwide car list was to distribute the the inherent challenge and usability 5.ment. Real-world authenticity is workload and take advantage of our becomes difficult if not impossible. a core characteristic of the PGR franchise: global resources. This ended up causing Schedule requirements limit valuable itera- real cars, real cities, real radio stations more problems than it solved. Crucial tion time, and members of the team can- with real DJs, and real music from real implementation and tuning time for each not represent the diversity of skill across bands. Unlike movie makers, game mak- car was sacrificed, as all cars were not

50 mar ch 2004 | game developer POSTMORTEM

A Porsche 911 GT1 burns rubber on the streets of Chicago. recorded to identical specifications and behind development. The 5,000-mile dis- pursued many different paths and game many cars were recorded very late. tance between teams greatly increased the modes before hitting on a concrete plan. Additionally, an essential member of the risk associated with any last-minute file Martyn Chudley, the creator of the PGR sound design team moved to California changes at the end of the project. franchise and head of Bizarre Creations, and attempted to continue to fill the role Though we made some great strides to played a critical part in stepping in to nar- part-time, adding friction to an already better synchronize implementation and row the focus of the overall game design weak process. A related problem also testing, our challenge moving forward in early 2003. occurred with the DJ scripts we recorded will be to hit our functionality and con- Iteration on the handling characteris- in each local city, as long contract and tent deadlines more effectively, increase tics of each vehicle proved incredibly recording schedules delayed implementa- the stability of the build process, and to challenging, and was completed very late tion beyond our desired dates. increase the scope of smoke tests in in the schedule. As with any simulation- This failure taught us two lessons. Liverpool before builds are sent over to quality racing game, each change in vehi- First, we needed to better synchronize the Redmond test team. cle handling has a significant knock-on content creation and implementation in effect on each car’s usability, the perform- the future. Second, we were reminded of Design took too long. ance of the AI, the class and competitive a challenge all Microsoft-published proj- 2.Incomplete design had the single categorization of each car, and the diffi- ects face: synchronization of the imple- largest impact on the slide of deliverables culty-level setting for each race. mentation and test teams. throughout our project schedule. Though Quadrupling the number of cars from Though all the developers, artists, and we all understood and agreed upon the PGR to PGR 2 more than quadrupled our designers at Bizarre were working in real fundamental vision for the franchise, we vehicle-tuning time, as the broader scope time within the same bug database as our started our PGR 2 design document very of vehicle content required far more dili- Redmond testers, licensing managers, and late, and we were still updating this docu- gence, testing, and tuning between indi- international localization teams, we did ment after feature freeze and E3 as new vidual vehicles and classes. We underesti- experience many setbacks. At 4GB, our ideas cemented. We ended up overhauling mated the initial scope of this effort. transfer time for builds was substantial, many elements of our design several times, To combat these problems we expanded and we were forced to change file transfer such as the user interface, Kudos reward the size of the test team, pulled in addition- tools three months before release. Some values, and overall game structure. al designers from other projects, and hired obvious bugs in Redmond were difficult There were many critical factors that a small team to execute specifically on the to reproduce in Liverpool, as the testers extended our design phase. Our designers play-balancing task. Moving forward onto verified builds that were at least a full day were spread too thin early on, and we future projects we will seek out additional

52 mar ch 2004 | game developer LEFT. The Bizarre Creations PGR 2 development team. RIGHT. The Microsoft Game Studios PGR 2 production and publishing team. solutions, such as fleshing out key sections of the design document upload a ghost replay, we needed to be able to guarantee each earlier, prototyping new gameplay elements in offline builds, and ghost would be a perfect replica of the actual race result. The proactively scheduling our extensive play-balance efforts. Iteration critical nature of this feature caused us to dedicate significant is a natural part of game development and we’re pleased with the attention from our test team. results, but getting there was no easy task. The replay subsystem served as the underlying framework for recording the ghost data. This legacy system was both complex Relying on new technologies from external and difficult to consistently debug. One late night two days 3.teams. PGR 2’s design called for online features to be before RTC (release to certification), our testers were in heated integrated within almost every area of the game. These features competition, challenging each other’s high score ghosts on one required APIs that were still in development by the Xbox Live team as our schedule progressed, raising many significant obstacles and time constraints. While our Scoreboards were an extension of the system shown in the 2002 Xbox Live starter kit’s MotoGP demo, we utilized them to a far greater extent than on other titles. Coupled with new Xbox Live features, such as stats with attach- ments (such as ghosts), and the high volume of Live users we anticipated, we knew that we had placed a big bet on bringing these emerging technologies to a usable and stable state. If any piece failed, the system would have failed. Implementing the new attachments APIs for uploadable and downloadable ghosts presented us with unpredictable problems, as we were pioneers in this space. The Xbox ATG team was extremely supportive, but they were also pushing hard to complete features for their own deadline. One example of a problem we should have been able to avoid was optimization of our interaction with the Live Scoreboards. During research done by the Xbox Live team late in our schedule, they found our code was making far too many calls to their Scoreboard servers, a capacity problem their servers would not have been able to handle after release. Though we fixed this problem, it raised significant production fears at the end of our schedule. In relying on critical technology from external teams, we have learned the importance of allocating an adequate schedule buffer to accommodate unforeseen problems, maintaining strong com- munication with all dependent parties, and gaining a deep under- standing of the technology.

Stability of the ghost replay system. In supporting 4.a feature where any Xbox Live user worldwide could w w w.gdmag.com 53 POSTMORTEM

process began to take longer. Everyone was working very long hours, late check- ins were made after build smoke tests, and additional steps were added in man- aging retrieval of assets from multiple game images, complicating the build process. The test image would often be pulled together and posted to the secure FTP site, only for the Redmond test team to find the build would not run when they came in the next morning, losing a day of testing on the latest bits. Moving forward, the Bizarre team will stick to two game images—one image for the team to post all code and content, and another image for all tested, ship- pable content. With less moving parts, A BMW M3 shines through a rainy night in Edinburgh. we expect the process to go more smoothly on future projects. of the arcade cone challenges, with sever- city models, and over 8,000 audio content al lead changes over the course of a few files. This caused tremendous confusion at hours. As one score became very difficult the end of the project, as our processes Final Lap to beat, one of the testers noticed the were not originally planned to handle this total Kudos shown in the ghost replay did scope of code and content. n the end, we are all very proud of the not match the value on the Scoreboard. At the beginning of the project we had I results we were able to achieve in Despite all the months of non-stop test- multiple teams uploading to one game PGR 2, and we hope gamers are too. As ing, a strong game only hours from ship- image. This led to significant content a team, we were able to deliver on our ping still had a very critical bug some- incompatibility problems in the build, vision and critical priorities for the game. where in the replay system. such as cars using incorrect engine audio, We were able to increase our quality bar Competitive gameplay is a very valu- city tracks without track-side barriers, by maintaining a strong balance between able part of the testing process during the and old bugs re-appearing as old content building upon the core fundamentals of endgame—this was how gamers were over-written as new. We planned to solve PGR gameplay and breaking new ground going to be playing our game! Although this by splitting up the original game in online multiplayer and scoreboards. we fixed it, we could have found that image into separate images for city data, We were also able to deliver the game to bug earlier, and in the future we will also audio and radio content, car content, and gamers on time. push to create more automation for criti- all source code. We also planned to cre- However, no project is perfect, and we cal areas such as this, including hooks ate a unique test image, where all content certainly had our share of hurdles to for specific test scripts, boundary, and would be copied before release to the overcome, many self-imposed. We grew a stress conditions. We’ll never be able to Redmond test team. lot as a combined Bizarre Creations and perfectly emulate the gameplay of mil- However, splitting the process this way Microsoft Game Studios team between lions of gamers, but by prioritizing our ended up causing more harm than good. PGR and PGR 2. Our challenge will be focus, expanding our automation suite, During the endgame, as the Bizarre team to continue that growth in the future, to and increasing the size and scope of our was doing builds every few days, the test learn from the success and failures of our endgame “bug bash” efforts to broader image had to be manually updated fre- past, and to work together to overcome internal groups and teams, we’ll be better quently. To ensure consistency and speed, future problems as they arise. armed to find and fix all show-stopping the team created a step-by-step process Looking back, the key to our success bugs before release. and set of batch files, to be run in a specif- was the team involved in bringing the game ic order each time. With a game so large to life. PGR 2 was built by smart, hard- Build process and source and server space at a premium, we could working people working together effectively 5.control. We had a substantial not use Alienbrain or another file manage- with a high degree of trust, open commu- number of assets to manage during content ment package. We were forced to dedicate nication channels, and a clear vision and creation at Bizarre Creations, including one person to this manual drag-and-drop goals. Easy things to say, but the magic over 100 cars with 3D model and dynam- process for creating the test image. was in the execution, as it will likely con- ics/handling files, 11 constantly evolving Disasters began to strike as the build tinue to be in the foreseeable future. q

54 mar ch 2004 | game developer