Running Quake II on a grid & As a genre of computer games, the massively multiplayer online game (MMOG) has the promise of enabling up to tens—or even hundreds—of thousands of simultaneous players. This paper describes how we began with an existing single-server online game engine and enhanced it to become a multiserver MMOG engine running on a grid. Other approaches require that a game be specifically designed to scale to MMOG G. Deen player levels. Our approach, using IBM OptimalGrid middleware (which provides an M. Hammer abstracted underlying grid infrastructure to an application) allowed us to reuse an J. Bethencourt existing game engine without the need to make any significant changes to it. In this I. Eiron paper we examine the design elements needed by an MMOG and provide a practical J. Thomas implementation example—the extension of the id Software Quake IIt game engine J. H. Kaufman using OptimalGrid middleware. A key feature of this work is the ability to programmatically partition a game world onto a dynamically chosen and sized set of servers, each serving one or more regions of the map, followed by the reintegration of the distributed game world into a seamless presentation for game clients. We explore novel features developed in this work and present results of our initial performance validation experiments with the resulting system. INTRODUCTION ness plan and renamed itself Emergent Game The computer game industry is deploying a new Technologies. At the time of the writing of this genre known as the massively multiplayer online paper, details about this new entity’s game archi- game (MMOG), characterized by large numbers of tecture were not available. The issues surrounding clients—ranging from several hundred to hundreds the building of MMOGs and large-scale virtual 2 of thousands—playing simultaneously. Until now, environments have become an area of interest and the creation of such a scalable game has been done study, with academia publishing studies on general 3 primarily by custom-designing game engines unique architecture, intelligent on demand provisioning, 4 to the specific MMOG. Efforts to create reusable server selection, distributed communications mod- MMOG game development and game engines have 1 been underway, Butterfly.net being an example of a ÓCopyright 2005 by International Business Machines Corporation. Copying in commercial game development environment for printed form for private use is permitted without payment of royalty provided that (1) each reproduction is done without alteration and (2) the Journal MMOGs. In its work, Butterfly.net obtained patents reference and IBM copyright notice are included on the first page. The title and abstract, but no other portions, of this paper may be copied or distributed related to the development of games and games royalty free without further permission by computer-based and other information-service systems. Permission to republish any other portion of the using grids. Subsequently, it has updated its busi- paper must be obtained from the Editor. 0018-8670/05/$5.00 Ó 2005 IBM IBM SYSTEMS JOURNAL, VOL 45, NO 1, 2006 DEEN ET AL. 21 5 6 els, distributed server location schemes, game specifically enhancing any one particular engine, artificial intelligence, and agent-based testing helping to broaden the applicability of the tech- frameworks for game developers. Additionally, the niques. application of MMOG technology to building large- 7 scale agent-based simulations for research on non- In this paper we report the results and lessons game-related commercial customer relationship learned in enhancing an existing single-server management systems is also underway. The com- multiplayer game engine to scale to the MMOG 8 plexities of large-scale group interactions within level. We chose the popular open-source Quake 12 online games, that is, exploring the communities II** game engine from id Software for this work, that come about within the virtual worlds of the but have attempted to develop reusable generalized games, has also become a topic of research. techniques in performing the enhancement. This enhancement was accomplished by using the 13 Many techniques have been developed to distribute OptimalGrid autonomic grid middleware from IBM both massively parallel applications and connected Research. This middleware was originally designed parallel applications, such as cellular automata and with scientific and engineering applications in mind; 9 finite element modeling. MMOGs—like other ap- however, the OptimalGrid object model and dy- plications that stress the resource limitations of a namic load-balancing features were easily adapted single system—are excellent candidates to which the to asynchronous online games. scalable computing power of grids can be applied, such as the approach taken by Emergent Technol- In this paper, we examine the key issues in 1 ogies and that presented in this paper. extending a game engine to running in a multiserver environment; we introduce the Quake II game MMOGs are especially demanding in their low engine and OptimalGrid middleware. We then communication-latency requirements. In comput- provide an overview of the resulting system, out- ing, as in other systems, increasing the number of lining its major components, followed by an parts, in this case the number of computers, exploration of the details of the design. We increases the likelihood of failure of a part. To introduce the runtime environment developed to provide a stable and long-lasting game for players, manage the execution of the MMOG game engine MMOGs running on a grid require fault tolerance and discuss the additional debugging challenges and dynamic load balancing, just like scientific grid inherent in a large distributed system, along with the applications. However, unlike scientific applications solutions we used. We then present the results of the that tolerate interruption for checkpoint and logging, performance experiments we performed on the online games do not permit interruption or latency system, and lastly, our conclusions and a brief degradation during load balancing or system main- outline of future work. tenance. MASSIVELY MULTIPLAYER ONLINE GAMES Because considerable performance optimization and Many types of parallel computational problems have game-play balancing have been done to single- been considered in research and implemented in server multiplayer engines such as id Software mature systems. They range from problems with 10,11 Quake** and Valve Half-Life**, it would be moderate connectivity, such as the simulation of desirable if the effort put into such engines could be finite element models, to massively parallel tasks, 14,15 reused to run MMOGs. Instead of designing new such as the search for optimal Golomb rulers. engines specifically for MMOGs, the reuse of MMOGs however, present a set of unique con- existing optimized engines would allow many of straints. Like many scientific simulations, the game today’s games to be scaled up to MMOG level and worlds found in MMOGs are often well-suited to allow game developers to continue using game division into spatial regions that may be run in engines with which they are familiar. Many popular parallel, albeit connected regions. engines apply similar architectures, offering the hope that techniques for scaling up of game engines Game-world partitioning to MMOG levels for one engine can be reapplied to Game worlds typically are continuous like the real other engines. Ideally, such scaling techniques world. This means that each spatial division has would be done in a generalized manner, rather than strong connectivity to the regions that neighbor 22 DEEN ET AL. IBM SYSTEMS JOURNAL, VOL 45, NO 1, 2006 (touch) it; actions and events in one region affect a single whiteboard as an intermediary, however, objects in neighboring regions. For example, a gun reduces the total connections to O(n), or one per fired in one region of the world may cause damage server. Of course, this improvement comes at the to a player in another region that lies along a line-of- cost of increasing latency in sending messages and sight path from the first. Long-range interactions limiting the collective bandwidth of all communi- between regions can exist, meaning in the example cation channels in the system to that of the that there may be multiple regions along the line-of- whiteboard. A more scalable approach is to use a sight path, with each region possibly being hosted number of whiteboards, each responsible for for- on a different server. Thus connectivity between warding messages between a set of servers that are regions is not limited to those that are ‘‘physically’’ nearby (in the sense of game-world regions). Each adjacent, but instead, connectivity exists between server then needs to connect only to several white- any region and all the regions that are visible to it. boards. For example, in long hallways and large open areas, a player’s local actions must be observable to Partition granularity players in any other visible region, regardless of A key problem involved in the partitioning of the which server is controlling that region. Any imple- game world is choosing the degree to which the mentation of spatial partitioning of the game world world is subdivided. For a one-time or static into discrete regions assigned to different servers partitioning of the game world, a significant trade- must provide a mechanism to propagate events to off must be made. If too few regions are used, then servers holding adjacent regions, and it must also fewer servers may be used, but regions can become have a mechanism to propagate such events along overloaded with players and other active entities. If the full path of connected regions, and events must too many regions are chosen, the area of the game be propagated in a timely manner so that no cause- world covered by a region becomes quite small. and-effect delay is introduced that exceeds that While this allows for more servers to be used, it can intended by the game designer or would be result in excessive data exchange among servers due experienced if the game regions were all hosted on a to the number of regions connected to each region; single server.