Databases for Online Gaming (MMOG)

[email protected] Manager Engineering, CARE Group

1 Agenda § Introduction § Persistent World § Strategies § Scalability § Distribution § High Availability

2 Versant Corporate Profile § Established 1988, HQ in Fremont, CA § 1996 NASDAQ: VSNT § Leading provider of 7th generation enterprise- level object database § Installed base is Global 500 “who’s-who” § Focus is on performance and scalability improvement of complex large scale enterprise systems § Offices: U.S., Germany, India, Japan

The main takeaway: Versant has delivered enterprise level solutions to the Global 500 for 14 years.

3 MMOG § Massively Multiplayer Online Game § is capable of supporting hundreds or thousands of players simultaneously on the Internet § is played in a giant persistent (fantasy) world § Role Games (MMORG) § Tycoon Games (MMOTG) and First Person Shooter (MMOFPS) § Real Time Strategy (MMORTS) § Sport Games (MMOSG) and Racing (MMOR) § Social Games (MMOSG) § Business Model § Most MMOGs require a monthly subscription fee § Some can be played for free

4 Persistent World § Is a virtual world § is always available § world events happen continually § events continue to develop even while some of the players are not playing their character § a player's character can also influence and change a persistent world § The state of the gaming world has to be maintained around the clock

Definition en.wiki.org: A persistent world is a virtual world (often in a fantasy setting) that is used as a setting for a role-playing game, often Online. The world is always available and world events happen continually. The persistency comes from maintaining and developing the state of the gaming world around the clock. Unlike with other games, a persistent world game's plots and events continue to develop even while some of the players are not playing their character. The comparison is to the real world where events occur that are not directly connected to a person, or continue to happen while a person sleeps, etc. Likewise, a player's character can also influence and change a persistent world. The degree to which a character can affect a world varies from game to game. (Since the game does not pause nor create player-accessible back-up files, a character's actions will have consequences that the player must deal with.)

5 Characteristics of MMOG § „Within minutes, players have an uncanny ability to discover the size of the possible space“, Will Wright § Players are the central focus § + artificial intelligence used in the older-technology games § Fulltime availability § „unlimited content“ § Performance, Performance, Performance § Market is highly competitive § MMOG need to adopt strategies to provide tremendous amounts of content to the users

6 Storing Persistent Worlds § MMOG is one of the most demanding, large scale applications in the world § Evolutionary game models using OO languages § Early models were quite simple and complexity is increasing § > 95% written in C++ § Scalability § High concurrency and transaction throughput § Real Time response § High Load at peak times § Large data set § Distributed State management § Parallel Processing § Data Clustering

A SUCCESSFUL Massive Multi-Player Online Game (MMOG) falls squarely into the category of one of the most demanding, large scale applications in the world. The MMOG's combined requirements of real- time response, high concurrency and large data sets can bring traditional technologies and architectures to their knees. This point is evidenced by the public pain of successful games like Ultima Online, EverQuest, and World of WarCraft™.

It is well documented that the effectiveness of the data tier will drive game scalability and system operating footprint and therefore operational expense. Traditional technology in the data tier consists of a combination of File System for static content and Relational Databases (RDB) for dynamic content. It is well documented that under the requirements of real-time response, high concurrency and large data, relational database technology does not scale well. It is also well documented in that the right Object Oriented Database (ODB), for reasons to be discussed here in, does scale well under those MMOG requirements.

7 Brief History of the Database

Model Strength Weakness Data

Hierarchical Performance Flexibility, queries Simple

Network Performance Flexibility, queries Simple

Relational Flexibility, Performance Simple queries ODBMS Performance, Random, Complex flexibility ad hoc queries

Very important for MMOG problem domains

8 Storing Objects Transient Objects Persistent World Persistent Objects Cache

Transparently Transparently fetches writes changes objects from back to the the database(s) Database(s)

Direct object persistence Navigational versus Vs. Mapping query-based access Versant

MMOGS and so the persistent world are mainly developed with C++, leveraging OO concepts. The persistent world and game logic uses the objects and the database provides a transparent access to virtually all objects. The developer can use the OO language constructs like Inheritance associations, collections and polymorphism to model the problem domains and implement it seamlessly. With Versant the persistent objects are loaded either as result of a query or most of the time as part of a navigation. The caching mechanism is very sophisticated and provides the developer with additional controls, so that he can develop own strategies. Once an object is modified it will be automatically be written to the database with the next commit. The objects will be stored as objects and no mapping is required. Versant provides 2 access patterns: navigation and selection. Often many operations are natural navigations like getting associated objects. With RDBMS these navigations have to be translated into query/join statements.

9 Using the RDBMS

SELECT FROM table1 ...; In-Memory SELECT FROM table2 ...; RDBMS convert_tables_to_memory();

BackPack bPack = new BackPack() for(i=0; i<100; i++) { BackPackItemp = new BackPackItem(); … bPack.addItem(p) }

convert_memory_to_tables(); Tables INSERT INTO table1 ...; UPDATE table2 ...;

§ Different in-memory and storage models § Significantly degraded performance when (de)composing data § Even more conversion code § Separate programming and data manipulation languages § Only suitable for simple models § Difficult if schema evolution is required Copyright © Versant 2007 - 10

10 ODBMS Solution VERSANT In-Memory

BackPack bPack = new BackPack() for(i=0; i<100; i++) { BackPackItemp = new BackPackItem(); … b.addItem(p) }

§ In-memory and storage models are the same § Orders of magnitude faster § No additional conversion code § Improved programmer productivity § Host programming language is also DML and DDL

Versant also provides advantages in the area of flexibility in evolving the existing database schema and in creating dynamic game object characteristics. This flexibility is achieved through support of both easier design and model extensibility. Versant deals with object oriented models in the proper polymorphic forms of the language and treats references to related objects orthogonal to the state of the objects involved. This means that you can write game logic to deal with interfaces or abstract classes that are then further derived to create the specialized classes for your game without introducing needlessly complex application and mapping code. Performance optimizations created at the abstract level allow you to add new game classes while keeping your performance optimizations intact. The original game logic incorporates new game object use with minimal code modification. Game object attributes can be easily managed through objectified relationships without introducing complicated coding semantics. A simple design example could be the modeling of a character Backpack. One obvious design when properly using OO would be to model the Backpack as an Item that contains other Items. The design uses both inheritance and self referencibility. This design is handled seamlessly using Versant and retrieval of a Backpack's contained Items requires nothing more than a message send in the game logic code. This same design in a relational form, even if flattening of the Backpack into a single Item table, forces performance draining JOIN operations due to the self referencibility. A more complex extensibility example could be the creation of an entirely new game character by subclassing an existing class that describes general character behavior. The addition of the new character class can be deployed to the database server on the fly. The Versant Object Database will incorporate the new schema and evolve the database automatically, and existing game logic will continue to operate uninterrupted bringing the new character into the game with minimal effort. 11 Example Architecture

The example MMOG Architecture is divided into three main layers: Game Server Layer ( clusterable game engine and the communications layers ) and the Database Layer. The game server is designed over a proxy data tier composing direct and clustered data access services. The architecture is designed to support the addition of new users ( via Game FE machines ) who can play seamlessly across the entire world with all other users within a Game Server Layer, without replicating the entire system hardware. This is in direct contrast to traditional MMOG architectures which scale by replicating the entire world ( hardware infrastructure ) and isolating new users to their given world. This approach enables in game events composed of 100's of thousands of users. Further, additional game server layers can be clustered and although direct game play across server layers is not possible, all game resources like money, land, weapons, etc are common across servers. The design provides for cost effective horizontal scalability enabled via a highly concurrent, super scalable database tier.

12 Scalability § Vertically § Single hardware node can maintain a set capacity of operations und required performance criteria by adding resources to the node § Direct correlation to costs , f.e. you can support 5000 concurrent user on a single Xeon CPU server with 2G RAM § Horizontal § When reaching a tolerable threshold of vertical scalability continued capacity of operations expansion can be achieved by adding additional hardware nodes. § Goal: § Maximize vertically scalability § Ensure Horizontal scalability to provide unlimited expansion and

When designing an MMOG software system for scalability, one should consider the ability to scale both vertically and horizontally. Vertical scalability means that a single hardware node in the software system can maintain a set capacity of operations under required performance criteria by adding resources to the hardware node. Horizontal scalability means that when reaching the tolerable threshold of vertical scalability, continued capacity of operations expansion can be achieved by adding additional hardware nodes. Maximizing Vertical scalability is important as this generally has a direct correlation to profit margins as a given configuration of hardware and software infrastructure supports a fixed number of users which translates into revenue. Additionally, maximizing vertical scalability reduces the complexity and overhead of maintainability and further improves margins. Ensuring horizontal scalability over vertical scalability means that unlimited expansion is possible with predictable margins. Versant enables developers of MMOG applications to implement both vertical and horizontal scalability through an advanced multi-threaded architecture integrated with a number of specific features described below.

13 Distribute data and workload § Strategies § Reduce the number of active data and users per database § Distributed data and load § Offload „Obsolete“ information that will be deleted after a while, migrate them to an archive db § Reduce number of conflicting operations (Query vs. Updates) § Data Caching § Caching and In Memory DBs

14 Shards or Grids § Shards § Split game into a number of parallel worlds through clustered servers § Distribute data on different databases § Users can „only“ see subset § need distributed access or propagation of new data § Grids § F.e. each resource is responsible for an square area in the persistent world § It knows it‘s direct 4 neighbors (right, left, up and down) and if a character moves to another square, the responsibility also moves.

A significant advantage of Versant is the ability to cache all or portions of the database in the game shard memory space and/or proxy data server for real-time read response. This shard caching is critical for meeting the minimized latency requirements of an MMOG and significantly improves vertical scalability by off-loading activity from the database server processes relieving contention for shared resources. The "native" persistence capabilities also provide transparency of access. For example, if a character game object is loaded into the shard, either by id or query, then in the game logic a message is sent to the charter to retrieve a wallet for money, if the wallet did not exist in the shard memory space, Versant will transparently retrieve that wallet from the database server memory space without the application executing any special query code for the wallet retrieval. Game objects are implicitly assigned system global id's so that they can be retrieved without impacting application logic. There are also capabilities in Versant so when a game object is retrieved from the server memory space, a configurable number of related game objects will automatically be retrieved into the shard to guarantee a subsequent call to the database server memory is not required, thereby avoiding unnecessary network calls.

15 A Transparently Distributed Database

Game A Shard B Relationships Cache maintained across D C persistent stores

TCP/IP

B Object ObjectObject ObjectObject Object A Object Object Object C D

Versant Versant Versant

Versant is a fully distributed architecture. The distribution is essentially transparent to the developer. Objects can reside in multiple persistent stores and be used transparently by a single or multiple clients. Objects in one persistent store may refer to objects in another persistent store and the bean developer can directly access the referenced objects without any additional coding. The only requirement is that your session have a connection to the persistent store where the object resides. Versant will then find it for you. Taking advantage of such transparent distribution allows for seem less implementation of data aging strategies and active data sets. This also allows for advanced partitioning strategies for disk bound systems where expensive options like EMC arrays are not feasible.

16 Parallel Processing with distributed databases (One Shard)

Parallel Processing: Versant provides the ability to incrementally add physical databases to a cluster which are treated as parallel processing nodes in a distributed system. Operations performed on the nodes are optimized using threading and asynchronous communications to provide parallel operations for optimal performance. For example, the Game logic can define many logical groupings of physical databases and treat them each as a virtual datastore. SQL92 type indexed queries can be targeted at a logical database and are executed in parallel within the physical in-memory databases representing the virtual datastore. By controlling the size of any physical database node within the cluster, the game logic can guarantee response times by controlling concurrent access and internal structure traversal times. Unlike in traditional relational databases where queries target a table and its columns, returning records that need to be mapped into your language objects, the Versant database queries target a class and its attributes, returning objects of that class without any intermediate application mapping. Once the queried objects are retrieved into the game logic, access to related objects occurs through the normal message send process of the language. An example is shown in the following figure. A query can be performed across many in-memory databases in parallel to find a particular player. Once the player object is returned by the query, the game logic sends a message to the player to get the related backpack. There can be specialized messages defined so that whenever a backpack is retrieved it's contained items are also automatically retrieved. The message getBackpack is sent to the player and the Versant database will ensure that the related backpack and it's contained items are retrieved into the shard cache. The Versant database handles all of the issues dealing with locality of the related objects in the physical cluster. There is no required code in the game logic to deal with determining where objects are physically located. Despite the fact that the backpack and its items are not in the same physical database as the player, they are still transparently returned on message send to the shard cache. There is no limit to the type of architectural flexibility gained with this design, allowing any type of partitioning scheme deemed appropriate for your game design. Versant guarantees uniqueness of object identity across the entire cluster and maintains the physical locality orthogonal to the logical identity. Even if external processes are put in place to move objects from one physical database to another, there will be no impact to gaming logic. Versant even handles the logical and physical consistency of backups across the distributed cluster. 17 Distributed State Management (Multiple Shards)

§ Multiple shards may use same data that can be changed on one side § Cache Coherence needs to be ensured

Versant has another feature working in concert with the real-time shard cache that facilitates distributed state management across multiple shards. Shared game objects in the shard memory space can be identified so that changes made to the shared game objects cause cache invalidation notifications to be broadcast to all shards. All shards in the cluster who have those shared objects will then automatically refresh the shared objects from the database server memory to ensure a consistent and up to date game state across the shard cluster. This allows unprecedented horizontal scalability for large numbers of users with controlled cost structure by allowing incremental addition of shards as the player demand and concurrency builds. Further, network communications routing can use load balancing to take advantage of this clustering by reassigning incoming requests for better response or failover without incurring significant performance penalties. In addition, this allows the design of a game where players and related game objects can effectively "walk" shard boundaries for uninhibited game play.

18 Optimizing TX throughput

§ In Memory database caches data read from disk § Typically the logging of tx is the bottleneck (ACID) § With RDBMS also the query execution and caching becomes a bottleneck Copyright © Versant 2007 - 19

Transaction throughput: Versant has the unique ability to provide in-memory performance characteristics while maintaining the ACID (Atomicity, Consistency, Isolation, Durability ) properties of an enterprise level database management system. Versant makes this possible by leveraging synchronous in memory databases coupled with the ability to recover in-flight transaction failures. Versant uses asynchronous socket communications to provide highly efficient 2-phase commit protocol across distributed in-memory databases. Further, commit calls to all databases are non-blocking allowing for immediate return to application control with only the overhead of a network call. Contents of commit operations are stored in memory based transaction log buffers which are asynchronously flushed to the disk based logging subsystem by background threads every N ( configurable ) number of commits. Versant maintains durability for transactions by ensuring that transactional content resides synchronously in multiple buffer spaces in a shared nothing architecture. The rapid database server response, due to this architecture, facilitates unprecedented vertical scalability by minimizing contention for internal structures in the server processes and eliminating waiting due to I/O operations. The result is a database node that can support thousands of concurrent update transactions. The game shard manager tracks all CRUD ( create, read, update, delete ) operations in the game logic. When the game logic calls commit, Versant uses asynchronous socket calls to get the changes ( results of the transactional CRUD operations ) to flow towards both in memory databases. The transactional changes are written into piggy backed double buffers . Piggy backed double buffers work in a way that a buffer is always available for writing even when one side of the buffer is being flushed to disk. Once the server threads have written into the piggy back buffers, control is immediately returned to the shard, so that real- time processing can continue. When the configured buffer flush threshold is reached, the piggy back buffers reverse, new transactions are written into the opposite side and an asynchronous thread flushes changes to disk. This process continues during normal operations. If any failure should occur, Versant recovers on the fly as described in the section below on high availability.

19 Fault Tolerant Database Configuration

High Availability: Using Versant, if a failure occurs in a database server process due to network, hardware, software process failure, the shard will transparently receive the failure notification and change it's mode of operation to continue the failed transaction only to the remaining live server. Versant handles in-flight failures from either primary or secondary site transparently. During failover operation, the shard sends additional transaction information to the live server to be used for automatic background restoration of the failed site. The live site forks a polling process that seeks to start the failed database server. When the failed site is reachable, the polling process uses the change set records to automatically resynchronize the failed site with the live site production database server. Once synchronized, the polling process sets to server to notify the shard to begin synchronous operations. The Recovery of the failed site requires no DBA intervention, unless of course something catastrophic at the hardware level has occurred like complete disk failure. If desired, during the failover operation period, the live server can be configured to automatically change modes to provide synchronous buffer flush on commit.

22 Summary

The MMOG application space demands a real-time, high concurrency, large scale, enterprise class database solution. These combined requirements do not fit well with traditional relational technology. The Versant database provides a combination of features specifically designed for the real-time, high scalability requirements of the MMOG application space.

23 For More Information § Visit Versant on the Web at: § http://www.versant.com § Technical Questions § [email protected] § Jobs § [email protected]

24 Even more information

§ Pushing the technological barriers of game design with IBM technology http://www-306.ibm.com/software/success/cssdb.nsf/CS/MCAG- 6FXRPV?OpenDocument&Site=default § Architecting Scalability for Massively Multiplayer Online Gaming Experiences http://www.gamesconference.org/digra2005/papers/68879030073efafba26fb47d37bb81b9.doc § SecondLife Design - Enabling Player-Created Online Worlds with Grid Computing and Streaming http://www.gamasutra.com/resource_guide/20030916/rosedale_pfv.htm § Powering next generation gaming with on-demand computing http://www-1.ibm.com/grid/pdf/butterfly.pdf § A scalable peer-to-peer network for virtual environments http://www.phys.sinica.edu.tw/~statphys/publications/2006_full_text/S_Y_ Hu_IEEE_July- August_22(2006).pdf § Adaptive middleware for multi-player games http://www. cs.cmu.edu/~rajesh/papers/middleware05.pdf § Object database -vs- Object-Relational databases http://ce.sharif.edu/courses/84-85/1/ce384/resources/root/Object%20Database%20vs_%20Object- Relational%20Databases.pdf § Comparing the performance of object databases and ORM tools http://www.odbms.org/download/027.01%20Zyl%20Comparing%20the%20Performance%20of%20O bject%20Databases%20and%20ORM%20Tools%20September%202006.PDF