Create a 20 Times Faster Database Engine Optimized to MMOGs

Shuichi Kurabayashi Technical Advisor/Director of Cygames Research Cygames, Inc. [email protected] Summary Legacy Full-Featured DBMS Many Business Functions Authentication Security A simple in-house Encryption Transaction database engine Data Mining Full SQL tailored to MMOGs is highly effective to provide Minimal & Tailored DBMS mobile MMOGs for multi- millions unique users. Simplified Relational Access Lock-Free I/O Operations Legacy Full-Featured DBMS Game developers do not Many Business Functions use almost of those Authentication Security Encryption Transaction business-oriented Data Mining Full SQL functions

It is easy to implement Minimal & Tailored DBMS the minimal database Simplified Relational Access management system. Lock-Free I/O Operations My message is: Quick Introduction Who are you? What is Cygames? What is problem? Introduction: Who I am Shuichi Kurabayashi, Ph.D.

• Technical advisor of Cygames, Inc. Also Director of Cygames Research.

• Project Associate Professor at the Graduate School of Keio University Introduction: Cygames is one of the largest mobile game developers in Japan Known as the developer of the wildly popular card battle game “Rage of Bahamut”.

Recently released “Shadowverse”. Background: We have been providing mobile MMOGs for 10 millions users.

10,000,000 9,000,000 8,000,000 7,000,000 6,000,000 5,000,000 4,000,000 3,000,000

2,000,000

The number of of number The

The number of of number The registered users registered registered users registered 1,000,000 0 0 0 5 10 15 20 25 months after release Background: Mobile gaming is an important subject of DB Research

Tokyo Stock Twitter Mobile game Exchange (Japan) 40,000/sec 6,700/sec messages at 100,000/sec (400,000/min) peak. transactions transactions in average. MMOG Problem Definition ● Modern MMOGs require … databases to support large- scale availability, strong availability consistency, and real-time consistency response. real-time ● It is difficult to support such capabilities efficiently by using conventional systems. CAP Theorem ● C: Consistency of data ● After data has been updated, if something else references that data, it will always be guaranteed that the Consistency updated data can be referenced. ● A: Availability of the system ● No matter what the current situation, the system will continue to Partition- Availability operate. tolerance ● P: Tolerance to network partitions

● Data can be distributed. http://berb.github.io/diploma-thesis/original/061_challenge.html CAP Theorem Out of these three guarantees, only two can ● C: Consistency of data be fulfilled at a time. ● After data has been updated, if something else references that data, it will always be guaranteed that the Consistency updated data can be referenced. ● A: Availability of the system ● No matter what the current situation, the system will continue to Partition- Availability operate. tolerance ● P: Tolerance to network partitions

● Data can be distributed. http://berb.github.io/diploma-thesis/original/061_challenge.html Choosing Appropriate System CP: Apache Hbase, Hadoop, CA: conventional Consistency BigTable, relational Good for database analytics systems, such as MySQL, Oracle, and SQL server, Partition- Availability strong tolerance consistency AP: SNS,NoSQL, eventual

http://berb.github.io/diploma-thesis/original/061_challenge.html consistency CAP Theorem CP: Apache Hbase, Hadoop, CA is the most BigTable, Consistency Good for important for analytics games, because games require strong Partition- consistency and Availability tolerance fast response AP: SNS,NoSQL, eventually

http://berb.github.io/diploma-thesis/original/061_challenge.html consistency Layer Analytics Layer Storage Live Typical Typical Backend Architecture Server

Partitioning Web (master) MySQL Fusion - io Cluster Cluster Set 1 Server Web (slave) MySQL replication Server Web MySQL (log) DataWarehouse Systems Statisticaland Analysis Systems Server Web (master) Server MySQL Fusion Environment Web Data Data Mining Cluster Cluster Set 2 - io (slave) MySQL Server Web Sharding MySQL (log) Server Web (master) MySQL Fusion Server Web Cluster Cluster Set 3 - io (slave) MySQL Support DB Customer Server Web MySQL (log) Server Web … Server Web (master) MySQL Fusion Cluster Cluster Set - io Server Web (slave) MySQL … 32 MySQL (log) Server Web Primitive Unit Web Server PHP

Main memory, memcached memcached stores static data shared This bandwidth is the among all the PHP processes. most precious resources.

MySQL MySQL MySQL MySQL stores status data, (master) (slave) (log) which are updated in a Fusion-io replication

real-time manner. Partitioning Cluster Set 1 Layer Storage Live decomposition decomposition of Combination Server

Partitioning Web (master) MySQL Fusion - io Cluster Cluster Set 1 Server Web (slave) MySQL replication Server Web MySQL (log) ） ） Server Web sharding brings high brings and (master) Server MySQL Fusion Web Cluster Cluster Set 2 - io partitioning (slave) MySQL Server Web Sharding MySQL (log) Server Web （ - horizontal (master) MySQL level parallelismlevel Fusion Server Web Cluster Cluster Set 3 - io (slave) MySQL Server Web MySQL (log) （ Server Web vertical … Server Web (master) MySQL Fusion Cluster Cluster Set - io Server Web (slave) MySQL … 32 MySQL (log) Server Web Layer Analytics Layer Storage Live database database asynchronously. masterthe arefromthat replicatedDBs by replica out operationscarriedJOINis including process Data mining

Partitioning (master) MySQL Fusion - io Cluster Set 1 (slave) MySQL replication DataWarehouse Systems Statisticaland Analysis Systems MySQL (log) (master) MySQL Fusion Environment Data Data Mining Cluster Cluster Set 2 - io (slave) MySQL Sharding MySQL (log) (master) MySQL Fusion Cluster Cluster Set 3 - io (slave) MySQL Support DB Customer MySQL (log) … (master) MySQL Fusion Cluster Cluster Set - io (slave) MySQL 32 MySQL (log) MySQL is working now. But… Operation can be highly inefficient and unprofitable

Not so Large Capacity per scalable maintenance machine is cost not so high RDB functionalities are appropriate for C-A requirements, but it is too fat. 3 Overheads that decrease RDB’s performance

Overhead-1: Too Generic Structure

Overhead-2: Not utilizing data access pattern in games

Overhead-3: Not utilizing modern hardware Overhead-1: Too Generic Structure! Data Model External External External supported Schema-1 Schema-2 Schema-n by Apps

Data Model supported by DBMS Conceptual Schema CREATE TABLE(…);

Data Model supported by storage Internal Schema TupleTableSlot *rec; Overhead-1:Too Generic Structure! SQL

● Planner (logical optimization) Planner ● Rewrites queries for better Internal declarative performance, by analyzing expression queries as relational calculus. ● Executor (physical Executer optimization) Procedural expression of relational algebra ● Executes query primitives such as relational algebras. ● Storage Subsystem Storage ● Reads and writes disk storages. I/O operations Let’s resolve Overhead-2: Not utilizing data access pattern in games Insert Update Select Real-time Consistency Query Type

Legacy Web Small Small Large No Strong Dynamic (e-commerce) Consistency (mutable) SNS(Large Scale Large Small Massively No Eventual Dynamic Web Apps) Large Consistency (mutable) IoT (Stream DB) Massively no Small Yes Application- Static Large Dependent (immutable) Mobile Game Small Massively Massively Yes Strong Static Large Large Consistency (immutable) We faces technical requirements that are essentially different from a conventional DBs Let’s resolve Overhead-3: Not utilizing modern hardware

The throughput performance becomes from 10 to 20 times faster, due to the multi-core CPU with the high parallelism and a SSD with the highly parallel access. Solution Query Pre-Compilation (Immutable Query)

Lower Footprint Lock-Free Thread query processing Scheduling Static Query Analysis is effective If we can know all the query to be processed, we can apply holistic optimization by analyzing their data access pattern and implicit race condition among them.

Query Query CySQL Params Params Query Query query query Params Immutable Params App Query Query Queries (Statically query query Params Params Server analyzed and query query compiled queries) Legacy Query Model

● Legacy query model cannot optimize holistic query, because queries are submitted dynamically.

SQL App SQL Legacy DB SQL Server Immutable Query Immutable query model requires the all queries to be defined at deploy time Deploy-Time Runtime

Query Query CySQL Params Params Query Query query query Params Immutable Params App Query Query Queries (Statically query query Params Params Server analyzed and query query compiled queries) compiling SQL into executable machine code. We can bypass the planner We by and the executor, Controlling Controlling directly manager transaction from the application, nearly we can achieve 10 than times performance SQL faster Planner Executer

Immutable

Query Apps

Storage SQL parsing parsing SQL and optimizing large cost takes Query Processor Bypassing

Introducing query pre-compilation and bypassing planner and executor

executor SQL executable storage planner SELECT * 0100100100101 FROM 0011101010010 storage table; 011101100011 SQL pre-compilation Bypassing SQL modules Lock-Free Transaction Processing Legacy RDB embraces a lot of overhead of sync inside the DB engine for lock mechanism. Each of those overheads is small, but small overhead will be accumulated, so each overhead is the cost which can't be ignored.

Row-level lock Snapshot creation for MVCC Buffer lock Can we realize Lock-Free? By using the thread scheduling specialized in the data access pattern specific to the game, multiple threads read/writes databases without locks. query I know • race condition, everything • dependency, about • priority Database queries between queries query Basic Concept of Lock-Free Thread Scheduling Query-1 Query-2 Query-3 Query-n Job Queue Job Queue Job Queue Job Queue

One query Job The scheduler recognizes dependency constructs one Job between jobs. It re-orders jobs according to queue. the those race conditions and atomicity.

Queueing Job Phase Job Depends on Job Job Job Job Depends on Job

Gate Keeper

Execution Job Job Job Job Phase Worker Thread Pool In the immutable query model, DB engine can schedule threads perfectly without synchronization lock.

● Because the DB engine can analyze race condition between queries statically. ● The thread scheduling strategy is easy: “Do not schedule the queries that cannot be executed simultaneously.” The number of CPU cores are increasing for both energy and performance efficiency.

10

8

4 image source: image source: https://www.qualcomm. http://mediatek- com/sites/ember/files/st helio.com/x20/img/X20. The Number of Cores of Number The yles/optimize/public/co png mponents/one-column- hdi/bottom/chip-tiny- 2 2 625_1.png?itok=RfSGm x-s

2010 2011 2012 2014 2015 Parallelism in I/O Devices

NAND Controller

NAND

NAND

HDD SSD Not so parallel Highly Parallel Parallelism in I/O Devices Only this Every chip parts NAND Controller can (head) can read/write read/write NAND data in data parallel NAND

HDD SSD Not so parallel Highly Parallel Comparison of the prototype and MySQL: UPDATE command 350 Executing UPDATE operations by using a 300 randomly generated value. 291

） 250

sec 200 （ 150 144 100

50 74

Throughput 3 6.3 10.4 0 500000 1000000 2000000 Update Count

Prototype MySQL Theoretical Conclusion Increase the number of threads as many as you can Implementation!

● Here I introduce a quick implementation method of read- only database engine. Use Case: https://shadowverse-portal.com/ Fast Search

Engine logical disjunction conjunction logical Query type detection

Those are sub-queries corresponds to the immutable queries Query type detection

When the target data is fixed, we can execute those sub queries before the runtime. Data Structure Each block represents an Attributes array that stores results of the specific queries … applied to the specific attributes. … Query … Results … … Queries Conclusion • You can implement tiny RDB within a hour.

• Increase the number of threads as many as you can Recommended Books

● Database Systems: The Complete Book (2nd Edition) 2nd Edition by Hector Garcia- Molina, Jeffrey D. Ullman, Jennifer Widom ● Introduction to Algorithms, 3rd Edition (MIT Press) 3rd Edition by Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, Clifford Stein