DOCSLIB.ORG

Spanner: Becoming a SQL System

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Spanner: Becoming a SQL System Spanner: Becoming a SQL System David F. Bacon Nathan Bales Nico Bruno Brian F. Cooper Adam Dickinson Andrew Fikes Campbell Fraser Andrey Gubarev Milind Joshi Eugene Kogan Alexander Lloyd Sergey Melnik Rajesh Rao David Shue Christopher Taylor Marcel van der Holst Dale Woodford Google, Inc. ABSTRACT this paper, we focus on the “database system” aspects of Spanner, Spanner is a globally-distributed data management system that in particular how query execution has evolved and forced the rest backs hundreds of mission-critical services at Google. Spanner of Spanner to evolve. Most of these changes have occurred since is built on ideas from both the systems and database communi- [5] was written, and in many ways today’s Spanner is very different ties. The first Spanner paper published at OSDI’12 focused on the from what was described there. systems aspects such as scalability, automatic sharding, fault tol- A prime motivation for this evolution towards a more “database- erance, consistent replication, external consistency, and wide-area like” system was driven by the experiences of Google developers distribution. This paper highlights the database DNA of Spanner. trying to build on previous “key-value” storage systems. The pro- We describe distributed query execution in the presence of reshard- totypical example of such a key-value system is Bigtable [4], which ing, query restarts upon transient failures, range extraction that continues to see massive usage at Google for a variety of applica- drives query routing and index seeks, and the improved blockwise- tions. However, developers of many OLTP applications found it columnar storage format. We touch upon migrating Spanner to the difficult to build these applications without a strong schema sys- common SQL dialect shared with other systems at Google. tem, cross-row transactions, consistent replication and a powerful query language. The initial response to these difficulties was to build transaction processing systems on top of Bigtable; an exam- 1. INTRODUCTION ple is Megastore [2]. While these systems provided some of the Google’s Spanner [5] started out as a key-value store offering multi- benefits of a database system, they lacked many traditional database row transactions, external consistency, and transparent failover features that application developers often rely on. A key example across datacenters. Over the past 7 years it has evolved into a rela- is a robust query language, meaning that developers had to write tional database system. In that time we have added a strongly-typed complex code to process and aggregate the data in their applica- schema system and a SQL query processor, among other features. tions. As a result, we decided to turn Spanner into a full featured Initially, some of these database features were “bolted on” – the SQL system, with query execution tightly integrated with the other first version of our query system used high-level APIs almost like architectural features of Spanner (such as strong consistency and an external application, and its design did not leverage many of the global replication). Spanner’s SQL interface borrows ideas from unique features of the Spanner storage architecture. However, as the F1 [9] system, which was built to manage Google’s AdWords we have developed the system, the desire to make it behave more data, and included a federated query processor that could access like a traditional database has forced the system to evolve. In par- Spanner and other data sources. ticular, Today, Spanner is widely used as an OLTP database management system for structured data at Google, and is publicly available in • The architecture of the distributed storage stack has driven beta as Cloud Spanner1 on the Google Cloud Platform (GCP). Cur- fundamental changes in our query compilation and execu- rently, over 5,000 databases run in our production instances, and tion, and are used by teams across many parts of Google and its parent com- pany Alphabet. This data is the “source of truth” for a variety of • The demands of the query processor have driven fundamental mission-critical Google databases, incl. AdWords. One of our large changes in the way we store and manage data. users is the Google Play platform, which executes SQL queries to These changes have allowed us to preserve the massive scala- manage customer purchases and accounts. Spanner serves tens of bility of Spanner, while offering customers a powerful platform for millions of QPS across all of its databases, managing hundreds of database applications. We have previously described the distributed petabytes of data. Replicas of the data are served from datacenters architecture and data and concurrency model of Spanner [5]. In around the world to provide low latency to scattered clients. De- spite this wide replication, the system provides transactional con- sistency and strongly consistent replicas, as well as high availabil- Permission to make digital or hard copies of part or all of this work for personal or ity. The database features of Spanner, operating at this massive classroom use is granted without fee provided that copies are not made or distributed scale, make it an attractive platform for new development as well for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. as migration of applications from existing data stores, especially For all other uses, contact the owner/author(s). for “big” customers with lots of data and large workloads. Even SIGMOD’17, May 14–19, 2017, Chicago, IL, USA “small” customers benefit from the robust database features, strong c 2017 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-4197-4/17/05. 1 DOI: http://dx.doi.org/10.1145/3035918.3056103 http://cloud.google.com/spanner consistency and reliability of the system. “Small” customers often parts ‘K’ are stored physically next to the parent row whose key is become “big”. ‘K’. Shard boundaries are specified as ranges of key prefixes and The Spanner query processor implements a dialect of SQL, preserve row co-location of interleaved child tables. called Standard SQL, that is shared by several query subsystems Spanner’s transactions use a replicated write-ahead redo log, and within Google (e.g., Dremel/BigQuery OLAP systems2). Standard the Paxos [6] consensus algorithm is used to get replicas to agree SQL is based on standard ANSI SQL, fully using standard features on the contents of each log entry. In particular, each shard of the such as ARRAY and row type (called STRUCT) to support nested database is assigned to exactly one Paxos group (replicated state data as a first class citizen (see Section6 for more details). machine). A given group may be assigned multiple shards. All Like the lower storage and transactional layers, Spanner’s query transactions that involve data in a particular group write to a logical processor is built to serve a mix of transactional and analyti- Paxos write-ahead log, which means each log entry is committed cal workloads, and to support both low-latency and long-running by successfully replicating it to a quorum of replicas. Our imple- queries. Given the distributed nature of data storage in Spanner, the mentation uses a form of Multi-Paxos, in which a single long-lived query processor is itself distributed and uses standard optimization leader is elected and can commit multiple log entries in parallel, techniques such as shipping code close to data, parallel process- to achieve high throughput. Because Paxos can make progress as ing of parts of a single request on multiple machines, and partition long as a majority of replicas are up, we achieve high availability pruning. While these techniques are not novel, there are certain as- despite server, network and data center failures [3]. pects that are unique to Spanner and are the focus of this paper. In We only replicate log records – each group replica receives log particular: records via Paxos and applies them independently to its copy of the group’s data. Non-leader replicas may be temporarily behind • We examine techniques we have developed for distributed the leader. Replicas apply Paxos log entries in order, so that they query execution, including compilation and execution of have always applied a consistent prefix of the updates to data in that joins, and consuming query results from parallel workers group. (Section3). Our concurrency control uses a combination of pessimistic lock- ing and timestamps. For blind write and read-modify-write transac- • We describe range extraction, which is how we decide which tions, strict two-phase locking ensures serializability within a given Spanner servers should process a query, and how to mini- Paxos group, while two-phase commits (where different Paxos mize the row ranges scanned and locked by each server (Sec- groups are participants) ensure serializability across the database. tion4). Each committed transaction is assigned a timestamp, and at any • We discuss how we handle server-side failures with query timestamp T there is a single correct database snapshot (reflect- restarts (Section5). Unlike most distributed query proces- ing exactly the set of transactions whose commit timestamp ≤ T). sors, Spanner fully hides transient failures during query exe- Reads can be done in lock-free snapshot transactions, and all data cution. This choice results in simpler applications by allow- returned from all reads in the same snapshot transaction comes ing clients to omit sometimes complex and error-prone retry from a consistent snapshot of the database at a specified times- loops. Moreover, transparent restarts enable rolling upgrades tamp. Stale reads choose a timestamp in the past to increase the of the server software without disrupting latency-sensitive chance that the nearby replica is caught up enough to serve the read.
Load more

Recommended publications
  • F1 Query: Declarative Querying at Scale
    F1 Query: Declarative Querying at Scale Bart Samwel John Cieslewicz Ben Handy Jason Govig Petros Venetis Chanjun Yang Keith Peters Jeff Shute Daniel Tenedorio Himani Apte Felix Weigel David Wilhite Jiacheng Yang Jun Xu Jiexing Li Zhan Yuan Craig Chasseur Qiang Zeng Ian Rae Anurag Biyani Andrew Harn Yang Xia Andrey Gubichev Amr El-Helw Orri Erling Zhepeng Yan Mohan Yang Yiqun Wei Thanh Do Colin Zheng Goetz Graefe Somayeh Sardashti Ahmed M. Aly Divy Agrawal Ashish Gupta Shiv Venkataraman Google LLC [email protected] ABSTRACT 1. INTRODUCTION F1 Query is a stand-alone, federated query processing platform The data processing and analysis use cases in large organiza- that executes SQL queries against data stored in different file- tions like Google exhibit diverse requirements in data sizes, la- based formats as well as different storage systems at Google (e.g., tency, data sources and sinks, freshness, and the need for custom Bigtable, Spanner, Google Spreadsheets, etc.). F1 Query elimi- business logic. As a result, many data processing systems focus on nates the need to maintain the traditional distinction between dif- a particular slice of this requirements space, for instance on either ferent types of data processing workloads by simultaneously sup- transactional-style queries, medium-sized OLAP queries, or huge porting: (i) OLTP-style point queries that affect only a few records; Extract-Transform-Load (ETL) pipelines. Some systems are highly (ii) low-latency OLAP querying of large amounts of data; and (iii) extensible, while others are not. Some systems function mostly as a large ETL pipelines. F1 Query has also significantly reduced the closed silo, while others can easily pull in data from other sources.
    [Show full text]
  • Containers at Google
    Build What’s Next A Google Cloud Perspective Thomas Lichtenstein Customer Engineer, Google Cloud [email protected] 7 Cloud products with 1 billion users Google Cloud in DACH HAM BER ● New cloud region Germany Google Cloud Offices FRA Google Cloud Region (> 50% latency reduction) 3 Germany with 3 zones ● Commitment to GDPR MUC VIE compliance ZRH ● Partnership with MUC IoT platform connects nearly Manages “We found that Google Ads has the best system for 50 brands 250M+ precisely targeting customer segments in both the B2B with thousands of smart data sets per week and 3.5M and B2C spaces. It used to be hard to gain the right products searches per month via IoT platform insights to accurately measure our marketing spend and impacts. With Google Analytics, we can better connect the omnichannel customer journey.” Conrad is disrupting online retail with new Aleš Drábek, Chief Digital and Disruption Officer, Conrad Electronic services for mobility and IoT-enabled devices. Solution As Conrad transitions from a B2C retailer to an advanced B2B and Supports B2C platform for electronic products, it is using Google solutions to grow its customer base, develop on a reliable cloud infrastructure, Supports and digitize its workplaces and retail stores. Products Used 5x Mobile-First G Suite, Google Ads, Google Analytics, Google Chrome Enterprise, Google Chromebooks, Google Cloud Translation API, Google Cloud the IoT connections vs. strategy Vision API, Google Home, Apigee competitors Industry: Retail; Region: EMEA Number of Automate Everything running
    [Show full text]
  • Spanner: Becoming a SQL System
    Spanner: Becoming a SQL System David F. Bacon Nathan Bales Nico Bruno Brian F. Cooper Adam Dickinson Andrew Fikes Campbell Fraser Andrey Gubarev Milind Joshi Eugene Kogan Alexander Lloyd Sergey Melnik Rajesh Rao David Shue Christopher Taylor Marcel van der Holst Dale Woodford Google, Inc. ABSTRACT this paper, we focus on the “database system” aspects of Spanner, Spanner is a globally-distributed data management system that in particular how query execution has evolved and forced the rest backs hundreds of mission-critical services at Google. Spanner of Spanner to evolve. Most of these changes have occurred since is built on ideas from both the systems and database communi- [5] was written, and in many ways today’s Spanner is very different ties. The first Spanner paper published at OSDI’12 focused on the from what was described there. systems aspects such as scalability, automatic sharding, fault tol- A prime motivation for this evolution towards a more “database- erance, consistent replication, external consistency, and wide-area like” system was driven by the experiences of Google developers distribution. This paper highlights the database DNA of Spanner. trying to build on previous “key-value” storage systems. The pro- We describe distributed query execution in the presence of reshard- totypical example of such a key-value system is Bigtable [4], which ing, query restarts upon transient failures, range extraction that continues to see massive usage at Google for a variety of applica- drives query routing and index seeks, and the improved blockwise- tions. However, developers of many OLTP applications found it columnar storage format.
    [Show full text]
  • Economic and Social Impacts of Google Cloud September 2018 Economic and Social Impacts of Google Cloud |
    Economic and social impacts of Google Cloud September 2018 Economic and social impacts of Google Cloud | Contents Executive Summary 03 Introduction 10 Productivity impacts 15 Social and other impacts 29 Barriers to Cloud adoption and use 38 Policy actions to support Cloud adoption 42 Appendix 1. Country Sections 48 Appendix 2. Methodology 105 This final report (the “Final Report”) has been prepared by Deloitte Financial Advisory, S.L.U. (“Deloitte”) for Google in accordance with the contract with them dated 23rd February 2018 (“the Contract”) and on the basis of the scope and limitations set out below. The Final Report has been prepared solely for the purposes of assessment of the economic and social impacts of Google Cloud as set out in the Contract. It should not be used for any other purposes or in any other context, and Deloitte accepts no responsibility for its use in either regard. The Final Report is provided exclusively for Google’s use under the terms of the Contract. No party other than Google is entitled to rely on the Final Report for any purpose whatsoever and Deloitte accepts no responsibility or liability or duty of care to any party other than Google in respect of the Final Report and any of its contents. As set out in the Contract, the scope of our work has been limited by the time, information and explanations made available to us. The information contained in the Final Report has been obtained from Google and third party sources that are clearly referenced in the appropriate sections of the Final Report.
    [Show full text]
  • Spanner: Google's Globally-Distributed Database
    Spanner: Google's Globally-Distributed Database Corbett, Dean, et al. Jinliang Wei CMU CSD October 20, 2013 Jinliang Wei (CMU CSD) Spanner: Google's Globally-Distributed Database Corbett,October 20,Dean, 2013 et 1 al./ 21 What? - Key Features I Globally distributed I Versioned data I SQL transactions + key-value read/writes I External consistency I Automatic data migration across machines (even across datacenters) for load balancing and fautl tolerance. Jinliang Wei (CMU CSD) Spanner: Google's Globally-Distributed Database Corbett,October 20,Dean, 2013 et 2 al./ 21 External Consistency I Equivalent to linearizability I If a transaction T1 commits before another transaction T2 starts, then T1's commit timestamp is smaller than T 2. I Any read that sees T2 must see T1. I The strongest consistency guarantee that can be achieved in practice (Strict consistency is stronger, but not achievable in practice). Jinliang Wei (CMU CSD) Spanner: Google's Globally-Distributed Database Corbett,October 20,Dean, 2013 et 3 al./ 21 Why Spanner? I BigTable I Good performance I Does not support transaction across rows. I Hard to use. I Megastore I Support SQL transactions. I Many applications: Gmail, Calendar, AppEngine... I Poor write throughput. I Need SQL transactions + high throughput. Jinliang Wei (CMU CSD) Spanner: Google's Globally-Distributed Database Corbett,October 20,Dean, 2013 et 4 al./ 21 Spanserver Software Stack Figure: Spanner Server Software Stack Jinliang Wei (CMU CSD) Spanner: Google's Globally-Distributed Database Corbett,October 20,Dean, 2013 et 5 al./ 21 Spanserver Software Stack Cont. I Spanserver maintains data and serves client requests.
    [Show full text]
  • Scalable Crowd-Sourcing of Video from Mobile Devices
    Scalable Crowd-Sourcing of Video from Mobile Devices Pieter Simoens†, Yu Xiao‡, Padmanabhan Pillai•, Zhuo Chen, Kiryong Ha, Mahadev Satyanarayanan December 2012 CMU-CS-12-147 School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 †Carnegie Mellon University and Ghent University ‡Carnegie Mellon University and Aalto University •Intel Labs Abstract We propose a scalable Internet system for continuous collection of crowd-sourced video from devices such as Google Glass. Our hybrid cloud architecture for this system is effectively a CDN in reverse. It achieves scalability by decentralizing the cloud computing infrastructure using VM-based cloudlets. Based on time, location and content, privacy sensitive information is automatically removed from the video. This process, which we refer to as denaturing, is executed in a user-specific Virtual Machine (VM) on the cloudlet. Users can perform content-based searches on the total catalog of denatured videos. Our experiments reveal the bottlenecks for video upload, denaturing, indexing and content-based search and provide valuable insight on how parameters such as frame rate and resolution impact the system scalability. Copyright 2012 Carnegie Mellon University This research was supported by the National Science Foundation (NSF) under grant numbers CNS-0833882 and IIS-1065336, by an Intel Science and Technology Center grant, by the Department of Defense (DoD) under Contract No. FA8721-05-C-0003 for the operation of the Software Engineering Institute (SEI), a federally funded research and development center, and by the Academy of Finland under grant number 253860. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily represent the views of the NSF, Intel, DoD, SEI, Academy of Finland, University of Ghent, Aalto University, or Carnegie Mellon University.
    [Show full text]
  • Develop and Deploy Apps More Easily with Cloud Spanner and Cloud Bigtable Updates | Google Cloud Blog
    8/23/2020 Develop and deploy apps more easily with Cloud Spanner and Cloud Bigtable updates | Google Cloud Blog Blog Menu D ATAB ASES Develop and deploy apps more easily with Cloud Spanner and Cloud Bigtable updates FinDde aenp tai rStricivlaes..t.ava Product Manager, Cloud Spanner Misha Brukman Latest storiPersoduct Manager, Cloud Bigtable October 11, 2018 Products Topics About One of our goals at Google Cloud is to offer all the database choices that you want as managed services in order to remove operational complexity and toil from your day. RSS Feed We’ve got a full range of managed database services to address a variety of your workload needs. We’re always working to add features to our databases to make your https://cloud.google.cwomo/rbklogb/puroildduicntsg/datapbpassese/daesveielorp-and-deploy-apps-more-easily-with-cloud-spanner-and-cloud-bigtable-updates 1/6 8/23/2020 Develop and deploy apps more easily with Cloud Spanner and Cloud Bigtable updates | Google Cloud Blog work building apps easier. Today at Next London ‘18, we’re excited to announce the general availability of two important new features for two of our cloud-native database offerings: Cloud Spanner Blog and Cloud Bigtable. Menu Cloud Spanner: We’re adding enhancements to Cloud Spanner’s SQL capabilities to make it easier to read and write data in Cloud Spanner databases using SQL, and use off-the-shelf drivers and tooling. Cloud Spanner’s API now supports INSERT, UPDATE, and DELETE SQL statements. Cloud Bigtable: We’re offering Key Visualizer so you can see key access patterns in heatmap format to optimize your Cloud Bigtable schemas for improved performance.
    [Show full text]
  • Trusting Your Data with Google Cloud Platform 2
    Google Cloud whitepaper September 2019 Trusting your data with Google Cloud Platform 2 Table of contents 1. Introduction 3 2. Managing your data lifecycle on GCP 4 2.1 Data usage 2.2 Data export/archive/backup 2.3 Data governance 2.4 Data residency 2.5 Security configuration management 2.6 Third party security solutions 2.7 Incident detection & response 3. Managing Google's access to your data 9 3.1 How does Google safeguard your data from unauthorized access? 3.2 Data export/archive/backup 3.2 Customer controls over Google access to data 3.3 Data access transparency 3.4 Google employee access authorization 3.5 What happens if we get a lawful request from a government for data? 4. Security and compliance standards 13 4.1 Independent verification of our control framework 4.2 Compliance support for customers Conclusion 14 Appendix: URLs 15 The information contained herein is intended to outline general product direction and should not be relied upon in making purchasing decisions nor shall it be used to trade in the securities of Alphabet Inc. The content is for informational purposes only and may not be incorporated into any contract. The information presented is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Any references to the development, release, and timing of any features or functionality described for these services remains at Google’s sole discretion. Product capabilities, time frames and features are subject to change and should not be viewed as Google commitments. 3 1. Introduction At Google Cloud we’ve set a high bar for what it means to host, serve, and protect customer data.
    [Show full text]
  • How to Integrate Google Cloud Spanner with Denodo
    How to integrate Google Cloud Spanner with Denodo Revision 20210421 NOTE This document is confidential and proprietary of Denodo Technologies. No part of this document may be reproduced in any form by any means without prior written authorization of Denodo Technologies. Copyright © 2021 Denodo Technologies Proprietary and Confidential How to integrate Google Cloud Spanner with Denodo 20210421 2 of 15 CONTENTS 1 INTRODUCTION.................................................................2 1.1 SAMPLE GOOGLE CLOUD SPANNER DATABASE.................................3 2 CREATING A SERVICE ACCOUNT AND KEY IN GOOGLE CLOUD SPANNER............................................................................4 3 CONNECTING TO GOOGLE CLOUD SPANNER USING THE JDBC DRIVER...............................................................................8 3.1 DOWNLOAD THE JDBC DRIVER........................................................8 3.2 INSTALL THE JDBC DRIVER IN THE DENODO PLATFORM....................8 3.3 CONNECT TO GOOGLE CLOUD SPANNER........................................10 3.4 CREATE A JDBC DATA SOURCE.......................................................10 3.5 CREATE A BASE VIEW...................................................................13 4 REFERENCES...................................................................16 1 INTRODUCTION Spanner is a NewSQL database designed, built, and deployed at Google for OLTP systems. It is a fully managed, mission-critical, relational database service that offers transactional consistency at a global
    [Show full text]
  • Spanner: Google's Globally- Distributed Database
    Today’s Reminders • Discuss Project Ideas with Phil & Kevin – Phil’s Office Hours: After class today – Sign up for a slot: 11-12:30 or 3-4:20 this Friday Spanner: Google’s Globally-Distributed Database Phil Gibbons 15-712 F15 Lecture 14 2 Spanner: Google’s Globally- Database vs. Key-value Store Distributed Database [OSDI’12 best paper] “We provide a database instead of a key-value store to make it easier for programmers to write their applications.” James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, JJ Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, “We consistently received complaints from users that Bigtable Peter Hochschild, Wilson Hsieh, Sebastian Kanthak, can be difficult to use for some kinds of applications.” Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak, Christopher Taylor, Ruth Wang, Dale Woodford (Google x 26) 3 4 Spanner [Slides from OSDI’12 talk] • Worked on Spanner for 4½ years at time of OSDI’12 • Scalable, multi-version, globally-distributed, Spanner: Google’s synchronously-replicated database – Hundreds of datacenters, millions of machines, trillions of rows Globally-Distributed Database • Transaction Properties – Transactions are externally-consistent (a.k.a. Linearizable) Wilson Hsieh – Read-only transactions are lock-free representing a host of authors – (Read-write transactions use 2-phase-locking) OSDI 2012 • Flexible replication configuration 5 What is Spanner?
    [Show full text]
  • Streaming Integration for Google Cloud Spanner
    Streaming Integration for Google Cloud Spanner Spanner needs an enterprise-grade streaming data integration solution BENEFITS that complements its global scalability, transactional consistency, and • Ingest real-time data for high availability advantages. The Striim® data integration platform operational reporting and share continuously ingests business-critical data from a wide range of up-to-date Cloud Spanner data cloud or on-premises source systems and continuously moves to with other services Cloud Spanner. Striim enterprise-grade platform includes end-to-end capabilities for real-time ingestion, in-flight processing, and continuous • Comply with regulations and delivery with sub-second latency. With real-time data feeds from the requirements around data Striim platform, high-value transactional and operational workloads privacy and security via in-flight can thrive on Cloud Spanner. encryption, data masking, and filtering Available on the Google Cloud Platform as well as on premises, Striim comes with change data capture (CDC) capabilities for non-intrusive • Migrate databases with minimal data ingestion from major relational databases. It ingests real-time risk, without downtime or data loss with real-time data pipeline change data from Oracle, SQL Server, HPE NonStop, MySQL, PostgreSQL, monitoring and alerts and MariaDB without impacting the performance of the source systems. Striim also moves data from log files, message queues such as Kafka, • Easily offload operational sensors, Hadoop, and NoSQL databases in real time. workloads to the cloud by moving data in real time and Online Database Migration to Cloud Spanner in the desired format With real-time data synchronization capabilities, Striim enables Cloud Spanner customers to achieve a seamless, online database migration.
    [Show full text]
  • This Document Describes Best Practices for Using Spanner As the Primary Backend Database for Game State Storage
    1/25/2020 Best practices for using Cloud Spanner as a gaming database This document describes best practices for using Spanner as the primary backend database for game state storage. You can use Spanner (/spanner/) in place of common databases to store player authentication data and inventory data. This document is intended for game backend engineers working on long-term state storage, and game infrastructure operators and admins who support those systems and are interested in hosting their backend database on Google Cloud. Multiplayer and online games have evolved to require increasingly complex database structures for tracking player entitlements, state, and inventory data. Growing player bases and increasing game complexity have led to database solutions that are a challenge to scale and manage, frequently requiring the use of sharding (https://wikipedia.org/wiki/Shard_(database_architecture)) or clustering (https://wikipedia.org/wiki/MySQL_Cluster). Tracking valuable in-game items or critical player progress typically requires transactions and is challenging to work around in many types of distributed databases. Spanner is the rst scalable, enterprise-grade, globally distributed, and strongly consistent database service built for the cloud to combine the benets of relational database structure with non-relational horizontal scale. Many game companies have found it to be well-suited to replace both game state and authentication databases in production-scale systems. You can scale for additional performance or storage by using the Cloud Console to add nodes. Spanner can transparently handle global replication with strong consistency, eliminating your need to manage regional replicas. This best practices document discusses the following: Important Spanner concepts and differences from databases commonly used in games.
    [Show full text]