Hands on Google Cloud SQL and Cloud Spanner Deployment, Administration and Use Cases with Python

Hands On Google Cloud SQL and Cloud Spanner Deployment, Administration and Use Cases with Python

Navin Sabharwal Shakuntala Gupta Edward Hands On Google Cloud SQL and Cloud Spanner Navin Sabharwal Shakuntala Gupta Edward New Delhi, India Ghaziabad, India

ISBN-13 (pbk): 978-1-4842-5536-0 ISBN-13 (electronic): 978-1-4842-5537-7 https://doi.org/10.1007/978-1-4842-5537-7 Copyright © 2020 by Navin Sabharwal, Shakuntala Gupta Edward This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: Celestin Suresh John Development Editor: Siddhi Chavan Coordinating Editor: Aditee Mirashi Cover designed by eStudioCalamar Cover image designed by Freepik (www.freepik.com) Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer- sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail [email protected], or visit http://www.apress.com/ rights-permissions. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the book's product page, located at www.apress.com/978-1-4842-5536-0. For more detailed information, please visit http://www.apress.com/source-code. Printed on acid-free paper Dedicated to the people I love and the God I trust. —Navin Sabharwal Dedicated to my family for being my rock and guiding light and to my God who shades every step of my life. —Shakuntala Gupta Edward Table of Contents

About the Authors �� ix About the Technical Reviewer �� xi Acknowledgments �� xiii Introduction ��xv

Chapter 1: Getting Started with GCP �� 1 Signing Up �� 3 Accessing Google Cloud Platform �� 6 Cloud Console �� 7 Cloud Shell �� 10 Cloud SDK �� 18 Project �� 18 Ownership: Projects and Billing �� 19 Projects and Isolations �� 20 Project as Namespace �� 20 Create a Project �� 23 Summary �� 26

Chapter 2: Cloud SQL �� 27 Relational Databases �� 27 Primary Key �� 29 Foreign Key �� 29 Constraints �� 30 Operations �� 31 Getting Started with Cloud SQL �� 36 Provision a MySQL Instance �� 37

v Table of Contents

Provisioning a PostgreSQL Instance �� 50 Summary �� 55

Chapter 3: Working with CloudSQL �� 57 Getting Started in Cloud Shell �� 57 Connect to the Database �� 58 Create Database �� 60 Create Table �� 61 Insert and Select �� 62 Load and Query Sample Dataset �� 63 Python Program - Feast Out �� 77 Database Schema �� 78 Provision Compute Instance �� 82 Set Up Python Development Environment �� 86 Enable APIs �� 87 Create Service Account �� 90 Install and Configure Cloud SDK �� 94 Python Program �� 103 Run and Verify �� 113 Wrap Up �� 126 Summary �� 128

Chapter 4: Administering CloudSQL �� 129 Authorization and Security �� 132 Authorized Network �� 132 Cloud Proxy �� 138 Private IP Address �� 139 Authorization �� 143 Backups and Restores �� 150 CloudSQL MySQL - Binary Logging �� 155 Maintenance �� 156 Managing Resources �� 159 vi Table of Contents

High Availability �� 166 Replication �� 166 Enable High Availability �� 172 Read Replicas �� 176 Logging �� 179 Summary �� 183 Chapter 5: Cloud Spanner �� 185 New in NewSQL �� 186 Origins of Cloud Spanner �� 187 Google Cloud Spanner �� 188 Spanner and CAP Theorem �� 189 Best Fit �� 192 Development Support �� 192 Summary �� 193 Chapter 6: Cloud Spanner Explained �� 195 Instance �� 195 Data Modeling �� 199 Databases and Tables �� 199 Primary Keys �� 200 Interleaving �� 203 Splits �� 206 Secondary Indexes �� 207 Replication �� 209 Transactions �� 213 Read/Write �� 213 Read Only �� 215 Handling Multiple Transactions �� 217 Distributed Transactions Explained �� 224 Writes �� 225 Reads �� 228 Summary �� 228 vii Table of Contents

Chapter 7: Getting Started with Spanner �� 229 Provisioning CloudSpanner Instance �� 229 Getting Started in Cloud Console �� 241 Connect to the Spanner Instance �� 242 Create Database �� 242 Create Schema �� 244 Reading and Writing Data �� 264 Python Program – Feast Out �� 277 Database Schema �� 277 Provision Compute Instance �� 282 Set up Python Development Environment �� 284 Install and Configure Cloud SDK �� 284 Python Program �� 289 Run and Verify �� 299 Wrap Up �� 311 Summary �� 312 Chapter 8: Best Practices �� 313 Cloud SQL Best Practices �� 313 Plan Resources Efficiently �� 313 Maintenance Activities �� 316 MySQL-Instance Specific Pointers �� 318 Cloud Spanner Pointers �� 319 Slower Transactions with Scale Out �� 319 Migrating Workloads from Existing SQL Setups Is a Cumbersome Task �� 320 Specify Referential Integrity at Design Time �� 320 Co-Located Joins Offer Performance Benefits �� 320 Be Careful When Choosing Your Primary Key �� 320 Specify the FORCE_INDEX Directive �� 321 Multitenancy Support Availability �� 321 Summary �� 324 Index �� 325 viii About the Authors

Navin Sabharwal has more than 20 years of industry experience and is an innovator, thought leader, patent holder, and author in the areas of cloud computing, artificial intelligence and machine learning, public cloud, DevOps, AIOPS, DevOps, infrastructure services, monitoring and managing platforms, Big Data analytics, and software product development. Navin is responsible for DevOps, artificial intelligence, cloud lifecycle management, service management, monitoring and management, IT ops analytics, AIOPs and machine learning, automation, operational efficiency of scaled delivery through Lean ops, and strategy and delivery for HCL Technologies. He is reachable at Navinsabharwal@gmail.com and https://www.linkedin.com/in/ navinsabharwal.

Shakuntala Gupta Edward is an accomplished consultant in the areas of data and analytics with more than 16 years of experience. Shakuntala is a Big Data architect and is responsible for database design, database architecture, best practices for Big Data technologies, product development using databases, Big Data, NoSQL, analytics, and machine learning technologies. She has authored books on Big Data analytics, including Practical MongoDB from Apress. She is reachable at https://in.linkedin.com/in/shakuntala- edward-70343627/.

ix About the Technical Reviewer

Piyush Pandey works as a deputy general manager in the HCL DRYiCE practice, focusing on creating solutions catering to cloud adoption (including cloud landing zone, migration, and operations), automation, orchestration, and cloud lifecycle management.

xi Acknowledgments

To my family, Shweta and Soumil, for being always by my side and letting me sacrifice their time for my intellectual and spiritual pursuits. For taking care of everything while I am immersed in authoring. This and the other accomplishments of my life wouldn't have been possible without your love and support. To my mom and sister for their love and support as always; without your blessings, nothing is possible. To my co-author, Shakutala. This is the third book we authored together and the journey over the last 10 years has been enriching. Thank you for your hard work and quick turnarounds. To my team here at HCL who has been a source of inspiration with their hard work, ever engaging technical conversations, and their technical depth. Your everflowing ideas are a source of happiness and excitement every single day. Piyush Pandey, Sarvesh Pandey, Amit Agrawal, Vasand Kumar, Punith Krishnamurthy, Sandeep Sharma, Amit Dwivedi, Gauarv Bhardwaj, Nitin Narotra, Shoyeb, Divjot, Nitin Chand, and Vivek— thank you for being there and making technology fun. To Celestine and Aditee and the entire team at Apress, for turning our ideas into a reality. It has been an amazing experience authoring with you over the years. The speed of the decision making and the editorial support have both been excellent. To all my co-authors, colleagues, managers, mentors, and guides, it was and is an enriching experience to be associated with you and to learn from you. Thank you goddess Saraswati, for guiding us to the path of knowledge and spirituality. असतो मा साद गमय, तमसो मा ज्योतिर् गमय, मृत्योर मा अमृतम् गमय (Asato Ma Sad Gamaya, Tamaso Ma Jyotir Gamaya, Mrityor Ma Amritam Gamaya) Lead us from ignorance to truth, lead us from darkness to light, lead us from death to deathlessness. —Navin Sabharwal

xiii Acknowledgments

First and foremost, my utmost thanks to my entire family. To my mom and dad (without you I am nothing), to Nitya (you are the very soul of my life, be you always), to Eddy (my husband, friend, and partner in crime), to Bindu, Eliz, and Vicky (for your love, being the energy in my life; you guys keep me grounded), to Amar, Jey, and Asper (for being there always), to my Savari mom and dad-in-law (for your support and blessings), and not to forget Ria, Asper, and Steve (my little angels). Without your unfaltering love and support, I would never be what I am today. Words cannot say enough. I would like to thank my co-author Navin for believing in me and pushing me always to deliver my best. Special mention to Ayush Aggarwal and Raja Bhattacharya, for helping me with the code in the book. Finally, I would like to give a big thanks to Celestin, Aditee, and the whole team at Apress for believing. —Shakuntala Gupta Edward

xiv Introduction

Back in 2009, a new term was coined—“Big Data”. Big Data is data that has a high volume, is generated at a high velocity, and has multiple varieties. This is data with the three Vs—Volume, Velocity, and Variety. Convergence of several technology trends fueled the progress further. On the one side, Big Data poses challenges to organizations to store, transport, secure, process, mine, and serve that data. Cloud Computing on the other hand provides support to address these challenges by provisioning shared resources, such as compute, database storage, networking, security, and analytical offerings. In effect, the combination of both is spearheading new innovations in technology. The Google Cloud Platform (GCP) is a suite of cloud computing services offered by Google. They run on the same infrastructure that Google uses internally for its own end-user products, such as Google search, Google Photos, YouTube, Gmail, etc. GCP services are well positioned for the modern application development user and have some unique offerings in the data storage, Big Data analytics, artificial intelligence, and containerization spaces. Google continues to innovate and strengthen its offerings. This book takes a deep dive into the relational database service offering of Google— Cloud SQL and Cloud Spanner. These database offerings from Google Cloud are increasingly being used by enterprises to develop their modern digital applications. The aim of this book is to provide guidance and best practices to architect applications using GCP database offerings. We have tried to cover topics that will enable developers to make informed choices and also help cloud administrators administer these databases and manage them on a daily basis. • Cloud SQL is Google’s fully managed relational database service with two choices of database engine—MySQL and PostgreSQL. • Cloud Spanner is Google’s fully managed NewSQL database service, offering high availability with an industry leading 99.999% availability SLA.

xv Introduction

Time is valuable, more so in today’s world of ever-increasing data. We thank you for spending your valuable time and investing in reading this book. We sincerely hope that you’ll find the coverage useful and will be able to architect, develop, and manage GCP database offerings. We hope that you enjoy reading the book as much as we enjoyed writing it. Wishing you the best on your cloud and Big Data journey.

xvi