Couchbase on Red Hat OpenShift How to avoid big database pain in microservices Steffen Schneider Senior Solutions Engineer +49 151 507 339 71 [email protected] linkedin.com/in/st-sch Speaker Intro • Cross disciplinary functional-technical profile • Over 10 years professional experience on end-customer side esp. executive department • Execution as Technical Lead with extended Project Management and Leadership experience • Dedicated responsibilities in the areas of o Business Intelligence o Data Warehousing o Data Processing/ETL o Processes o Technology Tools • Since 2017 on software vendor side as Solution Architect Steffen Schneider Finally switch into Big Data world to Couchbase as leading NoSQL Senior Solutions Engineer • technology platform vendor Couchbase – Central Europe 2 Agenda 1. Setting the Stage – Technology 2. Setting the Stage – Real World Use Cases 3. Challenges of today’s Corporate Architectures and its Solutions 4. Prerequisites 5. Containerized Database Handling for Microservices 6. Q&A 3 Agenda 1. Setting the Stage – Technology 2. Setting the Stage – Real World Use Cases 3. Challenges of today’s Corporate Architectures and its Solutions 4. Prerequisites 5. Containerized Database Handling for Microservices 6. Q&A 4 Couchbase – Brief Company Overview • Founded 2011 Manchester • >500 Empolyees San Francisco Paris • 70%+ new business growth London • 100%+ growth in total Santa Clara - HQ contract value • 55%+ billings growth Bangalore • 50%+ growth in average deal size Santa Clara – HQ San Francisco London Paris Manchester Bangalore 3250 Olcott Street 450 Mission Street #200 11-21 Paul Street 40 Rue du Collisée 1A Tariff St 10 Museum Road Santa Clara, CA 95054 San Francisco, CA 94105 London EC2A 4JU 75008 Paris Manchester M1 2FF Bangalore 560001 United States United States United Kingdom France United Kingdom India Couchbase Behind Today’s Business-Critical Applications Customers Application Caching & session Real-time pricing, Flight availability, Customer 360 single Real-time crew store for single view product catalog, booking, pricing view, unified notes management, scheduling inventory management analytics, etc. and resources Performance 2M+ 10M+ 15M 210M 41K reads/sec. unique SKUs ops / second documents pilots and crew 10M 35K <2.5ms 100K 148M queries/sec. requests/sec. response time users travelers in 2017 Infrastructure Developer Agility Performance at Scale Manageability Security + Availability A Proven Enterprise Solution Chosen by Industry Leaders Retail & E- Travel & Financial Media & Healthcare Telecom Gaming Industrial IoT Commerce Hospitality Services Entertainment 3 of the Top 10 3 of the Top 3 3 of the Top 3 3 Fortune 500 6 of the Top 10 6 of the Top 10 2 of the Top 2 eCommerce GDS Credit RePorting Healthcare Broadcast Online Casino IoT ComPanies ComPanies ComPanies ComPanies ComPanies Gaming ComPanies Platforms Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2019. All rights reserved. Value Proposition for Red Hat & Couchbase 8 Agenda 1. Setting the Stage – Technology 2. Setting the Stage – Real World Use Cases 3. Challenges of today’s Corporate Architectures and its Solutions 4. Prerequisites 5. Containerized Database Handling for Microservices 6. Q&A 9 SOLUTION: Field Service APPLICATION: Package tracking USE CASE(S): Caching Document store Operational dashboarding Requirements Outcomes • Changing state and serving inquiries for • Persistent caching with Couchbase (3rd ABOUT: billions of packages generation of tracking architecture) United Parcel Service is an American multinational • Billions of documents • Near-linear scalability package delivery and supply chain management • Tens of terabytes of data • Flexibility in supporting new requirements using N1QL company. • Tens of thousands of operations per second • Better TCO with reduction in the number • 30+ million document inserts and updates per of nodes from ~100 to ~20 day Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. SOLUTION: Customer 360 Catalog & Inventory Field Service APPLICATION: Inventory/price Reservation & PNR Crew services Requirements Outcomes USE CASE(S): • Optimize performance as “look to book” • Fast performance delivers superior User profile store ratio continues to climb: 7m requests per customer experience: 8m ops/sec., 99% Mobile wallet sec., 300k data requests before a booking <2.5 ms response time Pricing Product catalog • Growth of over 100% annually • Multidimensional scaling adds improveD agility: 17TB data set, 180TB usable • Decrease operational costs for ABOUT: storage Leading provider of travel Memcached and MySQL nodes • Data replicateD via XDCR to provide data software and technology • Simplify management anD replication solutions for the global locality in the Google East data center travel industry. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. Agenda 1. Setting the Stage – Technology 2. Setting the Stage – Real World Use Cases 3. Challenges of today’s Corporate Architectures and its Solutions 4. Prerequisites 5. Containerized Database Handling for Microservices 6. Q&A 12 Challenges with a classic RDBMS for microservices SPOF Workload competition Master Data (core) Service Query Service RDBMS as Rigid Index Services Monolith schema Analytics Services … Slave Slave … Virtualized Virtualized RePlica/ RePlica/ 13 Shard Shard Agenda 1. Setting the Stage – Technology 2. Setting the Stage – Real World Use Cases 3. Challenges of today’s Corporate Architectures and its Solutions 4. Prerequisites 5. Containerized Database Handling for Microservices 6. Q&A 14 Break database challenges with cloud-native behaviors SPOF Workload competition Master Data (core) Service Query Service RDBMS as Rigid Index Services Monolith schema Analytics Services … Slave Slave … Virtualized Virtualized RePlica/ RePlica/ 15 Shard Shard DEVELOP WITH AGILITY, DEPLOY AT ANY SCALE Couchbase is ONE technical product! Combining natively integrated microservices CLOUD-NATIVE Interlinked via native stream Accessible with preferable tool: UI, CLI or SDK (all REST based) providing DATABASE same functionalities Architecture ANY CLOUD, ANY DEVICE, MANY SERVICES 16 Designed for Cloud (native) Application(s) Automatic failover Database Database Database Database Shared-nothing architecture Instance Instance Instance Instance No Single Point of Failure Local Local Local Local Storage Storage Storage Storage 17 High Performance and scalability: Modern Architecture – Multi-Dimensional Scaling Scale-up and -out without overlapping services‘ nodes capacities (independently scalable services) 18 Data Everywhere for High availability and Disaster Recovery Cross-cluster (data center) active-active replication (XDCR) No need for replication software because it is already built-in Europe USA Asia 19 Red Hat OpenShift with Couchbase Autonomous Operator Cross Cluster Replication XDCR: Cross Data Center Replication Infrastructure-Agnostic – No Vendor Lock-In Hybrid Cloud & Multi-Cloud Strategy • Cloud-agnostic application deployment + management platform • Treats cloud providers like commodities • Enables you to migrate between clouds freely 21 Easy Data Access: fully compliant SQL for JSON flexible data schema Combination of all advantages of Input: SQL query Output: JSON document flexible JSON data with all common SQL query commands for example o SELECT/FROM/WHERE/GROUP BY/ORDER BY/LIMIT o JOINs incl. extensions for handling JSONs o USE KEYS (From clause) o NEST (From clause) o UNNEST (From clause) o MISSING (Where-clause) o DDL (e.g. CREATE) and DML (e. g. UPDATE) are also supported 22 Agenda 1. Setting the Stage – Technology 2. Setting the Stage – Real World Use Cases 3. Challenges of today’s Corporate Architectures and its Solutions 4. Prerequisites 5. Containerized Database Handling for Microservices 6. Q&A 23 Does this modernization challenge sound familiar? What happens with the database? ● Should we put it in a VM on the Kubernetes cluster? ● Let’s just put it into a container - there are options for persistence, right? ● Hm, what about security, performance, licensing … ○ And what’s the benefit? ○ Let’s start with the low-hanging fruits and leave it as-is for now? 24 Modernization Phase 1: Keep the database as is, but close to the new platform 25 Source: https://resources.couchbase.com/c/kubernetes-ebook?x=gDJUDM Modernization Phase 2: Redesign data layer according to microservices principles 26 Source: https://resources.couchbase.com/c/kubernetes-ebook?x=gDJUDM Full Kubernetes Adoption for database shown with Couchbase Clusters Data-oriented Microservices • Reduced DevOps workload (TCO) • Running Couchbase as a fully managed stateful database application • Next to your microservices applications on the same container platform 27 Source: https://blog.couchbase.com/why-couchbase-is-the-best-nosql-option-for-red-hat-openshift/ Containerized Database Architecture managed by Couchbase Operator OpenShift Namespace: User-Defined Cluster Couchbase Cluster Couchbase Master Autonomous Operator Scalable Service Options — On 1 or Many Pods Namespace: Full-Text Kube-System Data Index Analytics Query Eventing Search Controls API/Authentication Data Storage/ POD1 POD 2 POD 3 POD… POD… POD… POD… POD… POD… POD… POD… POD… POD… Services Couchbase Configuration Pods Scheduler Volumes Management/ Gluster Storage Ceph Storage Other Storage Vendors Replication Persistent Storage 28 Favorable Database