How to Avoid Big Database Pain in Microservices

Couchbase on Red Hat OpenShift How to avoid big database pain in microservices

Steffen Schneider Senior Solutions Engineer

+49 151 507 339 71 [email protected] linkedin.com/in/st-sch Speaker Intro • Cross disciplinary functional-technical profile • Over 10 years professional experience on end-customer side esp. executive department • Execution as Technical Lead with extended Project Management and Leadership experience • Dedicated responsibilities in the areas of o Business Intelligence o Data Warehousing o Data Processing/ETL o Processes o Technology Tools • Since 2017 on software vendor side as Solution Architect Steffen Schneider Finally switch into Big Data world to Couchbase as leading NoSQL Senior Solutions Engineer • technology platform vendor Couchbase – Central Europe

2 Agenda

1. Setting the Stage – Technology

2. Setting the Stage – Real World Use Cases

3. Challenges of today’s Corporate Architectures and its Solutions

4. Prerequisites

5. Containerized Database Handling for Microservices

6. Q&A

3 Agenda

1. Setting the Stage – Technology

2. Setting the Stage – Real World Use Cases

3. Challenges of today’s Corporate Architectures and its Solutions

4. Prerequisites

5. Containerized Database Handling for Microservices

6. Q&A

4 Couchbase – Brief Company Overview

• Founded 2011

Manchester • >500 Empolyees

San Francisco Paris • 70%+ new business growth London • 100%+ growth in total Santa Clara - HQ contract value • 55%+ billings growth Bangalore • 50%+ growth in average deal size

Santa Clara – HQ San Francisco London Paris Manchester Bangalore 3250 Olcott Street 450 Mission Street #200 11-21 Paul Street 40 Rue du Collisée 1A Tariff St 10 Museum Road Santa Clara, CA 95054 San Francisco, CA 94105 London EC2A 4JU 75008 Paris Manchester M1 2FF Bangalore 560001 United States United States United Kingdom France United Kingdom India Couchbase Behind Today’s Business-Critical Applications

Customers

Application Caching & session Real-time pricing, Flight availability, Customer 360 single Real-time crew store for single view product catalog, booking, pricing view, unified notes management, scheduling inventory management analytics, etc. and resources

Performance 2M+ 10M+ 15M 210M 41K reads/sec. unique SKUs ops / second documents pilots and crew 10M 35K <2.5ms 100K 148M queries/sec. requests/sec. response time users travelers in 2017

Infrastructure

Developer Agility Performance at Scale Manageability

Security + Availability A Proven Enterprise Solution Chosen by Industry Leaders

Retail & E- Travel & Financial Media & Healthcare Telecom Gaming Industrial IoT Commerce Hospitality Services Entertainment

3 of the Top 10 3 of the Top 3 3 of the Top 3 3 Fortune 500 6 of the Top 10 6 of the Top 10 2 of the Top 2 eCommerce GDS Credit Reporting Healthcare Broadcast Online Casino IoT Companies Companies Companies Companies Companies Gaming Companies Platforms

8 Agenda

1. Setting the Stage – Technology

2. Setting the Stage – Real World Use Cases

3. Challenges of today’s Corporate Architectures and its Solutions

4. Prerequisites

5. Containerized Database Handling for Microservices

6. Q&A

9 SOLUTION: Field Service

APPLICATION: Package tracking

USE CASE(S): Caching Document store Operational dashboarding Requirements Outcomes • Changing state and serving inquiries for • Persistent caching with Couchbase (3rd ABOUT: billions of packages generation of tracking architecture) United Parcel Service is an American multinational • Billions of documents • Near-linear scalability package delivery and supply chain management • Tens of terabytes of data • Flexibility in supporting new requirements using N1QL company. • Tens of thousands of operations per second • Better TCO with reduction in the number • 30+ million document inserts and updates per of nodes from ~100 to ~20 day

APPLICATION: Inventory/price Reservation & PNR Crew services Requirements Outcomes

USE CASE(S): • Optimize performance as “look to book” • Fast performance delivers superior User profile store ratio continues to climb: 7m requests per customer experience: 8m ops/sec., 99% Mobile wallet sec., 300k data requests before a booking <2.5 ms response time Pricing Product catalog • Growth of over 100% annually • Multidimensional scaling adds improved agility: 17TB data set, 180TB usable • Decrease operational costs for ABOUT: storage Leading provider of travel Memcached and MySQL nodes • Data replicated via XDCR to provide data software and technology • Simplify management and replication solutions for the global locality in the Google East data center travel industry.

1. Setting the Stage – Technology

2. Setting the Stage – Real World Use Cases

3. Challenges of today’s Corporate Architectures and its Solutions

4. Prerequisites

5. Containerized Database Handling for Microservices

6. Q&A

12 Challenges with a classic RDBMS for microservices

SPOF Workload competition Master Data (core) Service

Query Service RDBMS as Rigid Index Services Monolith schema

Analytics Services

…

Slave Slave …

Virtualized Virtualized Replica/ Replica/ 13 Shard Shard Agenda

1. Setting the Stage – Technology

2. Setting the Stage – Real World Use Cases

3. Challenges of today’s Corporate Architectures and its Solutions

4. Prerequisites

5. Containerized Database Handling for Microservices

6. Q&A

14 Break database challenges with cloud-native behaviors

SPOF Workload competition Master Data (core) Service

Query Service RDBMS as Rigid Index Services Monolith schema

Analytics Services

…

Slave Slave …

Virtualized Virtualized Replica/ Replica/ 15 Shard Shard DEVELOP WITH AGILITY, DEPLOY AT ANY SCALE Couchbase is ONE technical product! Combining natively integrated microservices CLOUD-NATIVE Interlinked via native stream Accessible with preferable tool: UI, CLI or SDK (all REST based) providing DATABASE same functionalities Architecture

ANY CLOUD, ANY DEVICE, MANY SERVICES

16 Designed for Cloud (native)

Application(s)

Automatic failover Database Database Database Database Shared-nothing architecture Instance Instance Instance Instance No Single Point of Failure

Local Local Local Local Storage Storage Storage Storage

17 High Performance and scalability: Modern Architecture – Multi-Dimensional Scaling

Scale-up and -out without overlapping services‘ nodes capacities (independently scalable services)

18 Data Everywhere for High availability and Disaster Recovery

Cross-cluster (data center) active-active replication (XDCR)

No need for replication software because it is already built-in

Europe

USA Asia

19 Red Hat OpenShift with Couchbase Autonomous Operator

Cross Cluster Replication

XDCR: Cross Data Center Replication Infrastructure-Agnostic – No Vendor Lock-In

Hybrid Cloud & Multi-Cloud Strategy • Cloud-agnostic application deployment + management platform • Treats cloud providers like commodities • Enables you to migrate between clouds freely

21 Easy Data Access: fully compliant SQL for JSON flexible data schema

Combination of all advantages of Input: SQL query Output: JSON document flexible JSON data with all common SQL query commands for example o SELECT/FROM/WHERE/GROUP BY/ORDER BY/LIMIT o JOINs incl. extensions for handling JSONs o USE KEYS (From clause) o NEST (From clause) o UNNEST (From clause) o MISSING (Where-clause)

o DDL (e.g. CREATE) and DML (e. g. UPDATE) are also supported

22 Agenda

1. Setting the Stage – Technology

2. Setting the Stage – Real World Use Cases

3. Challenges of today’s Corporate Architectures and its Solutions

4. Prerequisites

5. Containerized Database Handling for Microservices

6. Q&A

23 Does this modernization challenge sound familiar? What happens with the database?

● Should we put it in a VM on the Kubernetes cluster?

● Let’s just put it into a container - there are options for persistence, right?

● Hm, what about security, performance, licensing …

○ And what’s the benefit?

○ Let’s start with the low-hanging fruits and leave it as-is for now?

24 Modernization Phase 1: Keep the database as is, but close to the new platform

Source: https://resources.couchbase.com/c/kubernetes-ebook?x=gDJUDM Modernization Phase 2: Redesign data layer according to microservices principles

Source: https://resources.couchbase.com/c/kubernetes-ebook?x=gDJUDM Full Kubernetes Adoption for database shown with Couchbase Clusters

Data-oriented Microservices

• Reduced DevOps workload (TCO) • Running Couchbase as a fully managed stateful database application • Next to your microservices applications on the same container platform

Source: https://blog.couchbase.com/why-couchbase-is-the-best-nosql-option-for-red-hat-openshift/ Containerized Database Architecture managed by Couchbase Operator

OpenShift Namespace: User-Defined Cluster Couchbase Cluster Couchbase Master Autonomous Operator Scalable Service Options — On 1 or Many Pods Namespace:

Full-Text Kube-System Data Index Analytics Query Eventing Search Controls API/Authentication

Data Storage/ POD1 POD 2 POD 3 POD… POD… POD… POD… POD… POD… POD… POD… POD… POD… Services Couchbase Configuration

Pods Scheduler

Volumes Management/ Gluster Storage Ceph Storage Other Storage Vendors Replication

Persistent Storage

28 Favorable Database Operator at highest level to fully unboarden your Ops

Couchbase Ease Of Management – Low operational costs

Autonomous Data Platform • Automated operational best practices • In efficiently deploying + managing Couchbase Data Platform • Reducing up to 95 % of operational complexity (TCO)

30 Couchbase Operator Usage Scenarios

● Automated cluster provisioning ● Automated failure recovery with custom Couchbase-specific automation ● Cross datacenter replication (XDCR) ● On-demand dynamic scaling ● Rack/zone awareness ● Auto-failover capabilities ● Production-grade supportability features ● Persistent storage support ● Centralized configuration management ● Enterprise-grade high availability features ● Fully automated upgrade of the Couchbase cluster ● Rolling upgrade of Kubernetes without affecting the Couchbase cluster ● Assisted deployment via Helm charts ● Logging and Monitoring (Prometheus and Grafana) 31 ● Multi-Cluster Management

Further details at: https://blog.couchbase.com/why-couchbase-is-the-best-nosql-option-for-red-hat-openshift/ Agenda

1. Setting the Stage – Technology

2. Setting the Stage – Real World Use Cases

3. Challenges of today’s Corporate Architectures and its Solutions

4. Prerequisites

5. Containerized Database Handling for Microservices

6. Q&A

32 Q & A

Your questions please :-)

33 Many Thanks for your Participation & Interest

Steffen Schneider Senior Solutions Engineer Couchbase – Central Europe