Data Architecture Dilemma Many Single-Purpose Databases? Or A Converged Autonomous Database? Tirthankar Lahiri Senior Vice President, Data and In-Memory Technologies 1 Copyright © 2019 Oracle and/or its affiliates. A Brief History of Enterprise Data Management Mainframe Era Client-Server Era Internet Era Cloud Era Mid 60s – Early 80s Early 80s – Mid 90s Mid 90s – Mid 2000s Mid 2000s – Present These are the Voyages of the Modern Enterprise On its continuing mission … To explore strange new data models and unique new workloads … To Boldly Go Where No Database has Gone Before Copyright © 2019 Oracle and/or its affiliates. Mainframe Era: The War of The Paradigms • Hierarchical (IMS) or Network Models (Codasyl) • Data organized as trees or graphs with links • Data navigation was imperative: • Programmers described how to access data • The RDBMS (e.g. System R) was an upstart competitor • Data is organized into logical sets with relationships • Data navigation is declarative • SQL describes the question, not how to get the answer Copyright © 2019 Oracle and/or its affiliates. Client-Server Era: The RDBMS Wins • The RDBMS became the defacto data management standard • Key reason: Developer Productivity • Declarative language and data model: Simple development • Standard language and APIs: Reuse developer skillsets • Consistent Transactions: Database guaranteed atomicity • Integrity of data automatically enforced by constraints • Developers can focus on Application Logic (i.e. Development) • Data access, cleanliness, consistency handled by the database Copyright © 2019 Oracle and/or its affiliates. Internet Era: New Extreme Requirements • “The internet changes everything” – Larry Ellison • “The network is the computer” – Scott McNealy New Requirements Volume Exponential growth of data Scalability Massive online user populations, much larger than enterprise Availability ReQuirement to be always on, 24x7 access to data Variety Many different data models (teXt, XML, Graph, Spatial) Object, XML databases appeared but vanished once mainstream DBs implemented them Security Much larger attack surface area than enterprise Copyright © 2019 Oracle and/or its affiliates. Cloud Era: Developers and Microservices “The Cloud Changes Everything” • Deployment, Administration, specially Development • Empowers and Democratizes Development • Developers are also users (of database services) • Developer communities grow to planet scale • Key transformation: Lightweight Microservices • Microservices encapsulate functionality of specific modules • Modular, loosely coupled and communicate via events • Agile: Can be developed and upgraded independently • Highly available: Microservices can fail independently Copyright © 2019 Oracle and/or its affiliates. Cloud Era: Issues with “Monolithic” Databases PRODUCT CATALOG CUSTOMERS ORDERS • Single Model database architecture sometimes slowed development: ANALYTICS RECOMMENDATIONS • Developers needed consensus on Data Model and Schema • Single Database architecture sometimes could not meet cloud scale reQuirements Copyright © 2019 Oracle and/or its affiliates. Cloud Era: Single Purpose Makes a Comeback EVENT EVENT • Trend started by web giants: PRODUCT CATALOG CUSTOMERS ORDERS • Build Custom DBs for specific uses • Google Big Table, Facebook Cassandra, etc. Document RDBMS Key Value DB • NoSQL products began to proliferate ANALYTICS RECOMMENDATIONS • MongoDB, Couchbase, Redis, Neo4J ... • Different data models, languages, APIs Big Data Stream DB • Provided scalability & performance • With tradeoffs in functionality Copyright © 2019 Oracle and/or its affiliates. Cloud Era: Oracle Becomes a Converged Database • Oracle Database continued to evolve far beyond a traditional Relational DBMS • Converges many data models and workloads: Relational • Class Leading Document Database with JSON, TeXt, XML Document • Class Leading Database for AI and Machine Learning Spatial l • Class Leading Spatial and Graph Database Graph • Class Leading support for IoT, Times Series, Binary Data, Database AI / ML Resident Filesystem, Objects, etc. IoT / Time Series File System Copyright © 2019 Oracle and/or its affiliates. Meet the Present Day Enterprise Data Architecture Dilemma Copyright © 2019 Oracle and/or its affiliates. Today’s Enterprise Data Architecture Dilemma MANY SINGLE-PURPOSE DATABASES OR A CONVERGED DATABASE ? REST API EVENT EVENT PRODUCT CATALOG CUSTOMERS ORDERS CATALOG CUSTOMERS ORDERS RECOMEND ANALYTICS AQ MSG AQ MSG CROSS PDB Document RDBMS Key Value DB QUERIES ANALYTICS RECOMMENDATIONS Big Data Stream DB Copyright © 2019 Oracle and/or its affiliates. Single Purpose Databases: Microservice Friendly? EVENT EVENT • Single-purpose engines may seem PRODUCT CATALOG CUSTOMERS ORDERS better for micro-services Document RDBMS Key Value • Each function has a separate DB database optimized for that use ANALYTICS RECOMMENDATIONS Big Data Stream • Functions are independent, and DB independent databases are also failure independent Copyright © 2019 Oracle and/or its affiliates. Single Purpose Databases: Separating Myth from Reality • Myth #1: Microservices need EVENT EVENT separate specialized databases PRODUCT CATALOG CUSTOMERS ORDERS Document RDBMS Key Value • Fact: They may need separate DB algorithms and data models ANALYTICS RECOMMENDATIONS Big Data Stream • Possible with single-purpose DB databases and a converged database Copyright © 2019 Oracle and/or its affiliates. Single Purpose Databases: Missing Functionality EVENT EVENT • Many of them lack: PRODUCT CATALOG CUSTOMERS ORDERS • Native data Integrity • Sophisticated Indexing Document RDBMS Key Value DB • Full featured Transactions ANALYTICS • Enterprise Security RECOMMENDATIONS • Standard Declarative Languages Big Data Stream DB • Each is uniQue, need special skills Copyright © 2019 Oracle and/or its affiliates. Single Purpose Databases: Infrastructural Downsides Administration, Integration challenges: EVENT EVENT PRODUCT CATALOG CUSTOMERS ORDERS • Consistency of data • Cross product compatibility Document RDBMS Key Value DB • Separate security patches • Separate upgrade cycles ANALYTICS RECOMMENDATIONS • Complex integration for Analytics Big Data Stream • Consistent backup of data is hard DB Copyright © 2019 Oracle and/or its affiliates. Single Purpose Databases: Macro-Complex Microservices • Microservices must manage enormous EVENT EVENT PRODUCT CATALOG CUSTOMERS ORDERS amounts of application state Document RDBMS Key Value • Application is responsible for: DB –Atomicity of workflows ANALYTICS RECOMMENDATIONS –Failure handling Big Data Stream –Consistent security policy DB Copyright © 2019 Oracle and/or its affiliates. Real-World Macro-Complexity Producers Event and data streams Consumers Macro-Complexity Pub/Sub Web servers Mobile App1 • Order service Mobile Platform Multiple technologies Search Portal Apache ELK • Multiple data stores API/Brokers Kafka Ops Mobile Apache Flink Dashboards • Data copied multiple IoT Realtime times to do analytics Analytics, AWS S3 Transactions Alerts • Compromises security Oracle ML Model • Compromises data Training consistency NoSQL Hadoop lake Adhoc Exploration • CompleX to maintain Analytic MySQL Reports • Need highly skilled developers to build & OLTP Vertica/Hive keep running Copyright © 2019 Oracle and/or its affiliates. 17 Can a Converged Database Ever Scale Like a Single- Purpose Database? Copyright © 2019 Oracle and/or its affiliates. 18 YES Copyright © 2019 Oracle and/or its affiliates. 19 Scalability of Oracle Database Myth #2: An RDBMS is too complex to scale effectively • Oracle Database has made major scalability advances over decades • Every Oracle Database release has tackled difficult scaling problems • Major architectural features for Scale-Up, Scale-Out, Fault tolerance Copyright © 2019 Oracle and/or its affiliates. Oracle Database: Decades of Scalability 2019 19c 18c • Fast Ingest for IoT • Adaptive SeQuences • Multi-Instance Redo Apply • RDMA Undo Read 12c • In-Memory Row Store • Commit Cache • Database In-Memory • In-Memory Columnar in Flash 11g • Sharding • Smart Fusion Block Transfer • Multitenant • Direct-to-wire Protocol • Application Continuity • Active Data Guard • Tiered Disk/ Flash 10g • Golden Gate • PCIe NVMe Flash. 2000 • Database Resident Connection Pool • Unified Infiniband • Advanced Compression • Data Guard • Network Resource Management 9i • Enterprise Manager Grid Control • IO Resource Management • Database Resource Manager • Hybrid Column Compression • Real Application Clusters • • Automatic Segment Space Management Scale-Out Storage Servers • Scale-Out DB Servers • Scalable Log Generation • Query offload to storage Database Software Exadata 2008 – First Release of Exadata Software and Hardware Copyright © 2019 Oracle and/or its affiliates. 21 Database In-Memory: Converged Analytics and OLTP • Fast OLTP and Real-Time Analytics in the In-Memory same converged database Buffer Cache Column Store OLTP • Industry-first dual format architecture: • Row format ideal for OLTP, column format is SALES SALES ideal for Analytics Row Column Format Format • Same table in row and column format simultaneously • No data movement needed • Analytics against against real-time data SALES Copyright © 2019 Oracle and/or its affiliates. The Forrester WaveTM: In-Memory Databases, Q1 2017 Oracle In-Memory Databases Scored Highest by Forrester on both Current Offering and Strategy http://www.oracle.com/us/corporate/analystreports/forrester-imdb-wave-2017-3616348.pdf The Forrester Wave™ is copyrighted by Forrester Research, Inc.