Couchbase Under the Hood An Architectural Overview Couchbase Under the Hood An Architectural Overview Table of Contents INTRODUCTION 3 Essential NoSQL requirements and features 3 Core design principles 5 JSON DATA MODEL AND ACCESS METHODS 5 JSON data model 5 Document access methods 7 Keys, values, and sub-documents 8 Key Couchbase concepts 9 COUCHBASE SERVICES 9 Data service and KV engine 10 Key-value data access 12 Query service 13 Index service for N1QL 14 Search service 15 Eventing service 16 Analytics 16 Mobile 17 DISTRIBUTED FOUNDATION 18 Node-level architecture 18 Cluster architecture 19 Client connectivity 20 Data transport via Database Change Protocol (DCP) 21 Multi-Dimensional Scaling (MDS) 22 Data distribution 23 Rebalancing the cluster 24 High availability 24 Security 29 Mobile client synchronization 30 RESOURCES 32 INTRODUCTION Couchbase is an open source, distributed, document-oriented, NoSQL database. Couchbase’s core architecture supports a flexible JSON data model, easy scalability, consistent high performance, mobile synchronization, always-on 24x365 characteristics, and advanced security. Couchbase supports multiple data access patterns on top of a flexible JSON data model. Couchbase consolidates multiple layers into a single platform that would otherwise require separate solutions to work together. Couchbase provides the performance of a caching layer, the flexibility of a source of truth, and the reliability of a system of record, eliminating the need to manage data models and consistency between multiple systems, learn different languages and APIs, and manage independent technologies. This paper describes how the internal components of the Couchbase database (Server and Mobile) operate with one another. It assumes you have a basic understanding of Couchbase and are looking for a greater technical understanding of how things operate beneath the surface. Essential NoSQL requirements and features NoSQL databases evolved from enterprise relational databases to address performance and delivery deficiencies. Relational databases tend to operate primarily as systems of record, maintaining transactional data in a highly consistent manner. But several architectural principles (normalization of objects, single node transactional design, two- phase commit) have made them difficult to scale to larger workloads while simultaneously delivering responsive and highly available applications. Pragmatic business needs for more advanced technical requirements have pushed NoSQL to the forefront. These modern requirements have driven Couchbase’s development from day one: • Agile development • Scalable performance • System manageability Couchbase has been focused on setting a high standard in each of these areas. The result is a robust and accessible set of features with solid performance and ease of management in a single NoSQL database. Learn more about how Couchbase delivers on these core database requirements in the following table and later in the text of this paper. 3 Develop with agility Perform at any scale Manage with ease • Flexible JSON data • Memory- and network- • Global deployment model supports centric architecture, with low write latency continuous delivery. with an integrated using active-active Make schema changes cache delivering high cross datacenter without downtime throughput and sub- replication millisecond latency • Leverage common • Infrastructure agnostic SQL query patterns • Always-on, fault support across for joins, aggregations, tolerant design physical, virtual, cloud and more and containerized • Consistent environments • Extract value using performance at a broad set of key any scale • Microservices capabilities (mobile architecture with sync, full-text search, • Isolated and built-in auto-sharding, real-time analytics, etc.) independent scaling replication, and of workloads, with failover • ACID transaction no downtime or support code changes • Full-stack security with end-to-end • No hassle scale-out encryption and role- based access control These principles have been built in the very core of the database engine to ensure low latency and reliable, yet easy to manage, replication. Around this core are a set of data access components that run and scale independently of each other. These are delivered through a unified programming API, established security capabilities, and external technology integrations. Core design principles To effectively deliver the above features, three guiding principles have been followed when developing Couchbase: memory and network-centric architecture, workload isolation, and an asynchronous approach to everything. Memory and network-centric architecture • The most used data and indexes are transparently maintained in memory for fast reads. • Writes are performed in memory and replicated or persisted synchronously or asynchronously. • Internal Database Change Protocol (DCP) streams data mutations from memory to memory at network speed to support replication, indexing and mobile synchronization. Workload isolation • All databases perform different tasks in support of an application. These include persisting, indexing, querying, and searching data. Each of these workloads has slightly different performance and resource requirements. • Multi-Dimensional Scaling (MDS) isolates these workloads from one another at both a process and a node level. • MDS allows these workloads to be scaled independently from one another and their resources to be optimized as necessary. • Couchbase manages the topology, process management, statistics gathering, high availability, and data movement between these services transparently. 4 Asynchronous approach to everything • Traditional databases increase latency and block application operations while running synchronous operations, for example, persisting data to disk or maintaining indexes. • Couchbase allows write operations to happen at memory and network speeds while asynchronously processing replication, persistence and index management. • Spikes in write operations don’t block read or query operations, while background processes will persist data as fast as possible without slowing down the rest of the system. • Durability and consistency options are available allowing the developer to decide when and where to increase latency in exchange for durability and consistency. rogramming d P Mo ifie de Un l Eventing B i g D nsactio Query a Mobile a n Tr s t F a u a ory- l m fi l n r e s - t s d M t a P S c e y Q k r c t i s i L t l S i i s s e I Key t b Analytics a c n e l a l u t Value n E e a ce c r g S i t r y a t R n i ep io o licat n s Full Text Indexing Search C o ts nt n ain me er loy and Cloud Dep Couchbase Server JSON DATA MODEL AND ACCESS METHODS This section outlines the foundational JSON data model handling in Couchbase, then introduces the multiple ways to access that data. These methods include basic key-value operations, SQL querying, full-text searching, real-time analytics, server-side eventing, and mobile application synchronization. JSON data model The JSON data model supports basic and complex data types: numbers, strings, nested objects, and arrays. JSON provides rapid serialization and deserialization, is native to JavaScript, and is the most common REST API data format. Consequently, JSON is extremely convenient for web application programming. Couchbase stores data as individual documents, comprised of a key and a value. When the value is JSON-formatted, Couchbase provides rich access capabilities; when not, the document is stored as a binary BLOB and has more limited access characteristics. 5 A document often represents a single instance of an application object (or nested objects). It can also be considered analogous to a row in a relational table, with the document attributes acting similar to a column. Couchbase provides greater flexibility than relational databases, allowing JSON documents with varied schemas and nested structures. Developers may express many-to-many relationships without requiring a reference or junction table. Subcomponents of documents can be accessed and updated directly as well, and multiple document schemas can be aggregated together into a virtual table with a single query. Flexible, dynamic schema In the document model, a schema is the result of an application’s structuring of its documents: schemas are defined and managed by applications. A document’s structure consists of its inner arrangement of attribute-value pairs. This is in contrast to the relational model where the database manages the schema. For example, both of the following JSON document examples are valid data models that Couchbase can manage and query. How the documents are designed or updated over time is up to the application developer – normalized or denormalized or a hybrid depending on the needs and evolution of the application. Normalized – 4 documents Denormalized – 1 document invoice1: invoice1: { { "BillTo": "Lynn Hess", "BillTo": "Lynn Hess", "InvoiceDate": "2018-01-15", "InvoiceDate": "2018-01-15", "InvoiceNum": "ABC123", "InvoiceNum": "ABC123", "ShipTo": "H. Trisler, 41 Oak Drive" "ShipTo": "H. Trisler, 41 Oak Drive", } "Items": [ invoice1:item1: { "Price": "100", "Product": "Brake { Pad", "Quantity": "24" }, "InvoiceId": "1", { "Price": "10", "Product": "Rotor", "Price": "100", "Quantity": "5" }, "Product": "Brake Pad", { "Price": "20", "Product": "Tire", "Quantity": "24" "Quantity": "2" } } ] } invoice1:item2: { "InvoiceId": "1", "Price": "10", "Product": "Rotor", "Quantity": "5" } invoice1:item3: { "InvoiceId":