Inside MarkLogic Server Its data model, indexing system, update model, and operational behaviors Revision 6 Jason Hunter, MarkLogic Corp Licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. Some features discussed are covered by MarkLogic patents. Download at http://developer.marklogic.com/inside-marklogic. Revision 6 Inside MarkLogic Server This paper describes the MarkLogic Server internals: its data model, indexing system, update model, and operational behaviors. It's intended for a technical audience — either someone new to MarkLogic wanting to understand its capabilities, or someone already familiar with MarkLogic who wants to understand what's going on under the hood. This paper is not an introduction to using MarkLogic Server. For that you can read the official product documentation. Instead, this paper explains the principles on which MarkLogic is built. The goal isn't to teach you to write code, but to help you understand what's going on behind your code, and thus help you write better and more robust applications. The paper is organized into sections. The first section provides a high-level overview of MarkLogic Server. The next few sections explain MarkLogic's core indexes. The sections after that explain the transactional storage system, multi-host clustering, and the various connection options. At this point there's a natural stopping point for the casual reader, but those who read on will find sections covering advanced indexing features as well as topics like replication and failover. The final section discusses the ecosystem built up around MarkLogic. 2 Revision 6 Table of Contents What Is MarkLogic Server?................................................................................................ 5 Document-Centric ........................................................................................................... 5 Transactional.................................................................................................................... 6 Search-Centric ................................................................................................................. 6 Structure-Aware .............................................................................................................. 6 Schema-Agnostic............................................................................................................. 7 XQuery- and XSLT-Driven............................................................................................ 8 High Performance............................................................................................................ 8 Clustered .......................................................................................................................... 9 Database Server ............................................................................................................... 9 Core Topics........................................................................................................................... 10 Indexing Text and Structure ............................................................................................. 10 Indexing Words ............................................................................................................. 10 Indexing Phrases............................................................................................................ 10 Indexing Longer Phrases............................................................................................... 12 Indexing Structure ......................................................................................................... 12 Indexing Values............................................................................................................. 13 Indexing Text with Structure ........................................................................................ 14 Index Size....................................................................................................................... 14 Reindexing ..................................................................................................................... 15 Relevance....................................................................................................................... 15 Lifecycle of a Query...................................................................................................... 15 Indexing Document Metadata........................................................................................... 17 Collection Indexes......................................................................................................... 17 Directory Indexes .......................................................................................................... 18 Security Indexes ............................................................................................................ 18 Properties Indexes ......................................................................................................... 18 Fragmentation .................................................................................................................... 19 Fragment vs. Document ................................................................................................ 19 Estimate and Count ....................................................................................................... 20 Unfiltered ....................................................................................................................... 20 The Range Index................................................................................................................ 21 Range Queries................................................................................................................ 22 Extracting Values .......................................................................................................... 22 Optimized "Order By"................................................................................................... 23 Using Range Indexes for Joins ..................................................................................... 24 Data Management.............................................................................................................. 25 What's on Disk: Databases, Forests, and Stands ......................................................... 25 Ingesting Data................................................................................................................ 25 Modifying Data.............................................................................................................. 27 Multi-Version Concurrency Control............................................................................ 27 Time Travel.................................................................................................................... 28 Locking .......................................................................................................................... 28 Updates........................................................................................................................... 29 3 Revision 6 Documents are Like Rows............................................................................................ 30 Lifecycle of a Document............................................................................................... 30 Clustering and Caching ..................................................................................................... 31 Cluster Management ..................................................................................................... 32 Caching .......................................................................................................................... 33 Cache Partitions............................................................................................................. 34 No Need for Global Cache Invalidation ...................................................................... 34 Locks and Timestamps in a Cluster ............................................................................. 34 Lifecycle of a Query in a Cluster ................................................................................. 35 Lifecycle of an Update in a Cluster.............................................................................. 36 Coding and Connecting to MarkLogic............................................................................. 36 XQuery and XSLT ........................................................................................................ 36 Modules and Deployment............................................................................................. 37 Output Options .............................................................................................................. 38 Single-Tier Web Deployment....................................................................................... 38 Multi-Tier Programmatic Deployment ........................................................................ 39 Remote Filesystem Access ........................................................................................... 40 CQ for Remote Coding ................................................................................................. 40 Advanced Topics.................................................................................................................. 41 Advanced Text Handling