Subversion FS Replication

Philip Martin, Subversion Committer, WANdisco Extending Apache Subversion to Enable Better Replication Subversion FS Replication Different kinds of Replication

• Read-only mirror • ApacheTM Subversion® write-through proxy • Multiple master (active-active) replication – Proxy replication – FS replication Subversion FS Replication Write-through Proxy Replication

Client

Sync Slave Sync

Client Master Slave Subversion FS Replication svnsync Replication

• Revisions are ordered and immutable • Easy recovery following an error • Revision properties are not ordered • More care needed to revision properties reliable • File locks not handled • Only replicate successful commits Subversion FS Replication Write-through Proxy Features

• HTTP and HTTPS only • No • Commits pass through slave • Commits have to be sent back to the slave • Different representations for revisions Subversion FS Replication Content and Representation

• Same content, different representation • svnsync is a different client • Representation order is client dependent • rep-cache differences – single-threaded svnsync slave – multi-threaded httpd master • Deliberately different content Subversion FS Replication Multiple Master Replication

Client

Master

Client Master Master Subversion FS Replication Paxos Consensus Agreement

• Well-known family of algorithms • Agree the order in advance • Propose commit until agreed • Handles unreliable nodes • Lots of variations and optimizations Subversion FS Replication WANdisco Proxy Replication

Client Proxy

hpd Repo

hpd Repo hpd Repo

Proxy Proxy Subversion FS Replication Proxy Operation

• Categorise HTTP requests into – reads such as GET, PROPFIND – writes such as MKACTIVITY, PUT • Read passes through to Apache • Write triggers the Paxos algorithm – proxy knows about other nodes • Wait for correct point in global sequence • Pass write through to Apache Subversion FS Replication WANdisco Proxy Replication Features

• HTTP required • No svn://, svn+ssh:// or file://, HTTPS hard • Proxy overhead on read • No file locking • Multiple agreements per-commit • Replicate unsuccessful commits • Different representations of revisions • Standard Apache Subversion • Proven reliability Subversion FS Replication Subversion Architecture

• Modular with most code in libraries • Applications are thin wrappers • Client-side libraries • Server-side libraries • Stable public APIs • Written in with bindings for other languages Subversion FS Replication Apache Subversion

svnserve

apache BDB Repo TSVN

client FS svn FSFS Repo swigpy svnadmin

script.py Subversion FS Replication Subversion FS API

• Lowest level public API • All repository access goes through the FS API • Opaque svn_fs_t object with a vtable • vtable populated when the FS is opened • Each backend provides its own vtable Subversion FS Replication New FSFSWD Backend

BDB RW BDB BDB hpd FS Repo FSFS RW FSFSWD R FSFS FSFSWD W Repo FSFS FSFSWD FSFSWD Repo FSFSWD W FSFS

replicator JNI FSFSWD Subversion FS Replication FSFSWD Write Processing

• Subversion process calls FS write function • vtable invokes FSFSWD code • FSFSWD code sends message to replicator • Replicator runs Paxos • Replicator uses JNI to invoke FSFS code • Replicator returns message to Subversion • FSFSWD code returns replicator result Subversion FS Replication WANdisco Proxy Replication

Client Proxy

hpd Repo

hpd Repo hpd Repo

Proxy Proxy Subversion FS Replication FSFSWD Replication

FSFSWD Repo Client hpd

replicator

replicator replicator FSFSWD Repo

Client svnserve FSFSWD Repo Subversion FS Replication FSFSWD Repository

• New repository type “fsfswd” • No repository access without FSFSWD installed • Cannot bypass replication using Subversion – “svnadmin load” is replicated • Script using bindings to access repository on server will be replicated • Trivial conversion back to “fsfs” repository Subversion FS Replication FSFSWD Commit Processing

• Intercept svn_fs_create_txn • Modify txn-id to be globally unique • Build up transaction using FSFS code • Early conflict checks abort transaction before replication • Intercept svn_fs_commit_txn • Send commit message to replicator Subversion FS Replication FSFSWD Transaction Processing

• Apache Subversion transaction ID is unique within repository • FSFSWD transaction ID is unique across all instances of replicated repository • Transaction to revision is deterministic – rep-cache can be enabled or disabled – repository can be packed Subversion FS Replication FS Replication Features

• Supports all protocols • Supports all clients over network protocols • Negligible overhead for read operations • Far fewer agreements per-commit • Supports file locks • Identical representations of revisions • Easy migration back to FSFS Subversion FS Replication FSFSWD Module Loading

• Apache Subversion supports loadable FS modules • 1.7 requires changes to support 3rd party modules • 1.8 supports 3rd party modules • No FSFSWD code changes in core Subversion • Install two shared libraries and one JAR. Subversion FS Replication FSFSWD Code

• FSFSWD about 1,000 lines compared to 30,000 lines for FSFS • Mostly marshalling messages over the socket • com..subversion JNI to bypass the FSFS vtable • Install two shared libraries and one JAR Subversion FS Replication FSFSWD Performance

• Read performance has negligible overhead – vtable diversion once per-object – no overhead using the object • Write performance small overhead – vtable diversion once per-object – less than 1ms IPC overhead – small FSFS cache overhead • Paxos/replication overhead depends on network topology Subversion FS Replication FSFSWD and Subversion 1.9

• FSFS file locking scales with total number of locks in repository • Unlocking post-commit impacts replication performance • Long-term solution will require a repository upgrade • Multiple lock and unlock FS operations improve 1.9 performance Thank You [email protected]