Subversion FS Replication
Philip Martin, Subversion Committer, WANdisco Extending Apache Subversion to Enable Better Replication Subversion FS Replication Different kinds of Replication
• Read-only mirror • ApacheTM Subversion® write-through proxy • Multiple master (active-active) replication – Proxy replication – FS replication Subversion FS Replication Write-through Proxy Replication
Client
Sync Slave Sync
Client Master Slave Subversion FS Replication svnsync Replication
• Revisions are ordered and immutable • Easy recovery following an error • Revision properties are not ordered • More care needed to make revision properties reliable • File locks not handled • Only replicate successful commits Subversion FS Replication Write-through Proxy Features
• HTTP and HTTPS only • No file locking • Commits pass through slave • Commits have to be sent back to the slave • Different representations for revisions Subversion FS Replication Content and Representation
• Same content, different representation • svnsync is a different client • Representation order is client dependent • rep-cache differences – single-threaded svnsync slave – multi-threaded httpd master • Deliberately different content Subversion FS Replication Multiple Master Replication
Client
Master
Client Master Master Subversion FS Replication Paxos Consensus Agreement
• Well-known family of algorithms • Agree the commit order in advance • Propose commit until agreed • Handles unreliable nodes • Lots of variations and optimizations Subversion FS Replication WANdisco Proxy Replication
Client Proxy
h pd Repo
h pd Repo h pd Repo
Proxy Proxy Subversion FS Replication Proxy Operation
• Categorise HTTP requests into – reads such as GET, PROPFIND – writes such as MKACTIVITY, PUT • Read passes through to Apache • Write triggers the Paxos algorithm – proxy knows about other nodes • Wait for correct point in global sequence • Pass write through to Apache Subversion FS Replication WANdisco Proxy Replication Features
• HTTP required • No svn://, svn+ssh:// or file://, HTTPS hard • Proxy overhead on read • No file locking • Multiple agreements per-commit • Replicate unsuccessful commits • Different representations of revisions • Standard Apache Subversion • Proven reliability Subversion FS Replication Subversion Architecture
• Modular with most code in libraries • Applications are thin wrappers • Client-side libraries • Server-side libraries • Stable public APIs • Written in C with bindings for other languages Subversion FS Replication Apache Subversion
svnserve
apache BDB Repo TSVN
client FS svn FSFS Repo swigpy svnadmin
script.py Subversion FS Replication Subversion FS API
• Lowest level public API • All repository access goes through the FS API • Opaque svn_fs_t object with a vtable • vtable populated when the FS is opened • Each backend provides its own vtable Subversion FS Replication New FSFSWD Backend
BDB RW BDB BDB h pd FS Repo FSFS RW FSFSWD R FSFS FSFSWD W Repo FSFS FSFSWD FSFSWD Repo FSFSWD W FSFS
replicator JNI FSFSWD Subversion FS Replication FSFSWD Write Processing
• Subversion process calls FS write function • vtable invokes FSFSWD code • FSFSWD code sends message to replicator • Replicator runs Paxos • Replicator uses JNI to invoke FSFS code • Replicator returns message to Subversion • FSFSWD code returns replicator result Subversion FS Replication WANdisco Proxy Replication
Client Proxy
h pd Repo
h pd Repo h pd Repo
Proxy Proxy Subversion FS Replication FSFSWD Replication
FSFSWD Repo Client h pd
replicator
replicator replicator FSFSWD Repo
Client svnserve FSFSWD Repo Subversion FS Replication FSFSWD Repository
• New repository type “fsfswd” • No repository access without FSFSWD installed • Cannot bypass replication using Subversion – “svnadmin load” is replicated • Script using bindings to access repository on server will be replicated • Trivial conversion back to “fsfs” repository Subversion FS Replication FSFSWD Commit Processing
• Intercept svn_fs_create_txn • Modify txn-id to be globally unique • Build up transaction using FSFS code • Early conflict checks abort transaction before replication • Intercept svn_fs_commit_txn • Send commit message to replicator Subversion FS Replication FSFSWD Transaction Processing
• Apache Subversion transaction ID is unique within repository • FSFSWD transaction ID is unique across all instances of replicated repository • Transaction to revision is deterministic – rep-cache can be enabled or disabled – repository can be packed Subversion FS Replication FS Replication Features
• Supports all protocols • Supports all clients over network protocols • Negligible overhead for read operations • Far fewer agreements per-commit • Supports file locks • Identical representations of revisions • Easy migration back to FSFS Subversion FS Replication FSFSWD Module Loading
• Apache Subversion supports loadable FS modules • 1.7 requires changes to support 3rd party modules • 1.8 supports 3rd party modules • No FSFSWD code changes in core Subversion • Install two shared libraries and one JAR. Subversion FS Replication FSFSWD Code
• FSFSWD library about 1,000 lines compared to 30,000 lines for FSFS • Mostly marshalling messages over the socket • com.wandisco.subversion JNI to bypass the FSFS vtable • Install two shared libraries and one JAR Subversion FS Replication FSFSWD Performance
• Read performance has negligible overhead – vtable diversion once per-object – no overhead using the object • Write performance small overhead – vtable diversion once per-object – less than 1ms IPC overhead – small FSFS cache overhead • Paxos/replication overhead depends on network topology Subversion FS Replication FSFSWD and Subversion 1.9
• FSFS file locking scales with total number of locks in repository • Unlocking post-commit impacts replication performance • Long-term solution will require a repository upgrade • Multiple path lock and unlock FS operations improve 1.9 performance Thank You [email protected]