Pangolin: A Fault-tolerant Persistent Memory Programming Library Lu Zhang and Steven Swanson Non-Volatile Systems Laboratory Department of Computer Science & Engineering University of California, San Diego 1 Persistent memory (PMEM) finally arrives CPU Caches • Working alongside DRAM Memory • New programming model Controller – Byte addressability DRAM DRAM DRAM – Memory semantics PMEM PMEM PMEM – Direct access (DAX) 2 Challenges with PMEM programming • Crash consistency • Fault tolerance – Volatile CPU caches – Media errors – 8-byte store atomicity – Software bugs MOV CPU L1 L1 L2 3 Persistent memory error types • Persistent memory and its controller implement ECC – ECC-detectable & correctable errors do not need software intervention Application Data Receiving good data PMEM Controller Data Error auto-corrected PMEM Data 4 Persistent memory error types • Persistent memory and its controller implement ECC – ECC-detectable & correctable errors do not need software intervention – ECC-detectable but uncorrectable ones require signal handling Application SIGBUS Receiving SIGBUS PMEM Controller Data Error Detected but PMEM Data uncorrectable 5 Persistent memory error types • Persistent memory and its controller implement ECC – ECC-detectable & correctable errors do not need software intervention – ECC-detectable but uncorrectable ones require signal handling – ECC-undetectable errors demand software detection and correction Application Data Receiving bad data PMEM Controller Data Error undetectable PMEM Data 6 Handle uncorrectable & undetectable errors • Prepare some redundancy for recovery • Implement software-based error detection and correction Application SIGBUS Receiving SIGBUS Application Data Receiving bad data PMEM Controller Data PMEM Controller Data Error Detected but Error undetectable PMEM Data uncorrectable PMEM Data 7 DAX-filesystem cannot protect mmap’ed data File System App. DAX-mmap App. • Some filesystems (e.g. read()/write() mmap(/mnt/pmem/file) NOVA) provide protection protected only via read()/write() User-space unprotected • No known filesystem can File System protect DAX-mmap’ed read()/write() mmap() PMEM data Persistent Memory 8 DAX-filesystem cannot protect mmap’ed data File System App. DAX-mmap App. • Some filesystems (e.g. read()/write() mmap(/mnt/pmem/file) NOVA) provide protection protected Pangolin only via read()/write() User-space • No known filesystem can File System protect DAX-mmap’ed read()/write() mmap() PMEM data Persistent Memory 9 Pangolin design goals • Ensure crash consistency • Protect application data against media and software errors • Require very low storage overhead (1%) for fault tolerance 10 Pangolin – Replication, parity, and checksums • Combines replication and parity as redundancy – Similar performance compared to replication – Low space overhead (1% of gigabyte-sized object store) Metadata Metadata Object Object Object Object Object Object Object Object Object Object Parity • Checksums all metadata and object data 11 Pangolin – Transactions with micro-buffering • Provides micro-buffering-based transactions – Buffers application changes in DRAM – Atomically updates objects, checksums, and parity DRAM Object Object PMEM Object Object Object Object Object Parity 12 Pangolin – Transactions with micro-buffering • Provides micro-buffering-based transactions – Buffers application changes in DRAM – Atomically updates objects, checksums, and parity DRAM PMEM Object Object ObjectObject Object Object Parity 13 Pangolin’s data redundancy • Reserve space for metadata replication and object parity • Organize object data pages into “rows” Application Address Space Mapped PMEM file Metadata Metadata Data Parity Replica Page 0 Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Row 0 Row 1 Row 2 Row 3 Row p Row size: default 160 MB (1% of a data “zone”) 14 Pangolin’s parity coding • Compute a parity page vertically across all rows • Afford losing one whole row of data • By default, Pangolin implements 100 rows per data zone Row 0 Page 0 Page 1 ⊕ ⊕ Row 1 Page 2 Page 3 ⊕ ⊕ Row 2 Page 4 Page 5 ⊕ ⊕ Row 3 Page 6 Page 7 = = Row p Page 8 Page 9 15 Micro-buffering provides transactions • Move object data in DRAM and perform data integrity check • Buffer writes to objects and write back to PMEM on commit • Guarantee consistency with redo logging (replicated) ptr1 2 CSUM CSUM DRAM obj 1 D1’ 3 5 1 Buffering CSUM Logging Writing back D1’ CSUM CSUM CSUM CSUM CSUM PMEM CSUM CSUM Replicating obj 1 D1’D1 obj 2 obj 3 obj 4 obj 5 Row 0 D1’ 4 Updating parity 16 Updating parity using only modified ranges ptr1 2 4 obj 1 D1’ ⊕ = Δ1 ⊕ = P1’ DRAM 1 Logging 3 5 Data D1’ obj 1 D1’D1 obj 2 obj 3 obj 4 obj 5 Row 0 obj 6 obj 7 Row 1 PMEM obj 7 obj 8 Row 2 obj 9 unused (zero bytes) Row 3 Parity PP1’1 Row p 17 Parity’s crash consistency depends on object logs • Apply all redo-logs (if exist) and then re-compute parity obj 1 D1’ ⊕ = Δ1 ⊕ = PowerP1’ failure DRAM Data D1’ obj 1 D1 obj 2 obj 3 obj 4 obj 5 Row 0 obj 6 obj 7 Row 1 PMEM obj 7 obj 8 Row 2 obj 9 unused (zero bytes) Row 3 Parity P1 Row p 18 Parity’s crash consistency depends on object logs • Apply all redo-logs (if exist) and then re-compute parity DRAM Data D1’ obj 1 D1 obj 2 obj 3 obj 4 obj 5 Row 0 obj 6 obj 7 Row 1 PMEM obj 7 obj 8 Row 2 obj 9 unused (zero bytes) Row 3 Parity PP1’ Row p 19 Multithreaded update – Lock parity ranges • Lock a range of parity and serialize parity updates Thread2 Thread1 D1’ ⊕ = Δ1 ⊕ = P1’ D7’ ⊕ = Δ7 ⊕ = P7’ DRAM 1 2 3 4 Data D1’ obj 1 D1 obj 2 obj 3 obj 4 obj 5 Row 0 obj 6 obj 7 Row 1 PMEM D7’ obj 7 D7 obj 8 Row 2 obj 9 unused (zero bytes) Row 3 Parity PPP171’1 Row p 20 Multithreaded update – Atomic XORs • Parity range can update, lock-free, with atomic XORs Thread1 Thread2 D1’ ⊕ = Δ1 D7’ ⊕ = Δ7 DRAM Data D1’ obj 1 D1 obj 2 obj 3 obj 4 obj 5 Row 0 obj 6 obj 7 Row 1 PMEM D7’ obj 7 D7 obj 8 Row 2 obj 9 unused (zero bytes) Row 3 Parity PP171 Row p 21 Multithreaded update – Hybrid scheme • Atomic XORs can be slower than vectorized ones • Use shared mutex to coordinate both methods • Small updates (< 8KB) – Take shared lock of a parity range (8 KB) – Update parity concurrently with atomic XORs • Large updates (≥ 8KB) – Take exclusive locks of parity ranges (8 KB each) – Update parity using vectorized XORs (non-atomic) 22 Performance – Single-object transactions Single-object Overwrite Latencies • Evaluation based on Intel’s libpmemobj libpmemobj-replication pangolin Optane DC persistent memory 20 15 • On average, Pangolin’s latency is 10 11% lower than libpmemobj with 5 0 replication. (microseconds) Latency 64 256 1024 4096 Object Size (bytes) 23 Performance – Multi-object transactions • Performance of Pangolin is 90% of libpmemobj’s with replication • Pangolin incurs about 100× less space overhead Average Insertion Latencies Average Removal Latencies libpmemobj libpmemobj-replication pangolin libpmemobj libpmemobj-replication pangolin 25 20 20 15 15 10 10 5 5 Latency (microseconds) Latency Latency (microseconds) Latency 0 0 ctree rbtree btree skiplist rtree hashmap ctree rbtree btree skiplist rtree hashmap 24 Conclusion • PMEM programming libraries should also consider fault tolerance for critical applications. • Parity-based redundancy provides similar performance compared to replication and significantly reduces space overhead. • Micro-buffering-based transactions can both support crash consistency and provide fault tolerance. 25.