Using Concurrent Relational Logic with Helpers for Verifying the AtomFS File System

Mo Zou1, Haoran Ding1, Dong Du1, Ming Fu2, Ronghui Gu3, Haibo Chen12 1 IPADS, Shanghai Jiao Tong University 2 Huawei Technologies Co. Ltd 3 Columbia University File systems are buggy and underspecified

• 40% of FS patches fix bugs [Lu et al., FAST’13] – 20% of the bugs are concurrency bugs • Hard to eliminate due to many possible interleavings

• POSIX is vague about concurrent behavior – E.g., unclear whether an operation should be atomic – Hard to reason about higher-level applications

2 Approach: formal verification

• Concurrent implementation meets specification – Under arbitrary interleavings – Proof checked by proof assistant (Coq) • Avoid large classes of bugs • Specification serves as a precise interface

3 Verification efforts

• File system verification – FSCQ project [SOSP’15,SOSP’17,Tej M.S. thesis] No fine-grained – Yggdrasil [OSDI’16] concurrency – Cogent [ASPOLOS’16] • Concurrent system verification – CertiKOS [OSDI’16] Not applicable – CSPEC [OSDI’18] to FS Goal: verify a fine-grained, concurrent file system 4 Contributions

• CRL-H: Concurrent Relation Logic with Helpers for concurrent file systems – Helper mechanism – Proofs mechanically checked by Coq • AtomFS: the first verified concurrent FS with fine-grained locking – Fine-grained: per-inode lock (no crash-safety) – Atomic interfaces – Verified directly in C language

5 How to specify “correct”?

mkdir(/a), succss unlink(/b), failure Sequential Sequential file system history

For a sequential file system, correct if sequential history is legal

unlink(/b), failure Concurrent Concurrent file system execution mkdir(/a), succss

How to describe concurrent via sequential?

6 This work: “correct” means linearizability

Linearizability: describe concurrent via sequential unlink(/b), failure Concurrent execution mkdir(/a), succss map to Sequential history

linearization point (LP)---effect happens atomically Correct if equivalent “sequential” history is legal

7 Prove linearizability via forward simulation

multi steps LP multi steps mkdir C0 Ci Ci+1 Cn Concrete state

Equivalent effect

Abstract MKDIR A0 A1 Abstract state MKDIR(/a/b) / / Define abstract a state and abstract a b operation Abstract file system

as a tree 8 Prove linearizability via forward simulation

multi steps LP multi steps mkdir C0 Ci Ci+1 Cn Concrete state effect

Abstract MKDIR A0 A1 Abstract state effect Decide linearization point Define abstraction relation mkdir MKDIR 9 Prove linearizability via forward simulation

Simulation proof unlink(/b), failure LP C0 Ci Ci+1 Cn Concurrent effect execution mkdir(/a), success Abstraction relation A0 A1 effect

Linearization Proven by Liang et al, PLDI’13 MKDIR(/a), success UNLINK(/b), failure Filipovic, ESOP’09

Construct linearization of abstract operation Linearizability following temporal order of LPs

10 Strawman: fixed LP in critical section

/* error and corner cases Pattern of path-based operations mkdir(path) handling omitted*/ def mkdir(path) split(path, dir, name); 1. Invocation begins

// traverse path from root look(root); 2. Pathname resolution fat = locate(root, dir);

// fat’s lock is held node = init(); 3. Lock-protected critical insert(fat, name, node); section (where updates LP of mkdir happen) unlock(fat); 4. Invocation returns return success; 11 Strawman: fixed LP in critical section

begin lookup LP return mkdir(path)

begin lookup LP return rename(src, dst)

If fixed LP is correct and We can construct linearization implementation is linearizable for any concurrent execution

12 Challenge: fixed LP could fail in linearization Two threads begin lookup LP success mkdir(/a/b/c)

rename(/a,/d) begin lookup LP success

/ a b Initially

13 Challenge: fixed LP could fail in linearization Two threads begin lookup LP success mkdir(/a/b/c)

rename(/a,/d) begin lookup LP success

/ / a a b b Initially mkdir traverses to b 14 Challenge: fixed LP could fail in linearization Two threads begin lookup LP success mkdir(/a/b/c)

rename(/a,/d) begin lookup LP success

/ / / d a a b b b Initially mkdir traverses rename finishes to b 15 Challenge: fixed LP could fail in linearization Two threads begin lookup LP success mkdir(/a/b/c)

rename(/a,/d) begin lookup LP success / / / / d d a a b b b b c Initially mkdir traverses rename finishes mkdir succeeds to b Legal interleaving 16 Challenge: fixed LP could fail in linearization begin lookup LP success mkdir(/a/b/c)

rename(/a,/d) begin lookup LP success Map to abstract operation

Not legal RENAME(/a, /d) MKDIR(/a/b/c) success success (Should be failure) 17 Challenge: fixed LP could fail in linearization begin lookup LP success mkdir(/a/b/c)

rename(/a,/d) begin lookup LP success ？ Legal Cannot obtain legal linearization using fixed LPs MKDIR(/a/b/c) RENAME(/a, /d) success success Check other cases All failed cases involve rename 18 Observation: rename modifies other Op’s traversed path

• We call this phenomenon path inter-dependency – Rename, only operation that can modify an internal inode

mkdir(/a/b/c) LP /

rename(/a,/d) LP d

b Path inter- dependency Allowed by fine-grained implementation 19 Should consider path inter-dependency in linearization

/ RENAME(/a,/d) Linearization strategy break MKDIR(/a/b/c)’s (linearize at LPs) is insufficient d path integrity b

linearize before Fix linearization strategy to consider path inter-dependency RENAME(/a, /d) MKDIR(/a/b/c)

Approach: also linearize when path inter-dependency happens 20 Approach: linearize when path inter-dependency happens

/ mkdir(/a/b/c)⤻ success fixed LP Concurrent a rename(/a, /d) success fixed LP execution b ⤻ rename helps external LP commit mkdir Sequential history MKDIR(/a/b/c) RENAME(/a, /d) success success

• The LP of Op1 (e.g., mkdir) resides in another Op2 (e.g., rename) – This kind of LP is called external linearization point • For path-based Op, LP could be internal (“fixed LP”) or external (triggered by rename) 21 Helping: linearize abstract operations of other threads

• Helping: linearize abstract operations of other threads [Liang et al, PLDI’13]

Shared thread pool E.g., (mkdir, args) or Exist logically in (mkdir end, ret) abstract model Thread ID AopState … …

Current thread t-2: rename Linearized t-1 (mkdir, args) t-1 (mkdir end, ret) Helping

22 File system-specific challenges

• Which threads to help (for a rename)? • Helping order? • Handle recursive path inter-dependency root rename

mkdir create stat Help many Decide helping set and order

23 File system-specific challenges

• Which threads to help (for a rename)? • Helping order? rename-1 • Handle recursive path inter-dependency

rename-2 Path inter- dependency … rename-3 Recursive path inter-dependency

24 Helpers: extend helping with file system-specific notions

• Helper metadata provides global information – E.g., add “lock path” in Descriptor to record traversed path

• Decide whether Op1 should be linearized before Op2 – E.g., rename can use “lock path” to decide which threads to help

root Helper metadata Linearize before rename Shared thread pool Thread ID AopState Descriptor … mkdir create stat Helper extensions Should be helped 25 Read paper Specifying and proving with CRL-H for details CRL-H framework: Concurrent Relation Logic with Helpers Rely; Guarantee; Invariant ⊢ {Pre ∗ (aop, args)} Code {Post ∗ (aop end, ret)}

Specification Implementation Abstraction Concurrent file Local rely guarantee Invariants R/G conditions and Aops system code for fine-grained concurrency Import to Coq Inference rule Proof modelled C

Simulation Compile with with helpers C compiler

Linearizability Fuse/Kernel FS 26 Read paper Specifying and proving with CRL-H for details CRL-H framework: Concurrent Relation Logic with Helpers Rely; Guarantee; Invariant ⊢ {Pre ∗ (aop, args)} Code {Post ∗ (aop end, ret)} RENAME rename(args) Specification Implementation Abstraction Concurrent file Local rely guarantee Invariants R/G conditions and Aops system code for fine-grained concurrency Import to Coq Inference rule Proof modelled C Relational reasoning

Simulation Compile with with helpers C compiler

Linearizability Fuse/Kernel FS 27 Read paper Specifying and proving with CRL-H for details CRL-H framework: Concurrent Relation Logic with Helpers Rely; Guarantee; Invariant ⊢ {Pre ∗ (aop, args)} Code {Post ∗ (aop end, ret)} RENAME rename(args) Specification Implementation Abstraction Concurrent file Local rely guarantee Invariants R/G conditions and Aops system code for fine-grained concurrency Import to Coq Inference rule Proof modelled C Relational reasoning Inference rule Simulation Compile with with helpers C compiler

Linearizability Fuse/Kernel FS 28 Read paper Specifying and proving with CRL-H for details CRL-H framework: Concurrent Relation Logic with Helpers Rely; Guarantee; Invariant ⊢ {Pre ∗ (aop, args)} Code {Post ∗ (aop end, ret)} RENAME rename(args) Specification Implementation Abstraction Concurrent file Local rely guarantee Invariants R/G conditions and Aops system code for fine-grained concurrency Import to Coq Inference rule Proof modelled C Relational reasoning Inference rule Simulation Compile with with helpers C compiler Correctness theorem

Linearizability Fuse/Kernel FS 29 Read paper Specifying and proving with CRL-H for details CRL-H framework: Concurrent Relation Logic with Helpers Rely; Guarantee; Invariant ⊢ {Pre ∗ (aop, args)} Code {Post ∗ (aop end, ret)} RENAME rename(args) Specification Implementation Abstraction Concurrent file Local rely guarantee Invariants R/G conditions and Aops system code for fine-grained concurrency Import to Coq Inference rule Proof modelled C Relational reasoning Inference rule Simulation Compile with with helpers C compiler Correctness theorem C language modeling Linearizability Fuse/Kernel FS 30 Invariants in proving AtomFS

Abstract-concrete relation Finding all and precise specification Non-bypassable invariant is difficult! Invariants Good-FS-Tree Helper-metadata-consistency Always … hold on Necessary for simulation proof Shared states Abstract & concrete tree Helper metadata

31 Operation bypassing leads to non-linearizability

/ Op-2 holds no Op-1 a lock

b

c Op-1 bypasses Op-2

32 Operation bypassing leads to non-linearizability

Read paper for a concrete case Path inter- Op-1 bypasses op-2 dependency and modifies the state op-2 will use

Concurrent rename op-1 execution op-2 op-2 ⤻ return Illegal Sequential Op-2 return RENAME Op-1 history ⤻

Construct a non-linearizable interleaving 33 Lock coupling forbids operation bypassing

2. Release current • Non-bypassable invariant to capture the / property Op-2 Op-1 a • Cons: reduce parallelism • Pros: ensure linearizability b 1. Acquire next – Easier to reason about for users

c Tradeoff between Forbid bypassing by always holding a lock performance and reasoning!

34 Implementing CRL-H and AtomFS in Coq

• 1.5 years of effort, including building the framework and proving AtomFS

• CRL-H, ~100k LOC Basic libraries 34,820 – Most can be reused AtomFS 63,099 • AtomFS Machine & – 673 lines of C code logic 38,904 – 2k lines of specification Proof automation – 60k lines of proof 24,387

35 Evaluation: AtomFS achieves reasonable performance

• Single core performance • Faster than DFSCQ (1.38x-2.52x) – Avoid Haskell overhead 20 dfscq 17.5 atomfs tmpfs • Slower than ext4 and tmpfs 15 ext4 12.5 – FUSE overhead 10 – 7.5 Simplified data strucutre 5 Running time (seconds) 2.5 0 largefile smallfile git-clone make-xv6 cp-qemu ripgrep 36 Evaluation: AtomFS achieves reasonable performance

• Multicore scalability 8 AtomFS ext4 7

• Better scalability than ext4 6 – Not bypass VFS-level path lookup 5 – Bottleneck: lock coupling traverse Speedup 4 3 • Worse performance than ext4 2 1 2 4 6 8 10 12 14 16 – 6.39x lower throughput with 16cores Thread Number – Not implement optimizations Speedup on Fileserver (compared to single core)

37 Conclusion

• CRL-H: specify and prove concurrent file systems – Path inter-dependency and external LP challenge – Helper mechanism • AtomFS: first verified concurrent FS with fine-grained locking – Atomic interfaces – Reasonable performance

https://ipads.se.sjtu.edu.cn/projects/atomfs Thanks!

38