Scalable Multi-Core Model Checking: Technology & Applications of Brute

UNIVERSITY OF TWENTE. Formal Methods & Tools.

Scalable Multi-core Model Checking: Technology & Applications of Brute Force Part III: Symbolic

Jaco van de Pol 30, 31 October 2014 VTSA 2014, Luxembourg ... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 2 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Sources on Binary Decision Diagrams

Papers/Tutorials (1990’s)

I H.R. Andersen, An Introduction to Binary Decision Diagrams

I R.E. Bryant, Symbolic Boolean Manipulation with Ordered Binary-Decision Diagrams

Tools I BDD-packages: BuDDy, CuDD, Java(B)DD, multi-core: Sylvan

I Symbolic model checker: nuSMV http://nusmv.fbk.eu/

I LTSmin: http://fmt.cs.utwente.nl/tools/ltsmin/

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 3 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Binary Decision Diagrams

Binary Decision Diagram

I A Binary Decision Diagram is a directed acyclic graph

I Its internal nodes are ordered, binary (called low, high)

I Its internal nodes are labeled by variables

I Its leaves are labeled by 0 or 1

Example Conventions I Internal nodes are drawn as circles X I High edges are drawn solid

I Low edges are drawn dashed Y Y I Leaves are drawn as boxes, with 0 or 1

I “If X is true, then high, else low branch” 1 0 I Formula on the left: X ⇔ Y

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 4 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... How to interpret a BDD?

Boolean Functions – or sets of Boolean vectors I Let X = {x1,..., xn} be Boolean variables I A valuation is a function X → {0, 1} I A BDD represents a set of valuations

I all valuations that lead from the root to leaf 1 are in the set I valuations that lead from the root to leaf 0 are not in the set n I Equivalently, a BDD represents a function {0, 1} → {0, 1} Hint X You can read the BDD as one of:

B1 B2

I If X then B1 else B2. Notation: X → B1, B2

I (X ∧ B1) ∨ (¬X ∧ B2).

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 5 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Examples

Basic Boolean Connectives x ¬x x ∧ y x ∨ y X X

X X Y Y

1 0 0 1 1 0 1 0

Propositional logic formulas

I Apparently, BDDs form an alternative to proposition logic.

I Recall negation ¬ and the binary connectives: ∧, ∨, ⇒, ⇔

I How many binary operators are possible? . . . suﬃcient?

I Introduce one ternary operator: x → s, t;... suﬃcient basis!

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 6 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... More Examples

Three times: (x ∧ y) ∨ z

X X X

Y Y Y Z Z

Z Z Z Z Y

1 1 1 0 0 1 0 1 0

Reduced BDDs: Ordered BDDs: I no duplicate nodes I The order of the vars is ﬁxed

I no redundant tests I The order impacts BDD size

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 7 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Reduced Ordered BDDs (ROBDD = OBDD)

Reduced BDDs A BDD is called reduced iﬀ:

I No duplicate leafs: There is at most one leaf with label 0 and one with label 1.

I No duplicate nodes: For all nodes v, w, if var(v) = var(w), low(v) = low(w) and high(v) = high(w), then v = w.

I No redundant tests: For all nodes v, low(v) 6= high(v).

Ordered BDDs A BDD is called ordered iﬀ

I there exists an ordering x1 < x2 < ··· < xn, such that I for all nodes v in the BDD, var(v) < var(low(v)) and var(v) < var(high(v))

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 8 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Stepwise transformation from BDD to (R)OBDD A BDD can (in principle) be transformed to an OBDD by repeated application of the following transformation rules:

Stepwise reduction Stepwise ordering I Re-order nodes (p < q) I Eliminate duplicate nodes: q p X X X

⇒ p C ⇒ q q A B A B A B A B C

I Eliminate redundant tests: I Eliminate double tests X p p ⇒ A p C ⇒ A C A A B

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 9 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Variable ordering can make an exponential diﬀerence

(x1 ⇔ y1) ∧ (x2 ⇔ y2) ∧ (x3 ⇔ y3) (edges to 0 are suppressed)

x1 x1

x2 x2 y1 y1

x3 x3 x3 x3 x2

y1 y1 y1 y1 y1 y1 y1 y1 y2 y2

y2 y2 y2 y2 x3

y3 y3 y3 y3

1 1 3.2n − 2 nodes 3.n + 1 nodes

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 10 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Theoretical Results

Existence and Uniqueness For a ﬁxed variable ordering (X , <):

I every Boolean function can be represented,

I by a canonical (unique up to isomorphism) OBDD Ordering

I The chosen ordering has a huge impact on the OBDD size

I Finding the optimal ordering is NP-hard I Some functions only admit exponentially large OBDDs

I E.g.: multiplication P(~x,~y,~z) such that

(x1 ... xn) ∗ (y1 ... yn) = (z1 ... z2n)

needs O(2n) OBDD nodes, whatever ordering is chosen

I In practice, many functions have small OBDD representations

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 11 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 12 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... OBDD packages

Regard OBDD as abstract datatype

I Manipulation of OBDDs through pointers / objects

I Basic constructors ensure invariant “Reduced & Ordered”

I Operations on OBDDs implement logical connectives: Illustration (5 < 100 functions in C-interface of BuDDy) BDD bdd_high (BDD r) BDD bdd_not (BDD r) BDD bdd_apply (BDD l, BDD r, int op) BDD bdd_exist (BDD r, BDD var) BDD bdd_relprod (BDD l, BDD r, BDD var)

Implementation

I Data structures (unique table, operation caches)

I Operations are based on a generic Apply-function

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 13 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Data structure: Unique Table

Keep maximal sharing and avoid redundant tests

I This is a hash table, to ensure unicity of all BDD nodes

I It assigns a unique number to each triple: N ↔ hvar, NL, NH i I One can lookup var(N), low(N), high(N) in O(1) time.

MakeNode(x, NL, NH ) = N (create new nodes)

Require: variable x, nodes NL, NH Ensure: a unique node N denoting (¬x ∧ NL) ∨ (x ∧ NH ) 1: if NL = NH then 2: N := NL 3: else if hx, NL, NH i is in the unique table then 4: N := lookup(x, NL, NH ) 5: else 6: N := insert new entry(x, NL, NH ) in the unique table 7: end if

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 14 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Naive function for conjunction: ∧

ApplyAnd(N1, N2) = N

Require: BDD nodes N1, N2 Ensure: BDD node N representing N1 ∧ N2 1: if N1 = 0, N1 = 1, N2 = 0, or N2 = 1 then 2: N := 0, N2, 0, N1, respectively 3: else 4: x1, l1, r1 := var(N1), low(N1), high(N1) 5: x2, l2, r2 := var(N2), low(N2), high(N2) 6: if x1 = x2 then 7: N := MakeNode(x1,ApplyAnd(l1, l2),ApplyAnd(r1, r2)) 8: else if x1 < x2 then 9: N := MakeNode(x1,ApplyAnd(l1, N2),ApplyAnd(r1, N2)) 10: else if x1 > x2 then 11: N := MakeNode(x2,ApplyAnd(N1, l2),ApplyAnd(N1, r2)) 12: end if 13: end if

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 15 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Problem with naive recursion

Naive Complexity

I Consider a BDD with n nodes, but a lot of sharing n I The BDD can have O(2 ) diﬀerent paths (!) n I Hence the naive APPLY takes O(2 ) recursive calls Solution: Dynamic Programming I Store all intermediate results in an operation cache

I ﬁrst check if the result is already in the cache I if not, compute it and put the result in the cache

I This is a well-known technique, e.g. Fibonacci sequence Ultimate Complexity

I Given OBDDs A with m nodes and B with n nodes,

I There can be at most m · n pairs of nodes from A and B

I So with dynamic programming, APPLY takes O(m · n) time

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 16 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Data structure: Operation Cache

Apply(op, N1, N2) = N ...... (recursive cases only)

1: if (op, N1, N2) is in the operation cache then 2: N := lookup(op, N1, N2) 3: else 4: x1, l1, r1 := var(N1), low(N1), high(N1) 5: x2, l2, r2 := var(N2), low(N2), high(N2) 6: if x1 = x2 then 7: N := MakeNode(x1,Apply(op,l1, l2),Apply(op,r1, r2)) 8: else if x1 < x2 then 9: N := MakeNode(x1,Apply(op,l1, N2),Apply(op,r1, N2)) 10: else if x1 > x2 then 11: N := MakeNode(x2,Apply(op,N1, l2),Apply(op,N1, r2)) 12: end if 13: add (op, N1, N2) 7→ N to the operation cache 14: end if

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 17 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Why is your BDD package better than mine?

Black magic

I Variable ordering: sometimes even dynamically reordered

I Garbage collection on the unique table

I Operation cache replacement strategy

I And even: eﬀect on L2 cache versus main memory

And what about the user?

I Performance depends on how the BDD package is used

I Start with a good initial variable ordering

I Think about the order of applying operations

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 18 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 19 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Breadth First Search, example

BFS traversal

0 1 3

1 4

2 3 5

I Eventually all states will be visited

I Shortest path will be detected

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 20 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Breadth First Search

Explicit-state BFS Set-based BFS (variant 1)

1: check that init 6∈ Error 1: Vis := Cur := {init} 2: Queue := [init] 2: while Cur 6= ∅ do 3: Visited := {init} 3: check Cur ∩ Error = ∅ 4: while Queue 6= [ ] do 4: Cur := Next(Cur, →) \ Vis 5: pick s from front of Queue 5: Vis := Vis ∪ Cur 6: for all t with s → t do 6: end while 7: if t 6∈ Visited then 8: check that t 6∈ Error Set-based BFS (variant 2) 9: put t to the end of Queue 10: add t to Visited 1: Vold := ∅ 11: end if 2: Vnew:= {init} 12: end for 3: while Vold 6= Vnew do 13: end while 4: Vold:=Vnew 5: Vnew := Vold ∪ Next(Vold, →) Set operations: ∩, ∪, S, =, Next 6: end while

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 21 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 22 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Kripke structures

Deﬁnition

A Kripke structure is a tuple (S, S0, R, AP, L), where I S is a set of states

I S0 ⊆ S is set of initial states I R ⊆ S × S is a total transition relation on S

I ∀s ∈ S. ∃t ∈ S. R(s, t)

I AP is a set of atomic proposition labels

I L : S → P(AP) assigns to each state a set of labels

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 23 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Example of Kripke Structure

Mutual Exclusion / Critical Section

s0 s1 s2

{T1} {C1}

s3 s4 s5

{T2} {T1, T2} {C1, T2}

s6 s7 s8

{C2} {T1, C2} {C1, C2}

Parts of the Kripke Structure tuple:

I States S = {s0,..., s8}; Initial states S0 = {s0}.

I R = {(s0, s1), (s1, s2),...}

I AP = {T1, C1, T2, C2} (trying, critical) I L(s0) = ∅; L(s4) = {T1, T2}; L(s7) = {T1, C2},...

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 24 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Boolean Encoding of Kripke Structure: states

Encoding in Booleans (=bits)

I Virtually everything can be encoded in bits (as you know)

I We would like to preserve structure as much as possible

I Here we choose the atomic propositions to encode states

Encoding of this example: use variables {T1, T2, C1, C2} I States correspond to formulas: s s s 0 1 2 s0 = ¬T1 ∧ ¬T2 ∧ ¬C1 ∧ ¬C2 {T } {C } 1 1 s4 = T1 ∧ T2 ∧ ¬C1 ∧ ¬C2

s3 s4 s5 I Set of states are also formulas:

{T2} {T1, T2} {C1, T2} {s1, s4, s7} = T1 {s6, s7, s8} = C2 s6 s7 s8 I Set of reachable states? {C2} {T1, C2} {C1, C2} ¬(C1 ∧C2)∧¬(T1 ∧C1 ∨T2 ∧C2)

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 25 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Boolean Encoding of Kripke Structure: transitions

Encoding of States and Transitions

I We have encoded sets of states as formulas P(T1, T2, C1, C2). 0 0 0 0 I Transitions relate (T1, T2, C1, C2) and (T1, T2, C1, C2) I Encode transitions as formulas: 0 0 0 0 Q(T1, T1, T2, T2, C1, C1, C2, C2).

Encoding of transitions:

s0 s1 s2 I Represent transitions by formulas: {T1} {C1} 0 0 I Green: ¬T2 ∧ ¬C2 ∧ T2 ∧ ¬C2 0 0 I Red: C1 ∧ ¬C2 ∧ ¬C1 ∧ ¬T1 s3 s4 s5

{T2} {T1, T2} {C1, T2} I This is not quite true: also ensure other variables don’t change: s s s 0 0 6 7 8 I Green: . . . ∧ T1 = T1 ∧ C1 = C1. {C } {T , C } {C , C } 0 0 2 1 2 1 2 I Red: . . . ∧ T2 = T2 ∧ C2 = C2.

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 26 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Next-state by Relational Product

Relational Product: the problem

I Given: some set of states S, and a relation R 0 I Represented by: P(~x) and Q(~x, x~ ).

I Required: the R-successors of S, i.e. {t | ∃s ∈ S. s R t}

Relational product: the solution 0 I Step 1: Simply take the conjunction: P(~x) ∧ Q(~x, x~ ) 0 I Step 2: Abstract previous states: ∃~x. P(~x) ∧ Q(~x, x~ )

I Step 3: Rename back to original state variable names: ∃~x. P(~x) ∧ Q(~x, x~0)[~x/x~0]

What about existential quantiﬁcation?

I ∃xi . P(~x) is simply: P[xi := 0] ∨ P[xi := 1] I Note: size can double, so ∃~x.P can be exponentially big!

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 27 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... We are ﬁnished!

Recall Set-BFS (variant 1) Recall Set-BFS (variant 2)

1: Vis := Cur := {init} 1: Vold := ∅ 2: while Cur 6= ∅ do 2: Vnew:= {init} 3: check Cur ∩ Error = ∅ 3: while Vold 6= Vnew do 4: Cur := Next(Cur, →) \ Vis 4: Vold:=Vnew 5: Vis := Vis ∪ Cur 5: Vnew := Vold ∪ Next(Vold, →) 6: end while 6: end while

Implementation with BDDs

I Vis, Cur: Binary Decision Diagrams (BDDs)

I ∩: BDDapplyAnd

I ∪: BDDapplyOr

I Next: BDDRelProd

I 6=: pointer comparison (unique representation!)

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 28 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 29 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Partitioning of the next-state function

Realistic systems are composed naturally

I Each component uses only a subvector of the state variables

I Each component i has its own (simple) next-state relation Ri

Synchronous systems: conjunctive partitioning

I In hardware: all components do a step at clock ticks

I So the relation of the system is: R1 ∧ R2 ∧ · · · ∧ Rn.

A-synchronous systems: disjunctive partitioning

I In parallel software: transitions are interleaved, non-determinism

I So the relation of the system is: R1 ∨ R2 ∨ · · · ∨ Rn. I (In practice also: “the other variables are unchanged”)

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 30 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Four tricks to make symbolic methods more eﬃcient

Disjunctive Partitioning

I Handle all subtransitions Ri separately

I Never compute the expensive R = R1 ∨ · · · ∨ Rn Chaining

I Apply R2 on the result of applying R1, etc. ∗ ∗ I Basically, compute (R2 ◦ R1) (S) instead of (R1 ∪ Rn) (S) Relational Product I Interweave the EXIST and APPLY operations

I Abstract variables as soon as they are introduced by APPLY Variable Ordering 0 I Keep variables from same component together; also x and x

I Or: dynamically reorder variables during computation

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 31 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Symbolic Reachability algorithm, revisited

Reachable(I , N, Ri ) = Vnew Require: I , BDD representing initial states Require: Ri , BDDs, representing subtransitions Ensure: Vnew BDD representing the reachable states from I by Ri Vold := BDDempty Vnew:= I while not BDDequal(Vold, Vnew) do Vold:=Vnew for i = 1 to N do Vnew := BDDapply(∨, Vnew, BDDrelprod(Vnew, Ri )) end for end while

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 32 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 33 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Recall LTSmin architecture and PINS interface

Specification mCRL2 Promela DVE UPPAAL Languages x y z PINS t1 r w – t2 – r w Reachability t w – rw Distributed Multi−core Symbolic 3 Tools

Advantages of tool and interface (LTSmin / PINS)

I General and ﬂexible: support for arbitrary state/edge labels

I Also: LLVM, parity games, Markov Automata, C-code, B||CSP I Indirectly: GSPN, xUML, Signalling Networks in Biology

I On-the-ﬂy API: next-state function to pull the implicit graph

I Eﬃciency: models expose locality in a dependency matrix

I How to do symbolic model checking on-the-ﬂy?

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 34 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... LTSmin architecture and PINS interface Blom, van de Pol, Weber [CAV’10], Laarman, van de Pol, Weber [NFM’11] http://fmt.cs.utwente.nl/tools/ltsmin/

Specification mCRL2 Promela DVE UPPAAL Languages

PINS

Pins2pins Transition Variable reordering Partial−order Wrappers caching Transition grouping reduction

PINS

Reachability Distributed Multi−core Symbolic Tools

Analysis BisimulationLTL mu−calculus Algorithms reduction / lumping

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 35 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Example Dependency Matrix int x=7; int y=3; int z=9; process p1() { process p2() { process p3() { do do do ::{x>0 => x--;y++} ::{y>0 => y--;x++} ::{z>0 => z--;x++} ::{x>0 => x--;z++} ::{y>0 => y--;z++} ::{z>0 => z--;y++} od } od } od }

Default Matrix Better Matrix init state= h7, 3, 9i

p1.1 x y z x y z h7, 3, 9i −→ h6, 4, 9i p1.1 p1 + + +  p1.1 + + −  h7, 3, ∗i −→ h6, 4, ∗i p2 + + +  p1.2 + − +    p3 + + + p2.1 + + −  p3.2   h7, 3, 9i −→ h7, 4, 8i p2.2  − + +  p3.2   h∗, 3, 9i −→ h∗, 4, 8i p3.1 + − +  p3.2 − + + Static Regrouping

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 36 / 61

x y z p1.1, 2.1 + + −  p1.2, 3.1 + − +  p2.2, 3.2 − + + ... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Caching Local Transitions

I Consider local transition in speciﬁcation: p3.2: atomic { z>0 -> z--; y++ } x y z I Dependency matrix row: p3.2 0 1 1

I Deﬁne projection: πp3.2hx, y, zi = hy, zi

Next, consider two consecutive calls to p3.2 ﬁrst call: hx, y, zi second call: hx00, y, zi successor: hx, y 0, z0i project: hy, zi project and hy, zi → cache lookup: → hy 0, z0i store in cache: hy 0, z0i expand: hx00, y 0, z0i

I Transition caching saves calls to the language module

I Memoization table cache[i] for each transition group i

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 37 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 38 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Multi-valued Decision Diagrams

4 3 4 5 6 4 4 1 3 4 8 4 4 1 5 4 5 3 5 5 4 4 1 4 5 6 4 1 3 6 8 5 3 4 8 4 1 3 3 5 5 4 1 3 4 5 4 5 6 5 6 3 3 4 8 5 6 3 41 6 3 5 5 5 6 3 1 3

I Every path in the MDD represents a concrete state vector

I Potential gain in memory saving: exponential (here: 54 → 15)

I Symbolic Reachability: explore sets of states stored as MDDs

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 39 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... On-the-ﬂy Symbolic Reachability

I L, V : MDDs for sets of long state vectors (level, visited)

I Ri : MDDs to store transition relation i on short vectors I Li , Vi : MDDs for sets of short state vectors (level,visited for i) symbolic-reachability: learning transition relation

(1) L := {InitState()}; V := L; all Ri := ∅; all Vi := ∅ (2) while L 6= ∅ do (3) for i ∈ groups do (4) Li := πi ([D]N×K , L) \ Vi ; Vi := Vi ∪ Li 0 0 (5) Ri := Ri ∪ {(s, s ) | s ∈ Li ∧ s ∈ NextState(i, s)} S (6) L := i (RelProd(Ri , L) \ V ); V := V ∪ L (7) return V

I InitState and NextState come from PINS I RelProd, Or and Diff are MDD operations

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 40 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Symbolic Reachability

Symbolic Reachability with PINS

I Global set of reachable states is computed as ﬁx point

I Stored as a multi-valued decision diagram (MDD)

I Learn symbolic sub-groups Ri on-the-ﬂy (via NextState)

Extensions

I Multiple exploration strategies: ∗ I Breadth-ﬁrst:( T1 + T2 + ··· + Tn) ∗ I Chaining:( T1 ◦ T2 ◦ · · · ◦ Tn) ∗ ∗ ∗ ∗ ∗ I Saturation: ((((T1 ◦ T2) ◦ T3) ··· ) ◦ Tn) I Static variable reordering boosts performance

I Multiple BDD packages: BuDDy, ListDD, Sylvan (later)

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 41 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Enjoy playing

50 I Now > 2 states in seconds, also for PROMELA and mCRL2

I Can connect to any other explicit-state language (ﬁx state vector, split next-state function, dependency matrix) I mCRL2 example: xUML Interlocking model

I level 16 has 165499132954291200000 states (7286 nodes) (so we switched to gmp-bignum:1 .6 · 1020 states) I Promela example: GARP protocol [Igor Konnov, Vienna]: 11 I LTSmin explored 3 · 10 states in < 3 min using < 300MB.

I Indication for success: sparse dependency matrix

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 42 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 43 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Earlier work: disappointing speedups

Earlier work in parallel BDDs

I early ’90s: vector machines, massive SIMD (not unlike GPU)

I late ’90s: virtual SMP, distributed BDDs (BDDnow)

Earlier work in parallel symbolic model checking

I ’00vv: Grumberg et al: vertical splitting of BDDs (distributed)

I ’00vv: Ciardo et al: horizontal splitting of BDDs (distributed)

I CAV’07: L¨uttgenet al: - parallelisation of saturation with Cilk

I PDMC’09: Ciardo - Diﬃcult, but what is the alternative?

Recent developments

I 2010: J. Ossowski - Jinc: Multi-threaded decision diagrams

I 2012: Tom van Dijk, Alfons Laarman, JvdP: Sylvan, a library for multi-core BDD operations (PDMC’12)

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 44 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Symbolic Model Checking based on BDDs Bryant [IEEE Trans. Comp.’86], Burch,Clarke, McMillan [LICS’90]

BDD data structures

I Unique Table (to store BDD nodes) I Computed Cache (apply operations) X

Symbolic Reachability (chaining strategy) Y Require: I : initial state, R1,..., RN : subtransitions S Ensure: Vnew: set of reachable states from I by Ri 1: Vold :=0 ; Vnew := I Z 2: while Vold 6= Vnew do 3: Vold:=Vnew 4: for i = 1 to N do 5: Vnew := Vnew Or RelProd(Vnew, Ri ) 1 0 6: end for 7: end while

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 45 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Multi-core Binary Decision Diagrams Tom van Dijk, Alfons Laarman, van de Pol [pdmc’12]

Multi-core BDDs

I Use shared hashtable for Unique Table, Operations Cache

I Parallelize computation tree of recursive operations (Apply)

Complications for Parallelism Solutions

I BDD nodes can be removed I Tombstones, garbage collect

I Irregular task graph I Work-stealing

I Fine-grained parallelism I Use split deque

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 46 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 47 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Reuse the Lockless Concurrent Hash Table?

Data structures for Binary Decision Diagrams

I Unique Table: ensure that BDD nodes exist at most once

I implements the maximal sharing requirement I necessary for constant equality check by pointer comparison I Computed Table: dynamic programming

I Used to store intermediate results to avoid recomputations I Needed to manipulate BDDs in polynomial time

I Both data structures are usually implemented as hash tables

I But: Unique Table requires garbage collection in practice

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 48 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Algorithm to put an entry in Computed Table

I Local bit-lock in hash array controls access to data array I Every cache-hit is nice, but don’t lose time I No waiting for locks: just give up I No hash collision resolution: just overwrite or give up

Input: key, data 1: hash ← calculate hash(key) 2: index ← hash % tablesize 3: hcurhash, curlocki ← hasharray[index] 4: if curlock = 1 then return NOTADDED 5: if curhash = hash then 6: if data matches data in data array then return NOTADDED 7: end if 8: if not compare and swap(hasharray[index], hcurhash, 0i, hhash, 1i) then 9: return NOTADDED 10: end if 11: write data to data array 12: hasharray[index] ← hhash, 0i {release lock} 13: return ADDED

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 49 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Unique table = lockless hashtable + garbage collection

I Use one bit of hash as lock, and two bytes as reference count I Buckets in the hash table are in one of these states: 1 EMPTY: Unused bucket, also end-of-search 2 TOMBSTONE: Unused bucket, but keep searching 3 WAIT(hash): Bucket being written 4 DONE(hash, count): Filled bucket

cas write data EMPTY WAIT(h) DONE(h,count)

cas delete +, − TOMBSTONE

I Rules to enforce correctness: 1 Inserting and deleting mutually exclusive (separate gc mode) 2 Transitions to WAIT use compare and swap

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 50 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... lookup or insert algorithm

First part of the algorithm: loop over the probe sequence to ﬁnd existing data.

Input: data 1: hash ← calculate hash(data) 2: for i ∈ probe sequence(data) do 3: if bucket[i] = EMPTY then break 4: if bucket[i] = WAIT(hash) or bucket[i] = DONE(hash, ∗) then 5: while bucket[i] = WAIT(hash) do nothing 6: if data matches data in data array then 7: increase(bucket[i]) 8: return i 9: end if 10: end if 11: end for

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 51 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... lookup or insert algorithm

Second part of the algorithm: insert the data!

9: for i ∈ probe sequence(data) do 10: value ← bucket[i] 11: if value = EMPTY or value = TOMBSTONE then 12: if compare and swap(bucket[i], value, WAIT(hash)) then 13: write data to data array at i 14: bucket[i] ← DONE(hash, 1) {release lock and set count to 1} 15: return i 16: end if 17: end if 18: if bucket[i] = WAIT(hash) or bucket[i] = DONE(hash, ∗) then 19: while bucket[i] = WAIT(hash) do nothing 20: if data matches data in data array then 21: increase(bucket[i]) 22: return i 23: end if 24: end if 25: end for

26: return FULL UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 52 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 53 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Parallelizing the BDD Operations

Parallel BDD operations

I Organize recursive calls to RelProd and Or in a task dependency graph

I Same task might be created several times: store result in the shared Computed Table I Fine-grained task-parallelism:

I Parent spawns children for subtasks, and waits upon their completion I Load balancing by work-stealing; use e.g. Cilk [Blumofe ’95] or Wool [Fax´en’08]

Split double-ended queue in public and private part t s h • • • • • • • • • • stolen stealable worker-private

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 54 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 55 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Recent extensions

I Parallelize Symbolic Reachability itself: more parallel work

I Parallel learning for all transitions I Parallel enumeration of states I Parallel RelProd for all transitions I Map-reduce type of union of results

I Redesign of Unique Table, mark-and-sweep garbage collection

I Implement parallel MDDs by List Decision Diagrams

x0 : 0 1 3 5 6

x1 : 0 2 4 0 1

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 56 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Table of Contents

1 Binary Decision Diagrams - storing vectors of sets concisely Deﬁnition Implementation

2 Symbolic Reachability Next-state by Relational Product Partitioning of next-state

3 Symbolic Model Checking in LTSmin PINS interface and Local Transition Caching Multi-valued Decision Diagrams

4 Sylvan: Multi-core BDDs Data Structures Work Stealing Parallelism at a higher level Experiments

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 57 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Results: speedup of BDD operations for model checking

Model 30 bakery.4 bakery.8 collision.5 Experiments 25 iprotocol.7 lifts.4 lifts.7 20 schedule world.2 I BEEM benchmarks, again schedule world.3

Speedup I On 4 × 12 = 48 core NUMA

10 I Speedup up to 32 (=66.7%)

5 I Small models don’t scale (time spent in work stealing)

0 10 20 30 40 Workers

Conclusion

I So far only speed up for the BDD-operations

I Even for large models, many small BDDs are involved

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 58 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Parallel learning/Reachability vs Parallel BDDs only

40 40 36 36 32 32 28 28 24 24 20 20 16 16 12 12 8 8 4 4 0 0 16 128 1024 16 128 1024 Speedup with 48Time workers with 1 worker (seconds) Speedup with 48Time workers with 1 worker (seconds) High-level parallelism Parallel BDDs only

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 59 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Comparing multicore BDDs with multicore MDDs

1024 32 28 128 24 20 (seconds) 16 16 48 12 T 8 2 4 LDD Speedup (48) LDD 0 2 16 128 1024 0 4 8 12 16 20 24 28 32 BDD T48 (seconds) BDD Speedup (48)

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 60 / 61

... Binary Decision Diagrams - storing vectors of sets concisely Symbolic Reachability LTSmin Sylvan: Multi-core BDDs ... Literature on LTSmin (symbolic reachability)

LTSmin toolset: symbolic reachability

I http://fmt.cs.utwente.nl/tools/ltsmin/ I Stefan Blom, Jaco van de Pol, ...... (ICTAC 2008) Symbolic reachability for process algebras with recursive data types I Stefan Blom, Jaco van de Pol, Michael Weber, LTSmin: Distributed and Symbolic Reachability ...... (CAV 2010) I Jeroen Meijer, Gijs Kant, Stefan Blom, Jaco van de Pol . . . . (HVC 2014) Read, Write and Copy Dependencies for Symbolic Model Checking

Multi-core BDDs

I Tom van Dijk, Alfons Laarman and Jaco van de Pol, Multi-core BDD Operations for Symbolic Reachability . . . . (PDMC 2012) I Tom van Dijk, Jaco van de Pol, ...... (MuCoCoS 2014) Lace: non-blocking split deque for work-stealing

UNIVERSITY OF TWENTE. Multi-core Model Checking 30, 31 October 2014 61 / 61