Consistency and Completeness of Temporal Logic Specifications 1 Outline • Introduction • Consistency of Temporal Specifications • Completeness of Temporal Specifications 2 Issues in Property Verification • It is hard to specify all properties to be verified – Users typically verify a few critical properties formally and also run simulation tests • It is difficult to verify all properties specified – State Explosion – FPV works well at the block level (< 10K gates) 3 How much have I specified ? • Coverage metrics quantify the coverage of a property specification • Coverage metrics may be – With respect to a implementation – Independent of an implementation 4 How much can I verify ? • Existing formal methods do not scale beyond RTL modules of moderate size • For systems consisting of multiple modules, state space explosion • A systematic methodology for modular validation of large designs is indispensable 5 Specification Issues for Open Systems • Temporal logic specifications which are satisfiable at the system level may be inconsistent when interpreted over an open system • Consistency Issues – Realizability – Sanity 6 Outline • Introduction • Consistency of Temporal Specifications – Closed systems and Open systems – Consistency problem in Closed systems • Satisfiability – Consistency problem in Open systems • Realizability • Sanity • Completeness of Temporal Specifications 7 Closed Systems and Open Systems • Closed system: – its behavior is completely determined by the state of the system • Open system: – interacts with its environment – its behavior depends on this interaction 8 Example of a Closed System • Think of a drink dispensing machine which – repeatedly boils water – makes an internal nondeterministic choice – … and serves either coffee or tea The environment can not modify any of the system variables 9 Example of an Open System • Think of another kind of drink dispensing machine which – repeatedly boils water – asks the environment to choose between coffee and tea – deterministically serves a drink according to the external choice The environment can modify some of the system variables 10 Consistency Problem in Closed Systems • Only consistency problem that arises in closed system is satisfiability – a formula is said to be satisfiable iff • there exists a closed system K which satisfies • Formally – Let the formula be (z) where z is the set of system variables – then (z) is satisfiable iff z (z) is true 11 Example • Let us consider the following LTL formula –G p F(p) • there exists no assignment of p along a path which satisfies this formula • Again consider the following CTL formula –EG(q) AF(p U q) • there exists no assignment of q along the computation tree branches from a state which satisfies this formula 12 Complex Example • Two or more satisfiable formulas may together result in an unsatisfiable specification • Consider the following specification – AG(ack done) – A(req U ack) –EG(done) It’s a nontrivial task to find whether a specification is satisfiable 13 Complexity • CTL model-checking is linear time solvable • Whereas CTL satisfiability checking is EXPTIME-complete • Both LTL model-checking and satisfiability checking is PSPACE-complete 14 Consistency Problem in Open Systems • An open system has to be correct with respect to any environment • Here comes the issue of implementability, commonly known as realizability [Pnueli’89] – a formula is said to be realizable iff • there exists a module M which satisfies under any environment • also M is not clairvoyant 15 Realizability is not same as Satisfiability • Satisfiable, but unrealizable r1 g1 G( r1 ( X(g1) X X( g1) ) ) r2 g2 • This property is satisfiable, -- consider input sequences where r1 is never asserted • It is not realizable for input sequences where r1 is asserted for two consecutive cycles • Is realizability = satisfiability over all inputs? – Not quite. Consider the property: G(g1 X r1) • The property is satisfiable for all given input sequences • The problem is in the inability to foresee the future inputs 16 Nontrivial Example • Consider the specification of an arbiter that receives requests from two masters ( req0 and req1 are the inputs and ack0 and ack1 are the outputs ) • G(req0 X ack0) • G(req1 X ack1) •G(ack0 ack1) If both req0 and req1 are asserted, there is no valid next state assignment for ack0 and ack1 17 Realizability: Formal Characterization G( req (X ( gnt ) X X( ¬gnt )) full x tree gnt = x1 0 gnt = x gnt = 1 1 0 gnt = 11 gnt = x0 gnt = x 1 0 1 0 1 0 1 0 gnt =. ? gnt =. ? gnt =. 0 gnt .= 0 gnt. = 1 gnt. = 1gnt. = x gnt. = x . 18 Realizability: Formal Definition • A formula (x, y) where x is the set of input variables and y the set of output variables is realizable iff – there exists a valuation of all the variables in y at each node of the full x tree • such that is not refuted in any path of the tree 19 Realizability is not Enough: Sanity Now, the inputs and outputs are busreq reversed Master Bus Device gnt Arbiter Output depends on future G( busreq X X gnt ) input !! Not a Sane formula for the master device 20 Formal Characterization of Sanity G( gnt X X req ) full x tree gnt = 0 gnt = 0 1 0 gnt = 0 1 0 gnt = 01 gnt = 00 gnt = 0 1 0 1 0 1 0 1 0 gnt .= 0 gnt =. 0 gnt =. 0 gnt .= 0 gnt. = 0 gnt. = 0gnt. = 0 gnt. = 0 . 21 Formal Characterization of Sanity G( gnt X X req ) full x tree gnt = 1 gnt = 0 1 0 gnt = 0 1 0 gnt = 01 gnt = 00 gnt = 0 1 0 1 0 1 0 1 0 gnt .= 0 gnt =. 0 gnt =. 0 gnt .= 0 gnt. = 0 gnt. = 0gnt. = 0 gnt. = 0 . 22 Sanity: Formal Definition • A formula (x, y) where x is the set of input variables and y the set of output variables is not sane iff – There exists a valuation of the variables in y at a particular node of the full x tree • which does not refute the formula at that point • but makes the formula unrealizable – But, some other valuation of y would have made the formula realizable 23 Outline • Introduction • Consistency of Temporal Specifications • Completeness of Temporal Specifications – With respect to the implementation – Independent of the implementation – By simulating the implementation 24 Completeness of Specifications: Motivation Fix the model/specification NO p p p specification YES Did I check Have I written everything enough that I meant Properties? to check? 25 Coverage of Formal Specifications • Coverage metrics – heuristic measures of comprehensiveness of a given test • Traditional coverage metrics for simulation are – code coverage, transition coverage, etc • Applying them blindly on model checking – gives a meaningless 100% coverage for every property • because model checking searches the implementation exhaustively 26 Coverage: Various Approaches •With respect to the implementation – introduce a small mutation in the system – check whether the mutant system is still correct with respect to the specification – if yes – this mutation is not covered • Independent of the implementation – check the completeness with respect to a high-level fault model •By simulating the implementation – inject some fault on the system – simulate the fault-injected system to see whether some property fails 27 Outline • Introduction • Consistency of Temporal Specifications • Completeness of Temporal Specifications – With respect to the implementation – Independent of the implementation – By simulating the implementation 28 Coverage Metric for Model Checking • Hoskote et al.’99 proposed a coverage metric for model checking – to identify the part of the state space which is critical for the truth of the specification • Basic idea – flip an atomic proposition at a state of the model – see whether any property fails in the perturbed model (by model checking) – if some property fails • the state is covered with respect to the atomic proposition 29 Example Flip p at S1 What is the coverage of p E(p U q) on this model? S1 p, q, r Is S1 covered S3 S2 with respect to p? p, q, r p, q, r Now E(p U q) fails!! S4 p, q, r S1 is covered with respect to p 30 Example What is the coverage of E(p U q) on this model? S1 p, q, r Is S4 covered S3 S2 with respect to p? p, q, r p, q, r Still E(p U q) passes!! p Flip p at S4 S4 p, q, r S4 is not covered with respect to p 31 Example What is the coverage of E(p U q) on this model? S1 p, q, r Is S4 covered S3 S2 with respect to q? p, q, r p, q, r Now E(p U q) fails!! q Flip q at S4 S4 p, q, r S4 is covered with respect to q 32 What Does this Coverage Metric Mean? • It can uncover functionality in the model not covered by any property – but can not point out functionality missing in the model • So, 100% coverage – does not guarantee completeness of the verification – nor correctness of the model • But, a low coverage – definitely implies incompleteness of properties 33 Further Improvements • Chockler et al.’01 introduced the distinction between – state based and logic based coverage – coverage with respect to inputs and outputs • Chockler et al.’03 – introduced falsity coverage and vacuity coverage – adapted coverage metrics for simulation to the formal verification setting 34 Outline • Introduction • Consistency of Temporal Specifications • Completeness of Temporal Specifications – With respect to the implementation – Independent of the implementation – By simulating the implementation 35 Use of a High-Level Fault Model • Existing FPV coverage metrics compare the specification with an implementation • At the highest level we do not (yet) have the implementation • How to check whether we have written enough properties? – Check coverage against a high-level fault model • The fault model could be a standard fault model such as a stuck-at fault model 36 Fault Coverage on Output Lines • Allows us to determine whether for every output line there exists a property in the specification that requires that output to take a specific value (0/1) under some input scenario.