Consistency and Completeness of Temporal Logic Specifications

1 Outline

• Introduction

• Consistency of Temporal Specifications

• Completeness of Temporal Specifications

2 Issues in Property Verification

• It is hard to specify all properties to be verified – Users typically verify a few critical properties formally and also run simulation tests

• It is difficult to verify all properties specified – State Explosion – FPV works well at the block level (< 10K gates)

3 How much have I specified ?

• Coverage metrics quantify the coverage of a property specification

• Coverage metrics may be – With respect to a implementation – Independent of an implementation

4 How much can I verify ?

• Existing formal methods do not scale beyond RTL modules of moderate size

• For systems consisting of multiple modules, state space explosion

• A systematic methodology for modular validation of large designs is indispensable

5 Specification Issues for Open Systems

• Temporal logic specifications which are satisfiable at the system level may be inconsistent when interpreted over an open system

• Consistency Issues – Realizability – Sanity

6 Outline

• Introduction • Consistency of Temporal Specifications – Closed systems and Open systems – Consistency problem in Closed systems • Satisfiability – Consistency problem in Open systems • Realizability • Sanity • Completeness of Temporal Specifications

7 Closed Systems and Open Systems

• Closed system: – its behavior is completely determined by the state of the system

• Open system: – interacts with its environment – its behavior depends on this interaction

8 Example of a Closed System

• Think of a drink dispensing machine which – repeatedly boils water – makes an internal nondeterministic choice – … and serves either coffee or tea

The environment can not modify any of the system variables

9 Example of an Open System

• Think of another kind of drink dispensing machine which – repeatedly boils water – asks the environment to choose between coffee and tea – deterministically serves a drink according to the external choice

The environment can modify some of the system variables

10 Consistency Problem in Closed Systems

• Only consistency problem that arises in closed system is satisfiability – a formula  is said to be satisfiable iff • there exists a closed system K which satisfies 

• Formally – Let the formula be (z) where z is the set of system variables – then (z) is satisfiable iff z (z) is true

11 Example

• Let us consider the following LTL formula –G p  F(p)

• there exists no assignment of p along a path which satisfies this formula

• Again consider the following CTL formula –EG(q)  AF(p U q)

• there exists no assignment of q along the computation tree branches from a state which satisfies this formula

12 Complex Example

• Two or more satisfiable formulas may together result in an unsatisfiable specification

• Consider the following specification – AG(ack  done) – A(req U ack) –EG(done)

It’s a nontrivial task to find whether a specification is satisfiable

13 Complexity

• CTL model-checking is linear time solvable

• Whereas CTL satisfiability checking is EXPTIME-complete

• Both LTL model-checking and satisfiability checking is PSPACE-complete

14 Consistency Problem in Open Systems

• An open system has to be correct with respect to any environment

• Here comes the issue of implementability, commonly known as realizability [Pnueli’89]

– a formula  is said to be realizable iff

• there exists a module M which satisfies  under any environment • also M is not clairvoyant

15 Realizability is not same as Satisfiability

• Satisfiable, but unrealizable r1 g1 G( r1  ( X(g1)  X X( g1) ) ) r2 g2 • This property is satisfiable, -- consider input sequences where r1 is never asserted • It is not realizable for input sequences where r1 is asserted for two consecutive cycles • Is realizability = satisfiability over all inputs? – Not quite. Consider the property: G(g1  X r1) • The property is satisfiable for all given input sequences • The problem is in the inability to foresee the future inputs

16 Nontrivial Example

• Consider the specification of an arbiter that receives requests from two masters

( req0 and req1 are the inputs and ack0 and ack1 are the outputs )

• G(req0  X ack0)

• G(req1  X ack1)

•G(ack0  ack1)

If both req0 and req1 are asserted, there is

no valid next state assignment for ack0 and ack1

17 Realizability: Formal Characterization

G( req  (X ( gnt )  X X( ¬gnt )) full x tree

gnt = x1 0 gnt = x

gnt = 1 1 0 gnt = 11 gnt = x0 gnt = x

1 0 1 0 1 0 1 0 gnt =. ? gnt =. ? gnt =. 0 gnt .= 0 gnt. = 1 gnt. = 1gnt. = x gnt. = x ...... 18 Realizability: Formal Definition

• A formula (x, y) where x is the set of input variables and y the set of output variables is realizable iff

– there exists a valuation of all the variables in y at each node of the full x tree • such that  is not refuted in any path of the tree

19 Realizability is not Enough: Sanity

Now, the inputs and outputs are busreq reversed Master Bus Device gnt Arbiter

Output depends on future G( busreq  X X gnt ) input !! Not a Sane formula for the master device

20 Formal Characterization of Sanity

G( gnt  X X req ) full x tree

gnt = 0 gnt = 0 1 0

gnt = 0 1 0 gnt = 01 gnt = 00 gnt = 0

1 0 1 0 1 0 1 0 gnt .= 0 gnt =. 0 gnt =. 0 gnt .= 0 gnt. = 0 gnt. = 0gnt. = 0 gnt. = 0 ...... 21 Formal Characterization of Sanity

G( gnt  X X req ) full x tree

gnt = 1 gnt = 0 1 0

gnt = 0 1 0 gnt = 01 gnt = 00 gnt = 0

1 0 1 0 1 0 1 0 gnt .= 0 gnt =. 0 gnt =. 0 gnt .= 0 gnt. = 0 gnt. = 0gnt. = 0 gnt. = 0 ...... 22 Sanity: Formal Definition

• A formula (x, y) where x is the set of input variables and y the set of output variables is not sane iff – There exists a valuation of the variables in y at a particular node of the full x tree • which does not refute the formula at that point • but makes the formula unrealizable – But, some other valuation of y would have made the formula realizable

23 Outline

• Introduction • Consistency of Temporal Specifications • Completeness of Temporal Specifications – With respect to the implementation – Independent of the implementation – By simulating the implementation

24 Completeness of Specifications: Motivation

Fix the model/specification

NO p p p specification YES  Did I check Have I written everything enough that I meant Properties? to check?

25 Coverage of Formal Specifications

• Coverage metrics – heuristic measures of comprehensiveness of a given test • Traditional coverage metrics for simulation are – code coverage, transition coverage, etc • Applying them blindly on model checking – gives a meaningless 100% coverage for every property • because model checking searches the implementation exhaustively

26 Coverage: Various Approaches

•With respect to the implementation – introduce a small mutation in the system – check whether the mutant system is still correct with respect to the specification – if yes – this mutation is not covered • Independent of the implementation – check the completeness with respect to a high-level fault model •By simulating the implementation – inject some fault on the system – simulate the fault-injected system to see whether some property fails

27 Outline

28 Coverage Metric for Model Checking

• Hoskote et al.’99 proposed a coverage metric for model checking – to identify the part of the state space which is critical for the truth of the specification

• Basic idea – flip an atomic proposition at a state of the model – see whether any property fails in the perturbed model (by model checking) – if some property fails • the state is covered with respect to the atomic proposition

29 Example

Flip p at S1 What is the coverage of p E(p U q) on this model? S1 p, q, r Is S1 covered S3 S2 with respect to p? p, q, r p, q, r Now E(p U q) fails!!

S4 p, q, r S1 is covered with respect to p

30 Example

What is the coverage of E(p U q) on this model? S1 p, q, r Is S4 covered S3 S2 with respect to p? p, q, r p, q, r Still E(p U q) passes!! p Flip p at S4 S4 p, q, r S4 is not covered with respect to p

31 Example

What is the coverage of E(p U q) on this model? S1 p, q, r Is S4 covered S3 S2 with respect to q? p, q, r p, q, r Now E(p U q) fails!! q Flip q at S4 S4 p, q, r S4 is covered with respect to q

32 What Does this Coverage Metric Mean?

• It can uncover functionality in the model not covered by any property – but can not point out functionality missing in the model • So, 100% coverage – does not guarantee completeness of the verification – nor correctness of the model • But, a low coverage – definitely implies incompleteness of properties

33 Further Improvements

• Chockler et al.’01 introduced the distinction between – state based and logic based coverage – coverage with respect to inputs and outputs

• Chockler et al.’03 – introduced falsity coverage and vacuity coverage – adapted coverage metrics for simulation to the formal verification setting

34 Outline

35 Use of a High-Level Fault Model

• Existing FPV coverage metrics compare the specification with an implementation • At the highest level we do not (yet) have the implementation • How to check whether we have written enough properties? – Check coverage against a high-level fault model • The fault model could be a standard fault model such as a stuck-at fault model

36 Fault Coverage on Output Lines

• Allows us to determine whether for every output line there exists a property in the specification that requires that output to take a specific value (0/1) under some input scenario.

• Otherwise a valid implementation exists that can forever hold that output at that specific value

37 Fault Coverage on Input Lines

• Allows us to determine whether for every input line there exists an input scenario with the input taking a specific value (0/1) that triggers one or more properties.

• Otherwise a valid implementation may ignore that input line altogether

38 Example: Output Fault Coverage

Have I written • Memory arbiter: enough properties mem-arbiter(input r1, r2 ; output g1, g2)

Priority of g1: G ((r1  r2)  X g1) Mutual exclusion: G (g1 g2)

• g1 s-a-0 is directly covered (input: r1=r2=1) • g2 s-a-1 is indirectly covered – it implies g1 s-a-0, which is directly covered • g2 s-a-0 is not covered – we can have a valid implementation that always deasserts g2 !! -- add a property for the scenario where g2 has to be high • G((r1  r2)  X g2) • now, g2 s-a-0 directly and g1 s-a-1 indirectly covered

39 Output Fault Coverage: Definition

• Let T be a given realizable property and let z be an output signal. Then T covers the fault z s-a-0, iff T  G (z) is unrealizable • T covers the fault z s-a-1, iff T  G (z) is unrealizable

• Challenge: Realizability checking is harder than satisfiability checking

40 Example: Input Fault Coverage

Have I captured • Memory arbiter: enough scenarios mem-arbiter(input r1, r2 ; output g1, g2)

Priority of g1: always (r1 & r2)  next g1 No starvation: always(!r1 & r2)  next g2 Mutual exclusion: always (!g1 | !g2)

• r1 s-a-0 and s-a-1 are covered • r2 s-a-0 is not covered – Nothing is said when r2 is low -- add a property for some scenario where r2 is low • always(!r2  next !g2) • r2 s-a-1 is covered.

41 Input Fault Coverage: Definition

0 •Tv: Specification T with v replaced by 0

1 •Tv: Specification T with v replaced by 1

• A s-a-0 fault on an input signal v is covered by a 0 realizable specification T, unless T v  T

• Likewise a s-a-1 fault on an input signal v is 1 covered by T, unless T v  T

42 Outline