Semi-Formal Reformulation of Requirements for Formal Property Verification

Semi-formal Reformulation of Requirements for Formal Property Verification

Katharina Ceesay-Seitz, Verification Engineer, CERN, Geneva, Switzerland ([email protected]) Hamza Boukabache, Project Manager, CERN, Geneva, Switzerland ([email protected]) Daniel Perrin, Section Leader, CERN, Geneva, Switzerland ([email protected])

Abstract— Ambiguously specified requirements can be a source of risk for safety-critical electronic designs. Requirement specifications in natural language are subject to misinterpretation. A method is proposed that reduces the risk of misinterpretations. Requirements are reformulated into semi-formal properties, which we call Natural Language Properties (NLPs). These statements are composed of natural language patterns which are then translated into SystemVerilog for formal verification. This reformulation is done by an independent verification engineer and then reviewed by the requirements engineer. Applying this technique for the verification of the CERN RadiatiOn Monitoring Electronics (CROME) led to the discovery of a safety-critical fault.

Keywords—semi-formal methods; formal property verification; SystemVerilog assertions; functional safety; independence

I. INTRODUCTION Project teams in international organizations and companies consist of many people of different backgrounds. They often have different native languages and have previously gained work experience in different countries and cultures. They also have been educated in different universities, which may use different notations and terms to describe the same topic. They may even use different unities to describe physical properties. Even persons with the same mother tongue, trained at the same university in the same subject and working in the same company may interpret the same sentence in different ways. A prominent example for a failure related to intercultural project teams is NASA’s Mars Climate Orbiter. The spacecraft was lost when entering Mars’ atmosphere. The cause of this loss was that the spacecraft’s on-board software used different measurement units than the ground- segment software. The former performed calculations in metric units whereas the latter used English units [1]. Such mismatches can have life threatening consequences. Requirements specifications are often written from the end user’s perspective, which means at system level. When the development happens on Register Transfer Level (RTL), e.g. in VHDL code, a much more detailed specification is needed. Safety standards like IEC 61508 (standard for functional safety of electrical/electronic/programmable electronic systems), ISO 26262 (standard for automotive) or DO-254 (standard for airborne electronics) all specify similar safety life cycles. Typically, these follow the V-model. Validation according to the V-model of IEC 61508 should be based on the functional and safety-requirements [10]. The verification against the design specification does not automatically mean that the system is validated against its requirements specification, even though the design specification is back-traceable to the requirements. The specification is often a refinement of the requirements. This step is usually done by the design engineer, who in many cases is a different person than the requirements engineer. At this point a misinterpretation with possibly fatal consequences can easily happen. A misinterpretation would turn into a legal specification statement against which verification will be performed. Ultimately this means that the implementation is verified to comply to a specification statement that does not reflect the original requirement as it was intended. The proposed technique of using Natural Language Properties (NLPs) aims at discovering such cases. Figure 1 shows where the technique can be located in the V-model. It could be applied directly after writing the FPGA/Software Safety Requirements Specification, before the system is designed and implemented. This has the advantage that the requirements are validated before any further work is based on them. Another option is to write NLPs when writing the verification

plan. The latter has the advantage that the design specification is already available. Knowledge of implementation details will often be necessary when writing formal properties. The obvious drawback of validation at a later stage is that its results will lead to modifications of the already implemented system. Ideally a verification engineer would perform the requirements validation with NLPs after the specification of the FPGA/Software Safety Requirements.

System Safety System Safety Validation Requirements Validation

Verification Programmable System Architecture Electronics Integration Specification Testing

FPGA/Software Safety Validation FPGA/Software System Requirements Validation

NLP

FPGA/Software System Verification Module Integration Design Testing

NLP

Module Design Verification Module/Unit Testing Specification Time Flow, Trace

Code Implementation 2

Figure, 1 V-Model with NLPs The usage of NLPs will be demonstrated on a part of the CERN RadiatiOn Monitoring Electronics (CROME) [2]. CERN, the European Organization for Nuclear Research, operates physics experiments such as the Large Hadron Collider (LHC). The LHC and other facilities accelerate beams of particles nearly up to the speed of light and collide them in order to study the fundamental laws of our universe. These experiments produce ionizing radiation. CERN’s personnel and the general public need to be protected from any unjustified exposure to ionizing radiation. The CROME radiation monitoring system has been newly developed for this purpose. It will replace the old existing systems. It consists of hundreds of autonomous units that are measuring radiation levels. These units are responsible for the automatic detection of dangerous radiation levels based on many parameters, which can be configured in a supervision software and transmitted to the monitors over a network connection. If the calculated ambient equivalent dose rate exceeds the configured limits, the CROME Measurement and Processing Units (CMPUs) notify the alarm units and trigger machine stops. The CMPUs implement several signal processing algorithms. One of them is called “integration”. This algorithm calculates the ambient equivalent dose received by an area. If this dose exceeds a configurable threshold, an alarm/interlock signal is set. Several remote commands can be sent, which can e.g. restart the integration calculation or deactivate the alarm function. The outcomes of 7 conditions at two consecutive measurement times influence the decision whether an alarm should be activated. This leads to 214 possibilities.

II. RELATED WORK Property Specification Patterns (SPS) were introduced in [3]. These are recurring patterns in requirement specifications. A pattern consists of a name, an intent and a common specification formalism. The latter could for example be a Linear Temporal Logic (LTL) property. In [4] this approach was assessed for a requirements

specification in the automotive domain. They concluded that 70% of the requirements could be mapped to such patterns. SPS could be well combined with the suggested Natural Language Properties. The authors of [5] proposed to use a subset of the English language for the specification of requirements in order to get a clear mapping from English to a formal language. A tool for the translation of such specifications into formal properties was presented in [6]. A limitation is that generated properties cannot span multiple clock cycles. Only 44% of the English language assertions could be automatically translated into SystemVerilog assertions, which means that this approach, though promising, is not yet ready to be applied in real projects. Another tool for the automated translation of English language into formal properties has been presented in [7]. This tool relies on the similarity of sentences in low-level specifications. Sentences like “Signal A remains high for 5 clock cycles” can be automatically translated. High-level statements still need to be translated manually. Another tool for automated translation of natural language comments in Hardware Description Language (HDL) code has been presented in [8]. While the translation works very well for the tested benchmark, the authors highlight a general limitation of automated approaches that use natural language processing. English language constructs like “must be” can refer to the present or the future. A tool needs to be able to correctly translate such statements. From these references it can be seen that the automated translation from requirements to formal properties is not trivial. There are currently no safety qualified tools available to perform this step. Therefore, the manual formulation used in our workflow does not add much overhead. Furthermore, it builds on another important concept in the development and verification of safety-critical systems: independence. Several safety standards require independent reviews and verification. The study presented in [9] also confirmed that the number of discovered faults by an independent verification team is higher than when the designers verify their own product.

III. APPLICATION A Natural Language Property (NLP) is a natural language version of a formal SystemVerilog property. Table I provides a one to one mapping from natural language constructs to SystemVerilog. The goal of using NLPs is to facilitate communication between engineers with different backgrounds and training in order to establish a consistent understanding of the system requirements. A central element of the workflow proposed below is the review of the NLPs by a requirements engineer, who might not be familiar with SystemVerilog syntax. Requirements are typically written at a higher level of abstraction than formal properties. NLPs are at the same level of abstraction as SystemVerilog Assertions (SVA). Reviewing the NLPs allows the requirements engineer to validate the refinement of the requirements without the need of learning a new language (SVA). He or she can focus on the content of the property without being distracted by interpreting the syntax of SystemVerilog. The presented method targets teams where the requirements engineer does not have the time to learn and get accustomed to SVA syntax. It might take a week or more to learn it by heart so that it comes to mind easily and it needs continuous practice. By using NLPs the requirements engineer can focus exclusively on the review. In the CROME project the review and therefore validation of roughly 30 NLPs took no more than 1 hour. Formal properties are written in a declarative style. A property must consider all parameters or signals that could possibly influence whether the property holds or not. This means that it does not allow implicit assumptions. For any signal that is not stated, a formal proof calculation considers every possible value. Therefore any signal that might change the outcome of the formal proof intent of a property needs to be specified. Since a NLP is a formal property stated in natural language, this advantage is kept. The use of fixed natural language patterns that can be directly mapped to SystemVerilog enforces unambiguous statements. The requirements engineer needs to be aware that a proven property is true for all possible values of any parameter which is not mentioned in the property declaration. Having this in mind the requirements engineer will be forced to mention implicit assumptions like a specific value of a certain parameter during the review.

The suggested workflow for using NLPs is as follows. Figure 2 depicts the workflow. 1. Requirements engineers specify the system’s requirements. 2. Verification engineers formulate one or more Natural Language Properties (NLPs) for each functional and safety requirement. Each of them can be traced back to at least one requirement. 3. The requirements engineers review the NLPs and confirm them. 4. The verification engineers translate the NLPs into SystemVerilog assertions and perform the formal verification.

Requirements Verification Engineer Engineer

Requirements Specification

Review of Requirements & Natural Language Property Specification

Review of NLPs Time

Review Meeting

Translation of NLPs into SystemVerilog

Figure 2, Workflow

The method seamlessly fits into the V-model as shown in Figure 1. It adds a layer of assurance and creates backward traceability from verification results to requirements as demanded by most safety standards. Even though the IEC 61508 only mentions traceability in its software part, it is a good practice to apply it for the HDL design of electronic designs as well. This method could also be used to complement or replace the Hardware Design Description (HDD) according to DO-254. The HDD is required to contain a textual specification at signal level. Table I maps some SystemVerilog statements to multiple natural language patterns. This allows a more flexible use of the English language. It can be further complemented by an application specific table. This allows an even broader use of natural language constructs while still maintaining a direct mapping to SystemVerilog. Table II provides an application specific mapping for the CROME Measuring and Processing Unit (CMPU). An example is provided afterwards.

Table I. General Mapping of Natural Language Constructs to SystemVerilog Constructs

Natural Language SystemVerilog Explanation Expr1 equals Expr2 Expr1 == Expr2 An Expression (Expr1) equals another Expression (Expr2). Expr1 does not equal Expr2 Expr1 != Expr2 An Expression (Expr1) does not equal another Expression (Expr2). Expr1 equals or is greater than Expr2 Expr1 >= Expr2 An Expression (Expr1) equals or is greater than another Expression (Expr2). Expr1 equals or is less than Expr2 Expr1 <= Expr2 An Expression (Expr1) equals or is less than another Expression (Expr2). Expr1 and Expr2 Expr1 && Expr2 Logical and of two expressions (Expr1,Expr2). Expr1 or Expr2 Expr || Expr Logical or of two expressions (Expr1, Expr2). (Expr) (Expr) The parenthesis emphasize that an expression (Expr) consisting of several parts belongs together. Sig does not change $stable(Sig) Signal (Sig) is stable, so it does not change its value. Prop Prop |-> Seq Property (Prop) implies that in the implies that: Seq same clock cycle the sequence (Seq) happens. "implies that:" is placed in a new line to indicate that the whole statement above implies the one that follows. Every time when Expr: Expr |-> Seq Another way of stating an implication. Seq At Expr: Seq Expr |-> Seq Another way of stating an implication.

Prop Prop |-> ##TimeSeq Seq Property (Prop) implies that the sequence implies that: in TimeSeq, Seq (Seq) happens after some cycles or condition indicated by TimeSeq1. TimeSeq after Expr: Seq Expr ##TimeSeq Seq Expression (Expr) is followed by sequence (Seq) after some cycles or condition indicated by TimeSeq1. in TimeSeq Seq ##TimeSeq Seq After some cycles or condition indicated by TimeSeq1 the sequence (Seq) happens. Prop1 until Prop2 Prop1 s_until Prop2 Property (Prop1) holds until another Property (Prop2) holds. They don’t hold at the same time. Prop1 as long as not Prop2 Prop1 until Prop2 Property (Prop1) holds until another Property (Prop2) holds. The whole statement is also true if Prop2 never holds. (“ ”) A comment added for review that describes e.g. the content of testbench internal signals. 1TimeSeq could be a number or range alone or it could also include a sequence or condition that should happen before sequence (Seq) happens. E.g. TimeSeq = ##[0:2] stating that Seq should happen 0 to 2 clock cycles after Expr E.g. TimeSeq = ##1 input1xDI == 0 ##[2:$] $rose(input1xDI) ##1 stating that after Expr one clock cycle later input1xDI should be 0, then 2 to inﬁnite clock cycles later input1xDI should have risen from 0 to 1 and 1 clock cycle later Seq should happen.

Table II. Application Specific Mapping of Natural Language Constructs to SystemVerilog Constructs

Natural Language SystemVerilog Explanation Cycle is a MC mcValidxDI == 1 This clock cycle is within a Measurement Cycle (MC). Cycle is no MC mcValidxDI == 0 This clock cycle is not within a Measurement Cycle (MC). Cycle is the start of a MC ($rose(mtValidxDI) This clock cycle is the start of a Measurement Cycle, which can last several ms to s. The time passed since integration start == etCount etCount is a signal of the formal verification suite. It is increased in each device cycle. At the previous MC signalNameLastMC1 Take a signal value that has been stored in the formal module at the last MC. At integration end time elapsedTimexDO >= integralTimexDI When integration time has been reached or exceeded. At the next MC ##[1:$] (mcValidxDI==0) The MC indication signal is low at ##[1:$] (mcValidxDI==1) some point, then it becomes high again. Once the calculation is ready ##`nrCUntilCalcRdy The number of clock cycles until a new integral value is calculated is left to the implementation, as long as it is shorter than 1 device cycle (several ms). Alarm is on alarmxDO == 1 Alarm is turned on.

Alarm is off alarmxDO == 0 Alarm is turned off

Alarm is/was configured as latched alarmLatchedxDI == 1 The alarm signal is/was configured as latched.2 Alarm is/was configured as free running alarmLatchedxDI == 0 The alarm signal is/was configured as free running.2 Alarm function is/was deactivated alarmActivexDI == 0 The alarm function is/was deactivated.2

Alarm function is/was activated alarmActivexDI == 1 The alarm function is/was activated.2

Integral value signed’(integralxDO) The radiation integral.

Threshold signed'(thresholdxDI) The radiation threshold.

The testbench internal integral value integralTB The testbench models the integration algorithm in an always block. 1Store the value of a signal at a Measurement Cycle (MC) in a test module internal variable and access it at the next MC. 2If the statement refers to the past, then a previously stored value has to be used instead of the actual signal, eg.alarmLatchedLastMC.

IV. EXAMPLES The following examples show requirements and their translation into NLPs and SystemVerilog assertions. The general translations of Table I are applied for fragments like “implies that: ”. The translations from Table II are applied for application specific terms like “alarm is off” and for mapping of English words to signal names.

A. Example I The following example uses a testbench internal signal for comparison. This signal is calculated in an always block in the testbench, which models the required integration algorithm. It has been independently developed from the VHDL design under verification. Requirement: “A manual zeroing of the integrated value shall be possible.” Natural language property:

Cycle is the start of a MC and integral reset at previous MC equaled 1 implies that: once the calculation is ready, integral value equals the testbench internal integral value (“calculated since integration start”)

SystemVerilog property:

property pIntManualResetNextMT(); ($rose(mtValidxDI) && intResLastMC == 1) |-> ##`nrCUntilCalcRdy integralxDO == integralTB; endproperty

This property uses many application specific terms. The addition ‘(“calculated since integration start”)’ is not reflected in the property. It is a description of how the testbench internal integral value is calculated. It is added to improve the review.

B. Example II

Requirement: "It shall be possible to manually trigger a reset of an integration alarm through the supervision software." There are several formal properties necessary to fully proof this requirement. Natural language property - 1:

"(Cycle is no MC and (alarm was configured as latched at the previous MC) and alarm is on and alarm reset equals 1 and (integral value is less than (threshold at previous MC) or alarm function was deactivated at previous MC)) implies that: (in one clock cycle, alarm is oﬀ)"

SystemVerilog property:

property pIntAlarmResetBetweenMT1(); (mcValidxDI == 0 && latchedLastMC == 1 && ALARMxDO == 1 && integralAlarmResetxDI == 1 && (signed’(integralxDO) < signed’(thresholdLastMc) || alarmActiveLastMc == 0)) |-> ##1 (ALARMxDO == 0); endproperty

It is important to note that the alarm had to be configured as latched at the previous Measurement Cycle (MC).

Natural language property - 2:

"(Cycle is no MC and alarm is on and alarm reset equals 1 and (integral value is greater than or equal to the (threshold at previous MC) and alarm function was activated at previous MC)) implies that: (in one clock cycle, alarm is on)"

SystemVerilog property:

property pIntAlarmResetBetweenMT2(); (mcValidxDI == 0 && ALARMxDO == 1 && integralAlarmResetxDI == 1 && (signed’(integralxDO) >= signed’(thresholdLastMC) && alarmActiveLastMc == 1)) |-> ##1 (ALARMxDO == 1); endproperty

Here it is not relevant whether the alarm was conﬁgured as latched or free running, because in both cases we expect the reset to have no eﬀect. These assertions have been proven on the CERN RadiatiOn Monitoring Electronics (CROME) [2].

V. RESULTS The technique of specifying Natural Language Properties (NLPs) has been used for the verification of a subsystem of the CERN RadiatiOn Monitoring Electronics (CROME) [2]. When the NLPs for CROME were reviewed by the requirements engineer, some minor inconsistencies and one severe safety-critical fault could be discovered. The requirement presented in the example II in section IV.B stated that a radiation alarm can be reset through the supervision software. This requirement, nor any other, mentioned a relationship of the reset signal to the alarm configuration. For the requirements engineer it was clear that a free running alarm cannot be influenced by a human operator. The VHDL design engineer as well as the verification engineer both did not understand it that way. The Natural Language Property – 1 presented in the example was therefore initially declared without this phrase:

“(alarm was configured as latched at the previous MC)” This meant that the property would have allowed a reset of a free running alarm. It would have been actually possible to proof this wrong property, because the design engineer had implemented the HDL code in the same way as the verification engineer expected. The requirements engineer though intended to specify that only alarms in latched mode can be reset. This misinterpretation could have had severe consequences on the safety of people working at CERN. The original design would have allowed a reset by a human operator in the CERN control center in a situation in which the requirements engineer, who in this case was the radiation expert, expected the alarm to be kept on. Without the review of the NLPs, this design would have been formally proven against the verification requirements, because the latter did not mention the latched configuration either. The corrected NLP therefore had to take the latest alarm mode setting into account, which is described by the phrase shown here. This example clearly demonstrates the capabilities of the proposed method and justifies the addition of one review step into the V-model. In some development projects this review step would take place anyway. According to IEC 61508 all development phases, including the requirements specification phases, need verification, which can be conducted in the form of reviews. In that case the specification and review of NLPs nearly adds no overhead.

VI. CONCLUSION Natural Language Properties (NLPs) and a workflow have been introduced. The main goal is to avoid misunderstandings between engineers of different backgrounds and with different views of the system under development. The technique can speed up the development lifecycle by reducing the number of needed iterations due to misunderstandings. It further reduces the risk of undiscovered faults.

Each functional and safety requirement is reflected by one or more NLPs, which are then used for formal verification. The suggested technique seamlessly fits into the V-model, which is commonly used for the development of safety-critical electronic systems. It adds two additional steps: 1. Formulation of NLPs 2. Review of the NLPs by the requirements engineer The first step is the specification of formal properties in natural language, which are later translated into SystemVerilog. It is part of the verification. The second step targets at establishing consistency between the verification and the functional and safety requirements specifications. It is therefore part of the functional safety validation. The requirements engineer validates the formal properties and therefore the verification results. Most safety standards require backward traceability from verification results to requirements. NLPs build a connection between these. The proposed technique and workflow have been successfully used for the partial verification of the CERN RadiatiOn Monitoring Electronics (CROME) [2]. It has been shown that this technique can help to discover safety-critical faults, which would not be detected by automated verification methods. There is potential for automation of this technique, especially the translation from natural language constructs to SystemVerilog. For the automated extraction of properties from a natural language specification a qualified and accurate tool would be required. Such a translation would remove the need for a validation of the formal properties. As long as there is no tool available, an independent review by the requirements engineer should be conducted. The proposed method could also be combined with Property Specification Patterns as introduced in [3], which might have potential for further automation.

REFERENCES [1] Mars Climate Orbiter Mishap Investigation Board, “Phase I Report,” Nasa, USA, November 1999. [2] H. Boukabache, M. Pangallo, G. Ducos, N. Cardines, A. Bellotta, C. Toner, D. Perrin, D. Forkel-Wirth, “Towards a novel modular architecture for CERN radiation monitoring,” Radiat. Prot. Dosim. 173 (2017), pp.240-244, 2017 [3] M. B. Dwyer, G. S. Avrunin, J. C. Corbett, “Patterns in Property Specifications for Finite-State Verification”, Proceedings of the 21st international conference on Software engineering, California, USA, pp.411-420, 1999. [4] P. Filipovikj, M. Nyberg, G. Rodriguez-Navas, “Reassesing the Pattern-Based Approach for Formalizing Requirements in the Automotive Domain,” 2014 IEEE 22nd International Requirements Engineering Conference (RE), Sweden, 2014. [5] A. Holt, E. Klein, “A semantically-derived subset of English for hardware verification,”, Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pp.451-456, USA, 1999. [6] I. G. Harris, “Capturing Assertions from Natural Language Descriptions,” Natural Language Analysis in Software Engineering (NaturaLiSE) 2013 1st International Workshop, pp. 17-24, May 2013. [7] M. Soeken, C. B. Harris, N. Abdessaied, I. G. Harris, R. Drechsler, “Automating the Translation of Assertions Using Natural Language Processing Techniques, ” Proceedings of the 2014 Forum on Specification and Design Languages (FDL), Munich, 2014. [8] C. B. Harris, I. G. Harris, “Generating Formal Hardware Verification Properties from Natural Language Documentation,”, Proceedings of the 2015 IEEE 9th International Conference on Semantic Computing (IEEE ICSC 2015), Anaheim, California, USA, February 2015 [9] J. D. Arthur, M. K. Groner, K. J. Hayhurst, C. M. Holloway, “Evaluating the effectiveness of independent verification and validation,” Computer 32, Nr. 10, pp 79-83, October 1999 [10] European Committee for Electrotechnical Standardization, “Functional safety of electrical/electronic/programmable electronic safety- related systems,” May 2010