DEGREE PROJECT IN INFORMATION AND COMMUNICATION TECHNOLOGY, SECOND CYCLE, 30 CREDITS STOCKHOLM, SWEDEN 2020 Verification of Formal Requirements through Tracing JORRIT J. OLTHUIS KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE ABSTRACT I Abstract Software development in the railway application is governed by strict stan- dards which aim to ensure safety. It is for example highly recommended to use formal methods when specifying requirements. Moreover, it is mandatory to have certain roles be fulfilled by different people. A common technique is developing software tests for the requirements. Mak- ing sure that software requirements are properly described, interpreted and implemented by different people is a major challenge. Tests fully depend on the tester to cover all scenarios. Having more methods that simplify require- ment tracing and that depends less on thoroughness of the tester would provide many benefits. This thesis investigates whether and how software tracing can be used to vali- date formal requirements of software. The goal is to perform trace validation such that it can be used to complement more traditional verification techniques. By verifying formal requirements on traces, the detection of errors depends on the events in the traces. As a consequence, more traces provide a higher chance of detecting errors. Thereby eliminating risk of the tester missing important cases. The presented verification approach firstly specifies requirements in lin- ear temporal logic and converts this specification to a non-deterministic Büchi automaton, or a finite state machine which is also evaluated. Secondly, the approach describes several alternatives for collecting traces and how to link those to the formal specification. Lastly, the verification approach proposes an algorithm which takes the Büchi automaton and a trace to detect violations of the requirement. The validation approach is implemented in form of multiple tools and its op- eration shown by means of a toy example. This example models a railway application such that its requirements can be verified using the tools. The re- sults are then used to show how these tools can be used in an actual railway application. Using these research outcomes, and the stand-alone tool, an implementation in Trace Compass is created. This can, just like the stand-alone tool, decide for each pair of trace and requirement whether the trace violates the requirement. II ABSTRACT Swedish abstract Programvaruutveckling i järnvägsapplikationen styrs av strikta standarder som syftar till att säkerställa säkerheten. Det rekommenderas till exempel starkt att använda formella metoder när krav anges. Dessutom är det obligatoriskt att vissa roller uppfylls av olika ingenjörer. En vanlig teknik är att utveckla programvarutest för kraven. Det är en stor utmaning att se till att programvarukrav beskrivs, tolkas och implementeras på rätt sätt av olika ingenjörer. Tester beror helt på testaren för att täcka alla scenarier. Att ha fler metoder som förenklar spårning av krav och som beror mindre på testarens noggrannhet skulle ge många fördelar. Denna avhandling undersöker om och hur spårning av programvara (software tracing) kan användas för att värdera formella krav på programvara. Målet är att utföra spårningsvalidering så att den kan användas för att komplettera mer traditionella verifieringstekniker. Genom att verifiera formella krav på spår (trace) beror upptäckten av fel på händelserna i spåren. Som en konsekvens ger fler spår större möjlighet för att detektera fel. Därmed elimineras risken för att testaren missar viktiga fall. Den presenterade verifieringsmetoden specifi- cerar först kraven i linear temporal logic och omvandlar denna specifikation till en icke-deterministisk Büchi-automat, eller en ändlig tillståndsautomat som också utvärderas. För det andra beskriver tillvägagångssättet flera alternativ för att samla in spår och hur man länkar dem till den formella specifikationen. Slutligen föreslår verifieringsmetoden en algoritm som tar Büchi-automaten och ett spår för att upptäcka överträdelser av kravet. Valideringsmetoden implementeras i form av flera verktyg och dess funktion visas med hjälp av ett leksaksexempel. Detta exempel modellerar en järn- vägsapplikation så att dess krav kan verifieras med verktygen. Resultaten an- vänds sedan för att visa hur dessa verktyg kan användas i en verklig järn- vägsapplikation. Med hjälp av dessa forskningsresultat och det fristående verktyget skapas en implementering i Trace Compass. Detta kan, precis som det fristående verktyget, avgöra för varje par av spår och krav om spårningen bryter mot kravet. CONTENTS III Contents 1 Introduction 1 1.1 Project goal . .2 1.2 Literature survey . .2 1.3 Structure . .4 2 Requirement specification survey 5 2.1 Formal definition . .5 2.2 Verification . .7 3 Tracing survey 10 3.1 Tracing tools . 10 3.2 Tracepoints . 12 3.3 Information reduction . 17 4 Method 18 4.1 BT’s constraints . 18 4.2 Tool architecture . 19 5 Requirement conversion evaluation 21 5.1 Specification . 21 5.2 FSM algorithm . 23 5.3 NBA algorithm . 24 6 Verification evaluation 27 6.1 State reconstruction . 27 6.2 Automaton execution . 27 7 Application 30 7.1 Bombardier Transportation . 30 7.2 Trace Compass . 32 8 Conclusion 33 9 Future work 34 9.1 Timing . 34 9.2 Undecidability . 35 9.3 Requirement specification . 35 IV CONTENTS A Requirement specification 43 A.1 Example requirement . 43 A.2 JSON Schema . 46 B Requirement negation 51 C NBA specification 53 C.1 Example NBA . 53 C.2 JSON Schema . 55 D NBA conversion 58 E Trace Compass 61 1 INTRODUCTION 1 1 Introduction In situations where traditional verification techniques do not produce satisfac- tory results, tracing may be part of the solution. Tracing can be used to obtain information on a process while it is executing. Generally, this information is used for debugging, profiling and verification. By inserting tracepoints in the source code of the program, or using existing tracepoints in the kernel, one is able to identify the state of the execution. Every time execution passes by this tracepoint, an event is generated. These events are shown in figure 1. Each event may be of a different type. Depending on the type they can have different parameters. Together these events form a trace. A trace is a representation of the execution of a process. In turn, this execution is a finite prefix of a possibly infinite run of that process. This project seeks to apply analysis techniques to traces to verify formal re- quirements of the program. This approach can then be used to complement other forms of software verification [28]. This project was proposed by Bombardier Transportation (BT). BT is a mo- bility solution provider with a broad rail industry portfolio. Its products range from trains to sub-systems and from signalling to complete transport systems. 500 million people in 60 different countries rely on a daily basis on BT’s tech- nologies for their commute. In the European Union, railway applications have to conform to several stan- dards, including EN 50128 [13]. Five safety integrity levels (SILs) are defined to indicate different sets of requirements that ensure different levels of safety. For a product to be used as a system critical component, the development pro- cess must conform to a higher SIL. Verification through traces is an uncommon method of verification. In case it would be used as the primary verification method in BT, it may be subject to Trace Event Event Event int string double Figure 1: Abstract representation of a trace and its run. 2 1 INTRODUCTION additional checking during the assessment phase since it has not been explic- itly pre-approved by the standard. Especially for higher SILs, this may cause development delays. Using tracing as a complementing verification method- ology should not cause any such problems. 1.1 Project goal The goal of this thesis is to investigate how traces can be used in order to verify requirements of a system. The primary goal of the requirements that will be checked is to specify the temporal behaviour (i.e. ordering of events) of the system. Some requirements may have an absolute time limit associated to it. Even though temporal and timing requirements sound similar, both describe very different types of requirements. This report will mainly investigate the trace validity problem [35]. Traces will be collected in software (i.e. not through hardware techniques [14]) on the Linux operating system running C, C++ or similar programs. Java programs are explicitly excluded, as they can use features available in the Java VM. The goal is to gather and verify event based traces. In other words, for each event that occurs, an event entry is added to the trace. The requirements that were specified should be verifiable based on these events. They may include timestamps which would, in the future, allow the verification of timing con- straints in requirements. The final goal of this project is to identify tracing techniques and use this knowledge to create a proof of concept system to verify requirements on BT’s software. Some reasoning and choices have been made according to their needs. These are specified in section 4.1. 1.2 Literature survey Related to this project, Butt [12] completed their master thesis on a similar topic also at BT. However, rather than to use tracing for verification, they use it for profiling. The report gives a useful overview of tracing tools on Linux and FreeBSD. Also, it provides some details on the architecture of one of BT’s systems. 1 INTRODUCTION 3 In 1998, Lee et al. [31] proposed the monitoring and checking (MaC) frame- work to verify a system at run-time. This led to the first Runtime Verification1 (RV) workshop in 2001. The objective of RV is “to monitor, analyze and guide the execution of programs” which is similar to the goal of this project. Since then, many more suitable techniques – which are based on MaC – have been proposed.