Locating Faults in External C Code

Automated fault localization in external C code of Eiffel programs

Reto Ghioldi

Master Thesis

Supervising Professor: Bertrand Meyer Chair of Software Engineering, ETH Zürich

Supervisors: Ilinca Ciupa, Andreas Leitner

March 2006 Masters of war

Come you masters of war How much do I know You that build all the guns To talk out of turn You that build the death planes You might say that I’m young You that build the big bombs You might say I’m unlearned You that hide behind walls But there’s one thing I know You that hide behind desks Though I’m younger than you I just want you to know Even Jesus would never I can see through your masks Forgive what you do

You that never done nothin’ Let me ask you one question But build to destroy Is your money that good You play with my world Will it buy you forgiveness Like it’s your little toy Do you think that it could You put a gun in my hand I think you will ﬁnd And you hide from my eyes When your death takes its toll And you turn and run farther All the money you made When the fast bullets ﬂy Will never buy back your soul

Like Judas of old And I hope that you die You lie and deceive And your death’ll come soon A world war can be won I will follow your casket You want me to believe In the pale afternoon But I see through your eyes And I’ll watch while you’re lowered And I see through your brain Down to your deathbed Like I see through the water And I’ll stand o’er your grave That runs down my drain ’Til I’m sure that you’re dead

You fasten the triggers For the others to ﬁre Bob Dylan, 1963 Then you set back and watch When the death count gets higher You hide in your mansion As young people’s blood Flows out of their bodies And is buried in the mud

You’ve thrown the worst fear That can ever be hurled Fear to bring children Into the world For threatening my baby Unborn and unnamed You ain’t worth the blood That runs in your veins iii Abstract

AutoTest is a fully automatic testing tool for Eiffel classes and is based on the ideas of Design by Contract. However, external code (such as external C/C++ code) is usually not equipped with contracts. This means that often the manifestation of a bug occurs far from the actual location of the fault in the software and there is no indication of the type of error that occurred. In this thesis we evaluate different approaches for gaining additional information about faults in external code and for tracing them back to the Eiffel code. The solution is based on a technique called dynamic binary analysis (DBA). We chose Valgrind’s Memcheck as the tool that our work is based on, designed and implemented a universal framework for automated DBA and ﬁnally built an extension to AutoTest in order to allow it to use this automated analysis. We successfully detected the real error causes in over 93% of our Eiffel test applications with different kinds of bugs in external code and could provide the programmer useful debugging information. iv Acknowledgment

This master thesis has been written during the winter of 2005/2006 at the Chair of Software Engineering, ETH Zürich. I am deeply thankful for the open Swiss educa- tional system. I would like to thank my supervisors, Ilinca Ciupa and Andreas Leitner, for their great support during these months, their comments and helpful suggestions. Further, I would like to thank Prof. Dr. Bertrand Meyer who made this thesis possible. I also thank Hans Dubach for his support during my time at ETH Zürich. Without him in the student administration, life would have been so much more stressful and complicate. I always will appreciate my parents and siblings for their love and for their support of my education right from the very beginning. Thank you Beatrice, Rinaldo, Andrea, Daniela and Stefan. Last but not least, I would like to thank my girlfriend Sandra Käser for her endless love, support and patience. Contents

1 Introduction & Overview 1 1.1 Testing ...... 2 1.1.1 Manual testing ...... 2 1.1.2 Automated testing ...... 2 1.1.3 Black and white box testing ...... 3 1.2 Design by Contract ...... 3 1.2.1 AutoTest: The Eiffel approach ...... 4 1.2.2 Korat: The Java approach ...... 5 1.3 Eiffel compilation process ...... 6 1.4 Scope of thiswork ...... 6 1.4.1 Theoretical part ...... 7 1.4.2 Practical part ...... 9 1.4.3 Test results ...... 10 1.4.4 Conclusion & Outlook ...... 10

2 Related Work 11 2.1 Testing Eiffel programs in general ...... 12 2.1.1 Test input ...... 12 2.1.2 Test results ...... 12 2.2 TestStudio ...... 13 2.2.1 Overview ...... 13 2.2.2 Advantages ...... 13 2.2.3 Limitations ...... 13 2.3 AutoTest ...... 14 2.3.1 Overview ...... 14 2.3.2 Advantages ...... 14 2.3.3 Limitations ...... 14

3 Dynamic binary analysis 17 3.1 Testing binary code in general ...... 18 3.1.1 Problems of testing binary code ...... 18 3.1.2 Scope of dynamic binary analysis ...... 19 3.2 Solutions and limitations ...... 20 vi CONTENTS

3.2.1 Replace memory allocators and de-allocators ...... 20 3.2.2 Dynamic binary compilation and instrumentation ...... 21 3.3 Available tools ...... 22 3.3.1 Valgrind ...... 22 3.3.2 DynamoRIO ...... 22 3.3.3 DynInst ...... 22 3.3.4 Pin ...... 23 3.3.5 DIOTA ...... 23 3.3.6 Strata ...... 23 3.3.7 Commercial and closed source tools ...... 23 3.4 A closer look at the Valgrind Framework ...... 24 3.4.1 Why Valgrind? ...... 24 3.4.2 Glossary ...... 24 3.4.3 The basic idea ...... 25 3.4.4 Example translation ...... 26 3.4.5 Example analysis ...... 28 3.4.6 Limitations ...... 29

4 Enspect 31 4.1 Overview ...... 32 4.2 Intended results ...... 32 4.2.1 Scope ...... 32 4.2.2 Architecture ...... 32 4.2.3 AutoTest integration ...... 32 4.2.4 Usecases ...... 33 4.3 Execution model ...... 33 4.3.1 Normal execution ...... 33 4.3.2 Execution under enspect ...... 33 4.4 Core - Plugin concept ...... 35 4.4.1 Core ...... 35 4.4.2 Deferred Plugin classes ...... 41 4.5 Further design aspects ...... 45 4.5.1 Structure of a bug situation ...... 45 4.5.2 Command line tool ...... 45 4.6 Example plugin for AutoTest ...... 46 4.6.1 Instruction execution ...... 46 4.6.2 Analysis of bugs ...... 51

5 Results 61 5.1 Synthetic test cases ...... 62 5.1.1 Bad free ...... 62 5.1.2 Bad address value ...... 62 5.1.3 Badjump ...... 63 CONTENTS vii

5.1.4 Badloop ...... 63 5.1.5 Bad read/write ...... 63 5.1.6 Bad C library calls to brk() and sbrk() ...... 64 5.1.7 Double free ...... 64 5.1.8 Custom allocators ...... 65 5.1.9 File descriptors ...... 65 5.1.10 Read/write using pointers ...... 66 5.1.11 Leak check ...... 66 5.1.12 Too long function match ...... 68 5.1.13 Overlapping src and dst ...... 69 5.1.14 Allocator errors ...... 70 5.1.15 Buffer overﬂow ...... 70 5.1.16 NULL pointers ...... 71 5.1.17 Results ...... 71 5.2 Real World testcase in Eiffel Media ...... 71

6 Conclusion and Outlook 73 6.1 Conclusion ...... 74 6.2 Limitations ...... 74 6.3 Further work & Outlook ...... 75

Bibliography 76

A Source Code 81 A.1 enspect: ESP_AUTO_TEST_ANALYZER ...... 82 A.2 AutoTest: AUT_C_EIFFEL_LOOKUP_TABLE ...... 87

B Documentation 93 B.1 Requirements ...... 94 B.2 Installation notes ...... 94 B.3 Usage ...... 95

C Project Plan 97 C.1 Project Description ...... 98 C.1.1 Overview ...... 98 C.1.2 Scope of the work ...... 98 C.1.3 Intended results ...... 98 C.2 Background Material ...... 98 C.2.1 Reading list ...... 98 C.2.2 Tools ...... 99 C.3 Project Management ...... 100 C.3.1 Objectives and priorities ...... 100 C.3.2 Criteria for success ...... 100 viii CONTENTS

C.3.3 Method of work ...... 100 C.3.4 Quality management ...... 100 C.4 Plan with Milestones ...... 101 C.4.1 Project steps ...... 101 C.4.2 Deadline ...... 101 C.4.3 Tentative schedule ...... 101 Chapter 1

Introduction & Overview

“A goal without a plan is just a wish.” (Antoine de Saint-Exupéry) 2 Chapter 1: Introduction & Overview

This chapter provides a short introduction to state of the art techniques for testing computer programs. The focus is set on the programing language Eiffel [20]. Further, an overview of this thesis is provided.

1.1 Testing

Testing plays an important role in software development [38]. Unfortunately due to time pressure or a lack of project management testing often is neglected. During the years several techniques have been developed to improve testing efﬁciency and make it more attractive to programers. In this thesis the idea to test a certain software system remains always the same:

1. setup the system under test 2. create a test case for this system 3. execute the test case on the system 4. check if the output corresponds to the systems speciﬁcation

These steps can either be processed manually by the tester or (semi-)automated.

1.1.1 Manual testing The simplest way to test a program is to just have a set of specified use cases (like sample input or a sequence of user interactions) and then check manually if they run down correctly by the system under test. Because this is very easy to realize, it’s a widespread approach. Also in complex systems like developed in the Mozilla project, they - among other proceedings - use test cases [12] or the human driven reporting tool Bugzilla [11] to discover and track bugs in the web browser Firefox or the mail client Thunderbird. Not all bugs can be found by manually testing a program and every time the underlying system changes, the test cases have to be rebuilt in order to match the new specifications. Thus programers find bugs only by chance and manual testing remains an awkward and time consuming task.

1.1.2 Automated testing Testing frameworks like JUnit [14] help the programer to ﬁnd bugs already during development phase. In this kind of frameworks, the programmer speciﬁes the expected response and the tool checks if the actual calculation result matches. Test execution and result checking is automated whereas the test cases themselves still need to be written by hand. Because these frameworks can be knitted only with a few lines of code, they exist for almost every available programing language. 1.2 Design by Contract 3

Current research projects try to improve automation even more and generate test cases, too. A formal specification of the software components must be provided (in a special language) by the programer. Analyzing these specifications, the tools are able to produce, compile, execute and verify the result of a test case. For Eiffel (see section 1.2.1), Java (1.2.2) and other languages, there exist implementations. It’s important to know that also fully automated testing has its limitations. For example, it relies on the quality of the mentioned software specification and also can’t cover all possible programing problem causes like scalability, security, compatibility with other programs, requirements for a complex execution environment or verification of human specific output like audio or video data.

1.1.3 Black and white box testing It’s important to keep in mind the two general possibilities to test a software system.

Black box testing The term black box indicates that the internal implementation of the program being executed is not examined by the tester. That’s also the reason why black box testing is not normally carried out by the programmer who has the knowledge of the application’s code. In most real-world engineering companies, one group designs work while another group does the testing. White box testing The term white box (or glass box) indicates that testing is done with knowledge of the code used to execute certain functionality. For this reason, the programmer who knows the internal structure of the system is usually required to perform white box tests himself. Because of this, the term structural testing is also used.

In this thesis we mainly deal with black box testing.

1.2 Design by Contract

The idea of Design by Contract (DbC) is viewing the relationship between a class and its clients as a formal agreement, expressing each party’s rights and obligations [27]. The source code is augmented by adding preconditions, postconditions, class invariants, loop (in)variants and check instructions. DbC is directly related to the idea of an Abstact Data Type (ADT), a mathematical speciﬁcation of a set of data and the set of operations that can be performed on the data. Eiffel supports the DbC software development methodology by integrating contracts into the language 1. But it also gains more and more popularity in other languages like Java or .NET. The following two tools implement automated testing based on the DbC principle.

1The contracts for pre-, postconditions and invariants are always present. If not explicitly written, some weak default contracts are assumed. 4 Chapter 1: Introduction & Overview

1.2.1 AutoTest: The Eiffel approach AutoTest has been developed at ETH Zürich by Andreas Leitner and Ilinca Ciupa [9]. It’s a successor of Eiffel TestStudio [8] and TestWizard [16]. The AutoTest command line tool checks automatically if an Eiffel system respects its contracts. Let’s assume we have the contracted Eiffel stack dispenser class from Meyer’s Eiffel book [27]. The implementation is omitted in this listing.

Listing 1.1: Eiffel stack dispenser with contracts § ¤ 0 indexing description: ''Stacks: Dispenser structures with a Last-In, First-Out access policy''

class STACK [G]

5 creation make

feature -- Initialization make (n: INTEGER) is 10 -- Allocate stack for a maximum of n elements require non_negative_capacity: n >= 0 do ... 15 ensure capacity_set: capacity = n end

feature -- Access 20 count: INTEGER -- Number of stack elements

capacity: INTEGER -- Maximum number of stack elements 25 item: G is -- Top element require not_empty: not empty -- i.e. count > 0 30 do ... end

feature -- Status report 35 empty: BOOLEAN is -- Is stack empty? do ... 40 ensure empty_definition: Result = (count = 0) end

full: BOOLEAN is 45 -- Is stack representation full? do ... ensure full_definition: Result = (count = capacity) 50 end 1.2 Design by Contract 5

feature -- Element change put (x: G) is -- Add x on top. 55 require not_full: not full do ... ensure 60 not_empty: not empty added_to_top: item = x one_more_item: count = old count + 1 end

65 remove is -- Remove top element. require not_empty: not empty -- i.e. count > 0 do 70 ... ensure not_full: not full one_fewer: count = old count ? 1 end 75 invariant count_non_negative: 0 <= count count_bounded: count <= capacity empty_if_no_elements: empty = (count = 0) 80 end ¦ ¥

Given the class name (and an Eiffel ACE file where the STACK class is part of the assembly), AutoTest first analyzes the system under test and builds a strategy for its testing. Secondly, it creates input for the test cases. The current implementation uses a random input generation strategy and hence does not always respect the require clauses of the individual features. Nevertheless it performs quite well. In a next step AutoTest executes the tests and finally verifies the result with the given ensure clause and the class invariant. As an example for a test case of feature put(), a (generic) input for the only argument x : G will be created. Because the precondition of the feature is independent from x, every instance of G fulfills the contract, only a stack validation, i.e. a capacity check is performed. After execution, the result is verified with the postcondition and class invariant. The only contract that checks the internals of x is added_to_top with its implicit call to x’s comparison feature. All other contracts concern only the stack’s capacity.

1.2.2 Korat: The Java approach Korat has been developed by Chandrasekhar Boyapati, Sarfraz Khurshid and Darko Marinov at the MIT [7]. Unfortunately there was never a public source code or binary release of their tool. Korat uses annotations in JML [21] and the JUnit framework to create and execute test cases. The test results are veriﬁed with automatically built test 6 Chapter 1: Introduction & Overview

oracles. The test cases as well as the test oracles are constructed in a very efﬁcient way. Due to the lack of a public available implementation it’s not possible to analyze this tool in all details and continue the development (such as for the new Java generics). Listing 1.2 shows an example of Java contracts in JML (source code comments). They are not part of the ofﬁcial language and hence are seldom seen in non-academic software. We won’t deal with Java any further in this thesis and focus on the Eiffel approach.

Listing 1.2: JML annotations from a Java function of a Heap class § ¤ 0 /* @ public normal_behavior @ requires size > 0; @ ensures \result == \old(array[0]); @ also public exceptional_behavior @ requires size == 0; 5 @ signals (IllegalArgumentException e) true; @*/ public Comparable extractMax() { // ... method body } ¦ ¥

1.3 Eiffel compilation process

Most Eiffel compilers use ANSI C code as intermediate representation when they build an executable program from Eiffel source code. There exist also approaches with a direct compilation process to machine code [34] but they never reached the same popularity as the two-step solutions. For example, the Eiffel Software compiler [35] uses the Melting Ice Technology to achieve immediate recompilation after a change, guaranteeing a recompilation time that’s a function of the size of the changes, not of the system’s overall size. In a process called “Freezing”, Eiffel code is compiled to ANSI C code and the C code is compiled by a C compiler which results in a monolithic executable machine code image. In “Melting”, small changes are compiled into an intermediate bytecode which replaces at runtime a portion of the frozen image. In fact, Eiffel Software’s compiler is actually a three-step compiler. However, with the Microsoft .NET platform, there may be a reversal of the trend to one-step compilers because single modules of a .NET assembly can be exchanged without touching the rest of the system.

1.4 Scopeof this work

This thesis presents an analysis of the problem (testing external code in Eiffel programs where no contracts are available) and a theoretical solution. In a second part it presents an implementation which turns those ideas into Eiffel code, i.e. the tool called enspect 1.4 Scope of this work 7

, and an integration into AutoTest. Last but not least some synthetic and real world tests have been preformed with the newly developed tool.

1.4.1 Theoretical part Limitations of AutoTest The AutoTest tool has some limitations in its approach. Of importance for this work is the missing ability to check external code of Eiffel programs where no contracts exist. This is the case whenever a feature’s behavior is imported with the external ''C'' clause. As an example let’s consider an Eiffel random number class RANDOM_NUMBER listed in 1.4 which makes use of the standard C random number generator by two internal wrapper features (c_rand() and c_srand()). These features simply forward the Eiffel call to external C functions with the same name. Note that with parameters or return values of more complex data types also the wrapping would become more difﬁcult. But to get the idea this example will do. On the C side we just redirect the Eiffel calls to the C standard library. You could also perform a rather complex calculation here. Either way, there are no contracts on the C side.

Listing 1.3: C code for a simple random number ”library” § ¤ 0 #include

int c_rand(void){ return rand(); } 5 void c_srand (unsigned int seed){ srand(seed); } ¦ ¥

Listing 1.4: Eiffel side of a C code based random number ”library” § ¤ 0 class RANDOM_NUMBERS

feature last_random: INTEGER;

5 next_random is do last_random := c_rand; end;

10 set_seed (new_seed: INTEGER) is do c_srand(new_seed); end; 8 Chapter 1: Introduction & Overview

15 feature{NONE}

c_rand : INTEGER is external "C" end; 20 c_srand (seed: INTEGER) is external "C" end; end ¦ ¥

In the above example, AutoTest won’t be able to follow the C calls and check them internally for correctness or side effects. Hence, if you are implementing an Eiffel program based on a corresponding C low-level library it will be very hard to ﬁnd the bugs on the C side. Also because you can’t count on the support of AutoTest’s automated testing ability. An Eiffel debugger like the one of ISE EiffelStudio [36] won’t help you either to scrutinize the external code. There may be no debugging information available in the external code.

Scope of the implementation Before doing some research about available tools to gain indications of faults in external code, we setup a scope of bugs we want to cover. Not all kinds of bugs can be found, even with the help of the best analysis tool. But after categorizing the types of errors it turns out that most of the bugs are directly or indirectly related to memory management, hence we focus on tools which are strong in this area.

Analysis of available tools To get indications of what happens in the external calls we examined several memory checkers, proﬁlers or debuggers. Not all programs satisfy our needs. In section 3.3 we look at the advantages and disadvantages of available implementations. Our choice for this work is the Valgrind supervision framework [31]. Valgrind ﬁnds memory-management problems by checking all reads and writes of memory, intercepting all calls to malloc/free for C and new/delete for C++ respectively. As a result, Valgrind can detect problems like

• The use of uninitialised memory

• Reading or writing memory after it has been free’d

• Reading or writing off the end of malloc’d blocks

• Reading or writing inappropriate areas on the stack

• Memory leaks

• Passing of uninitialized and/or unaddressible memory to system calls 1.4 Scope of this work 9

Valgrind tracks each byte of memory with nine status bits: one tracks addressabil- ity, and the other eight track validity. As a result, it can also detect the use of single uninitialized bits, and does not report spurious errors on bitﬁeld operations. Valgrind debugs almost any dynamically-linked executable, even without modiﬁcation or recompilation.

Architecture for a robust implementation In the last theoretical part we present an architecture for a robust implementation. In- stead of integrating Valgrind directly in our application or linking against its libraries, we use standard POSIX interprocess communication to control Valgrind’s memory tracer plugin Memcheck. On the Eiffel side Berend de Boer’s e-POSIX library binding is used [10]. We use, like the Valgrind authors also do, a core-plugin approach to unify our software in a very flexible way. The enspect core provides a base infrastructure, sets up the whole test environment, controls the program flow and delegates the input and output of the tool. It further deals with error handling and logging. A separate plugin (we provide one for AutoTest test cases) inspects a fault in the external code, analyzes the collected meta data and provides debugging hints for the programer in a bug report. Thanks to this flexible architecture, we can, by writing an additional plugin, use almost every third party analysis tool to gain the bug meta data.

1.4.2 Practical part For the practical part of this work, we created a set of test cases which are not traceable with AutoTest but should be with our new tool, named enspect . From our theoretical solution we derive a core application and a sample plugin for the AutoTest-Valgrind tool combination. All together can be seen as an extension to the current AutoTest version.

Implementation Our implementation enspect turns the idea of the theoretical part into Eiffel code. In a first run, a system (i.e. its ACE file) is tested with AutoTest. In the second run, the enspect command line tool is fed with an AutoTest invalid or bad response case, packed as (minimized) sequence of AutoTest interpreter commands [24]. Together with the ACE file of the system under test and AutoTests interpreter, enspect generates a bug report with Eiffel stack trace and debugging hints.

Synthetic testcases

The synthetic test cases make use of the external ''C'' clause in Eiffel and contain a bug. The fault lies completely on the C side. Some of these bugs like memory 10 Chapter 1: Introduction & Overview leaks aren’t even recognized in normal testing or debugging. Other test cases result in a segmentation fault or any similar bad error state... and lead in most cases into a complete program crash.

Real world tests with EiffelMedia Since three years ETH students lead by Ph.D. student Till Bay and Michela Pedroni are developing Eiffelmedia [3], an Eiffel multimedia library, based on the highly portable Simple Directmedia Layer (SDL) [30]. SDL is written in C, hence Eiffelmedia components widely use the external ''C'' clause. In fact, Eiffelmedia generates the wrapper classes automatically with the Eiffel Wrapper Generator tool [22]. Does an Eiffelme- dia programmer call a SDL libary function with faulty parameters, it may fail. Because of the external ''C'' clause debugging is hard and hints like the one from enspect may be essential to ﬁnd the error.

1.4.3 Test results We will present the results of our synthetic and real world tests in a short overview and show possible hints enspect is able to provide to the programmer. Further, we will discuss possibilities to improve the tool.

1.4.4 Conclusion & Outlook Looking back at the whole project and comparing the results with the intended results in appendix C, we will make a ﬁnal conclusion. We will also discuss future development and features of enspect in a ﬁnal outline. Chapter 2

Related Work

“Study the past if you would deﬁne the future.” (Confucius) 12 Chapter 2: Related Work

The presented tools in this chapter, TestStudio (see section 2.2) and AutoTest (2.3), are the base for the theoretical and practical part of this work. Hence we will present in a short summary how they work and what their limitations are. Section 2.1 explains some basic knowledge about automated push-button testing in general.

2.1 Testing Eiffel programs in general

2.1.1 Test input The testing tools TestStudio and AutoTest use the same process for a testing session:

1. Select software elements to test

2. Build test cases

3. Execute the test cases

4. Compare actual results with expected results (as given by oracle)

5. Measure execution characteristics like time, space, etc. (this is optional)

All these steps run without any user interaction. In TestStudio the user also has the possibility to specify the range of the base test input data (like INTEGER, REAL, CHARACTER, etc.) and to enter other parametrisations. On the other hand, AutoTest always chooses the test input according to its test strategy in an automated manner.

2.1.2 Test results Every test response, with granularity down to feature calls, falls into one of the following four categories:

Passed The test succeeded and there has no exception occurred (for example an as- sertion violation).

Failed The test has failed because of one of the following reasons:

• The feature or one of its subfeatures 1 violated a postcondition. • A call to a subfeature didn’t fulﬁll the precondition.

• There was a check instruction violated. • During or after the test, a class invariant has been broken and/or a loop variant/invariant was violated. 1 subfeature: a feature (recursively) called by the feature under test 2.2 TestStudio 13

Invalid The test could not be executed due to a violated precondition of the feature or a broken class invariant prior to the feature execution.

Bad response The response of the execution process of the test tool doesn’t make any sense. The process may has triggered a timeout event and therefore been killed.

Together with this information the programer is able to narrow down the location where the error occurs to only a few lines of code and also gets a ﬁrst attempt to a reusable counter-example to the correctness of his code.

2.2 TestStudio

2.2.1 Overview TestStudio written by Ilinca Ciupa [8] is an advancement of Nicole Greber’s Test- Wizard [16]. Both are based on Design by Contract and use this Eiffel source code information to generate test cases automatically. Where TestWizard was at ﬁrst a simple pioneer approach, TestStudio is a whole testing environment including test project management, graphical user interface (GUI) and a detailed report of the test results. A command line version is available too.

2.2.2 Advantages TestStudio is a very comfortable way to test an Eiffel program. The process of creating, compiling and executing tests is completely automatic. Detailed result information is provided and tweaking the test input as well as scope is allowed. Test projects may be saved and later restored for a feature recheck as soon as the bug has been ﬁxed.

2.2.3 Limitations TestStudio and the underlying TestWizard library have the following limitations:

1. They cannot automatically test features in descendants of a class. The tester must include them manually into the test case.

2. Inﬁnite loops block TestStudio and abort further testing.

3. Acalltothe die() feature of the EXCEPTIONS class would also terminate TestStudio and thus is prohibited.

4. They can’t test expanded types.

5. TestStudio skips external ''C'' features because they could result in a runtime panic and thus kill the TestStudio process. 14 Chapter 2: Related Work

6. Limited CAT-call (Changing Availability or Type) checking.

7. Randomly created input for test (when no input is provided by the user) won’t be saved. Hence the test is not repeatable.

8. Cannot handle nested genericity.

9. Objects mutate only by calling routines without return value. Some of these calls may also be redundant.

10. Information about the object which produces the bug is not stored.

2.3 AutoTest

2.3.1 Overview In September 2005 Ilinca Ciupa and Andreas Leitner released a completely rewritten version of their automated testing tool with the name AutoTest [9]. It’s a command line tool with a more ﬂexible master-slave architecture [25] using the Eiffel Reﬂection Library Generator (Erl-G) package [24]. The actual feature execution is performed in a dynamically built Eiffel interpreter. The test input data is built randomly. Although this is just a temporary solution and there will be used a cleverer strategy in the future, this simple approach produces surprisingly good results. Further, a pool of already built and mutable objects acts as a cache to improve performance.

2.3.2 Advantages Thanks to the master-slave architecture, AutoTest is very stable and remains as ﬂexible and adjustable as TestStudio was. Additionally, the test driver (master process) may kill the executing interpreter (slave process) after a timeout event occurred or may restart the interpreter if it died for some other reason. Thus, inﬁnite loops or even segmentation faults on operating system level won’t harm the automated test process anymore.

2.3.3 Limitations Although AutoTest solves many limitations of TestStudio, it has its own drawbacks:

1. The current random input generation strategy doesn’t take preconditions into account. A planning based strategy which solves this problem is in development [25]. 2.3 AutoTest 15

Figure 2.1: AutoTest presents its test result in a well structured HTML report. More detailed logﬁles are also provided.

2. Aborting a ﬁnite loop too early and reporting it as inﬁnite will produce a false positive. A clean solution for this problem has not yet implemented in AutoTest.

3. After the non-answering interpreter process has been killed, there is no information about the cause available. Although an Eiffel stack trace can be provided after a runtime crash, no details about the real cause are available. Often just a segmentation fault is reported. The excessive use of external ''C'' clauses in many applications or libraries only aggravates this problem.

For limitation 1, work is in progress. Problem 2 has a more theoretical background and hence is not easy to be solved (if possible at all 2) [39]. For limitation number 3 this work presents a solution.

2Alain Turing’s halting problem has the consequence that in general an automated veriﬁcation and decision whether a program terminates or not, is impossible. 16 Chapter 2: Related Work Chapter 3

Dynamic binary analysis

“Where facts are few, experts are many.” (Donald R. Gannon) 18 Chapter 3: Dynamic binary analysis

The main goal of this thesis, to gather information about faulty external ''C'' calls in Eiffel programs, contains innumerable low-level and operating system aspects. It doesn’t make sense to reinvent the wheel with deadline constraints, hence we use an existing solution. In this chapter we will give reasons for having chosen Valgrind and explain how the tool works and what its limitations are.

3.1 Testing binary code in general

3.1.1 Problems of testing binary code Testing binary code, like third-party libraries or already compiled program components, is exactly what’s needed when we have to gather information about faulty external ''C'' calls in Eiffel programs. The programmer would like to know whether the origin of the bug was in an invalid actual parameter of his feature call or in the external code itself. Also hints about the kind of error are of interest.

Missing source code The main problem of testing external (binary) code is deﬁnitely the missing source code. Even when you disassemble machine code, there may be a lot of important information lost during compilation and optimization. There can never be a completely automated disassembly tool which always outputs correct source code because the disassembly process reduces to the impossible-to-solve halting problem 1. Deeper knowledge of reverse-engineering and human creativity are needed. That is why we can only approximate a solution to the problem. The exact cause of the bug in the external code will remain uncertain. With the help of debugging information and a symbol table in the external code some additional (human readable) information could be gained. In commercial or optimized code, no debugging information and sometimes not even the symbol table are present and hence we often know just the nameless memory address of a program part together with a sequence of assembly instructions. A static check of the code without debugging information becomes impossible as we cannot recognize pointers to static data objects. On the other hand, a stripped symbol table further complicates error checking because function entry and exit points on the stack can’t be detected.

Loss of program ﬂow Not only missing source code navigates analysis of external code through shallow depth water. Also the risk of losing control over program ﬂow is omnipresent. When-

1This problem could be solved when an interpreter or a just-in-time (JIT) compiler would produce the external machine code at runtime. “Unfortunately” because of performance reasons today’s JIT compilers use lossy intermediate code as input. Some ﬁrst-level compilers even obfuscate the intermediate code to make de-compilation nearly impossible. 3.1 Testing binary code in general 19 ever (in a second, third, etc. level) a system or further library call is performed you may not follow it. For example, the code of a system call runs in the processor’s kernel space mode which a user space mode program has no access to. In such cases you can only pray that the call will return correctly from the callee 2. If these (system) calls are asynchronous, the issue is even more delicate. Particu- larly in network programming or other synchronization tasks, most calls don’t return immediately but sometime in the future. The subroutine indicates its result in general by a system signal or a highlevel programming framework event. The question is how to catch up on the result and who it passes its original destination to.

Tied to the operating system Besides system calls many other aspects are operating system related and important to know for a binary analysis. The memory layout differs sometimes even between the versions of an operating system and the loader and linker can work completely diverse. Also the process and thread models differ. Sometimes there are operating system interface standards like POSIX [2] which ease the problem. But they don’t solve it in all its aspects.

3.1.2 Scope of dynamic binary analysis Before we look at a possible existing solution, we state what programming errors we want to detect in the external code. This requirements speciﬁcation will help us amongst others in section 3.3 to ﬁnd the optimal base for our further work. We categorize bugs in the following way with a focus on the programming language C

• Memory, Stack and Cache: faults dealing with storage in memory or the processor cache

– Use of uninitialized memory

– Reading/writing memory after it has been free’d (i.e. writing a free’d zone)

– Double free or realloc of already freed pointers – Overﬂow and underﬂow to memory buffers

– Memory leaks (i.e. where pointers to malloc’d blocks are lost forever) – Passing uninitialized and/or unaddressable memory to system calls

– Mismatched use of malloc/new/new[] v.s. free/delete/delete[]

2When the current function (caller) wants to execute a subroutine (callee) in a normal function stack based computer architecture, it pushes arguments and other parameters on the stack and gives program ﬂow to the subroutine. The callee does some more or less interesting things with this input and (eventually) returns control to the caller. 20 Chapter 3: Dynamic binary analysis

– Overlapping src and dst pointers in memcpy() and related functions

– Dereferencing NULL pointers – Memory layout corruption – Reading/writing inappropriate areas on the stack – Cache trashing

• Structural bugs: logical problems in the software

– Inﬁnite loops (this kind of bug is rather academic and we won’t preponder- ate it) – Bad aliasing (interface by-passing, etc.) – Insufﬁcient access rights – Unwanted side effects (i.e. outer invariant violations)

• High-level bugs:

– Array indices out of bound – Invalid type casts – Errors caused by unreliable resources (network, harddrive, hardware failure)

We won’t concern ourselves with errors which appear when dealing with several processes and threads like deadlocks, race conditions, starvation or unfair scheduling. Further, faults of the second and third of the above categories are all reducible to invalid memory access. Consequently, the better we can track the state of the memory the more errors we will detect and we are going to focus on this area.

3.2 Solutions and limitations

We found several solutions recording and verifying memory access. Often the equation “the more complex the technique is, the more details on a fault will be available” applies. After some simple approaches to control access to memory, we focus on larger and very robust binary analysis frameworks.

3.2.1 Replace memory allocators and de-allocators Normal C programs use the four C standard library calls to allocate and de-allocate memory: calloc() Allocates and clears a two-dimensional chunk of memory. 3.2 Solutions and limitations 21 free() Returns previously allocated memory to the operating system. malloc() Allocates memory. realloc() Changes the size of previously allocated memory.

C++ programs make use of the language built-in keywords new/delete to achieve the same goal (for arrays the equivalent new[]/delete[] is taken). After replacing these functions with your own implementation, it’s possible to keep track of (unfortunately not all) dynamic memory changes in your program. Most tools just wrap the standard functions: they catch the original call with their wrapper function, update some meta data and redirect the call to its intended receiver. Thus memory access remains unrecognized if a linked third party library or a garbage collector like the one of Hans Boehm [6] is used. Some implementations of memory tracers like ccmalloc [5] need a recompilation of your source code and require to link it against your tool. Others like Leak Tracer [1] are more ﬂexible and hook in during the loading process, normally by using the operating system’s LD_PRELOAD variable. Although these tools may be useful for searching memory leaks in your applications, they can’t check for illegal access or static memory problems, i.e. memory failures on the stack. Even worse, they can’t check access and modiﬁcations on bit level. A more sophisticated technique is needed for such cases.

3.2.2 Dynamic binary compilation and instrumentation In order to know about every single memory access, knowledge over every processor instruction is required. In fact, better tools don’t execute the original binary code directly but kind of interpret it. The interpretation may also include several translation steps from the source language (often x86, PPC or SPARC) to the execution language. The execution language is normally the same as the source language but this is not compulsory. Because this translation is performed during runtime, it is called dynamic binary compilation. In fact thistechnique does the same a JIT compiler does and hence these tools are actually some kind of virtual machines. An interpretation of the source language alone offers no additional information. All tools presented in section 3.3 collect therefore meta data about the memory state and add additional analysis code, so called instrumentation, before executing the code. The tool can make as many instrumentation passes over the code as it likes, including optimization passes before generating the ﬁnal executable code. It’s obvious that this process slows down execution. In practice a slowdown factor of 2 − 30, depending on the complexity of the instrumentation and the tool, is today’s standard. In the next section, we will present several different tools which can perform such a dynamic binary analysis of a program by instrumenting the code with additional instructions and will report on faulty memory access. 22 Chapter 3: Dynamic binary analysis

3.3 Available tools

Some of the presented tools are rather frameworks than just applications and present a perfect base to build your own analysis tool on. Nevertheless they all offer powerful debugging functionalities.

3.3.1 Valgrind

Valgrind has originally been written by Julian Seward who also developed the bzip2 data compressor or the cache profiler cacheprof. It’s a free programming tool for memory debugging, memory leak detection and profiling, has an excellent reputation and is widely used by Linux programmers. In section 3.4 we take a closer look on how it works. Valgrind is a mature robust system. It’s used by thousands of programmers on a wide range of software types, including the developers of notable software projects such as OpenOffice, Mozilla, KDE, Gnome, MySQL, Perl, Samba, Unreal Tournament and for NASA’s Mars landers and rovers. Valgrind is available under the GNU public license.

3.3.2 DynamoRIO

DynamoRIO [17], derived from Hewlett-Packard’s Dynamo, is a dynamic binary optimisation and instrumentation framework enhanced at the MIT. It also supports active analysis code. It was ﬁrst released in 2001, instrumentation support was added in 2002 and is available for x86 Windows and Linux systems. DynamoRIO’s speed and exact representation of the original client code make it ideal for building lightweight dynamic binary analysis tools, tools using active analysis code such as sandboxes, or for experimenting with dynamic binary optimization. Unfortunately DynamoRIO is available under a non-transferable binary-only licence (free of charge).

3.3.3 DynInst

DynInst is an Application Program Interface (API) to permit the insertion of code into a running program. The goal of this API is to provide a machine independent interface to enable programers to create tools and applications that use runtime code patching. This API is based on the idea of dynamic binary instrumentation and is available for most platforms: Alpha, MIPS, Power/PowerPC, SPARC, x86 (Linux and Windows) and IA64 (Linux). Thanks to adding the instrumentation code only when needed, the slowdown factor without any analysis is nil. The more analysis code is added, the slower execution becomes. Further, it’s possible to instrument already running programs without meta information about the past execution being available. Source as well as binary releases for all major platforms are available. 3.3 Available tools 23

3.3.4 Pin Pin performs also dynamic instrumentation of programs and supports Linux on all In- tel architectures (Xscale, x86, IA64) [19]. The tool is provided free of charge by Intel and can be redistributed. Pin was designed to provide functionality similar to the pop- ular ATOM toolkit for Compaq’s Tru64 Unix on Alpha. Arbitrary C or C++ code can be injected at arbitrary places in the system under test. Unlike Atom, Pin does not instrument an executable statically by rewriting it, but rather adds the code dynamically while the executable is running. This also makes it possible to attach Pin to an already running process. Pin is a nice framework for doing ATOM-style instrumentation, although it might not be suitable for very heavyweight dynamic binary analysis (ex. memory tracking on bit level). Furthermore the instrumentation engine is closed source (the example tools are open source under a BSD-style license).

3.3.5 DIOTA DIOTA is a tool for instrumenting x86 code under Linux [15]. The tool correctly deals with programs that contain traditionally hard to instrument features such as data in code and code in data. The technique does not require reverse engineering, program understanding tools or heuristics about the compiler or linker used. The basic idea is that instrumented code is generated on-the-ﬂy, while the original process is used for data accesses. DIOTA comes with a number of useful backends to check programs for faulty memory accesses, data races, deadlocks and to perform basic tracing operations. Robustness and performance are similar with Valgrind’s, however its instrumentation support is much more limited. DIOTA is, like Valgrind, available under the GNU public license.

3.3.6 Strata Kevin Scott et al. at the University of Virginia are developing the dynamic transla- tor implementation infrastructure called Strata. Strata provides a common framework in which researchers can build dynamic translators. Strata uses several novel techniques to reduce the cost of dynamic code translation, including partial inlining, indirect branch translation caching, fast returns, instruction trace formation and fragment linking. To gather run-time information, Strata has facilities for dynamic instrumentation. Because instrumentation can be very expensive, Strata uses several novel optimizations that target at instrumentation overhead. Unfortunately we couldn’t ﬁnd neither a source nor a binary version even though it should be open source.

3.3.7 Commercial and closed source tools All of the following tools have a commercial background. They are out of the question for our work but are worth mentioning. 24 Chapter 3: Dynamic binary analysis

IBM Rational Purify Purify is a memory debugger program and was originally written by Reed Hastings of Pure Software. After some interstation IBM bought the software and now offers it under the name Rational Purify(Plus) [18]. Purify exists for AIX/PPC, x86 (Windows, Linux), SPARC, IRIX/MIPS and HP-UX/PA-RISC. Purify is very robust and offers a good performance. It was something of the father of all binary instrumentation tools. Its functionalities are similar to those of Valgrind.

Vulcan Microsoft uses the Vulcan tool for internal static and dynamic binary instrumentation [28]. It runs under Windows and supports the architectures x86, IA64 and MSIL (Mi- crosoft Intermediate Language, .NET). The tool is robust. No details are known about the performance.

3.4 A closer look at the Valgrind Framework

3.4.1 Why Valgrind? From the above tools, we restricted our choice to free, open source software. Further, we wanted a robust tool which offers good analysis results. DynamoRIO and Pin drop out because of their license model. DIOTA has a good platform support but lacks analysis quality. For Strata, neither source nor wide enough public distribution is given. We would have chosen DynInst because of its outstanding platform support but it has a restrictive license. So we chose Valgrind which is one of the most robust tools available, has a high-performance, is widely used, provides many possibilities to parametrize its behavior and is available under the GNU public license. Either way, we designed our implementation enspect, presented in chapter 4, as a core-plugin application. By writing a further plugin (we deliver one for Valgrind) every other tool can be used!

3.4.2 Glossary The following glossary from a Valgrind publication [29] provides a short lookup possibility for often used terms when talking about Valgrind.

Basic block A straight-line sequence of machine code whose head is jumped to and which ends in a control ﬂow transfer such as a jump, call, or return. Note: a jump can land in the middle of a previously seen basic block.

Binary analysis The analysis of programs at the level of machine code. Contrast with source analysis. 3.4 A closer look at the Valgrind Framework 25

Client A program running under the control of a Valgrind tool.

Core The main part of Valgrind and the major component in every Valgrind tool, doing most of the work except for instrumentation which is done by a tool plugin.

Dynamic analysis Run-time techniques for remembering approximations to the set of values or behaviors seen so far when executing a program, i.e. the analysis of programs at run-time. This involves the use of analysis code and metadata and contrasts with static analysis.

Dynamic binary analysis The intersection of dynamic analysis and binary analysis.

Instrumentation The act of adding analysis code to a program.

Metadata Approximations to the set of values or behaviors seen so far when executing a program and the key component of dynamic analysis.

Tool plug-in A component of a Valgrind tool which deﬁnes an instrumentation pass over UCode and plugs into Valgrind’s core.

UCode The intermediate representation used by Valgrind. Tool plug-ins perform instrumentation of UCode. Valgrind’s core performs several passes over UCode, including optimisation, register allocation and code generation.

3.4.3 The basic idea Valgrind uses dynamic binary compilation and caching. It hooks itself into the client process during startup, recompiles the client code just-in-time basic block by basic block and executes it. Compilation involves disassembling the client’s machine code (x86) into an intermediate representation (UCode), instruments it and ﬁnally converts it back to executable code (x86). A cache for already translated basic blocks is used (a 300’000 entry hash table with a hit rate of 98%). Running a program thus becomes an execution of linked translated basic blocks. Valgrind can be seen as an additional (transparent) layer between client program and operating system. The instrumentation of the UCode is done by a Valgrind plugin, all the rest is performed by Valgrind’s core. The developers provide some useful default plugins:

Memcheck detects memory-management problems. All reads and writes of memory are checked (nine status bits per original byte) and calls to malloc, new, free, delete are intercepted. Memcheck can detect

• The use of uninitialised memory

• Reading or writing memory after it has already been free’d

• Reading or writing besides malloc’d blocks 26 Chapter 3: Dynamic binary analysis

• Reading or writing inappropriate areas on the stack • Memory leaks • Passing invalid memory to system calls • Mismatched use of allocation v.s. de-allocation

• Invalid use of memcpy() and related functions

• Some misuses of the POSIX pthreads API

We will use this plugin for our implementation described in chapter 4.

Addrcheck is a light version of Memcheck with twice the performance but the plugin detects fewer errors than its “big brother”.

Cachegrind is a cache proﬁler.

Massig is a heap proﬁler.

Helgrind is a thread debugger which ﬁnds data races in multithreaded programs.

There exist also a bunch of third-party plugins which we won’t consider in detail.

3.4.4 Example translation To give a better idea of what happens during recompilation, we translate a small basic block from x86 to UCode and back to x86. UCode is a simple RISC like register independent intermediate representation which can be easily augmented with instrumentation code. A basic block is a sequence of x86 code whose head is jumped to and which ends in a control ﬂow transfer such as a jump, call or return. Let’s assume we have these four lines of disassembled client code

Listing 3.1: Basic block in original x86 assembler code § ¤ 0 pushl %eax andl %eax, %ebx addl $0x3,%ecx jmp-8 0x8048307 ¦ ¥

Valgrind’s core will translate it to a sequence in UCode, where processor registers (here %EAX, %EBX, %ECX) are straightforward renamed to virtual registers (%t0, %t2, etc.). x86 instructions are mapped to one or more equivalent UCode instructions. PUT and GET move values in and out of virtual registers whereas INCEIP marks the original x86 instruction’s end (the argument indicates the length of the original x86 instruction). ST does a store. Each UCode instruction that affects the %eflags register is marked as such; the -wOSZACP sufﬁx indicates that the AND and ADD instruction updates all six ﬂags. 3.4 A closer look at the Valgrind Framework 27

Listing 3.2: To UCode translated basic block § ¤ 0 GETL %EAX, t0 ; pushl %eax GETL %ESP, t2 SUBL $0x4,t2 PUTL t2,%ESP STL t0,(t2) 5 INCEIPo $1 GETL %EBX, t4 ; andl %eax,%ebx GETL %EAX, t6 ANDL t6, t4 (-wOSZACP) PUTL t4,%EBX 10 INCEIPo $2 GETL %ECX, t10 ; addl $0x3, %ecx ADDL $0x3, t10 (-wOSZACP) PUTL t10,%ECX INCEIPo $3 15 JMPo $0x8048307 ; jmp-8 0x8048307 ¦ ¥

This UCode block is now optimized and passed to the Valgrind plugin which can add additional commands and update its meta data about memory state, cache, etc. and ﬁnally gives it back to the framework’s core. The instrumented UCode is translated back to x86 and executed. A large cache stores this recompiled basic block for future executions.

Listing 3.3: Register re-allocated UCode (using Nulgrind plugin, no instrumentation was added) § ¤ 0 GETL %EAX, %eax GETL %ESP, %ebx SUBL $0x4,%ebx PUTL %ebx, %ESP STL %eax, (%ebx) 5 INCEIPo $1 GETL %EBX, %ecx ANDL %eax, %ecx PUTL %ecx, %EBX INCEIPo $2 10 GETL %ECX, %edx ADDL $0x3,%edx (-wOSZACP) PUTL %edx, %ECX INCEIPo $3 JMPo $0x8048307 ¦ ¥

Listing 3.4: Final recompiled x86 code § ¤ 0 ; prologue movl 0x0(%ebp), %eax movl 0x10(%ebp), %ebx subl $0x4,%ebx movl %ebx, 0x10(%ebp) 5 movl %eax, (%ebx) movl $0x8048300, 0x24(%ebp) movl 0xC(%ebp), %ecx andl %eax, %ecx 28 Chapter 3: Dynamic binary analysis

movl %ecx, 0xC(%ebp) 10 movb $0x2, 0x24(%ebp) movl 0x4(%ebp), %edx addl $0x3,%edx movl %edx, 0x4(%ebp) movb $0x5, 0x24(%ebp) 15 pushfl ; popl 32(%ebp) movl $0x8048307, %eax movl %eax, 0x24(%ebp) call VG_(patch_me) ¦ ¥

Even if the plugin doesn’t add any instrumentation in our example, the code blows up by a factor of ﬁve. Caching and optimizations can compensate this overhead so that nil instrumentation slows down only by a factor of 1-2. Plugins like Memcheck which add a lot of instrumentation code and update meta data have a slow down factor of 20-30.

3.4.5 Example analysis To clarify which errors can be found with Valgrind, we present a small C program and list Valgrind’s analysis reponse. For the instrumentation we used the Memcheck plugin.

Listing 3.5: C sample program with illegal memory access § ¤ #include

int static_array[10];

5 int main(void) { int i; int* heap_array = malloc(10*sizeof(int));

10 for (i=0; i<=10; i++) { heap_array [i] = 0; // overrun when i==10 static_array[i] = 0; // overrun when i==10 } free(heap_array); 15 heap_array[0] = 0; // block has been freed } ¦ ¥

Compiled with all debugging information, this C program results in the following Valgrind output

... ==16884== Invalid write of size 4 ==16884== at 0x8048398: main (bad.c:11) ==16884== Address 0x40D1C040 is 0 bytes after the expected range, ==16884== the 40-byte heap block allocated ==16884== at 0x400216E7: malloc (vg_replace_malloc.c:161) ==16884== by 0x8048375: main (bad.c:8) ==16884== ==16884== Invalid write of size 4 3.4 A closer look at the Valgrind Framework 29

==16884== at 0x80483A2: main (bad.c:12) ==16884== Address 0x80495E8 is 0 bytes after the expected range, ==16884== a 40-byte static array in the main executable ==16884== ==16884== Invalid write of size 4 ==16884== at 0x80483C5: main (bad.c:15) ==16884== Address 0x40D1C018 is 0 bytes inside the once-expected range, ==16884== the 40-byte heap block freed ==16884== at 0x40021CE9: free (vg_replace_malloc.c:187) ==16884== by 0x80483BE: main (bad.c:14) ... As you can see, Valgrind found the two illegal array accesses (out of valid bound) on line 11 and 12 as well as the bad free on line 15. Unfortunately the quality of the feedback decreases if there is no debugging information available. The illegal access to the static array wouldn’t be found and the link between fault and C code would be missing too. Thus consider this example as an optimal case.

3.4.6 Limitations Besides Valgrind’s nice beneﬁts there is of course two sides to every coin. Some client code may lead Valgrind into problems that it cannot handle correctly. But there are also limitations which originate from a more architectural or “political” aspect.

Operating systems Valgrind runs under Linux on x86, AMD64 and PowerPC32 machines. There will probably never be a Windows version. The architectural differences between these operating systems are too big and could only be solved by rewriting Valgrind from scratch. As an example, Valgrind has to handle the clients system calls. On Linux there exist around 100 different ones. On Windows we have some thousands. Valgrind can’t trace into the kernel and thus has to know all system calls to execute them untranslated doing the following steps 1. Save Valgrinds stack pointer 2. Move the client’s state (without the program counter value) into the real processor registers 3. Delegate the system calls execution to the operating system kernel 4. After return from the call, Valgrind has to copy the client’s state (except the program counter) out to memory 5. Restore Valgrind’s stack pointer The system call will be performed directly on the client’s stack. All the same, some system calls like mmap() or open()/close() must be wrapped to prevent a damage of Valgrind’s memory or internal data. 30 Chapter 3: Dynamic binary analysis

Debugging information To dramatically increase Memcheck’s error report quality and get human readable output, debugging information and symbol tables in the client code are required. Other- wise it reports just memory addresses and - far more important - misses faulty instructions like bad memory access on static data areas, i.e. on the stack.

FP, MMX, SSE, etc. Valgrind can’t check other than integer arithmetic. All other special units like ﬂoating- point, MMX or SSE1/2/3 (also called SIMD for single instruction, multiple data) are performed directly on the CPU. Valgrind just records the change of the status bits. This approach keeps the handling of FP and SIMD instructions simple. However, it prevents plugins from instrumenting these instructions in meaningful ways. Chapter 4

Enspect

“When one admits that nothing is certain one must, I think, also admit that some things are much more nearly certain than others.” (Bertrand Russell) 32 Chapter 4: Enspect

4.1 Overview

One major limitation of AutoTest is the missing information about faults in external C code; the fault may also occur somewhere in a calling chain inside the feature under test. We present a general core-plugin based solution, named enspect (a mixture of Eif- fel and to inspect), which is able to use any available dynamic binary analysis (DBA) tool to gather additional data about the error. An implementation of the core application and a plugin for the DBA tool Memcheck of the Valgrind project is provided. Section 4.2 sets up some intendend results for enspect , section 4.3, 4.4 and 4.5 describe how the execution model and the architecture of enspect look like, whereas section 4.6 deals with an effective implementation and extensions added to the Au- toTest tool.

4.2 Intended results

4.2.1 Scope The intended results for our solution are tied directly to the abilities of the available DBA tools. In our case this will be Valgrind. Hence we focus on the memory and high-level errors described in the previous section. Valgrind’s Memcheck is able to trace them all except cache trashing (but the Cachegrind plugin of Valgrind does) and illegal stack access if debugging information is missing. Although some tools provide additional analysis such as threads, network access and others, we treat those cases as special and optional ones. They will probably be integrated in a future version of enspect .

4.2.2 Architecture The architecture of enspect should be designed as flexible as possible in relation to the used DBA tool. Although our prototype has Linux as a target platform, this limitation is only due to the use of Valgrind. With our flexible design, we are able to support any other platform where DBA tools exist (like Windows, Mac OS X or Solaris). Further, a version change of the underlying tool should not affect enspect , as well as the user should be independent from a special file organization or other technical requirements.

4.2.3 AutoTest integration enspect was projected from the beginning as extension to AutoTest and hasn’t been enforced to be integrated directly in the source code. There should rather exist two independent applications that communicate over a well defined interface. This makes it possible to start enspect from AutoTest as a separate process, feed it with input and handle its response while preventing at the same time AutoTest’s process from being 4.3 Execution model 33 blocked if enspect had to trace an error that would lead into an infinite loop or another non-responding state. During design phase we noted that some functionalities better fit into AutoTest itself than into enspect . Hence we have touched AutoTest’s source code all the same and added some new classes to the existing application (see 4.6.2).

4.2.4 Use cases To test enspect we built a set of Eiffel use cases containing common pitfalls in the external ''C'' clause code. We were able to trace these faults back to their origin and provide the programer with some additional debugging hints. In a last step, we applied enspect also to the real world framework Eiffelmedia [3].

4.3 Execution model

4.3.1 Normal execution Before we present our solution let’s recapitulate what happens when an application fault occurs. In ﬁgure 4.1 you can see the normal execution situation: the user or another process of the machine controls the application by giving new input and reacting on the application’s output. After a malfunction this communication is interrupted. But if we are unlucky, this malfunction manifests much later and the communication continues as if nothing had happened.

4.3.2 Execution under enspect Execution under enspect is slightly different and shown in ﬁgure 4.2. The base is also the operating system, but input and output with the application under test is au- tonomously done by enspect . Not a single user interaction is needed. The input must be provided as a kind of bug report in XML format. It can be created by hand or by another application. enspect runs the application under test (or a proxy for it like an interpreter) under Valgrind according to the information in the bug report. In this additional DBA layer, meta data about the applications like memory state or access is collected. Information about faults is constantly reported to enspect . This process runs down in a stepwise manner and ends with the bug instruction which terminates the application under test. If no step resulted in a bug, the application will be terminated all the same. The collected error information is analyzed and set in relation to the triggering Eiffel code (remember that the real error occured in an Eiffel external ''C'' clause) and ﬁnally provided as enspect output. 34 Chapter 4: Enspect

User, otherapplication

Input/Output

Application

OperatingSystem

Figure 4.1: Normal applications run directly on the operating system.

enspect aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaenspectaaaaaaaaaapluginaaaaaaaaaaaBugreportaaaaaaaaaaaaaaaaaaaaaaaa a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaInput/Outputaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaApplication(interpreter)aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaDBAaaaaaa toolaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

OperatingSystem

Figure 4.2: Applications traced by enspect run through a DBA tool like Valgrind. Input and output is automatically handled by enspect . For our AutoTest extension we use an interpreter instead of the real application. 4.4 Core - Plugin concept 35

4.4 Core - Plugin concept

In order to gain as much flexibility as possible, we have designed enspect as a core - plugin application which provides a common infrastructure (core) for automated dynamic binary analysis. Based on this core system, we have written a plugin for the Valgrind-AutoTest combination. The user is free to write his own plugin and profit from the standardized core infrastructure (see figure 4.3).

4.4.1 Core enspect ’s core deals only with the general tasks, like input handling or plugin management. In this section we present the provided functionalities.

Bug history file parser The input file of enspect is XML using the following simple XML definition

Listing 4.1: XML DTD deﬁnition for enspect’s input format § ¤ 0 5

]> ¦ ¥

We forbear from using XML Schema deﬁnition because of simplicity and ease to handle. We also leave it to the user to ensure that there exists only one bug event. A missing bug tag is interpreted as no to prevent code blowup. The element’s meaning is the following

enspect_storyline We call the whole bug report a storyline because all events before the bug instruction may have inﬂuence on the actual fault as well and hence are part of the whole “story”. This inﬂuence is usually exerted by unwanted side effects which invalidate the system’s invariant.

title A bug storyline can have a title, but this is optional.

description The description of the bug provides some additional details but remains unused in our current implementation.

history The history element is just a non empty sequence of instruction elements where at least one element must have the bug attribute set. Executing this special 36 Chapter 4: Enspect

instruction will result in the monitored fault. We named this sequence history because in normal case the isolated bug instruction won’t result in a failure. In normal cases, all preceding instructions play an important role too. Hence also all instructions are needed to understand the fault’s story. They build a whole bug history together. instruction An instruction is a single execution step in a sequence of instructions used by enspect ’s plugin to lead the application under test into the buggy state. The actual faulty instruction has to be marked with the attribute bug=''yes''.

The use of the instruction element allows a flexible description of a single execution step. Depending on the implemented plugin, this may be a command sent to the application’s STDIN file descriptor, but it may also be any other textual description of a (high-) level input command like a mouse movement (mouse_move (32, -8)) or a key press (key_down (``y'')). In our implementation of the AutoTest plugin we use the Erl-G interpreter language [23]. Listing 4.2 shows a valid input file. Notice that this sequence is probably not minimized to the instructions that are absolutely necessary for the bug event. At first sight the first three instructions seem independent from the actual bug instruction v_3.start. But instructions like the first one (create {ANY} v_1) cannot be dropped without being considered because unexpected side effects during v_1’s creation may influence the bug situation decisively. For enspect , we treat the input instruction sequence as already minimized and leave this task to the user or an external sequence minimizer.

Listing 4.2: Example input for enspect ’s AutoTest plugin § ¤ 0 Test case 1 Bad free in external C code 5 create {ANY} v_1 v_2 := Void :type v_2 create {EXTERNAL_C_TESTCASE} v_3 v_3.start 10 ¦ ¥

Environment The architecture of enspect contains two big parts (see ﬁgure 4.3). On the one hand, we have the application’s core system dealing with input and output aspects, error handling, logging or interprocess communication. On the other hand, there’s the enspect test environment where the bug under inspection will be executed and where the DBA tool’s process resides. 4.4 Core - Plugin concept 37

Core Testenvironment enspect DBA input input Plugin TestDriver

DBA tool Plugin DBA Analyzer output Application enspect undertest Plugin ReportWriter output

Figure 4.3: Architectural overview of enspect .

The plugin, we are going to describe in detail later, hooks into both parts. The environment and the core will instantiate the plugin’s components on demand according to the given command line option. The main task of the environment is to control the plugin’s test driver. The interface of the test driver is a small subset from DS_TRAVERSABLE and DS_CURSOR of the Gobo Eiffel Structure Library [4]. Only initialization, forward iteration and bound check need to be implemented by the plugin because the environment only needs to step through the bug’s history once in one direction (from the beginning to the bug instruction). The Eiffel inspect_bug feature in listing 4.3 uses this functionality. The inspect_bug feature performes not much more than an iteration over the instructions leading into the fault under supervision. The ﬁrst two statements in the feature are also of interest and we will consider them in the next section.

Listing 4.3: inspect_bug feature of the ESP_TEST_ENVIRONEMENT class. § ¤ 0 inspect_bug is -- collect meta data of the bug history and the bug itself require history_is_loaded: history_is_loaded history_not_void: history /= void 5 local factory: ESP_PLUGIN_FACTORY do create factory.make testdriver := factory.generate_test_driver (history) 10 -- execute the bug's history from testdriver.start until not testdriver.has_more_instructions 15 loop 38 Chapter 4: Enspect

testdriver.execute_instruction testdriver.forth end -- execute the bug instruction 20 testdriver.execute_bug_instruction bug_is_inspected := true ensure testdriver_not_void: testdriver /= void bug_is_inspected: bug_is_inspected 25 end ¦ ¥

Plugin factory Whenever a plugin of enspect is needed, we create an instance using the well known factory pattern. Real objects will be created only by the factory. It returns instances conforming to a well deﬁned class interface (these instances have in fact the type of a descendant of the interface class). The choice of which dynamic type will be created by the factory, depends on enspect ’s command line parameter --plug-in=<...> (the default plugin is auto_test). We describe the exact class interfaces in a next section and now have a closer look at the factory implementation itself. As you have seen in listing 4.3, the factory is of type ESP_PLUGIN_FACTORY and has a parameterless constructor make. It checks for the plugin name given by the user or takes the default value if it hasn’t been speciﬁed. After this setup, a call to the generate_xxx feature returns a corresponding plugin instance. Listing 4.4 lists the available generate features

Listing 4.4: Available generate features in ESP_PLUGIN_FACTORY. § ¤ 0 class ESP_PLUGIN_FACTORY

feature {ANY} -- generation of plugins

5 generate_test_driver (bug_history: ESP_BUG_HISTORY) : ESP_ABSTRACT_TEST_DRIVER is -- returns a instance of the selected plugin's test driver do -- ... 10 end

generate_analyzer (bug_history: ESP_BUG_HISTORY) : ESP_ABSTRACT_ANALYZER is -- returns a instance of the selected plugin's test driver 15 do -- ... end

generate_report_generator (bug_analysis: ESP_ABSTRACT_BUG_ANALYSIS) : 20 ESP_ABSTRACT_REPORT_GENERATOR is -- returns a instance of the selected plugin's test driver do -- ... end 25 4.4 Core - Plugin concept 39

end ¦ ¥

Global program settings

Besides the exact interface of the ABSTRACT_xxx classes, the choice for the dynamic plugin instance is another open question. In listing 4.4, no parameter about the plugin name is handed to the generate features. How does the factory know which type it must build? This information is stored in the global enspect settings. Therefore a special class named ESP_GLOBAL_SETTNGS exists. This class contains a bunch of once features, returning a application wide setting value like the command line options or an error handler. Every class which needs access to these settings inherits from the ESP_GLOBAL_SETTNGS class. The reason why we use once features is the following: the value of most of the settings depends on the command line options. Instead of parsing them with a special feature call, say parse_command_line, we perform this operation on demand. But only once. Otherwise every feature that uses the global setting would first need to check if the command line already has been parsed. If not, the parse_command_line has to be called, if yes, everything is fine and we can access the setting’s value. This is nasty style and blows up the require clause needlessly. A solution would be to parse the command line once during enspect ’s startup. But this makes the root class definite and contradicts the idea of flexible object oriented programming. To solve this problem, we implement the very nice idea of Bertrand Meyer’s article about global variables [26]. We have a special class, as already said we named it ESP_GLOBAL_SETTNGS, which parses the command line once, but on demand. As example, we have a setting plug_in_name (listing 4.5) which in fact is a once feature and accesses the command line in the first instruction. The command line variable itself is again a once feature. This makes sense because only one command line per application process exists.

Listing 4.5: plug_in_name feature of the ESP_GLOBAL_SETTINGS class. § ¤ 0 plug_in_name : STRING is -- name of the selected plug_in (default: "auto_test") require has_valid_argument_number: valid_argument_number local 5 value : STRING once value := cmd_line_parser.get_option_value ("plug-in") if value /= void then result := value.twin 10 else value := cmd_line_parser.get_option_value ("p") if value /= void then result := value.twin else 15 result := "auto_test" 40 Chapter 4: Enspect

end end ensure result_not_void: result /= void 20 end ¦ ¥

The cmd_line_parser feature is shown in listing 4.6. The ﬁrst time, this feature is called, the command line gets parsed by a call to the parse feature of ESP_COMMAND_LINE _PARSER. If a class needs access to the global settings, it simply inherits from ESP_GLOBAL _SETTINGS and uses the now available values without bothering wether the command already has been parsed or not.

Listing 4.6: cmd_line_parser feature of the ESP_GLOBAL_SETTINGS class. § ¤ 0 feature {NONE} -- Internal implementation

cmd_line_parser : ESP_COMMAND_LINE_PARSER is -- command line is parsed only once (on first access to a global value) once 5 create result.make result.parse ensure command_line_is_parsed: result.is_parsed end ¦ ¥

Error handler & Logﬁle Like global setting, some other global constructs are handled in the same way as described before. We use a so-called error handler for error and information reporting during execution. Instead of passing this object (and there is only one needed) as parameter to every module, we added it as an attribute to the global settings class (see listing 4.7) as well.

Listing 4.7: error_handler feature of the ESP_GLOBAL_SETTINGS class. § ¤ 0 error_handler : ESP_ERROR_HANDLER is -- create global error handler (singleton) once create result.make end ¦ ¥

Like the global settings, log ﬁle writing is based on the same idea. By inheriting from ESP_LOGGER, every class of enspect is able to write strings or whole lines to the application wide log ﬁle. First we planned to move the logging logic also into the ESP_GLOBAL_SETTINGS but then we gave up this approach for simplicity reasons. Other- wise the programmer would have to write a too long statement for a simple debugging output (logger.log() instead of just log()). 4.4 Core - Plugin concept 41

4.4.2 Deferred Plugin classes After this detailed description of enspect ’s core, we take a closer look at the plugin interfaces. As we said in the last section, the factory creates instances of plugin components which represent implementations of well defined interfaces. In this section we present the deferred classes that such a plugin has to implement. Keep in mind that the core of enspect is completely independent from a specific DBA tool and only the plugin defines the (limited) part of the application which may contain tool dependent code.

Test driver The test driver component of the plugin is as close to the DBA tool as no other. In fact, the test driver manages the whole input-output communication between enspect and the DBA tool. The class interface presented in listing 4.8 is similar to a container cursor with which you can iterate over data (in our case this will be a sequence of instructions). We have a start feature that (re)starts the whole iteration process, a forth feature to move the “cursor” and a has_more_instructions query to check if all instructions have been executed. An instruction can be executed with the execute_instruction feature and the special bug instruction straightforward with the execute_bug_instruction.

Listing 4.8: Deferred test driver component for enspect plugins. § ¤ 0 deferred class ESP_ABSTRACT_TEST_DRIVER

feature -- Creation

5 make (a_bug_history : ESP_BUG_HISTORY) is -- create a new test driver with the given bug_history require a_bug_history_not_void: a_bug_history /= void do 10 bug_history := a_bug_history ensure bug_history_not_void: bug_history /= void bug_history_set: bug_history = a_bug_history end 15 feature -- Access

start is -- sets testdriver to: no instruction executed yet 20 deferred end

forth is -- goes to next instruction 25 deferred end

execute_instruction is -- executes the next instruction of the bug_history 30 deferred 42 Chapter 4: Enspect

end

has_more_instructions : BOOLEAN is -- indicates if there are unexecuted instructions left 35 deferred end

execute_bug_instruction is -- executes the critical bug_instruction 40 deferred end

feature {NONE} -- Implementation

45 bug_history: ESP_BUG_HISTORY -- parsed instruction sequence and bug instruction

invariant bug_history_not_void: bug_history /= void 50 end ¦ ¥

Let’s throw a glance at the ESP_BUG_HISTORY class in order to understand the depen- dencies of this plugin interface class. As you can see in listing 4.9, the ESP_BUG_HISTORY is just a descendant of an array based list whose elements are of type ESP_BUG_HISTORY _INSTRUCTION. This is a storage class for DBA and application-under-test instructions or meta data together. Hence an implementer of ESP_ABSTRACT_TEST_DRIVER should fill in the corresponding data of output and error after the execution of an instruction under supervision and finally set the is_executed flag to true.

Listing 4.9: Classes ESP_BUG_HISTORY and ESP_BUG_HISTORY_INSTRUCTION. § ¤ 0 class ESP_BUG_HISTORY inherit DS_ARRAYED_LIST [ ESP_BUG_HISTORY_INSTRUCTION ] -- ... 5 end

class ESP_BUG_HISTORY_INSTRUCTION 10 -- ... setter/getter features

feature {NONE} -- Implementation

15 instruction : STRING -- instruction which will be sent to the interpreter

output : STRING -- output from the interpreter 20 error : STRING -- error message from the binary intrumentation tool (i.e. 'valgrind') -- empty sting if none occured

25 is_executed : BOOLEAN 4.4 Core - Plugin concept 43

-- indicates if the instruction has already been executed

end ¦ ¥

Analyzer The collected DBA output alone won’t lead the programmer to the source of the error. Although many DBA tools already provide some hints about the actual fault, this information is described in a language close to the machine code. In contrast, we are interested in the source of the error on the Eiffel side. Based on the low-level information from the DBA tool, we create the corresponding hint on the Eiffel side. For this task, an enspect plugin has to implement the deferred class ESP_ABSTRACT_ANALYZER. The very simple class contracts can be seen in listing 4.10, actually the class de- ﬁnes only the generate_bug_analysis feature which returns an analysis result of type ESP_ABSTRACT_BUG_ANALYSIS.

Listing 4.10: Classes ESP_BUG_HISTORY and ESP_BUG_HISTORY_INSTRUCTION. § ¤ 0 deferred class ESP_ABSTRACT_ANALYZER

feature -- Creation

5 make (a_bug_history : ESP_BUG_HISTORY) is -- initialize 'current' and store reference to the bug history deferred end

10 feature -- Access

generate_bug_analysis :ESP_ABSTRACT_BUG_ANALYSIS is -- analyzes valgrind's output and returns the result deferred 15 end

end ¦ ¥

The ESP_ABSTRACT_BUG_ANALYSIS is a data structure independent intermediate representation interface for the analyzed DBA result. Details can be found in the next section. Instances of this type can be passed to the report generator. We decided to use this intermediate step because one may have several different analysis results or different report generators per plugin.

Bug analysis

The ESP_ABSTRACT_BUG_ANALYSIS class is intentionally kept very simple. By limiting the interface to some container access features, the implementer is free to use any container and element structure he needs. This is very comfortable because most DBA tools will 44 Chapter 4: Enspect

allow different analysis, so the result will depend on the choice of the tool. The use of a list or an array would be too restrictive. Also, for custom containers, iterators are not always available, hence we let the implementer do the mapping to the interface features (see listing 4.11).

Listing 4.11: Classes ESP_BUG_HISTORY and ESP_BUG_HISTORY_INSTRUCTION. § ¤ 0 deferred class ESP_ABSTRACT_BUG_ANALYSIS

feature -- Element access

5 after : BOOLEAN is deferred end

item: ANY is 10 deferred end

forth is deferred 15 end

start is deferred end 20 end ¦ ¥

Report generator

Above we presented a light-weighted access interface for a generic bug analysis result. In order to get output readable for humans or another application, this analysis must be turned into a speciﬁc format like XML, HTML, PDF, etc.

Listing 4.12: The ESP_ABSTRACT_REPORT_GENERATOR creates human or machine readable bug report (i.e. enspect ’s output). § ¤ 0 deferred class ESP_ABSTRACT_REPORT_GENERATOR

generate_bug_report (a_bug_analysis : ESP_ABSTRACT_BUG_ANALYSIS) is -- generates an (output) report for 'a_bug_analysis' 5 deferred end end ¦ ¥ 4.5 Further design aspects 45

Messages

The last deferred class to implement is the ESP_ABSTRACT_MESSAGES. This is just an empty class ﬁle that can be ﬁlled with resources like strings or numerals used by the plugin. The idea is to give the programmer just the hint, not to use hardcoded values in its code. We won’t mention this any more because it seems obvious to us that a program (component) should always be implemented like a piece of art.

4.5 Further design aspects

In this section we want to present two further design and implementation aspects of enspect .

4.5.1 Structure of a bug situation

As already mentioned in the last section, the fault state of the application under test is described in a sequence of instructions where one special instruction fires the bug and results in an application error. We call the whole sequence, including the bug instruction, a storyline of the bug, the sequence before the error occurs is named the bug’s history and the key instruction for the bug is simply the bug instruction. If theinput of enspect (i.e. a counter example for the correctness of a feature) is not minimized, not every instruction necessarily influences the application’s state. For example, the last instruction in figure 4.4 has no effect on the state, thus is redundant and can be dropped unconsidered. But this is the task of an external counter example minimizer which could use a technique like Delta Debugging [40] or similar. If a non-redundant instruction is missing, the final state of the application is different and probably no bug will occur - if the bug instruction can be reached at all. After all, the cycle of the bug’s history has been broken by dropping a crucial element. Figure 4.4 illustrates the above terms in the whole storyline context.

4.5.2 Command line tool

In order to improve automation, we designed enspect as command line tool. An instance of enspect can thus be started easily as new process via the operating system’s exec() system or API call. Further, the tool needs less resources and is independent of graphical frameworks or widget libraries. All central behaviors can be controlled by an external application using enspect ’s command line options. 46 Chapter 4: Enspect

bughistory

instruction ... instruction instruction instruction buginstruction

Applicationstate

storyline

Figure 4.4: Terminology overview of a bug.

4.6 Example plugin for AutoTest

In this section we present an example plugin for AutoTest. It uses Valgrind and makes enspect to an extension of AutoTest, the automated testing tool of Andreas Leitner and Ilinca Ciupa [9]. Firstly, we describe how the the plugin executes the instructions of the enspect input ﬁle and how it fetches the DBA output. In the second part we describe how the plugin gains additional Eiffel related information about the fault. During this process we also did some re-engineering on the AutoTest tool and added some new functionalities.

4.6.1 Instruction execution AutoTest interpreter

The execution of the instructionsis not performed on a real application, but on a special interpreter process which is part of every AutoTest run. This interpreter offers access to all the Eiffel reflectable classes in the application’s universe. The language that the interpreter understands is described in [23]. All instructions of enspect ’s input file must be valid interpreter commands. Because our plugin is an extension to AutoTest, we assume the interpreter as existing. In fact, its file path must be handed to enspect as compulsory command line option. Keep the flexibility of our architecture in mind by looking at this example: the kind of application that is inspected using automated dynamic binary analysis is left completely open. It can be any executable like a real applications or a system interpreter. 4.6 Example plugin for AutoTest 47

It doesn’t even have to be written in Eiffel.

Valgrind As DBA tool we use Valgrind. To analyze an application, instead of a normal program startup like

$ program_name

you simply pass these values to valgrind

$ valgrind program_name

In order to gain some more ﬂexibility, you can tune Valgrind’s behavior with several command line options and ﬁnally get the following startup call structure

$ valgrind program_name

The effective call in the plugin is shown in listing 4.13

Listing 4.13: Valgrind can be tuned with several command line options. § ¤ 0 -- ...

valgrind: EPX_EXEC_PROCESS -- valgrind's external process

5 -- ...

create valgrind.make_capture_all ("valgrind", <<"--tool=memcheck", 10 "--xml=yes", "--suppressions="+enspect_files_path+"/"+core_messages.tool_name+".sup", interpreter_path >>)

-- ... ¦ ¥

Details about the --xml=yes option can be seen later. Bellow, we explain also the mechanism of suppressions and how we use it automated. The real execution is ﬁnally started with a valgrind.execute call.

Communication with Valgrind enspect communicates with Valgrind by using POSIX interprocess communication with the ePOSIX library. The plugin’s test driver implementation inherits from EPX_CURRENT _PROCESS and so gains access to enspect ’s own process. Then Valgrind is started as executable process EPX_EXEC_PROCESS as described above. Communcation between these two processes is performed over the standard ﬁle descriptors 48 Chapter 4: Enspect

STD_IN Commands to the Valgrind process are sent using the standard input ﬁle descriptor. All commands will be forwarded to the AutoTest interpreter.

STD_OUT The output of the AutoTest interpreter can be read from the standard output ﬁle descriptor. Because execution slows down due to the Valgrind overhead and because the valgrind process is autonomous, we use a non blocking interprocess communication line reader to fetch the output.

STD_ERR Valgrind’s analysis output can be read from the standard error ﬁle descriptor. As we will describe before, the format for data exchange will be XML.

To enable data capturing fromor sending to these ﬁle descriptors, we activate them with the valgrind.set_capture_error, valgrind.set_capture_input and valgrind.set_capture _output features. Otherwise no communication between the processes is possible.

Error suppressions The Eiffel runtime startup already produces some errors. For example, the runtime resets all system signal handlers. Also those who can’t be set like SIGKILL (which causes a process to terminate) or SIGSTOP (which causes a process to stop). Other library errors during startup (such as those from the libc) may confuse the user even more because they have nothing to do with the bug under supervision. Hence we use a special feature of Valgrind, called error suppression, to filter out these environment errors. In a first pass, we record all the runtime errors during startup and build a so-called suppression file. No instruction of the actual bug history will be executed. This dynamically created suppression file is then fed back to Valgrind in a second pass, where we really reproduce the faulty situation. Listing 4.14 shows you such a suppression file including some further explanations.

Listing 4.14: Automatically generated error suppression ﬁle for Valgrind. § ¤ 0 ##------## # # Errors to suppress for enspect tool (automatically generated file) # Format of this file is: # { 5 # name_of_suppression # tool_name:supp_kind # (optional extra info for some suppression types) # caller0 name, or /name/of/so/file.so # caller1 name, or ditto 10 # (optionally: caller2 name) # (optionally: caller3 name) # } # # For Memcheck, the supp_kinds are: 15 # # Param Value1 Value2 Value4 Value8 Value16 # Free Addr1 Addr2 Addr4 Addr8 Addr16 # Cond (previously known as Value0) # 4.6 Example plugin for AutoTest 49

20 # and the optional extra info is: # if Param: name of system call param # if Free: name of free-ing fn) # # see also: 25 # http://valgrind.org/docs/manual/manual-core.html#manual-core.suppress # ##------##

# enspect suppression 1 30 { enspect startup suppression 1 Memcheck:Cond obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 35 obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 40 obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so } # enspect suppression 2 { 45 enspect startup suppression 2 Memcheck:Cond obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 50 obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 55 obj:/lib/ld-2.3.5.so } # enspect suppression 3 { enspect startup suppression 3 60 Memcheck:Cond obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 65 obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 70 } # enspect suppression 4 { enspect startup suppression 4 Memcheck:Cond 75 obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 80 } # enspect suppression 5 { enspect startup suppression 5 Memcheck:Cond 50 Chapter 4: Enspect

85 obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 90 } # enspect suppression 6 { enspect startup suppression 6 Memcheck:Cond 95 obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 100 } # enspect suppression 7 { enspect startup suppression 7 Memcheck:Cond 105 obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 110 } # enspect suppression 8 { enspect startup suppression 8 Memcheck:Cond 115 obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so obj:/lib/ld-2.3.5.so 120 } ¦ ¥

Although the suppressions seem very similar, their structure differs and Valgrind knows how to handle them correctly. Of course it’s possible to drop the first pass and always use an already prepared suppression file, but this won’t guarantee that the suppressions are correct on every system and with every Eiffel runtime or loaded libraries during application startup. Also the first pass takes only some seconds and is therefore a negligible contribution to the whole execution time.

XML exchange format

As mentioned above, we use XML as an exchange format between enspect and the Valgrind process. Valgrind provides this well known and easy to process format as from version number 3.0 or above. It can be turned on with the command line option --xml=yes. Using the XML utilities of the GOBO library, access to the XML element tree also becomes very easy and comfortable. Thanks to this format, we are not depending on a special Valgrind version output, but really on the analysis’ structure. Also in a long term view, this stable exchange structure deﬁnitely makes sense. The structure of an error in Valgrind’s XML format can be found in listing 4.15. 4.6 Example plugin for AutoTest 51

Listing 4.15: Code snippet of a Valgrind error report in XML. § ¤ 0 0x12 1 InvalidFree Invalid free() / delete / delete[] 5 0x1B90144F /usr/lib/valgrind/vgpreload_memcheck.so free 10

/build/buildd/valgrind-3.0.1/coregrind/m_replacemalloc vg_replace_malloc.c 235 15 Address 0x87654321 is not stack'd, malloc'd or (recently) free'd 20 ¦ ¥

4.6.2 Analysis of bugs In this last last section about enspect , we present a mechanism to get out of the machine code based DBA analysis some Eiffel related debugging information.

Detectable sorts of memory nasties To ﬁnd out the exact error type of a system failure, we depend very much on Valgrind’s output. All meta data about the external fault is collected inside the Memcheck Val- grind plugin and hence we won’t be able to get any more information about the actual memory violation than Memcheck. That’s why we hand Valgrind’s guess for the error untranslated to the report writer. Keep in mind that despite considerable sophistication under the hood, Memcheck can only really detect two kinds of errors. First the use of illegal addresses and second the use of undeﬁned values. Nevertheless, this is enough to discover the listed sorts of memory management nasties.

1. Illegal read or write errors Output: Invalid read of size 4, Invalid write of size 8

2. Use of uninitialised values Output: Conditional jump or move depends on uninitialised value(s)

3. Illegal frees Output: Invalid free() 52 Chapter 4: Enspect

4. When a block is freed with an inappropriate deallocation function Output: Mismatched free() / delete / delete []

5. Passing system call parameters with inadequate read or write permissions Output: Syscall param write(buf) points to uninitialised byte(s) or Syscall param exit(error_code) contains uninitialised byte(s)

6. Overlapping source and destination blocks Output: Source and destination overlap in memcpy(0xbffff294, 0xbffff280, 21)

7. Memory leak detection Output: 88 (8 direct, 80 indirect) bytes in 1 blocks are definitely lost in loss record 13 of 14

For some errors, Valgrind is able to provide additional hints about the error (Address 0xBFFFF0E0 is not stack'd, malloc'd or free'd). At ﬁrst sight, all this information has little in common with the original Eiffel code. Hence we developed a link mechanism which is presented in the next section.

C to Eiffel stack mapping Together with the information from Valgrind, enspect provides a stack trace of the Eiffel side. Thanks to this information, the tester is able to ﬁnd the cause of the error in a (sub) feature call of a faulty Eiffel feature under test, i.e. in its external ''C'' clause. The example bellow illustrate this mapping from the tester’s point of view. The error was triggered by a faulty external ''C'' function in a small Eiffel application. A reason for the rather complex feature nesting is the use of the Eiffel Wrapper Generator [22] for an automated integration of external C code and its linking constructions. EWG builds for a C header ﬁle and it’s implementation wrapper classes on the Eiffel side. These wrapper classes provide Eiffel applications a clean access to the original C code. The automation is very useful if the C code interfaces change frequently and also the wrapper classes go out of date very easily, or if there are hundreds of C library functions to wrap. Without EWG this work would be very dull and error-prone. First consider the following C stack trace:

C stack trace of a bad free bug

STACK TRACE ======Function : free Location : @0x1B90144F Object : /usr/lib/valgrind/vgpreload_memcheck.so'

Directory : /build/buildd/valgrind-3.0.1/coregrind/m_replacemalloc File : vg_replace_malloc.c Line :235 ------4.6 Example plugin for AutoTest 53

Function : testcase Location : @0x846CEA3 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fac50pj Location : @0x8449D66 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fac7bkt Location : @0x8449ED7 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fcd3pfq Location : @0x80FC5BC Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fbjojz8 Location : @0x804D4DB Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fbjml6d Location : @0x804DA98 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fbi8dek Location : @0x84267D1 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fbjzjwv Location : @0x82971DA Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : _fAaax7_n Location : @0x8467C05 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fbjpuof Location : @0x83D145E Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fbjo6qt Location : @0x8424838 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ======

The C function names in the stack trace are generated automatically by the Eiffel compiler and thus are very cryptic (we use the ISE compiler ec). It’s near impossible to reconstruct the semantics of the original program ﬂow. After a mapping to Eiffel, the stack trace looks slightly different. Instead of functions, we have features and classes. Of course the object and instruction pointer of a stack frame remain unchanged. Note the gaps in the Eiffel mapping, highlighted with a WARNING message: in consequence of using AutoTest which dynamically creates new classes in the application’s universe, we are not able to map all functions back to Eiffel 1.

1AutoTest even creates its interpreter helper application on the ﬂy and compiles it before the ﬁrst test run. Details about this drawbacks can be found at the end of the chapter. 54 Chapter 4: Enspect

Mapped Eiffel stack trace in enspect

STACK TRACE ======Function : free WARNING No corresponding Eiffel feature found! Location : @0x1B90144F Object : /usr/lib/valgrind/vgpreload_memcheck.so'

Directory : /build/buildd/valgrind-3.0.1/coregrind/m_replacemalloc File : vg_replace_malloc.c Line :235 ------Function : testcase WARNING No corresponding Eiffel feature found! Location : @0x846CEA3 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Feature : testcase_external Class : TESTCASE_HEADER_FUNCTIONS_EXTERNAL Location : @0x8449D66 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Feature : make Class : EXTERNAL_C_TESTCASE_STARTER Location : @0x8449ED7 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fcd3pfq WARNING No corresponding Eiffel feature found! Location : @0x80FC5BC Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Feature : rout_obj_call_function Class : FUNCTION Location : @0x804D4DB Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Feature : apply Class : FUNCTION Location : @0x804DA98 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Feature : call Class : ROUTINE Location : @0x84267D1 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : Fbjzjwv WARNING No corresponding Eiffel feature found! Location : @0x82971DA Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Function : _fAaax7_n WARNING No corresponding Eiffel feature found! Location : @0x8467C05 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Feature : rout_obj_call_procedure Class : PROCEDURE Location : @0x83D145E Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ------Feature : apply Class : PROCEDURE 4.6 Example plugin for AutoTest 55

Location : @0x8424838 Object : /tmp/test_badfree/auto_test_gen/interpreter/EIFGEN/F_code/interpreter' ======

But even if some single stack frames can’t be mapped back to Eiffel, we think that a stack trace in Eiffel code offers a good approach to ﬁnd the real cause of the bug. Of course, extra information would help additionally. For example an external debugger can hook into Valgrind’s process and read out some values of Eiffel variables.

AutoTest extension The technique we use in enspect to map C code back to Eiffel in order to create the Eiffel stack trace is based on an extension we have written for AutoTest. Without this extension, the AutoTest interpreter knows nothing about it’s C code. With our extension it can, provided an Eiffel class and feature name, give the corresponding C function. The same works the other way around: given a C function name, the interpreter is able to provide its client the corresponding Eiffel class and feature (ET_BASE_TYPE and ET_FEATURE). Listing 4.16 shows the additional interpreter code.

Listing 4.16: Extension to AutoTest’s AUT_INTERPRETER_PROXY class. § ¤ 0 class AUT_INTERPRETER_PROXY

-- ...

feature {ANY} -- C <--> Eiffel Mapping 5 c_to_eiffel_table: AUT_C_EIFFEL_LOOKUP_TABLE -- mapping of C functions <-> Eiffel features

10 eiffel_feature_for_c_function (a_function_name: STRING): DS_LINKED_LIST [ DS_PAIR [ET_BASE_TYPE, ET_FEATURE] ] is 15 -- empty list if no feature exists require c_to_eiffel_not_void: c_to_eiffel /= void a_function_name_not_void: a_function_name /= void do 20 result := c_to_eiffel_table.eiffel_feature_for_c_function (a_function_name) ensure list_not_void: result /= void list_valid: not result.has (void) end 25

c_function_for_eiffel_feature (a_feature: ET_FEATURE; a_type: ET_BASE_TYPE): DS_LINKED_LIST [ 30 AUT_C_FUNCTION ] is -- empty list if no function exists require c_to_eiffel_not_void: c_to_eiffel /= void 56 Chapter 4: Enspect

35 a_type_not_void: a_type /= void a_feature_not_void: a_feature /= void do result := c_to_eiffel_table.c_function_for_eiffel_feature (a_feature, a_type) ensure 40 list_not_void: result /= void list_valid: not result.has (void) end

-- ... ¦ ¥

As you can see, the eiffel_feature_for_c_function not just returns the name of the Eiffel feature and class, but really some objects. Their types, ET_FEATURE and ET_BASE_TYPE, are part of the Eiffel Tools Library from the Eiffel Gobo project which allows a runtime access to the application’s universe. To understand how the two mapping directions in the AUT_C_EIFFEL_LOOKUP_TABLE class are calculated (when calling the eiffel_feature_for_c_function and c_function_for _eiffel_feature feature), consider figure 4.5. The illustration shows a processing step during ISE Eiffel compilation: when you freeze a system, the Eiffel compiler generates C code and compiles it 2. For every Eiffel feature there will be at least one C function. In order to speed up execution, ISE’s compiler generates sometimes more than one C function for a feature of a generic Eiffel class. The compiler stores all these mappings (in most systems around 5000 − 20000 entries) in a file called TRANSLAT in the EIFGEN subdirectory. The most important part of our AutoTest extension is the AUT_C_EIFFEL_LOOKUP_TABLE class. It parses the Eiffel to C translation file and provides services based on this information to AUT_INTERPRETER_PROXY. In order to speedup a lookup in these thousands of mappings, we use hash tables (one table per mapping direction). These tables are built on their first access in order not to waste memory. The parsing and building process takes just some milliseconds and hence is negligible. In return for a C function name, eiffel_feature_for_c_function hands back a list of possible Eiffel features together with their corresponding class (we return a list for a consistent access interface). If the list is empty, no mapping feature was found. c_function_for_eiffel_feature returns, given a Eiffel class and one of its features, a list of matching C functions of type AUT_C_FUNCTION. If there is no such function the list is empty. Remember that the mapping is bijective and there may be several C functions for an Eiffel feature. Listing 4.17 shows the primitive structure of the AUT_C_FUNCTION class.

2Freezing is a proprietary form of compilation of Eiffel Software’s Eiffel Studio and part of the Melting Ice Technology [37]. Freezing generates C code from the active system, and then compiles it into machine code. 4.6 Example plugin for AutoTest 57

. . : :

C_STRING,read_substring_into c_157.x,FanmwcuC2 C_STRING,read_string_into c_157.x,FannjagC2 C_STRING,share_from_pointer c_157.x,FankamcC2 LIST [REFERENCE],after li49.x,Fam9amvC1 LIST [REFERENCE],is_equal li49.x,Fam8no8C1 LIST [INTEGER],after li83.x,Fam9ansC1 LIST [INTEGER],is_equal li83.x,Fam8np5C1 LIST [NATURAL_8],after li115.x,Fam9aonC2 : : HASH_TABLE[INTEGER,REFERENCE],prune ha87.x, Fa1t1f4C1 HASH_TABLE[INTEGER,REFERENCE],occurrences ha87.x,Fa0d_p5C1 : : HASH_TABLE[INTEGER,REFERENCE],prune ha87.x, Fa2pu4aC1 HASH_TABLE[INTEGER,REFERENCE],remove ha87.x,Fa0ut1bC1

: : . . Figure 4.5: Mapping from Eiffel to C created by the Eiffel Software compiler. In order to speed up execution of generic classes this mapping is not bijective (for example class HASH_TABLE [INTEGER, REFERENCE], feature prune). Left side: Eiffel class and feature, Right side: C ﬁle and function.

Listing 4.17: Class AUT_C_FUNCTION which represents a C function. § ¤ 0 class AUT_C_FUNCTION

create make 5 feature {ANY} -- Initialization

make (a_name, a_filename: STRING) is -- Initialize `Current' 10 require a_name_not_void: a_name /= Void a_filename_not_void: a_filename /= Void do create name_attr.make_from_string (a_name) 15 create filename_attr.make_from_string (a_filename) end

feature -- Access 20 name : STRING is -- gives the C function name do 58 Chapter 4: Enspect

Result := name_attr.twin 25 end

filename : STRING is -- gives the C function file name do 30 Result := filename_attr.twin end

feature {NONE} -- Implementation 35 name_attr: STRING -- storage attibute

filename_attr: STRING 40 -- storage attibute

invariant name_attr_not_void: name_attr /= void filename_attr_not_void: filename_attr /= void 45 end ¦ ¥

Limitations

We already mentioned that our technique has a drawback which manifests in WARNING messages in the Eiffel stack trace. In this section we explain this problem in more detail and propose a solution. To understand the problem, we must look at in more detail how AutoTest works.

1. AutoTest parses the given (X)ACE ﬁle using the Gobo Eiffel Tools.

2. This results in an Eiffel Universe (ET_UNIVERSE) which provides access to every class of the system under test. 3. Given the universe, AutoTest builds additional Eiffel classes for testing using the reﬂection mechanism provided by Erl-G. These classes won’t be inserted in the existing universe (and that’s the critical point!). 4. The interpreter helper application is composed and compiled. The interpreter contains the classes under test as well as the newly created ones.

5. Compilation results in the TRANSLAT ﬁle that we parse in our extension class AUT_C_EIFFEL.

In order to access the AutoTest interpreter in enspect , we rebuild the universe and pass it to AutoTest’s interpreter. In fact, the interpreter is built by a factory which takes the universe as argument. All the same, we don’t create the dynamically built classes again. Hence they got lost because they aren’t in the (X)ACE ﬁle’s universe. We guess that this limitation will be eliminated in the next AutoTest release and the dynamic classes are injected in the application’s class universe before the interpreter 4.6 Example plugin for AutoTest 59 is built. Of course we could also return just the plain name of the feature and class (taken from the TRANSLAT ﬁle) in our AutoTest extension. But this won’t conform to a fairly long-term development process where the enspect response will be reused by AutoTest. Keep in mind, that some of the WARNING messages will never vanish because they are plain C functions and hence no equivalent Eiffel feature exists. Examples are the free or testcase function in the above Eiffel stack trace. 60 Chapter 4: Enspect Chapter 5

Results

“If your success is not on your own terms, if it looks good to the world but does not feel good in your heart, it is not success at all.” (Anna Quindlen) 62 Chapter 5: Results

In orther to test how enspect performs on bug tracking, we run it on several applications with faulty external ''C'' clauses. First we prepared some rather synthetic test cases, second we were also looking for bugs in the Eiffelmedia library.

5.1 Synthetic test cases

In this section we present the faulty C part of our16 tiny test applications and if enspect found a bug on this input. All these test cases were built with the Eiffel Wrapper Generator [22] and contain a C function with the signature void testcase (void). It’s called from Eiffel and always contains at least one bug.

5.1.1 Badfree

§ ¤ void testcase(void) { void* p = (void*)0x87654321;

5 /* Free a pointer to Never-Never Land */ free(p); } ¦ ¥

§ ¤ void testcase(void) { int q[] = { 1, 2, 3 };

5 /* Free a pointer to a stack block */ free(q); } ¦ ¥

5.1.2 Bad address value

§ ¤ void testcase(void) { char* aa = malloc(8);

5 /* invalid address value */ aa[-1] = 17;

if (aa[-1] == 17) printf("17\n"); else printf("not 17\n"); 10 } ¦ ¥ 5.1 Synthetic test cases 63

5.1.3 Badjump

§ ¤ void testcase(void) { char* p = (char*)0xE000000; int result = ((int(*)(void)) p) (); 5 } ¦ ¥

5.1.4 Badloop

§ ¤ void testcase(void) { int a[5]; int i, s; 5 /* skip index '2' and leave value uninitialized */ a[0] = a[1] = a[3] = a[4] = 0; s = 0; for (i = 0; i < 5; i++) s += a[i]; 10 if (s == 377) printf("sum is %d\n", s); return; } ¦ ¥

5.1.5 Bad read/write

§ ¤ void testcase(void) { void* x = malloc(10);

5 int *x4; short *x2; char *x1; int y4; short y2; 10 char y1;

x4 = x-4; x2 = x-4; x1 = x-1; 15 // Invalid reads and writes of sizes 4, 2, 1 y4 = *x4; *x4 = y4;

20 y2 = *x2; *x2 = y2;

y1 = *x1; *x1 = y1; 25 } ¦ ¥ 64 Chapter 5: Results

5.1.6 Bad C library calls to brk() and sbrk()

§ ¤ #include #include #include #include 5 #define EOL ((void*)( ~(long)0 ))

void testcase(void) { 10 int i; void* orig_ds = sbrk(0); // origin of data segment void* ds = orig_ds; void* vals[10]; void* res; 15 vals[0] = (void*)0; vals[1] = (void*)1; vals[2] = ds - 0x1; // small shrink vals[3] = ds; vals[4] = ds + 0x1000; // small growth 20 vals[5] = ds + 0x40000000; // too-big growth vals[6] = ds + 0x500; // shrink a little, but still above start size vals[7] = ds - 0x1; // shrink below start size vals[8] = EOL;

25 for (i = 0; EOL != vals[i]; i++) { /* use: '#define __NR_brk 45' from syscall.h */ res = (void*) syscall(__NR_brk, vals[i]); }

30 assert( 0 == brk(orig_ds) ); // libc brk()

for (i = 0; EOL != vals[i]; i++) { res = (void*)brk(vals[i]); } 35 } ¦ ¥

5.1.7 Double free

§ ¤ void testcase(void) { int i; void* p = malloc(177); 5 for (i = 0; i < 2; i++) free(p); return; } ¦ ¥ 5.1 Synthetic test cases 65

5.1.8 Custom allocators

See the source code package or Valgrind’s vg_replace_malloc.c ﬁle for more details.

§ ¤ void testcase(void) { int* array; int* array3; 5 array = custom_alloc( sizeof(int) * 10 ); array[8] = 8; array[9] = 8; array[10] = 10; // invalid write (ok w/o MALLOCLIKE -- in superblock) 10 custom_free(array); // ok

custom_free(NULL); // invalid free (ok without MALLOCLIKE)

15 array3 = malloc(sizeof(int) * 10); custom_free(array3); // mismatched free (ok without MALLOCLIKE)

make_leak();

20 int result; result = array[0]; // use after free (ok without MALLOCLIKE/MAKE_NOACCESS)

// leak from make_leak() } ¦ ¥

5.1.9 File descriptors

§ ¤ #include #include #include #include 5 void testcase(void) { int fd, n; char buf[10]; 10 fd = open("foo/bar/xyzzy", O_RDONLY); // fails printf("fd = %d\n", fd); n = read ( fd, buf, 10 ); // illegal fd printf ("n = %d\n", n); return; 15 } ¦ ¥ 66 Chapter 5: Results

5.1.10 Read/write using pointers

§ ¤ void testcase(void) { volatile double d; volatile float f; 5 double* dp = malloc(sizeof(double)); float* fp = malloc(sizeof(float)); int* ip = (int*)0x1234567;

d += 1.0; 10 f += 10.0; *dp += ( d ? 2.0 : 3.0 ); *fp += ( f ? 20.0 : 21.0 );

free(dp); 15 free(fp);

*dp += 3.0; //error: mem already freed *fp += 30.0;

20 free(ip); //error: illegal addr ip = malloc(sizeof(int));

* ((double*)ip) = 1.2 + d; //error: illegal cast 4<->8 bytes } ¦ ¥

5.1.11 Leak check This test case shows incompetent memory management. It simulates what happens when we try to manage a dense branchy cyclic structure with reference counting. This code is based on an example from the Valgrind authors.

§ ¤ #include #include "memcheck.h"

#define ITER 100000 5 #define N 1000

static int bytes, blocks; struct node { struct node *l, *r; int ref; }; static void ref(struct node *n) { if (n) n->ref++; } 10 static void unref(struct node *n) { if (n && --n->ref == 0) { bytes -= sizeof(*n); blocks--; free(n); } }

15 static struct node *alloc() { struct node *n = malloc(sizeof(*n)); n->l = n->r = NULL; n->ref = 0; bytes += sizeof(*n); blocks++; 20 return n; }

static void assign(struct node **np, struct node *n) { 5.1 Synthetic test cases 67

ref(n); unref(*np); *np = n; 25 }

static int rrange(int range) { long long ret = rand() * (long long)range; return (int)(ret / RAND_MAX); 30 }

static struct node *nodes[N];

static struct node *mk() { 35 int idx = rrange(N); struct node *n; /* 50% recycle, 25% alloc new, 25% return NULL */ if (rand() > RAND_MAX/2) { n = nodes[idx]; /* recycle existing block */ 40 } else if (rand() > RAND_MAX/2) { n = alloc(); /* alloc new block */ assign(&n->l, mk()); assign(&n->r, mk()); } else { n = NULL; /* no block */ 45 } assign(&nodes[idx], n); return n; }

50 void testcase(void) { int i; long base_definite, base_dubious, base_reachable, base_suppressed; long definite, dubious, reachable, suppressed; 55 int total;

/* we require these longs to have same size as a machine word */ assert(sizeof(long) == sizeof(void*));

60 /* get a baseline in case the runtime allocated some memory */ VALGRIND_DO_LEAK_CHECK; base_definite = base_dubious = base_reachable = base_suppressed = 0; VALGRIND_COUNT_LEAKS ( base_definite, base_dubious, 65 base_reachable, base_suppressed);

for(i = 0; i < ITER; i++) { mk(); 70 if ((i % (ITER/10)) == 0) { if (0) printf("%d living blocks, %d bytes\n", blocks, bytes); VALGRIND_DO_LEAK_CHECK; } 75 }

/* "free all memory" */ for(i = 0; i < N; i++) assign(&nodes[i], NULL); 80 if (0) printf("FINISHED: %d living blocks, %d bytes\n", blocks, bytes); VALGRIND_DO_LEAK_CHECK;

85 /* Shouldn't be necessary, but COUNT_LEAKS doesn't define its result values */ definite = dubious = reachable = suppressed = 0; VALGRIND_COUNT_LEAKS(definite, dubious, reachable, suppressed); 68 Chapter 5: Results

definite -= base_definite; 90 dubious -= base_dubious; reachable -= base_reachable; suppressed -= base_suppressed;

total = definite+dubious+reachable+suppressed; 95 if (0) printf("leaks: definite %d, dubious %d, reachable %d, suppressed %d = %d\n", (int)definite, (int)dubious, (int)reachable, (int)suppressed, total);

100 if (reachable != 0) printf("FAILED: I freed everything, but there's still %d bytes reachable\n", (int)reachable); else if (total != bytes) printf("FAILED: I count %d bytes, leakcheck says %d\n", 105 bytes, total); else printf("PASS\n"); } ¦ ¥

5.1.12 Too long function match

§ ¤ //too long function match

static int a1234567890123456789012345678901234567890123456789012345678901234567890 5 1234567890123456789012345678901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 10 1234567890123456789012345678901234567890123456789012345678901234567890 123456789012345678901234567890123456789012345678901234567890123456789 (void) { return *(int *)0; } 15 void testcase(void) { exit( a1234567890123456789012345678901234567890123456789012345678901234567890 20 1234567890123456789012345678901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 1234567890123456789012345678901234567890123456789012345678901234567890 25 1234567890123456789012345678901234567890123456789012345678901234567890 123456789012345678901234567890123456789012345678901234567890123456789() ); } ¦ ¥ 5.1 Synthetic test cases 69

5.1.13 Overlapping src and dst

§ ¤ char b[50];

void reset_b(void) { int i; 5 for (i = 0; i < 50; i++) b[i] = '_'; b[49] = '\0'; }

void reset_b2(void) { 10 reset_b(); strcpy(b, "ABCDEFG"); }

void testcase(void) { 15 char x[100]; char a[] = "abcdefghijklmnopqrstuvwxyz"; int i;

/* testing memcpy/strcpy overlap */ 20 for (i = 0; i < 50; i++) { // don't put any zeroes in there x[i] = i+1; } 25 for (i = 50; i < 100; i++) { // because of the errors, the strcpy's will overrun, so put some // zeroes in the second half to stop them eventually x[i] = 0; } 30 memcpy(x+20, x, 20); // ok memcpy(x+20, x, 21); // overlap memcpy(x, x+20, 20); // ok memcpy(x, x+20, 21); // overlap 35 strncpy(x+20, x, 20); // ok strncpy(x+20, x, 21); // overlap strncpy(x, x+20, 20); // ok strncpy(x, x+20, 21); // overlap 40 x[39] = '\0'; strcpy(x, x+20); // ok

x[39] = 39; 45 x[40] = '\0'; strcpy(x, x+20); // overlap

x[19] = '\0'; strcpy(x+20, x); // ok 50 } ¦ ¥ 70 Chapter 5: Results

5.1.14 Allocator errors

§ ¤ void testcase ( void ) { int i; char* p = malloc(10); 5 for (i = 0; i < 10; i++) p[i] = 'z'; free(p); p[1] = 'z'; // already free'd p = malloc(10); 10 p[2] = 'z'; p[-1] = 'z'; // illegal address } ¦ ¥

§ ¤ /* Illegal allocation args */ void testcase ( void ) { char* p; 5 /* returns a null pointer or a unique pointer that can be passed to free() */ p = malloc(0); printf("malloc(0) = %p\n", p); free(p); 10 p = malloc(-1); // illegal printf("malloc(-1) = %p\n", p); free(p);

15 /* returns a null pointer or a unique pointer value that can be passed to free() */ p = calloc(0,1); printf("calloc(0,1) = %p\n", p); free(p);

20 p = calloc(0,-1); // illegal printf("calloc(0,-1) = %p\n", p); free(p);

p = calloc(-1,-1); // illegal 25 printf("calloc(-1,-1) = %p\n", p); free(p); } ¦ ¥

5.1.15 Buffer overﬂow

§ ¤ void testcase(void) { char buffer[10]; char* input = "A too long string"; 5 strcpy(buffer, input); } ¦ ¥ 5.2 Real World testcase in Eiffel Media 71

5.1.16 NULL pointers

§ ¤ typedef struct { int x, y; } Point;

void testcase(void) { 5 Point* p1 = NULL; Point* p2 = NULL;

p1 = (Point*) cmalloc(sizeof(Point)); p1->x = 5; 10 p1->y = 72;

p2->x = 24; //NULL pointer access p2->y = 42; } ¦ ¥

5.1.17 Results The following table lists which faults we were able to trace and which not. The debugging hints we constructed were enough to locate the actual cause of the bug. Only one test case remained undetected by enspect (the illegal brk() and sbrk() calls). We checked if the plain Valgrind tool is able to detect this fault, but also Valgrind’s Mem- check plugin didn’t succeed. Thus this limitation is caused rather by the underlying DBA tool than by enspect itself. Table 5.1 sums up our results.

5.2 Real World testcase in Eiffel Media

For a real world test, we were looking for faults in the Eiffelmedia library - in fact, in its external ''C'' clauses. First we analyzed the official sample applications with AutoTest. Even though we ran the tool over days and produced some gigabytes of logging data, we haven’t found any bug. We guess that the reason is in the nature of Eiffelmedia: most features need a rather complex infrastructure to work on. For example to draw on the screen, a surface has to be prepared and the screen has to be initialized. To communicate over a network, a connection has to be established first. Such tasks are too complex to be performed by AutoTest and its current random input generator strategy. A solution to this problem would be to parametrize the infrastructure setup, say by a feature setup(arg1 : TYPE1; arg2: TYPE2; ... ) that returns a object of type EM_TEST _INFRASTRUCTURE. A feature under test (featXY) is then wrapped by another feature (featXY_wrapper) which takes an infrastructure instance as first argument. This is similar to the Current reference in an object oriented function call. With this mechanism, we could ensure that a feature call will always be performed on a valid infrastructure. Unfortunately, there wasn’t enough time to implement and also test this idea. Thus we can’t provide real world test results of enspect . But with our results of the synthetic 72 Chapter 5: Results

Table 5.1: Test results of the synthetic test cases.

Test case Bug detected Bad free Yes Bad address value Yes Bad jump Yes Bad read/write Yes Bad C library calls to brk() and sbrk() No Double free Yes Custom allocators Yes File descriptors Yes Read/write using pointers Yes Leak check Yes Too long function match Yes Overlapping src and dst Yes Allocator errors Yes Buffer overﬂow Yes NULL pointers Yes

test cases, we educe that bugs in real world applications should be traced correctly by enspect too. Chapter 6

Conclusion and Outlook

“I never think of the future - it comes soon enough.” (Albert Einstein) 74 Chapter 6: Conclusion and Outlook

6.1 Conclusion

In this thesis we have shown that it is possible to automatically gather additional debugging information about faults in external ''C'' clauses of Eiffel programs. Un- certainty about the real cause of the bug remains because there are no contracts for automated testing available in C and C++. When already compiled third party C libraries are wrapped and used by Eiffel programs, the C source code is missing too. Even though we successfully used a more low-level approach called dynamic binary analysis (DBA). Based on a set of reference test cases, we evaluated several DBA tools for their ability to locate the errors. Robustness, performance, license and platform independence were further aspects we took into account. We chose the supervision framework Valgrind to be the best solution for our intention to build an extension for AutoTest, the automated (push-button) testing tool from ETH Zürich. We designed a ﬂexible core-plugin architecture as a proof-of-concept implementation with the name enspect . This architecture makes it possible to exchange the underlying DBA tool by simply writing a new plugin and has no inﬂuence on the application itself. We implemented the core framework of enspect and a plugin for the AutoTest/Valgrind combination and showed that 15 out of 16 types of faults in our test suite could be found. In a re-engineering process we moved some functionalities of our enspect plugin to the AutoTest tool. This extension provides a lookup mechanism for C function and Eiffel feature mappings.

6.2 Limitations

Even though 15 out of 16 error categories could be traced by our proof-of-concept implementation enspect , there remain several limitations. Our choice for a DBA tool, Valgrind, is only available under Linux. There are other solutions that work also under Microsoft Windows or Apple’s OS X. Because we have implemented only the AutoTest/Valgrind plugin, we weren’t able to check for platform specific problems in enspect . Further our extension to AutoTest works fine but is based on Eiffel Software’s compiler, i.e. its Eiffel-to-C TRANSLAT file generated when freezing an Eiffel system. Thus the SmartEiffel and Visual Eiffel compilers are not supported. At least in SmartEiffel we couldn’t find accessible and comparable compiler information. enspect ’s input is required to be an already minimized counter example for the correctness of the Eiffel code. The current version of AutoTest doesn’t provide such counter examples but they can be extracted by hand from its interpreter log file. This missing link prevents us from creating a fully automated testing process of faults in external ''C'' clauses. However, there will be further attempts to solve this problem by an intermediate data exchange format for AutoTest helper applications. 6.3 Further work & Outlook 75

Last but not least, our enspect implementation doesn’t use a time-out mechanism to prevent the system under test and enspect ’s client from ending up in an endless loop. In our opinion, the ﬁxing of a concrete time value is rather the task of the client application (such as AutoTest) than the one of enspect .

6.3 Further work & Outlook

As mentioned above, the integration of enspect in AutoTest should be improved. An intermediate data exchange format and a enspect plugin which works also on Windows will be the obvious first steps. But also a refinement of the debugging information is possible. There may be other possibilities to find out more about the Eiffel code than a stack trace. For example, Val- grind has a built-in support for the GNU debugger gdb [13] which allows to gain more information about the value of a certain memory address. But also here, a platform and compiler independent solution would be optimal. 76 Chapter 6: Conclusion and Outlook Bibliography

[1] Erwin Andreasen. Leaktracer. http://www.andreasen.org/LeakTracer. Consulted in September 2005.

[2] IEEE Standards Association. Ieee standard 1003.1-2001 posix. http://standards.ieee.org/regauth/posix/. Consulted in February 2006.

[3] Till Bay and Michela Pedroni. Eiffelmedia. http://eiffelmedia.origo.ethz.ch. Consulted in February 2006.

[4] Eric Bezault and Others. Gobo eiffel tools and libraries. http://www.gobosoft.com/. Consulted in October 2005.

[5] Armin Biere. ccmalloc. http://www.inf.ethz.ch/personal/biere/projects/ccmalloc. Consulted in September 2005.

[6] Hans Boehm. A garbage collector for c and c++. http://www.hpl.hp.com/personal/Hans_Boehm/gc. Consulted in September 2005.

[7] Chandrasekhar Boyapati, Sarfraz Khurshid, and Darko Marinov. Korat: Automated testing based on java predicates. http://www.cag.lcs.mit.edu/∼marinov/. Consulted in September 2005.

[8] Ilinca Ciupa. Teststudio: An environement for automatic test generation based on design by contract. Master’s thesis, ETH Zürich, Switzerland, July 2004. http://se.inf.ethz.ch/projects/ilinca_ciupa/teststudio/.

[9] Ilinca Ciupa and Andreas Leitner. Autotest. http://se.inf.ethz.ch/people/leitner/auto_test/. Consulted in September 2005.

[10] Berend de Boer. e-posix: The deﬁnitive and complete eiffel to standard c and posix 1003.1 binding. http://www.berenddeboer.net/eposix/. Consulted in November 2005.

[11] The Mozilla Foundation. Bugzilla. http://www.bugzilla.org. Consulted in February 2006. 78 BIBLIOGRAPHY

[12] The Mozilla Foundation. Mozilla project: Gecko testcases. http://www.mozilla.org/newlayout/testcases/. Consulted in February 2006.

[13] Free Software Fundation. Gdb: The gnu project debugger. http://www.gnu.org/software/gdb/. Consulted in December 2005.

[14] Erich Gamma and Kent Beck. Junit. http://www.junit.org. Consulted in February 2006.

[15] Belgium Ghent University. Diota. http://www.elis.ugent.be/diota. Consulted in September 2005.

[16] Nicole Greber. Testwizard: Automatic test generation based on design by contract. Master’s thesis, ETH Zürich, Switzerland, January 2003. http://se.inf.ethz.ch/projects/nicole_greber/.

[17] HP and MIT. Dynamorio. http://www.cag.lcs.mit.edu/dynamorio. Consulted in September 2005.

[18] IBM. Purify(plus). http://www-306.ibm.com/software/awdtools/purify/. Consulted in September 2005.

[19] Intel and University of Colorado. Pin. http://rogue.colorado.edu/Pin. Consulted in September 2005.

[20] Ecma International. Standard ecma-367, eiffel analysis, design and programming language. http://www.ecma-international.org/publications/standards/Ecma-367.htm, June 2005. Consulted in February 2006.

[21] Gary Leaven. Jml. http://www.cs.iastate.edu/∼leavens/JML/. Consulted in February 2006.

[22] Andreas Leitner. Eiffel wrapper generator (ewg). http://ewg.sourceforge.net. Consulted in October 2005.

[23] Andreas Leitner. Erl-g: Language understood by the erl-g interpreter. http://se.inf.ethz.ch/people/leitner/erl_g/interpreter_language.html. Consulted in October 2005.

[24] Andreas Leitner. Erl-g: The eiffel reﬂection library generator. http://se.inf.ethz.ch/people/leitner/erl_g/. Consulted in October 2005.

[25] Andreas Leitner. Strategies to automatically test eiffel programs. Master’s thesis, Technische Universität Graz, Austria, December 2004. http://www.ist.tugraz.at/verify/view/Projects/TestStudio. BIBLIOGRAPHY 79

[26] Bertrand Meyer. Bidding farewell to globals. Journal of Object-Oriented Programming (JOOP), 1(1), 1988. ֓←/http://archive.eiffel.com/doc/manuals/technology/bmarticles joop/globals.html. [27] Bertrand Meyer. Object-Oriented Software Construction. Prentice Hall, 2nd edition edition, 1997.

[28] Microsoft. Vulcan: Binary transformation in a distributed environment. Consulted in September 2005.

[29] Misc. Academic publications written by valgrind developers. http://www.valgrind.org/docs/pubs.html. Consulted in September 2005.

[30] Misc. Simple directmedia layer (sdl). http://www.libsdl.org. Consulted in February 2006. [31] Nicholas Nethercote and Julian Seward. Valgrind, a suite of tools for debugging and proﬁling linux programs. http://www.valgrind.org. Consulted in September 2005.

[32] The GNU Project. Gcc, the gnu compiler collection, front ends for c, c++, objective-c, fortran, java and ada, as well as libraries for these languages (libstdc++, libgcj, etc). http://www.gnu.org/software/gcc/gcc.html. Consulted in September 2005. [33] The GNU Project. The gnu concurrent versions system. http://www.nongnu.org/cvs/. Consulted in September 2005.

[34] Fridtjof Siebert. Implementierung eines eiffel-compilers für sun/sparc. Master’s thesis, Universität Stuttgart, Fakultät Informatik, May 1997.

[35] Eiffel Software. Eiffel compiler. http://docs.eiffel.com/eiffelstudio/tools/eiffelstudio/reference/30_compiler/. Consulted in September 2005.

[36] Eiffel Software. Eiffelstudio - a complete integrated development environment for eiffel. http://www.eiffel.com. Consulted in September 2005.

[37] Eiffel Software. Melting ice technology. http://docs.eiffel.com/eiffelstudio/tools/eiffelstudio/reference/30_compiler/. Consulted in September 2005.

[38] Wikipedia. Software testing. http://en.wikipedia.org/wiki/Software_testing. Consulted in February 2006.

[39] Wikipedia. Turing’s halting problem. http://en.wikipedia.org/wiki/Halting_problem. Consulted in February 2006. 80 BIBLIOGRAPHY

[40] Andreas Zeller. Delta debugging. http://www.st.cs.uni-sb.de/dd/. Consulted in February 2006. Appendix A

Source Code 82 Appendix A. Source Code

We list only two classes of our source code in this thesis appendix. They are of central interest for the mapping from C to back to Eiffel. ESP_AUTO_TEST_ANALYZER is taken from the AutoTest plugin of enspect whereas AUT_C_EIFFEL_LOOKUP_TABLE is from our AutoTest extension and performs the actual C function ↔ Eiffel feature mapping. For all details see the released source code package.

A.1 enspect: ESP_AUTO_TEST_ANALYZER

§ ¤ 0 indexing

description:

"'enspect' plugin: implementation for auto_test_plugin's analyzer" 5 " (uses auto_test's interpreter to gain access to C <-> Eiffel mapping" -- -- XML structure is (example): -- root ::= -- 10 -- -- 0x13 -- 1 -- InvalidFree -- Invalid free() / delete / delete[] 15 -- -- -- 0x1B90144F -- /usr/lib/valgrind/vgpreload_memcheck.so -- free 20 --

/build/buildd/valgrind-3.0.1/coregrind/m_replacemalloc -- vg_replace_malloc.c -- 235 -- -- 25 -- ... -- -- ... -- -- 30 -- -- -- ... -- -- ... 35 --

class ESP_AUTO_TEST_ANALYZER

inherit 45 ESP_ABSTRACT_ANALYZER redefine generate_bug_analysis end

50 ESP_GLOBAL_SETTINGS -- access to global settings redefine plugin_messages end 55 ESP_LOGGER -- logging ability undefine plugin_messages 60 end

create make

65 feature -- Creation A.1 enspect: ESP_AUTO_TEST_ANALYZER 83

make (a_bug_history : ESP_BUG_HISTORY) is -- analyzes the given bug history 70 require else a_bug_history_not_void: a_bug_history /= void do create analysis.make_from_bug_history (a_bug_history) create tree_callbacks.make 75 create parser_factory error_handler.report_info_message (plugin_messages.universe_build_start) create auto_test_interpreter.make error_handler.report_info_message (plugin_messages.universe_build_end) end 80

feature -- Execution

85 generate_bug_analysis : ESP_ABSTRACT_BUG_ANALYSIS is -- performes a bug analysis and returns the result local cursor: DS_ARRAYED_LIST_CURSOR [ ESP_AUTO_TEST_BUG_ANALYSIS_ELEMENT ] entry: ESP_AUTO_TEST_BUG_ANALYSIS_ELEMENT 90 do -- Analyse meta data from cursor := analysis.new_cursor cursor.start 95 until cursor.after loop entry := cursor.item -- has this instruction got a valgrind output 100 if not entry.error_data.is_empty then if count_errors (entry.error_data) > 0 then entry.debugging_hint.copy ( analyse_error_data (entry) ) end end 105 cursor.forth end -- error instruction if count_errors (analysis.bug.error_data) > 0 then analysis.bug.debugging_hint.copy ( analyse_error_data (analysis.bug) ) 110 end result := analysis ensure then result_set: result = analysis end 115

count_errors (xml: STRING) : INTEGER is -- returns how many errors have been occured local 120 document: XM_DOCUMENT elements: DS_LIST [XM_ELEMENT] cursor: DS_LIST_CURSOR [XM_ELEMENT] root, element: XM_ELEMENT do 125 document := parse_valgrind_output (xml) root := document.root_element elements := root.elements if not elements.is_empty then -- has tags 130 if root.has_element_by_name ("error") then cursor := elements.new_cursor from cursor.start until 135 cursor.after loop element := cursor.item if element.name.is_equal("error") then -- error tag found 140 result := result + 1 end cursor.forth end -- loop else 145 end end ensure tree_callbacks_void: tree_callbacks = void parser_void: parser = void 150 end

155 84 Appendix A. Source Code

get_stack_trace (a_xml_error_stack: XM_ELEMENT): DS_LINKED_LIST [ESP_AUTO_TEST_STACK_FRAME] is -- get the stack trace of the stack as linked list -- first list element = top of the stack 160 -- last element = bottom of the stack (error root -> eiffel feature) require a_xml_error_stack_not_void: a_xml_error_stack /= void a_xml_error_stack_not_empty: not a_xml_error_stack.elements.is_empty local 165 frames: DS_LIST [XM_ELEMENT] frame: XM_ELEMENT cursor: DS_LIST_CURSOR [XM_ELEMENT] stack_frame: ESP_AUTO_TEST_STACK_FRAME do 170 frames := a_xml_error_stack.elements from cursor := frames.new_cursor cursor.start create result.make_default 175 until cursor.after loop frame := cursor.item create stack_frame.make ( frame.element_by_name ("ip").text, 180 frame.element_by_name ("obj").text, frame.element_by_name ("fn").text ) if frame.has_element_by_name ("dir") then stack_frame.directory_name.copy (frame.element_by_name("dir").text) end 185 if frame.has_element_by_name ("file") then stack_frame.file_name.copy (frame.element_by_name("file").text) end if frame.has_element_by_name ("line") then stack_frame.source_code_line.copy (frame.element_by_name("line").text) 190 end result.force_last (stack_frame) cursor.forth end ensure 195 result_not_void: result /= void result_not_empty: not result.is_empty end

200 analyse_error_data (a_element : ESP_AUTO_TEST_BUG_ANALYSIS_ELEMENT) : STRING is -- get's a hint about the error (i.e. eiffel class/feature name) require a_element_not_void: a_element /= void a_element_error_data_not_void: a_element.error_data /= void 205 a_element_error_data_not_empty: not a_element.error_data.is_empty a_element_contains_errors: count_errors (a_element.error_data) > 0 local document: XM_DOCUMENT elements: DS_LIST [XM_ELEMENT] 210 cursor: DS_LIST_CURSOR [XM_ELEMENT] root, stack, element: XM_ELEMENT stack_trace, eiffel_stack_trace: DS_LINKED_LIST [ESP_AUTO_TEST_STACK_FRAME] error_number: INTEGER do 215 document := parse_valgrind_output (a_element.error_data) root := document.root_element elements := root.elements from cursor := elements.new_cursor 220 cursor.start create result.make_empty until cursor.after loop 225 element := cursor.item if element.name.is_equal("error") then -- error tag found error_number := error_number + 1 if error_number > 1 then 230 -- not the first error -> padding result.append ("%N%N") end stack := element.element_by_name ("stack") stack_trace := get_stack_trace (stack) 235 result.append (plugin_messages.stack_trace_c + " " + error_number.out + ":%N%N") result.append ( output_c_format (stack_trace) ) eiffel_stack_trace := convert_to_eiffel (stack_trace) result.append ("%N") result.append (plugin_messages.stack_trace_eiffel + " " + error_number.out + ":%N%N") 240 result.append ( output_eiffel_format (eiffel_stack_trace) ) end cursor.forth end -- loop A.1 enspect: ESP_AUTO_TEST_ANALYZER 85

ensure 245 result_not_void: result /= void end

output_c_format (stack: DS_LINKED_LIST [ESP_AUTO_TEST_STACK_FRAME]) : STRING is 250 -- renders a stack trace to user friendly output format require stack_not_void: stack /= void stack_not_empty: not stack.is_empty local 255 cursor: DS_LINKED_LIST_CURSOR [ESP_AUTO_TEST_STACK_FRAME] optional_output: STRING do create result.make_from_string ("STACK TRACE%N======%N") from 260 cursor := stack.new_cursor cursor.start until cursor.after loop 265 result.append ("Function : " +cursor.item.function_name+"%N") result.append ("Location : @"+cursor.item.instruction_pointer+"%N") result.append ("Object : " +cursor.item.object_path+"'%N") create optional_output.make_empty if not cursor.item.directory_name.is_empty then 270 optional_output.append ("%NDirectory : " + cursor.item.directory_name) end if not cursor.item.file_name.is_empty then optional_output.append ("%NFile : " + cursor.item.file_name) end 275 if not cursor.item.source_code_line.is_empty then optional_output.append ("%NLine : " + cursor.item.source_code_line) end if not optional_output.is_empty then result.append (optional_output + "%N") 280 end result.append ("------%N") cursor.forth end ensure 285 result_not_void: result /= void end

output_eiffel_format (stack: DS_LINKED_LIST [ESP_AUTO_TEST_STACK_FRAME]) : STRING is 290 -- renders a stack trace to user friendly output format require stack_not_void: stack /= void stack_not_empty: not stack.is_empty local 295 cursor: DS_LINKED_LIST_CURSOR [ESP_AUTO_TEST_STACK_FRAME] optional_output: STRING do create result.make_from_string ("STACK TRACE%N======%N") from 300 cursor := stack.new_cursor cursor.start until cursor.after loop 305 if cursor.item.feature_name.is_empty then -- use C information result.append ("Function : " +cursor.item.function_name+"%N") result.append (" WARNING : No corresponding Eiffel feature found!%N") result.append ("Location : @"+cursor.item.instruction_pointer+"%N") 310 result.append ("Object : " +cursor.item.object_path+"'%N") else -- use Eiffel information result.append ("Feature : " +cursor.item.feature_name+"%N") if not cursor.item.class_name.is_empty then 315 result.append ("Class : " + cursor.item.class_name+"%N") end result.append ("Location : @"+cursor.item.instruction_pointer+"%N") result.append ("Object : " +cursor.item.object_path+"'%N") end 320 create optional_output.make_empty if not cursor.item.directory_name.is_empty then optional_output.append ("%NDirectory : " + cursor.item.directory_name) end if not cursor.item.file_name.is_empty then 325 optional_output.append ("%NFile : " + cursor.item.file_name) end if not cursor.item.source_code_line.is_empty then optional_output.append ("%NLine : " + cursor.item.source_code_line) end 330 if not optional_output.is_empty then result.append (optional_output + "%N") 86 Appendix A. Source Code

end result.append ("------%N") cursor.forth 335 end ensure result_not_void: result /= void end

340 convert_to_eiffel (stack: DS_LINKED_LIST [ESP_AUTO_TEST_STACK_FRAME]) : DS_LINKED_LIST [ESP_AUTO_TEST_STACK_FRAME] is -- converts a stack trace to an eiffel stack trace (i.e. replace function names) local 345 cursor: DS_LINKED_LIST_CURSOR [ESP_AUTO_TEST_STACK_FRAME] mapping_list: DS_LINKED_LIST [DS_PAIR [ET_BASE_TYPE, ET_FEATURE]] entry: DS_PAIR [ET_BASE_TYPE, ET_FEATURE] do create result.make_from_linear (stack) 350 from cursor := result.new_cursor cursor.start until cursor.after 355 loop mapping_list := auto_test_interpreter.map_c_to_eiffel (cursor.item.function_name) if mapping_list.count > 0 then if not mapping_list.is_empty then -- take first mapping 360 entry := mapping_list.first cursor.item.class_name.copy (entry.first.name.name) cursor.item.feature_name.copy (entry.second.name.name) end end 365 cursor.forth end ensure result_not_void: result /= void end 370

feature -- XML stuff

parse_valgrind_output (data : STRING) : XM_DOCUMENT is 375 -- parses a xml string to an xml document require data_not_void: data /= void local root: XM_ELEMENT 380 do parser.parse_from_string ("%N" + data + "%N") if tree_callbacks.error.has_error then error_handler.report_error_message ("Bad valgrind output:%N " + tree_callbacks.last_error) error_handler.report_error_message ("Input was:%N" + data) 385 exceptions.die (error_numbers.invalid_valgrind_output) end result := tree_callbacks.document root := result.root_element if not root.is_root_node then 390 -- register as official root node result.set_root_element (root) end ensure result_not_void: result /= void 395 end

parser: XM_PARSER is -- XML parser do 400 if parser_factory.is_expat_parser_available then result := parser_factory.new_expat_parser else create {XM_EIFFEL_PARSER} result.make end 405 result.set_callbacks (tree_callbacks.start) end

410 feature -- AutoTest interpreter

auto_test_interpreter : ESP_AUTO_TEST_INTERPRETER -- access to mapping using auto_tests interpreter proxy

415

feature -- Attributes

analysis: ESP_AUTO_TEST_BUG_ANALYSIS A.2 AutoTest: AUT_C_EIFFEL_LOOKUP_TABLE 87

420 -- created analysis report

plugin_messages : ESP_AUTO_TEST_MESSAGES is -- create global message pool for plugin component (singleton) 425 once create result.make ensure then result_not_void: result /= void end 430 tree_callbacks: XM_TREE_CALLBACKS_PIPE -- Tree generating callbacks

parser_factory: XM_EXPAT_PARSER_FACTORY 435 -- Expat XML parser factory

invariant analysis_not_void: analysis /= void 440 tree_callbacks_not_void: tree_callbacks /= void parser_factory_not_void: parser_factory /= void auto_test_interpreter_not_void: auto_test_interpreter /= void end ¦ ¥

A.2 AutoTest: AUT_C_EIFFEL_LOOKUP_TABLE

§ ¤ 0 indexing description:

"performs a mapping: C function <-> Eiffel feature"

10 class AUT_C_EIFFEL_LOOKUP_TABLE

inherit KL_SHARED_EXCEPTIONS 15 -- for 'die()'

creation make

20 feature {ANY} -- Creation

make (a_interpreter_path: STRING; a_universe: ET_UNIVERSE; a_error_handler: AUT_ERROR_HANDLER) is -- Init a mapping, path is the path to the interpreters TRANSLAT -- The System must be built already (EIFGEN directory must exist) 25 require a_interpreter_path_not_void: a_interpreter_path /= void a_universe_not_void: a_universe /= void a_error_handler_not_void: a_error_handler /= void local 30 file: KL_TEXT_INPUT_FILE read_error: UT_CANNOT_READ_FILE_ERROR do -- store reference to auto_test's universe and error_handler universe := a_universe 35 error_handler := a_error_handler -- copy path create translat_filename.make_from_string (a_interpreter_path) -- try to open the file create file.make (translat_filename) 40 file.open_read if not file.is_open_read then create read_error.make (translat_filename) error_handler.report_error (read_error) exceptions.die (1) 45 end -- clean up file.close ensure translat_filename_not_void: translat_filename /= void 50 error_handler_not_void: error_handler /= void 88 Appendix A. Source Code

end

parse_mapping (a_line: STRING): AUT_C_EIFFEL_MAPPING is 55 -- Initialize a mapping from a line of the 'TRANSLAT' file -- -- Line ::= Eiffel_Cluster TAB Eiffel_Class TAB Eiffel_Feature TAB C_Function TAB C_File -- -- PS: You may also parse the string linearly by hand until the next TAB occurs. 60 -- require a_line_not_void: a_line /= void a_line_not_empty: not a_line.is_empty local 65 i: INTEGER k,l : INTEGER line: STRING do create result.make 70 create line.make_from_string (a_line) -- *** cluster *** i := line.index_of ('%T', 1) if i <= 1 then error_handler.report_error_message (eiffel_c_mapping_file_error) 75 error_handler.report_error_message (" '" + line + "'") exceptions.die (1) end result.eiffel_cluster.copy (line.substring (1,i-1) ) line.remove_head (i) 80 line.prune_all_leading('%T')

-- *** class *** i := line.index_of ('[', 1) if i = 0 then 85 -- not generic -- parse... -- only class name i := line.index_of ('%T', 1) if not (i > 1) then 90 error_handler.report_error_message (eiffel_c_mapping_file_error) error_handler.report_error_message (" '" + line + "'") exceptions.die (1) end result.eiffel_class.copy (line.substring (1,i-1) ) 95 line.remove_head (i) else -- generic class: "CLASSNAME [ G list ]" -- parse... -- (a) class name 100 -- (b) list of generic parameters result.make_eiffel_class_generic -- (a) class name check i > 2 end result.eiffel_class.copy (line.substring (1,i-2) ) 105 line.remove_head (i) from k := 1 until k > line.count or else k > line.index_of(']' ,1) 110 loop l := line.index_of(',', k) if l = 0 then -- only one generic parameter left result.eiffel_class_generic_parameters.force_last( line.substring(k,line.index_of(']',k)-1) ) 115 -- abort loop k := line.count + 1 else -- parse parameter and advance check k > 1 and l 0 end line.remove_head (line.index_of (']', 1)) end line.prune_all_leading('%T') 130 -- *** eiffel feature *** i := line.index_of ('%T', 1) if i <= 1 then error_handler.report_error_message (eiffel_c_mapping_file_error) 135 error_handler.report_error_message (" '" + line + "'") exceptions.die (1) end result.eiffel_feature.copy (line.substring (1,i-1) ) A.2 AutoTest: AUT_C_EIFFEL_LOOKUP_TABLE 89

line.remove_head (i) 140 line.prune_all_leading('%T')

-- *** c_function *** i := line.index_of ('%T', 1) if i <= 1 then 145 error_handler.report_error_message (eiffel_c_mapping_file_error) error_handler.report_error_message (" '" + line + "'") exceptions.die (1) end result.c_function.copy (line.substring (1,i-1) ) 150 line.remove_head (i) line.prune_all_leading('%T')

-- *** c_file *** i := line.index_of ('%T', 1) 155 if i > 0 then -- there should be no TAB left error_handler.report_error_message (eiffel_c_mapping_file_error) error_handler.report_error_message (" '" + line + "'") exceptions.die (1) 160 end result.c_file.copy ( line ) ensure result_not_void: result /= void end 165

feature {ANY} -- Access

170 eiffel_feature_for_c_function (a_function_name: STRING): DS_LINKED_LIST [ DS_PAIR [ET_BASE_TYPE, ET_FEATURE]] is -- returns a eiffel type and feature corresponding to a given C function -- empty list -> no class in universe found require 175 a_function_name_not_void: a_function_name /= void local mapping_list: DS_LINKED_LIST [AUT_C_EIFFEL_MAPPING] mapping_cs: DS_LINKED_LIST_CURSOR [AUT_C_EIFFEL_MAPPING] mapping: AUT_C_EIFFEL_MAPPING 180 -- all_classes : DS_HASH_TABLE [ET_CLASS, ET_CLASS_NAME] class_: ET_CLASS feature_: ET_FEATURE feature_name: ET_IDENTIFIER 185 i: INTEGER do create result.make_default mapping_list := c_to_eiffel (a_function_name) -- is there a mapping for this c function? 190 if mapping_list /= void then -- look build class/feature for every mapping entry from -- get all classes all_classes := universe.classes 195 mapping_cs := mapping_list.new_cursor mapping_cs.start until mapping_cs.after loop 200 mapping := mapping_cs.item class_ := universe.class_by_name (mapping.eiffel_class) if class_ = void then -- retry: check for Erl-G class name which are created dynamically by AutoTest -- and aren't present in the Eiffel universe, yet. 205 -- "ERL_TYPE_IMP_18_EXTERNAL_C_TESTCASE_STARTER" --> "ERL_TYPE" i := a_function_name.substring_index ("ERL_TYPE", 1) if i = 1 then -- "ERL_TYPE" in the beginning class_ := universe.class_by_name ("ERL_TYPE") 210 else -- check for implementation class -- "ERL_MY_CLASS_NAME_IMP" --> "ERL_MY_CLASS_NAME" i := a_function_name.substring_index ("_IMP", 1) if i > 1 then 215 class_ := universe.class_by_name (mapping.eiffel_class.substring (1,i-1)) end end end -- check also for corresponding class feature 220 if class_ /= void then create feature_name.make (mapping.eiffel_feature) feature_ := class_.named_feature (feature_name) if feature_ /= void then -- add to result list 225 result.force_last ( create {DS_PAIR [ET_BASE_TYPE, ET_FEATURE]}.make (class_, feature_) ) else 90 Appendix A. Source Code

end end mapping_cs.forth 230 end end ensure result_not_void: result /= void end 235

c_function_for_eiffel_feature (a_feature: ET_FEATURE; a_type: ET_BASE_TYPE): AUT_C_FUNCTION is -- returns a c function (and its file name) corresponding to a given Eiffel type/feature 240 -- void -> not found require a_feature_not_void: a_feature /= void a_type_not_void: a_type /= void local 245 mapping_list: DS_LINKED_LIST [AUT_C_EIFFEL_MAPPING] mapping: AUT_C_EIFFEL_MAPPING do mapping_list := eiffel_to_c (a_type.name.name, a_feature.name.name) if not mapping_list.is_empty then 250 mapping := mapping_list.first create result.make (mapping.c_function, mapping.c_file) end end

255 feature {NONE} -- Internal access

c_to_eiffel (c_function: STRING) : DS_LINKED_LIST [AUT_C_EIFFEL_MAPPING] is -- get's the corresponding eiffel feature to 'c_function' 260 require c_function_not_void: c_function /= void do if table_by_c_function = void then table_by_c_function := build_table_by_c_function 265 end if table_by_c_function.has (c_function) then result := table_by_c_function.item (c_function) else result := void 270 end end

eiffel_to_c (eiffel_class, eiffel_feature: STRING) : DS_LINKED_LIST [AUT_C_EIFFEL_MAPPING] is 275 -- get's the corresponding C function to 'eiffel_feature' of 'eiffel_class' -- may be more than one C require eiffel_class_not_void: eiffel_class /= void eiffel_feature_not_void: eiffel_feature /= void 280 do if table_by_eiffel_class_feature = void then table_by_eiffel_class_feature := build_table_by_eiffel_class_feature end if table_by_eiffel_class_feature.has (eiffel_class+"::"+eiffel_feature) then 285 result := table_by_eiffel_class_feature.item (eiffel_class+"::"+eiffel_feature) else result := void end end 290

feature -- Table insertion

insert_into_hash_table ( 295 table : DS_HASH_TABLE [DS_LINKED_LIST [AUT_C_EIFFEL_MAPPING], STRING]; entry : AUT_C_EIFFEL_MAPPING; key : STRING) is -- insert 'entry' with 'key' into 'table', the hash table may have several entries for 'key' -> list require 300 table_not_void: table /= void entry_not_void: entry /= void local list: DS_LINKED_LIST [AUT_C_EIFFEL_MAPPING] do 305 if table.has (key) then -- already in table -> append to list list := table.item (key) list.force_last (entry) else 310 -- first entry -> create list and append create list.make_default list.force_last (entry) table.force (list, key) end A.2 AutoTest: AUT_C_EIFFEL_LOOKUP_TABLE 91

315 end

feature {NONE} -- Implementation

320 build_table_by_c_function: DS_HASH_TABLE [DS_LINKED_LIST [AUT_C_EIFFEL_MAPPING], STRING] is -- stores the "C -> EIFFEL" mappings local entry: AUT_C_EIFFEL_MAPPING filename: STRING 325 file: KL_TEXT_INPUT_FILE read_error: UT_CANNOT_READ_FILE_ERROR do error_handler.report_info_message (build_c_eiffel_mapping) create file.make (translat_filename) 330 file.open_read if not file.is_open_read then create read_error.make (filename) error_handler.report_error (read_error) exceptions.die (1) 335 end -- read entries from create result.make(1024) file.read_line 340 until file.end_of_file loop -- create new entry entry := parse_mapping(file.last_string) 345 -- insert entry c->eiffel table -- make hash key not changeable --> twin it insert_into_hash_table (result, entry, entry.c_function.twin) -- next line file.read_line 350 end -- loop file.close error_handler.report_info_message (build_c_eiffel_mapping_done) ensure result_not_void: result /= void 355 end

build_table_by_eiffel_class_feature: DS_HASH_TABLE [DS_LINKED_LIST [AUT_C_EIFFEL_MAPPING], STRING] is -- stores the "EIFFEL -> C" mappings (EIFFEL_HASH_STRING = CLASSNAME + "::" + FUNCTION_NAME) 360 local entry: AUT_C_EIFFEL_MAPPING filename, tmp: STRING file: KL_TEXT_INPUT_FILE read_error: UT_CANNOT_READ_FILE_ERROR 365 do error_handler.report_info_message (build_eiffel_c_mapping) create file.make (translat_filename) file.open_read if not file.is_open_read then 370 create read_error.make (filename) error_handler.report_error (read_error) exceptions.die (1) end from 375 create result.make(1024) file.read_line create tmp.make_empty until file.end_of_file 380 loop -- create new entry entry := parse_mapping (file.last_string) tmp.wipe_out tmp.append (entry.eiffel_class + "::" + entry.eiffel_feature) 385 -- make hash key not changeable --> twin it insert_into_hash_table (result, entry, tmp.twin) -- next line file.read_line end -- loop 390 file.close error_handler.report_info_message (build_eiffel_c_mapping_done) ensure result_not_void: result /= void end 395

table_by_c_function: DS_HASH_TABLE [DS_LINKED_LIST [AUT_C_EIFFEL_MAPPING], STRING] -- stores the C -> Eiffel mappings

400 table_by_eiffel_class_feature: DS_HASH_TABLE [DS_LINKED_LIST [AUT_C_EIFFEL_MAPPING], STRING] -- stores the C -> Eiffel mappings 92 Appendix A. Source Code

translat_filename: STRING -- (absolut) path to the TRANSLAT file 405 universe: ET_UNIVERSE -- reference to Eiffel universe

error_handler: AUT_ERROR_HANDLER 410 -- auto_test's error handler

invalid_c_eiffel_translation_file : STRING is do 415 result := "Two identical entries in the C <-> Eiffel mapping file (TRANSLAT):" end

build_c_eiffel_mapping : STRING is do 420 result := "Building C function to Eiffel feature mapping..." end build_c_eiffel_mapping_done : STRING is do result := "--> C function to Eiffel feature mapping built successfully." 425 end

build_eiffel_c_mapping : STRING is do result := "Building Eiffel feature to C function mapping..." 430 end build_eiffel_c_mapping_done : STRING is do result := "--> Eiffel feature to C function mapping built successfully." end 435 eiffel_c_mapping_file_error : STRING is do result := "Invalid C <-> Eiffel mapping file structure (TRANSLAT):" end 440 invariant universe_not_void: universe /= void error_handler_not_void: error_handler /= void

445 translat_filename_not_void: translat_filename /= void translat_filename_not_empty: not translat_filename.is_empty

table_by_c_function_not_void: table_by_c_function /= void table_by_eiffel_class_feature_not_void: table_by_eiffel_class_feature /= void 450 table_by_c_function_not_void_implies_not_empty: table_by_c_function /= void implies not table_by_c_function.is_empty table_by_eiffel_class_feature_not_void_implies_not_empty: table_by_eiffel_class_feature /= void implies not table_by_eiffel_class_feature.is_empty 455 end ¦ ¥ Appendix B

Documentation 94 Appendix B. Documentation

B.1 Requirements enspect should run under most current Linux distribution and requires the following tools if it’s compiled from the source:

• ISE Eiffel 5.6 or above → http://www.eiffel.com

• Gobo Eiffel Library → http://www.gobosoft.com/eiffel/gobo/index.html

• ePOSIX 2.2 or above → http://www.berenddeboer.net/eposix/

The enspect core framework should also run under Windows, though we haven’t tested it.

B.2 Installation notes

1. Preparation Install and if necessary compile the required tools from section B.1. Extract the source code of enspect from the archive package to a separated writable directory of your choice.

2. Environment variables Make sure you have set the enspect environment variables. Add the following lines to your terminal configuration file (here, enspect ’s source files are stored in /home/ghioldir/enspect).

export $ENSPECT=/home/ghioldir/enspect export $PATH=PATH:$ENSPECT/bin

3. Compilation Prepare and install enspect with Gobo Eiffel Ant by entering the commands bellow in your terminal

cd $ENSPECT geant install geant compile

If you would like to compile enspect in Eiffel Studio, you can build an ACE ﬁle in the bin subdirectory by calling B.3 Usage 95

cd $ENSPECT geant install geant build_ace_ise cd bin estudio &

B.3 Usage usage: enspect [--help] [--version] [--verbose | -v] [--output-file=<...> | -o=<...>] [--plug-in =<...> | -p=<...>] ace_file interpreter / executable input file

To use enspect, follow the above calling convention. The XML input ﬁle format is described in listing 4.1.

The output of a (verbosed) enspect run would look like the following listing.

Initialize system...

Parse '/home/ghioldir/enspect/example/auto_test/bug_history.xml' ... --> Input file successfully parsed. Collecting meta data for a bug analysis...

Phase 1 - Generate suppression file Execute 'create {ANY} dummy_object' Generate suppression file... 8 error suppressions have been recorded. --> Suppression file generated! Execute ':quit'

Phase 2 - Real bug execution Execute 'create {ANY} v_1' Execute 'v_2 := Void' Execute ':type v_2' Execute 'create {EXTERNAL_C_TESTCASE_STARTER} v_3.make' Execute ':quit' 96 Appendix B. Documentation

--> Meta data about bug has been collected.

Analyse collected meta data... Build Eiffel universe and AutoTest interpreter... --> Universe and AutoTest interpreter created successfully! --> Meta data analyzed.

Generate report... --> Report generated. Appendix C

Project Plan 98 Appendix C. Project Plan

C.1 Project Description

C.1.1 Overview Eiffel Test Studio is a tool for fully automatic testing of Eiffel classes, based on the contracts present in them [8]. However, external code (e.g. external C/C++ code) is usually not equipped with contracts. This means that often the manifestation of a bug occurs far from the actual location of the bug in the software and there is no indication of the type of error that occured. There are tools such as memory proﬁlers or memory leak tracers that are able to detect some types of these software faults.

C.1.2 Scope of the work The aim of this project is to research if and how such a tool can be integrated into Test Studio to provide information about faults occurring in external code.

C.1.3 Intended results After the project’s ending the following should be provided: • an analysis of theoretical and technical aspects of possible approaches.

• requirements for external testing.

• a list of third-party tools able to fulﬁll these requirements.

• an implementation written in Eiffel/C.

• a set of use and test cases.

• a lasting link concept between Eiffel Test Studio and possible third-party analysis tools.

• a documentation and API description.

• an extensive technical report of all the results.

C.2 Background Material

C.2.1 Reading list Object-Oriented Software Construction Several chapters from the well known book Object-Oriented Software Construction [27], in particular: C.2 Background Material 99

• Chapter 1: Software quality

• Chapter 8: The run-time structure: objects

• Chapter 9: Memory management

• Chapter 11: Design by Contract: building reliable software

• Chapter 12: When the contract is broken: exception handling

• Chapter 17: Typing

• Chapter 28: The software construction process

• Chapter 34: Emulating object technology in non-O-O environments

• Chapter 35: Simula to Java and beyond: major O-O languages and environments

Valgrind publications The Valgrind developers [29] have written some interesting academic publications like

• Bounds-Checking Entire Programs Without Recompiling by Nicholas Nether- cote and Jeremy Fitzhardinge [2004].

• Dynamic Binary Analysis and Instrumentation by Nicholas Nethercote [2004].

• Valgrind: A Program Supervision Framework by Nicholas Nethercote and Julian Seward [2003]

C.2.2 Tools Various tools will be used during the project time. The following incomplete list provides a shortened overall view

• Eiffel Studio 5.6 [36]

• Eiffel Test Studio 1.0+ [8]

• Valgrind 3.01+ (and other binary analysis tools) [31]

• GNU Compiler Collection (GCC) 3.4+ [32]

• ReadHat Linux

• LATEX 100 Appendix C. Project Plan

C.3 Project Management

C.3.1 Objectives and priorities

Objective Priority Careful project schedule 1 Test requirements and tool selection which meets them 1 Technique to detect real bugs in external code 1 Generic adapter classes from TestStudio to an external tool 3 Platform independence (related to the binary analysis tool) 3 Set of use cases and test classes 2 API description and further documentation 2 Technical report 1

C.3.2 Criteria for success

With a proper integrated binary analysis tool and in connection with Eiffel Test Studio, it should be possible to detect bugs in external code. The test output should support the programer with comments and hints to crystallize the real bugs in extrenal code if there is any ambiguity. The new feature has to be integrated in the existing Eiffel Test Studio in a seamless and ﬂexible way. The performance penalty of the tool integration is reduced to a minimum.

C.3.3 Method of work

As development platform, Read Hat Linux with Eiffel Studio 5.6 will be used. The focus of the binary analysis tool will be on Valgrind [31]. For the source code exchange a concurrent versions system CVS [33] will be used.

C.3.4 Quality management

Weekly meetings and a careful schedule should help to reach the project goals and result in high quality code.

Documentation

The documentation will be provided by a technical report, as well as an API description of the implementation and a carefully commented code in the sense of design by contract [27]. C.4 Plan with Milestones 101

Validation steps

A set of test classes and use cases will be developed. The implemented features should be able to detect more and more of these constructed programming errors. The number of detected bugs and the tools output quality (comments and hints) can be used as a quality measure.

C.4 Plan with Milestones

C.4.1 Project steps

After an analysis of the current system and the state of the art techniques for testing external code, a proof of concept implementation will be written. With such a base, the implementation should become as platform independant as possible and shouldn’t rely heavily on a third-party tool. Last but not least the comments and hints on real bugs should be improved.

C.4.2 Deadline

On 9. March 2006 the project has to reach its end.

C.4.3 Tentative schedule

In the following project schedule, each task is splitted up into phases

1. Analysis & Reading of background material

2. Design

3. Implementation

4. Testing

5. Review

6. Documentation

A loopback to an earlier phase is always possible if the quality of the output isn’t satisfactory. Grey frames in the following Gant chart represent actual implementation work, milestones are printed black. 102 Appendix C. Project Plan

Month 1 Month 2 Month 3 Month 4 Month 5 Month 6 Setup Environment

M1: Ready to Run

Valgrind Analysis

Eiffel Compilation

TestStudio Analysis

M2: Valgrind Presentation

Adapter Framework

Reference Test Cases

M3: Framework

Adapter Implementation Project Core

Time Buffer

M4: Prototype

Output Reﬁnement

Thesis Report

Documentation

M5: Paper Thesis Report

M6: Final Presentation

The milestones of the projects in detail are the following:

• M1 (Ready to Run): 2005 - 09 - 11

• M2 (Valgrind, Presentation): 2005 - 10 - 03

• M3 (Framework, Presentation): 2005 - 11 - 07

• M4 (Prototype, Presentation): 2006 - 01 - 09

• M5 (Printed Version of the Thesis Report): 2006 - 03 - 02

• M6 (Final Project Presentation): 2006 - 03 - 09