DOCSLIB.ORG

Deepstate: Symbolic Unit Testing for C and C++

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Deepstate: Symbolic Unit Testing for C and C++ DeepState: Symbolic Unit Testing for C and C++ Peter Goodman Alex Groce Trail of Bits, Inc. School of Informatics, Computing & Cyber Systems [email protected] Northern Arizona University [email protected] Abstract—Unit testing is a popular software development larger audience of developers. DeepState makes it possible methodology that can help developers detect functional regres- to write parameterized unit tests [38] in a Google Test- sions, explore boundary conditions, and document expected be- like framework, and automatically produce tests using angr, havior. However, writing comprehensive unit tests is challenging and time-consuming, and developers seldom explore the obscure Manticore, or Dr. Memory’s fuzzer, Dr. Fuzz [2]. (and bug-hiding) corners of software behavior without assistance. DeepState also targets the same space as property-based DeepState is a tool that provides a Google Test-like API to give testing tools such as QuickCheck [10], ScalaCheck [28], C and C++ developers push-button access to symbolic execution Hypothesis [26], and TSTL [17], [20], but DeepState’s test engines, such as Manticore and angr, and fuzzers, such as Dr. harnesses look like C/C++ unit tests. The major difference Fuzz. Rather than learning multiple complex tools, users learn one interface for defining a test harness, and can use various from previous tools is that DeepState aims to provide a methods to automatically generate tests for software. In addition front-end that can make use of a growing variety of back- to providing a familiar interface to binary analysis and fuzzing end methods for test generation, including (already) multiple for parameterized unit testing, DeepState also provides constructs binary analysis engines and a non-symbolic fuzzer. Developers that aid in the construction of API-sequence tests, where the who write tests using DeepState can expect that DeepState tool chooses the functions or methods to call, allowing for even more diverse and powerful tests. By serving as a front-end to will let them, without rewriting their tests, make use of new multiple tools, DeepState additionally provides a way to apply binary analysis or fuzzing advances. The harness/test definition (novel) high-level strategies to test generation, and to compare remains the same, but the method(s) used to generate tests effectiveness and efficiency of testing back-ends, including binary may change over time. In contrast, most property-based tools analysis tools. only provide random testing, and symbolic execution based approaches such as Pex [37], [39] or KLEE [8], while similar I. INTRODUCTION on the surface in some ways, always have a single back-end for A key limitation in the advancement of binary analysis and test generation. Even a system such as TSTL, which aims to other approaches to improving software security and reliability provide a common interface [14] to different testing methods is that there is very little overlap between security experts assumes that all of those methods will be written using the familiar with tools such as angr [34], [35], [33], Manticore TSTL API and (concrete) notion of state. [27], or S2E [9], and the developers who produce most code In addition to letting developers find the best tool for that needs to be secure or highly reliable. testing their system (or find different bugs with different Developers, as a class, do not know how to use binary tools), the ability to shift back-ends effortlessly addresses a analysis tools; developers, as a class, seldom even know how core problem of practical automated test generation. Most of to use less challenging tools such as fuzzers, even relatively the tools in wide use are research prototypes with numerous push-button ones such as AFL [41]. Even those developers bugs. In fact, in our own early efforts to test a file system whose primary code focus is critical security infrastructure using DeepState, we discovered a show-stopping fault in such as OpenSSL are often not users, much less expert users, angr [3]. Without DeepState, encountering such a bug means of such tools. stopping testing efforts until the bug can be fixed (which Developers do, however, often know how to use unit testing can be difficult, especially if reporting a bug arising from frameworks, such as JUnit [12] or Google Test [1]. DeepState sensitive code is at issue), or re-writing your tests (possibly aims to bring some of the power (in particular, high quality after learning a new tool). With DeepState, one simply types automated test generation) of binary analysis frameworks to a deepstate-manticore instead of deepstate-angr to switch from angr to Manticore. Moreover, sharing a single notion of test harness makes it possible to implement high-level test generation strategies once and thus avoid the effort of re-implementing such ap- proaches for every tool (or, worse yet, every individual testing effort, as happens with many strategies). In this paper we describe the design and implementation of Network and Distributed Systems Security (NDSS) Symposium 2018 18-21 February 2018, San Diego, CA, USA DeepState (Section II) and show how DeepState’s approach ISBN 1-1891562-49-5 enables API-call sequence testing of a simple user mode http://dx.doi.org/10.14722/ndss.2018.23xxx www.ndss-symposium.org DeepState.hpp libdeepstate.a 1 bool IsPrime(const unsigned p) { #define TEST(... DeepState_Assume: for (unsigned i = 2; i <= (p/2); ++i) { #define TEST_F(... ... 2 #define ASSUME(... DeepState_Pass: 3 if (!(p % i)) { #define EXPECT(... ... #define ASSERT(... DeepState_SoftFail: 4 return false; #define LOG(... ... DeepState_ 5 } 6 } 7 return true; 8 } Tests.cpp Tests.o Tests 9 #include <DeepState Unit_Name1_Test: 0101010 call 1010101 10 TEST(PrimePolynomial, OnlyGeneratesPrimes) { TEST(Unit, Name1) { DeepState_Int 0010101 11 symbolic_unsigned x, y, z; symbolic_int x; cmp eax, 0 0110101 ASSUME(x > 0); jg .L0 0100101 12 ASSUME_GT(x, 0); ... call DeepState_.. 0101010 } .L0: 1010101 13 unsigned poly = (x * x) + x + 41; ... 0101010 ASSUME_GT(y, 1); TEST(Unit, 1010101 14 0101010 15 ASSUME_GT(z, 1); 16 ASSUME_LT(y, poly); 17 ASSUME_LT(z, poly); 18 ASSERT_NE(poly, y * z) Fig. 1. The DeepState workflow mirrors that of Google Test. The developer 19 << x << "ˆ2 + " << x << " + 41 is not prime"; includes the DeepState API header file (DeepState.hpp) into their testing 20 ASSERT(IsPrime(Pump(poly))) code (Tests.cpp), and uses the various macros, such as TEST, to organize 21 << x << "ˆ2 + " << x << " + 41 is not prime"; the code to be tested. The compiled tests (Tests.o) are linked against the 22 } DeepState runtime (libdeepstate.a), producing a test suite binary (Tests). This 23 binary can be run independently, or under the control of programs like 24 int main(int argc, char *argv[]) { DeepState_InitOptions(argc, argv); deepstate-angr. 25 26 return DeepState_Run(); 27 } filesystem (Section III). Finally, we show how the DeepState Fig. 2. A simple test case that asserts that every number generated by the approach enables novel solutions to scalability challenges in polynomial x2 + x + 41 is prime. The Pump function (described in Section symbolic execution (Section IV). IV-A) “pre-forks” the executor on possible assignments to poly, thereby enabling many state forks to simultaneously execute IsPrime at native speeds on many different concrete assignments to poly. II. IMPLEMENTATION A. Architecture by the DeepState library to define and run tests. The key DeepState is made of two components: i) a static library elements of a DeepState-based test suite are the definition of linked into every test suite (Figure 1), and ii) the executor, tests and test fixtures, the sources of symbolic values, and the which is a lightweight orchestrator around symbolic execution pre- and post-conditions that constrain the properties to be engines like Manticore and angr, or fuzzing engines like tested. Figure 2 shows how these basic components combine Dr. Fuzz. Adding new back-ends involves producing a new into a unit test. executor, but does not require changes to the static library. 1) Library: The DeepState library is approximately 1200 B. Tests lines of C, with an additional 600 lines of C++ which wrap Unit tests are functions that are defined using the core C APIs and provide the conveniences of a Google the special TEST macro. This macro denotes the Test-like system. The majority of the DeepState library code is unit name (PrimePolynomial) and the test name unrelated to the core task of testing. In fact, a lot of complexity (OnlyGeneratesPrimes) accordingly. Contained within in DeepState relates to ensuring that the common practice of the typical body of a TEST-defined function is code that sets printf-like logging inside of unit tests does not degrade up the environment, executes the functions to be tested, and performance. asserts any pre- or post-conditions that must hold. 2) Executor: The orchestration engine is approximately The TEST macro wraps the test code in a function that calls 350 lines of Python, with an additional 250 lines each for the DeepState_Pass function as its last action. The macro enabling Manticore and angr support. DeepState uses APIs also ensures that the wrapped function is registered at runtime already provided by angr and Manticore to locate and hook into a global linked list of all test cases. At runtime, the key APIs and find all test cases to run. In the case of executor simulates the program’s normal execution (including Dr. Fuzz, a special test case harness function is defined, test registration) up until the call to DeepState_Run. At DrMemFuzzFunc, which is Dr. Fuzz’s standard “entry-point” this point, the executor takes control, and forks execution for into a code-base and the source of mutated input bytes. each registered test. The executor will exhaustively explore Eventually we plan to integrate libFuzzer [32] (and perhaps the state space of each test case, up until the execution other fuzzers) in a similar way.
Load more

Recommended publications
  • Chopped Symbolic Execution
    Chopped Symbolic Execution David Trabish Andrea Mattavelli Noam Rinetzky Cristian Cadar Tel Aviv University Imperial College London Tel Aviv University Imperial College London Israel United Kingdom Israel United Kingdom [email protected] [email protected] [email protected] [email protected] ABSTRACT the code with symbolic values instead of concrete ones. Symbolic Symbolic execution is a powerful program analysis technique that execution engines thus replace concrete program operations with systematically explores multiple program paths. However, despite ones that manipulate symbols, and add appropriate constraints on important technical advances, symbolic execution often struggles to the symbolic values. In particular, whenever the symbolic executor reach deep parts of the code due to the well-known path explosion reaches a branch condition that depends on the symbolic inputs, it problem and constraint solving limitations. determines the feasibility of both sides of the branch, and creates In this paper, we propose chopped symbolic execution, a novel two new independent symbolic states which are added to a worklist form of symbolic execution that allows users to specify uninter- to follow each feasible side separately. This process, referred to as esting parts of the code to exclude during the analysis, thus only forking, refines the conditions on the symbolic values by adding targeting the exploration to paths of importance. However, the appropriate constraints on each path according to the conditions excluded parts are not summarily ignored, as this may lead to on the branch. Test cases are generated by finding concrete values both false positives and false negatives. Instead, they are executed for the symbolic inputs that satisfy the path conditions.
    [Show full text]
  • Practical Unit Testing for Embedded Systems Part 1 PARASOFT WHITE PAPER
    Practical Unit Testing for Embedded Systems Part 1 PARASOFT WHITE PAPER Table of Contents Introduction 2 Basics of Unit Testing 2 Benefits 2 Critics 3 Practical Unit Testing in Embedded Development 3 The system under test 3 Importing uVision projects 4 Configure the C++test project for Unit Testing 6 Configure uVision project for Unit Testing 8 Configure results transmission 9 Deal with target limitations 10 Prepare test suite and first exemplary test case 12 Deploy it and collect results 12 Page 1 www.parasoft.com Introduction The idea of unit testing has been around for many years. "Test early, test often" is a mantra that concerns unit testing as well. However, in practice, not many software projects have the luxury of a decent and up-to-date unit test suite. This may change, especially for embedded systems, as the demand for delivering quality software continues to grow. International standards, like IEC-61508-3, ISO/DIS-26262, or DO-178B, demand module testing for a given functional safety level. Unit testing at the module level helps to achieve this requirement. Yet, even if functional safety is not a concern, the cost of a recall—both in terms of direct expenses and in lost credibility—justifies spending a little more time and effort to ensure that our released software does not cause any unpleasant surprises. In this article, we will show how to prepare, maintain, and benefit from setting up unit tests for a simplified simulated ASR module. We will use a Keil evaluation board MCBSTM32E with Cortex-M3 MCU, MDK-ARM with the new ULINK Pro debug and trace adapter, and Parasoft C++test Unit Testing Framework.
    [Show full text]
  • Parallel, Cross-Platform Unit Testing for Real-Time Embedded Systems
    University of Denver Digital Commons @ DU Electronic Theses and Dissertations Graduate Studies 1-1-2017 Parallel, Cross-Platform Unit Testing for Real-Time Embedded Systems Tosapon Pankumhang University of Denver Follow this and additional works at: https://digitalcommons.du.edu/etd Part of the Computer Sciences Commons Recommended Citation Pankumhang, Tosapon, "Parallel, Cross-Platform Unit Testing for Real-Time Embedded Systems" (2017). Electronic Theses and Dissertations. 1353. https://digitalcommons.du.edu/etd/1353 This Dissertation is brought to you for free and open access by the Graduate Studies at Digital Commons @ DU. It has been accepted for inclusion in Electronic Theses and Dissertations by an authorized administrator of Digital Commons @ DU. For more information, please contact [email protected],[email protected]. PARALLEL, CROSS-PLATFORM UNIT TESTING FOR REAL-TIME EMBEDDED SYSTEMS __________ A Dissertation Presented to the Faculty of the Daniel Felix Ritchie School of Engineering and Computer Science University of Denver __________ In Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy __________ by Tosapon Pankumhang August 2017 Advisor: Matthew J. Rutherford ©Copyright by Tosapon Pankumhang 2017 All Rights Reserved Author: Tosapon Pankumhang Title: PARALLEL, CROSS-PLATFORM UNIT TESTING FOR REAL-TIME EMBEDDED SYSTEMS Advisor: Matthew J. Rutherford Degree Date: August 2017 ABSTRACT Embedded systems are used in a wide variety of applications (e.g., automotive, agricultural, home security, industrial, medical, military, and aerospace) due to their small size, low-energy consumption, and the ability to control real-time peripheral devices precisely. These systems, however, are different from each other in many aspects: processors, memory size, develop applications/OS, hardware interfaces, and software loading methods.
    [Show full text]
  • Advanced Systems Security: Symbolic Execution
    Systems and Internet Infrastructure Security Network and Security Research Center Department of Computer Science and Engineering Pennsylvania State University, University Park PA Advanced Systems Security:! Symbolic Execution Trent Jaeger Systems and Internet Infrastructure Security (SIIS) Lab Computer Science and Engineering Department Pennsylvania State University Systems and Internet Infrastructure Security (SIIS) Laboratory Page 1 Problem • Programs have flaws ‣ Can we find (and fix) them before adversaries can reach and exploit them? • Then, they become “vulnerabilities” Systems and Internet Infrastructure Security (SIIS) Laboratory Page 2 Vulnerabilities • A program vulnerability consists of three elements: ‣ A flaw ‣ Accessible to an adversary ‣ Adversary has the capability to exploit the flaw • Which one should we focus on to find and fix vulnerabilities? ‣ Different methods are used for each Systems and Internet Infrastructure Security (SIIS) Laboratory Page 3 Is It a Flaw? • Problem: Are these flaws? ‣ Failing to check the bounds on a buffer write? ‣ printf(char *n); ‣ strcpy(char *ptr, char *user_data); ‣ gets(char *ptr) • From man page: “Never use gets().” ‣ sprintf(char *s, const char *template, ...) • From gnu.org: “Warning: The sprintf function can be dangerous because it can potentially output more characters than can fit in the allocation size of the string s.” • Should you fix all of these? Systems and Internet Infrastructure Security (SIIS) Laboratory Page 4 Is It a Flaw? • Problem: Are these flaws? ‣ open( filename, O_CREAT );
    [Show full text]
  • Unit Testing Is a Level of Software Testing Where Individual Units/ Components of a Software Are Tested. the Purpose Is to Valid
    Unit Testing is a level of software testing where individual units/ components of a software are tested. The purpose is to validate that each unit of the software performs as designed. A unit is the smallest testable part of software. It usually has one or a few inputs and usually a single output. In procedural programming a unit may be an individual program, function, procedure, etc. In object-oriented programming, the smallest unit is a method, which may belong to a base/ super class, abstract class or derived/ child class. (Some treat a module of an application as a unit. This is to be discouraged as there will probably be many individual units within that module.) Unit testing frameworks, drivers, stubs, and mock/ fake objects are used to assist in unit testing. METHOD Unit Testing is performed by using the White Box Testing method. When is it performed? Unit Testing is the first level of testing and is performed prior to Integration Testing. BENEFITS Unit testing increases confidence in changing/ maintaining code. If good unit tests are written and if they are run every time any code is changed, we will be able to promptly catch any defects introduced due to the change. Also, if codes are already made less interdependent to make unit testing possible, the unintended impact of changes to any code is less. Codes are more reusable. In order to make unit testing possible, codes need to be modular. This means that codes are easier to reuse. Development is faster. How? If you do not have unit testing in place, you write your code and perform that fuzzy ‘developer test’ (You set some breakpoints, fire up the GUI, provide a few inputs that hopefully hit your code and hope that you are all set.) If you have unit testing in place, you write the test, write the code and run the test.
    [Show full text]
  • Including Jquery Is Not an Answer! - Design, Techniques and Tools for Larger JS Apps
    Including jQuery is not an answer! - Design, techniques and tools for larger JS apps Trifork A/S, Headquarters Margrethepladsen 4, DK-8000 Århus C, Denmark [email protected] Karl Krukow http://www.trifork.com Trifork Geek Night March 15st, 2011, Trifork, Aarhus What is the question, then? Does your JavaScript look like this? 2 3 What about your server side code? 4 5 Non-functional requirements for the Server-side . Maintainability and extensibility . Technical quality – e.g. modularity, reuse, separation of concerns – automated testing – continuous integration/deployment – Tool support (static analysis, compilers, IDEs) . Productivity . Performant . Appropriate architecture and design . … 6 Why so different? . “Front-end” programming isn't 'real' programming? . JavaScript isn't a 'real' language? – Browsers are impossible... That's just the way it is... The problem is only going to get worse! ● JS apps will get larger and more complex. ● More logic and computation on the client. ● HTML5 and mobile web will require more programming. ● We have to test and maintain these apps. ● We will have harder requirements for performance (e.g. mobile). 7 8 NO Including jQuery is NOT an answer to these problems. (Neither is any other js library) You need to do more. 9 Improving quality on client side code . The goal of this talk is to motivate and help you improve the technical quality of your JavaScript projects . Three main points. To improve non-functional quality: – you need to understand the language and host APIs. – you need design, structure and file-organization as much (or even more) for JavaScript as you do in other languages, e.g.
    [Show full text]
  • Test Generation Using Symbolic Execution
    Test Generation Using Symbolic Execution Patrice Godefroid Microsoft Research [email protected] Abstract This paper presents a short introduction to automatic code-driven test generation using symbolic execution. It discusses some key technical challenges, solutions and milestones, but is not an exhaustive survey of this research area. 1998 ACM Subject Classification D.2.5 Testing and Debugging, D.2.4 Software/Program Veri- fication Keywords and phrases Testing, Symbolic Execution, Verification, Test Generation Digital Object Identifier 10.4230/LIPIcs.FSTTCS.2012.24 1 Automatic Code-Driven Test Generation In this paper, we discuss the problem of automatic code-driven test generation: Given a program with a known set of input parameters, automatically generate a set of input values that will exercise as many program statements as possible. Variants of this problem definition can be obtained using other code coverage criteria [48]. An optimal solution to this problem is theoretically impossible since this problem is undecidable in general (for infinite-state programs written in Turing-expressive programming languages). In practice, approximate solutions are sufficient. Although automating test generation using program analysis is an old idea (e.g., [41]), practical tools have only started to emerge during the last few years. Indeed, the expensive sophisticated program-analysis techniques required to tackle the problem, such as symbolic execution engines and constraint solvers, have only become computationally affordable in recent years thanks to the increasing computational power available on modern computers. Moreover, this steady increase in computational power has in turn enabled recent progress in the engineering of more practical software analysis techniques. Specifically, this recent progress was enabled by new advances in dynamic test generation [29], which generalizes and is more powerful than static test generation, as explained later in this paper.
    [Show full text]
  • Testing and Debugging Tools for Reproducible Research
    Testing and debugging Tools for Reproducible Research Karl Broman Biostatistics & Medical Informatics, UW–Madison kbroman.org github.com/kbroman @kwbroman Course web: kbroman.org/Tools4RR We spend a lot of time debugging. We’d spend a lot less time if we tested our code properly. We want to get the right answers. We can’t be sure that we’ve done so without testing our code. Set up a formal testing system, so that you can be confident in your code, and so that problems are identified and corrected early. Even with a careful testing system, you’ll still spend time debugging. Debugging can be frustrating, but the right tools and skills can speed the process. "I tried it, and it worked." 2 This is about the limit of most programmers’ testing efforts. But: Does it still work? Can you reproduce what you did? With what variety of inputs did you try? "It's not that we don't test our code, it's that we don't store our tests so they can be re-run automatically." – Hadley Wickham R Journal 3(1):5–10, 2011 3 This is from Hadley’s paper about his testthat package. Types of tests ▶ Check inputs – Stop if the inputs aren't as expected. ▶ Unit tests – For each small function: does it give the right results in specific cases? ▶ Integration tests – Check that larger multi-function tasks are working. ▶ Regression tests – Compare output to saved results, to check that things that worked continue working. 4 Your first line of defense should be to include checks of the inputs to a function: If they don’t meet your specifications, you should issue an error or warning.
    [Show full text]
  • The Art, Science, and Engineering of Fuzzing: a Survey
    1 The Art, Science, and Engineering of Fuzzing: A Survey Valentin J.M. Manes,` HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo Abstract—Among the many software vulnerability discovery techniques available today, fuzzing has remained highly popular due to its conceptual simplicity, its low barrier to deployment, and its vast amount of empirical evidence in discovering real-world software vulnerabilities. At a high level, fuzzing refers to a process of repeatedly running a program with generated inputs that may be syntactically or semantically malformed. While researchers and practitioners alike have invested a large and diverse effort towards improving fuzzing in recent years, this surge of work has also made it difficult to gain a comprehensive and coherent view of fuzzing. To help preserve and bring coherence to the vast literature of fuzzing, this paper presents a unified, general-purpose model of fuzzing together with a taxonomy of the current fuzzing literature. We methodically explore the design decisions at every stage of our model fuzzer by surveying the related literature and innovations in the art, science, and engineering that make modern-day fuzzers effective. Index Terms—software security, automated software testing, fuzzing. ✦ 1 INTRODUCTION Figure 1 on p. 5) and an increasing number of fuzzing Ever since its introduction in the early 1990s [152], fuzzing studies appear at major security conferences (e.g. [225], has remained one of the most widely-deployed techniques [52], [37], [176], [83], [239]). In addition, the blogosphere is to discover software security vulnerabilities. At a high level, filled with many success stories of fuzzing, some of which fuzzing refers to a process of repeatedly running a program also contain what we consider to be gems that warrant a with generated inputs that may be syntactically or seman- permanent place in the literature.
    [Show full text]
  • Background Goal Unit Testing Regression Testing Automation
    Streamlining a Workflow Jeffrey Falgout Daniel Wilkey [email protected] [email protected] Science, Discovery, and the Universe Bloomberg L.P. Computer Science and Mathematics Major Background Regression Testing Jenkins instance would begin running all Bloomberg L.P.'s main product is a Another useful tool to have in unit and regression tests. This Jenkins job desktop application which provides sufficiently complex systems is a suite would then report the results on the financial tools. Bloomberg L.P. of regression tests. Regression testing Phabricator review and signal the author employs many computer programmers allows you to test very high level in an appropriate way. The previous to maintain, update, and add new behavior which can detect low level workflow required manual intervention to features to this product. bugs. run tests. This new workflow only the The product the code base was developer to create his feature branch in implementing has a record/playback order for all tests to be run automatically. Goal feature. This was leveraged to create a Many of the tasks that a programmer regression testing framework. This has to perform as part of a team can be framework leveraged the unit testing automated. The goal of this project was framework wherever possible. to work with a team to automate tasks At various points during the playback, where possible and make them easier callback methods would be invoked. An example of the new workflow, where a Jenkins where impossible. These methods could have testing logic to instance will automatically run tests and update the make sure the application was in the peer review.
    [Show full text]
  • Symstra: a Framework for Generating Object-Oriented Unit Tests Using Symbolic Execution
    Symstra: A Framework for Generating Object-Oriented Unit Tests using Symbolic Execution Tao Xie1, Darko Marinov2, Wolfram Schulte3, David Notkin1 1 Dept. of Computer Science & Engineering, Univ. of Washington, Seattle, WA 98195, USA 2 Department of Computer Science, University of Illinois, Urbana-Champaign, IL 61801, USA 3 Microsoft Research, One Microsoft Way, Redmond, WA 98052, USA {taoxie,notkin}@cs.washington.edu, [email protected], [email protected] Abstract. Object-oriented unit tests consist of sequences of method invocations. Behavior of an invocation depends on the method’s arguments and the state of the receiver at the beginning of the invocation. Correspondingly, generating unit tests involves two tasks: generating method sequences that build relevant receiver- object states and generating relevant method arguments. This paper proposes Symstra, a framework that achieves both test generation tasks using symbolic execution of method sequences with symbolic arguments. The paper defines sym- bolic states of object-oriented programs and novel comparisons of states. Given a set of methods from the class under test and a bound on the length of sequences, Symstra systematically explores the object-state space of the class and prunes this exploration based on the state comparisons. Experimental results show that Symstra generates unit tests that achieve higher branch coverage faster than the existing test-generation techniques based on concrete method arguments. 1 Introduction Object-oriented unit tests are programs that test classes. Each test case consists of a fixed sequence of method invocations with fixed arguments that explores a particular aspect of the behavior of the class under test. Unit tests are becoming an important component of software development.
    [Show full text]
  • Introduction to Unit Testing
    INTRODUCTION TO UNIT TESTING In C#, you can think of a unit as a method. You thus write a unit test by writing something that tests a method. For example, let’s say that we had the aforementioned Calculator class and that it contained an Add(int, int) method. Let’s say that you want to write some code to test that method. public class CalculatorTester { public void TestAdd() { var calculator = new Calculator(); if (calculator.Add(2, 2) == 4) Console.WriteLine("Success"); else Console.WriteLine("Failure"); } } Let’s take a look now at what some code written for an actual unit test framework (MS Test) looks like. [TestClass] public class CalculatorTests { [TestMethod] public void TestMethod1() { var calculator = new Calculator(); Assert.AreEqual(4, calculator.Add(2, 2)); } } Notice the attributes, TestClass and TestMethod. Those exist simply to tell the unit test framework to pay attention to them when executing the unit test suite. When you want to get results, you invoke the unit test runner, and it executes all methods decorated like this, compiling the results into a visually pleasing report that you can view. Let’s take a look at your top 3 unit test framework options for C#. MSTest/Visual Studio MSTest was actually the name of a command line tool for executing tests, MSTest ships with Visual Studio, so you have it right out of the box, in your IDE, without doing anything. With MSTest, getting that setup is as easy as File->New Project. Then, when you write a test, you can right click on it and execute, having your result displayed in the IDE One of the most frequent knocks on MSTest is that of performance.
    [Show full text]