RI14008 2 June 2014 Computer Science IBM Research Report Test Generation from Business Rules Simon Holm Jensen Samsung Research America USA Suresh Thummalapenta Microsoft Corporation USA Saurabh Sinha IBM Research { India Satish Chandra Samsung Research America USA IBM Research Division Almaden - Austin - Beijing - Delhi - Haifa - T.J. Watson - Tokyo - Zurich LIMITED DISTRIBUTION NOTICE: This report has been submitted for publication outside of IBM and will probably be copyrighted if accepted for publication. It has been issued as a Research Report for early dis- semination of its contents. In view of the transfer of copyright to the outside publisher, its distribution out- side of IBM prior to publication should be limited to peer communications and specific requests. After out- side publication, requests should be filled only by reprints or legally obtained copies of the article (e.g., payment of royalties). Copies may be requested from IBM T.J. Watson Research Center, Publications, P.O. Box 218, Yorktown Heights, NY 10598 USA (email: [email protected]). Some reports are available on the internet at http://domino.watson.ibm.com/library/CyberDig.nsf/home. ABSTRACT in the relevant test steps to exercise a business rule, or a scenario Enterprise applications are difficult to test because their intended thereof. Without systematic test creation, testers may end up creat- functionality is either not described precisely enough or described ing multiple tests that exercise the same business rule, or a scenario in cumbersome business rules. It takes a lot of effort on the part thereof, over and over again without any additional benefit (espe- of a test architect to understand all the business rules and design cially if they are incentivised by the number of tests created rather tests that “cover” them, i.e., exercise all their constituent scenarios. than quality of the created tests), and more problematically, may Part of the problem is that it takes a complicated set up sequence neglect to create tests for some other business rules. to drive an application to a state in which a business rule can even In practice, due to time pressures, testers are more often ad-hoc fire. In this paper, we present a business rule modeling language than methodical in creating test scenarios and test data. One part that can be used to capture functional specification of an enterprise of the problem is that a realistic system might have hundreds of system. The language makes it possible to build tool support for business rules written out in plain text, and it is difficult to grasp rule authoring, so that obvious deficiencies in rules can be detected a global view of how the rules together describe the application mechanically. Most importantly, we show how to mechanically behavior. A related part of the problem is that it may need complex generate test sequences—i.e., test steps and test data—needed to reasoning to piece together test sequences that would cover each exercise these business rules. To this end, we translate the rules scenario. The net result is that, despite a lot of resources spent in into logical formulae and use constraint solving to generate test testing, bugs still escape into the field. sequences. One of our contributions is to overcome scalability is- Our vision is to make testing of enterprise software more tool- sues in this process, and we do this by using a novel algorithm for based, by adapting technology developed for automated and sys- organizing search through the space of candidate sequences to dis- tematic test generation for programs. In this vision, business rules cover covering sequences. Our results on three case studies show would be written in a structured notation that allows mechanized the promise of our approach. analysis. Special editors could be created to enable non-programm- ers to capture business rules in a structured notation; this is an in- dependent challenge in end-user programming. A tool would vali- 1. INTRODUCTION date business rules and point out any ambiguities or omissions that A business rule articulates some aspect of the expected func- it can detect. After the business rules pass validation, another tool tional behavior (or a requirement) of an enterprise application. Here would generate test sequences and test data to exercise the applica- is a simple business rule that determines how an invoice total is de- tion thoroughly as well as without redundancies. termined in a billing application: We have built a system to partially fulfil this vision. In the rest The Balance Type of a customer affects how invoice of this introductory section, we give an overview of our system, total is computed; it can be one of the following: describe some of the challenges in automating test generation for covering business rules, and briefly summarize our results. None: The customer’s account will not hold a balance; instead all charges accrued in an order will be included in the next invoice; 1.1 An Overview Credit: The customer’s account may accrue charges Enterprise systems of interest to us are transaction-oriented, which e.g., up to the set credit limit. Charges will automatically means that they consist of a set of transactions or operations ( be paid from the users credit pool until the set limit is create a customer, add an item to an order, and so on) that oper- reached. Users are responsible for paying their credit ate on databases. A business rule applies to a particular operation debt as well as any overages. supported by the system. Formally, a business rule describes the relation between the database state before and after the operation. We will examine this rule closely later; for now, suffice it to say Figure 2 shows formalization of the business rule quoted informally that the requirements of an enterprise system are typically captured at the beginning of the introduction. It says that the operation refers by a large number (often, hundreds) of business rules such as the to an invoice record, inv, and modifies specific attributes of inv. one above. There are three scenarios that occur in this rule. The first scenario It is reasonable to expect functional testing of an enterprise sys- applies when the customer to which the invoice refers has balance tem to cover its business rules, which is to say, testing would exer- type None; the precondition is shown on the left of the first arrow. cise every distinct scenario described in each of its business rules. Note that the invoice has references to the customer to which this For example, in the preceding rule, one of the scenario to be exer- invoice pertains and an order created in the system; in a relational cised is that a customer’s balance type is credit, and that the order database, these would be foreign keys in the customer and order amount exceeds the customer’s credit limit. A test that exercises tables. The second scenario applies when the customer has balance this scenario would set up a customer with the balance type as type Credit and the credit limit is sufficient to cover the order to- credit as well as a certain credit limit, say, 100, create an order tal. The third scenario applies when the customer has balance type and add items to the order to bring a total to, say, 120, to exceed Credit, but the order total exceeds the credit limit. In each scenario, the credit limit for that customer, and then finally create an invoice the effect of the operation is to compute invoice total and update for that order,and verify the invoiced amount. Figure 1 illustrates the customer’s residual credit limit; this effect, or postcondition is this flow. Although the values credit, 100, and 120 can be iden- shown to the right of the arrow in each case. Note that business tified just from this rule (by constraint solving), identifying a test rules refer to the state observable at transaction boundaries; inter- sequence is also important to apply those values at the right fields mediate program states encountered in the implementation while a on the appropriate screens.1 transaction is in process, are not important to business rules. It requires a carefully thought-out test scenario, i.e., a sequence Covering a business rule means to exercise each of its constituent of test steps as well as associated test data, i.e., values to be entered scenarios, referred henceforth as rule parts. To cover these three 1Not all fields on the screens in Figure 1 are constrained from the point rule parts in the business rule of Figure 2, it is necessary to create of view of exercising a particular scenario, but the application might still (separately) each of the three preconditions and, in each case, ver- demand sensible values for them. The tester is expected to make these up. ify that the postcondition holds after the operation has completed. 1. Create Customer 2. Enter Customer Details 3. Create Order for Customer 4. Add Item to Order 5. Generate Invoice 6. Verify Invoice Total Figure 1: Test sequence for exercising a business rule from the jBilling application. Compute Invoice Total problem—practioners tend to write business rules with an intention Input Invoice inv to be complete—but their intended use as input to test generation Modifies inv.total, inv.ord.cust.creditLimit process does increase expectations from the rules and, therefore, inv.ord.cust.balanceType = None ⋀ from the analysts who write them. inv.total = inv.ord.total inv.ord.Total > 0 1.2 Our Approach and Results inv.total = 0 ⋀ inv.ord.cust.balanceType = Credit ⋀ inv.ord.cust.creditLimit = At a high level, the idea is to use backward analysis to piece inv.ord.cust.creditLimit ≥ inv.ord.total inv.ord.cust.creditLimit – inv.ord.total together a sequence of operations to arrive at a desired state.