ATCG: AN AUTOMATED TEST CASE GENERATOR Journal of Information Technology Management ISSN #1042-1319 A Publication of the Association of Management ATCG: AN AUTOMATED TEST CASE GENERATOR CHARLES P. MORGAN FEDEX SERVICES CORPORATION [email protected] MARK L. GILLENSON UNIVERSITY OF MEMPHIS [email protected] XIHUI ZHANG UNIVERSITY OF NORTH ALABAMA [email protected] SON N. BUI TEXAS A&M UNIVERSITY–COMMERCE [email protected] EUNTAE “TED” LEE UNIVERSITY OF MEMPHIS [email protected] ABSTRACT Testing software to validate its functionality requires the development of carefully crafted test cases. Test cases can be developed algorithmically, they can be taken from existing data, or they can be developed from requirements. In this paper, we describe an Automated Test Case Generator (ATCG) that takes requirements statements as inputs and creates test cases as outputs. The ATCG represents a better, systematic way to develop test cases automatically from requirements, and provides multiple benefits and in the meantime, alleviates some potential issues with manually developing test cases from requirements. Keywords: Software testing, test case, test case generator, requirements document. software, whether it is systems software or application INTRODUCTION software. Software can vary from simple applications to very complex applications and systems, and it all must be Every newly developed product of any kind has tested. The fact that in today’s business environment, to be tested to ensure that it correctly performs the func- software applications have to function on or be accessed tions for which it was designed. This is true of an airplane, from a panoply of devices ranging from smart phones to an oil refinery, or a washing machine. It is also true of Journal of Information Technology Management Volume XXVII, Number 3, 2016 112 ATCG: AN AUTOMATED TEST CASE GENERATOR mainframes, each with a variety of operating systems and If necessary test cases are not included, it versions, multiplies the testing problem considerably [6]. can lead to defects not caught that may show Testing software to validate its functionality re- up in production. quires the development of carefully crafted test cases. A This paper describes an Automated Test Case test consists of executing a test case and comparing the Generator (ATCG) that takes requirements statements as actual result to the expected result. Broadly speaking, test inputs and creates test cases as outputs. We start by pre- cases can be developed in one of three major ways. They senting a review of the related work with a focus on can be developed algorithmically using techniques such as scripted testing and model-based testing. We then de- pairwise analysis [6] [9], they can be taken (possibly with scribe the Automated Test Case Generator (ATCG), a modifications) from data from an existing application that software tool that can read requirements documents and is being replaced or upgraded [6] [15], or they can be de- automatically detect and generate test cases. And finally, veloped from requirements [5] [6] [8] [9] [13]. This paper we summarize the paper, discuss the advantages of using will focus on developing test cases from requirements. the ATCG, and outline future research. Developing test cases from requirements has several distinct advantages [5] [6] [8] [9] [13]: RELATED WORK The assurance that the software that supports every requirement is being tested. Software testing is indispensable in ensuring The involvement of the business personnel software quality [2]. It has been estimated that software who requested the application in the testing testing uses up to 50% of the overall development cost process. [10] and the testing activities consume approximately Traceability of a defect, based on a test case 40% of the overall development time and effort [14]. To failure, to the requirement and thus to the improve the effectiveness and efficiency of testing, testers portion of the software that was written for need to create high-quality test cases. Writing test cases, the requirement. however, is an arduous task and often times involves hu- However, there are also some potential issues man errors. Thus, it is crucial that we can find a way of with manually developing test cases from requirements [5] automatically generating high-quality test cases which can [6] [8] [9] [13]: be used in making testing activities more effective and The process is labor-intensive and can be efficient to assure software quality. unacceptably time consuming. Automatically generating test cases for testing If the person writing the test cases is not one purposes is not a new idea. For instance, Weyuker et al. of the business people who developed the [15] introduced an approach to automatically generate test requirements (e.g., a tester) then there is a sets that would be highly effective at detecting faults. In significant time element in reviewing and requirement-based testing, the need of automation is even understanding the requirements. more serious. Requirement-based testing normally re- The requirements may not have been broken quires multiple manual processes where software testers down into fine enough divisions for a com- have to design test cases from requirements, run the test prehensive set of test cases to be developed. cases, and verify whether requirements of the soft- A test case developed from a requirement ware/system under test are met. If the requirement-based may not correctly test it. testing processes do not run correctly or consistently, the Too much time may be required to review size of the test cases may be too enormous to complete requirements documents and ask for clarifi- testing in reasonable time or the test cases cannot provide cations. the expected results [1]. To minimize the effort and cost on testing, many companies have started investing in the Additional time may be required to enter test automation of testing processes and tools. Automation has cases in a test case management system. proven to be an effective way to reduce cost and shorten Sometimes there are missed test cases re- the product release time to market, and will be a major quirements due to miscommunication in the factor for the success of software testing [13]. requirements or a missed requirement itself. In general, this effort can be divided by two types Extra time is required to add missed test cas- of automated test case generation techniques: white-box es to the test bed and re-test. (code based approach) and black-box (specification based Test cases that affect multiple systems some- approach). While each of the two techniques has its own times are either missed or not tested. advantages and disadvantages and both are used common- Journal of Information Technology Management Volume XXVII, Number 3, 2016 113 ATCG: AN AUTOMATED TEST CASE GENERATOR ly in automated test case generation, the emphasis in this quirements to test case specification and test procedure paper is on black-box test automated test case generation specification in order to automatically execute the test [13]. cases [3]. There are several black-box test approaches to automatically generating test cases in requirement-based Model-Based Testing testing. These methods are semi-automated ones that re- Another popular black-box automated test case quire software/system specifications expressed in precise generation method is model-based testing. Model-based notations. However, most of the software/system require- testing is the method that automatically generates test cas- ments currently used in the software industry are written es from models derived from system requirements and in natural language and thus normally require software system behaviors [4]. Typically in order to generate test testers or systems analysts to translate them into precise cases, software testers need to develop models to repre- software/system specifications. The two most common sent how the intended systems could work from system requirement-based testing methods that can semi- requirements documents. The goal of the model is to cap- automatically generate test cases, scripted testing and ture the functional requirements of the system in a clear, model-based testing, are reviewed as follows. concise and executable way that the language cannot ex- press [5]. The requirements model can be used to evaluate Scripted Testing the completeness of the system when developing and test- Scripted testing follows a single path that is writ- ing test cases. In order to develop a requirements model, ten by software testers. The path normally follows multi- modeling languages such as Unified Modeling Language ple steps and phrases that test cases are designed to fol- (UML) and SysML are typically used to create interac- low. tions and descriptions for system behaviors [8] [12]. Scripted testing has three important characteris- For example, Nebut et al. [11] introduced a semi- tics: repeatability, objectivity, and auditability. Repeata- automated method of generating test cases by using the bility refers to the ability to execute the scripts multiple UML method. This approach creates simulation models times in an identical way. If product A passed the first test from systems requirements to automatically generate test but failed the second test, the scripted testing has to be cases and execute them. able to reproduce the testing process to test product A. While much of this method is automated, an im- Objectivity refers to the ability to follow good test design portant phase that converts the requirements language to principles rather than well-skilled testers. For example, system models is manually done by the software testing scripts are used not only in early testing phases but also in team. This process, however, is very time-consuming and later testing activities to catch defects. Finally, auditability prone to error. refers to the ability to trace from requirements, to design, In summary, neither scripted testing nor model- to execution, and back again.