A Lightweight Test-Driven Approach to Test Indie Software Products

Yong Lai, and Hassan Reza School of Aerospace Sciences Department of Computer Science University of North Dakota Grand Forks, ND 58201, USA [email protected]

Abstract In this work, we propose a lightweight testing method In this work, we discuss the application of testing and guidelines for indie software developers aiming to method to a typical independent (or indie) software improve the software quality, simplify the testing process, development. Toward this goal, an experimental indie and reducing the cost of testing. Toward this goal, we are software ShoppingList is developed and tested using the applying our proposed method to a typical indie software proposed techniques. Some testing frameworks and tools application to examine its feasibility. are also utilized to facilitate the process. Based on our In what follows, we first describe the test-driven results we concluded that the test-driven development development method [1]. This discussion will assist us to does have a positive effect on reducing the cost of indie effectively incorporate the methodology into our proposed software development.. software testing approach. Next, our lightweight testing management process is discussed. We then discuss our decision on programming language, testing framework Key Words: Test-driven Development, Extreme and tool support. A model-view-controller [3] testing Programming (XP), Lightweight testing, JUnit, model is developed to depict the feasibility of our Jemmy, Indie Software Development and Testing. proposed testing process. More specifically, the feasibility of our method is examined using a simple indie software

application known as ShoppingList that manages a list of 1. Introduction groceries the user wants to shop.

Comparing to well-established software testing 2. Background and Related Works method (e.g., white-box, black-box, etc.) used in the traditional software development methodologies (e.g., Test-driven development (TDD) refers to a testing waterfall model, spiral model, etc.) indie software testing method that requires testing a requirement (or code) first is a new concept and it is at its primitive stage. and implementing that requirement (or code) when the Informally, the indie software development refers to the test fails [1]. More specifically, TDD is an evolutionary independent development of software systems by very and specification development method in which writing small team of talented but market savvy software code and testing the corresponding code are carried out in developers having limited human and financial resources. interleaved fashion [1]. The approach originally Examples of indie software developers include introduced as part of agile development such as Extreme independent game and mobile applications (or mobile Programming [4] that often works with automated apps) developers. software testing such as Junit [10]. Test-driven software The front-end and back-end developments of these development process heavily relies on the repetition of a types of software require agilities and speed due to market very short development cycle: first developer writes a short windows of opportunities. Traditional heavy failing test case describing the new functionality, then weighted and plan driven software development process s/he writes the code to pass that test, and finally, s/he such as waterfall model are very expensive and hence are refactors (or improve) the new code according to ill-suited for development of these types of systems for acceptable standards [1,4]. Testing-driven development the following reasons: demands complete, consistent, and (TDD) is especially efficient in solving the prevailing correct requirements, testing process takes place only problem by indie software developers consisting of small after the software has been written, lack of systematic test number of talented programmers with limited resources. case design and testing plan, combined debugging and A test-driven development cycle includes the testing approach, etc [4]. following steps [4]: 1) Identify new functionality and generating the the following steps: 1) test planning, 2) test design, 3) test corresponding test cases. setup, 4) test execution, 5) test result analysis, and 6) test 2) Write the code: Initially, the test case will fail and reporting. In our experiment, we simplified the task of corresponding code must be written and implemented each stage to create a light-weight testing management to pass subsequent test case. process. To further simplify the testing process, we have 3) Refactor the code: After all test cases passed integrated the elements of lightweight testing process with successfully, we need to clean up the code along with the steps of test-driven development (TDD) as follows: the test cases as necessary. 1) Test planning and test design. 4) Repeat: The previous three steps are repeated to push These two stages are integrated in the first step of forward the whole development process until the test-driven development. Different from the test whole program is written. planning and test design stages in traditional testing management process, which usually requires the Figure 1 shows a typical test-driven development cycle. requirements to be complete into consideration. In addition, test planning and design at each cycle of test-driven development focuses on a small chunk of requirements. Since the test planning and design only consider a small part of the functionality of the software, a detailed test schedule is minimized or becomes unnecessary. Also, because of the small scale of indie software, other steps such as deciding test strategy, test requirements and items can be skipped in order to keep the testing management process simple and agile. 2) Test setup and test execution. For the test-driven indie software development, test setup and execution are related to writing product code to pass the newly added test case. Again, because of the small scale of indie software, test setup is trivial. After the product code is written, the new test cases and old test cases Figure 1. Test-driven development cycle. all need to be executed to ensure the new code doesn’t introduce errors (regression testing). With the advancement of the development, test execution will 3. A light-weight testing method be a tedious task as the number of test cases keeps growing. Normally, automated testing tools [10] are The well managed testing processes such as waterfall used to ease the task of test execution. The test models have been proven to be very effective but very entrance criterion is easy to define: when the new test expensive in the development of commercial software cases are written. So as the test exit criterion: the new product because the approach demands complete, product code passes both the new test cases and old consistent, and correct requirements before the validation test cases. and verification (V&V) begin [4]. The testing (static and 3) Test result analysis and reporting. The task of this dynamic) activities usually are normally performed by stage is to write test report, analyze defects, make independent testing teams aiming at showing both correct changes, and prevent defects from reoccurring. In and incorrect behaviors of the systems under test. Because test-driven indie software development, this stage of the cost and human resource attributed to these should happen before or after the third step. The most approaches, it is impractical to directly apply them in the important task for indie developers at this stage is to indie software development. Therefore, it makes sense to document the test case design and test results adopt a lightweight testing process that shall be cheaper including the effectiveness of test cases, whether new and yet effective for indie software developers with product code passes the old test cases, the choice of limited resources. test data, etc. The output of this stage is very A lightweight methodology shall be a flexible important as well since it provides insightful development process using a handful of rules to manage guidance for future software development. the unstable requirements and environments. Examples of light-weight approach include adaptive software A modified test-driven development incorporating a development [5] and extreme programming (XP) [6], light-weight testing management process is shown in feature driven development [7], etc. figure 2. In general, the testing management process includes 4. Testing tools and techniques popularity of Android platform development and mobile apps. Besides testing methodology, testing tools also play an important role in the indie software development. 4.2 Unit testing framework Appropriate choice of testing tools can greatly enhance the productivity of indie developers because they can We choose JUnit [10] as our testing framework. JUnit is a minimize the test cost by providing sophisticated testing unit and regression testing framework; it is commonly frameworks and saving programmers’ effort in creating, considered as de facto tool for test-driven development organizing, and executing test cases. For different that works with java language. The key justifications for programming languages and platforms, there are many using Junit include providing a complete assertion tools to different testing tools available for indie developers to validate actual result against expected results; generating choose. Some of them are open source, while others are Java test cases by API; and reporting and aggregation commercial software. The factors influencing the choice supports. of testing tools include but not limited to individual’s A high-level class diagram of JUnit framework is styles, budget, tools, appropriateness to the project, etc. shown in figure 3.

Repeat

(Re)Write a test Test succeeds

Check if the test fails Test planning and design Test fails

(Re)Write a test Test(s) fail

Check if the test fails Test setup and execution All tests succeed

Document test cases and test results

Clean up code

Figure 3. High-level class diagram of JUnit framework.

Figure 2. An Improved test-driving development cycle The core classes of JUnit framework include the using light-weight following: • Test: Test is the interface that all test cases must implement. In JUnit framework, two classes 4.1 Programming language TestCase and TestSuit implemented the Test interface. • TestCase: TestCase is the simplest type of Test. We choose Java to implement our experimental project Most test case written by indie developers are going because as a programming language, Java is comparable to extend this class. A concrete TestCase class with any other high level programming languages. Java contains methods (with signature testXXX) has the advantage of being used as a prototype language. implementing individual tests as well as optional Also, as a mature programming language, Java enjoys the setup (for setting up test environment) and teardown availability of many well-established frameworks and (for cleaning up test environment) methods. tools support used in test-driven development and test • TestSuite: Like TestCase, TestSuite also implements automation. More importantly, Java is representative the Test interface. Like a container, TestSuite is used among the programming language adopted by indie to collect together tests: TestCases, other TestSuites, developers especially in the current trend of increasing or any combination of the two. The purpose of TestSuite is organizing logically related testCases • Mocks: a class in which testers can set expectations into individual groups. regarding what methods are calls, with what • Assert: Assert is the superclass of TestCase that parameters, how often, return values for various provides all kinds of assertion methods, which are calling situations, etc. It also provides a way to verify used to decide if a test passes, or fails. whether the expectations are met. • TestFailure: TestFailures encapsulates an error or Despite the differences between the semantics of mock failure that occurs during a test run. It keeps track of objects, they have a common target that is to take the the test cases that failed and the exception that was place of real objects for the purpose of testing some responsible for the error or failure. functionality that interacts with and is dependent on the • TestResult: It accumulates the results of running the real object [2]. There are two major approaches to create tests and is responsible for informing the start and mock objects: 1) coding them from scratch and 2) end of a test. generating those using frameworks or tools. • TestListener: Any class that wishes to track the A mock object written from scratch looks like a progress of a test run can implement this interface. It traditional class having some methods. It is just that the has methods for notification of the start and end of logic contained in the methods are greatly simplified to each test, as well as occurrence of errors and failures. return desired value or performing simplified actions that • TestRunner: TestRunner is used to run specified test a real object would do in place. Hand-coding all the mock cases. It expects the name of a TestCase class as objects can be a very time consuming task when the argument. If this class defines a static suite method it number of test cases is large. will be invoked and the returned test is run. There are many frameworks and tools have been Otherwise all the methods starting with “test” having created to make mock objects creation easier. EasyMock no arguments are run. is one of such frameworks which generate mock objects at runtime using Java’s proxy mechanism. EasyMock fits JUnit is not an automated testing tool: developers still very well with test-driven development because the have to write the test cases themselves. But by providing generated mock objects will not be affected by refactoring testing support library and runtime framework, JUnit does due to its unique style of recording expectations. Other enable developers’ write their test cases more similar frameworks and tools include MockMaker and conveniently, execute the test cases in batch, or run the MockObjects project. test cases automatically with the help of Java Another Neat Tool (ANT) build script. At each test run, JUnit will 4.4 Automated testing techniques report how many tests are executed, how many of the test methods in a test succeed, and how many failed. The For indie software developers, two major automated detail of each failure will be reported in a form of a print testing techniques are used more often than others: batch of Java’s stack track leading to the location of the failure. test and automated GUI test.

4.3 Mock objects 4.4.1. Batch test. Batch test refers to the ability to execute a lot of test cases together [4]. It is usually supported by a Test-driven development (TDD) focuses on a small part testing framework. In our experiment, the batch test of the functionality of the software one at a time. The functionality is provided by the JUnit framework. direct result is numerous independent, small, but focused test cases. Some test cases are context sensitive and hence 4.4.2 Automated GUI test. Most indie software involves require system to be in the specific state. This makes it intense graphical user interface (GUI) interaction with the very impractical to set the system in a specific state to user. Thus, GUI testing is inevitably an important part of meet the requirement of each test case since it may greatly the overall testing process. However, using test-driven increase the number of test cases. The other problem is development for GUI testing is tricky because automated lack of proper coverage criteria to measure the adequacy testing of user interface is not as intuitive as other non- of test cases when context is very important. interface related testing. Fortunately, there are many Mock objects are shown to be very effective when the automated GUI testing tools have been developed to help objects are not currently available (e.g., Database). Mock developers make the task easier. objects are interpreted in three different ways [2]: In this work, we choose Jemmy to implement our • Stubs: a class with methods that do nothing. They are automated GUI testing. Jemmy is a Java library that is created simply to allow the system to compile and used to create automated tests for Java GUI application. run. Using Jemmy, each component in the Swing API has a • Fakes: a class with methods that return a fixed value corresponding Operator class in the Jemmy API. An or values that can either be hardcoded or set operator is used to wrap a corresponding Swing object on programmatically. the GUI to be tested. The developer can specify the behavior of each operator and wrap all these code in a Using our proposed method, the first step is to identify single testXXX method of a TestCase class which can be smell set of functionalities and corresponding test cases executed by JUnit framework automatically. according to MVC. Figure 4 shows a screenshot of the project structure and the functionalities to be tested 4.5. A model-view-controller testing model according to MVC.

Model-view-controller (MVC) is a classic design pattern often used by indie and GUIs application developer which requires the ability to maintain multiple model-view-controller views of the same data [3][8]. testing model To test the application designed using MVC pattern, we identify three classes of test cases corresponding to the essential elements of the MVC pattern, namely, test cases to test model, test cases to test controller, and test cases to test GUI. This test partitioning, in turn, allows us to organize test cases in manner that traceability between code implementing the requirements and test cases to validate these code can be established. Furthermore, it helps to simplify the testing process because each test group is independent of other test suites and hence reduces the duplicated test. This should increase the effectiveness of test cases to uncover possible software defects. JUnit library 5. Case Study EasyMock library A simple mobile application, namely, ShoppingList is used to show the feasibility of our proposed method. Automated GUI testing library ShoppingList attempts to help the shopper to keep track of the groceries (items) she/he wants to shop. The main Figure 4. Project “ShoppingList” structure. features of the program are as follows: • It keeps a list of groceries items the user needs to Step 2 requires to test setup and test execution buy. New groceries can be added to the list and environment. To this end, JUnit framework is used which existing groceries can be removed from the list. allows multiple test cases be executed at one time (i.e., • The name of each grocery in the list is unique. batch testing). Upon completion of step 2, the final step, Attempt to add duplicated grocery to the list should test result analysis and reporting, is performed. The fail. testing result will be provided immediately after the • Each grocery in the list has a priority which helps the testing is done. Using this feature, we can test our test user decide which groceries are needed more than cases in different groups. For example, we can choose to others. The priority of a grocery can be changed by execute all the test cases in the model testing group. After the user. all the test cases in the model testing group passed, we • Groceries belong to different categories such as can start executing all the test cases in the GUI testing diary, baked food, vegetable, meat, frozen food, etc. group and leave the test cases in the controller testing The user can manage the category to accomplish group for the last. In this way, program defects are tasks such as adding new categories, delete existing isolated and the errors in one group won’t affect other categories, or modifying categories. If a category groups. containing groceries is deleted, those groceries’ Figure 5 shows a JUnit testing report for a successful category property will be automatically changed to test run. The listbox shows that there are total 4 test cases: “uncategorized”. TestEmptyGroceryList contains one test “testSize”; • The grocery list can be sorted by name, category, or TestGroceryListWithOneGrocery contains one test priority. The order can be ascending or descending. “testSize”; TestGroceryListWithTwoGroceries contains two tests “testSizeAfterAddingTwo” and “testContents”. The project “ShoppingList” is developed using the Figure 6 shows a JUnit testing report for a failed test Eclipse java development tool [11] in Java language. The run. It shows that the test “testSize” in application is designed using the MVC design pattern [8]. TestEmptyGroceryList failed.

It shows that there are total 4 tests being executed. No errors No failures

Green bar shows that all tests passed.

Figure 5. JUnit testing report for a successful test run.

5.2 Automated GUI test

In our experimental project, the automated GUI test is implemented by Jemmy framework [9]. In this work, we Figure 8. The automated GUI testing code using Jemmy. are going to test the functional features such as adding/deleting a grocery item by user interface. Figure 7 shows a simplified GUI.

One failure

Failed test case

Failed test

Figure 9. The testing report for adding new groceries.

Figure 9 shows the testing report for the automated GUI Figure 6. JUnit testing report for a failed test run. test. It shows that there is one test case in the test run. The test case contains two tests “testListContents” and “testAdding”. The green bar shows that all tests passed. New grocery name The console output of the testing execution is shown in Grocery list Figure 10.

Add button

Figure 7. The simplified user interface (GUI)

The automated GUI testing code of adding new groceries using Jemmy is shown in figure 8.

[2] Astels, D. (2003). Test-driven development: Practical Guide. Prentice Hall PTR. (2003).

[3] R Taylor, N. Medvidovic, E. Dashofy. Software Architecture, Foundation, Theory, and Practice. Wiley, 2010. ComponentOperator wrapper provided by Jemmy [4] I. Sommerville. Software Engineering, 9th edition. Addison wisely, 2012. [5] J. Highsmith. Adaptive Software Development : Collaborative approach to development of complex, 1999. Systems.

[6] K. Beck. Extreme Programming Explained: Embrace Change. 2nd edition. Addison-Wesley, 2005.

[7] S. Palmer, S.R., and J. Felsing. A Practical Guide to Feature-Driven Development. Prentice Hall, 2002.

[8]. G. Krasner, S. Pope. A cookbook for using the model-view controller user interface isn Smalltalk-80. Figure 10. The console output of test automated GUI Journal of Object-Oriented Programming, 1(3):26-49, testing of adding new groceries. 1998.

6. Conclusions and Future works [9]. Jemmey framework . http://jemmy.java.net/ [10]. Junit Framework. http://junit.org/ Through our experiment, we found out that test-driven development does have a positive effect on indie software [11]. The Eclipse Java Development Tool. development. First, the introduction of testing at the early http://www.eclipse.org/jdt/ stage of software development helps developers to better understand the requirement specification and hence prevent design errors from being propagated into the later stage of development. This, in turn, decreases the development cost and hopefully increase the productivity. Also, appropriate choice of testing framework and tools can help developers implement effective test automation and simplify the task of regression test. In addition, a light-weight testing management process helps to enhance the test-driven development process because it enforces the regulation of test design, documentation, execution, and reporting. Last but not the least, the development of innovative testing models should always be encouraged since it may help to simplify the testing process. This work by no means is complete and therefore it warrants additional work and experimentations to validate our process. Therefore, future work includes, among other things, more case studies and experimentations.

References

[1]. K. Beck. Test-Driven Development by Example, Addison Wesley, 2003.