Nuts Bolts Bolts Nuts How To Programmers can build high-quality tests by following certain basic steps, outlined here. The author also discusses the cost, time, and personnel resources required for debugging and testing. How To Design Practical Test Cases Tsuneo Yamaura, Hitachi Software Engineering t Hitachi Software, our software has attained such high quality that only 0.02 percent of all bugs in a software program emerge at the user’s site.1 A If a typical project—10 engineers working for 12 months to develop 100,000 lines of code—contains 1,000 bugs, at most one will surface at the user’s site. We do not use sophisticated tools or state-of-the-art methodology— we simply test programs and fix the bugs detected. Our secret is that we document all test cases before we start debugging and test- ing. The written test cases provide significant advantages, and all our quality assur- ance activities start from this point. Figure 1 shows the percentage of defects detected at each stage in a 1990 Hitachi software development project. (More recent data, still being collected, will not show drastic changes.) Of all the bugs detected since the project’s inception, only 0.02 percent came out at the customer’s site. Such high quality could never have been attained if we had not employed written test cases. 30 IEEE Software November/December 1998 0740-7459/98/$10.00 © 1998 . 100 99.98 100 90 94.7 95.5 91.4 91.4 80 90.7 70 60 50 56.6 40 35.1 Percentage 30 28.0 21.5 20 10 6.1 4.5 0.7 3.3 0.8 0.02 0 0 Desk Unit Combination System Quality ”Cramming” Inspection ”Cramming” Field debugging debugging debugging test probe before during failure inspection inspection Figure 1. Bug detection during the development phases of a 1990 project. Documenting Test Cases grammers to a project that is behind schedule fre- quently causes more delays because the project en- Documented test cases can benefit your devel- gineers must spare precious time to educate the opment process in several key ways: new personnel. If the test cases are properly docu- ♦ Designing test cases gives you a chance to an- mented, however, the new staff can run the test alyze the specification from a different angle. cases as written. ♦ You can repeat the same test cases. ♦ Somebody else can execute the test cases for Validating test case quality you. You must, of course, test the test cases to ensure ♦ You can easily validate the quality of the test that they visit all the features implemented in the cases. software. Check whether they are well-balanced ♦ You can estimate the quality of the target soft- among normal, abnormal, and boundary cases, and ware early on. evaluate their overall sufficiency. Ad lib or random testing will never suffice—if you do not document A new viewpoint the test cases clearly, you cannot precisely measure Test cases provide another rendition of the func- their quality metrics and success. tional specification. Designing the test cases will give you pause: “Aha, I didn’t consider such and such Estimating software quality conditions. Was that specification really correct?”The If the test cases are properly developed, you can bugs revealed in this way would be harder to detect easily estimate the quality of the target software in and would require enormous time and money to fix the midst of debugging by applying the fish-in-the- later, such as in the system integration phase. pond analogy: If you detect four bugs after execut- Roughly speaking, you can catch 10 percent of a sys- ing 100 test cases out of 1,000, common sense says tem’s bugs when you are designing the test cases— the software will carry about 40 bugs altogether. a significant advantage. State-of-the-art software reliability theory is not that simple, but this quick and easy estimation gives you Repeating test scenarios a rough idea of the software’s quality. You can easily reiterate the same test cases if everything is documented. Reusing test cases lets you reproduce bugs, which helps ensure that de- Schedule, Cost, and tected bugs are fixed properly. Personnel Passing the test along You see how useful it is to document the test If you specify the test conditions, input data set, cases. There is no free lunch, however—you must and expected outcome, you can ask somebody else invest time, money, and personnel to enjoy the ad- to execute the test—which can prove particularly vantages. The question is how much you will need. valuable on a project running late. Adding pro- Here is a rough idea, with some statistical data. November/December 1998 IEEE Software 31 . Nuts olts B B olts uts N Program checking list 1 0 1 1 9 8 7 6 5 4 3 2 Checking ID 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 L L L L L L L L L L L E E E E E E E E E E Condition and expected output E C C C C C C C C C C C Format for control display Display for command view port Zoom Close n o i t Transition of command Dictionary i d view port Data m o C Code SR Verb Display of control port Display for command view port Standard status Zoom status Charge of Close status command view port Dictionary status Code status SR status Display Control View Test title (Command View Port) Test Spec. ID System ID ABCDE/01-00 Date DC Control ID Command Proc. Checked MC Management ID Parameter Priority A A B B B B B B B B B PMCard ID 0040-0199 Category N N N N N N N N N N E Written by K. Moriya Notes(#) Certified by N. Sakamoto Hitachi Working Sheet 10810040 1 of 99 Reviewed by A. K. Onama Software Number No. Figure 2. A matrix-based test sheet. Assume that an average 12-month project with 10 months evaluating whether the software is ship- engineers develops C-based software with 100,000 pable. Since the testers must be unbiased and un- lines of code. An average project might apportion derstand the customer’s requirements, they never 12 months among the following phases: reuse the programmers’test suites; they redesign all ♦ requirement specification: two months the test items from scratch. ♦ designing: three months The test-case density—“how many test cases ♦ coding: two months do we need?”—is one of the most critical engi- ♦ debugging: three months neering issues in testing. Too few test cases will ♦ testing (handled by the quality assurance leave easy bugs undetected, and too many will run team): two months short of time. At Hitachi Software, when the project completes Our 30-year empirical study based on trial and debugging, the product is sent to the QA depart- error found that the proper test case density con- ment where testers spend approximately two verges on the standard of one test case per 10 to 15 32 IEEE Software November/December 1998 . Nuts olts B B olts uts N LOC. This means that a product with 100,000 LOC And you must specify the testing priority, which needs approximately 10,000 test cases—1,000 per shows the testing order. programmer. Of 1,000 test cases, approximately 100 In the early 1970s, we employed natural lan- will be checked in the code inspection phase, and guage to describe the stepwise conditioning of the all of them will be checked in the machine debug- test cases. For example: ging phase (which means that the first 100 cases for ♦ When the person in the form is 65 years or older: code inspection overlap with the machine debug- ♦ When the person’s annual income is $10,000 or ging). Based on our study, these figures do not look less: unrealistic. An average programmer takes two ♦ When the person lives in area A-1: … (the nest- weeks to execute 100 test cases in the code inspec- ing of the conditions goes deeper) tion phase (10 cases per day), and two months to Since this approach frequently caused us to over- execute 1,000 test cases in the machine debugging look various combinations of the conditions (or phase (25 cases per day). Ten programmers will take holes in the test items), we migrated to matrix-based two weeks to design the test cases (assuming 100 test design in the mid-1970s. test cases a day per programmer), or 3.8 percent of Equivalent partitioning,2 a well-known testing the entire project. technique, uses a single value to represent the same domain. Suppose, for example, an admission fee varies by age, such that there is no fee for age 6 or Steps for Debugging and under, a $5 fee for 12 years old or younger, $8 for 18 Testing years or under, and $10 for 19 and above. The test cases you pick up from each domain can be “age 2” Systematic testing follows six core procedures: for the domain of 0 ≤ age ≤ 6, “age 10”for 6 < age ≤ 1. Design a set of test cases. 12, “age 14” for 12 < age ≤ 18, and “age 43” for 18 < 2. Check the test cases. age. Picking up “age 43” and “age 50” is redundant 3. Do code inspection based on the test cases. unless each represents a different domain in terms 4.