Testing & Analysis We’ve only got , so we shouldn’t waste it on ineffective test design. Follow this to The Bold and the Beautiful world of Testing

BY HANS BUWALDA

30 BETTER SOFTWARE FEBRUARY 2004 www.stickyminds.com TEST PROCESSES ARE SUCCESSFUL IF THEY are effective, efficient, manageable, and fun. A major factor in how well these four objectives are achieved is the ap- proach to making test cases. One such approach is “soap opera” testing.

As many readers know, soap operas are dramatic shows that were originally sponsored by soap vendors. They depict life in a way that viewers can relate to, but the situations portrayed are typically condensed and exaggerated. In one episode, more things happen to the characters than most of us will experience in a lifetime.

Opinions may differ about whether soap operas are fun to watch, but it must be great fun to write them. Soap opera testing is similar to a soap opera in that tests are based on real life, exaggerated, and condensed.

Info to Go Tests should be fun and aggressive. Add structure to retain, and even improve, manageability. Write scenarios that are reality-

based, exaggerated, and condensed. BY LISA HENDERLING ILLUSTRATIONS

www.stickyminds.com FEBRUARY 2004 BETTER SOFTWARE 31 Testing & Analysis

The idea for turning test cases into to five people each) and come up with call the “mechanical approaches.” soap operas came to me while testing a stories based on the most extreme exam- By a mechanical approach, I mean de- new financial system. A group of end ples that had happened, or that could veloping tests following a simple, pre- users was mobilized to come up with a happen in practice. Imagination was in- structured process. For example, consid- large number of test cases quickly. It was vited and exaggeration was welcome. To er the following process. Start with a important that the test cases be very help the process, I asked the groups to finite number of screens or requirements. good and very aggressive, even though imagine that they were writing soap op- For each screen (or requirement) in a sys- time was limited due to extra pressure eras. This helped create lean, mean test tem, make a test case that inputs values from the potential Y2K problem. Be- cases fast. It also made the whole experi- in that screen. After entering the values cause the system had to do sensitive ence a bit more fun, which was important for each screen, verify that those values work such as calculating pensions for re- in this situation, where end users had to appear in the system. Then, check tirees, errors were a big no-no. put in a lot of time testing—something whether the system has caught illegal in- The end users were the people with not in their job descriptions. put, such as mandatory fields that the most practical knowledge, but they haven’t been filled in. If the above tests had no IT or QA background. The testers Mechanical Approaches are completed successfully, testing is lacked the proper financial background to Testing complete. and the day-to-day experience. To solve To set the stage for soap opera testing, I These tests may be developed “top this, the testers and end users were asked want to contrast it with the more common down”—break down functional specifica- to sit together in small work groups (four ways test cases are usually made, which I tions and/or requirements until the items are elementary enough to create tests for them. Or they may be developed “bottom up”—make one or more tests for each small unit in a system, Count the Goodies then organize those tests into larger suites. These are analyti- BY HANS BUWALDA cal ways of thinking, common Our soap, called “Count the Goodies,” is about the Goody family. for IT people. For testing, however, a bet- ter direction is what I like to call “outside In this episode, the Goodys have won $750,000 in the lottery. They decide to buy that big in”—work from the business environment house on the corner of the street that they’ve always dreamed about. Father, Bing Goody, toward the system under test. This is the goes to the banking office to make the necessary arrangements. Of course, the bankruptcy essence of soap opera testing. that Bing had two years ago complicates the approval, but Jim, the guy at the bank, is very I’m not saying that mechanical ap- helpful and even sells Bing on the idea of a beautiful vacation property in Mexico. Bing has a proaches aren’t any good. On the con- brilliant idea. Why not ask neighbor Jones if he is interested in co-owning the vacation prop- trary, mechanical approaches tend to be erty? Like everything else with Mr. Jones, his credit is perfect. straightforward and give reasonable con- One week later: After a small party with some new friends, only about $720,000 of fidence that the functionalities in the the prize money is left, so the mortgage arrangements need to be changed. Jim, system under test are in good Mr. Jones, and Bing decide to meet the next morning to make the necessary adjustments. Since it was a great party, they are not at their best when they meet, so many mistakes are made. They correct some immediately, others they miss and will need to fix later. They find that an additional second mortgage on Mr. Jones’s home is needed in order to still qualify for the vacation home. A story like this can potentially test all kinds of objectives, such as Entering a customer with a big family Mortgage arrangements including a first and a second house Property abroad Both a primary residence and a vacation property Weighing of the income of a second owner Co-ownership on the second house only Establishing a second mortgage to finance a vacation property Bankruptcy two years ago Modification of the down payment Correcting errors upon entry Corrections after the application has been processed

32 BETTER SOFTWARE FEBRUARY 2004 www.stickyminds.com Testing & Analysis

enough shape for normal, everyday us- tiveness. However, in exploratory test- age. For many systems, such knowledge ing, test design and text execution inter- relieves a lot of concerns; if anything mingle. The soap opera testing tech- does go wrong after release, it can proba- nique, on the other hand, has separate bly be fixed with manageable impact. test design and test execution phases. Also, since mechanical test cases tend Tests are described and reviewed by to correspond to tangible things like stakeholders first, before they are (po- screens or requirements, their de- tentially much later) executed, prefer- velopment and execution are easy ably by automation. to manage. It is possible to predict how many test cases have to be de- Imagine, Weed, and Feed veloped and executed at any point in Soap opera testing gives you all of the the project. But this also leads to a confidence of mechanical testing but disadvantage of mechanical test adds the complexity necessary for highly cases: Since the individual tests are ambitious testing. It also allows you to usually small, they come in great maintain separation between test design numbers, which can be and test execution. Your testing can be hard to organize and automated but not bogged down in a Disorder Depot track, particularly when large number of tests to organize and there are multiple ver- track. Finally, it allows you to have fun BY LISA CRISPIN sions of a system under and be creative, key to getting your test- There are 20 preorders for Elite Force Avi- test, such as foreign languages. For exam- ing mind flowing. ator George W. Bush Action Figure in En- ple, when a system needs to undergo To create effective soap opera tests, terprise awaiting the receipt of the items adaptations, it will be hard to know first write the situations without worry- in the warehouse. Finally, the great day which tests are impacted. Getting the test ing too much about the technical details arrives, and Jane at the warehouse re- cases up-to-date again can become a pro- of the system under test. Try to immerse ceives 100 of the action figures as avail- hibitive task. yourself into the business the system was able inventory against the purchase or- The biggest concern I have with me- designed for and take it from there. der. She updates the item record in chanical testing, however, is the typical- Don’t try to invent completely new sto- Enterprise to show it is no longer a pre- ly low “ambition level.” By that I mean ries every time. Instead, do the same order. Some time passes, during which the degree of aggressiveness designed thing “real” soap operas do: Invent a the Enterprise background workflow to into a test to find hidden problems. fixed set of characters that experience release preorders runs. The 20 orders are Most tests I see in organizations are not (are subjected to) various events. This pick-released and sent down to the ware- overly ambitious; if all requirements are way the test cases become like “epi- house. Meanwhile, Joe loses control of met and every screen works, the tester sodes,” which are easier to come up his forklift and accidentally drives it into and the QA manager are happy. That with. For instance, in the case of a finan- the shelf containing the Bush action fig- might be fine at a certain point in the cial system, the cast could be a family ures. All appear to be shredded to bits. process, but later on some tests should with parents, children, grandparents, etc. Jane, horrified, removes all 100 items be run that aggressively pressure the sys- We’ll call them the Smith family. Depend- from available inventory with a miscella- tem to find hidden problems. Even if ing on the scope of the tests needed, sur- neous issue. Meanwhile, more orders for you don’t fix the issues immediately, it is round them with neighbors (“the Jones- this very popular item have come in to always valuable to know what can es”), the local bank office, and in case Enterprise. Sorting through the rubble, break your system. security is an issue, add bank robbers, Jane and Joe find that 14 of the action Last but not least: Always following a con men, tax officials, etc. The resulting figures have survived intact in their box- predefined process is just not much fun. “world” is rich enough to produce finan- es. Jane adds them back into available It doesn’t stimulate creativity, which is a cially relevant situations with ease. inventory with a miscellaneous receipt. key asset for a good software tester. In one episode, one of the Smiths’ You may be wondering, “What about children is getting engaged to one of the This scenario tests exploratory testing? It typically does not Joneses’ children. The couple looks for a Preorder process follow a predefined process and also house and begins applying for a mort- PO receipt process leads to testing with a higher ambition gage and a loan. However, somewhere in Miscellaneous receipt and issue level.” I agree. Exploratory testing is “ad the process the engagement is called off. Backorder process hoc” in the good sense. Thanks to the Similarly, the children might drop out of Pick-release process work done by James Bach and others, school, start working, or study abroad. Preorder release process procedures for exploratory testing are Various family members marry and di- Warehouse cancels now available that easily outperform vorce, become ill and die—all in one mechanical tests in efficiency and effec- hour, just like on television.

34 BETTER SOFTWARE FEBRUARY 2004 www.stickyminds.com Testing & Analysis

At this point, the events have not nec- not relevant, and come up with addi- nancial systems usually don’t like roll- essarily been translated into financial tional events that create extra trouble for backs). And the bank subsidiary is in the terms, let alone into transactions for the the system under test. For example, fa- middle of a re-org, resulting in changes system under test. The next step is to do ther does the finances, but he often lives in authorizations, creating even more these translations, while at the same time with his head in the clouds, thus making mistakes. A robbery results in having to weeding and feeding. Leave out what is mistakes that have to be corrected (fi- recover lost data, etc. You may have to gather more infor- mation in order to feed your tests. I’ve been surprised by how hard it can be to base tests on “real life.” In more than one case I Flights of Fancy have had a hard time placing BY HANS SCHAEFER myself into a particular envi- ronment with enough feel to work with it Hans likes to travel in the cheapest possible way, so he often combines business and holiday creatively. That’s when cooperation with trips. He lives in Norway but has to give a speech at a conference in Amsterdam. Afterwards, others, such as end users and subject- he wants to go for a holiday in Beijing. He also wants maximum benefits through his airline matter experts, is so helpful. If chal- bonus program, so he prefers to use “Star Alliance” flights. lenged, they can come up with all kinds He asks for a tourist-class flight to Amsterdam, returning through Copenhagen, Denmark. Ad- of nice anecdotes that provide excellent ditionally, he asks for a tourist-class flight from Amsterdam to Beijing two days later, returning inspiration for mean tests. in two weeks. He wants the cheapest possible price, so he is flexible about the return date, In other words, you cannot always asking for prices for three different days. His other hidden agenda is cutting corners on the just stay in your cubicle and think hard. way back from China: If the flight goes through Copenhagen, he would not need to go all the It might be necessary to go out, talk to way to Amsterdam but could just fly back home directly. people, and watch what is going on. This might need some management ap- He gets a few flights and prices, but finds it too expensive. He asks for other flights between proval (show this article to your super- Amsterdam and China, not only through Copenhagen. There is one, considerably cheaper, visor), since it looks time-consuming, through Frankfurt, Germany. He chooses this one. but it is much more likely to save signif- Then a German customer wants to schedule a meeting. Hans decides to put it into his travel icant time and money, as tests can be plans and changes the return flight, with a several-hour break in Frankfurt. A week later, the produced faster, and the tests are leaner deadline for printing the tourist-class ticket runs out, but Hans does not yet want to get the fi- and meaner. From a business perspec- nal ticket. He changes the reservation again, changing the date for the outbound journey Am- tive, there are even more savings and sterdam–Beijing to one day later. The new deadline for printing the ticket comes up, and fi- mitigation of risks and problems be- nally all the tickets are printed and paid for. cause more bugs are found and they are found earlier. Now comes the flight. The first leg is overbooked, so Hans chooses to accept the airline’s offer and takes the next Add Structure flight. However, his baggage is lost. It turns up the next day in Amsterdam, just in time for Soap operas are a kind of scenario test. checkin for the Beijing flight. This flight, however, is can- (For a more general treatment of sce- celled due to aircraft failure. His flight is rebooked to a nario testing, see Cem Kaner’s article different airline the same day (for free). This flight in the September 2003 issue of goes through Paris, but the first leg is de- STQE.) As with other kinds of sce- layed, so he does not reach the connect- nario testing, there can be pitfalls. ing flight to Beijing. Again, a rebooking For instance, manageability is a results and he finally makes it. risk for soap opera testing. Do soap operas help when we Conditions tested include have to test a large amount of Searching for alternative flights functionality? Can the Complicated connections process be kept under con- Rebookings trol from a managerial per- Not using booked flights spective? What about cover- Overbooking age? It can be hard to predict Delays and related service and control the number of tests to Lost baggage be developed and how much time it Rebooking on the way will take; plus, it is not clear what the Ticketing deadline coverage of the tests is since they are not developed systematically.

www.stickyminds.com FEBRUARY 2004 BETTER SOFTWARE 35 Testing & Analysis

To address this problem, let me first onds when 5000 users visit our home It can make sense to relate objectives talk about test objectives. Test objectives page in a 5-minute period and tests in a late stage; first develop the are basically a list of statements, usually tests, and then determine whether these about the behavior of the system, that the HELP key works for each screen in cover the objectives. When not all objec- describe minute properties that the tests subsystem B tives are matched, either extend the exist- minimally need to address. ing scenarios and/or develop additional Here are some examples of test objec- In the case of a mechanical test tech- ones. This late matching can be serendip- tives: nique, this connection will typically be a itous: things are tested that weren’t in- one-to-one relationship between test ob- tended, but it might uncover unexpected a negative value is not allowed for the jectives and tests (see Figure 1): one ob- problems (in fact, bugs are unexpected age field jective is met by one test case, and each by definition; otherwise they wouldn’t be test case addresses one objective. For a there, would they?). Another idea is to if a person works part time, pension creative technique like soap opera test- use different people for the tasks; for ex- rights are built up for the total of the ing, however, this relation is more likely ample, a more analytical person to do the hours worked to be many-to-many: one objective can test objectives and a more creative one to be tested in several scenarios, and a single come up with the stories. when server X goes down, server Y scenario might capture a multitude of ob- A system of any substantial size will takes over jectives (see Figure 2). By relating tests to need so many tests and test objectives test objectives, you can ensure coverage that you can’t tractably treat them as a the response time stays below 8 sec- for soap opera tests. single lump. Instead, break them into manageable clusters. A cluster is a set of tests with a similar and specific scope. Such a scope could be “functional tests Objectives Tests for module XYZ in system ABC” or “performance tests for Web page W.” When clustering, it is essential to not follow an exclusively top-down hierar- chy but to divide the tests according to a combination of different criteria, such as which functional area of the system is addressed or what the intended ambi- tion level of a test is. Clusters with a low ambition level and with small simple test cases typically would go first in the execution. If possible, clustering should be done prior to developing individual tests. Therefore, the result of the clustering is a Figure 1: Mechanical model—tests produced one by one from objectives list of potential cluster candidates. For each individual cluster on the list, quick decisions can be made as to if and when Objectives Soap Opera Tests it would need to be developed and exe- cuted, and who would be involved in do- ing so. Also, a choice can be made about the testing techniques to apply, like soap opera testing. I call this combination of soap opera testing and traceability to test objectives “structured soap opera testing” (see Fig- ure 3). It is lightweight enough to en- courage creativity, and also complements the soap opera technique.

Vary Complexity Since scenarios combine a lot of features, early in the test cycle when a system still Figure 2: Soap opera tests have a many-to-many relationship has a lot of simple bugs, such as non-

36 BETTER SOFTWARE FEBRUARY 2004 www.stickyminds.com Testing & Analysis

The Rented and the Wrecked

BY BRIAN MARICK

A customer named Marick hires a car for a three-day business trip. Midway through the rental, he extends it for another week. (This, by the way, gives him enough rental points to reach Preferred status.) Several days later, he calls to report that the car has been stolen. He in- sists that the Preferred benefit of on-site replacement applies, even though he was not Preferred at the start of the rental. A new car is deliv- ered to him. Two days after that, he calls to report that the “stolen” car has been found. It turns out he’d mis-remembered where he’d parked it. He wants one of the cars picked up and the appropriate transaction closed. Oh, and one other thing: the way he discovered the mislaid car was by back- ing into it with its replacement, so they’re both damaged. This scenario would test the following conditions Upgrading to Preferred status (during rental) Extending a rental Handling a stolen car On-site replacement Undoing on-site replacement Undoing handling of stolen car Return of a damaged car functioning buttons, they can be a chal- Give It a Try lenge to execute. To say it in soap opera Try to write scenarios that ited number of test cases—few- terms: It doesn’t make sense to write are (1) based on real life, (2) ex- er, more efficient tests are easier about a Hollywood wedding in a beauti- aggerated, and (3) condensed into a lim- to manage, can be executed faster, and ful church if the lock on the church door ited number of events. Cover functionali- produce fewer results to be analyzed. Use is broken and you can’t get in. The best ties that either are in the system under structure to overcome coverage pitfalls. way I know to limit this is to have both test or, even better, should have been in Finally, remember to try the lock on the simple clusters to test the basics of a sys- the system, but weren’t recognized in the church door before the big day arrives, tem and more advanced clusters, for ex- design process. Test cases should be effi- making sure nothing stands in the way of ample with soap operas, to run after the cient in design and able to inflict a lot of the great happiness that, as every soap basic tests have passed. stress with minimal actions. Create a lim- opera writer knows, is so typical for Hol- lywood marriages. {end}

CLUSTERING Hans Buwalda, now at LogiGear in Cal- ifornia (and formerly at LogicaCMG in Cluster Cluster Cluster Europe), is the chief architect for Action Develop in Parallel Based Testing (ABT), a comprehensive ...... method that is based on his ideas for cre- ative and manageable testing and test au- Test Objectives Soap Operas tomation, such as action words and soap opera testing. You can reach him through www.logigear.com.

Sticky rt Notes For more on the following topics go to Cross-Reference When Ready www.stickyminds.com/bettersoftware Soap operas and action words Figure 3: Structured soap opera testing

www.stickyminds.com FEBRUARY 2004 BETTER SOFTWARE 37