The 10-Minute Test Plan

Feature: Software Testing

perience suggested that test plans were The 10-Minute created with an uncritical eye. They’re simply expected. The first question asked when the subject of testing comes up is, “What’s the plan?” Then inevi- Test Plan tably someone accepts the task of creating it. The process of doing so creates value: writing the test plan forces James A. Whittaker, Google us to think through testing problems and potential solutions. But the actual value is in the test plan activity and // Test plans are perhaps the least appreciated of all not the test plan artifact, which often sits unused once development starts in supporting software development artifacts, so let’s spend earnest. Test planning is an incredibly as little time as possible on them—say, 10 minutes. // useful exercise, but the plan itself is of questionable value. Keeping a test plan up to date through all the various requirements, specification, and product changes is simply of too little value to be worth anyone’s time. As someone who’s worked in the testing field for much of my career, I wanted to find out why test plans die so prematurely. If test planning is truly of little value, should we even bother? Software is ubiquitous. It runs and shifts the focus—rightly so, in my Why grant tenure to a low-value pro- corporations, governments, and mar- opinion—to the care and feeding of cess simply because it’s familiar? And kets, as it has for decades, but it’s also source code. After all, the source code if test planning is valuable, how do we central to personal and social commu- becomes the product that users covet. get to that value quickly and efficiently nication, entertainment, travel, and lei- Spending a moment more on a test plan without generating an artifact that’s sure. Software is no longer a tool lim- than is absolutely necessary is a mo- not worth maintaining? I wanted to re- ited to the 40-hour work week; it’s a ment taken away from making the code think test planning from the ground up. service consumed at work, home, and the best it can be. A theoretical exercise was outside play. It isn’t a stretch to say that users Writing voluminous test plans may the skill set of my engineering team. not only depend on software but also be a prescribed burden in the life of Asking them to sit around and brain- fall in love with it. Developing beauti- software engineers working in regu- storm about why our test plans were ful and engaging software creates an lated industries or dealing with safety- dying would have produced scant in- emotional bond between your applica- critical issues. But for the compressed sight. Instead, I took a more aggressive tion and its users. shipping cycles of modern mobile and approach, using a very industry-ori- But how many users have ever fallen Web apps, we need something simpler. ented technique: I gave a group of peo- in love with a test plan? Of all the sup- A solution that compresses the time ple who report to me a nebulous task porting artifacts of software develop- spent developing and maintaining a test with unrealistic time constraints. ment (specifications, designs, product plan might even have useful takeaways plans, and so forth), the lowly test plan for those who require more rigor and Ten Minutes and No More is probably the hardest to love. Test adherence to standards.1 I chose my directions carefully: plans are written, reviewed, and often “Build a test plan for Google App En- left to rot in place as the hustle and bus- I See Dead Test Plans gine; you have 10 minutes.” Such a tle of software development progresses When I first arrived at Google, my ex- directive has no ambiguity. The peo-

70 IEEE Software | published by the IEEE computer society 0740-7459/12/$31.00 © 2012 IEEE ple I asked to participate in this ex- need verification? App Engine needed • Avoid prose and favor bulleted periment were used to taking work re- to be available, secure, and responsive. lists. Not all testers possess the quests from me. Test planning was an Chrome Web Store needed to be in- skills to adequately capture a prod- activity they understood. App Engine tuitive, appealing, and worthwhile. In uct’s purpose or testing needs in was a product they understood. As I other words, the adjectives and adverbs narrative form. Prose can be hard closed the door, sequestering the team that describe why a user would use the to read and easy to misinterpret. in their conference room, there were service in the first place are an impor- • Don’t bother selling. A test plan no questions to ask. tant part of the planning process. They isn’t a marketing document or a Ten minutes later, I opened the door represent the characteristics we’re try- place to talk about the importance and there was, unsurprisingly, no test ing to validate during testing. We must of a product’s market niche or how plan. The excuses were predictable: understand the system properties that cool it is. Test plans aren’t for cus- “We didn’t think you really meant 10 need verification. tomers or analysts; they’re for minutes.” “That isn’t enough time to Second, what parts of the applica- engineers. even document our intent.” “Dude, tion need verification? App Engine • No fluff. Test plans aren’t school seriously?” has components for creating, hosting, term projects with specific length I reset the clock but changed the ap- and scaling applications. Chrome Web expectations. Bigger is not better. plication: “10 minutes, Chrome Web Store…. Go!” You get the idea. Every 10 minutes, I changed the application target, but Every 10 minutes, I changed the task remained the same: build a the application target, but the task test plan. As the minutes ticked by, my remained the same: build a test plan. team got the message. They quickly dis- carded any idea of describing the problem, the application, or finally, any- thing at all. There wasn’t enough time. Store has a data store, discovery en- A plan’s size is related to the size They set aside prose in favor of bul- gine, and download manager. These of the testing problem, not the au- leted lists. They learned to ignore any are the nouns that name components of thor’s propensity to write. information that wasn’t relevant to test- the system under test. We must know • If it isn’t important and action- ing—meaning that if it wouldn’t appear the parts of the application that require able, don’t put it in the plan. Not a in an important test case, ignore it. If it testing attention. single word should garner a “don’t didn’t affect testing in some fundamen- Finally, what purpose does the ap- care” reaction from a potential tally important way, leave it out. By the plication serve for users? App Engine stakeholder. fifth attempt, they were able to get 80 lets users write code, serve application • Make it flow. Each section should percent through the test-planning pro- instances, and scale to meet demand. In expand on earlier sections so that cess in 30 minutes or less, documenta- other words, the verbs describing what readers can stop at any time and tion included. Not exactly a 10-minute capabilities users receive are the actions have a clear picture of the product’s test plan, but a success nonetheless. We that eventually get translated into test functionality. If they need more de- boiled test planning down to its very cases. We must know what actions the tail, they just continue reading. essence. system performs. • Guide a tester’s thinking. A good We experimented with various com- planning process helps a tester Attributes, Components, binations of these three properties and think through functionality and and Capabilities settled on a process called ACC: attri- test needs and so leads logically In all the documentation and notes cre- butes (the adjectives and adverbs), com- from higher-level concepts to lower- ated during both the false-start and the ponents (the nouns), and capabilities level details that engineers can im- 80-percent-complete test plans, three (the verbs), to be documented in that plement directly. requirement categories surfaced. order and, we hoped, in 10 minutes or • The outcome should be test cases. First, what application properties less. ACC has seven guiding principles: By the time the plan is complete,

November/December 2012 | IEEE Software 71 Feature: Software Testing

Figure 1. “Welcome to Google Sites” screenshot. Most of the product attributes are listed under “Quick facts.” Many end-user applications include similar pages that effectively identify attributes for testing.

it should clarify not just what test- Why are we building this thing? What we want to attach test cases to the ating needs to be done but also how core value does it deliver? Why is it in- tributes, so we know how much testing to write the test cases. A plan that teresting to customers? We’re not look- we’ve done to demonstrate their mani- doesn’t lead directly to tests is a ing to either justify or explain these festation in the product. waste of time. things, only to label them. Presumably, Typically, a product manager will the product planners have done their have a hand in narrowing down the This last point is crucial. If the test job in coming up with a product that list of system attributes. Testers often plan doesn’t have enough detail to tell will matter in the marketplace. From a get this “list” by reading the product us what test cases we need to write, it testing perspective, we just need to cap- requirements document or the team’s hasn’t served its primary purpose of ture and label attributes so we can en- vision/mission statement or even by helping test the application we’re build- sure they’re accounted for when we test simply listening to a salesperson de- ing. Test planning should put us in a the product. scribe the system to a prospective cus- position to know exactly what tests Attributes are the qualities and char- tomer. Indeed, at Google, we find that must be written. ACC accomplishes acteristics that promote the product salespeople and product evangelists are this by guiding the planner through and distinguish it from the competi- an excellent source of attributes. Just three views of a product: its attributes, tion. In a way, they explain why people imagine back-of-the-box advertising components, and capabilities. would choose to use your product over or think about how the product would a competitor’s. For example, Chrome be pitched, and you’re getting the right A Is for Attributes is designed to be fast, secure, stable, mindset to list the attributes. Attributes describe why the product is and elegant, so these are the attributes Some tips on coming up with attri- important to users and to the business. ACC is trying to document. Eventually, butes for your own projects:

72 IEEE Software | www.computer.org/software • Keep them simple. If it’s taking more than a few minutes, you don’t understand the product well enough. • Keep them accurate. Make sure they come from documentation or marketing information that your team already accepts as truth. • Keep moving. Don’t worry if you missed something. If it’s not obvi- ous later that you missed something, it probably wasn’t that important. • Keep it short. No more than a dozen attributes is a good target. We boiled Chrome’s operating system down to 12 key attributes and, in retrospect, should have short- ened that list to eight or nine.

As an example, consider the attributes for the Google Sites product, which is a freely available application for building a shared community website. Google Sites, as you’ll find with many end-user applications, is Figure 2. Google Sites attributes as documented in the Google Test Analytics (gTA) kind enough to give you most of its tool. “Searchable,” “sharing,” “quick,” and so forth are the actual attributes we selected to attributes in its own documenta- demonstrate in subsequent testing. tion, as Figure 1 shows in the “Quick facts” list. Indeed, most applications have some that together constitute a system and just ask each developer, “What compo- sort of “Getting started” page or sales- implement the attributes. Components nent are you working on?” oriented literature that will often do the are the shopping cart and the checkout As with attributes, the level of detail work of identifying attributes for you. feature for an online store. They’re the in identifying product components is If they don’t, then talking to a salesper- formatting and printing features of a critical. Too much detail becomes over- son or, better yet, watching a sales call word processor. They’re the core chunks whelming and brings diminishing re- or demo, will get you the information of code that make the software what it turns. Too little detail, and there’s sim- you need. is. Indeed, they’re the very things that ply no reason to bother in the first place. At Google, we use any number of testers are tasked with testing. Keep the list small: 10 components are tools for documenting ACC, from Components are generally easy to good, 20 are probably too many un- specification-like documents to spread- identify and often already cast in a de- less the system is very large. It’s okay sheets to a custom tool built by some sign document somewhere. For large to leave minor things out: if they’re mi- enterprising engineers called Google systems, they’re the big boxes in an ar- nor, then they’re either part of another Test Analytics (gTA). Figure 2 shows chitectural diagram and often appear component or lacking enough end-user the attributes for Google Sites as docu- in bug database labels or get called out value to focus on them. mented in gTA. explicitly in project pages and docu- Indeed, you should be able to enu- mentation. For smaller projects, they’re merate both attributes and components C Is for Components the code classes and objects that devel- in minutes. If you’re struggling to come The next enumeration targets are opers are creating. You’ll get the list up with components, then you’re not nouns—the component building blocks without having to do much else if you familiar enough with your product and

November/December 2012 | IEEE Software 73 Feature: Software Testing

the user a capability. Chrome renders a webpage fast and plays a Flash file se- curely. If your product does something that isn’t covered by an attribute-component intersection, it probably war- rants raising the question of why you’re bothering to include it in the product. A capability that doesn’t serve a core product value sounds like fat that you can trim—either that or an explanation for the capability exists but you don’t understand it. Not understanding your product is unacceptable. Here are some example capabilities for an online shopping site:

• Add/remove items to/from the shopping cart. A cart component intersects with an intuitive UI attribute. • Collect credit card and verification data. The cart component meets the convenient and integrated (with the payment system) attributes. • Processes monetary transactions Figure 3. Google Sites components as documented in gTA. “Nav bar,” “Sitemap,” and so using HTTPS. The cart component forth are blocks of actual code modules. Each of these components is visible from the Sites UI meets the secure attribute. or immediately available in specification documents. • Provide suggestions to shoppers based on the products they’re view- ing. The search component meets need to spend some time using it to get They represent the actions the system the convenient attribute. quickly to a power-user level. Any ac- performs in response to user inputs. • Calculate shipping cost. The FedEx tual power user should be able to list Indeed, users choose your software be- integration component meets the attributes immediately, and any project cause they want some specific function- fast and secure attributes. insider with access to the source code ality that it provides. • Display available inventory. The and its documentation should be able Chrome, for example, has capabili- search component meets the conve- to list the components quickly as well. ties to render a webpage, play a Flash nient and accurate attributes. Finally, don’t worry about complete- file, synchronize between clients, and • Defer a purchase for a later date. ness. The whole ACC process is based download a document. These capa- The cart component intersects with on doing something quickly and then bilities and many more represent the the convenient attribute. iterating as you go. If you miss an at- browser’s full functionality. A shop- • Search for items by keyword, tribute, you might discover it as you’re ping app, on the other hand, has ca- stock-keeping unit, and category. listing the components. ACC’s capabili- pabilities to perform a product search The search component intersects ties process should shake out any attri- and complete a sale. If an application with the convenient and accurate butes or components you missed earlier. can perform a task, then the task is one attributes. (In general, we prefer Figure 3 shows how Google Sites of its capabilities. treating each search category as a components are documented in gTA. Capabilities lie at the intersection separate capability.) of attributes and components. Compo- C Is for Capability nents perform some function to satisfy Obviously, the capabilities list can Capabilities are the system’s verbs. a product attribute and the result gives be long. If you feel as if you’re listing

74 IEEE Software | www.computer.org/software Figure 4. Capabilities grid. The numeric value indicates the number of capabilities that a particular component (y-axis) provides to satisfy a particular attribute (x-axis). The higher the number, the more test points for that intersection. Blank entries indicate no attribute-component link and therefore no test requirement. everything you could test, then you’re Testability Continuing with the Google Sites getting the hang of ACC—to list, Most importantly, capabilities must example, Figure 4 shows a grid with at- quickly and succinctly, the most impor- be testable. They’re verbs because they tributes across the x-axis and compo- tant system capabilities that need verifi- require action on our part—specifi- nents across the y-axis. This is the way cation to be in working order. cally, to write test cases that will deter- ACC links capabilities back to attri- Capabilities are generally user-ori- mine whether each capability is imple- butes and components. The large num- ented and written to convey a user’s mented correctly and whether the user ber of empty squares is typical because view of what the system does. Whereas will find the experience useful. This is not every component has an impact on brevity rules the attribute or compo- the primary reason to write them in an every attribute. For example, only some nent ACC stages, capabilities should active voice. of Chrome’s components are respon- describe everything the system can do. Capabilities aren’t meant to be test sible for making it fast or secure; the They’re far more numerous than attri- cases that contain all the information others will have blanks at the attribute butes and components, and their num- necessary to run as actual tests. They intersection points, representing no im- ber increases with an application’s fea- don’t require exact input values and pact and therefore no need to test this ture richness and complexity. Smaller data. If the capability is to let users particular attribute-component pair. applications I’ve worked on at Google shop, the test case will specify what they Each row or column in the capabili- have dozens of capabilities, and the shop for. Capabilities are general con- ties grid represents a slice of function- more complicated systems tend to have cepts of actions the software can take or ality that’s related in some fashion. A hundreds (Chrome OS has over 300, a user can request. They imply tests and single row or column is a good way to for example). values but aren’t tests themselves. break the application’s functionality

November/December 2012 | IEEE Software 75 Feature: Software Testing

deal of work to be done here. If you believe that some monetary transactions might be missed by, say, a new tester on the team, then you need to replicate this capability to expose the various transaction types; if not, then the general level of abstraction is good enough. Likewise, if HTTPS is something the team understands well, this capability is fine as it’s worded. Capabilities are supposed to be abstract, so don’t fall into the trap of trying to document every detail as a capability. Leave it to the test cases or to the exploratory testers themselves to provide that level of detail. (Leaving such variations to the tester also creates variety in how capabilities are interpreted and cast into Figure 5. Heat map of risk areas per attribute-component pair, for inherent risk, bugs, actual test cases, which in turn trans- and code churn. The darker the box, the greather the risk and thus the more testing required. lates to better coverage.) When tests are successfully executed, the dark areas become lighter. Finally, a capability should be composable with other capabilities. In fact, a user story2 or use case3 (or into testable sessions. A test manager the page-view/sharing pair. We can ei- whatever terminology you may prefer) might assign each row to a separate test ther write test cases for them directly should be describable in a series of team or have a bug bash to really hit or test a combination of capabilities by capabilities. If you can’t write a user a row or column hard. The grid also combining them into a larger use case story with only the existing capabili- reveals targets for exploratory testing or test scenario. ties, then either some capabilities are and, when each exploratory tester takes missing or they’re written at too high a different row or column, helps man- Documenting Capabilities an abstraction level. age overlap and improve coverage. Writing good capabilities requires Transforming a set of capabilities The numeric values in Figure 4 rep- some discipline. Three properties into user stories is an optional interim resent the number of capabilities pro- we’ve found to be useful include, first, step that can add a great deal of flexi- vided by the component on that row writing a capability as an action that bility to testing. In fact, several Google to satisfy the attribute in that column. conveys the sense of the user accom- groups prefer more general user sto- The higher the number, the more test plishing some activity, as in the page- ries over more specific test cases when points for that particular intersection. view/sharing examples. engaging with external contractors For example, the page-view compo- Second, a capability should pro- or when organizing a crowdsourced4 nent addresses the sharing attribute in vide enough guidance for a tester to exploratory testing effort. Test cases three capabilities: understand the variables involved that are specific can become boring as in writing test cases for the behav- a contractor executes them over and • Make the document accessible to ior it describes. For example, “Pro- over, whereas a user story provides collaborators. cess monetary transactions using enough leeway in deciding specific be- • Share page-management duties HTTPS” requires the tester to under- haviors to make testing more fun and with a collaborator. stand what types of monetary trans- less prone to mistakes. • View collaborator position within a actions the system can perform and Whether your ultimate goal is user page. to define a mechanism that validates stories, test cases, or both, we devel- whether the transaction occurs over oped three general guidelines at Google These capabilities must be tested for HTTPS. Obviously, there’s a great for translating capabilities to test cases:

76 IEEE Software | www.computer.org/software • Every capability should be linked About the Author to at least one test case. If the capability is important enough to document, it’s important enough to test. James A. Whittaker was an engineering director at Google before return- • Many capabilities require more ing to Microsoft as a development manager. His research interests are in devel- than one test case. Variation in the oper platforms and application development. Whittaker has a PhD in computer science from the University of Tennessee. He’s written several books, including inputs, input sequences, system vari- How to Break Software, its series follow-ups, and Exploratory Software Testing; ables, data used, and so forth re- he also wrote cowrote How Google Tests Software (Addison-Wesley, 2012). quire multiple test cases. The attacks Contact him at docjamesw@gmail.com. in How to Break Software5 and the tours in Exploratory Software Test- ing6 offer guidance on selecting test cases and thinking through data and inputs in ways that are more do is reenter a value, the impact is low. • outsource or crowdsource certain likely to turn a capability into a test If the user loses data, the impact is high. groups of capabilities. case that finds a bug. Together, these variables represent • Not all capabilities are equal; some the minimum amount of thinking a tes- ACC makes it easy to ensure that are more important than others. ter must perform from a prioritization each type of testing has minimal overlap and risk perspective. If we give these or that high-risk areas have purposeful Once the ACC document is com- two factors numerical weights and sum overlap. The idea is to be mindful of all plete, it specifies everything we could them, we can group the capabilities into the testing that takes place either by ac- test if budget and time weren’t a prob- higher- and lower-risk categories and tual testers or by collaborating groups lem. Given that both are major prob- make more organized decisions about such as beta users or a dog food team.7 lems, the next step is to associate the test coverage. We can even get views of Then you can map the testing back to capabilities with a risk and distinguish testing needs based on these values, as ACC to see just how well the actual their importance. Figure 5 shows. testing covers the testing surface.

Prioritization and Risk Our 10-minute goal leaves too little bviously, regulated or safety- References time to stack-rank the capabilities to critical software must do 1. IEEE Std. 829: Software Test Documenta- tion, IEEE, 1998. determine the order in which we con- far more than what I’ve pre- O 2. K. Beck, Extreme Programming Explained, vert them to tests. A more realistic goal scribed here. However, the core of 2nd ed., Addison-Wesley Professional, 2004. is to assess the relative importance of test planning is about discovering and 3. A. Cockburn, Writing Effective Use Cases, each capability to the overall mission. documenting testing effort, and the Addison-Wesley Professional, 2000. 4. J. Winsor, “Crowdsourcing: What It Means We do this with two simple variables: 10-minute test plan represents the core for Innovation,” Business Week, 15 June expected frequency of failure and fail- activity that must be performed. 2009, http://innovbfa.viabloga.com/files/ BusinessWeek___Crowdsourcing___What_it_ ure impact. A complete ACC is a guide for either means_for_Innovation___june_2009.pdf. With regard to expected frequency, performing testing or simply organiz- 5. J. Whittaker, How to Break Software: A Prac- as a function of complexity, usage ing it. It lets you clearly scrutinize each tical Guide to Testing, Addison-Wesley, 2002. 6. J. Whittaker, Exploratory Software Testing: rates, and so forth, how often do you area for the type of testing you want to Tips, Tricks, Tours, and Techniques to Guide anticipate the capability to fail? A ca- do—for example, Test Design, Addison-Wesley Professional, pability that establishes a network con- 2009. 7. W. Harrison, “Eating Your Own Dog Food,” nection might be expected to fail fairly • automate tests for certain groups of IEEE Software, vol. 23, no. 3, 2006, pp. 5–7. often. Capabilities that represent the high-risk capabilities; work of a new developer or that depend • write test scripts for specific groups on flaky external resources may also of capabilities that you want to test fail often. frequently; With regard to impact, if the capa- • select capability sets for exploratory Selected CS articles and columns bility does fail, how dire are the conse- testing without any further docu- are also available for free at quences to a user? If all the user has to mentation beyond the ACC; and http://ComputingNow.computer.org.

November/December 2012 | IEEE Software 77 Copyright of IEEE Software is the property of IEEE Computer Society and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.