A Publication P B RA E Un ST it T T IC es ES ti : ng

VOLUME 3 • ISSUE 1 • JANUARY 2006 • $8.95 www.stpmag.com

XP’s Balanced Approach to Test Stressing With Open Source Tools

A MMethodethod to Build Visibility Into Your DevelopmentDevelopment Process

MakingMaking YourYour QAQA EEfffoforrttss FFlyly The Importance of Life Cycle Management

A MESSAGE FROM THE EDITOR

VOLUME 3 • ISSUE 1 • JANUARY 2006

Publisher Editorial Director Better Life Cycle Ted Bahr Alan Zeichick +1-631-421-4158 x101 +1-650-359-4763 [email protected] [email protected]

Editor Director of Events Management Lindsey Vereen Donna Esposito [email protected] +1-415-785-3419 [email protected] Associate News Editor Everyone is familiar with the ground. While you Director of Circulation Alex Handy Agnes Vanek the rule of thumb that says can’t download a hard- [email protected] +1-631-421-4158 x111 at each step of the way, the ware patch, neither can [email protected] Art Director cost to fix a problem you get into a software LuAnn T. Palazzo Circulation Assistant increases tenfold, and hav- system with a wrench. [email protected] Advertising Traffic Phyllis Oakes ing to fix a problem in the And getting to the site Copy Editor +1-631-421-4158 x115 field is the most expensive. can pose a challenge. George Ellis [email protected] [email protected] (Granted, the Internet It is an unfortunate fact Office Manager/ somewhat mitigates the of life that despite lofty Contributing Editors Marketing Scott Barber Cathy Zimmermann cost of patches for de- intentions, test cycles will [email protected] [email protected] ployed software, but still, Lindsey Vereen always get truncated. And Esther Schindler Customer Service/ what patch doesn’t intro- Editor as Feldstein points out in [email protected] Subscriptions duce a glitch somewhere?) his article, “No matter +1-847-763-9692 Contributing Writers [email protected] No matter that we already know the how well the software is tested, some Alan Berg Jeff Feldstein Controller rule; we still have to file this one in the bugs will inevitably escape the testing Matt Hargett Viena Isaray lessons-we-have-to-learn-over-and-over process.” And you know those will be Steve Lemme [email protected] Tracy Ragan folder. In this month’s lead story, Jeff the expensive ones to fix. Article Reprints Feldstein reminds us that too many The solution is successful manage- Director of Editorial Lisa Abelson Operations Lisa Abelson & Co. bugs are still found late in the process, ment of the application development David Rubinstein +1-516-379-7097 when they are expensive to fix. Serious cycle. And the inducement for man- +1-631-421-4158 x105 fax +1-516-379-3603 [email protected] [email protected] performance issues or integration aging it well is an understanding of issues are uncovered in the latter part the business value for doing so. In Cover Photograph by Peter Nguyen of the development cycle and can managing the application life cycle, require a costly rework of large parts of product quality is only part of the Advertising Sales Manager the code, which will have repercus- equation, but it is undeniably a signif- David Karp +1-631-421-4158 x102 sions for the product team and the cus- icant part. [email protected] tomers and will affect when the com- Quality assurance is about doing pany receives revenue. things right the first time. While it’s a My favorite example of the increas- good idea to make sure that a product ing cost of repair comes from the can be repaired in the field, it’s a bet- realm of hardware: the Hubble Space ter idea to make sure it doesn’t need Telescope. Shortly after the Hubble to be. That means taking control of President BZ Media LLC was deployed, a flaw was discovered in the application life cycle and estab- Ted Bahr 7 High Street, Suite 407 its ability to focus—one thing you have lishing quality as a priority. That way, Executive Vice President Huntington, NY 11743 Alan Zeichick +1-631-421-4158 high expectations for a telescope to be you can catch the bugs before they fax +1-631-421-4045 www.bzmedia.com able to do. The problem got through get out the door. As Feldstein says, “A [email protected] because of—what else?—a truncated carefully planned application devel- test cycle. Getting it repaired required opment life cycle is a key requirement

Software Test & Performance (ISSN #1548-3460, sending field service out to the site— to successful delivery of on-time, qual- USPS #78) is published 12 times a year by BZ via the Space Shuttle. ity software.” Media LLC, 7 High Street, Suite 407, Huntington, NY 11743. Periodicals privileges pending at There are software problems in Success takes a team effort, involv- Huntington, NY and additional offices. space as well. Perhaps the most famil- ing product management, develop- POSTMASTER: Send address changes to BZ Media, iar is the software bug that caused the ment, test and documentation. 7 High Street, Suite 407, Huntington, NY 11743. Ride along is included. Mars Polar Lander to crash. Feldstein understands this. At his

©2006 BZ Media LLC. All rights reserved. Software I’m not sure you can say that soft- company, he says, everyone is in- Test & Performance is a registered trademark of BZ Media LLC. ware problems are easier to fix than volved with quality. That’s a competi- hardware problems, in space or on tive advantage. ý

JANUARY 2006 www.stpmag.com • 5

VOLUME 3 • ISSUE 1 • JANUARY 2006

Contents A Publication

COVER STORY Life Cycle Management: Make Your 12 Quality Assurance Efforts Fly A carefully planned application life cycle is key to delivering on-time, quality software—and the test team should be involved in each step. By Jeff Feldstein

18 XP’s Balanced Approach to Test

Better software quality will be your reward if you can achieve the right bal- ance between unit and system tests. By Matt Hargett

26 Make Your Development Process Departments More Transparent 5 • Editorial You can use Eclipse to make your Managing your software development process understandable by everyone life cycle. from developers to QA managers to business managers, even if they’re not 8 • Out of the Box IT experts. By Tracy Ragan New products for developers and testers. Compiled by Alex Handy 10 • Peak Performance 31Stress Testing, Alberto Savoia’s hot old topics seem Open Source Style fresher than ever. By Scott Barber This exercise walks you through the process of using Eclipse for controlling 36 • Best Practices Unit testing’s like a box of chocolates. Ant and JMeter to generate and perform By Esther Schindler automated stress tests. By Alan Berg 38 • Future Test Effective management of database performance. By Steve Lemme

JANUARY 2006 www.stpmag.com • 7 Out of the Box Compiled by Alex Handy Appscan 6.0 Sniffs Out Security Holes

Waltham, Mass.-based Watchfire has been completely re- (www.watchfire.com) has pushed its designed, with an emphasis Appscan security tool to version 6.0. The on ease of use and compre- most significant of the new features are hensible results. fresh remediation capabilities designed “Security teams today are to hold developers’ hands as they mud- under intense pressure, and dle through the security testing process. many cannot keep up with AppScan can detect security holes in Web applications Appscan now helps testers and develop- the volume of applications quickly and easily. ers focus on and fix the issues it finds, they need to test. Currently, instead of simply sounding an alarm bell security professionals are either catching tion testing with innovative capabilities and leaving the rest up to them. issues late in the development cycle or, that not only identify critical application Appscan is a Web applications securi- often, not at all,” said Michael Weider, weaknesses, but also provide intelligent ty testing tool that is typically known for chief technology officer at Watchfire. fix recommendations and new remedia- its speed. With version 6.0, Watchfire has “We spent more than a year working tion capabilities, improving the ease and also added 31 compliance-based tests to on this release, and we were laser-focused speed with which users are able to under- ensure that applications function within on eliminating barriers to fixing critical stand, prioritize and remediate critical the bounds laid out by the SEC, Congress security flaws, boosting the level of Web application security issues.” and other business regulatory bodies. automation and improving efficiency. Appscan 6.0 is available now directly To help users better understand and AppScan 6.0 is a significant milestone and from Watchfire. It is priced at US$15,000 target found holes, Appscan’s interface sets a new benchmark for Web applica- per seat. RadView Secure Splits Up CodeAssure The analysis tool CodeAssure with how and where to address the appli- Offers New has been updated by its creators, Secure cation security dilemma, and we believe Software (www.securesoftware.com). The a one-size-fits-all solution does not Performance tool checks code to find security flaws, address the key use cases,” said Kevin and it alerts developers and testers to Kernan, CEO of Secure Software. “Our problems that could arise as a result of expanding product suite enables every- Testing Tool sloppy code, unrestricted inputs and one, from an individual developer or unchecked registers. architect to an organization with hun- WebLOAD Analyzer J2EE Edition is now “Many organizations are still wrestling dreds or thousands of people, to fully available from RadView Software realize the tremendous cost (www.radview.com). The tool is designed and quality benefits of locat- to identify performance bottlenecks with- ing and fixing vulnerabili- in Web applications, and it can compare ties in software as early as test results from multiple machines and possible in the development angles. process.” WebLOAD Analyzer offers logging The company is now capabilities designed to help users dig making CodeAssure avail- into the nitty-gritty of each test case to see able in three different ver- just where slowdowns occur. The software sions—the Solo, Team and is also capable of recording information Enterprise editions—to bet- from multiple running Web applications ter target it toward different- in order to pinpoint which one is caus- size teams. They are priced ing problems, and why. at US$399, $3,295 and WebLOAD Analyzer J2EE Edition is $46,995, respectively. These available today; pricing starts at With a lower price tag, CodeAssure Solo is perfect for one- prices are for a one-year sub- US$31,854. person development teams worried about security. scription to the software.

8 • Software Test & Performance JANUARY 2006 AccordSQA Offers New Making Manual Testing Tool Mountains

AccordSQA (www.accordsqa.com) has works: “The first mode is very early testing,” announced the release of SmartCapture, he said, “where what you’re doing is Out Of a new manual testing and documentation exploratory testing; you may not even have tool aimed at the do-it-yourselfers in your requirements, you may be working on pro- Anthills QA department. The tool watches and totypes and very thinly implemented user records actions taken by a tester, and then interfaces—things that you would not want Urbancode Inc. has introduced a new it allows those actions to be played back to automate, because by the time you run version of Anthill Pro, the Cleveland- and documented for use by the rest of the the test again they may be different. based company’s flagship build manage- department. Anyone who has attempted “In that first mode, SmartCapture com- ment system. Version 2.5 of Anthill Pro to document a manual test case knows that pletely disappears from view; the process adds multiple new drivers for PVCS, this can be a laborious process. Smart- allows the tester to focus on the application ClearCase, Visual SourceSafe, Sub- Capture is designed to cut the process alone, not the testing too. SmartCapture version, , CVS, BitKeeper and down substantially, making it easier and sits in the background, captures the tester’s StarTeam. more comprehensible to all those involved. keystrokes, takes discreet screen shots, and “Most of the driver improvements are David Gilbert, lead developer on Smart- captures a desktop view at a frame rate that’s based on customer-based requests. Capture, said in an interview that “We configurable by the user. It uses predefined When you buy Anthill Pro, you become strongly believe in automation, but what test cases. When the tester tells Smart- part of our community and you can par- we found was there is a great majority of Capture he’s done with the test case, then ticipate in the process of improvement,” testers and organizations out there that do SmartCapture reveals itself and creates a said the company’s president, Maciej an enormous amount of manual testing. set of icons for each window in the order Zawadzki. We think that’s a good thing. Manual test- they were interacted with. Under each win- “Furthermore, Anthill has been ing is the place where the experience and dow a series of steps, keystrokes or mouse around since 2001 and has many more the skill and the knowledge of the tester clicks represents all the actions the user customers than anyone else in the mar- bring more to bear on the process than a took. Clicking on any of those nodes in the ket, so our community is the biggest and rote, methodical process. The problem we trees will bring you to the screen shot. So, has the most to offer,” he added. saw with manual testing was that everyone what that provides is a very accurate, trace- “Chances are, if you experience a prob- does it differently; there’s no standardiza- able audit trail of the transaction, and it val- lem, someone has already addressed it.” tion of method, no standard of output. idates that testing did occur and was com- Anthill Pro 2.5 is available directly SmartCapture tries to bring that organiza- plete and accurate.” from Urbancode’s Web site (www tional methodology of automation tools to SmartCapture costs between US$590 .urbancode.com) and costs US$2,499 the manual process.” and $700 per seat and is available now from for a single install, with a full year’s sup- Gilbert described how SmartCapture AccordSQA. port and updates included. Software AG Releases SOA Competency Center to the North American Market Software AG (www.softwareag.com) has Achievement Report and a host of oth- ly from Software AG. “It will be offered announced the American release of the er tools that automatically evaluate the free along with the products to which it SOA Competency Center, a large toolset information transaction systems imple- is enabled,” said Jim Fowler, the compa- designed to assist organizations in the mented within an organization. In addi- ny’s director of media and analyst rela- implementation and evaluation of serv- tion, SOACC also provides tools for tions. “So, for example, Software AG’s ice-oriented architectures. The Center developing new services based on blue- Enterprise Information Integrator or is designed to help organizations gain prints included with the SOACC. Enterprise Process Manager—those two long-term value from their SOA proj- Software AG also announced, in products now come enabled with ects, and reduce time and cost in imple- October, a new repository called CentraSite and SOACC.” mentation. CentraSite, which it developed along- Send product announcements to The SOACC includes SOA Discovery, side Fujitsu. [email protected] SOA Readiness Assessment, SOA The SOACC is available now direct-

JANUARY 2006 www.stpmag.com • 9 Peak Performance Remember Yesterday

If you don’t have a refer- experts championing for Today, these are the kinds of issues that ence folder, I highly recom- better methods, tools and development teams detect before they mend starting one. That’s heuristics to convert even tell the performance tester that the folder where I stow all today’s stumbling blocks they have put out a build that is stable the files that I can’t quite into tomorrow’s building enough to start building performance figure out where else to put, blocks for quality perform- scripts against. but know that I will want ance testing. That is a far Savoia’s presentations—“The Science again someday. Admittedly, cry from debating their and Art of Web Site Load Testing,” I rarely find what I go in importance. “Predicting How Your Web Site Will there looking for, but I The topics might sur- Respond to Stress,” “Three Web Load never fail to find something Scott Barber prise you, both because I’m Testing Blunders and How to Avoid interesting enough to make certain that you are facing Them,” “Understanding and Measur- me forget what I went in there to find in at least some of the challenges related ing Performance Test Results” and the first place…at least for a little while. to these topics on your current project, “The Science of Web Site Load Test- I stumbled across a Zip file in my ref- and because of just how far ahead of the ing”—were all based on the same erence folder the other day. This particu- rest of us Savoia was in 2000-1. I spent underlying themes: lar file caught my eye because it was sim- that period finishing up my first “offi- • Acceptable performance is defined ply called “Savoia.zip.” It was dated cial” performance testing projects. by system users, not by stakehold- 2/2/2004. “Hmmm…,” I thought as I During the ones that actually made it ers or metrics. opened the file, “What do we have here?” far enough into the development cycle • System usage models have to be There I found four articles and a presen- for the stakeholders to start focusing on realistic in order to generate mean- tation written by Alberto Savoia. The old- performance, I did a respectable job of ingful results. est was from STAREAST in May of 2000, finding critical performance issues that • To create realistic usage models when he was still CTO of Velogic Inc. could have added to the list of spectacu- and interpret the results, we need The newest was from the July/August lar launch day performance disasters to understand and account for 2001 issue of STQE Magazine (now that seemed so common at the time. both human psychology and called Better Software) after he had Of course, thinking back, I know technically complex systems. become CTO of Keynote (still before his now that I found them by applying • Without at least a reasonable days at Google, and probably before he nothing more scientific than sheer knowledge of and aptitude in had even dreamed of Agitar). dumb luck. On one project it was mathematics and statistics, results As I skimmed through the docu- insufficient network bandwidth; on will be misinterpreted. ments, I couldn’t decide whether I was another it was a dramatically under- • Not accounting for user abandon- depressed by the fact that we perform- sized thread pool; and another time it ment will render your results not ance testers are still struggling with was an 8-second round trip to the just inaccurate, but deceptive. many of the same challenges he mainframe, located on another conti- • The real reason we performance addressed, or encouraged by the fact nent. My all-time favorite was when test is to proactively address and that, by and large, it is no longer just the someone forgot to install the perma- mitigate business risks. elite performance testers who are trying nent license keys on the servers! It was While Savoia didn’t specifically men- to educate developers, testers and man- years before I figured out just how tion usability studies, he did refer to agers about the importance of address- lucky I actually had been in terms of online behavior profiling and user ing these challenges while wrestling averting performance disasters. demographics, which are fundamental with them in virtual isolation. While we These were issues that could have components of a usability study. In the still have a long way to go, it seems that easily been detected by sending an e- same presentation, he categorized currently these are the challenges that mail to the network admin to request numerical response times as “Good,” software professionals worldwide are bandwidth stats, monitoring of Scott Barber is the CTO at PerfTestPlus discussing, innovating around and col- resources while the five functional Inc. His specialty is context-driven perform- laborating over. Indeed, these are the testers were doing their initial inspec- ance testing and analysis for distributed challenges that have ever-increasing tion of the prototype, pinging the main- multiuser systems. Contact him at sbarber numbers of performance testing frame or validating software installs. @perftestplus.com.

10 • Software Test & Performance JANUARY 2006 “Borderline,” “Unacceptable” or “Virtu- materials available today focus on a par- for—user abandonment on perform- ally offline,” then left the numbers ticular system component. The way that ance test results. This is the area of behind to focus instead on these user Savoia focused on the entire system was Savoia’s work in which I’ve noticed the perception categories. In a subsequent what impressed me about his work least progress being made since I article, he expressed users’ tolerance in when I first encountered it in late 2002. became active in the performance test- terms of their expectations, perceptions Recently, another hot topic has been ing community. Maybe I should ask and tendency to stick with what they the mathematics behind performance Alberto to contribute a guest column know, as well as their uncanny ability to testing. In my opinion, no other single to reinvigorate this area. blame the owner of a Web site for poor item causes more performance-related Finally, Savoia never forgot to tie les- performance even if the issue was their surprises than “fuzzy math” (for exam- sons back to the reason we do perform- own 2400-baud modem. These mes- ple, not understanding the not-so-sub- ance testing in the first place, by demon- sages clearly demonstrate that he was tle differences between strating how a poorly per- incorporating usability information col- averages and percentiles, forming Web site can lected by someone. the relationship between adversely affect business via One of Savoia’s signature contribu- outliers and sample sizes, “major eBusiness disasters tions to performance testing is his or the intuitively mislead- • and/or chronic minor loss- “Website Usage Signature” (WUS) tech- ing effect of averaging es.” We all know stories nique, used to create usage models for averages). Savoia only In my about tragic software per- performance testing. This technique addressed this topic formance failures, yet most has been referenced in several books explicitly once, when he opinion, no of our clients and/or boss- and has been used as a basis for subse- said “the greatest oppor- es still ask us for return on quent models and techniques, because tunity for voluntary, or other single item investment analysis before it captures the essence of what matters involuntary, misuse of sta- they approve the budget most in designing performance tests: tistics is related to average causes more for performance testing realism. More than a few vocal individu- page response time.” But their applications. als (and at least a couple of tool ven- of all the books, articles performance- I’ve yet to read or dors) still challenge the importance of and presentations I’ve develop an approach to this realism in usage modeling by quot- encountered about per- related surprises calculating the ROI for ing anecdotes about critical perform- formance testing, his have performance testing that ance defects that were uncovered by the best balance of neces- than “fuzzy doesn’t involve some fuzzy executing unrealistic models. sary and appropriately math of its own. Maybe it These critics seem to overlook the applied math. math.” would be more effective to fact that Savoia advocated “unrealistic” One of the few com- use Savoia’s examples in stress and endurance tests as well for plaints I do hear about response to the ROI just that reason. The point that Savoia Savoia’s work is about the requests; he is a corporate makes about realistic usage models is math involved, yet it really • executive at least four that without realism in the usage is nothing more than basic times over, so if the exam- model, there is no way to determine algebra. People are always trying to sim- ples are good enough for him, they what the performance will actually be plify his math further. The problem is should be good enough for other corpo- in production. In the same way that that Savoia has already simplified the rate executives, right? users don’t care whose fault poor per- math from the integral calculus that it’s You know, I think I should drop formance is, they also don’t care how derived from to make it field-expedient Alberto a note and let him know that I many performance issues were found enough to apply “in Internet time,” yet think his “old” performance testing arti- and resolved if their overall experience still accurate enough to be reliable. cles just added about six items to my list is unacceptable. Several years ago I had the pleasure of column topics for this year. I’m hear- Even when Savoia focused on entire of applying some of Savoia’s work on ing lots of folks speculating that ’06 is systems, he never forgot to acknowl- user abandonment to the load genera- going to be a significant year in terms of edge users as an integral part of each tion tool I was using at the time, which advancements in performance testing. I system. This total system approach to was how our ongoing dialog began. hope what that means is that we will close performance testing is part of what Savoia didn’t demonstrate how to out this year significantly closer to actual- makes Savoia’s work so valuable. code user abandonment in tools. He izing the vision Alberto Savoia put before Certainly, performance testing is not a did something more important by us more than five years ago. “black box” activity, but the vast majori- explaining the implications of poorly Now, what was I looking for in my ty of the training and instructional accounting for—or not accounting reference folder? ý

JANUARY 2006 www.stpmag.com • 11 Life Cycle Management Will Help You Achieve Total Software Quality, From Beginning to End carefully planned application development life cycle coherence and overall quality of the final product from a customer’s point of view. A is a key requirement to successful delivery of on-time, It is widely understood that the earli- er a bug is found, the cheaper it will be quality software—and the test engineering team should be to fix. Bugs can be detected and resolved even in the requirements phase. Fixing involved each step of the way. (or avoiding) a bug during this phase By Jeff Feldstein ensures that it will be fixed as inexpen- sively as possible. At first glance, a test Software quality is a team effort. Every finding bugs, ship the product. engineer might ask how he can find bugs member of the product team, whether The problem, of course, is that many on paper without even a prototype, but they are a product marketing manag- bugs are found too late in the process, they are there in the requirements, if you er, development engineer, test engi- when they are expensive to fix. Serious know what to look for. neer, technical writer or customer sup- performance issues or integration issues When looking for requirements port engineer, plays a vital role in the are uncovered too late and can require a bugs, my recommendation to test engi- quality of the software delivered. Each significant rework of major sections of neers reviewing a PRD is to ask them- person involved in the project must the code, causing repercussions through- selves the following questions: constantly keep quality at the fore- out the product team and the customers, • Are the requirements clear and front of their thinking. and possibly even impacting the compa- understandable? Most software development projects ny’s revenue flow. • Are the requirements testable? are made up of four major teams: prod- A carefully planned application devel- • Are there conflicting requirements? uct management, development, test and opment life cycle is a key requirement to • Do the requirements, taken as a documentation. Holding the teams to- successful delivery of on-time, quality whole, describe a cohesive system, gether and keeping them on track is software. This article focuses on the role or do they seem fragmentary? usually the role of a program manager. the test engineering team plays during If the test engineer does not under- Each of these teams, as well as the pro- each step of the product life cycle. stand the requirement as written, it is gram manager, looks at a software proj- probably unclear to others as well. Each ect from different viewpoints, but each Life Cycle Phases person who reads the PRD must inter- of these viewpoints plays a key part in The product development life cycle con- pret it in exactly the same way. If a devel- assuring software quality. Other teams sists of four broad phases: requirements, oper interprets a requirement differently that may be involved in developing the development, test and post-ship. Each then a tester, there is a high risk that the product include customer support, phase has important activities that di- requirement either is not being met to value-add partners and, at times, cus- rectly affect the quality of the delivered the product manager’s expectations, or tomers themselves. The members of software. This article will explore each is not being properly verified. This can these teams can also contribute to ensur- phase in detail from a software quality lead to bugs or issues remaining hidden ing software quality. perspective, describing the role that a until very late in the process. This bug Quality cannot be tested into a prod- test engineer or a quality assurance de- will be found only after the requirement uct; it must be emphasized, monitored partment would play during each phase. is developed, the test is written, and the and measured from the beginning of the entire product is handed off to test; the project. The idea of “quality control,” Requirements Definition test is run, and a bug is opened. where a person or a team checks at the The software life cycle typically starts with The first opportunity to start the dis- very end of the manufacturing or devel- a product requirements document, or cussion about the various interpretations opment process for quality, then accepts PRD. This document, authored by a of the requirement is happening ex- or rejects the final product based on a product manager from the marketing tremely late in the process. Ensuring that series of measurements, does not work department, describes the product the requirement is clear and understand- well for software development projects. requirements. The developer reads able, before even a line of code is writ- Bugs, performance issues and system through the document trying to figure ten—or for that matter, before even a integration issues are far too expensive to out how to build what the product man- functional specification or test plan is correct so late in the product life cycle. ager wants in the product. The develop- written—can avoid a potentially expen-

l Traditionally, the test team’s role in er might immediately be thinking in sive fix much later in the product devel- h o P ensuring product quality did not start terms of objects, data flow, methods, pro- opment life cycle. m i T

d until the product was handed off to test. cedures and data structures that might One aspect of the test team’s job is to n a

e Testers would install the product (if be needed to meet the requirements. He verify that a product meets the require- c i t n

e they could); try out all the features; per- might also immediately start considering ments specified in the PRD. In order to r P form a round of system testing, includ- resource utilization and performance m a h Jeff Feldstein manages a team of 40 test a ing performance checks; try to break issues such as CPU, memory, user net- r

G engineers for Cisco Systems working in the y the product by acting like a dumb user work utilization, security and user b

s U.S., India and Israel. A frequent speaker at h p or giving bad input; maybe supervise response times. With all of these deci- a

r test and user conferences throughout the g o some external testing (beta or early sions to make, it is often difficult to step t U.S., he can be reached at [email protected]. o h

P field trial)—and, when they got tired of back and think about the testability,

JANUARY 2006 www.stpmag.com • 13 LIFE CYCLE MANAGEMENT ensure verification, the requirements through, no matter how carefully or how explained further below. need to be deterministic and “testable.” many times the document is reviewed. When reviewing the functional spec- A testable requirement is one that can But any conflict caught and rectified dur- ification (FS), test engineers should ask be verified (labeled Pass or Fail) by a ing the product definition phase will themselves the following questions: test case or a suite of test cases. ensure smoother execution of the later • Is the specification clear and An example of a bad requirement phases of the product life cycle. understandable? would be: “The GUI must be intuitive Another valuable input that a test • Does the specification address all of and easy to use.” Since this requirement engineer can give while reviewing a PRD the requirements listed in the PRD? is subjective and not precise, there is no is to provide feedback on whether the • Are there any extra functions not way to verify whether the product is meet- requirements, taken as a whole, define a required by the PRD? ing this requirement. A better require- cohesive system. A requirements docu- • Does the specification address the ment might be: “All data entry screens ment often begins with some introducto- related requirement fully? should have a button labeled ‘View ry information on the philosophy of the Shopping Cart,’ which resides in a con- product, and customer concerns and stant location in the window. ‘View needs that the product being described Shopping Cart’ must lead the user, with a will address. A test engineer can use his single click, to their current shopping or her knowledge and experience of the cart.” To verify that this requirement is customer, previous versions of the prod- met, a test engineer might design a sce- uct and familiarity with the competition nario that visits every GUI screen, ensures to inject an impression as to whether the that the button is in the correct place, specific requirements, taken as a whole, and makes sure that clicking on the but- meet the goals described. This impres- ton navigates to the correct screen. sion, input and feedback might not be as The detailed description of the rigorous as the previous questions, but requirement also suggests that the when taking this view, a test engineer requirement has been more thoroughly might be able to spot a missing require- thought through, and that there is a ment or a requirement that is insuffi- higher degree of likelihood the product ciently stated. will meet both the product manager’s For example, if the introductory sec- and customer’s expectations. tion describes “serving a company with It is especially important to ensure 10,000 to 30,000 employees with up to that expected performance require- one-third of these employees accessing ments, such as hardware sizing, user the system at a given time,” but the details response times, database capacities and later say that the “system needs to scale to scalability requirements, are clearly 25,000 simultaneous users with no signif- defined and testable. icant performance degradation,” the test Any environmental assumptions, such engineer may have spotted a problem as required operating systems, browsers, concerning the cohesiveness of the Web servers, database engines and other described system. The distinction here software, need to be clearly defined in between conflicting requirements and f f o t

the PRD. Without this detailed informa- cohesiveness might blur at times, but it is s e W

tion, not only will verification be difficult, important to catch both of these error l e a h but the test engineer will not be able to types as early as possible. c i M

accurately estimate the test effort. y b

h

PRDs often contain individual re- Development p a r g

The development phase typically starts o quirements numbering in the hundreds. t o h

When a large number of requirements when the PRD is approved and the pro- P are present, it is not hard for one require- gram or project is committed. Within • Does the specification describe a ment (or set of requirements) to be in the development phase, some variation testable architecture? conflict with requirements in another of the following steps usually occurs: • Is all the information required to section of the document. It is important • Functional specification write test cases present in the spec? to have at least one test engineer read • Coding Just as the PRD must be understand- through the entire PRD with an eye for • Unit test able, the functional specification must be checking for conflicting requirements. • Integration test clear to all readers, not just the develop- Examples of conflicting requirements • Base-level performance test ers. All of the consequences of an unclear might be a screen mock-up that is missing Note that although three of the steps PRD can happen during the FS review as the required “View Shopping Cart” but- above contain the word “test,” we have well. It is important to encourage the test ton, or has it showing in the wrong loca- not yet reached the test phase. It is engineers to ask any questions they might tion relative to other screen shots. important from a quality standpoint that have about unclear sections. If they do In this exercise, success is not neces- unit, integration and rudimentary per- not wish to ask a particular question in a sarily measured by avoiding all PRD con- formance testing be done during the large review situation or in a meeting flicts; a few will probably always slip development phase. This concept is with many others present, they can either

14 • Software Test & Performance JANUARY 2006 LIFE CYCLE MANAGEMENT talk with another test engineer to see if through all subsequent steps. It is best to interpretation of the FS than a product that person has the same question, or have a database to track these issues, but manager or another developer may seek out the author of that section to dis- when one does not exist, it should still be have. cuss the confusing issue. done manually, by spreadsheet or by any The specification should describe a Even when there is a formal review other means available. testable architecture. The testability of phase, with tracked comments and Traceability is important, because you the application can have a profound questions, there is still no substitute for do not want the product manager or, effect on its overall quality. An applica- having test engineers and development worse, the customer finding out after the tion that is designed for testability will engineers working one-on-one during product has been developed that one of allow bugs to be found earlier, will the very early phases of the project. the requirements has not been met. enable a greater degree of automation Formal process is not a substitute for a Again, it is much cheaper to find and fix (increasing the accuracy and efficiency close working relationship in a team the functionality holes during the speci- of repeated tests) and will free up the fication phase of the product than after test engineer to write more sophisticated coding is done. testing instead of spending time figuring Extra functionality should not be out a way to automate a simple scenario. overlooked as a serious quality concern. The FS review is, of course, far too It is often tempting for an innovative late in the development process to and creative development engineer to begin discussing product testability, and add functionality that has not been some testability issues will not be deter- asked for. The problem with this is that minable until a more detailed design is it will distract the development team produced, but this is a good place for a from writing the required features, it checkpoint designed to ensure that pre- will introduce more bugs into the sys- vious discussions were incorporated. tem, and, since the requirement is miss- A full discussion of testable architec- ing, it will be difficult to understand tures is not appropriate here, but for how to test the feature, because there is the most part, the characteristics of a no requirement to test it against. testable system also make sense from a The proper place to add new prod- software engineering point of view and uct features is in the PRD, and the will have benefits throughout the prod- development engineer had the oppor- uct. One example of this is building a tunity to pitch the idea to the product user interface (graphical, command- manager during the review of the PRD. driven, voice-activated, etc.) that is sep- Extra features added at this point are arate from the actual business logic. already more expensive than before, The interface sections should simply because it requires another revision of have the job of taking input from and dis- the PRD, and that section, at least, will playing output to the user. Any other need to be reviewed by the other teams, work (input verification, business logic, with the same questions previously error checking, etc.) should be done by raised and answered. the back end or the engine. The UI and It’s important to determine that the back end should communicate through a specification addresses the related re- well-defined and fully specified API. This quirement fully. When reviewing the architecture makes sense even if testabili- specification of a feature, the test engi- ty were not an issue, because it allows UI neer may wish to go back and reread the changes (or managing multiple UIs) to requirement. This will ensure that the occur in isolation from any engine work. feature as specified completely meets the Test can also make use of this requirement it addresses. If the require- approach by using the API to fully test atmosphere. Close interaction allows ment is only partially met or does not the back end of the product, independ- the engineers to begin to come togeth- seem to be met at all, the section will ently test the UI portion, and bring the er as a team, develop a relationship with need to be clarified. As test engineers, we two together for testing only after each each other and work toward a common are hoping that the product manage- piece is working independently of the goal. All of this will pay off when you ment team is reading the FS as carefully other. Since UI design is often fluid, encounter the frictions that inevitably as we are, but we should not count on it. controversial and subjective, having a arise in the testing cycle. In case an important detail is missed, this solid separation between front and back Requirements traceability is an im- is a good time to find it, and it really does ends makes sense for both the develop- portant part of the product development not matter which department requests ment and test teams. Other examples of life cycle. Traceability is the ability to the clarification. testability often have the same effect. trace a requirement from the PRD to the In addition, test engineers are con- It’s important to ensure that all the FS, to the test plan (including specific stantly keeping in the back of their information required to write test cases test cases) and to the test results. There mind the set of test cases they will write is present in the specification. Detailed must be a direct connection for every to verify, stress, scale and break this fea- test plans are typically written based on requirement so that it can be tracked ture, which forces a slightly different the PRD and the FS. Within these two

JANUARY 2006 www.stpmag.com • 15 LIFE CYCLE MANAGEMENT

quent product builds. customer support) a view into how test- Integration test is built on the same ing will be performed and allows them to technology and automated for the same be sure that all of the major areas are reasons as the unit test. The difference covered. Review of the test strategy pro- here is one of emphasis on what is being vides a forum for feedback and sugges- tested. For integration testing, develop- tions on test approaches and techniques. ers write code to test the integration The detailed test plan describes the between two units. In other words, if unit specific script or scenario for each case. A calls unit B for certain kinds of trans- The word “script” refers to detailed doc- actions, an integration test of unit A will umentation that describes step-by-step execute those transactions. In a unit test, instructions for validating one or more unit B’s code will usually be stubbed out features of the software. We are not in order to perform a test on unit A in referring here to automation, which is isolation. In the integration test, both sometimes referred to as “scripts.” The units will be executed. test plan is the test department’s equiva- It is also advantageous to the entire lent of the FS. It needs to specify the test- project if the development team runs a ing that will occur as completely as possi- set of performance tests before handing ble. All preconditions, the script details off the code to test. This will check the and the expected results should be pres- reasonableness of the performance of ent in the test plan. The test script the functions and will allow develop- should be written regardless of whether ment to have its first early examination the test case is going to be manual or of overall performance. This can then automated, but the plan should indicate be matched against the requirements whether the test case will be automated from the PRD to see if the project is on for this release. track. This is also a good time to deter- The test plan should include “nega- documents must be all of the informa- mine whether issues can be resolved tive” tests (meaning handling errors, tion required to write test cases. Writing with minor changes or if major rework unexpected events and bad input the detailed test plan—and, often, is required. properly without adversely effecting implementing automation—need to The test department will typically the system). The importance of nega- take place at the same time as the system write two major documents. The first is tive tests should not be overlooked. is being coded. If a test department is the overall test strategy. This is authored Many bugs are discovered by a cus- planning on API testing, the FS must during the writing of the tomer not understanding contain the complete function or functional specification a feature and selecting method signature of all published meth- and is produced shortly wrong options, or by an ods. Expected outputs and handling of after the functional speci- unexpected input from errors should be detailed in the FS so fication. It will outline • the environment. Low that test plans can be generated from it. goals for automation, how memory or disk resources, Ideally, coding of the application and automation will occur, the Requirements user-response timeouts the writing of test cases (both manual tools to be used, the test and network interrup- and automated) will happen simultane- lab equipment and topol- traceability is an tions are a few examples ously. For scheduling reasons, these ogy, and how the system of environment problems activities should not occur sequentially. will be tested for security important part that the software should The more time spent up front on design- vulnerabilities. This docu- be able to protect itself ing the application, the smoother the ment describes how the of any software against. execution of the coding phase will be. test will occur and the Just as documents writ- When a developer checks in his mod- major test types, including life cycle. ten by marketing and ule or unit, the unit test software should functional, system, per- development are ap- be checked in at the same time. This formance, scale, stress and proved by test, both the ensures that a certain number of bugs soak. It will describe, in a test strategy and the have already been found, and some cod- general sense, how these • detailed test plan should ing has taken place to check for future tests will occur. be reviewed and approved regressions on that unit. By writing the The other goal of this by the entire product unit test code in a system such as JUnit document is to have enough informa- team. This is marketing’s opportunity to (for Java applications), and by deliver- tion in it so that the test effort can be esti- obtain a level of confidence that the ing the test code with the application mated. Test effort includes number of product will meet its requirements. code, the development team is thus testers, time to implement and run the It is a good idea for developers to freed up from the laborious task of doc- tests, money and time to set up the test pay careful attention to the test plan, umenting and demonstrating its unit laboratory, the strategy for external test- because they can give good feedback testing results. Automating the unit tests ing (beta and/or early field trials). This on whether the test cases are valid (Do allows for an automatic unit-regression document also gives the other teams the scripts make logical sense?) while facility that can be run with all subse- (product management, development, checking for missing test cases. In addi-

16 • Software Test & Performance JANUARY 2006 LIFE CYCLE MANAGEMENT tion, simply by reading and under- • Upper management pressures to Post-Ship standing the test plan, the developer release for business or revenue Many people might ask: Why continue may code in such a way as to be sure reasons. testing after a product is released? This the test case will pass, and therefore • Product managers pressure for is, however, a good opportunity to find avoid a bug report entirely. release (they are anxious to please additional bugs. The automation can be their customers). run continuously, perhaps using some Test Execution • Fatigue sets in for the development model-based testing if it is available, for Now that all of the documents are writ- and test teams. an extended time. Typical bugs found ten and approved, the timelines are The best way to avoid shipping after shipping might be very slow mem- understood, and deliverables are because of the latter three reasons is to ory leaks, new security vulnerabilities or detailed, it is time for the test execution ensure that the measurable, objective performance issues related to running phase. During this time, developers are product quality goals are agreed upon the software continuously over an coding, and testers are coding the at the beginning of the project. extended period of time. With good automation. This will continue until Some suggestions for shipping automation, the cost of this is relatively development is ready to hand off either goals are: small; just set up a system, check it peri- a distinct component or possibly the • All system crash and “must fix” odically, and learn from what happens. entire product to test. This milestone, bugs are resolved and verified. Lessons learned during this phase can Handoff to Test (HOT), is an be brought into prerelease important date from a project testing of future versions. management and scheduling No matter how well the point of view. It says that the software is tested, and no software is complete, all fea- • matter how carefully the prin- tures are implemented, and ciples above are followed, the package is ready for for- No matter how well the software is some bugs will escape the mal testing. Any bugs found in testing process. These bugs or after HOT should be for- tested, some bugs will inevitably will be found and reported by mally tracked in the bug track- the customer. It is important ing system. escape the testing process. to have a process in place to Product teams (including constantly monitor incoming test) tend to want to jump to customer-found bugs, analyze the execution phase as quickly them, figure out why the as possible. Management, in- escape occurred, and ensure vestors and customers often • that this bug is caught in test- pressure teams for demonstra- ing subsequent releases by tions and visible progress. The problem • All fixed bugs are verified. adding test cases to future test plans. is that the faster the software moves to • All customer-found bugs, from the execution phase, the higher the risk previous releases, are fixed and A Team Effort of poor quality, serious performance verified. I will end where I began: Software qual- issues, integration mismatches or sched- • All performance goals are met and ity is a team effort. Having just one team ule delays. results documented. assuring software quality does not work Bugs found during the execution • Total number of open bugs is not as well as having everybody on the larg- phase of the project are inevitable, and to exceed XX. er team dedicated to it. Each individual although they are more expensive to fix • Defect density is not to exceed X team in the development process at this point (a bug is filed, code must bugs per thousand lines of code should have an equal say in all major be changed, regression testing reexe- (KLOC). product decisions. cuted and the fix verified), it is still • Code and branch coverage of tests The best way to achieve high-quali- cheaper to fix bugs now than after the is performed to specific targets. ty, on-time software is to expect owner- product ships. Bugs that are shipped The specific numbers above might ship of all goals and schedules from incur the additional expenses of inter- vary for each project. A new product or each member of the team. Each action with customer support person- company, where immediate revenue is department’s work is reviewed by all of nel, interrupting development and test important and the customers (possibly the rest to get inputs from the entire activities on future releases to repro- early adopters) may be able to tolerate a team. Buy-off and agreement are duce the issue in the lab, fixing the bug, higher bug rate, may set different goals emphasized, as opposed to dictating verifying it, rebuilding the system, and for the release than a company upgrad- either of these on the part of upper rerelease and redistribution of the fix. ing an established product, where cus- management or various department In addition, customer satisfaction can tomers have a large investment in the members. By following the guidelines be adversely effected with any bug they current product and upgrades can be above, the hope is that we can improve might find on their end. expensive. The emphasis, however, quality, meet or exceed customer Now that we have been running tests should be toward the team committing to expectations and significantly con- and finding and fixing bugs, four things objective goals and continual improve- tribute to increased customer satisfac- will occur simultaneously: ment, and ensuring that the product’s tion, leading to improved business or • Product quality improves. quality increases with each release. department success. ý

JANUARY 2006 www.stpmag.com • 17 Software Quality Benefits If You Weigh In With Both Unit and System Tests

tay on the ball by “unit test first,” because of the high code coverage that can result due to S maintaining the right testability being designed in and imme- diately used. A nice side effect of this is balance of programmer that objects are loosely coupled, allow- ing for easier reuse, and other design best practices fall out as well. and user tests. By Matt Hargett Unit tests are also fast. We ran about a thousand unit tests, which executed in Test-driven development (TDD) requires approximately 10 seconds, that helped that a test to verify functionality be writ- us catch numerous regressions as we ten before the code that implements were coding. Because unit tests run so that functionality is written. When some fast, developers can run them many people implement this idea, they lean times a day as they code. And since they toward the idea of “unit test first,” which are in code themselves, and since there requires that a failing and sometimes are free and open-source unit-test noncompiling unit test be written frameworks for nearly every language before writing the code that would and environment, developers don’t make the test pass or compile. have to bother themselves with overly While unit tests are widely viewed as distracting and expensive tools. a testing technique, those of us inti- The overall advantage is cleaner mate with these practices rely on them code up front; the ability to fix bugs and more in the sense of a design tech- optimize that code without the fear of nique. In Extreme Programming (XP) regression; the ability to refactor and terminology, these unit tests are called eliminate duplication more easily programmer tests. On the other side of thanks to a loosely coupled design—in the spectrum are what XP calls customer short, develop better code, faster. In my tests, system-level tests that can be exe- experience in several product cycles, cuted against the software, usually in an you end up moving more quickly automated fashion, to verify that the despite the extra work, because many of requested functionality is in place. XP the things that slow development down requires that you do both programmer near the end of the cycle are greatly tests and customer tests, for a number minimized. of good reasons. Customer (or system) tests allow the customer to give the team a benchmark Why Do Both? for completion. When customer tests Programmer (or unit) tests allow pro- are automated—definitely a worthwhile grammers to put a safety net in place investment—you preemptively prevent that allows for refactoring, optimiza- regressions in customer-specified func- tion, and bug fixing to take place more tionality. This can prevent a great deal quickly and without fear of unknown of unnecessary back-and-forth commu- regressions. This is especially true of nication overhead during development,

18 • Software Test & Performance JANUARY 2006 and can guarantee that when you think the components testable in this way. you are done (that is, the customer tests It may sound like I am confusing pass), you really are done. programmer tests, which require clean That isn’t to say that there shouldn’t code-level interfaces, with customer be good, healthy communication dur- tests. Customer tests can’t be automated ing the project—just that you can save with scripts if everything is in a Java yourself some time while doing it. This Applet in a browser. You could certainly effort can reduce fear and allow for invest umpteen thousand dollars into a more aggressive planning, assuming GUI testing tool, but the time would your estimates are proving to be accu- probably be better spent at this stage of rate. Moreover, extra speed is gained, development by creating a command- even though more work is being done. line, HTTP, SOAP or other open inter- Building this kind of automated test- face to the business logic. Sure, a com- ing into a project up front can be a great ponent of the customer’s acceptance boon to QA engineers, who can thereby tests might be a cornflower blue button focus their efforts on testing infrastruc- in the lower right corner of the dialog ture and documentation until the cus- box with the title “About.” There are tomer tests pass. Testing a product that tools for doing that kind of thing, but doesn’t pass minimal automated tests is they don’t require a lot of infrastruc- usually not a wise use of QA time, in my ture beyond an automated build and experience. The overall advantages are deployment of the application being making sure the customer gets what he developed. wanted on the first try, building in auto- The second, perhaps non-obvious, mated tests up front, and making effec- answer is political resistance from tive use of QA engineers’ time in the developers, or the perception that writ- early stages of development. ing unit tests slows down development When trying TDD, why do most overall. An excuse I have often heard in teams not employ both practices? The the past was that we would need to first and most obvious answer is time build a test infrastructure, and we did- pressure. Managers and engineers n’t have time to do that. This excuse is might begrudgingly try one of the prac- pretty bogus, since there are many free tices, having enough long-term vision open source frameworks for unit test- to see the upside, but not enough ing for nearly every language and plat- vision to see the upside of using both. form imaginable. If people choose one, they generally Another common perception is that lean toward customer tests rather than a given problem domain or environ- programmer tests. In my experience, ment is “too complex” to have TDD this is usually due to cultural issues that applied. I have applied TDD to a bina- boil down to the mantra “All testing ry static analysis product that had a belongs in QA.” common framework decoupled from When teams having this attitude the platform-specific (SPARC, MIPS, (which generally trickles down from Java, x86, etc.) emulators—generally management) implement TDD, they more complex than the usual examples generally miss out on many of the major of where TDD supposedly cannot

advantages. Also, many organizations do n a

Matt Hargett has successfully architected, T

n

not have QA engineers with the techni- n

implemented and executed black-box, white- y J

cal skill or domain knowledge to write box and security QA methodologies over his k o K

automated customer tests. QA engi- 10-year career. In recent years he has begun y b

n

creating security QA tools and implementing o neers might be able to pick up an exist- i t a

agile software processes such as Extreme r t

ing suite and interfaces, but it is really s u l

Programming. You can contact him at l i

up to the developers to create and main- o t

[email protected]. o h

tain the external interfaces that make P

JANUARY 2006 www.stpmag.com • 19

Register Now! Early Bird Rates End Jan. 20! www.S-3con.com TEST-DRIVEN DEVELOPMENT apply. On the embedded software side, That was a well-funded start-up The test fails—none of the charac- my partner has applied TDD tech- company, though. This time, we were ters are being filtered! I went to MSDN niques across two different embedded barely a company at this stage. I con- and reread the documentation for the proprietary systems software projects. vinced myself to forgo TDD and unit String.Replace class more carefully. I These perceptions about programmer tests and implement the minimal user couldn’t believe how dumb my mistake tests are understandable to some stories and ship them as fast as we was until I realized I had just been work- degree, and can generally be resolved could. To help make up for this, we ing too hard for too long. The fix to with a bit of developer education and a had a fairly open beta and invited our make the test pass looked like this: team with an overall good attitude. peers in the security community to play string filterSQL(string tainted) When people are re- with the application via { sistant to customer tests, it the Web UI while it was in string = string.Replace(‘\’’, ‘_’); is usually an indicator of a progress. This helped us string = string.Replace(‘;’, ‘_’); string = string.Replace(‘%’, ‘_’); more sinister problem: find a great number of the building of a wall • bugs, which was a mostly return string; } between your team and positive experience. the customer. This sounds Having a small Unfortunately, even It was at this point that I decided we unlikely, but it does hap- though I had a great deal couldn’t afford not to do TDD, and pen, and it’s not accept- team with few of experience in securing thenceforth we did it faithfully until the able for several clear rea- (and breaking the security project ended. sons. Sometimes this programmers of) Web applications from It never ceases to amaze me the little occurs on the customer the previous job, there was problems that can conspire to be a big side, and generally not doesn’t mean you a massive oversight in the problem, and how reliably TDD can from malicious intent. Web UI’s code, which was catch things before they become a big Getting customer tests can’t afford to do written in C# and problem. We shipped our product on from the customer, even if ASP.NET. One of our time and had several customers buy as your team has to imple- TDD; it means peers from the security soon as we released. We had met our ment them, can help pre- community was trying to hard ship date (we were almost out of vent scheduling issues you can’t afford SQL-inject various fields seed money), while doing TDD for four and the circus that might in the Web UI and out of the six months of our develop- result. If the customer not to do it. thought he had found our ment time. can’t specify their accept- application to be vulnera- Since this was a security product, a ance criteria, you (again) ble. We actually weren’t public vulnerability in the product once probably have a disaster using any SQL—just XML it shipped would have meant massive lying in wait. files—but either source of brand damage. We wouldn’t have been “We don’t have time • data is accessed the same able to make the volume of sales we did for this” is not a reason. in the ADO.NET APIs. if we had to spend time recovering from When first starting a self-funded This is what caused the SQL injec- such an exposure. company, I was the sole developer. It was tion attack to yield a SQL-like error Because of this, I now tell everyone the first time I had done development from ADO.NET. This was reported on a that having a small team with few pro- on something totally new, building from private mailing list of peers, but I was grammers or no QA doesn’t mean you the ground up. I had only done profes- still highly embarrassed, because I was can’t afford to do TDD; it means you sional development in the guise of sure I was filtering the user input cor- can’t afford not to do TDD. white-box QA previously, although I had rectly before passing it on to ADO.NET. extensive experience on open source I read and reread the code below trying Know the Boundary projects. Having been in QA for seven to figure out the problem: Where is the boundary between system years before this, much of it in white- tests and unit tests? string filterSQL(string tainted) box QA, I was seeded with the experi- { One of the primary rules for pro- ence of finding problems in code and string.Replace(‘\’’, ‘_’); grammer tests (a.k.a. unit tests) is that therefore had some idea how to avoid string.Replace(‘;’, ‘_’); they need to run really fast. The pri- string.Replace(‘%’, ‘_’); the problems. Being the only developer mary reason for this is so that program- and having a limited amount of seed return string; mers will be able to run them easily } money from my business partners, there many times during their programming was no time to waste—we had to ship I was really stumped, and getting very day. Long-running tests should be a and sell a product in six months. frustrated—the code was so simple! I part of the customer tests (a.k.a. system At my previous job, we had just start- decided I would write a little unit test for tests). ed applying unit tests to some legacy this very simple function using NUnit. Customer tests, while generally auto- C++ code to allow for safe optimiza- mated to the same degree as program- [Test] tions, which resulted in a 50-fold per- public void filterSQL() mer tests, can take much longer to run. formance improvement with no func- { For instance, we applied TDD tech- tional regression. (A junior QA engi- const string tainted = “‘;%”; niques, as a part of our XP efforts, to Assert.AreEqual(“___”, filterSQL(tainted)); neer and I pair-programmed those unit } our binary static analysis product. We tests together.) wrote both programmer tests and cus-

22 • Software Test & Performance JANUARY 2006 TEST-DRIVEN DEVELOPMENT tomer tests: The programmer tests were started with a failing unit test. After a little while, we were disas- written in NUnit (www.nunit.org), an We wrote a unit test that would veri- sembling the functions for valid files open-source unit testing framework fy that the binary was valid. If the bina- and rejecting invalid files. This had for .NET; the customer tests were test ry was invalid, an exception would be worked out quite well in unit tests so programs (based on real vulnerabili- thrown. far, so we saw no reason to deviate from ties, or the actual vulnerable programs this behavior—that is, until we actually [Test] themselves) compiled with different public void validElfBinary() started implementing and testing a compilers at different optimization { more complicated analysis across lots levels and customer-specified target Binary binary = new of files. programs. Binary(@”c:\valid\bin”); We were loading the file, disassem- Sounds pretty straightforward, but } bling, and sometimes deeply analyzing in the beginning, the line between pro- Once we created the scanner object a file for each test. Pretty soon our unit grammer tests and customer tests be- with the appropriate constructor, this tests took minutes to execute! This came somewhat blurred as we heinous- test immediately passed. The object and doesn’t seem like a lot, but when unit ly violated the primary rule mentioned constructor didn’t do anything, but the tests take that long to run, they get run above. test only required the object not to less often. When they get run less often, Unit testing frameworks are very crash. The next obvious step is to give it little problems build up until they are common and exist for just about any an invalid binary and make sure it does run, and then programmers lose pro- language and environment you can crash. ductivity as they unravel their changes think of. PyUnit for Python, JUnit for since the last passing test run to figure [Test] Java, NUnit for .NET, cppunit for C++ [ExpectedException(typeof(InvalidFileException))] out where they broke things. What do and so forth. I have personally used public void invalidElfBinary() we do now? NUnit and cppunit with great success on { The end-to-end analysis of binaries, Binary binary = new Binary(@”c:\windows both preexisting and new code. They \notepad.exe”); we realized, it totally inappropriate for a are both free, open source, and mature } unit test. This analysis actually tests the enough for enterprise use. On the bina- ry static analysis project, which was writ- This test fails because no exception ten in C# for deployment on Linux/ is thrown. So we add some logic to the mono (www.mono-project.com), we constructor to see if the file specified used NUnit. begins with “ELF” and throws the Thankfully, mono packages the exception expected if it does not. Both NUnit assemblies by default, so just tests now pass. Next test: Throw an installing the mono packages allows you exception if the ELF header is valid but to get going with TDD. We also com- the platform specified in the headers is piled and ran our code in Microsoft’s not SPARC. See it fail, make it pass. .NET implementation for testing and Next test: Get the offsets in the file debugging, but the results were almost where the functions it contains begin. always exactly the same as with mono. See it fail, make it pass. Next test: Verify With our tools set up, we were ready to the first byte in the first function. tackle our first user story. [Test] The first user story was to find a bug public void verifyFirstFunctionByte() in a small program compiled for the { whole system: header parsing, control- Binary binary = new Binary(@”c:\valid\bin”); SPARC hardware platform with various Function function = binary.Functions[0]; flow analysis, emulation, data-flow compilers at various optimization levels. analysis, evaluating signatures, dynami- Assert.AreEqual(0x2f, function.Bytes[0]); First, we broke up this story into several } cally creating signatures, and so on. So tasks: Make sure we can read the bina- we needed to refactor our unit tests into ry’s (ELF) headers correctly; make sure We now have some duplication in customer (or system-level) tests. The we can disassemble the first instruction; our tests: the creation of the scanner for book “Extreme Programming Adven- then disassemble the first function, our valid file. We refactor to make tures in C#” by Ron Jeffries offered use- then the rest of the functions; and final- “binary” a class-level variable, and put ful guidance for this in its XML ly find the bug specified. its initialization into a [SetUp]. Notepad example. Breaking up a seemingly large story We decided we would annotate the into smaller tasks made us relax a bit, [TestFixture] source code for our test program, and public class Tests as trying to figure out where to start to { extend the analysis engine to optionally show immediate value was already giv- Binary binary; read in a file of annotations and present ing us a bit of stress. With the tasks [SetUp] the user with an error if the analysis broken up, we compiled the small pro- public void setUp() { binary = new Binary(@ reports did not match the annotations. “c:\valid\bin”); } gram specified on the user story card We made a user story card for this new and then went to write our first test. [Test] public void validElfBinary() { .... } annotation checking feature, broke it

Er, how did we do that? We ... up into small tasks, and wrote the first stopped, went to study our XP and } failing unit test for parsing the annota- TDD books, and then came back. We tions in a given file.

JANUARY 2006 www.stpmag.com • 23 TEST-DRIVEN DEVELOPMENT

We circled the wagons in the usual decoupled and the tests run very fast, primary rule: The unit tests need to write-a-failing-test, see-it-pass and elimi- and it gives programmers the confi- run very fast. nate-duplication pattern. We were then dence to work very quickly when refac- Next, we saw that refactoring them able to take out the slow system-level toring, optimizing, debugging and out into system tests and taking the tests that had been inappropriately enhancing code. time to come up with a good frame- inserted in the unit tests and instead A brief example of a new unit test work for doing so was worth the effort put them into annotations. We then we wrote to fill in the hole left by our in both the long and short term. made a simple script to run the analysis programmer-test to customer-test tran- Last, we saw that system, or cus- on all the files with the annotation sition is below. We wanted to be able to tomer, tests don’t eliminate the need checking turned on. easily specify a small array of bytes and for unit tests that attack the same tar- The unit tests were again running pretend that it was a real function. get from a different angle. in a few seconds, making them easier The configuration of our Function to run all the time; and the time we objects actually didn’t permit this in Achieve The took to make the annotation language the design, so the unit tests drove us to Right Testing Balance made it much easier for the customer refactor. You will discover an improvement in to specify tests, ultimately saving us lots When we enhanced the code later, software quality by applying test-driven of time in the long run. The downside this decoupling was very handy, in development whether you are part of a is that now we have large swaths of addition to providing us the added large development team or working in code that were no longer tested in the testability: a small shop. You will see these gains unit tests. whether you are working on enter- [Test] One might say that since they’re public void disassembleAdd() prise software or writing embedded being tested in the system tests, why { systems code. would you spend time writing another Function function = new Function(“name”, As I have explained in this article, new byte[] { 0x2e, 0x2f, 0x30, 0x31 }); test for the same functionality? In fact, you will benefit by performing both why write unit tests at all, since you can Assert.AreEqual(“addx”, unit tests and system tests, so long as function.GetInstructions()[0].ToString()) just write system tests? Or, better yet, you’re clear on the proper boundary } someone else can write system tests! between them. Neither one should be You can see the slippery slope that As we have seen, it can be easy to neglected, no matter how strong the this seemingly innocent argument “misplace” system-level tests in the temptation. starts you down. Writing unit tests is unit tests. Sometime this might be All in all, it’s the balance between important for making sure objects are okay, so long as it doesn’t violate the them that counts. ý

24 • Software Test & Performance JANUARY 2006

ow do you make a process visible enough that it upper management perceives the applica- H can be reviewed and understood by everyone tion development pro- cess. It is no longer sim- from developers to QA teams to business managers, even ply something that each unique development if they are not IT experts? By Tracy Ragan team within the organi- zation defines and con- trols. CIOs, VPs and When the left hand doesn’t know what directors are beginning the right hand is doing, someone needs to perceive the software to bring some clarity to the situation, development process as especially for QA teams. Because of new a “business process” IT governance regulations, CIOs, VPs that needs to be tightly and directors now have to understand managed and consistently repeated, the details of how applications are writ- regardless of the development tool, the ten, tested and deployed, and an Eclipse or a unique application subproject may help shed some light. team. With this new IT governance envi- According to the IT Governance Insti- ronment, IT professionals from develop- tute Web site (www.itgi.org), “The over- ment teams to QA teams must create a all objectives of IT governance activities transparent process that can be reviewed are to understand the issues and the and understood by anyone, even if they strategic importance of IT, to ensure that are not IT experts. the enterprise can sustain its operations In the past, creating a truly transpar- and to ascertain that it can implement ent development process has never been the strategies required to extend its activ- possible. When a production problem ities into the future.” does occur, it is the QA team that is ques- Consequently, IT professionals rang- ing from development teams to QA Tracy Ragan is the CEO of Catalyst Systems teams must create a transparent process Corp. and has been a member of the Eclipse that can be reviewed and understood by Organization since 2002. She has a back- ground in testing, and build anyone, even if they aren’t IT experts. management. There is a paradigm shift as to how

26 • Software Test & Performance JANUARY 2006 tioned first as to what went wrong. QA practices of each of the unique develop- teams are often looked at by upper man- ment teams and report to auditors, agement as the team with the most infor- CIOs and VPs about what when wrong One Eclipse mation about how the application got to with the release. production. They are expert in creating In order to report on such a broad a standard and repeatable method of problem, testers often need informa- testing, but they’re always at the mercy of tion that is out of their jurisdiction. Project Will the development teams to deliver “clean” Testers do not always have access to the executables to them on schedule so that array of tools that just may contain an they may complete testing before the important clue as to what went wrong Help Meet IT production due date. with a particular release. Research may When everything doesn’t fall into begin with some questions that should place, even the most efficiently defined have been asked before the executables QA process can fail as days of scheduled even arrived on your desk. For example: Governance testing turn to mere hours of real test- 1. What were the requirements that ing. The result will be an inadequate initiated the new release of the software? product in the hands of the end users. 2. What additional features were As the tester, you are often expected to added to the release, and who request- Regulations understand the different development ed these new features?

JANUARY 2006 www.stpmag.com • 27 PROCESS MANAGEMENT

3. Were the new features reported to questions with just one tool. Management” will allow requirements testing, and were test cases created? The Eclipse Foundation has recently tools to pass requirements information 4. What was the impact of the approved and launched the Application to testing tools, allowing testers to better changes to the overall application, and Lifecycle Framework (ALF) project. This define new test cases for new require- was the impact of the changes added to project is good news for testers who often ments. Getting this information early, as the timeline? find themselves in front of upper man- soon as the requirement is defined, will 5. Were any unexpected changes ini- agement trying to explain why a release give testers an immediate notification tiated by the development team without went out that was less than perfect. that new test cases are required. the direction of the end user? The ALF project will allow tools that Both of these examples demonstrate 6. Who approved the changes to be support the application life cycle process how the Eclipse ALF project will create promoted to the new release, and do to communicate with one another using a level of transparency in the overall these changes match the approved SOAP transactions via a common com- software development process. The requirements? munications framework. This means that examples provide testers with informa- 7. Where is the approved source code even when a development organization tion they need to manage the testing managed and stored? Are all third-party uses application life cycle tools from dif- process, and they indicate why testers libraries also managed and stored? ferent vendors, the detailed information must be concerned about the trans- 8. How did the application build pro- about why, who and when can be passed parency of software development. cess verify matching source to Developers don’t need to executables, and is all source use the Eclipse IDE in order code used in the build coming for life cycle teams to take from the approved and man- advantage of the Eclipse ALF. aged source code repository? • ALF will exist outside of the 9. Based on the impact of well-known Eclipse IDE. the source code changes, The ALF project will allow tools that ALF is not a development what test cases were selected tool. It is a life cycle tool. If you as “critical”? support the application life cycle process are a user of a life cycle tool 10. Who approved the re- such as a requirements gather- lease to production? to communicate via SOAP transactions. ing, issue tracking, version The first two questions can control, configuration man- be answered using a require- agement, build management, ments gathering tool. Ques- testing or deployment tool, tion 3 needs information ALF will potentially change from both a requirements • the way you work. tool and a testing tool. Most important to note is Question 4 can be answered using a between tools. The functionality of the that no single team will use all of these commercial build management tool that tools can be extended by asking other tools independently. Organizations break provides impact analysis information. tools for information. down these responsibilities into diverse To answer Questions 5 through 8, a To achieve interaction, the Eclipse teams. For example, requirements gath- combination of tools must be used, in- ALF project will define a common vocab- ering is often done by a software analyst cluding a requirements tool, a configu- ulary between application life cycle tools. who is part of the development team. ration management tool and a build Tools will communicate via SOAP trans- Issue tracking and deployment may be management tool. actions over the Eclipse Application used and managed by the production Question 9 may require knowledge Lifecycle framework. ALF will also define management team. Testing teams use from both an impact analysis build-man- “service flows.” An example of a service testing tools, while the central configura- agement feature and a test tool. And flow would be build/deploy/test. In tion management department is respon- finally, Question 10 involves a release answer to Question 9 above, the build sible for version control. Build-manage- management tool. management tool could send a list of the ment tools may be used by several areas. To get the information you need to affected executables to the test tool. The Developers perform unit and pre-test answer these questions, you must work test tool could then determine which test builds. Testing initiates a QA build, and with a variety of application life cycle cases must be retested prior to produc- production management may perform a tools that share no common vocabulary tion release based on the list provided by production build. All of these teams must or data. You need access to all of these the build management tool’s impact work together, and through ALF, the tools to get to the data needed to analysis function. tools they use will have the potential to report on the most basic of develop- Another example of how ALF might communicate as well. ment metrics. change the way a tester works is the bet- This sharing of information from These are not difficult or unreason- ter use of requirements. Often, testers application developers to the configura- able questions to ask. The problem is do not have a complete picture of the tion management and testing teams, that for most of the questions, it requires requirements of a new release. In regard down to production control, is what will cooperation between multiple teams to to Question 3, “Were the new features provide the transparency needed for determine the complete answer. No sin- reported to testing, and were test cases reporting on the slightest details of the gle team within the IT organization can created?”: The ALF service flow called software development process. quickly get answers to most of these “Requirements Management to Test I’ve watched how application life cycle

28 • Software Test & Performance JANUARY 2006 PROCESS MANAGEMENT tools are responding to the needs of their you can define that the first test case to be tester, you should be concerned about customers, based on the new IT gover- executed is a QA build, which includes a the transparency of the application pro- nance rulings, such as Sarbanes-Oxley. full impact analysis report for future use. cess and not feel you are asking for too Some tools are expanding their basic fea- You should request that your develop- much when you ask for more than just a tures to provide better reporting, while ment teams assist you in setting up this copy of the installer to begin testing. others are simply emphasizing those fea- build. By doing this, you are requiring As all members of the IT organization tures that already provide critical audit the development teams to be more become more involved in achieving a data. The Eclipse ALF project will com- methodical about how builds are done. If level of transparency in the development plete the picture for life cycle tools, with you do not do a QA build independent of process, reaching the goals of IT gover- critical information being shared the development team, at a minimum nance will be more easily realized. In ad- between tools regardless of vendor. you should require verification of how dition, as you begin requesting more in- It may seem out of the realm of a the build was completed, as well as valida- formation about how an application was tester to worry about the transparency of tion showing that only approved source created before you begin testing, you will the application development process. code was used in the build. begin to see a pattern emerge regarding However, you may want to consider start- You should request access to require- the potential quality of any single release. ing to request some basic reports that ments-gathering tools and issue-tracking As application life cycle tools be- can assist you in your testing efforts tools to provide you with early informa- come more integrated with ALF, you immediately. You don’t have to wait for tion about new features and enhance- will be able to gather information that the automation of ALF. If you simply ments that will be coming your way. will help you determine the quality of look at an area where you are most By asking for this type of information, the software even before you execute affected, such as the late delivery of exe- you are placing more responsibility on your first test case. If substandard soft- cutables to testing before production the development teams to deliver a qual- ware sneaks through the door, you will release, you may find you have new tools ity product before you begin testing. This be able to follow the ALF trail back to in hand to help mitigate these problems. is the goal of IT governance. By more where the breakdown may have oc- Let’s look at the most usual suspect in clearly defining a repeatable process in curred. Fixing the process may not late deliveries to QA: the clean build. It is software development, the quality of soft- seem to be the QA department’s job, often the case that developers manually ware improves. If there are aspects of the but who else is so worried about quality? develop scripts to execute the build, software development process that are For more information about ALF, go which can trigger many problems, mysterious or overcomplicated, as the to www.eclipse.org/alf/. ALF is in its early including the use of incorrect source application build process often is, the stage of development. The proof of con- code, causing builds to break. As a tester, quality of the software decreases. As a cept is scheduled for early 2006. ý

JANUARY 2006 www.stpmag.com • 29

Stress Testing: Take An Open Source Approach

ith Eclipse, you W can control Ant and JMeter to generate and perform automated

stress tests. By Alan Berg

Within the Eclipse environment, it is possible to control Ant and JMeter so that they generate and perform auto- mated stress tests. This article looks at what the open source approach can achieve within the realm of stress test- ing. I will explain the basics of combin- ing the power of Eclipse, Ant and JMeter, which when combined create a feature-rich and relatively easy-to-con- trol stress testing environment. JMeter is a powerful, fully featured open-source stress testing tool. As a developer, I have had the pleasure of watching JMeter in action. This tool has n a i

found bottlenecks for me in unexpected r d A

places, stressed infrastructure to levels o t r e

higher than any potential loads found in b o R

production situations, and has generally y b

h p

laid waste to negative, undeserved back- a r g o

room gossip about architectural pat- t o h

terns. As an open source tool, JMeter has P been through many life cycles, with sup- one central and visual location. I will • Jmeter binary distribution: jakarta port from a broad community. explain how to glue Eclipse and JMeter .apache.org/site/downloads Depending on whom you ask, Eclipse together through Ant, another mature /downloads_jmeter.cgi is a framework, an open source commu- open source tool. Eclipse includes a • The example project for this tutori- nity or an IDE. In this article I am native editor and browser for Ant. al: www.stpmag.com/downloads addressing Eclipse only as an IDE. For /stp-0601_berg.zip example, I don’t use the graphical Try It Yourself After installing and downloading the libraries to create applications. Eclipse is I hope you will get your hands dirty by tools, import the example project into popular among code bashers, especially working with the tools mentioned, and Eclipse as an Archive file and then read those tainted by Java. The IDE has all the to aid in the cause, I’ve included a sam- the README contained within the top features one would expect from its pedi- ple Eclipse project. However, actively fol- level of the expanded project. gree. Eclipse allows control of many fea- lowing the coding is not strictly necessary Note: The Ant task contained within tures of the coder’s environment from to gain insight into the methodology. the example project home page can be You will find downloads for the tools found at www.programmerplanet.org Alan Berg, Bsc., MSc., PGCE, has been a at the following locations: /ant-jmeter/. lead developer at the Central Computer • Ant binary distribution: ant.apache Services at the University of Amsterdam for Workflow the past seven years. You can contact him at .org/bindownload.cgi [email protected]. • Eclipse SDK: www.eclipse.org The motivation for the Eclipse world- /downloads view is to pull as many parts of the work-

JANUARY 2006 www.stpmag.com • 31 OPEN SOURCE TOOLS

FIGURE 1:WORKFLOW FOR STRESS TESTING WITHIN ECLIPSE tweaking the environment to a fine level of granularity. An Apache project, JMeter is the (1) Make test plan premiere open-source stress testing in JMeter tool. Apache is a container for numer- ous projects generated by like-minded, freely giving developers. Tomcat and (2) Template test plan the Apache Web server are two of the best-known products out of this stable. JMeter is similar to Ant; JMeter also (3) Make Ant build reads XML configuration files. file via Eclipse However, to our advantage, we will run JMeter through a GUI, which signifi- Run test plan through cantly decreases the associated learn- Change variables in Ant in Eclipse ing curve. You create a test plan, and Properties file the GUI generates an equivalent XML configuration file. Once you are satis- Generate human-readable fied that the test plan is actually what results and process results you want, you need only change a cou- ple of variables to allow JMeter to No attack the specific infrastructure. Finished? To tie JMeter to Eclipse, we can run Yes JMeter as an Ant task. This method removes the GUI from the action of running. However, please note that in this exercise we will take advantage of flow as possible together into a visual the organization. To install Eclipse for the GUI to build a template test plan environment. This provides the pro- the first time, you will need to have that Ant will later modify. grammer immediate control and one installed Java. After that, it is a case of Furthermore, it is important to be location for all possible updates and downloading a large Zip file and aware that JMeter is not limited to test- modifications. Centering on Eclipse has unpacking. The same is true for Ant ing HTTP; the tool can also test data- many advantages; for example, one can and JMeter. All three products are OS- bases, ftp, LDAP, JMX and Web servic- edit numerous file formats, each format agnostic. In the worst-case situation, es. The list is expanding with every new edited with a specifically tailored editor. switching from Windows to Solaris or release. Eclipse has a plug-in infrastructure Linux or the other way around is mere- JMeter is not limited to testing from that allows for ready extensibility, ly a question of a new download and one client machine. You may run and future-proofing the IDE to a high the importing of projects. control numerous instances of JMeter degree. The IDE also has built-in CVS Ant is a Java-aware build tool that across a network. A standard PC with a and Ant support. Of further signifi- reads in an XML configuration file and fast network connection may simulate cance is the fact that many program- enacts the tasks set within it. The tool is as many as 300 concurrent HTTP mers tend to stay loyal to one particu- extendable; many contributors have browsers, and the capacity increases lar editor or IDE. If the environment helped increase its vocabulary and with each new computer generation. you find yourself working in is oriented reach. It is popular with Java program- toward Eclipse, the workflow will readi- mers, because Ant understands the A Generic Workflow ly lend itself to dissemination within Java environment rather well. Eclipse Figure 1 describes a generic workflow. has a specific Ant build file Throughout this article, I mean to FIGURE 2: PROXY SERVER SETUP DIALOG editor that has syntax high- keep the examples as simple as possi- lighting and, more impor- ble so as not to distract from the main tant, context completion. theme, which is the bonding of JMeter Context completion gives to Eclipse through Ant. hints that allow the program- The three main processes of the mer to learn on the job, workflow are: without returning to refer- 1. Make a test plan. Within JMeter, ence books. Running a build you may start up a proxy service that file in Eclipse is a simple captures browser events. This allows matter of right-clicking on for quick building of HTTP-based test the build file and selecting plans. This process will be explained in Run As followed by the detail in the next section. desired target. Eclipse allows 2. Template. Converting the test for excellent control of the plan into a generic template is most Ant run process, and there easily done through the JMeter GUI. are dialogs within Eclipse for Templating is a question of replacing

32 • Software Test & Performance JANUARY 2006 OPEN SOURCE TOOLS actual values—for example, a FIGURE 3:THE THREADED GROUP DIALOG BOX still gives you an idea how the URL, or the number of threads plumbing works. If you want, to run with tokens that Ant sub- follow the spirit of this section stitutes at runtime. and generate your own test 3. Make an Ant build file via plan relevant to your own Eclipse. The Ant build file is the immediate infrastructure. Play- scaffolding around JMeter. Ant ing around in this way is in will change the template into itself quite educational. an actual test plan, run the plan 2. Visit the application, and and then transform the results fill in and return the form. stored from XML into human-readable JMeter. Proxying allows for the capture Remember to turn the proxy off in HTML summaries. of HTTP requests and responses JMeter after use, or you may end up The rest of the workflow mostly between the Web browser and the serv- with many irrelevant entries. involves iterating through various vari- er for later reuse within JMeter. To 3. Make a test plan. This can be bro- ables—for example, changing the num- achieve this, you will need to modify the ken up into a series of substeps: ber of concurrent visits to the Web site, settings of the browser. I generally use • First, make a Thread group by or the delay between simulated browser Firefox for building test plans. It is stan- right-clicking on the Test Plan node events. The process of changing vari- dards-pure in its HTTP requests. To set and choosing ADD/Thread Group. ables is done by hand in this example. up Firefox (version 1 or greater), you Fill in the details as described in However, Ant is capable of iteration will need to visit the Firefox menu Figure 3. At runtime, Ant will through multiple parameters. We avoid /tools/options/connection, tick Manu- replace the @threads@ token with a doing so here, as writing such a task al Proxy configuration, and set the meaningful value. would be verbose and would detract value of HTTP Proxy to localhost and • Next, add the HTTP Parameter from the simplicity of our discussion. the port to 8099. Modifier via ADD/Pre Processor The application tested in this article To run the proxy server in JMeter, /HTTP Parameter Modifier. Fill in the is a form that sends a username. A PHP right-click on WorkBench and select filename “articles1.xml.” script welcomes the user. The applica- Add/Non-Test Elements/HTTP Proxy • (Optional) Add a delay to slow Server. The proxy server dialog is now down the hitting of the Web server LISTING 1 open. Fill in the details as shown in via ADD/Timer/Gaussian Random A simple form that posts the users Figure 2. Timer. Remember, 300 agitated name Notice that the port number is set automated monkeys are probably Software to 8099. By default, JMeter sets the equivalent to 1,000 normal hu- testing port to 8080, which is in conflict with mans. Delays are a force for good.

the default settings of a standard instal- • Move the captured events /example Username: lation of Tomcat. The “patterns to /index.html and /example/process.php types that you do not want to record, placing them via drag-and-drop after
because the entries would clutter your the Gaussian random timer. test plan results. For example: the pat- • Add the “article1.xml” file from tern .*\.jpg is a regular expression for Listing 3 to the JMeter bin directory. tion is mentioned in Listings 1 and 2. ignoring any returned response with a Modify the HTTP request dialog I’m assuming that JMeter and the URL that has any character followed by /examples/process.php with the details: Apache Web server with PHP exten- .jpg. Hence, no jpeg files are recorded. • HTTP Request/Path: @url@ sions are run on the same machine. To build a test plan via the proxy, /process.php This is, of course, bad practice. You you will need to visit the target applica- • Send parameters with request should never have the potential for tion. First, browse to the Index page, • Name username. Change the denial of system resources of the tester then enter a username; allow the PHP value to blank from the tested. However, the elemen- script to process the form data and The HTTP Parameter Modifier tary infrastructure does allow one to return a response. You should immedi- changes the values by substituting get a feel for the concepts mentioned. ately see the responses appear within parameters mentioned in the “arti- the workbench area. cle1.xml” file that sits in the bin direc- JMeter Test Plans OK, let’s prepare a test plan for tory of JMeter. JMeter captures Web browser events templating. The recipe is: Congratulations, you now have a plan. via its own proxy server. This section 1. Install the application. I assume If your master plan fails, compare it to the will explain how to make test plans and that you have a PHP-enabled Apache LISTING 2 template a plan. You can find an exam- server, and that the index.html and ple test plan under the testplans direc- process.php files sit under the The trivial PHP script that process- tory in the example project. You may Example directory under the htdocs es the form data load this directly into JMeter for refer- directory. If they do not, the test plan Welcome
First, you will need to make sure mentioned in your summary reports. BACK your Web browser is proxied through That is OK for experimentation and

JANUARY 2006 www.stpmag.com • 33 OPEN SOURCE TOOLS plan contained in the example project. overview of the inner LISTING 4 In descriptive terms, the test plan structure, it is best to 1 achieves the following: The thread use the Eclipse 2 group defines how many concurrent Outline window. To 3 sessions are running. The HTTP activate the outline, 4 5 Parameter Modifier changes the user- double-click on the 6 name parameter in the simulated build file, thus activat- 7 8 HTTP requests. The HTTP request ing the Eclipse Ant 9 component makes requests based on editor. 10 11 the @token@ with real values, JMeter defines macro proper- 13 runs the tests to definition. ties for the whole proj- 14 15 Please note that by adding the listen- ect. In the current situ- 16 er view tree results, results are readily ation, testPage (line 1) 17 18 observable live. is the default target 19 run. 20 21 Running Ant A property file (line 22 This section details how to combine 3) is a file with a series 23 24 the power of Ant with Eclipse. of properties contained 25 To run Ant from Eclipse, right- within. Once loaded, 26 click on the build.xml file and choose these properties are vis- 27 28 ing you targets from within the proj- the build.xml file. 30 ect to run. Run the already ticked Next (lines 6-8), we 32 default target. want Ant to be able to 33 34 I will describe the build.xml config- find the JMeter library 35 uration file next. However, before that, JAR file so that we can 36 I wish to mention the build.properties later define a new task. 37 38 build.xml file sees the properties and new task (lines 11-14), 40 41 values contained within. By having a we associate an Ant tag 42 properties file, you separate the code with the relevant Java 43 44 example, you may wish to change the classes within the JAR 46 47 host name or file locations without library. 48 modifying the code. In the properties Creating a viable 49 file, a variable foo would be defined as: instance of the test plan 50 51 foo=value. In the build, you may use involves copying the 52 Now that we have seen how to same time, filtering 55 57 look at the build.xml file in detail. It is suppose you have the 58 my hope that you will find parts of the token @delay@ in a tem- 59 plate, then the following 60 An example parameter file defining 64 one username Test1 65 value="30" /> store results in XML format and not in username In the build.xml file, on line 34, the more common comma-delimited Tester1 delay is expanded to the value of format. thread.delay that is mentioned in the Within lines 60-63, the XSLT tag build.properties file. takes the results example.jtl and trans- The JMeter tag (lines 52-57) is the forms the results to example__${thread build useful enough to reuse. First, kernel of the build file. The tag orders .max}.html, where, of course, the we’ll list the whole build for complete- Ant to run the stamped template exam- ${threads.max} is expanded from the ness (Listing 4), and then we’ll break ple.jmx, and the results are then stored build.properties file. The JMeter- the listing down into smaller parts in example.jtl. The only detail that may results-report.xsl file is the applied with an explanation for each. XML is be hard to follow here is that a specific transformation. not always the most digestible format property jmeter.save.saveservice.output In summary, within this section we for human readability. To achieve an _format is set to xml to force JMeter to have described a typical Ant build file

34 • Software Test & Performance JANUARY 2006 OPEN SOURCE TOOLS that does a number of helpful tasks. dard slave can run 300 or more simul- However, looking at the build as a taneous threads. For complex process- whole can be confusing. This is es involving XLM, such as SOAP mes- because Ant does a lot, and the config- saging, this number is significantly uration in XML format is not readily lower. viewable. Remember that a thread in JMeter Eclipse helps us here with a tree is consistently faster than a typical user view of the structure, and it also pro- would be, so obviously, 300 threads is vides a decent editor that eases man- not equivalent to 300 simultaneous agement tasks. users. It is more likely to feel like 1,000. With this, you can really hit your infra- Make Eclipse the Boss structure and explore where the archi- I hope this discussion has given you the tectural bottlenecks are. understanding that making Eclipse the Applying stress is one issue, but to boss and running JMeter through Ant interpret the results is somewhat more offers you a controllable and maintain- difficult. First, you may need to able environment. observe key real-time metrics such as What we have not explored in any memory usage. Further, you may be detail is the manipulation of configura- required to monitor log files live. In tion through JMeter assets such as the Java universe, frameworks such as “Config Elements.” By using the log4j and JMX enable the required “HTTP Request Default” dialog, we can fine level of monitoring. set the host name in one location and JMeter is a powerful tool in the arse- allow JMeter to modify the behavior of nal of software testers. Stress testing all other elements. This implies that in can give you a feeling of confidence the template we would also have only about the quality and stability of a one place necessary to replace tokens, given structure. thereby easing readability and thus improving maintainability. In Control As you look forward to additional This article has described control of possibilities, you will see other poten- JMeter from Ant within Eclipse, an tial experiments that await you. For environment in which a rapidly example, you can set up JMeter to run increasing number of developers are in a distributed manner. As long as a developing proficiency. While it is true series of JMeter instances are run on that Ant build files can be hard to read, the same subnet, you can control the Eclipse has built-in functionality to slave JMeters from a master. I illustrate ease the pain of editing. this theme for a typical topology in I wish you much luck with your Figure 4. For HTTP requests, a stan- experiments. ý

FIGURE 4: A SAMPLE DISTRIBUTED JMETER TEST ENVIRONMENT

Slave Apache

Tomcat Slave

DB Master Slave

LDAP

Slave

JANUARY 2006 www.stpmag.com • 35 Best Practices Unit Testing: It’s Like a Box Of Chocolates

Nominally, unit testing is a ing,” says Taylor, “it is always where changing code broke existing QA function. After all, it’s a high priority for us to (1) valid scenarios.” part of the process of ensur- move unit testing from the Efficiency and early feedback are ing that an application QA team to the developer also the major advantages for Ilja works correctly. Developers team and (2) automate the Preuss, a software developer at disy are responsible for creating unit tests.” Moving unit test- Informationssysteme GmbH in Ger- the application in the first ing to developers, he feels, many. He explains, “I need to have new place—and surely that’s enables QA to focus on tests for my new code as early as possi- enough to keep track of. higher-level testing, such as ble, and I need to be able to run all the As a result, it seemed like integration testing and tests whenever I want.” an obvious question to ask Esther Schindler acceptance testing. “Their Gerardo Tasistro, research and devel- both developers and testing productivity goes through opment director for Asdeporte CIE in professionals how their workflow breaks the roof,” he reports. Mexico City, is more concerned about down. That is, who does the unit testing? That’s not to say that everyone the overall development process. Accord- Who should? I imagined a knock-down, believes that developers should write ing to Tasistro, when code units are test- drag-out fight between the two commu- and run their own tests. Some believe ed by developers, it reduces the number nities, possibly including shouting and that the programmer’s peer should do of problems QA has to deal with. Also, he name-calling, tufts of hair fluttering in the unit testing. For example, Asif Patel, says, doing so benefits code reusability. the breeze, and chairs thrown across the a software tester at Anker Systems in the “This is the most important reason. You room. It would all be very entertaining. U.K., advocates a level of test independ- don’t want to reuse bad code.” So much for that idea. ence. “Ideally, the author should have a Allan Caplan, architect at Workbrain As it turns out, developers and QA ‘buddy’ developer who is able to test the Inc. in Toronto, has three reasons. First, testers have achieved an almost unani- code. There can be a degree of emo- he says, unit testing is faster than inte- mous consensus: Unit testing is the tional attachment.” gration testing. “I’ve watched as devel- developer’s responsibility. What I found Overall, however, I found more opers go through this cycle [write the interesting, though, are the differing agreement on this topic than on most fix, compile/deploy the fix, start up opinions on the reasons why. others I’ve encountered. (Though per- your app server, navigate to the page, One consultant is unrepentant in his haps the question “Is chocolate a good and perform a set of actions] ad nause- viewpoint. Says Brian Crook, freelance thing?” would have received a more uni- um. With unit testing, you define what writer and IT consultant in the Rocky form response.) the correct behavior is. You can then Mountains, “Any developer who claims Developers and testers have their own write your code, and if you’re using an to be ‘above’ unit testing is not to be reasons for developing and using unit editor with JUnit integrated, you just trusted. A developer should be able to tests, and they see different benefits. keep clicking that Run button until [the demonstrate that his work conforms to Developers view unit testing primarily code passes the tests]. Only then are requirements as s/he understands them. as a way to improve their own efficiency. you usually ready to deploy to the server So developers are responsible for unit Programmer Christof Wollenhaupt and attempt an integration test.” testing; always have been.” Nonetheless, claims that unit tests are a worthwhile Also, says Caplan, unit testing en- he admits, “Conformity has been spotty replacement for traditionally written test forces developer accountability, and the sometimes.” programs. “Unlike these one-time test tests provide an additional level of doc- When he first encounters them, says programs that help in writing the code in umentation. Kevin P. Taylor, principal consultant for the first place,” he says, “unit tests are And, of course, there’s always the cyn- Obtiva Corp. in Naperville, Ill., most of repeatable.” Unit tests have enabled him Contributing editor Esther Schindler is still his clients run unit tests manually by QA to improve his own code quality. For disappointed that nobody engaged in shouting teams, with test plans in MS Word or instance, he says, “A modification in one or hair-pulling. If you want to pick a fight with Excel. “Since my company specializes in class caused a side effect on another class her,perhaps about the virtues of chocolate, you agile methodology training and mentor- that I didn’t foresee, as well as many bugs can reach her at [email protected].

36 • Software Test & Performance JANUARY 2006 Best Practices

ical (or is it realistic?) point of view. One developer who relies on TDD is “Writing the unit test first forces me to According to programmer Scott Selikoff, Ernest Friedman-Hill, who works for write testable code. And the most im- from a management perspective, unit Sandia National Laboratories in portant property of well-testable code is testing can be more about PR and ac- Livermore, Calif. His main project is a that it is decoupled, so that you can test countability than it is about quality, con- small one, with about six developers. units in isolation. Writing the tests in trol or good coding. “I’ve seen managers The developers do all the unit testing, parallel to the code also helps with get- talk about it as a buzzword in the same and the tests are written before the code. ting high code coverage,” he says. way they talk about J2EE,” Selikoff says. Doing it first changes everything, says But TDD isn’t for everyone. Selikoff “Most of the time, they have no notion of Friedman-Hill. “I’ve run plenty of proj- says that while unit testing has advan- what unit testing is, nor can they proper- ects in the past where developers wrote tages, test-driven development is funda- ly formulate the desires to the developer tests after the fact. These invariably were mentally flawed. It’s especially problem- as far as what to test and how.” more application tests than unit tests, atic in J2EE, he says, where service con- Despite his remarks, Selikoff ac- because if you don’t design for testability, text, database dependency and transac- knowledged unit testing’s benefits. “It you often end up with code that’s hard to tional requirements make it unrealistic makes sloppy developers think first test ‘in the small.’ You get big tests with to run good unit tests. “For example, if before they code,” he says. lots of setup that don’t isolate faults and a line of code was only supposed to be While it’s obvious that unit testing is tend to give worse test coverage.” Writing run inside a transaction, it is very diffi- considered an essential part of the de- tests first leads to better design, says cult to successfully unit test. I think the velopment process, there’s less agree- Friedman-Hill, because it forces you to people who thought of test-driven devel- ment about when those tests should be think about an interface for each class, opment were smart and had excellent written. The test-driven development allowing that class to stand and be tested intentions; I see the practical applica- (TDD) methodology suggests that the in isolation. tion of it being a complete mess and giv- tests be written before you begin cod- Preuss finds that writing the tests ing management a false sense of securi- ing. Not everyone sees it the same way. helps him write well-designed code. ty, at least in the near future.” ý

Index to Advertisers

Advertiser Page

Code Project www.codeproject.com 30

Entrek www.entrek.com 29

ITKO www.itko.com 35

LogiGear www.logigear.com 29

Mindreef Inc. www.mindreef.com/go 4

Parasoft Corp. www.parasoft.com/SoftwareTestAndPerformance 2

Perforce Software www.perforce.com 25

Seapine Software Inc. www.seapine.com/stp75 6

Segue Software Inc. www.segue.com/lh?sorry 3

Software Security Summit www.s-3con.com 20-21

Software Test & Performance Conference www.stpcon.com 39

SpeeDEV www.speedev.com/white-papers.asp 24

Software Quality Solutions (SQS) www.sqs.com 40

JANUARY 2006 www.stpmag.com • 37 FutureFutur e Test Test

es at once, and often spends more than half of his or her time manually manag- ing, monitoring and tuning them. If a Managing The DBA is spending more than 15 percent of his time tweaking or writing scripts, it is an indication that automation is needed. Automated notification eliminates the Performance Of need to dedicate resources 24x7 to watch administration consoles for system mes- sages. Instead, error and status monitor- ing should be automated and critical alert information sent via e-mail or Databases mobile devices. DBAs receiving alerts can then determine the severity of the prob- If you manage databases, aiding a symptom. lem and when to repair it without being you’re probably called upon In short, once a problem on site. Although remote notification from time to time to accom- is identified and isolated to reduces the need for dedicated on-site plish tasks that seemingly the database domain, it’s es- staff and decreases problem response would take a superhero. Yet sential to quickly know when time, it does not provide analysis or cor- armed with a few industry it started; trends or patterns rective action capabilities. Advanced noti- best practices and the help associated with it; and pro- fication capabilities should be employed of technology, you can ac- cesses and sessions impact- with multilevel thresholding, event man- complish far more then you ed. Only then can the prob- agement and correlation technology to thought was possible. Even lem be properly repaired. further reduce the need for manual man- with the most complex envi- To be proactive, organi- agement, enabling the DBA to more ronments, there are four Steve Lemme zations need to recognize effectively pinpoint and correct issues. simple practices you can follow to greatly and resolve issues early, to ascertain what In order for autonomic or grid com- improve your database performance needs management, and to gain control puting to gain acceptance, the infrastruc- management today: fast problem deter- of that by employing automation. Under- ture that supports business applications mination and resolution, consolidated standing the databases that exist today, and the data center operations must be and automated monitoring, automated especially their capacity and utilization, broad, flexible and designed with contin- problem notification and automated cor- is a critical first step in the automation ual change in mind—in other words, rective actions for preventive self-healing. process. This also helps us identify how service-oriented. Database performance Once an issue is reported, it is essen- many inactive or underutilized databases management is an essential component tial that the IT organization promptly could be consolidated to reduce costs. of these initiatives. The key is to leverage identify the cause of the problem and It’s important to capture and trend a technologies that improve productivity ascertain the best way to resolve it. End- baseline of current database utilization and ensure absolute availability regard- lessly transferring the trouble ticket from and capacity. The objectives are to iden- less of the level of demand. department to department, or closing tify consolidation targets; spot potential Database monitoring and perform- the ticket with no validation that the issue performance problems and capacity ance technology can help organizations has been resolved, are not good prac- planning issues; provide data to compare meet availability and performance service tices. Companies cannot afford to have and contrast conformance of the data- levels for mission-critical database appli- time wasted by staff guessing without hav- base to performance objectives, and to cations. It is also an essential element of a ing fact-based information or doing indi- indicate any deviations from a standard services-oriented architecture to enable vidual manual isolation testing. operating range for service-level-agree- the proper provisioning of resources Since reactive problem response is ment compliance, reconciliation and regardless of the business demand. such a time-consuming activity, the staff reporting; analyze information collected Technology like this, combined with gets engulfed and doesn’t have time to in real time; and over a period of time the best practices described above, will concentrate on strategic projects—and adjust thresholds for corrective action as improve the effectiveness of your over- becomes trapped in an endless loop per- required to meet performance objectives all database performance management petuated by ineffective methods. The and for capacity planning. efforts. Who knows—it might even keys to breaking this logjam are to rec- It is not uncommon for data collec- make management think you’re some ognize issues quickly, to have quality tion and monitoring automation to be set kind of superhero. ý data on hand, and to possess the capabil- up by database administrators using cus- Steve Lemme is director of product manage- ity to quickly recognize a performance tom scripts in the absence of the correct ment at Computer Associates. He can be problem created by a database—and fix management technology. A typical DBA reached at [email protected]. the actual cause instead of merely band- can manage, say, five production databas-

38 • Software Test & Performance JANUARY 2006 Mark your calendar!

IS COMING TO BOSTON IN 2006!

November 7-9, 2006 The Hyatt Regency Cambridge