TSP Teams Transform Test 2009 QUEST Chicago James D. McHale Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 [email protected] © 2008-09 Carnegie Mellon University A Nightmare Scenario The development team has been promising their “final” code drop for weeks, ever since their latest due date (months after the original date). Your test team has been filling in with clean-up, make-work tasks for most of that time. Every day represents one less day for your test team to complete their work, since the final ship date is not moving. Finally an email arrives handing off the code. All the proper links to the configuration management system are there… But they point to software that doesn’t build. A few more days pass. Another email. The software builds this time. The first day of testing cannot complete due to irrecoverable defects… and management wants you to commit to the original ship date. TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 2 © 2008-09 Carnegie Mellon University Current Test Reality Requirements, Design, Code, TEST, TEST, TEST, TEST… The focus is on writing code and getting it into test. Most working software developers learned this model implicitly, either in school or on the job, and have no other mental model available. Counting ALL of the testing costs for most modern-day applications still shows that 50% or more of development effort is spent in debugging. Is the purpose of testing to find defects? Or to verify function? TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 3 © 2008-09 Carnegie Mellon University The Problem With Testing Overload Hardware Configuration failure Good region = tested (shaded) Unknown region = untested (unshaded) Resource Operator contention error Data error TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 4 © 2008-09 Carnegie Mellon University Testing Takes a Long Time Magellan spacecraft – 22,000 LOC 200 180 160 140 120 All 100 Non-Crit 80 Critical Cumulative Defects 60 40 20 0 1 37 46 55 64 73 82 91 10 19 28 100 109 118 Weeks TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 5 © 2008-09 Carnegie Mellon University Answers For Testing? Testing is necessary (and not a necessary evil). However there are few good answers for test teams in an organization whose management allows poor quality software into test. • Typical development practices rely on test to find all but the simplest defects. • Incentives to “get into test” are unwittingly counter-productive. • With very buggy code, it is sometimes difficult to separate development defects from testing defects. Testing better and faster only encourages the root problem. TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 6 © 2008-09 Carnegie Mellon University Where Do Defects Come From? While a relatively small number of expensive defects are inherent in requirements (often outside the control of development), most defects are injected by the developers. For a small program (100 – 200 LOC), it may be only 5 or 10 defects, and many of those are caught by modern IDEs. (This is the mode in which almost everyone learns to program, regardless of academic discipline.) For a larger program (1000 – 2000 LOC), 50 to 100 defects is still tractable given those modern IDEs. For a small system (10,000 – 100,000 LOC), 500 to 5,000 defects cause long nights, lost weekends, and death march projects. For large systems (>1 million LOC), 50,000 defects per MLOC (even with 80% caught “automatically”) translates to worst-nightmare scenarios. TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 7 © 2008-09 Carnegie Mellon University What Can Be Done About Defects? Test teams cannot prevent the “import” of defects that development teams “export”, they can only highlight the problem. Development teams cannot today buy tools that reduce the number of defects going to test by a factor of 10. But… a different attitude on the part of developers concerning quality CAN reduce defects going to test by that factor of 10, or better. How can this be achieved? TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 8 © 2008-09 Carnegie Mellon University Personal Software Process (PSP) The Personal Software Process (PSP) • developed by Watts Humphrey at the SEI in the early 1990s • an interlocking set of measurement, quality, and planning frameworks • applicable to the smallest project (one person) PSP is taught incrementally through a series of 7 or 8 programming assignments, typically two one-week blocks about a month apart. One version of the training aims at starting to use a subset of these skills immediately following the first week. TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 9 © 2008-09 Carnegie Mellon University Basic PSP Skills Team Software PSP3 Process Cyclic development •Teambuilding •Risk management •Project planning and tracking PSP2 PSP2.1 Introduces quality •Code reviews Design templates management and design •Design reviews PSP1.1 Introduces estimating and PSP1 •Task planning •Size estimating • Schedule planning planning •Test report PSP0.1 •Coding standard PSP0 •Process improvement Introduces process discipline •Current process •Basic measures proposal and measurement •Size measurement TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 10 © 2008-09 Carnegie Mellon University PSP Estimating Accuracy 40 Majority are underestimating PSP 0 20 0 -200%-100% 0% 100% 40 Balance of over- and under- PSP 1 estimates 20 0 -200%-100% 0% 100% 40 PSP 2 Much tighter balance around 20 zero error 0 -200%-100% 0% 100% Effort EstimationTSP Teams Accuracy Transform Test James D. McHale, 2009 QUEST Chicago 11 © 2008-09 Carnegie Mellon University Improving Estimating Accuracy Effort Estimation Accuracy Trend 0.7 PSP0 PSP1 PSP2 0.6 0.5 Mean Time Misestimation PSP Level Average 0.4 0.3 298 Engineers 0.2 0 1 2 3 4 5 6 7 8 9 10 11 Program Number Estimated Minutes – Minutes / (Actual Estimated Minutes) TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 12 © 2008-09 Carnegie Mellon University PSP Quality Results Defects Per KLOC Removed in Compile and Test C 120 O L 110 K r 100 e P 90 s t Mean Compile + Test 80 c PSP Level Mean Comp + Test e PSP2 f 70 e D 60 f o 50 PSP1 r e 40 b PSP0 m 30 u N 20 n 10 a 298 Engineers e 0 M 0 1 2 3 4 5 6 7 8 9 10 11 Program Number TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 13 © 2008-09 Carnegie Mellon University PSP Productivity Results Lines of (New and Changed) Code Produced Per Hour of Total Development Time 30 PSP0 PSP1 28 r u o H 26 / Mean LOC/Hour C PSP Level Average O L 24 n a e M 22 PSP2 298 Engineers 20 0 1 2 3 4 5 6 7 8 9 10 11 Program Number TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 14 © 2008-09 Carnegie Mellon University Problems With PSP In the classroom, PSP works brilliantly. In practice, PSP works only in a supportive environment. That environment must acknowledge • that the individual’s contribution to quality is critical • that the people doing the work must have the proper quality training • that the project team is the minimum support structure for such work • “Quality without numbers is just talk.” – Watts Humphrey TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 15 © 2008-09 Carnegie Mellon University Team Software Process (TSP) The Team Software Process (TSP) was developed in the late 1990s to address the problems seen in early field use of the PSP. TSP applies the PSP measurement, quality, and planning framework at the level of a project team, adding • a team-owned project management method that generates extremely accurate and reliable tracking and exception information • a team commitment discipline that involves each individual in defining and accepting the group’s responsibilities, not just for timely delivery but also for the quality of the final product • a team-working structure based on pre-defined roles (similar to the positions on a sports team or the sections of an orchestra) TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 16 © 2008-09 Carnegie Mellon University Building High-Performance Teams Team communication Team Team coordination Management Project tracking Risk analysis TSP Goal setting Team Role assignment Building Tailored team process Detailed balanced plans Team Process discipline Team Performance Member Performance measures Estimating & planning skills PSP Skills Quality management skills TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 17 © 2008-09 Carnegie Mellon University Improved Quality (Delivered Defect Density) An analysis of 20 projects in 13 organizations showed TSP teams averaged 0.06 defects per thousand lines of new or modified code. Approximately 1/3 of these projects were defect-free. Defects/KLOC 8 7.5 7 6.24 6 4.73 5 4 3 2.28 2 1.05 1 0.06 0 Level 1 Level 2 Level 3 Level 4 Level 5 TSP Source: CMU/SEI-2003-TR-014 TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 18 © 2008-09 Carnegie Mellon University Reviews and Inspections Save Time Xerox found that TSP quality management practices reduced the cost of poor quality by finding and removing defects earlier when costs are lower. Defect Removal Time by Phase 1600 1405 1400 1200 1000 800 Minutes 600 400 200 5 22 2 25 32 0 Design Design Code Code Unit System Review Inspect Review Inspect Test Test Removal Phase TSP Teams Transform Test James D. McHale, 2009 QUEST Chicago 19 © 2008-09 Carnegie Mellon University Intuit Quality Improvement TSP reduced defects found in system test by 60% over the previous two releases of QuickBooks 2007.