W12 Concurrent Session Wednesday 05/07/2008 3:00 PM Systematic Test Design … All on One Page Presented by: Peter Zimmerer Siemens AG Presented at: STAREAST Software Testing Analysis & Review May 5-9, 2008; Orlando, FL, USA 330 Corporate Way, Suite 300, Orange Park, FL 32043 888-268-8770 904-278-0524 [email protected] www.sqe.com Peter Zimmerer Peter Zimmerer is a Principal Engineer at Siemens AG, Corporate Technology, in Munich, Germany. He studied Computer Science at the University of Stuttgart, Germany and received his M.Sc. degree (Diplominformatiker) in 1991. He is an ISTQBTM Certified Tester Full Advanced Level. For more than 15 years he has been working in the field of software testing and quality engineering for object-oriented (C++, Java), distributed, component-based, and embedded software. He was also involved in the design and development of different Siemens in-house testing tools for component and integration testing. At Siemens he performs consulting on testing strategies, testing methods, testing processes, test automation, and testing tools in real-world projects and is responsible for the research activities in this area. He is co-author of several journal and conference contributions and speaker at international conferences, e.g. at Conference on Quality Engineering in Software Technology (CONQUEST), SIGS-DATACOM OOP, Conference on Software Engineering (SE), GI-TAV, STEV Austria, SQS Software & Systems Quality Conferences (ICS Test), SQS Conference on Software QA and Testing on Embedded Systems (QA&Test), Dr. Dobb’s Software Development Best Practices, Dr. Dobb’s Software Development West, Conference on Testing Computer Software, Quality Week, Conference of the Association for Software Testing (CAST), Pacific Northwest Software Quality Conference (PNSQC), PSQT/PSTT, QAI’s Software Testing Conference, EuroSTAR, and STARWEST. He can be contacted at [email protected]. Internet: http://www.siemens.com/research-and-development/ http://www.siemens.com/corporate-technology/ 1 Corporate Technology Systematic Test Design . All on One Page STAREAST 2008 Orlando, FL, USA Peter Zimmerer Principal Engineer Siemens AG, CT SE 1 Corporate Technology Corporate Research and Technologies Software & Engineering, Development Techniques D-81739 Munich, Germany [email protected] http://www.siemens.com/research-and-development/ http://www.siemens.com/corporate-technology/ Copyright © Siemens AG 2008. All rights reserved. Contents Introduction Test design methods Here: methods, paradigms, techniques, styles, and ideas to create, derive, select, generate a test case Examples and references Problem statement Poster Test Design Methods on One Page Guidelines and experiences Summary Page 2 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology What is a Test Case? A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. ISTQB 2007, IEEE 610 A Test Case should include unique identification – who am I? test goal, test purpose – why? test conditions – what? preconditions – system state, environmental conditions test data – inputs, data, actions execution conditions – constraints, dependencies expected results – oracles, arbiters, verdicts, traces postconditions – system state, traces, environmental conditions, expected side effects, expected invariants Page 3 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology Introduction – Test design methods Good test design, i.e. high-quality test cases, is very important There are many different test design methods and techniques Static, dynamic Black-box, white-box, grey-box Based on fault model, experience, exploratory Statistical (user profiles), random (monkey) The tester‘s challenge is to adequately combine these methods dependent on the given problem, domain, and requirements This is art as well! Black-box test design methods are often based on models – model-based testing Page 4 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology Some systematic methods for test design Black-box (models, interfaces, data) Selection, usage and Requirements-based (traceability matrix) applicability depends on the Use case-based testing, scenario testing specific domain (domain Design by contract knowledge is required!) Equivalence class partitioning used software technology Classification-tree method test requirements: required Boundary value analysis test intensity, quality criteria, State-based testing risks Cause-effect graphing existing test basis: Decision tables, decision trees specifications, documents, Combinatorial testing (n-wise) models White-box (internal structure, paths) project factors: constraints Control flow testing and opportunities Data flow testing Page 5 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology Example – There are always too many test cases ... Page 6 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology Examples – Demo Microsoft PowerPoint Microsoft Word 2002 Page 7 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology Test effectiveness and formal (systematic) test design There are studies showing advantages of systematic test design. There are also studies showing advantages of random testing. Æ But do you really want to design your test cases only randomly? Formal test design was almost twice as effective in defect detection per test case as compared to expert (exploratory) type testing, and much more effective compared to checklist type testing. Bob Bartlett, SQS UK, 2006 Page 8 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology Some references Many testing books cover test design to some extent Boris Beizer: Software Testing Techniques Lee Copeland: A Practitioner's Guide to Software Test Design Rick D. Craig, Stefan P. Jaskiel: Systematic Software Testing Tim Koomen et. al.: TMap Next: For Result-driven Testing Glenford J. Meyers: The Art of Software Testing Torbjörn Ryber: Essential Software Test Design James Whittaker: How to Break Software James Whittaker, Herbert Thompson: How to Break Software Security Standard for Software Component Testing by the British Computer Society Specialist Interest Group in Software Testing (BCS SIGIST) (see http://www.testingstandards.co.uk/) There are many different training offerings by different providers Page 9 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology Problem statement Starting from a risk-based testing strategy an adequate test design is the key for effective and efficient testing. Automation of bad test cases is a waste of time and money! There are many different test design methods around for a long time (perhaps too many?) and a lot of books explain them in detail. There are different categorizations, classifications, and dimensions naming, interpretations, and understandings of test design methods which does not simplify their usage … When we look into practice we can see that often there is quite limited usage of these test design methods at all. What are the reasons behind that? How can we overcome this and improve our testing approaches? Page 10 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology Possible reasons and consequences (1) Possible reasons Engineers do not have / spend time to read a book on testing Missing the big picture Information is not available at a glance Which test design methods are there at all? Which test design method should I use in which context? Consequences A specific test design method is not “available” when needed A specific test design method is too detailed or too complicated to be used in practice Often the focus is on depth and on perfectionism Æ Remark: But that can be especially required, e.g. in a safety critical environment! Page 11 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology Possible reasons and consequences (2) Test design tools are typically focused and implement only a few test design methods, e.g.: ATD – Automated Test Designer (AtYourSide Consulting, http://www.atyoursideconsulting.com/): Cause-effect graphing BenderRBT (Richard Bender, http://www.benderrbt.com/): Cause-effect graphing, quick design (orthogonal pairs) CaseMaker (Diaz & Hilterscheid, http://www.casemaker.de/): Business rules, equivalence classes, boundaries, error guessing, pairwise combinations, and element dependencies CTE (Razorcat, http://www.ats-software.de/): Classification tree editor Reactis (Reactive Systems, http://www.reactive-systems.com/): Generation of test data from, and validation of, Simulink and Stateflow models TestBench (Imbus, http://www.testbench.info/, http://www.imbus.de/): Equivalence classes, work-flow / use-case-based testing Page 12 May 7, 2008 Peter Zimmerer, CT SE 1 © Siemens AG, Corporate Technology Poster Test Design Methods on One Page (1) Idea: Systematic, structured, and categorized overview about different test design methods on one page Focus more on using an adequate set of test design methods than on using only one single test design method in depth / perfection Focus more on concrete usage of test design methods than on defining a few perfect test design methods in detail which are not used then in the project Focus more on breadth instead on depth Do not miss breadth because of too much depth Do not miss