M.A.S.E testing (Manual Automated Scripted Exploratory) 20 June 2013 Keith Stobie So*ware Quality Engineering Architect, TiVo Copyright © 2013 Keith Stobie, TiVo testmuse.wordpress.com 1 Keith Stobie ASQ CSQE. BBST Foundaons graduate. ISTQB FL. Member of AST, ASQ, ACM, and IEEE, also SASQAG and PSQNC 20 June 2013 Protocol Quality Assurance Process Model-Based Quality Assurance of Protocol Documentaon, STVR, 2011 Too Darn Big to Test – ACM Queue vol.3 (#1)-Feb.2005 Tesng for Excepons, STQE, July/Aug. 2000, 12-16 Keynoted at CAST 2007 and MBT-UC 2012 Test Architecture, Test Design, Test Automaon Copyright © 2013 Keith Stobie, TiVo Test Methodology TiVo, Bing, Microso* servers & tools, BEA, Informix, Tandem 2 Context • Scripted versus exploratory dimension and its interacHon with manual versus automated. • Learn about the forces that influence when automaon 20 June 2013 or manual tesHng is most appropriate and • When confirmatory (scripted) or bug finding (exploratory) is most appropriate. • The role and benefit of each type (manual scripted, automated scripted, manual exploratory, automated exploratory). Copyright © 2013 Keith Stobie, TiVo 3 Perfect Software Testing So#ware Tes+ng – Cem Kaner Perfect Tes+ng – James Bach an empirical TesHng is the infinite process technical of comparing the invisible invesHgaon to the ambiguous 20 June 2013 conducted to provide stakeholders so as to avoid the unthinkable with informaon happening to the anonymous about the quality of the product or service under test tesng. IEEE Standard for so*ware and system test documentaon (IEEE Std 829-2008) Copyright © 2013 Keith Stobie, TiVo (A) An acHvity in which a system or component is executed under specified condiHons, the results are observed or recorded, and an 4 evaluaon is made of some aspect of the system or component. (B) To conduct an acHvity as in (A). Exhaustive vs. Pragmatic Testing Exhausve / Complete Tesng execute program on all possible inputs and compare actual to expected behavior. • Could “prove” program correctness. 20 June 2013 • Not pracHcal for any non-trivial program. Comprehensive Tesng • Usually some (coverage) criteria. Pragmac tesng Select a vanishingly small % of all possible tests. Copyright © 2013 Keith Stobie, TiVo Goal: ExecuHng the Hny % of test will uncover a large % of the defects present. 5 A tes6ng method is essenHally a way to decide which Hny % to pick. Associations Are you familiar with the term: Manual Tes+ng? 20 June 2013 AutomateD Tes+ng? Scripted Tesng? Exploratory Tesng? It is tradiHonal for testers to be untrained in what I consider the basic cogniHve skills of excellent tesHng. It is tradiHonal, I suppose, for testers to have almost no ability to defend what they do, except through the use Copyright © 2013 Keith Stobie, TiVo of clichés and appeals to authority and folklore. That tradiHon has poorly served our industry, in my opinion. I'd like to replace that with a tradiHon 6 of context-driven, skilled tesHng -- James Bach hnp://www.sasfice.com/blog/archives/58 Manual testing What Do you think of? 20 June 2013 Copyright © 2013 Keith Stobie, TiVo 7 Automated testing What Do you think of? 20 June 2013 Copyright © 2013 Keith Stobie, TiVo 8 Scripted testing What Do you think of? A test script in so*ware tesHng is a set of instrucHons that will be 20 June 2013 performed on the system under test to test that the system funcHons as expected. -- hnp://en.wikipedia.org/wiki/Test_script Test script is a set of instrucHons (or steps) to follow in a prescribed sequence. Copyright © 2013 Keith Stobie, TiVo 9 Exploratory testing What Do you think of? Term was coined by Cem Kaner in 1983 in his book TesHng Computer So*ware : “Trust your insHncts” 20 June 2013 Updated descripHon by James Bach: “Exploratory tesHng is simultaneous learning, test design, and test execuHon, with an emphasis on learning” Design as you test No test scripts Learn as you test Copyright © 2013 Keith Stobie, TiVo Adapt as you test Agile testing No detailed test planning 10 Lightweight test Find new bugs all the time hp://blogs.msdn.com/b/anunhara/archive/2011/10/20/exploratory-tesHng-introducHon.aspx hp://swtester.blogspot.com/2012/05/what-is-exploratory-tesHng.html Continuum hp://swtester.blogspot.com/2012/05/ what-is-exploratory-tesHng.html 20 June 2013 Copyright © 2013 Keith Stobie, TiVo 11 Test & Test Case (IEEE) 3.1.39 test: (A) A set of one or more test cases. (B) A set of one or more test procedures. 20 June 2013 (C) A set of one or more test cases and procedures. (D) The acHvity of execuHng (A), (B), and/or (C). test case. IEEE Standard for so*ware and system test documentaon (1) (IEEE Std 610.12-1990) A set of test inputs, execuHon condiHons, and expected results developed for a parHcular objecHve, such as to exercise a parHcular program path or to verify compliance Copyright © 2013 Keith Stobie, TiVo with a specific requirement. (2) (IEEE Std 829-2008) Documentaon specifying inputs, predicted 12 results, and a set of execuHon condiHons for a test item. ScriptedóExploratory Continuum vague fragmentary freestyle exploratory pure scripted scripts test cases charters tours roles 20 June 2013 hp://www.qualitestgroup.com/media/presentaons/ PPT_Exploratory_TesHng_Explained.pdf exploratory side of the con6nuum Itkonen, Juha; Mika V. Mäntylä and Casper Lassenius (2007). "Defect DetecHon Efficiency: Test Case Based vs. Exploratory TesHng" no significant differences in defect detecHon efficiency between {manually execu6ng} test case based tesHng (TCT) and {manual} exploratory tesHng (ET). [However TCT took more 6me – to prepare the test cases] Copyright © 2013 Keith Stobie, TiVo no benefit in terms of defect detecHon efficiency of using predesigned test cases in comparison to an exploratory tesHng approach. – BJ Rollins, 13 hp://www.sigist.org.il/_Uploads/dbsAnachedFiles/Empirical_Evaluaoness.pdf Defect Detection Efficiency: Test Case Based vs. Exploratory Testing 79 advanced so*ware engineering students performed manual funcHonal tesHng on an open-source applicaon with actual and seeded defects. 20 June 2013 Phase Group 1 Group 2 Preparaon Test cases for feature set A Test cases for feature set B session 1 Test case based tesHng Feature Exploratory tesHng TesHng 90 min set A Feature set A session 2 Exploratory tesHng TesHng Test case based tesHng Feature 90 min Feature set B set B The distribuHons of detected defects did not differ significantly Copyright © 2013 Keith Stobie, TiVo regarding technical type, detecHon difficulty, or severity. TCT proDuced significantly more false defect reports than ET. 14 Surprisingly, our results show no benefit of using predesigned test cases in terms of defect detecHon efficiency, emphasizing the need for further studies of manual tesHng. Empirical Evaluation of Exploratory Testing Effectiveness No significant differences between the two approaches in terms of the detected defect types, severiHes, or detecHon difficulty. 20 June 2013 Difference in criHcality of defects between the 2 approaches is stasHcally significant • ET more likely to idenHfy behavioral issues (e.g., “look and feel”) • Scripted tests more likely to idenHfy technical defects (computaonal / code level) • Both approaches may miss criHcal defects The overall effecHveness of both exploratory tesHng and Copyright © 2013 Keith Stobie, TiVo designing effecHve scripted tests depends heavily upon the inDiviDual tester’s professional knowleDge of the system and 15 tesHng! Automated óManual Continuum? Rule #1C: If you have a great automated test, it’s not the same as the manual test that you believe you were automa6ng. 20 June 2013 Manual Tests Rule #1B: If you can truly automate Cannot Be a manual test, it couldn’t have AutomateD been a good manual test. hnp://www.sasfice.com/ blog/archives/58 If you read and hand-execute the - James Bach code– if you do exactly what it “human eyes, ears, hands, tells you– then congratulaons, and brains are engaged in you will have performed a poor Copyright © 2013 Keith Stobie, TiVo the execuHon of the test” – manual test. Michael Bolton 16 #1: A good manual test cannot be automated. Technique vs. Approach • A technique is a specific method that is used to accomplish a specific goal. • An approach is the overall manner in which you act. 20 June 2013 • Exploratory TesHng is an approach to tesHng. • All test techniques (i.e. manual, funcHonal) can be done in an exploratory way or a scripted way (predefined test procedures, whether manual or automated) Copyright © 2013 Keith Stobie, TiVo • You can work in an exploratory way at any point in tesHng1 hp://www.qualitestgroup.com/media/presentaons/ 1 hp://www.tesHngeducaon.org/BBST/exploratory/ 17 PPT_Exploratory_TesHng_Explained.pdf BBSTExploring.pdf Random vs. Ad Hoc vs. Exploratory “To the extent that the next test we do is influenced by the result of the last test we did, we are doing exploratory tesHng. “ – James Bach 20 June 2013 Copyright © 2013 Keith Stobie, TiVo 18 hp://qatestlab.com/services/No-Documentaon/ad-hoc-tesHng/ Coder - Domain Expert continuum Automated Manual 20 June 2013 Coder Domain Expert xUnit Keyword Acon word Gherkin Rspec test Copyright © 2013 Keith Stobie, TiVo libraries FIT scriptless flowchart 19 Automation goal 20 June 2013 Copyright © 2013 Keith Stobie, TiVo 20 hp://www.methodsandtools.com/archive/archive.php?id=94 Continuum combinations Exploratory – find bugs does not guarantee any type of coverage Numerous heurisHcs 20 June 2013 AutomateD Exploratory Manual Exploratory AutomateD ScripteD Manual ScripteD Scripted – Confirmaon requires some knowledge of upfront results Automated – requires hardware and so*ware Copyright © 2013 Keith Stobie, TiVo may miss side-effect issues 21 Manual – requires human (even if computer assisted) may miss minor issues . Keith’s History & Experience Mostly an automated script tester of Systems and Services (backend). Wrote tools for exploratory API tesHng. Did large scale system tesHng using random load 20 June 2013 generators.