Keith Stobie – Scripted Manual Exploratory Testing

M.A.S.E testing (Manual Automated Scripted Exploratory) 20 June 2013

testmuse.wordpress.com 1

Keith Stobie

ASQ CSQE. BBST Foundaons graduate. ISTQB FL. Member of AST, ASQ, ACM, and IEEE, also SASQAG and PSQNC 20 June 2013 Protocol Quality Assurance Process Model-Based Quality Assurance of Protocol Documentaon, STVR, 2011 Too Darn Big to Test – ACM Queue vol.3 (#1)-Feb.2005 Tesng for Excepons, STQE, July/Aug. 2000, 12-16 Keynoted at CAST 2007 and MBT-UC 2012 Test Architecture, Test Design, Test Automaon Copyright © 2013 Keith Stobie, TiVo Test Methodology TiVo, Bing, Microso servers & tools, BEA, Informix, Tandem 2

Context

• Scripted versus exploratory dimension and its interacon with manual versus automated.

• Learn about the forces that influence when automaon 20 June 2013 or manual tesng is most appropriate and • When confirmatory (scripted) or bug finding (exploratory) is most appropriate. • The role and benefit of each type (manual scripted, automated scripted, manual exploratory, automated exploratory). Copyright © 2013 Keith Stobie, TiVo

3 Perfect Software Testing

Soware Tesng – Cem Kaner Perfect Tesng – James Bach an empirical Tesng is the inﬁnite process technical of comparing the invisible invesgaon to the ambiguous 20 June 2013 conducted to provide stakeholders so as to avoid the unthinkable with informaon happening to the anonymous about the quality of the product or service under test

tesng. IEEE Standard for soware and system test documentaon

(IEEE Std 829-2008) Copyright © 2013 Keith Stobie, TiVo (A) An acvity in which a system or component is executed under speciﬁed condions, the results are observed or recorded, and an 4 evaluaon is made of some aspect of the system or component. (B) To conduct an acvity as in (A). Exhaustive vs. Pragmatic Testing

Exhausve / Complete Tesng execute program on all possible inputs and compare actual to expected behavior.

• Could “prove” program correctness. 20 June 2013 • Not praccal for any non-trivial program. Comprehensive Tesng • Usually some (coverage) criteria. Pragmac tesng Select a vanishingly small % of all possible tests. Copyright © 2013 Keith Stobie, TiVo Goal: Execung the ny % of test will uncover a large % of the defects present. 5 A tesng method is essenally a way to decide which ny % to pick.

Associations

Are you familiar with the term: Manual Tesng? 20 June 2013 Automated Tesng? Scripted Tesng? Exploratory Tesng?

It is tradional for testers to be untrained in what I consider the basic cognive skills of excellent tesng. It is tradional, I suppose, for testers to have almost no ability to defend what they do, except through the use Copyright © 2013 Keith Stobie, TiVo of clichés and appeals to authority and folklore. That tradion has poorly served our industry, in my opinion. I'd like to replace that with a tradion 6 of context-driven, skilled tesng -- James Bach hp://www.sasﬁce.com/blog/archives/58 Manual testing What do you think of? 20 June 2013 Copyright © 2013 Keith Stobie, TiVo

8 Scripted testing What do you think of?

A test script in soware tesng is a set of instrucons that will be 20 June 2013 performed on the system under test to test that the system funcons as expected. -- hp://en.wikipedia.org/wiki/Test_script

9 Exploratory testing What do you think of?

Term was coined by Cem Kaner in 1983 in his book Tesng Computer Soware : “Trust your insncts” 20 June 2013 Updated descripon by James Bach: “Exploratory tesng is simultaneous learning, test design, and test execuon, with an emphasis on learning” Design as you test No test scripts

Learn as you test Copyright © 2013 Keith Stobie, TiVo Adapt as you test Agile testing No detailed test planning 10 Lightweight test Find new bugs all the time hp://blogs.msdn.com/b/anuhara/archive/2011/10/20/exploratory-tesng-introducon.aspx hp://swtester.blogspot.com/2012/05/what-is-exploratory-tesng.html Continuum

11 Test & Test Case (IEEE)

3.1.39 test: (A) A set of one or more test cases. (B) A set of one or more test procedures. 20 June 2013 (C) A set of one or more test cases and procedures. (D) The acvity of execung (A), (B), and/or (C). test case. IEEE Standard for soware and system test documentaon (1) (IEEE Std 610.12-1990) A set of test inputs, execuon condions, and expected results developed for a parcular objecve, such as to exercise a parcular program path or to verify compliance Copyright © 2013 Keith Stobie, TiVo with a speciﬁc requirement. (2) (IEEE Std 829-2008) Documentaon specifying inputs, predicted 12 results, and a set of execuon condions for a test item. ScriptedóExploratory Continuum vague fragmentary freestyle exploratory pure scripted scripts test cases charters tours roles 20 June 2013 hp://www.qualitestgroup.com/media/presentaons/ PPT_Exploratory_Tesng_Explained.pdf exploratory side of the connuum

Itkonen, Juha; Mika V. Mäntylä and Casper Lassenius (2007). "Defect Detecon Efficiency: Test Case Based vs. Exploratory Tesng" no significant differences in defect detecon efficiency between {manually execung} test case based tesng (TCT) and {manual} exploratory tesng (ET). [However TCT took more me – to prepare the test cases] Copyright © 2013 Keith Stobie, TiVo no benefit in terms of defect detecon efficiency of using predesigned test cases in comparison to an exploratory tesng approach. – BJ Rollins, 13 hp://www.sigist.org.il/_Uploads/dbsAachedFiles/Empirical_Evaluaoness.pdf

Defect Detection Efficiency: Test Case Based vs. Exploratory Testing

79 advanced soware engineering students performed manual funconal tesng on an open-source applicaon with actual and seeded defects. 20 June 2013 Phase Group 1 Group 2 Preparaon Test cases for feature set A Test cases for feature set B session 1 Test case based tesng Feature Exploratory tesng Tesng 90 min set A Feature set A session 2 Exploratory tesng Tesng Test case based tesng Feature 90 min Feature set B set B The distribuons of detected defects did not differ significantly Copyright © 2013 Keith Stobie, TiVo regarding technical type, detecon difficulty, or severity.

TCT produced significantly more false defect reports than ET. 14 Surprisingly, our results show no benefit of using predesigned test cases in terms of defect detecon efficiency, emphasizing the need for further studies of manual tesng.

Empirical Evaluation of Exploratory Testing Effectiveness

No significant differences between the two approaches in terms of the detected defect types, severies, or detecon difficulty. 20 June 2013 Difference in cricality of defects between the 2 approaches is stascally significant • ET more likely to idenfy behavioral issues (e.g., “look and feel”) • Scripted tests more likely to idenfy technical defects (computaonal / code level) • Both approaches may miss crical defects The overall effecveness of both exploratory tesng and Copyright © 2013 Keith Stobie, TiVo designing effecve scripted tests depends heavily upon the individual tester’s professional knowledge of the system and 15 tesng! Automated óManual Continuum? Rule #1C: If you have a great automated test, it’s not the same as the manual test that you believe

you were automang. 20 June 2013 Manual Tests Rule #1B: If you can truly automate Cannot Be a manual test, it couldn’t have Automated been a good manual test. hp://www.sasﬁce.com/ blog/archives/58 If you read and hand-execute the - James Bach code– if you do exactly what it “human eyes, ears, hands, tells you– then congratulaons, and brains are engaged in you will have performed a poor Copyright © 2013 Keith Stobie, TiVo the execuon of the test” – manual test. Michael Bolton 16 #1: A good manual test cannot be automated. Technique vs. Approach

• A technique is a speciﬁc method that is used to accomplish a speciﬁc goal.

• An approach is the overall manner in which you act. 20 June 2013 • Exploratory Tesng is an approach to tesng. • All test techniques (i.e. manual, funconal) can be done in an exploratory way or a scripted way (predeﬁned test procedures, whether manual or automated) Copyright © 2013 Keith Stobie, TiVo • You can work in an exploratory way at any point in tesng1 hp://www.qualitestgroup.com/media/presentaons/ 1 hp://www.tesngeducaon.org/BBST/exploratory/ 17 PPT_Exploratory_Tesng_Explained.pdf BBSTExploring.pdf Random vs. Ad Hoc vs. Exploratory “To the extent that the next test we do is inﬂuenced by the result of the last test we did, we are doing exploratory tesng. “ – James Bach 20 June 2013 Copyright © 2013 Keith Stobie, TiVo

hp://qatestlab.com/services/No-Documentaon/ad-hoc-tesng/ Coder - Domain Expert continuum

Automated Manual

20 June 2013

Coder Domain Expert

xUnit Keyword Acon word Gherkin Rspec

20 hp://www.methodsandtools.com/archive/archive.php?id=94 Continuum combinations

Exploratory – ﬁnd bugs does not guarantee any type of coverage Numerous heuriscs

20 June 2013 Automated Exploratory Manual Exploratory

Automated Scripted Manual Scripted

Scripted – Conﬁrmaon requires some knowledge of upfront results

Automated – requires hardware and soware Copyright © 2013 Keith Stobie, TiVo may miss side-eﬀect issues 21 Manual – requires human (even if computer assisted) may miss minor issues . Keith’s History & Experience Mostly an automated script tester of Systems and Services (backend). Wrote tools for exploratory API tesng. Did large scale system tesng using random load 20 June 2013 generators. Rare bouts of manual tesng: Bing, Doyenz, TiVo Three years of Model Based Tesng to automacally generate test scripts. Limited GUI tesng (manual or automated) [not my focus] Copyright © 2013 Keith Stobie, TiVo Manual Scripted: 5% Automated Scripted: 85% 22 Manual Exploratory: 5% Automated Exploratory: 5% Manual Exploratory

Google “Z” safe oﬀ Bing – Defects every week

Ken Johnston 20 June 2013 Single leer queries – inappropriate results Bing log analysis : Big Data Abandonment analysis – navigaonal queries on images.

24 TiVo Manual Exploratory bugs

Take home builds

Click 25 Login Manual Detail Charter 20 June 2013

Enter Username and Password Outline

• Click mouse in username ﬁeld to place cursor in username ﬁeld. Type “testuser”.

• Click mouse in password field to place cursor in Copyright © 2013 Keith Stobie, TiVo Specific password field. Type “Pa$$w0rd” • Click on login buon 26 Specific Acon LoginEnteringUsernameAndPassword • • •

string username, string password) Username.Enter Username.Enter Mouse.Position Mouse.Position LoginButton.Click Automated Details (“ (“Pa$$w0rd”) (password).Click (username).Click testuser

Objecve Verify that the TCD is able can make a recording from live TV

Steps 20 June 2013 1. Press the Live TV buon 2. Press the Channel Up buon to change channels and verify A 3. Wait awhile to build up some video in the cache 4. Press the Record buon 5. Press select on the "Record this showing" menu item 6. Allow some me to pass 7. Press the Record buon 8. Press the down buon then select to select "Stop the current recording" 9. Press the TiVo buon (you'll go to the main screen) 10. Press the Select buon. You should be on My Shows (or what ever its name for that product) 11. Verify B Copyright © 2013 Keith Stobie, TiVo

Expected Results A. Verify the new channel has video that is correctly displayed 28 B. Verify the recorded show is present at the top of the show list if you are sorted by date

TiVo Manual Script Test Case 108864 Direct Tuning - cable channels

Objecve

Verify able to direct tune cable channels. 20 June 2013 Steps 1. key in a cable channel number from liveTV such as 201 2. verify A Expected Results

29 TiVo Automated script Test Case 108864 Direct Tuning - cable channels

$controller->gotoScreen('central’)

$controller->gotoLiveTVChannel($channel); 20 June 2013 $result = $controller->{'screendump'} ->parseScreenDump(ﬁleﬁlter => "livetv.xsl"); $tuned_channel = $result->{'channel-number'}; if ( $channel ne $tuned_channel ) { $self->setError("Not able to tune to Channel.");

return FAILURE; Copyright © 2013 Keith Stobie, TiVo } 30 return SUCCESS; Automated Scripted Testing Pros • Low learning curve, no special tools • Low risk, done it for years • Reach, everyone can do it, no tools required • Predictable, repeatable 20June 2013 • Success, proven mechanism to ﬁnd bugs

Cons • Tests require a lot of code, every scenario • Expensive to maintain, cleanup actually lasts for years • Test issues, far surpass product bugs • Low coverage, out of the box Copyright © 2013 Keith Stobie, TiVo • Rigid, diﬃcult to improve, change, or just keep up to date • Stac, stop ﬁnding bugs, same scenarios, missing bugs • Doesn’t scale, increasing feature sets, complexity, dependencies 31

Ed Triou What’s a Model?

A model: Is an abstracon or simpliﬁed representaon of the system from a 20 June 2013 parcular perspecve Supports invesgaon, discovery, explanaon, predicon, or construcon May be expressed as a descripon, table, graphical diagram, or quantave mathemacal model Is not necessarily comprehensive

Inspired by :“Modeling: A Picture's Worth 1000 Words” Copyright (c) 2002, Quality Tree Soware, Inc. Manual Model Exploration End API Simple Finite State Machine (FSM) of an API. Not Started Test Cases (input sequences) 1. begin; end; end endAPI

2. begin; begin beginAPI 20 June 2013 3. end; 4. begin; end; begin; end Proces- sing // Some WCF service interface [ServiceContract] public interface IService begin { API

[OperationContract(AsyncPattern = true)] Copyright © 2013 Keith Stobie, TiVo IAsyncResult BeginGetCustomers(AsyncCallback callback , object state); List EndGetCustomers(IAsyncResult result); 34 } hp://www.codeproject.com/Arcles/56175/A-Generic-Class-for-Wrapping-Asynchronous-Begin-En Automated Exploratory Mine

Accidental – throled process creaon

Dumb Monkey & Fuzzing – Random input 20 June 2013 Random failures – similar to Chaos monkey Random stress tesng

Fuzz Testing providing invalid, unexpected, or random data to the inputs of a computer program

History – a dark & stormy night

Fuzz Revisited: A Re-examinaon of the Reliability of UNIX ... 20 June 2013 by BP Miller, et al hp://www.opensource.org/advocacy/fuzz-revisited.pdf hps://www.owasp.org/index.php/Fuzzing Tools list

36 Fuzz Testing

Goal: push invalid data as far into system as possible. Unexpected input : length and content 20 June 2013 Both Dumb and Smart fuzzing valuable! • Dumb generates data with no regard to the format (data form of dumb Monkey tesng) • Smart requires knowledge of data format or how data is consumed. The more detailed the knowledge – the beer. Generate - creates new data from scratch Mutaon - transforms sample input data to create new input data : add/change/delete data Copyright © 2013 Keith Stobie, TiVo Most fuzzing tools are a mix of each approach 37

Monkey Testing

hp://www.quesoningsoware.com/2007/02/ people-monkeys-and-models.html

Dumb / Wild / Unmanaged

banging on the keyboard by randomly generang input (key- 20 June 2013 presses; and mouse moves, clicks, drags, and drops) Smart / Trained / Managed detects available opons displayed to the user and randomly enters data and presses buons that apply to the detected state of the applicaon Chaos Monkey (nelix.com) hp://techblog.nelix.com/2012/07/chaos-monkey-released-into-wild.html Copyright © 2013 Keith Stobie, TiVo Chaos Monkey is a service which runs in the Amazon Web Services (AWS) that seeks out Auto Scaling Groups (ASGs) and 38 terminates instances (virtual machines) per group.

Flavors of Stress Tests

Load Ensures that system can perform “acceptably” under Input and Resource load both Constant and Varying When capacity\limits are known. Limit 20 June 2013 Equivalent to boundary testing Breakpoint When capacity\limits are not known. Find heaviest load the component can handle, before failure Torture Taking component beyond known or pre-defined min\max limits Duration To run for some amount of time without undue load or failures. (aka “soak” testing)

Synchronization Flush out timing\synchronization\concurrency issues, e.g., Copyright © 2013 Keith Stobie, TiVo multi-thread tests Deterministically inject faults, simulate error conditions Fault Injection 39 Chat Slice where local consistency does matter 20 June 2013 Copyright © 2013 Keith Stobie, TiVo

40 Spec Explorer

point and shoot hp://msdn.microso.com/en-us/library/ee620432.aspx

Move to a parcular state (point), and then explore it

(shoot). 20 June 2013

Also “On The Fly” tesng – connue generang inputs unl me runs out. (Smart “state” monkey).

41 www.SowareTesngSoware.com Manual Testing Advantages Disadvantages Leverage experience of Testers to me consuming find Defects usable for majority of soware not suitable for Performance Tesng, tesng types Stress Tesng and Soak Tesng 20 June 2013 performable in different phases of ROI Is low for tests like Smoke or SDLC Regression Tests performable even when Can lead to defect leakage due to human applicaon is not stable miss or error Effecve when requirements are Not effecve if tests require validaon of volale large data sets Effecve when UI changes are Cannot perform most of the tests related frequent to Security Tesng Testers can lose interest running same tests over longer period of me Success depends on experience and knowledge of testers Copyright © 2013 Keith Stobie, TiVo

42 Manual/Automated Effectiveness

Manual Eﬀecve Automated Eﬀecve Funconal Sanity Unit Smoke 20 June 2013 Exploratory Database Ad-Hoc Regression Integraon API User Acceptance Web Services

hp://www.sowaretesngsoware.com/manual-tesng/ Differences Between Scripted and Exploratory Testing

Scripted Tesng Exploratory Tesng Directed from requirements Directed from requirements and exploring during 20 June 2013 Determinaon of test cases well in Determinaon of test cases during advance tesng • Funconality 1 • Funconality 1 Scripted Conﬁrmaon of tesng with the Invesgaon of system or applicaon • Funconality 4 requirements Tesng • Funconality 2 Exploratory Emphasizes on predicon and decision Emphasizes on adaptability and making • Funconality 3 learning Tesng • Involves conﬁrmed tesng Involves Invesgaon Funconality 2 • Funconality 4 • Funconality 3

Is about Controlling tests Is about Improvement of test design Copyright © 2013 Keith Stobie, TiVo Like making a speech — you read from Like making a conversion — it’s a dra spontaneous 44 The script is in control The tester’s mind is in control

hp://tesnginterviewsquesons.blogspot.com/2013/01/exploratory-tesng.html Beneits of Computer-Assisted Testing • Easy test case maintenance • Reduced costs “You can start with a small model and build in the direcon you think will do the most good 20 June 2013 • More test cases for you. “ • Early bug detecon • Increased bug count • Time savings • Time to address bigger test issues Copyright © 2013 Keith Stobie, TiVo • Improved tester job sasfacon Harry Robinson Exploratory Test Automaon slides from CAST 2010 45 Issues with Random Testing

Veriﬁcaon of the result. Easier to have stac tests with pre-computed outputs.

Very low probability sub-domains 20 June 2013 • likely to be disregarded by random tesng but cost of failures in those sub-domains may be very high • good reason for using it as a complementary rather than the sole tesng strategy. Copyright © 2013 Keith Stobie, TiVo

46 Validation & Veriication

Validaon (requirements conﬁrmaon)

• Establishing the ﬁtness of a soware product for its use. 20 June 2013 • “Are we building the right product?” • Requires interacon with customers

Verificaon (design confirmaon) • Establishing the correspondence between the soware and its specificaon. Copyright © 2013 Keith Stobie, TiVo • “Are we building the product right?” • Requires interacon with soware 47

hp://www.construx.com/Praccing_Earl/Doing_Jusce_to_V_V/ … Conﬁrmaiton Analyze these claims: Becoming a Test Expert – James Bach

You should write a test plan

It’s important that tesng be repeatable 20 June 2013 Each test case should have an expected result Test automaon saves money and me All tesng is based on a model of what is being tested Good enough quality is not good enough An undocumented test cannot be improved Exploratory tesng is a useful pracce Copyright © 2013 Keith Stobie, TiVo

It’s beer to use the term defect than bug 48 Ambiguity should be removed from requirements Conirmation & Bug Finding

Conﬁrm Find bugs

49 Automated Business Facing Manual & Manual

Exploratory Tesng Inc

Funconal Tests Scenarios Examples Usability Tesng Story Tests DragonFire User Acceptance Tesng Prototypes Critique (UAT) 20 June 2013 Simulaons Alpha / Beta Q2 Q3 Product Q1 Q4 used with permission.

Supporting the Team Team the Supporting Performance & Load Unit Tests Tesng Component Tests Security Tesng Copyright © 2013 Keith Stobie, TiVo Copyright 2009: Lisa Crispin, Janet Gregory – hp://warmelon.com/2011/06/10/yet-another-soware-tesng-pyramid/ “ility” Tesng 50

Automated Agile Testing Quadrants Tools Technology Facing Construx Validation Test list Level Test Type Description Notes 0 Unit Do individual code units (i.e., Typically done by developer to modules) work by themselves? check the integrity and functionality of code units. 0 Code Do individual code units work well Typically done by developer to Integration with one another? insure code units interface 20 June 2013 successfully with one another. 1 Build Can the application be successfully Done by Tester Acceptance installed? 1 Smoke Does the build work well enough to Done by Tester be tested? 1 Regression Have bug fixes or design Executed immediately after a enhancements in the code caused successful Smoke Test. Tester other parts of the program to reruns tests proved successful in

malfunction? the previous build. Copyright © 2013 Keith Stobie, TiVo 2 Functional Are all the features and functions Should cover both described in the Functional Front End (UI, GUI) as well as 51 Specifications implemented and Backend (i.e., data validation). working properly? Conducted by QA as Black Box testing. Construx Validation Test list Level Test Type Description Notes 3 Desktop Does the program: Integrates the program into the Configuratio 1) Work in the communications, hardware environment, n/Host hardware and software environment confirms that it can run Integration described in the Requirements concurrently with other typical documents? software and that all 20 June 2013 2) Run successfully with other typical necessary resources (local programs loaded? and remote) can be accessed. 3) Interface successfully with remote Conducted by QA as Black network resources (e.g., database Box testing. servers) and local peripherals (e.g., printers and modems)? 4 Load / Does the program: Finds out how well the Performance 1) Work acceptably fast for the user program handles under less under typical conditions? than optimal conditions. 2) Work acceptably well when subject Managed by QA but often to extreme processor activity or involves automation or Copyright © 2013 Keith Stobie, TiVo volumes of inputs/outputs? volunteer ad hoc testers and 3) Work acceptably well with limited network personnel to create 52 resources? atypical conditions. Level Level 5 4 6 Construx Validation Test list

Production Test in ance Accept- Compliance Security Test Type

Is the service: policy/procedure. organizational and government threaten the product Does 2) 1) to expose: threaten the program Does confirmed? systembeen to security end end Has Description with real load load real with 1) path. for error every instructions 5) scenarios user typical 4) functions) and features desired 3) 2) 1) Private Customer Information? Customer Private assets? Corporate Does the service operate correctly correctly operate the service Does offering driven: Error flow and options efficiently driven: flow Control handles user all (includes Correct? Complete? (if applicable) Customizable? to use? Comfortable

continual satisfaction satisfaction continual the system for Monitors satisfaction. user ultimate for the program Checks procedures. and policies organizational and industry governmental, with compliance assures and access from unauthorized information user private assets and corporate of protection Insures Notes

Stac

• Soware inspecons 20 June 2013 • Stac analysis of source code • control/data ﬂow analysis Dynamic • Defect tesng – look for errors in funconality

• Load tesng – look for errors in scalability, performance, Copyright © 2013 Keith Stobie, TiVo reliability. 54 An Evaluaon Scheme of Soware Tesng Techniques, H-D. Chu, Univ. of Newcastle, Dept. of CS, TR 583, June 1997 Software Testing Techniques

Soware tesng techniques no must execute the soware? yes Stac Dynamic no examine yes no code yes 20 June 2013 code syntax? examine? Syntacc Semanc Black-box White-box

How is test data selected? Random Funconal Structural

Discrete Connuous Fault-based Error-based What do you think?

• Is it “tesng” if you simply run the program to see what

it does? 20 June 2013 • Is requirements conﬁrmaon done best with stac or dynamic tesng? • Is design conﬁrmaon done best with stac or dynamic tesng?

56 A Beginners Guide to Tesng – Phillip Johnson, Informaon & CS, U. of Hawaii Acceptance Breadth Equivalence Model-Based … Testing Accessibility Business Logic Paroning Mutaon Acve Code-driven Fault injecon Modularity-driven Stac Agile Compability Formal veriﬁcaon Non-funconal Stability Age Comparison Funconal Negave Smoke Ad-hoc Component Fuzz Operaonal Storage Alpha Conﬁguraon Gorilla Orthogonal array Stress Asseron Condion Gray Box Pair Structural

API Coverage Glass box Passive System 20 June 2013 All-pairs Compliance GUI soware Parallel System Integraon Automated Concurrency Globalizaon Path Tesng in Basis Path Conformance Hybrid Integraon Penetraon Producon Backward Context Driven Integraon Performance Top Down Compability Conversion Interface Qualiﬁcaon Integraon Beta Decision Coverage Install/uninstall Ramp Thread Upgrade Benchmark Destrucve Internaonalizaon Regression Big Bang Integraon Dependency Inter-Systems Recovery Unit Binary Portability Dynamic Keyword-driven Requirements User Interface Copyright © 2013 Keith Stobie, TiVo Black box Domain Load Security Usability Boundary Value Error-Handling Localizaon Sanity Volume Vulnerability Boom Up End-to-end Loop Scenario 57 White box Integraon Endurance Manual Scripted Scalability Workﬂow Branch Exploratory Manual-Support Statement Manual Exploratory

+ Cost eﬀecve one-shot bug discovery - No coverage guarantees, varies every me (for beer or worse) 20 June 2013

Tester requires extensive background or knowledge Numerous heuriscs

58 Manual Scripted for rapidly changing product + can guarantee some coverage ⬇ Manual exploratory doesn’t guarantee coverage 20 June 2013 ⬇ Automated scripted maintenance can be too high + Costly automaon vs. cheap human Moravec's paradox : reasoning vs. sensorimotor, e.g., sound quality, video quality, captcha - May miss “minor” changes, e.g., typos, wrong font, … Copyright © 2013 Keith Stobie, TiVo • Requires upfront result knowledge Asperger syndrome highly talented specialist people 59

Automated Scripted

+ Guaranteed coverage for slowly or non-changing product - Misses many “side” observaons, and/or very fragile 20 June 2013 • Requires upfront result knowledge

60 Regression vs Latent

Wring tests and running tests both have costs. Run same tests, reduce regression defects. Regression: old code fails due to new code 20 June 2013 Automaon mostly helps here Create new tests, reduce latent defects (always there). Manual tesng can most help here Copyright © 2013 Keith Stobie, TiVo Regression Latent 61 Old Code New Code Regression Risk vs Code Base

250 Example for 10 iteraons when each new 20 units of code 100% has 5% errors and old code 1% regression errors.

200 20 June 2013 Old Code % of bugs New Code 150 Units of Code Regression Risk 50% due to regression 100

0 0% 1 2 3 4 5 6 7 8 9 10 11 62

Iteraon / Sprint / Release Automated Exploratory

+ Finds defects that other approaches miss - Incomplete, can’t cover most of products today 20 June 2013 • Numerous heuriscs

63 For each Type or Technique an approach to testing

Automated Exploratory Manual Exploratory Pex (pexforfun.com) ModelJUnit, genec Modifying in the debugger Many tools, e.g. xUnit Unit / Structural Debugger script 20 June 2013 Automated Scripted Manual Scripted

Automated Exploratory Manual Exploratory Broad, not deep Smoke Primary paths (for slow changing) most basic use cases (for fast changing) Automated Scripted Manual Scripted Copyright © 2013 Keith Stobie, TiVo Automated Exploratory Manual Exploratory Model Based Tesng heuriscs Funconal / Black Box 64 for most common/crical no ROI to automate Automated Scripted Manual Scripted Automated Unit Testing Current automac unit generaon test techniques: • Simple known failure modes (e.g., null parameters as inputs) •

analysis of the data structures used. 20 June 2013 Data structure generaon can be based on Constraint Solvers, Models, and symbolic execuon You may see 100% code coverage, with no results checking! Be aware of results checking thoroughness; not just code coverage. Eclat creates approximate test oracle from a test suite, then uses it to aid in generang test inputs likely to reveal bugs or expose new behavior while ignoring those that are invalid Copyright © 2013 Keith Stobie, TiVo inputs or that exercise already-tested funconality 65 Manual Script Exactness Avoiding Overkill in Manual Regression Tesng - Lisa Shepard hp://www.uploads.pnsqc.org/2012/papers/t-46_Shepard_paper.pdf

Jonathan Kohl, hp://www.methodsandtools.com/archive/archive.php?id=65 Marick hypothesis:

hp://www.exampler.com/blog/2008/03/23/an-alternave-to-business-facing-tdd/

An applicaon built with programmer TDD, whiteboard-

style and example-heavy business-facing design, 20 June 2013 exploratory tesng of its visible workings, and some small set of automated whole-system sanity tests will be cheaper to develop and no worse in quality than one that diﬀers in having minimal exploratory tesng, done through the GUI, plus a full set of business-facing TDD tests derived from the example-heavy design.

69 Automated Exploratory Testing

Genec Algorithms -- global opmizaon vs random

Unit, Funconal, Concurrency 20 June 2013 Guided -- Kyle A. Larsen – coverage exploraon of virus detecon Don Sleuts – SQL grammar & negave grammar

Large (e.g. nested) SQL legal inputs Copyright © 2013 Keith Stobie, TiVo SQL almost correct (SQL monkey, Fuzzing) 70 Basic outline of Genetic Algorithm Input vars == genes, set of inputs == chromosomes.

Inialize (populaon) -- global opmizaon

Evaluate (populaon) 20 June 2013 While (stopping condion not sasfied) { Selecon (populaon) -- most fit => most likely to swap Crossover (populaon) -- swap “genes” Mutate (populaon) -- small changes to small group Evaluate (populaon) -- kill off less fit Copyright © 2013 Keith Stobie, TiVo } 71 A Survey on Soware Tesng Techniques using Genec Algorithm JCSI Internaonal Journal of Computer Science Issues, Vol. 10, Issue 1, No 1, January 2013 ISSN (Print): 1694-0784 | ISSN (Online): 1694-0814 hp://ijcsi.org/papers/IJCSI-10-1-1-381-393.pdf Testing real-time systems using genetic algorithms

use genec algorithms to ﬁnd inputs with the longest or

shortest execuon mes automacally. 20 June 2013 check whether they produce a temporal error. The ﬁtness funcon is the execuon me measured in processor cycles. Experiments using genec algorithms on a number of programs with up to 1511 LOC and 843 integer input

Soware Quality Journal, 1997, Volume 6, Issue 2, pp 127-135 Machine vs. Human

Machine Human Chess Play (deep search) Chess Play (paern match) Learning (training data) Learning 20 June 2013 Exploring Exploring

Advanced Chess is a form of chess developed in 1998 by Kasparov where a human plays against another human, and both have access to computers to

“Computer assisted tesng

that supports learning of new informaon 20 June 2013 about the quality of the soware under test” -- Douglas Hoﬀman & Cem Kaner “automated” tests are enrely scripted tests – Michael Bolton

74 For each Type or Technique an approach to testing

Automated Exploratory Manual Exploratory (aka Stress!) Performance Poke & Measure Many tools “exercises” 20 June 2013 Automated Scripted Manual Scripted

Automated Exploratory Manual Exploratory Reachability, GUITAR, AUTO Usability Works for me? Automated checks Speciﬁcs to look for Automated Scripted Manual Scripted Copyright © 2013 Keith Stobie, TiVo Automated Exploratory Manual Exploratory Dynamic fuzzing Social aacks Security 75 Script kiddies Captcha Automated Scripted Manual Scripted Memes about Test Automation

“Automated tests represent the automaon of a manual process.” – Harold F. Tipton, Micki Krause 20 June 2013 “The most important benefit of automated tesng over convenonal manual tesng is the minimizaon of costs over repeated tests.” – Markus Helfen, Michael Lauer “I have never been convinced that finding ‘new’ bugs is a realisc expectaon for test automaon.” – I.M. Testy “Aer you have run your automated tests for the first me, you are done finding new bugs. Never again will you find a new issue. ” – Steve Rowe Copyright © 2013 Keith Stobie, TiVo

“Running automated test scripts can not be used to ﬁnd new 76 bugs in the soware ...” – Cordell Vail

The Netlix Simian Army hp://techblog.nelix.com/2011/07/nelix-simian-army.html

Chaos Monkey randomly disables producon instances

Freely available on github 20 June 2013 Latency Monkey induces arficial delays Security Monkey finds security violaons or vulnerabilies 10-18 Monkey (short for Localizaon-Internaonalizaon, or L10n-i18n) detects configuraon and run me problems in instances serving customers in mulple geographic regions, using different languages and character sets. Copyright © 2013 Keith Stobie, TiVo Chaos Gorilla simulates an outage of an enre Amazon 77 availability zone. Hybrid Approach An Innovave Approach to Model-based Tesng Heurisc Search Phases

Random Search Phases • Hybrid approach combines pure random tesng with heurisc search strategies in interleaving phases to increase overall coverage • User can choose between various coverage goals, exit criteria and search strategies

TestIstanbul Conferences 2012 Fake Trafic (capacity test)

User Frontend Backend Service Requests service service 20 June 2013 replayed Requests Data Data Feed Producon

79 Capacity test a single service, or enre oﬀering (via the frontend) From: Greg Veith – Workshop on Performance & Reliability 2010 (WOPR14) Modern Cap Generation Toolset DC1 DC3 Monitoring system Monitoring system

Capacity clients Capacity clients

VIP VIP 20 June 2013

Bing Entry point Controller Bing entry point

DC2 DC…n Monitoring system Monitoring system

Capacity clients Capacity clients

VIP VIP

1. Controller drives each of the clients across any/all DCs 80 2. Capacity is generated within each datacenter to avoid cross-DC network ulizaon 3. Controller pulls data from monitoring system auto-throling based on SLA 4. 2 -3 engineers required for a typical full scale test Harry Robinson Rules of Thumb

Consistency Heurisc

Birthdates 20 June 2013

Abraham Lincoln 2/12/1809 Abe Lincoln 12/2/1809

81 Exploratory Test Automation www.kaner.com/pdfs/highvolCSTER.pdf Examples hp://www.associaonforsowaretesng.org/wp-content/files/ Kaner_Hoffman-ExploratoryTestAutomaon.pdf 1. Disk buffer size 2. Simulate events with diagnosc probes

3. Database record locking 20 June 2013 4. Long sequence regression tesng 5. Funcon equivalence tesng (sample or exhausve comparison to a reference funcon) 6. Funconal tesng in the presence of background load 7. Hosle data stream tesng 8. Simulate the hardware system under test (compare to actual system) 9. Comparison to self-verifying data Copyright © 2013 Keith Stobie, TiVo 10. Comparison to a computaonal or logical model or some other oracle 11. State-transion tesng without a state model (dumb monkeys) 82 12. State-transion tesng using a state model (terminate on failure rather than aer achieving some coverage criterion) 13. Random inputs to protocol checkers For More Information

83 References Exploratory Tesng hp://blogs.msdn.com/b/anuhara/archive/2011/10/20/exploratory- tesng-introducon.aspx hp://tesnginterviewsquesons.blogspot.com/2013/01/exploratory- 20 June 2013 tesng.html hp://warmelon.com/2011/06/10/yet-another-soware-tesng- pyramid/ hp://swtester.blogspot.com/2012/05/what-is-exploratory- tesng.html hp://www.sasﬁce.com/arcles/what_is_et.shtml hp://www.goforexam.in/2012/01/what-is-diﬀerence-between- scripted.html Copyright © 2013 Keith Stobie, TiVo hp://www.methodsandtools.com/archive/archive.php?id=65 84 hp://www.tesngeducaon.org/BBST/exploratory/BBSTExploring.pdf

References Automated Exploratory hp://www.associaonforsowaretesng.org/wp-content/

ﬁles/Kaner_Hoﬀman-ExploratoryTestAutomaon.pdf 20 June 2013 kaner.com/pdfs/VISTACONexploratoryTestAutmaon.pdf www.kaner.com/pdfs/highvolCSTER.pdf

85 References Model Based Tesng

Hendrickson, E. “A Picture’s Worth 1000 Words,” STQE

Magazine, September/October 2002 20 June 2013 PNSQC 2002 Winant, B. “Modeling Pracce and Requirements,” Sckyminds.com. Available: hp://www.sckyminds.com/ se/s3565.asp Harry’s Model-Based Tesng website:

86 References Manual vs Automated hp://www.sowaretesngsoware.com/manual-tesng/ hp://www.methodsandtools.com/archive/archive.php? 20 June 2013 id=94 hp://qatestlab.com/services/We-Are-Professionals-in/ automated-tesng/ Copyright © 2013 Keith Stobie, TiVo