Software and Systems Verification (VIMIMA01) Testing process and levels Zoltan Micskei, Istvan Majzik Budapest University of Technology and Economics Fault Tolerant Systems Research Group Budapest University of Technology and Economics Department of Measurement and Information Systems 1 Main topics of the course . Overview (1) o V&V techniques, Critical systems . Static techniques (2) o Verifying specifications o Verifying source code . Dynamic techniques: Testing (7) o Developer testing, Test design techniques o Testing process and levels, Test generation, Automation . System-level verification (3) o Verifying architecture, Dependability analysis o Runtime verification 2 TESTING PROCESS 3 Learning outcomes . Explain the activities and tasks in the typical test process (K2) 4 Overview of testing concepts Software testing Testing Test design Testing in Test levels Test types process techniques the lifecycle 5 Software testing Test design Testing in the Testing process Test levels Test types techniques lifecycle Planning and Control Analysis and Design Imlementation and Execution Evaluating Exit Criteria and Reporting Test closure Source: ISTQB CTFL syllabus 6 Software testing Test design Testing in the Testing process Test levels Test types techniques lifecycle Planning and Control Test planning: Scope, risks, objectives • Test approach Analysis and • Test strategy and/or test policy Design • Required test resources like people, test environments, etc. Imlementation and Execution • Schedule of test analysis and design tasks, test implementation, execution Evaluating Exit Criteria and and evaluation Reporting • Exit criteria such as Coverage criteria Test control: Monitoring, corrections Test closure 7 Test strategy . Guidelines for . Possible example: o What methodology? o Test-driven development o What kinds of tests? o Module & system o What tools? o JUnit & GUI Tester o Who will test? o Developers & test engineers o Exit criteria? o At least 90% statement coverage & every use case requirement o What o Test Report according to documentation? IEEE 829 8 Test plan . Mapping test strategy to the actual test project o Test objectives o Test objects, test environment o Resources, roles, o Schedules . Defining test phases o Length of phase o Exit criteria o Measuring quality of testing 9 Test documentation . IEEE 829 - Standard for Software and System Test Documentation (1998) o Test Plan (SPACEDIRT: Scope, People, Approach, Criteria, Environment, Deliverables, Incidentals, Risks, Tasks) o Test specifications: Test Design, Test Case, Test Procedure Specifications o Test reporting: Test Item Transmittal Report, Test Log, Test Incident Report, Test Summary Report 10 Google “10 minute test plan” . Why do write a plan that is not used and updated? . Keep only the most important o Attributes, Components, Capabilities (ACC) Source: Google Test Analytics - Now in Open Source 11 Software testing Test design Testing in the Testing process Test levels Test types techniques lifecycle Planning and Control What can and should be tested? Analysis and Design • Designing and specifying test cases • Goal Imlementation • Preconditions and Execution • Test steps, test data • Expected results, checks Evaluating Exit Criteria and • Before writing the test code Reporting • Systematic techniques Test closure 12 Software testing Test design Testing in the Testing process Test levels Test types techniques lifecycle Planning and Control • Manual or automatic • Not everything is worth Analysis and automating Design • Executing tests Implementation • Logging and Execution • Time, test environment • Version SUT and system Evaluating Exit Criteria and • Outputs Reporting • … • Incident reporting Test closure 13 Software testing Test design Testing in the Testing process Test levels Test types techniques lifecycle Planning and Control Analysis and Design Implementation • Decision on when to stop testing and Execution • Evaluating metrics Evaluating Exit • Summary reports Criteria and Reporting Test closure 14 Software testing Test design Testing in the Testing process Test levels Test types techniques lifecycle Planning and Control Analysis and Design Implementation and Execution • After major milestones • Collecting experience and feedback Evaluating Exit Criteria and • Finishing and storing reusable Reporting testware (tools, environment) Test closure 15 Software testing Testing Testing Test design Testing in Test types process levels techniques the lifecycle Unit / Module Integration System Acceptance Alpha and beta 16 Software testing Testing Testing Test design Testing in Test types process levels techniques the lifecycle Functional Non- functional Regression … 17 Software testing Testing Testing Test design Testing in Test types process levels techniques the lifecycle Functional Non- • After changes functional • Previous functions are still working • Only subset of tests Regression (test selection) • Test minimization … 18 Software testing Testing Test design Testing in the Testing levels Test types process techniques lifecycle Experience- based Specification- based Structure- based Fault-based Probabilistic 19 Software testing Testing Test design Testing in the Testing levels Test types process techniques lifecycle Experience- based • Ad hoc testing • Exploratory testing Specification- based • Freedom and responsibility of tester Structure- • Test design, execution and based interpretation in parallel Fault-based Probabilistic 20 Software testing Testing Test design Testing in the Testing levels Test types process techniques lifecycle Experience- based • Error guessing Specification- • Previous or typical faults based • Mutation testing Structure- • Mutating the code based – Evaluating tests • Mutating the tests Fault-based – Creating new tests Probabilistic 21 Software testing Testing Testing Test design Testing in Test types process levels techniques the lifecycle V modell Agile … 22 Software testing Test design Testing in the Testing process Testing levels Test types techniques lifecycle Planning and Experience- Unit / Module Functional V modell Control based Analysis and Specification- Integration Non-functional Agile Design based Implementation System Regression Structure-based … and Execution Evaluating Exit Criteria and Acceptance … Fault-based Reporting Test closure Alpha and beta Probabilistic 23 TESTING PRACTICES 24 Testing @ Microsoft . Software Developer Engineer in Test (SDET) . Same career paths as developers o Testing is not an entry position o Test manager is not a promotion but a different path . „Hiring testers to pound quality into a product after it’s been developed is a waste of money.” . 10 year support cycle for major releases o Worth investing in good test automation „How we test software at Microsoft”, Microsoft Press, ISBN 0735624259, 2008. 25 Testing @ Google Roles o Test Engineer (TE) o Software Engineer in Test & Infrastructure (SETI) • Engineering Productivity „The burden of quality is on the shoulders of those writing the code.” „Do not hire too many testers.” 26 Testing Quadrants Source: http://angryweasel.com/blog/riffing-on-the-quadrants/ 27 TEST LEVELS 28 Learning outcomes . Distinguish the scope of different test levels (K2) . Describe the different integration testing approaches (K2) . Recall the goals of system verification and system validation testing (K1) 29 Characteristics of tests in different levels Recommendations from How Google Tests Software: Small Medium Large Execution time < 100 ms < 1 sec As fast as poss. Time limit (kill) 1 minute 5 minutes 1 hour Resource Small Medium Large Network (socket) Mocked only localhost Yes Database Mocked Yes Yes File access Mocked Yes Yes System call No Not recommended Yes Multiple threads Not recommended Yes Yes Sleep No Yes Yes System properties No Yes Yes 30 Testing and test design in the V-model Operation, maintenance Requirement System val. System analysis design validation System System test System specification design verification Architecture Integration test System design design integration Module Module test Module design design verification Module implementation 31 Integration testing Testing the interactions of modules . Motivation o The system-level interaction of modules may be incorrect despite the fact that all modules are correct . Methods o Functional testing: Testing scenarios • Sometimes the scenarios are part of the specification o (Structure based testing at module level) . Approaches o “Big bang” testing: integration of all modules o Incremental testing: stepwise integration of modules 32 Integration testing approaches Integration testing Big bang Incremental Top-down Bottom-up 33 “Big bang” testing . Integration of all modules and testing using the external interfaces of the integrated system . External test executor . Based of the functional specification of the system . To be applied only in case of small systems A Tester1 B Tester2 C Debugging is difficult! D 34 Incremental integration and testing . In case of complex systems (supports debugging) . Adapts to module hierarchy (calling levels) A A A3 A31 A311 A312 A1 A2 A3 A31 A32 A33 A311 A312 A313 35 Top-down integration testing . Modules are tested from the caller modules . Stubs replace the lower-level modules that are called . Requirement-oriented testing . Module modification: modifies the testing of lower levels A Tested module: test executor A1 A2 A3 Module A31 A32 A33 to be tested A311 A312 A313 Test Test Test stub stub stub 36 Bottom-up integration testing . Modules use already tested