How Code Coverage Analysis can Improve Software Testing (Wie Codeabdeckungsanalyse den Softwaretest verbessern kann) Ingo Nickles Sr. Field Application Engineer Vector Software © Vector Software Agenda . Vector Software – Who we are . Basics of code coverage analysis . Unit-, integration, and system tests . Change-based test selection . White-box tests with the Vector toolchain . Live Demo © Vector Software Vector Software – Who we are © Vector Software Vector Software . Founded in 1990 • First product released for Lockheed Martin’s C-130J Super Hercules . Corporate headquarters in Rhode Island, USA • US HQ – Providence, Europe HQ – London, Asia HQ – Yokohama • Worldwide Presales, Sales, and Technical Support – Direct and Indirect • Development in US, UK, and India • 90+ Employees • 100+ Partners TÜV SÜD Certified • 900+ Customers Software Tool for Safety Related . Global Services Development • On-site training, best practice workshops, project delivery . Since 2017 part of Vector Informatik, Stuttgart © Vector Software Where does VectorCAST Fit in Your Environment? VectorCAST Requirements Gateway • Dassault Systèmes® Reqtify® • IBM Rational DOORS® Requirements System • IBM Rational RequisitePro® Specification Testing • Jama Software® Compiler Integrations • Microsoft Excel® • Analog Devices V-DSP • Microsoft Word® • ARM RealView • Polarion® REQUIREMENTS™ • Cosmic Software • PTC Integrity® • FreeScale Code Warrior • Siemens ® Teamcenter™ • Fujitsu Software • Standard XML Integration • GNU / GNU Cross • Visure IRQA Architecture Testing • Green Hills MULTI • Hightec • IAR • Intel VectorCAST Integrations • Keil uVision • Atego Artisan Studio • Metaware • IBM Rational Rhapsody® System Unit • Microchip • MathWorks Simulink® • Microsoft VisualStudio Design Testing • NEC • Paradigm C++ Pro. • Renesas • STMicroelectronics • TASKING Compatible Static Analysis Tools • Texas Instruments CCS • Coverity® Test Advisor™ • Wind River • Gimpel Lint™ • Klocwork Insight™ Coding • MathWorks Polyspace® • PRQA™ QA·C|QA·C++ © Vector Software Testing: Early, Plenty, Thoroughly and Continuously Integration Static Analyis Unit Testing System Testing Testing Requirements Based Testing Code Coverage Based Testing Robustness Testing New Release: Regression Testing Continuous Integration: „Clean base“ Testing of each code change Parallel Testing Change Based Testing © Vector Software Basics of Code Coverage Analysis © Vector Software Why measure Code Coverage? . An important metric for certification . An important metric in testing • Testing (In-)Completeness • Quality/Adequacy of test cases • Requirements Traceability . An important metric for change based testing © Vector Software Code Coverage Level . Function coverage • Each function should be called . Function call coverage • Each function call should be executed . Statement coverage • Each line of code should be executed . Decision (Branch) coverage • Each decision should be executed with outcome True and False . Condition coverage • Each condition in each decision should be executed with outcome True and False . Condition/Decision coverage • Combination of Condition and Decision Coverage . Modified condition/decision coverage (MC/DC) • Condition/Decision Coverage with verification that conditions are independant © Vector Software Code Coverage and Certification High Software Quality DAL/SIL/Level/Class Code Coverage Level Code Coverage Low Software Quality © Vector Software Why achieve 100% Code Coverage? . 100% Code Coverage means • Each part of the code was executed without crashing • There is no unreachable code • We tested thoroughly and archieved a minimum number of test cases • “Program elements […] covered by any test case see about half as many bug-fixes as those not covered” * . 100% Code Covearge does not mean • The code is correct • The code is complete • The test is complete *Source: Can Testedness be Effectively Measured? Rahul Gopinath; Oregon State University; based on 250 real world programs © Vector Software Unit Test Integration Test System Test Coding © Vector Software How often should we test the software? Phase of the S/W Dev Relative cost to fix a bug* Testing Phase ~Cost/Bug** Maintenance 100x System Test $5000 Testing 15x Integration Test $500 Implementation 6.5x Full Build $50 Design 1 Unit Test $5 . As soon and as often as possible! . Find any bug in early stage => reduce the cost of fixing the bug Cost of fixing a bug depending of time passed since introduction. * DevOps: Shift left with continuous testing by using automation and virtualization (IBM; Dibbe Edwards) ** How Google Tests Software (James A. Whittaker, Jason Arbon, and Jeff Carollo) © Vector Software Functional Testing needs Requirements Customer Request Requirements System Analysis Testing . Basis for SW Development System Link Test Cases Requirements . Basis for SW Tests (functional) High Level Integration . Allows Traceability Design Testing • Requirement <-> Test Case High-Level Link Test Cases • Requirement <-> Source Code Requirements . Allows Change Impact- Analysis Detailed Unit • When source code changes: Design Testing • -> what requirement might be affected Low-Level Link Test Cases • When a requirement changes: Requirements • -> what source code might be affected • -> what test case might be affected Coding © Vector Software Change-based Test Selection Test Less, Fail Faster © Vector Software Test Cases in a Project Test Cases System System Requirements Test Cases High-Level Integration Requirements Test Cases Low-Level Unit Requirements Test Cases Coding Source Code © Vector Software Change-based Test case selection . What can change? • Requirement • Invalidate Test Cases linked to that Requirement • Source Code Traceability can give a clue what code needs to be changed • Test Case • Re-run the changed Test Case • Source Code • A real challenge • Do we need to execute all test cases? • Tools can help reducing the test execution times with an intelligent Test Case Selection © Vector Software Continuous Testing of Code Changes Test Cases System System Requirements Test Cases High-Level Integration Requirements Test Cases Low-Level Unit Requirements Test Cases Coding Source Code Code Change © Vector Software White-box tests with the Vector toolchain © Vector Software vTESTstudio Demo Setup VectorCAST VectorCAST Source Code Instrumented Source Code Execute Test-Add Coverage CANoe Instrument Execute Test-Multiply Coverage Execute Test-Subtract Coverage Reporting © Vector Software Live Demo © Vector Software © Vector Software .