Vector India Conference 2019
Embedded Testing for the Complete Application
V0.1 | 2019-05-10 Agenda
➢ VectorCAST/QA and Functional Test
➢ Change Based Testing at Black Box
➢ Gray Box Testing using VectorCAST/Probe
➢ Demo
➢ Q&A
2 VectorCAST/QA and Functional Test
3 VectorCAST/QA and Functional Test What is Functional Or Black Box Test?
Functional testing is a way of checking software to ensure that it has all the required functionality that's specified within its functional requirements.
Black box testing is defined as a testing technique in which functionality of the application under test is tested without looking at the internal code structure, implementation details and knowledge of internal paths of the software.
4 VectorCAST/QA and Functional Test Test Coverage
REQUIRMENTS and CODE are two different entities and validating just one of these would be unfair to the other one.
Test Coverage = Requirement Coverage + Code Coverage
Requirement coverage is ensured by tracing the requirements to your test cases and ensuring user needs are met but what about the traceability between the code and the test. In other words, what happens if there is code which is not exercised by your test and results into unexpected behaviour at customer’s end.
5 VectorCAST/QA and Functional Test Test Coverage
Test measurement allows better product quality: So, how do you measure test effectiveness? Counting number of lines covered by tests or taking a functional approach and see if all the features are covered.
Test Coverage: Test Coverage aims to measure of the effectiveness of testing in a qualitative manner. It determines whether the test cases are covering entire functional requirements. You can think of it as a kind of black box testing, where test cases are not written based on code but based on user requirements or expected functionality.
6 VectorCAST/QA and Functional Test Who should use VectorCAST/QA
▪ Recurring quality issues
▪ Measure effectiveness of functional test
▪ Wanting to do more with their QA resources in less time
▪ Starting a new project and want to reduce risk as it grows and changes
▪ Taking over a legacy product and need to minimize change risk
▪ Trying to improve the entire application life-cycle management process
▪ Wanting to increase communication between developers and QA for faster cycle times
▪ Outsourcing development and wanting to verify quality of testing
7 VectorCAST/QA and Functional Test Why VectorCAST/QA? (1\2)
▪ Built specifically for teams and organizations
▪ Easy to set up and configure- whether end environment is embedded target or x86 host
▪ Adopted to existing workflow with minimal changes
▪ Can be used by both development and QA teams making consistent reporting simple
▪ Works with any type of testing from manual testing to automated
8 VectorCAST/QA and Functional Test Why VectorCAST/QA? (2\2)
▪ Support wide variety of compiler ▪ Proven tool in safety critical domain like DO-178B/C (Avionics) ISO-26262 (Automotive) IEC-61508 (Industrial) EN-50128 (Railway) IEC-62304 (Medical) ▪ Provides all coverage metrics such as statement, Branch, MC/DC, Function, Function Call, Basis Path ▪ Web based dashboard for key metric analysis which, combined with code coverage, allow for quick targeting of high risk, problem areas for refactoring, increased testing, or both ▪ HTML/Text reports are sharable
9 VectorCAST/QA and Functional Test VectorCAST/QA Key Features
VectorCAST/QA enables teams to implement consistent and repeatable processes for managing test activities and reporting key quality metrics.
▪ Code Coverage Analysis ▪ Continuous Testing ▪ Test Collaboration ▪ System Test Automation ▪ Change-Based Testing ▪ Change Impact Analysis ▪ Parallel Testing ▪ Web-based Quality Dashboard ▪ Quality Trend Analysis
10 VectorCAST/QA and Functional Test VectorCAST/QA Environment
11 VectorCAST/QA and Functional Test VectorCAST/Analytics
12 Change Based Testing at Black Box
13 Change Based Testing at Black Box Change Based Testing
Running the complete set of tests for software applications can take days. Obviously running a full set of tests prior to a Production Release is prudent, but how about for intermediate builds as changes are made.
14 Change Based Testing at Black Box Automatically Identify the Minimum Tests for Each Code Change
For most software changes, you would ideally like to choose a sub-set of tests to run which will still adequately test the changes being made.
Choosing which tests to run is not as simple as finding tests that invoke a function directly. How about all of the tests that call functions that call that function, or that are called by the changed function, they are likely to be impacted too.
In fact most regressions occur because of these transitive relationships.
15 Change Based Testing at Black Box Change Impact Report
16 Change Based Testing at Black Box Build Execute-Incremental
17 Gray Box Testing using VectorCAST/Probe
18 Gray Box Testing using VectorCAST/Probe What is Gray Box Testing?
▪ Gray-box testing is a combination of white-box testing and black-box testing. The aim of this testing is to search for the defects if any due to improper structure or improper usage of applications
▪ Gray box testing, also know as gray box analysis, is a strategy for software debugging in which the tester has limited knowledge of the internal details of the program. A gray box is a device, program or system whose workings are partially understood.
▪ Gray-box testers require both high-level and detailed documents describing the application, which they collect in order to define test cases.
19 Gray Box Testing using VectorCAST/Probe VectorCAST/Probe
▪ One of the challenges with resolving defects in a fully integrated system is how to capture the data needed to understand the root cause of the problem. Using a debugger often changes the timing of the system in a way that masks the bug or prevents the system from running properly.
▪ VectorCAST/Probe addresses these challenges using our existing code instrumentation and target support technology to make the process of adding trace code as simple as possible.
20 Gray Box Testing using VectorCAST/Probe Features
▪ Dynamically instruments device software to isolate defects ▪ Inserts probes during Unit, Integration, or System Testing ▪ Captures internal data values ▪ Records detailed control flow ▪ Injects Faulty Values to test error handling ▪ Debugs hard to trigger race conditions ▪ Supports all embedded targets
21 Gray Box Testing using VectorCAST/Probe VectorCAST/Probe Example
22 Demo
23 Q&A
24 Author: Yadav, Sunil Vector India
25