Pareon Verify

Test suite for dynamic Find the hard-to-fi nd errors software verifi cation other tools don’t fi nd

“We are going to defi nitely incorporate the Vector Fabrics toolset as an important part of our quality oﬀ ering“

Test suite for dynamic software verification finds difficult-to-detect software errors and makes lengthy debugging sessions a thing of the past.

The amount of software in embedded Cost of an error systems, cell phones, or transportation vehicles such as cars and airplanes is A study by NASA [1] shows the cost of fixing a software error found in the field is 100x more expensive than exploding in size and importance. This the same error found already during development. development has led to a exponential growth in the number of potential software errors or “bugs”.

It is more important than ever to effectively and thoroughly test software before it is shipped. With the common use of C++ nowadays, it is also common to have more dynamic behavior and multithreading. This leads to a new category of errors that is hard to detect using the traditional testing tools and methods. Development Testing In the field

Errors are expensive The mounting costs of a software defect. Having to repair software errors post-release in pro- duction or in the field is a costly process. Expensive, even without taking into account any damage to the brand and the public image of the manufacturer.

Thus it is crucial that as many errors in the code as possible are detected and corrected before the product enters the market.

2 Copyright © 2015 Vector Fabrics B.V Car Sofware % Single Average high-end car 8,5 8,5 processor 14 14 F-35 Fighter Jet 30,1 30,1 % Multi 24 24 2013 19,4 19,4 processor Boeing 787 % Multi core 100 100Total fl ight software *million lines 21,4 20,621,4 20,6 % Multi core & of code Multi processor % Don’t know

Growing size and complexity of code increases the number of errors in code.

100 million lines of complex code

It is not uncommon for a car or an airplane to have For instance, a Heisenbug could relate to a timing over 100 million lines of code running. As the amount problem: in a debugger, timings change and the and the of complexity of code increases, so does the bug does not appear. This makes the subsequent number of errors in the code. detection and resolution of these errors an ex- tremely complex and thus costly process. Another component to drive up the complexity and the number of errors in code is multicore usage. More and more processors are already multicore, which For the reasons mentioned above, validating software brings new challenges to software correctness and without dynamic analysis is no longer an eff ective verifi cation. strategy. Software projects are growing in size, the software itself is becoming more dynamic and the Increasing dynamic behavior hardware used to operate the software is becoming Modern software testers are challenged to cope with increasingly sophisticated. Without employing dynam- the ever increasing complexity and the ever increasing ic analysis, the complex software errors will remain dynamic behavior of the software. Dynamic behavior undetected in the product to cause problems later. - for example dynamic memory use and multithreading - is hard to test for. Bugs in dynamic behavior tend Automated dynamic analysis by Pareon Verify to be hard to reproduce. Examples of these errors This trend led Vector Fabrics to develop Pareon Verify include buff er overfl ows, memory leaks, and invalid - a tool for dynamic software verifi cation. Pareon memory accesses. Verify allows for automated dynamic analysis of coding errors in large volumes of source code. This process is Heisenbugs able to detect software errors that cannot be identi- A whole new category of software errors is nowadays fi ed using other tools or methods. labelled as a “Heisenbug”: named after the uncer- tainty principle elaborated by the German theoretical physicist Werner Heisenberg. Heisenbugs are hard to diagnose because they tend to disappear when under scrutiny: debuggers or so-called “printf-debugging” tend to adjust the run-time circumstances enough to make the bug disappear.

Increasing size and complexity of applications cause more errors to occur in software.

Dynamic analysis - or runtime analysis, as it’s some- Dynamic analysis looks at the entire application and is times called - can be seen as an MRI scanner for not limited to a local unit scope in its error detection. software. By looking inside the application while it Additionally, as it looks inside the application to see executes, it is possible to detect errors in the errors in individual operations in a white-box fashion, internal behavior. it reports errors in an application even if that error does not cause the application to fail its black-box test Dynamic analysis observes at the application’s internal on a particular test case. structures and behavior and observes where errors originate. In doing so, it pinpoints the primary cause of Therefore, dynamic code analysis is a part to a the error. This way, dynamic analysis identifies errors comprehensive software testing process. It com- that remain undetected during other tests. plements static code analysis and unit testing. Dy- namic analysis offers the key advantage: it reveals Limits of existing methods the most expensive errors that would otherwise Errors in dynamic behavior are notoriously difficult remain undetected. to detect with traditional testing tools and methods. Errors in the dynamic behavior don’t trigger when the application is run in its normal testing environment, Dynamic analysis Heisenbugs but cause problems when environment, timing or

circumstances change. Unit & system tests Functional errors

Static analysis is limited in its capability to predict the Static analysis Coding errors behavior of dynamic structures and cannot foresee all Coding style checks Coding style violations possible combinations of inputs, code paths, dynamic Cost of an error memory and thread timing in the full application scope. Compilation Syntax errors

Unit testing is limited in scope to a smaller part of the application, and cannot detect errors that are caused The role of the dynamic analysis in the software by the global behavior or the environment of the verification process. application.

Dynamic analysis to help Dynamic analysis is not hindered by these limitations: as it observes the application’s actual operation while being executed, it can see all the combinations of dynamic and global behavior that are triggered during the execution, regardless of complexity.

Dynamic analysis works by instrumenting the target Pareon Verify detects a range of software errors, in- application to see the behavior inside the application. cluding Heisenbugs.. Pareon Verify can be completely The instrumentation in the application binary creates integrated into the development process so that new a stream of events, or a trace, that is captured by the software changes can be continuously screened for analyzer and studied for errors in the behavior. The new errors. analyzer then reports the found errors in a clear and concise way with detailed references to source code for easy correction of the error.

The workflow of Pareon Verify is as follows: 1. Compile Use Pareon’s instrumenting C and C++ compiler as a drop-in replacement for your (cross)compiler. Set-up your build in minutes. 2. Execute Execute the instrumented program on a host PC or target embedded board. Run with representa- tive input data or test suite. 3. Analyze Conveniently analyze the resulting execution trace on your host PC. Minimal memory and performance impact from the analysis on the target! 4. Resolve bugs Get detailed messages for each error. Instantly see how to fix the problem. Variable names, full call stacks. No false positives!

Invalid memory access Invalid pointer usage and buffer overflows e.g. through bad pointer arithmetic

Array out-of-bounds Access outside a buffer or using wrong indexes into a multidimensional array Using variables that have not been written yet, generally leading to data corruption Uninitialized memory reads or crashes

Use-after-free Use of dangling pointers, pointing to memory that no longer exists Multiple threads accessing the same data without proper synchronization, Race conditions generally leads to data corruption

Memory leaks Will cause system to slow down or run out of memory

Hard-to-detect errors Pareon Verify finds.

Write Computer hardware increasingly uses multicore 10 processors. Not just PCs, but also cell phones have up 10 to 8 cores inside nowadays. This causes problems on the software side as multicore software development is harder. So-called “data races” already occurred on single core CPUs when interacting with interrupts, but Read are now muchRead more widespread due to the multi- threaded programs actually running threads in parallel. Write 10 A data race arises when two threads access the same 10 variable without proper synchronization. As a result, threads could read stale or just wrong data - data corruption will happen.

Shared Shared Thread A memory Thread B Thread A memoryIn general,Thread multi-threaded B programs will have more Time Time dynamic behavior and thus will be more susceptible to Read and write from different threads to dynamic software errors. shared memory.

“We are glad we are now able to ? ? catch some really difficult bugs much earlier“ Write 10

10 Daniele Lacamera Lead developer / PicoTCP

Read Read

Write 10 10

Shared Shared Thread A memory Thread B Thread A memory Thread B Time Time

Data race, read of uninitialized data because the read happens before the write is done.

Many professional software development compa- Less errors in the field nies deploy a development process called “contin- By capturing the errors before the hit the field, uous integration”. Upon on check-in of a new piece companies can reduce maintenance costs of of code by a developer or at least every night, a software. The customer satisfaction and the quality central server will take all source code, build it and image of the brand will improve when errors are run a set of tests on it. If any of the tests break, the no longer triggered in the field. An error found just submitted code is rejected and the develop- during testing is 5x more expensive than one er has to fix it. This allowed to find a lot of errors found during development, and it can be up to earlier in the process: whilst the developer is still 100x more expensive when found in the field - not working on it, or just finished. The code is still fresh to mention impacts to customer satisfaction and on his mind, fixes will be faster. brand image.

Using Pareon Verify, it is now also possible to Improved time to market deploy dynamic error finding during continuous in- Often, during the integration and system testing tegration. Instead of just receiving a “pass” or “fail” phase’ the number of bugs found by the test team from the automated tests, Pareon Verify will also only grows. Developers cannot tell when they are report if the code flow leading to the verdict was done fixing all the issues as one issue might be correct. Pareon Verify reports if the test result is hiding the next. As a result, project and release only correct by accident. Pareon Verify also reports schedules tend to slip, delaying the release or the if there were unwanted side effects. The benefit of end of the project. With Pareon Verify, companies this early detection is that developer can correct will be able to cut significantly back on the time the error immediately - even before it triggers a spent on debugging and correcting errors, which test case to fail and a crash to happen, saving a lot in turn reduces the risk of project delays. In prac- of time and costs in the future. tice, this can mean weeks’ worth of time-saving and a more predictable test and integration phase. To reduce maintenance costs, it is essential that errors be removed from a software design Summary as early as possible. A well-known Japanese car By using Pareon Verify, software can be brought to manufacturer notably had to issue recalls for market more quickly, cheaper and with fewer error problems that were linked to software errors in reports from customers. Software developed with a number of its models. It is estimated that the Pareon Verify is of a higher quality and is complet- recalls cost the company two billion dollars, not ed sooner. This allows companies to improve the to mention the damage to its corporate image. quality image of their software.

http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100036670.pdf

For more information about Pareon Verify, please visit: http://pareonverify.com

Software is an essential part of virtually every modern product or service. Vector Fabrics specializes in development and testing tools for complex software that is used in embedded systems, including automotive, IOT devices, network equipment, consumer electronics, aerospace, medical and industrial systems, and more.

The Pareon family of dynamic analysis tools provides customers with deep insight into how their software will actually behave when running, ensuring a higher level of reliability, safety and performance. Using these tools, businesses are able to prevent costly software bugs and improve the time-to-market for new products.

Phone +31 40 8200 960 Email [email protected] Hogeweg 39A 5301 LJ Zaltbommel The Netherlands