Pareon Verify

Pareon Verify Test suite for dynamic Find the hard-to-fi nd errors software verifi cation other tools don’t fi nd “We are going to defi nitely incorporate the Vector Fabrics toolset as an important part of our quality oﬀ ering“ Jan Asselman Quality team leader / Altran Intelligent Systems Copyright © 2015 Vector Fabrics B.V Avoid software errors and high repair costs with Pareon Verify Test suite for dynamic software verification finds difficult-to-detect software errors and makes lengthy debugging sessions a thing of the past. The amount of software in embedded Cost of an error systems, cell phones, or transportation vehicles such as cars and airplanes is A study by NASA [1] shows the cost of fixing a software error found in the field is 100x more expensive than exploding in size and importance. This the same error found already during development. development has led to a exponential growth in the number of potential software errors or “bugs”. It is more important than ever to effectively and thoroughly test software before it is shipped. With the common use of C++ nowadays, it is also common to have more dynamic behavior and multithreading. This leads to a new category of errors that is hard to detect using the traditional testing tools and methods. Development Testing In the field Errors are expensive The mounting costs of a software defect. Having to repair software errors post-release in pro- duction or in the field is a costly process. Expensive, even without taking into account any damage to the brand and the public image of the manufacturer. Thus it is crucial that as many errors in the code as possible are detected and corrected before the product enters the market. 2 Copyright © 2015 Vector Fabrics B.V Car Sofware % Single Average high-end car 8,5 8,5 processor 14 14 F-35 Fighter Jet 30,1 30,1 % Multi 24 24 2013 19,4 19,4 processor Boeing 787 % Multi core 100 100Total fl ight software *million lines 21,4 20,621,4 20,6 % Multi core & of code Multi processor % Don’t know Growing size and complexity of code increases the number of errors in code. 100 million lines of complex code It is not uncommon for a car or an airplane to have For instance, a Heisenbug could relate to a timing over 100 million lines of code running. As the amount problem: in a debugger, timings change and the and the of complexity of code increases, so does the bug does not appear. This makes the subsequent number of errors in the code. detection and resolution of these errors an ex- tremely complex and thus costly process. Another component to drive up the complexity and the number of errors in code is multicore usage. More and more processors are already multicore, which For the reasons mentioned above, validating software brings new challenges to software correctness and without dynamic analysis is no longer an eff ective verifi cation. strategy. Software projects are growing in size, the software itself is becoming more dynamic and the Increasing dynamic behavior hardware used to operate the software is becoming Modern software testers are challenged to cope with increasingly sophisticated. Without employing dynam- the ever increasing complexity and the ever increasing ic analysis, the complex software errors will remain dynamic behavior of the software. Dynamic behavior undetected in the product to cause problems later. - for example dynamic memory use and multithreading - is hard to test for. Bugs in dynamic behavior tend Automated dynamic analysis by Pareon Verify to be hard to reproduce. Examples of these errors This trend led Vector Fabrics to develop Pareon Verify include buff er overfl ows, memory leaks, and invalid - a tool for dynamic software verifi cation. Pareon memory accesses. Verify allows for automated dynamic analysis of coding errors in large volumes of source code. This process is Heisenbugs able to detect software errors that cannot be identi- A whole new category of software errors is nowadays fi ed using other tools or methods. labelled as a “Heisenbug”: named after the uncer- tainty principle elaborated by the German theoretical physicist Werner Heisenberg. Heisenbugs are hard to diagnose because they tend to disappear when under scrutiny: debuggers or so-called “printf-debugging” tend to adjust the run-time circumstances enough to make the bug disappear. Increasing size and complexity of applications cause more errors to occur in software. Copyright © 2015 Vector Fabrics B.V 3 The advantages of dynamic code analysis Dynamic analysis - or runtime analysis, as it’s some- Dynamic analysis looks at the entire application and is times called - can be seen as an MRI scanner for not limited to a local unit scope in its error detection. software. By looking inside the application while it Additionally, as it looks inside the application to see executes, it is possible to detect errors in the errors in individual operations in a white-box fashion, internal behavior. it reports errors in an application even if that error does not cause the application to fail its black-box test Dynamic analysis observes at the application’s internal on a particular test case. structures and behavior and observes where errors originate. In doing so, it pinpoints the primary cause of Therefore, dynamic code analysis is a part to a the error. This way, dynamic analysis identifies errors comprehensive software testing process. It com- that remain undetected during other tests. plements static code analysis and unit testing. Dy- namic analysis offers the key advantage: it reveals Limits of existing methods the most expensive errors that would otherwise Errors in dynamic behavior are notoriously difficult remain undetected. to detect with traditional testing tools and methods. Errors in the dynamic behavior don’t trigger when the application is run in its normal testing environment, Dynamic analysis Heisenbugs but cause problems when environment, timing or circumstances change. Unit & system tests Functional errors Static analysis is limited in its capability to predict the Static analysis Coding errors behavior of dynamic structures and cannot foresee all Coding style checks Coding style violations possible combinations of inputs, code paths, dynamic Cost of an error memory and thread timing in the full application scope. Compilation Syntax errors Unit testing is limited in scope to a smaller part of the application, and cannot detect errors that are caused The role of the dynamic analysis in the software by the global behavior or the environment of the verification process. application. Dynamic analysis to help Dynamic analysis is not hindered by these limitations: as it observes the application’s actual operation while being executed, it can see all the combinations of dynamic and global behavior that are triggered during the execution, regardless of complexity. 4 Copyright © 2015 Vector Fabrics B.V How it works? Dynamic analysis works by instrumenting the target Pareon Verify detects a range of software errors, in- application to see the behavior inside the application. cluding Heisenbugs.. Pareon Verify can be completely The instrumentation in the application binary creates integrated into the development process so that new a stream of events, or a trace, that is captured by the software changes can be continuously screened for analyzer and studied for errors in the behavior. The new errors. analyzer then reports the found errors in a clear and concise way with detailed references to source code for easy correction of the error. The workflow of Pareon Verify is as follows: 1. Compile Use Pareon’s instrumenting C and C++ compiler as a drop-in replacement for your (cross)compiler. Set-up your build in minutes. 2. Execute Execute the instrumented program on a host PC or target embedded board. Run with representa- tive input data or test suite. 3. Analyze Conveniently analyze the resulting execution trace on your host PC. Minimal memory and perfor- mance impact from the analysis on the target! 4. Resolve bugs Get detailed messages for each error. Instantly see how to fix the problem. Variable names, full call stacks. No false positives! Invalid memory access Invalid pointer usage and buffer overflows e.g. through bad pointer arithmetic Array out-of-bounds Access outside a buffer or using wrong indexes into a multidimensional array Using variables that have not been written yet, generally leading to data corruption Uninitialized memory reads or crashes Use-after-free Use of dangling pointers, pointing to memory that no longer exists Multiple threads accessing the same data without proper synchronization, Race conditions generally leads to data corruption Memory leaks Will cause system to slow down or run out of memory Hard-to-detect errors Pareon Verify finds. Copyright © 2015 Vector Fabrics B.V 5 Example Multicore errors ? ? Write Computer hardware increasingly uses multicore 10 processors. Not just PCs, but also cell phones have up 10 to 8 cores inside nowadays. This causes problems on the software side as multicore software development is harder. So-called “data races” already occurred on single core CPUs when interacting with interrupts, but Read are now muchRead more widespread due to the multi- threaded programs actually running threads in parallel. Write 10 A data race arises when two threads access the same 10 variable without proper synchronization. As a result, threads could read stale or just wrong data - data corruption will happen. Shared Shared Thread A memory Thread B Thread A memoryIn general,Thread multi-threaded B programs will have more Time Time dynamic behavior and thus will be more susceptible to Read and write from different threads to dynamic software errors. shared memory. “We are glad we are now able to ? ? catch some really difficult bugs much earlier“ Write 10 10 Daniele Lacamera Lead developer / PicoTCP Read Read Write 10 10 Shared Shared Thread A memory Thread B Thread A memory Thread B Time Time Data race, read of uninitialized data because the read happens before the write is done.

Load more