Static Analysis for FDA Software Validation Compliance

Home , Parasoft, Regression testing, SOAtest

Technical Whitepaper

EXECUTIVE SUMMARY

Static analysis, when deployed as part of a continuous quality process, But even with the ideal mixture of testing techniques, quality software has proven to be a very efficient way for developers to expose and cannot be delivered by testing alone. Quality software is delivered prevent many critical defects as the code is being written. The FDA consistently via a solid, repeatable process. Such a process must has publicly recommended that software developers use static embrace everything from software testing and analysis, to quality analysis for ensuring medical device software safety and reliability, planning, to requirements traceability, to change management. This and has touted static analysis as a critical component of an effective process needs to be visible, measurable, and improvable. software development process. To assist organizations that are exploring static analysis for FDA But static analysis is only one piece of the software development compliance, this paper describes Parasoft’s static analysis capabilities puzzle. The FDA also recommends that medical device software in the context of FDA compliance. Because we recognize that static development teams take a software development lifecycle (SDLC) analysis is not a silver bullet for FDA compliance, this paper also approach that integrates risk management strategies with principles describes Parasoft’s broader software solution for medical device for software validation. An integrated SDLC merges validation and software development. Parasoft’s reporting and analytics system verification activities, including unit testing, peer code reviews, extends Parasoft’s static analysis capabilities by combining a pre- static analysis, manual testing, and regression testing. The result of configured system with processes and best practices. The result such an approach is an emphasis on planning, verification, testing, is that organizations are able to produce medical device software traceability, and configuration management. consistently and efficiently with freedom from unacceptable risks.

THE FDA COMPLIANCE PARADOX

Before jumping into static analysis, it is worth discussing some of the challenges associated with adopting the FDA’s recommended software development approach for gaining approval. Developing software for medical devices that complies with the FDA’s Quality System regulation is a challenging endeavor that’s as much a business issue as it is an engineering feat. One reason for this is the vagueness with which the FDA outlines compliance.

The agency does not prescribe specific practices, tools, coding methods, or any other technical activity, but instead prescribes the seemingly innocuous concept of the Least Burdensome Approach. In this approach, organizations determine, and strictly adhere to, their self-defined validation and verification processes. Development activities and outcomes must be clearly defined, documented, verified, and validated against the organization’s process.

The goal of this approach is to give medical device makers enough rope to determine how to best ensure public safety. But in practice, the effect has been that organizations have enough rope to hang themselves. This is because the requirements, expressed in FDA 21 CFR, represent extensive planning and testing, which require validation. The following examples are just a few of the challenges software engineers must overcome:

• The software validation process cannot be completed without an established software requirements specification, which specifies the intended use. Results must not only verify that the specifications are met, but they must be reproduced consistently (validation). Testing methods like regression testing can be implemented to meet the requirement.

• Validation must be established and re-established for even small changes. This means that validation activities, including static analysis, unit testing, code review, etc., must be THE VALUE OF PARASOFT’S repeated if the code has changed. Furthermore, as software continues to become more STATIC ANALYSIS TOOLSET and more complex, tests that validate the changes should be conducted in scale with the application to ensure that no other part of the system is affected. Parasoft provides early and effortless detection of errors that might otherwise take • Changes to the requirements deemed significantly different enough from the originally weeks to find, through: registered design may require the product to be re-registered per FDA Section 501(K). • STATIC CODE ANALYSIS • There are no “FDA certified” tools or methods. No person, organization, or tool can claim Static analysis prevents defects using any form of some supposed FDA certification. However, any software used to automate thousands of rules tuned to find and any part of the device process or any part of the quality system must also be validated. prevent coding issues. Over 15 years You must be able to run any tools used to assist in the verification and validation efforts of R&D have gone into fine-tuning on a control code base and confirm that the results are consistent, which may affect your Parasoft’s rule set. time-to-market. • DATA FLOW ANALYSIS This technology simulates test case The FDA has established grounds for approval in a way that effectively puts the responsibility execution and detects errors across of ensuring quality and public safety back to the device makers. But device makers are multiple units, components, and files, often dealing with a bigger challenge: bridging the gap between business goals and the exposing runtime errors without hav- development process. ing to run the code.

• METRICS ANALYSIS LACK OF SOFTWARE DEVELOPMENT POLICY Calculate a customizable set of industry Software engineers often either don’t know what’s expected or do not understand the business standard metrics and identify pieces objective behind the guidelines driving their products. They are expected to write code that of code that exceed industry standard meets the requirements, without necessarily understanding why requirements have been or custom metrics thresholds. This ex- established in the first place. poses brittle or overly-complex code that could be dangerous to reuse, At Parasoft, we believe that the best way to overcome these challenges, while satisfying the extend, or maintain. FDA’s requirements for medical device software development, is to drive the development process in a platform based on policy-driven development. Policy-driven development involves: To improve overall team productivity, a con- (1) clearly defining expectations and documenting them in understandable polices, (2) training tinuous process ensures that scanning and engineers on the business objectives driving those policies, and (3) monitoring policy adherence remediation tasks are not only deployed in an automated, unobtrusive way. across the SDLC, but also integrated into the team’s workflow. Managers set ex- Integrating these principles into the development process gives businesses the ability to pectations by defining a code compliance accurately and objectively measure productivity and application quality. In addition to reducing policy (e.g., by enabling or customizing risk, the result is lower cost over the total software development lifecycle from build to support. Parasoft’s pre-configured FDA compliance Adopting a policy-driven development process is crucial for achieving the following goals: template). A daily process automatically monitors policy compliance at all layers of • Ensuring that engineers don’t make trade-offs that potentially compromise reliability and the application stack, identifies non-com- performance pliant code, and collects process metrics. • Ensuring that engineers build security into the application, safeguarding it from potential Management gains real-time visibility into attacks overall code-compliance status, allowing • Preventing defects that could result in costly recalls, litigation, or a damaged market teams to document improvements and de- position termine additional actions to ensure appli- • Accurately and consistently applying quality processes cation safety and reliability. • Gaining the traceability and auditability required to ensure continued policy compliance Developers simply respond to tasks report- Software engineers are continually making business decisions. Every line of code, test ed from the automated scan. They can also conducted (or left undone), and guideline or standard followed (or ignored) have profound perform interactive static analysis directly effects on the business. With public safety, potential litigation, market position, and other from their IDE: detected compliance issues consequences on the line, it behooves software development teams and people in traditional are prioritized and automatically assigned business management positions to come together on policy, and implement the strategy into to the developer who introduced it, with di- their software development lifecycle. rect links to the problematic code and an explanation of how to fix it.

THE IMPORTANCE OF A CONTINUOUS STATIC ANALYSIS PROCESS

Having static analysis tightly integrated into the SDLC as described Policy management lies at the core of such an inline process. in the column above — rather than only as an audit at the end of the Parasoft allows you to easily configure policies for specific projects process — provides two distinct benefits. without compromising the integrity of the corporate objectives, to easily deploy and update both project-specific and organization- First, developers adopt better coding habits that help them write wide policies, and automate their application for rapid scanning and better code, faster. The process of fixing a static analysis violation reporting. is more meaningful and more educational when it is done in the context of what the developer was trying to achieve. Therefore, in a A carefully-defined and implemented set of policies establishes continuous process, it is more likely that the developer will learn from a knowledge base that guides developers to start writing safe and his mistakes and eventually start writing compliant code as a matter reliable code as a matter of habit. of habit. With a policy established, putting it into practice involves workflow Second, developers remediate problems faster and easier. If the code management: defining, automating, and monitoring a workflow that is still fresh in the developer’s mind when the problem is reported, improves development productivity and forms the foundation for a the developer doesn’t need to waste time trying to remember what sustainable process. Moreover, tasks to support quality policies must the code was supposed to do, why he wrote it the way he did, what be optimized so they can feasibly become an integral part of the impacts he needs to consider when modifying the code to meet team’s existing workflow, ensuring that the static analysis process is compliance expectations, and so on. As you can imagine, resolving both sustainable and scalable. A lack of automation, repeatability, or the same issue weeks or months later would require significantly consistency will degrade any quality initiative that the organization more work to achieve the same outcome. intends to deploy.

KEEPING RESULTS RELEVANT AND MEANINGFUL One way that Parasoft helps organizations optimize the workflow is THROUGH CONTEXT by using static analysis in tandem with other analysis capabilities, such as metrics analysis and peer code review. This allows you to Parasoft promotes a policy-based approach to static analysis: use better optimize developer time by focusing their efforts on more static analysis to monitor a non-negotiable set of expectations severe or complex scenarios first. For example, if you are alerted that around code reliability, security, performance, and maintainability. a certain piece of code has a high level of Cyclomatic Complexity With this approach, a violation of a particular guideline is not just as well as high severity security issues, you would ultimately want to another suggestion for people building software in an ivory tower, point developers to that region first. This is a prime example of how but a notification that the code failed to meet the organization’s code analysis can help you zero in on items that should be discussed expectations. Otherwise, rule violations are perceived as suggestions during the peer code review. for general code improvements, as opposed to critical coding issues that need to be addressed immediately. Other ways that Parasoft helps organizations optimize the workflow include: Effective policy management allows an organization to bridge the gap between management expectations and developer performance. If • Routing each reported issue directly to the responsible a static analysis rule enforces something that is part of the policy, developer, as well as customizing issue prioritization to suit fixing a violation of that rule is non-negotiable. If a developer fails to your policy priorities, ensures that your most critical issues are satisfy the defined policy, he is not executing his job as expected by addressed in a timely manner. management. • Centralized configuration management ensures that rule sets SUSTAINABLE STATIC ANALYSIS THROUGH PROCESS are applied consistently and can be updated effortlessly as AND WORKFLOW priorities and processes evolve.

To ensure that an inline, policy-driven static analysis process is easy • Using automated refactoring whenever feasible helps the to introduce and sustainable to adopt, Parasoft focuses on policy team correct rule violations as fast as possible. management, workflow management, and workflow optimization. The strategies we implement in these areas have been optimized through 20+ years of experience working with Fortune 500 companies.

© 2017 Parasoft Corporation 4 Beyond Static Analysis: INTEGRATED TEST BUILDING A SOLID PROCESS AND ANALYSIS As described above, the FDA recommends not only that testing involve a mixture of test and analysis methods applied throughout the SDLC, but also that a broad set of software lifecycle Static analysis is just one component management and risk-management activities be integrated across the process to ensure the of Parasoft’s integrated testing and delivery of safe and reliable software. Parasoft addresses both of these expectations as detailed analysis software toolset. Parasoft’s in the column to the right. reporting and analytics dashboard brings all these elements together, A COMPREHENSIVE QUALITY-DRIVEN PROCESS providing integrated support for:

Extending beyond the test and analysis component of compliance, Parasoft supports the • Static analysis FDA’s vision of an integrated SDLC for C, C++, Java, and .NET with a software development management platform designed for medical device software development, pre-configured with • Peer code reviews processes and best practices described in the FDA guidelines and medical device industry • Unit and component testing standards. Parasoft’s solution features: • Integration and functional • Built-in configurable templates for FDA, ANSI 62304, IEC, SIL and more testing • Process, project, and task management • Comprehensive requirements traceability • Testing on the host, simulator, • Integrated defect prevention, validation, and verification and target device • A continuous policy-driven compliance process with real-time visibility • Runtime error detection during • Correlation of all key artifacts, from tests, to requirements, to code, to builds, to project test execution (for all tests from tasks the unit level to the system This powerful platform ensures that software can be produced consistently and efficiently, while level) simultaneously preventing unacceptable risks. • Code coverage racking (for all tests from the unit level to the The Details: system level) Memory error detection PARASOFT’S STATIC ANALYSIS CAPABILITIES • • Message/protocol testing Parasoft’s static analysis ensures that code meets expectations for reliability, security, performance, and maintainability. It eliminates the root cause of crashes, deadlocks, erratic • Regression testing behavior, performance degradation, security/privacy issues, and more. Our static analysis Penetration testing capabilities can be divided into three core categories: • • Application behavior emulation • Pattern-based static analysis • Flow-based static analysis • Load and performance testing • Metrics-based static analysis • Business process testing

Since 1994, Parasoft has been leading the industry in static analysis, especially in how it can • Manual user acceptance testing be integrated successfully into the SDLC. With years of experience working with over 10,000 • Requirements-based testing customers, we know what’s required for sustainable success with static analysis: • Change-based testing • Seamless integration into any development infrastructure • Automated workflow driven by an effective policy • Making analysis and remediation a natural part of the development process

In addition to static analysis, data flow analysis, and code metrics, Parasoft’s integrated multi- language solutions also facilitate code review, unit testing, regression testing, runtime error detection, manual testing, and SOA/web/cloud functional and load testing.

PATTERN-BASED STATIC CODE ANALYSIS DATA FLOW STATIC ANALYSIS

Parasoft’s pattern-based static code analysis monitors whether code Parasoft’s data flow static analysis provides automated detection of follows industry-standard or customized rules for ensuring that code runtime errors without requiring the software to actually be executed. meets uniform expectations around reliability, security, performance, Flow-based static analysis is the technique of logically executing and maintainability. Parasoft’s static code analysis solutions feature: the program to track the propagation of data values, their effects on control flow, and the legality and cleanliness of data at multiple • A centralized, integrated system for automated monitoring of points in the code. Flow analysis attempts to simulate the runtime code compliance across heterogeneous environments (Java, condition of data objects across functions, modules, and files. The C/C++, C#, VB.NET, JavaScript, etc.); FDA and other core goal is to uncover code problems such as memory corruption, leaks, industry standards; and organization-specific policies (security, invalid pointer dereferences, unsafe or tainted data propagation, and branding, etc.) security vulnerabilities. Whereas pattern-based analysis focuses on • Rule sets that are the most comprehensive in the industry and looking for local syntactical anomalies, flow analysis explores the are constantly being extended potential execution paths of the larger code context. This defect- • Fast, easy ways to define and check custom static analysis prevention technique is powerful because it does not depend on user rules that prevent application-specific errors from re-occurring, input to identify defects that, in reality, are data-dependent. and monitor adherence to organization-specific policies • Automated task assignment and customizable issue Combining path flow analysis with interprocedural and intraprocedural prioritization to ensure that the most critical issues are analysis, Parasoft statically simulates application execution paths— addressed in a timely manner which may cross multiple units, components, and files—to identify • Extensive support for policy management, workflow paths that could trigger runtime errors such as: management, and workflow optimization to facilitate code improvement without impeding project progress or team • C/C++: uninitialized or invalid memory, null pointer productivity dereferencing, array and buffer overflows, division by zero, memory and resource leaks, dead code, security Problems detected and prevented include: vulnerabilities, and concurrency defects • Exceptions • .NET: NullReferenceExceptions, ArgumentNullExceptions, • Buffer overflows resource leaks, division by zero, dereferencing before • Deadlocks, race conditions, and other concurrency defects checking for null, SQL injections, XSS, other security • Memory & resource leaks vulnerabilities, and concurrency defects • Erratic behavior • Performance degradation • Java: NullPointerExceptions, memory & resource leaks, • Data corruption accessing arrays out of bounds, unvalidated input in array • Security/privacy compromises indexes, incorrect Iterator usage, division by zero, SQL injections, XSS, other security vulnerabilities, and concurrency defects

REAL-WORLD EXAMPLE: STAYING AHEAD OF COMPETITION WITHOUT IMPACTING COMPLIANCE

The competitive pressures of the medical device industry required one organization to produce new products at an incredibly fast pace. Adding to the challenges of meeting FDA regulations and staying on top of the competition, their products included complex embedded devices and web-application monitoring tools, and their development teams were distributed across the world.

To meet all of these challenges head-on, the organization turned to Parasoft. Initially, they used Parasoft solutions to streamline and standardize their Java testing efforts. Next, they became an early adopter of Parasoft’s reporting and analytics dashboard for software development management. They have since standardized all development teams on Parasoft C++test, Parasoft Jtest and Parasoft SOAtest, and with Parasoft’s software development management platform, they now enforce quality standards, increase productivity, and track project budgets with superior efficiency.

To simplify defect analysis, a complete analyzed path trace for each potential defect is reported in the IDE, and cross-links to code help you quickly jump to any point in the highlighted analysis path. This ability to expose errors without executing code is especially valuable for teams with REAL-WORLD EXAMPLE: legacy code bases lacking robust test suites or embedded code, where runtime analysis and detection of such errors is not always effective or possible. CUTTING VERIFICATION AND VALIDATION COSTS CODE METRICS IN HALF Parasoft’s code metrics analysis calculates a customizable set of industry-standard metrics, as well as identifies specific pieces of code that exceed industry-standard or customized metrics thresholds. This helps organizations identify brittle or overly-complex code that is difficult and dangerous A manufacturer of electrosurgical to maintain, extend, and reuse. products decided to introduce a Parasoft’s metrics calculation capability provides organizations with visibility into code complexity new user-interface based product and the potential impacts of an anticipated code change. This enables them to make better in- to their line of surgi-center and formed decisions about how to modify, refactor, and test it. Provided metrics calculations include hospital-based electrosurgical Cyclomatic Complexity, Inheritance Depth, and Nested Block Depth. generators and accessories. Parasoft’s limit-based analysis capability enables organizations to customize the boundaries and They had previously relied on an thresholds for available metrics so that team members are alerted when metrics fall outside of the outside vendor for their embedded prescribed range. Leveraging this automation, the team is freed to focus on analyzing and improv- software testing; however, for this ing the problematic code, i.e. the task that requires human intelligence to complete. project, which involved both C++ and C# code, they wanted to move Policy and Workflow Management / Optimization testing in-house to reduce costs. Parasoft helped them achieve The most critical component to making quality a continuous process is to establish organization- wide quality policies that can be objectively defined, realistically applied, and automatically this goal. They not only cut their monitored for compliance. This involves rules management, workflow management, and workflow verification and validation costs optimization. approximately in half, but also went to market six months sooner than RULES MANAGEMENT they could have with their previous Parasoft provides centralized management of organization-level and team-level policies for process. ensuring application security, reliability, performance, and maintainability—as well as complying with FDA and other regulations. Policies are easily deployed and customized for specific projects, without compromising the integrity of the corporate objectives. Policy management is centralized for simple access, and policy application is automated for rapid scanning and reporting.

Organization/Team Policy Manager: Parasoft’s policy manager provides a fast and easy way to ensure that policy enforcement is standardized both globally and locally. This can include a fixed set of organizational policies, as well as subsets of policies customized to suit the needs of specific projects and teams. Policy configurations for thousands of desktops can be implemented or updated once, then distributed to thousands of machines with a single click.

Rule Library: Each language-specific rule library provides configurable sets of hundreds of rules. Parasoft’s rule library is the most comprehensive in the industry, and is constantly being extended. To help developers understand and achieve the benefit that the rules deliver, detailed documentation for each rule is provided, including an explanation of the rule requirements and customizable enforcement parameters, the rationale behind the rule, a sample violation, and an explanation of how to bring code into compliance.

Rule Configuration/Parameterization: Since goals and technologies vary from project to project, Parasoft allows teams to customize the logic of library rules to suit the unique demands of specific projects and priorities. Rules are REAL-WORLD EXAMPLE: customized using GUI controls, and no coding or scripting is required. Moreover, rule names, EXTENDING BEYOND descriptions, and severities can be matched to your organization’s policies, establishing a fully- STATIC ANALYSIS customized policy-management and reporting interface.

Graphical Rule Editor: In the event that certain internal policies cannot be matched to library rules, Parasoft provides An organization that provides fast and easy ways to extend the rule set. Custom rules can be built graphically or generated services in medical imaging, automatically, by providing code that demonstrates a sample rule violation. This flexible framework information technologies, medical supports the definition of even the most complex policy requirements. diagnostics, and disease research WORKFLOW MANAGEMENT & drug discovery needed to automate and streamline their Parasoft defines, automates, and monitors a workflow that improves development productivity and forms the foundation for a sustainable quality process. As quality tasks are defined and automat- FDA compliance efforts in order ed, process control guidelines are established to measure process progress as well as software to reduce time-to-market without quality. compromising their commitment to safety and reliability. Quality Process Definition: Quality processes and tasks are defined in the system and the workflow associated with the They contacted Parasoft and process is automated. The organization’s expectations around quality tasks are visible and embarked upon a pilot project for monitored. This significantly reduces the resources required to establish and maintain the quality introducing Parasoft’s C and C++ process, resulting in increased team productivity. solutions across the entire MRI Quality Process Metrics: team, which has development distributed across the US and Asia. Process control guidelines are established to measure both process progress and software quality. By establishing quality gates at specific points in the SDLC and establishing quality thresholds, From there, the MRI team organizations and teams gain the ability to track progress and ultimately improve the software realized the benefit of the whole development process. toolset. They decided to move WORKFLOW OPTIMIZATION beyond static analysis and started also utilizing unit testing Tasks to support quality policies must be optimized so they can become an integral part of the capabilities, brought together with team’s existing workflow. The lack of automation, repeatability, or consistency will degrade any Parasoft’s software development quality initiative that the organization intends to deploy. Parasoft optimizes the workflow to ensure that the quality process is both sustainable and scalable. management dashboard, gaining valuable insights to establish and For example, Parasoft provides the following: drive an auditable quality process with complete visibility into their • Routing each reported issue directly to the responsible developer, with customizable issue compliance efforts. prioritization to ensure that your most critical issues are addressed in a timely manner.

• Centralized configuration-management to ensure that test configurations are applied consistently and can be updated effortlessly as priorities and processes evolve.

• Monitoring manual processes that require human intelligence, then converting them into automated processes, relieving the team from having to repeat many manual efforts.

Seamless Integration: Static analysis can be performed directly from the IDE (Eclipse, Visual Studio, Wind River, ARM, TI, REAL-WORLD EXAMPLE: and more). It can also run in batch mode, where new and modified code is scanned completely and BRINGING LEGACY automatically at regular intervals (e.g., nightly or as part of a continuous integration process). Each PRODUCTS INTO issue detected is prioritized according to configurable settings, automatically correlated to the developer who introduced it, and then distributed to his or her IDE with direct links to the COMPLIANCE problematic code and a description of how to fix it.

Issue Prioritization: A maker of blood-management Results are reported as a prioritized task list to help the team focus their attention and resources products that have been on the on the issues expected to have the greatest impact on quality. To ensure that the most critical market since the early 90s needed policy violations are identified immediately and promptly resolved, Parasoft allows organizations to customize issue severities to match their unique project needs and priorities. to update one of their legacy products. When the product was Context Suppression: originally released, the current Context-sensitive suppressions allow teams to prevent rules from firing under specific circum- FDA software validation guidelines stances where they do not apply. This reduces the level of noise, ensuring that every violation were not in place. In order to keep reported is a real problem that should be fixed. Suppressions can be defined in the code to ensure up with their competition, they they are always visible when code is reviewed and modified. needed to upgrade the product Quick Fix and update their technology. The Parasoft Quick Fix automatically refactors code to conform with certain coding policies. This en- company was surprised to learn ables the development team to rapidly remediate many policy violations. that these modifications would make their legacy product subject MOVING FORWARD to the current FDA guidelines. The real-world examples described highlight how Parasoft has assisted organizations developing Parasoft helped them jumpstart medical device software, to establish auditable quality processes that include complete visibility compliance by introducing static into compliance efforts. For more information about how to deploy Parasoft’s complete software analysis scans on their software, development management platform for the medical device software development market, visit parasoft.com. then placing static analysis in the context of an end-to-end compliance process.

ABOUT PARASOFT www.parasoft.com Parasoft helps organizations perfect today’s highly-connected applications by automating time- Parasoft Headquarters: consuming testing tasks and providing management with intelligent analytics necessary to focus +1-626-256-3680 on what matters. Parasoft’s technologies reduce the time, effort, and cost of delivering secure, Parasoft EMEA: reliable, and compliant software, by integrating static and runtime analysis; unit, functional, +31-70-3922000 and API testing; and service virtualization. With developer testing tools, manager reporting/ Parasoft APAC: analytics, and executive dashboarding, Parasoft supports software organizations with the +65-6338-3628 innovative tools they need to successfully develop and deploy applications in the embedded, enterprise, and IoT markets, all while enabling today’s most strategic development initiatives Copyright 2017. All rights reserved. Parasoft and all Parasoft products and services listed within are trademarks or registered trademarks of Parasoft Corporation. All other products, services, and companies are trademarks, registered — agile, continuous testing, DevOps, and security. trademarks, or servicemarks of their respective holders in the US and/or other countries.