Accelerating Devsecops with Containers and Continuous Testing PARASOFT.COM

WHITEPAPER Accelerating DevSecOps With Containers and Continuous Testing PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

DevSecOps represents a shift in software development processes that stresses collaboration between end users, developers, and IT professionals. Software test automation can enhance these connections and help organizations achieve desired secure software development acceleration.

Software test automation plays an important role but it’s just one piece of the DevSecOps puzzle. Testing is often one of the greatest constraints in the software development life cycle (SDLC) so optimizing security processes that allow testing to begin earlier, as well as shrink the amount of testing required, has a significant impact on the security of the software and development efficiency.

Containers play an important role in the secure deployment of software, providing a standardized, lightweight, and reproducible execution environment for applications in a DevSecOps pipeline. Containers are also becoming helpful in development as containerized development and testing environments are providing benefits like those in deployment. There is an important role for containers to play in modern DevSecOps development as containers can help shift-left security processes in software development.

Adopting a continuous testing process (more than just automated tests running regularly) helps promote all of the core pillars of DevSecOps:

» Collective responsibility

» Collaboration and integration

» Pragmatic implementation

» Compliance and development

» Automation and measurement

» Monitoring and reporting

This paper looks at the role of containers in development and testing as part of a DevSecOps pipeline in the context of a continuous testing regimen for achieving both quality and security.

2 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

CONTAINERS FOR SECURE DEPLOYMENT Containers as an execution and deployment approach is well established. They provide lightweight isolation for applications and (micro)services. Unlike more heavyweight options such as virtual machines, containers virtualize the operating systems and are considered more portable and efficient.

Containers are a perfect fit with continuous integration (CI) and continuous deploy (CD) pipeline. They support rapid deployment with portability across different host environments with support for version control. They are also important for security since it’s possible to provide a reproducible application environment with built-in security controls. For example, controlling and arbitrating the communication between components, referred to as the side car pattern, is recommended as part of the DoD Enterprise DevSecOps initiative.

MANAGING COMPLEX DEV ENVIRONMENTS Containerized deployments of development tools are becoming popular with development teams. Although containers were initially developed to solve problems with the deployment of microservices and web-based applications, they recently gained popularity with software teams to standardize and manage their complex toolchains. This becomes especially useful in embedded software development where toolchains are complex and must support multiple target environments.

When it comes to managing complex development environments, teams usually struggle with the following challenges.

» Synchronizing upgrades for the entire team to a new version of a tool like a compiler, build toolchain, and so on.

» Dynamically reacting to a new security patch for the library or software development kit (SDK), and the like.

» Assuring consistency of the toolchain for all team members and the automated infrastructure (CI/CD).

» Ability to version the development environment and restoring it to service the older version of the product.

» Onboarding and setting up new developers.

Containers for development environments are also suitable for SAST (static application security testing) and test automation tools. In fact, it’s possible to create a standardized, easily deployable development and test environment using containers. This ensures every developer works from the same versions of tools with the same configurations.

3 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

CONTAINERS SHIFT-LEFT SECURITY IN DEVSECOPS The drive to shift-left security in the SDLC comes from the desire to find and fix bugs and security vulnerabilities as early as possible. Issues are much easier, cheaper, and less risky to fix early, not later. Common sense, but the software industry is full of examples where critical defects caused catastrophic results.

Figure 1: Finding and fixing security vulnerabilities early is cheaper and less risky. Source: Applied Software Measurement, Capers Jones.

The essential requirements to shift-left security center around the need to incorporate security into any and all applications at the very beginning. Security can’t be added on, it must be built in. Here are some of the steps you can take to shift-left security in the DevSecOps pipeline and provide the necessary platform for continuous testing.

Use SAST as early as possible. Modern SAST tools find deep reliability and security issues hidden within the code that are often overlooked by human inspection or dynamic testing. Part of the adoption process should focus on a relevant coding standard. For regulated industries, coding standards (MISRA C 2012 or AUTOSAR C++14) are already defined for safety critical applications. For other types of applications, it's a good idea to consider coding standards focused on secure code development. The popular options are SEI CERT, OWASP Top 10, and CWE Top 25 and On the Cusp.

Introduce unit testing early. In the early stages of development, it's important to start unit testing and adoption of continuous testing practices. A foundation of unit tests ensures that code is functional, well designed and meets requirements, including security requirements. The emphasis should be on proactively using unit testing from the beginning, which results in better code that is easy to maintain, debug, and regression test. Unit testing is the ultimate shift-left component to focus the use of tools and test automation. Equally important is understanding how tests correlate back to the original work items. In other words, the requirements traceability to understand which functionality is tested and validated.

4 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

Focus on API testing. Early end-to-end testing of the assembled application can be done earlier and more effectively with API testing. These tests can be deployed in virtual or production environments via containers as appropriate. Using functional test automation, it is possible to validate the application at the API level, whether it is REST APIs, web services, or any other protocol message level testing. API level testing provides the ability to test system functionality with more reusability, shorter execution times and less impact from changes at the UI level.

Maintain a centralized view of security and quality. Combining several test automation techniques results in a lot of quality information. This data is useful, but the sheer volume makes it difficult to gain insight into the security and quality of your deliverables and make decisions about the release or changes in the development process. To support decision making this data must be aggregated into a centralized view of quality. Graphical dashboards provide a top-level view of testing progress and correlating quality and security data to your requirements and user stories provide true bidirectional traceability so you can assess the business risk and impact of gaps in your testing practices.

5 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

CONTAINERIZED DEVELOPMENT ENVIRONMENT Containers in the development context support these goals in several ways, most importantly, by providing a repeatable, versioned, and lightweight development environment. The containerized development environment includes all the necessary tools with the same version and configuration for each developer. The development environment can include the correct versions and configurations of SAST, unit test, build toolchain, and other tools as needed. Team leads and security analysts provide the necessary tools and configurations for these environments based on observed results and feedback from the continuous processes, illustrated below.

Policy Improvements

Dev Stream

SAST

IDE QA/Integration Security Deploy Build Unit Test

Code traceability Containerized Development Environment provides root- cause analysis

This feedback loop is critical to the success of continuous development and DevSecOps. Containerized environments boost this by providing Figure 2: Containers can be used for a known starting point for improvement. Removing the possible build, SAST analysis, and unit and API testing among others. Security policies differences each developer may have in their desktop environment and improvements are fed back into the removes the possibility of errors of tool version mismatches. The development environment. containerized development environment can be deployed on a regular basis, updated with new configurations, and redeployed.

6 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

CONTINUOUS TESTING TO ACCELERATE SECURITY IN DEVSECOPS Modern DevSecOps initiatives require the ability to assess the risks associated with a release candidate—instantly and continuously. Continuous testing provides an automated, unobtrusive way to obtain immediate feedback on the security risks associated with a software release candidate. It guides development teams to meet security requirements and helps managers make informed trade-off decisions to optimize the release candidate.

Continuous testing is not simply more test automation. Continuous testing delivers a quantitative assessment of risk as well as actionable tasks that mitigate risks before they progress to the next stage of the SDLC. The goal is to eliminate meaningless activities while improving quality and security and driving development towards a successful release.

CONTINUOUS TESTING AS A DEVSECOPS ENABLER Continuous integration, continuous deployment, continuous release, and continuous delivery are key DevSecOps enablers. Automated testing involves automated, CI/CD-driven execution of whatever set of tests the team has accumulated. However, if one of these tests fails, what does that really mean?

Does it indicate a critical business risk or just a violation of some naming standard that nobody is really committed to following anyway?

And what happens when it fails? Is there a clear workflow for prioritizing defects vs. business risks and addressing the most critical ones first?

For each defect that warrants fixing, is there a process for exposing all similar defects that might already have been introduced, as well as preventing this same problem from recurring in the future?

This is where the difference between automated and continuous becomes evident.

Continuous testing is a solution to the issues faced by software teams adopting DevOps and agile development. Wikipedia defines Continuous testing as “… the process of executing automated tests as part of the software delivery pipeline to obtain immediate feedback on the business risks associated with a software release candidate.” Despite the straightforward definition, implementing continuous testing and optimizing it over time is another thing entirely.

Moving from automated testing to continuous testing outlined above is a big leap. But it is one that yields significant benefits from a DevSecOps perspective. Through objective real-time validation of whether software meets security requirements via several “gates,” organizations can automatically and confidently promote release candidates through the delivery pipeline.

7 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

The benefits of this automated assessment include:

» A central view of quality throughout the process provides instant access to feedback on whether requirements are satisfied, enabling team leaders and managers to make informed decisions.

» An instant, objective assessment at the critical “go/no go” decision time, reducing the business risk of a fully automated continuous delivery process.

» Eliminates defects at the point when they are easiest, fastest, and least costly to fix — a prime principle of “shift-left”.

» Continuous measurement means continuous feedback, which can be shared and used to refine the process.

With a broad set of automated defect prevention and detection practices throughout the SDLC, both the product and the process are continuously measured. A detected vulnerability, for example, is more than just a bug to fix. Consider each problem found an opportunity to re- examine and optimize the process itself — a defect prevention feedback loop that incrementally improves the process.

LEVERAGING CONTAINERS TO ACHIEVE SECURITY AT SPEED WITH CONTINUOUS TESTING Continuous testing provides a framework to shift testing earlier in the life cycle. With the right application of automation and focused on the highest risk areas of the application, it’s possible to streamline testing to become much less of an inhibitor in Agile and continuous processes. Containers play an important role in this shift-left of testing and the success of continuous testing and DevSecOps in general. Consider the following three areas that will be expanded on later in the paper:

Containerize security practices. By taking advantage of the consistency and reusability that containers provide, continuous security testing is better served by making sure each developer and tester on the team is working from the same configuration. Moving tools and associated configurations into containers ensure consistent use and any changes propagate immediately. A smarter DevSecOps workflow means that security is “baked in” from the start.

Remove constraints in the DevSecOps pipeline. Despite the desire to accelerate security and quality, Agile and continuous processes hit barriers in early adoption and in day-to-day operations. Testing is, by far, seen as the culprit in slowing down Agile processes. The problem arises because teams don’t understand exactly what to test during each iteration. They either guess, try to test everything, or don’t test nearly enough. All three outcomes are unacceptable, particularly from a security standpoint. Teams can leverage smart test execution (STE) to focus their testing efforts and enable continuous testing.

8 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

Containerize secure development at every developer’s desktop. To achieve an efficient DevSecOps pipeline, developers need access to automation that removes the tedious aspects of their day-to-day work. Successful security — and quality and safety — development requires quick feedback from unit testing, API testing, UI testing, and SAST results. Plus, fine tuning security controls based on the most recent findings by the security team (see Figure 2).

Let’s discuss each of these in more detail.

CONTAINERIZE SECURITY PRACTICES Security requirements add additional workload for developers. How this is managed makes the difference between an on-time, secure product, and a late, insecure one. A critical requirement is to integrate security into the existing development process — to make security part of quality and the overall workflow.

The workflow starts with the secure coding policy. The Architect or Lead creates a configuration (possibly based on coding guidelines such as CERT, CWE, OWASP driven by standards like UL-2900 or PCI DSS) for the rest of the team to leverage directly within their IDE. This gives the developer the ability to check the code locally on their machine before committing to source control — catching and fixing security violations where and when it’s cheaper and easier to do so.

The same configuration is replicated through container images, providing consistency, then leveraged by analysis executed as part of the build process. This comprehensive analysis goes beyond the scope of the developer's locally modified code and provides a safety-net to gate the delivery pipeline to ensure that insecure code does not get promoted to later stages.

Lastly, the results of the analysis are sent back to the developer's IDE via the centralized reporting and analytics dashboard, where progress can be tracked, course corrections made, and audit reports generated in real time.

The full DevSecOps workflow looks like this:

Container Image Repository Pre-Commit Post-Commit Source Automation Repository Pull Images Unit SAST Test IDE Results

Parasoft DTP Developers Security & Quality Tasks

Figure 3: A DevSecOps workflow Manager / Lead with containerized implementation of security controls and processes. Architect / Lead Test Configuration(s) 9 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

Managers and security leads can now assess projects based on security standards such as CWE, OWASP Top 10 or SEI CERT in central dashboards. These dashboards can show trending information and answer questions, such as, “Is the project improving or getting worse?” or “Which areas of the code are causing the most issues?” Being able to answer these and other questions, and take action, transforms the development team from DevSecOps to SecDevOps.

REMOVE CONSTRAINTS IN THE DEVSECOPS PIPELINE The ideal test pyramid defines where it’s best to invest time and effort in a project. In the ideal pyramid, you invest your valuable time and effort into a comprehensive suite of unit tests at the foundation of the pyramid, which is backed up by API and service tests. At the top of the pyramid is a much smaller number of system and GUI-based testing.

Figure 4: The test pyramid. The ideal pyramid is built on a foundation of unit tests on up. Superimposed is the inverted pyramid which mirrors what many organizations are currently doing.

However, this pyramid is often inverted into an ice cream cone. Teams spend too much time and effort on brittle and complex system-level GUI tests that require full functionality to be implemented and integrated, resulting in tests that cannot be executed continuously during the earlier stages of the SDLC.

This is especially problematic with security. It's costly and risky to leave security testing to such a late stage of development. The key to achieving successful continuous testing is to melt the ice cream cone andfocus on creating automated unit and API tests for functionality and security that can be executed continuously as developers are implementing the new functionality.

10 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

The best approach to un-invert the test pyramid — which aligns nicely with the best practices needed for DevSecOps mentioned already — is to adopt these testing best practices:

» Use SAST as early as possible to find deep reliability and security issues hidden within the code that are often overlooked by human inspection or unit testing.

» Introduce unit testing early to ensure that code is functional, well designed, and meets requirements, including security requirements.

» Focus critically on API testing deployed in a virtual or production environment via containers as appropriate.

API-level testing provides the ability to test system functionality with more reusability and less impact from changes at the UI level. This means a lot of data is generated by the automated process. To support decision making, this data must be aggregated into a centralized view of quality. Graphical dashboards provide a top-level view of testing progress. You can create additional work items based on review of the data, identified defects, and other issues.

A way to focus testing is missing from the best practices listed above and begs an answer to the question, "What to test?" Focusing testing on exactly what needs to be reviewed after each code change is critical to accelerating testing, enabling continuous testing, and accelerating the DevSecOps pipeline.

11 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

LEVERAGE SMART TEST EXECUTION TO ACCELERATE AGILE To accelerate testing in an Agile or continuous pipeline, smart test execution on a per-build basis is needed to reduce the set of tests required to be executed to address the risk that each new iteration has introduced. The analytics provided by test impact analysis are key to making testing focused on only what absolutely needs to be tested rather than the shotgun approach used otherwise.

Only through smart, can data-based decision-making enable continuous testing. Focusing the development team on the minimum set of tests to ensure proper coverage at each iteration is the key to bring the agility back to Agile development methods.

Figure 5: Accelerated security in a DevSecOps pipeline based on the test pyramid. A solid foundation of early stage tests, a focus on API testing, and reserving system-level and UI tests for specific cases.

Smart test execution is extended with plugins for CI systems (Jenkins, TeamCity, Bamboo, and so on) for advanced functionality to help software development organizations reduce bottlenecks associated with running continuous builds. The same capabilities are available inside IDE environments with dedicated plugins that access a centralized coverage image through a REST API, and determine which tests need to be executed locally inside the IDE to verify all changed code.

On the development side, experienced developers might apply the proper structure in organizing their tests and run only a subset of them manually, but they still might not know which tests need execution to verify ALL changes.

Teams using CI might rely on nightly builds to execute all tests automatically overnight and get feedback the next day, but only if it's possible to execute the total number of tests in under 12 hours.

Unfortunately, most software development teams are running their day-to-day operations accepting those unscalable testing practices. The situation becomes even more difficult when manual testing is involved. “Traditional” software development organizations are still following testing practices represented by an inverted pyramid, which emphasizes running manual tests over automated tests for one reason or another.

12 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

Figure 6: An example high-level chart provided by Parasoft DTP showing the distribution of tests that passed, failed, are incomplete, and need retesting. Users can click the pie chart to view more details.

Smart test execution uses test impact analysis to trace the execution of manual tests against applications and associated, captured code coverage information with those tests. Similar technology is used for automated testing. This analysis figures out which manual tests need execution to access changed functionality delivered with every new build. Thus, smart test execution is critical at the developer and tester levels in their local IDEs. It enables them to focus the testing where it’s needed, removing guesswork and extra “just in case” work.

Figure 7: An example of Parasoft DTP analysis of the impact on tests, per file, detected due to recent changes to the code base.

13 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

CONTAINERIZE SECURE DEVELOPMENT AT EVERY DEVELOPER’S DESKTOP The benefit of containers for development teams extends to the developer’s desktop by providing a consistent and reusable toolchain (including testing/ SAST configuration) across developer, build, integration, and deployment environments. Rather than updating local workstations each time configurations are updated, only the container image needs to be updated in a container-based development. The entire team can benefit.

Team-based tool deployments can use containers to host repositories, different toolchains (especially important for embedded software development), SAST, and test tools. An illustrative example of a possible containerized development environment is shown below.

Automation Container Image Repository Unit SAST Test Pull Images

Parasoft DTP IDE

Figure 8: Containers Source Build provide an efficient way Repository for developers to have a consistent development and testing environment.

Where developers really start to accelerate secure development is with local “workstation” adoption of smart test execution. Reinforcing good security and engineering practices (to perform security analysis and test execution locally and often) between code modifications and before committing any changes to the source control system. This process significantly reduces the feedback loop on the security and quality of modified code, generally speeding up the “code, test, submit” workflow.

Development teams can use a continuous testing workflow to help focus testing efforts and make sure teams are performing only necessary tests. In addition, constant monitoring of code security with SAST tools is critical to ensure that coding standards are adhered to and that tricky vulnerabilities aren’t eluding security tests.

14 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

Pre-Commit Workflow

1. Decide on a security standard that suits the need of the project and organization like OWASP, CWE, CERT, and so on. You can customize and combine levels of compliance and composition of coding standards to suit your needs — security and quality standards, for example.

2. Encapsulate the security policy in a test configuration. This is usually done centrally for an entire project by a team lead. Update necessary containers to use this configuration so each image pulled by the team has the correct version.

3. Make defined configuration(s) available for developers to use when they're writing and testing code. The key benefit here is getting standards in place on every developer’s desktop environment, so everyone is working to the same standard.

4. Apply checkers to code before check-in. Our experience shows that the best practice is to not make compliance with the test configuration a gate to check-in. Doing so impacts the team’s efficiency and adoption of secure coding practices. Code that isn’t fully compliant is often checked in for various reasons that tools can’t necessarily evaluate. The purpose of the post-commit part of the workflow is to perform a "complete analysis" and act as a safety net.

Post-Commit Workflow

1. Build code, run existing tests, and performproject-wide static analysis.

2. Inspect results published to the security dashboard to determine areas of concern.

3. Use smart test execution to assign new work items for the team. High risk areas receive the most attention via re-tests, updated tests, or actual code changes to correct errors.

4. Analyze results, prioritize SAST violations, and assign them accordingly in the form of tasks for the appropriate developer. Instant feedback is available via pre-set and customizable dashboards.

5. Take actions to address the warnings and violations that are published and available in every IDE for review.

Quick feedback and analysis of SAST and test results reduces the friction in continuous testing. The consistency provided by container-based development environments means results are the same across the team, reducing the need to “debug” different development environments when results don’t match. Bringing the power of smart test execution to the developer’s desktop truly accelerates secure software development.

15 PARASOFT.COM

Accelerating DevSecOps With Containers and Continuous Testing

SUMMARY Containers as an execution and deployment approach is well established. They provide lightweight isolation for applications and (micro)services. Containers are also a perfect fit with continuous integration (CI) and continuous deploy (CD) pipeline. They support rapid deployment and portability across different host environments with support for versioning and centralized control. CI/CD are also important for secure development in a DevSecOps pipeline since it’s possible to provide a reproducible application environment with built-in security controls.

Continuous testing provides a framework to shift testing earlier in the life cycle. With the right application of automation and focus on the highest risk areas of the application, it’s possible to streamline testing to be less of an inhibitor in Agile and continuous processes. Development use of containers extends benefits of versioning and reusability to the developer desktop by providing a consistent toolchain, testing/SAST configuration, development and (local) deployment, and test environment.

Leverage containers to make continuous testing a reality and your DevSecOps initiative a success. Parasoft is here to help you get started. Talk to one of our experts today.

ABOUT PARASOFT Parasoft helps organizations continuously deliver quality software with its market-proven, integrated suite of automated software testing tools. Supporting the embedded, enterprise, and IoT markets, Parasoft’s technologies reduce the time, effort, and cost of delivering secure, reliable, and compliant software by integrating everything from deep code analysis and unit testing to web UI and API testing, plus service virtualization and complete code coverage, into the delivery pipeline. Bringing all this together, Parasoft’s award winning reporting and analytics dashboard delivers a centralized view of quality enabling organizations to deliver with confidence and succeed in today’s most strategic ecosystems and development initiatives — cybersecure, safety-critical, agile, DevOps, and continuous testing.