
White Paper Continuous Testing: The Good, The Bad, and the Ugly The field of software functional testing is undergoing a major transformation. What used to be an onerous manual process took a big step forward with the advent of Selenium and other testing tools. But those tools remain heavily developer-centric and are primarily tethered to rigid script-based approaches that do not scale. And now, with the rise of cloud, big data and machine learning, companies need to re-think their testing strategies and technologies yet again. The core concept behind this new approach is that, in this era of agile development and rapid customer response, testing can no longer be an adjunct to the development process: it needs to be an integral and strategic component of the workflow. Functionize is doing just that with the introduction of our ground-breaking scalable autonomous test platform. The driving force behind this platform is Adaptive Event Analysis TM, a patented and proprietary hybrid of algorithms which span supervised, unsupervised, reinforcement learning and computer vision, enabling lightning-fast test creation and execution, self- healing maintenance, and actionable analytics. Continuous testing is a framework for running automated tests—as early as practicable and across the product delivery pipeline—in which the results of these tests quickly provide risk exposure feedback on a specific software release candidate. The promise of continuous testing is faster delivery of higher quality software. Functionize can help you move faster toward a continuous testing framework that will progressively deliver better products to your customers. The Promise of Continuous Testing The promise of continuous testing is faster delivery of higher quality software. Test automation produces a set of failure/acceptance data points that correspond to product requirements. Continuous testing has a broader scope across more of the development cycle, focuses more on business risk, and provides more insight on the probability that a product is going to be shippable. Continuous testing is a framework for running automated tests—as early as practicable and across the product delivery pipeline—in which the results of these tests quickly provide risk exposure feedback on a specific software release candidate. Generally, the goal is to achieve higher speed with higher quality, by moving testing upstream and testing with a higher degree of frequency. It’s easy to ship a software product if testing is minimal; and it’s easy to get out good software if you’ve got a whole year to dump a feature. Test early, test often, test exhaustively, and get the payoff in higher quality products that potentially release sooner. The price for all of this? You’ll need to reconfigure your delivery pipeline. Full- bore continuous testing includes not only code coverage, functional quality, and compliance, but also impact analysis, and post-release testing. The Need for Continuous Testing Changes in software development continue to increase stress on testing teams—like never before. Also, the complexity of newer technologies and components present more challenges in achieving test automation with conventional methods and tools. Extensive, complex application architectures: Software tools and technologies continue to become more complex, cloud-connected, distributed and expansive with APIs and microservices. There is an ever increasing number of combinations of innovations, application components, and protocols that interact within a single event or transaction. Frequent releases/continuous builds: DevOps and Agile continue a big push toward continuous delivery, which has brought the industry to the point at which no small number of applications release builds many times per day. This is only possible when significant effort has been put into the product lifecycle to automate testing and assess risk of failure. It also means that end-of-cycle testing time must have a much short duration. Managing risk: Software is a primary business interface, so any application failure translates directly to a failure for the business. A “minor” glitch will have a serious negative impact if it significantly affects user experience. For many software vendors and service providers, application integrity risks are now a critical concern for all business leaders. Learn more at: www.functionize.com How does CT differ from Testing Automation? We can categorize the main differences between test automation and continuous testing with categories: risk, broader coverage, and time. Minimizing Business Risk Today, most businesses not only expose many elements of their internal applications to external end users, they have also built many types of additional software that ex- tend and complement those internal applications. Major software application failures have brought serious repercussions to the extent that that software-related risks are now high-profile aspects in many business financial filings. On average, recent statistics suggest that notable software failures result in an average 4% stock price decline—about $2.5 billion reduction in total market capitalization. This is a direct hit to the bottom line, so business leaders are putting more pressure on their IT leaders to find a remedy. With continuous testing, if your test cases haven’t been built to readily assess busi- ness risk, then the results won’t provide the feedback necessary to continually assess risk. The design of most tests is to provide low-level detail on whether requirements/ specifications have been met. Such tests give no indication of how much risk the business would take if the software were released today. Think about this: Could your senior management intelligently make a decision to cancel a release according to test results? If the answer is no, then your tests are out of alignment with your business risk assessment criteria. Broader Coverage Even when a company manages to avoid the detriments of large-scale software failures, it remains true that a supposedly minor defect can cause major problems. If a user evaluation results in an unsatisfactory experience or fails to meet expectations, there is a real risk that the customer will consider your competitors. There is also the risk of damage to the brand if any user takes his complaints to news media. Merely knowing that a unit test fails or an interface test passes doesn’t tell you the extent to which recent app changes will affect user experience. To maintain continuity and satisfaction for the user community your tests must be sufficiently broad to detect application changes that will adversely impact functionality on which users rely. Time The speed at which organizations ship software has become a competitive differentiator, so a majority of companies are looking to DevOps, Agile, and other methodologies to optimize and accelerate delivery pipelines. In its infancy, automated testing brought testing innovations to internal applications and systems that were built with conventional, waterfall development procedures and processes. Since these systems were fully under the control of the organization, everything was dev-complete and test-ready at the designated start of the testing Learn more at: www.functionize.com phase. With the rise of Agile and DevOps, the expectation is forming in many companies that testing must start very soon after development begins. Some highly-optimized DevOps teams are actually realizing continuous delivery with consistent success. These teams can often deliver releases every hour of the day—or more frequently. Feedback at each step in the process must be virtually instantaneous. If quality isn’t a critical concern at your company—minimal disincentive for rolling back when defects are found in production—then it might be sufficient to quickly run some unit and smoke tests on the release. If, on the other hand, your management and your team have got to the level of frustration that drives you to minimize the risk of releasing defective software to customers, then you might be searching for a way to achieve solid risk mitigation. For testing, there are a number of significant impacts: • To be effective in continuous delivery pipelines, testing has to become an integral activity for the entire development cycle—instead of continuing to be seen as a hygiene activity that occurs post-development. • As much as possible, tests should be built concurrently and be ready to execute very soon after the new functions or features are built. • The entire team should work together to analyze and determine which tests should be run at specific points in the delivery pipeline. • Each test suite should be configured to run fast enough to avoid any bottleneck in a particular stage in the software delivery pipeline. • Environment stabilization is important to prevent constant changes from raising false positives. Minimizing Risk By Moving Testing Further Upstream Development teams need to improve their testing abilities and add test automation wherever it’s necessary in the delivery pipeline. One approach is known as service virtualization, which simulates systems or components that are unavailable. This enables a team to test further upstream in the pipeline and test more frequently. When combining service virtualization with test automation, a development team can achieve immediate feedback on quality, which translate into faster and less costly issue resolution. By pursuing and cultivating this approach in their development teams, businesses can bring
Details
-
File Typepdf
-
Upload Time-
-
Content LanguagesEnglish
-
Upload UserAnonymous/Not logged-in
-
File Pages6 Page
-
File Size-