Identifying Application Performance Risk

Performance Engineering

COLLABORATIVE WHITEPAPER SERIES COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk

Under what conditions will an application require a performance test? Not all applications require performance testing, and the same application may not require repeated performance testing for every release. The selection process for performance testing frequency must consider user population, application type, technology, changes to features and function, and how non- functional requirements are monitored in the software development life cycle (SDLC).

I. Introduction A typical enterprise has thousands of applications. In a business unit, there can be several hundred applications. As the business continues to evolve and change, so will the applications evolve and change to support the business needs. The changes made to the application may increase the risk to performance, scalability, or stability. Depending on the business’s tolerance for disruption, these changes may require some level of performance and scalability testing to verify the application can still process the accepted business volumes while staying within service level agreements. These application changes are typically scheduled on a release calendar. Customer facing applications should be considered critical to the business, while internal facing applications may be less critical. However, what if the internal application is supporting the executive level of the company and provides key information for decision-making? Some enterprises believe inherently that performance and scalability testing are required, while others may leave the decision to the business units. Enabling the enterprise for performance engineering requires an investment. This investment is considered to be an indirect investment, as it may not be immediately linked to a revenue generating process. The value of the investment in performance engineering can be quantified. For example, when performance engineering recommends design changes allowing a revenue- generating website to process more orders, the return on investment is clear. In a large enterprise, performance engineering resources are often in short supply, and budgets are generally under constant pressure. This leads IT and business decision- makers to ask certain questions, such as: What guidelines can you use to help allocate these resources across the project portfolio? How can you make sure you have not missed an application that required performance testing, and how can you make sure you’re not testing the wrong applications? Are you sure the results from performance are accurate and allow management to make informed decisions?

2 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk

II. What factors should you consider when or replacing one of the tiers, there may be a great risk and selecting an application for performance testing? thus testing is required. Likewise, a significant upgrade to a vendor product could warrant performance testing. It is Not all applications require performance testing, and the important to consider the scope and impact of changes to same application may not require repeated performance any key components. testing for every release. The risk factors you should use to evaluate your application include: 4. Application features and functions 1. User population The amount of modified code or new code in an application can create new performance risks. Understanding the The people who use your application are critical to the impact of the changes is critical to determining if decision. Questions may include: Who are the users of performance testing is required. Potential analysis questions your applications, how many concurrent users are there, may include: How has the new or modified business feature and is the number of users increasing? Are the users changed the behavior of the application? Were the changes purchasing products or services from your business? Are extensive and across the client, application services, and they external or internal users? How easy is to for your database? What percentage of the code was impacted by user base to switch to a competitor if you website is the new or modified services? not performing well? How is the application accessed? Access types may include browsers (both desktop and 5. Software development process mobile), call center applications (both client server, legacy, Analysts may consider questions such as: Does the SDLC and web), and even internal voice recognition (IVR) track non-functional requirements during the lifecycle? clients. The amount of transactions for each access type How are those non-functional requirements communicated and application must be considered when defining the from the requirements, design, development, testing workload for the user population. and deployment teams? Have key business transactions 2. Application type or services been identified with stringent response time requirements, or strict throughput requirements? What The application type itself can dominate the risk architectural risk analysis, prototyping, or other types of factors. For instance, questions to ask may include: Is testing have been done throughout the lifecycle that may the application an online retail website? Is there both mitigate the need for formal performance and scalability a desktop and mobile website/mobile app? Typically a testing efforts? revenue generating website requires a performance test for every release. Is the application a key component in 6. Production issues with the last release the enterprise architecture that other applications use? If Recent history can be an indicator for the future. If the so, this is deemed a critical application and may require application went into production and the last release had performance testing for every release. Is the application performance, scalability, or stability issues, then it may a batch process with a strict window of processing? How require a closer look at the application to determine if the critical is this application to the business, and how is it issues have been truly mitigated. Otherwise, performance rated? testing is required. Similarly, resource utilization patterns 3. Application technology and trends may be used to assess the need for further risk mitigation. The state of the application technology stack can be a significant risk factor. Generally, the technology platform does not change from release to release. If the underlying technology is stable and is well known to the application development team, a performance test might not be required. However, if a new technology is being introduced

3 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk

7. The schedule of performance tests uses this application? What is the behavior of the users’ workload? In an analysis, we may ask: What would be the Applications will undergo performance testing at different business impact if this application exhibited performance times during their lifetime. The application can be tested problems? Is your business in a highly competitive market, before it is ever released into production; performance where dissatisfied users can switch to you competitor? testing can be scheduled for every major release; or For instance, if the response time doubled or tripled from performance testing can be scheduled based on the normal, how would the user experience degrade, and what extent of the application’s changes. would be the financial impact of service calls and shopping cart abandonment? How long would this condition be III. Risk factors elaborated tolerated and under what conditions would it be tolerated? 1. User population The matrices shown below may be used to identify and The key information to capture is: How many people use assess potential performance engineering risks, using a the application on average and during peak time? Who simple 1-2-3 scoring system:

Risk Level User Population Characteristics 1: requires a performance check Figure 1: Figure title 2: depends on other extenuating factors 3: does not require a performance test

Business internal: Senior management or 2 board level, enterprise-wide

Business external 2

Consumers 1

1: Thousands Average number of concurrent users 2: Hundreds

Peak number of concurrent users relative to 1: Application peak TPS is more than five normal business load based on month-end or times the typical TPS for the business periodic event that occurs regularly

1: If population growth doubles in less than nine months 1: A significant increase is expected due to Rate of user population growth the business winning a new contract or large client 3: No user population growth

Flash or sudden spikes in concurrent users, above and beyond your current peaking factor: • Based on a new business model of broadcasting messages to your client base telling them to all come to your website 1: Upcoming flash or spike events will add to existing application workload • Currently a novelty event for many businesses, not accounted for in the system design • Eventually folded into the traditional peaking factor

4 If you have a large number of consumers that is growing, you would require performance and scalability testing. For example, flash events are a key risk area because they are occurring more frequently for businesses on application platforms that were neither designed nor tested for sudden workload bursts. 2. Application type Most businesses will rate their applications based on criticality to the business. The rating is typically for availability and recovery time. Ratings can range from high availability (almost no down time) rated A, to minutes of down time rated B, to hours rated C, to days rated D. These ratings factor into your selection criteria. Should a C rated application be performance tested? Under specific conditions it should; for instance, it is a major application upgrade or if it is chronically slow for the users to the point that it is interfering with their workflow and productivity.

Risk Level Application Characteristics 1: requires a performance check 2: depends on other extenuating factors 3: does not require a performance test

Online website retailing (may include both desktop browser website and mobile 1 website/app)

Financial transactions 1 (brokerage, 401K, etc.)

1: Enterprise-wide use Corporate support 2 or 3: Group-wide

1: Open enrollment is a key business Customer portal: This is where the customer process that occurs once per year can maintain their account information, view 1: Most customer facing systems require their history of transactions, or check the some level of performance testing, the end- status of an order user response time is critical to how your customers view your business

Business portal: This is where the business partner can maintain their account 1: The business partner is responsible for information, view their history of transactions, driving revenue to your business or check the status of an order. It could be for 2: The business partner provides a service an insurance agency with a large number of to your business that is part of a workflow independent agents

1: Service level penalties if late Business intelligence and analytics: Large 1: First mover advantage, business, volumes of data analysis, tight reporting competitive edge deadlines or regulatory filings 2 or 3: Weekly, monthly, quarterly reporting

Date hub or integration hub: A key component where many applications are 2 dependent upon

Informational or marketing: User experience 2 or 3 is important if this is for consumers

Batch processing: Constrained processing Rating depends on extent of penalties windows for violating SLA’s

1: Timeliness of documents is critically Document management important, e.g. news agency or day-trading investment advisor 5 2 or 3: Otherwise

Imaging systems: These are typically scanning systems that are used in mail order 1: Part of a time critical workflow. The processing applications, or where signature is documents must be scanned quickly and required. They are part of an essential made available quickly workflow

A core Enterprise-wide component, used by 1 or 2 many other applications Risk Level Application Characteristics 1: requires a performance check 2: depends on other extenuating factors 3: does not require a performance test

Online website retailing (may include both desktop browser website and mobile 1 website/app)

Financial transactions 1 (brokerage, 401K, etc.)

1: Enterprise-wide use Corporate support 2 or 3: Group-wide

COLLABORATIVE WHITE PAPERDate SERIES: hub or integration hub: A key Identifying Applicationcomponent Performance where Risk many applications are 2 dependent upon

Informational or marketing: User experience 2 or 3 is important if this is for consumers

Batch processing: Constrained processing Rating depends on extent of penalties windows for violating SLA’s

1: Timeliness of documents is critically Document management important, e.g. news agency or day-trading investment advisor 2 or 3: Otherwise

A core Enterprise-wide component, used by 1 or 2 many other applications

3. Application technology When you are using unfamiliar technologies, you need proactive performance management. In general, the application technology stack does not change for every release. Nevertheless, it is important to consider the cumulative effect of small changes over time with respect to online features and batch processes.

Risk Level Technology Characteristics 1: requires a performance check 2: depends on other extenuating factors 3: does not require a performance test

Well-known and stable technology stack 3

Accessed via more than one client presentation tier (i.e., classic browser 1 website, mobile website, web app)

New technology for a key component: For instance replacing the WebSphere application 1 server with Jboss, or replacing a C++ batch process with Java batch process

Significant technology upgrade: For instance, upgrading the WebSphere application server, upgrading the Database to a new major 1 release. The key here is to identify impact and determine the proper level of testing

Significant application upgrade: This is typically for a third party vendor who made significant architecture changes, particularly 1 if the new version of the application has not been demonstrated at full business volume New component added to the application: 1 or 2: It is critical to understand the number The application has new component that will of transactions that use this component and become part of most of the key business the nature of the transactions transactions

6 4. Application features and functions This section considers the magnitude of the changes made to the application for this release cycle. The goal is to assess the risk the new change poses to the performance, stability, and scalability of the application. • Where new database tables were added or existing ones modified, did this impact the database partitioning scheme? • On the application server, were new services introduced, or existing ones modified? Will this impact the application server’s scalability profile? • Did the web client change to introduce asynchronous web server requests? How will this impact the transaction arrival rate to the web server and application servers? • Has this application experienced performance or stability issues in the recent past? Is there a history of performance issues after a new release? • Is the application sensitive to a sudden increase in user load or transactions per second? Under what conditions does this occur?

Risk Level Impact 1: requires a performance check 2: depends on other extenuating factors 3: does not require a performance test

1 or 2: Introducing asynchronous Web server Web client requests

1: New plug-ins or new connections to Web Server application servers

1: Services added or changed with Application server suspected performance impact

1: Impact to database partitioning scheme 1/2: New tables added to high volume Database server transaction 1: New requirements for keeping and accessing historical information

All tiers impacted by change 1

Percentage of overall code changes or new code developed. For instance, if there are 1: More than 15% 100,000 lines of code, what percentage was 2: 5%-to 15% modified?

Batch processing – where new jobs were introduced, was the concurrency model 1: Concurrency model impacted impacted?

Reporting server 1: New tools providing ad hoc and wider or deeper queries

7 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk

5. Software development process nature of the production issues related to performance. It will allow them to build a historical profile for the production The software development process should identify behavior of the application. non-functional requirements, such as performance, scalability, maintainability, usability, and others. During the This linkage of the performance team to the operations team requirements phase, any key business use cases that have provides a feedback loop to the performance team. This response time requirements must be identified and tracked information will allow the team to adjust the performance through the process. Also, key business use cases for test scenarios based on changing workloads and new or batch process throughput should be identified. In general, modified business functions. activities that support performance engineering should be 7. The schedule of performance tests conducted throughout the entire lifecycle, and they should be defined in a standards document so that assessment of Applications will undergo performance testing at different performance risk can be understood. times during their lifetime. The application can be tested before it is ever released into production; performance Coordinated Performance tests: The performance team testing can be scheduled for every major release; or will lay out a plan to coordinate the goals and test cases performance testing can be scheduled based on the for performance in architecture proof-of-concepts (POCs), extent of the application’s changes. unit testing, functional quality assurance (QA), and performance testing. The development/build phase has a a. Project-based schedule: unit testing activity, the functional QA testing phase will Project-based performance tests can be driven by the nature have a performance test activity, and the performance of the change to the application and technical architecture, testing process will require the outcome of those two rather than by the profile of the user population. This activities. Early in the lifecycle, an architectural POC may approach is appropriate if the user population is not growing, be required, and it should include a set of performance, but the technology is changing. If the application uses stability, and scalability tests. The performance team can a vendor product as a key part of the workflow, and the unify the strategy of introducing performance testing into vendor has introduced a new major release, the level of risk the other testing activities. may be high. Before agreeing to forgo in-house performance The performance team can introduce “micro” performance testing, the application owners must be more than satisfied tests into the other phases of the SDLC where they that the vendor had done proper performance testing. Once already have testing activities, for architecture/design, the new version has been verified not to harm the response development, and QA. time or throughput, the performance test project is over. 6. Recent release and production issues Oftentimes, the application is custom-developed by the IT organization for the business. The custom-developed The recent past can be an indicator of the future. If application may be in need of technology upgrade or performance, scalability, or stability issues arose from replacement. This would require a project-based approach the last release of the application, performance testing to performance testing, possibly coupled with a release- is needed on the next release. In some instances, the based approach to the schedule. The custom-developed application development team may have a poor track application may be scheduled for iterative deployment record for managing application performance, resulting across different release cycles. in frequent performance issues. The risk of the new technology or upgrade must be The performance team must see the application understood. What did the vendor or IT development team performance management reports from the production change? You must understand how the changes impact operations team. This will allow them to determine the the users’ workflow. You must measure the before and

8 after performance of the system. In order to do a proper done in isolation or behind closed doors. The selection comparison, you must understand the key business process and the risks must be visible, and there must be transactions’ response times and the batch process communication between the performance team and the throughput of the existing version. business, ultimately to answer the questions of “why are we testing?” and “what are we getting from those tests?” b. Release-based schedule Establish a transparent selection process, as some Release-based tests are recurring and are scheduled for application owners budget for performance testing and every release. Since the application has been deemed others do not. This selection process will allow you to critical, it requires a baseline test, performance validation, work with the applications’ owners who do not budget for or performance characterization for every release. The performance testing, providing them a structured process for type of test may change from release to release due to identifying risks to the business as a basis for selecting the the nature of the change. The test scenarios can change applications to include in performance testing. based on observations from the production environment. Likewise, a changing production workload or changing Is the current vendor telling your business they must user behavior will require the performance test cases to upgrade? Often software products are selected by the change. The performance team must have a link to the business because they provide a set of key business production application-monitoring and capacity-planning functions, while little attention is paid to the non-functional team. requirements of the business. This process will allow you to assess the risk introduced by a vendor solution, either The user population could be increasing, and the business during the selection process or with an established vendor may have introduced new features or functions that have who wants you to upgrade to the new version. changed since the last baseline test. This approach provides the criteria for defining the c. Production support schedule performance risk or scalability risk the application may Production support test cases are required to help triage or may not present to the business. For instance, you production issues. When issues arise due to performance, should always be looking out for business initiatives scalability, or stability, the root-cause must be found, and that introduce a new extreme workload, such as a new one or more solutions must be defined. These tests are marketing campaign that will encourage customers to login disruptive to the planned performance testing schedule to see the new product. Then in response, you must design and environment. Oftentimes, the performance team has and execute a scalability test to verify the behavior of the the only properly sized environment to reproduce the application under the anticipated flash-type workload. production issue. Release and configuration management Most importantly, this approach provides visibility into the become critical in this situation. You need mature performance testing risk-based selection process. This processes to be able to suspend the current performance approach allows you to communicate the risk to business testing schedule and execute on the production issues. and other key stakeholders by showing the connection These test cases are created on-demand and adjusted as between the business risk and the testing plan to reduce new information is uncovered. The tests are created to that risk. either locate the root-cause or to help define a solution. Now that you have identified the applications that should be performance tested, what types of tests should you be IV. Conclusion designing and executing? See the next white paper from This approach will allow you to allocate the right resources Collaborative Consulting on the different categories of to the right projects and applications. Performance performance tests. engineering and performance testing should not be

9 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk

Collaborative Consulting is a leading information technology services firm dedicated to helping our clients achieve business advantage through the use of strategy and technology. We deliver a comprehensive set of solutions across multiple industries, with a focus on business process and program management, information management, software solutions, and software performance and quality. We also have a set of offerings specific to the life sciences and financial services industries. Our unique model offers both onsite management and IT consulting as well as U.S.-based remote solution delivery.

To learn more about Collaborative, please visit our website at www.collaborative.com, email us at [email protected], or contact us at 877-376-9900.