Identifying Application Performance Risk
Total Page:16
File Type:pdf, Size:1020Kb
Identifying Application Performance Risk Performance Engineering COLLABORATIVE WHITEPAPER SERIES COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk Under what conditions will an application require a performance test? Not all applications require performance testing, and the same application may not require repeated performance testing for every release. The selection process for performance testing frequency must consider user population, application type, technology, changes to features and function, and how non- functional requirements are monitored in the software development life cycle (SDLC). I. Introduction A typical enterprise has thousands of applications. In a business unit, there can be several hundred applications. As the business continues to evolve and change, so will the applications evolve and change to support the business needs. The changes made to the application may increase the risk to performance, scalability, or stability. Depending on the business’s tolerance for disruption, these changes may require some level of performance and scalability testing to verify the application can still process the accepted business volumes while staying within service level agreements. These application changes are typically scheduled on a release calendar. Customer facing applications should be considered critical to the business, while internal facing applications may be less critical. However, what if the internal application is supporting the executive level of the company and provides key information for decision-making? Some enterprises believe inherently that performance and scalability testing are required, while others may leave the decision to the business units. Enabling the enterprise for performance engineering requires an investment. This investment is considered to be an indirect investment, as it may not be immediately linked to a revenue generating process. The value of the investment in performance engineering can be quantified. For example, when performance engineering recommends design changes allowing a revenue- generating website to process more orders, the return on investment is clear. In a large enterprise, performance engineering resources are often in short supply, and budgets are generally under constant pressure. This leads IT and business decision- makers to ask certain questions, such as: What guidelines can you use to help allocate these resources across the project portfolio? How can you make sure you have not missed an application that required performance testing, and how can you make sure you’re not testing the wrong applications? Are you sure the results from performance are accurate and allow management to make informed decisions? 2 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk II. What factors should you consider when or replacing one of the tiers, there may be a great risk and selecting an application for performance testing? thus testing is required. Likewise, a significant upgrade to a vendor product could warrant performance testing. It is Not all applications require performance testing, and the important to consider the scope and impact of changes to same application may not require repeated performance any key components. testing for every release. The risk factors you should use to evaluate your application include: 4. Application features and functions 1. User population The amount of modified code or new code in an application can create new performance risks. Understanding the The people who use your application are critical to the impact of the changes is critical to determining if decision. Questions may include: Who are the users of performance testing is required. Potential analysis questions your applications, how many concurrent users are there, may include: How has the new or modified business feature and is the number of users increasing? Are the users changed the behavior of the application? Were the changes purchasing products or services from your business? Are extensive and across the client, application services, and they external or internal users? How easy is to for your database? What percentage of the code was impacted by user base to switch to a competitor if you website is the new or modified services? not performing well? How is the application accessed? Access types may include browsers (both desktop and 5. Software development process mobile), call center applications (both client server, legacy, Analysts may consider questions such as: Does the SDLC and web), and even internal voice recognition (IVR) track non-functional requirements during the lifecycle? clients. The amount of transactions for each access type How are those non-functional requirements communicated and application must be considered when defining the from the requirements, design, development, testing workload for the user population. and deployment teams? Have key business transactions 2. Application type or services been identified with stringent response time requirements, or strict throughput requirements? What The application type itself can dominate the risk architectural risk analysis, prototyping, or other types of factors. For instance, questions to ask may include: Is testing have been done throughout the lifecycle that may the application an online retail website? Is there both mitigate the need for formal performance and scalability a desktop and mobile website/mobile app? Typically a testing efforts? revenue generating website requires a performance test for every release. Is the application a key component in 6. Production issues with the last release the enterprise architecture that other applications use? If Recent history can be an indicator for the future. If the so, this is deemed a critical application and may require application went into production and the last release had performance testing for every release. Is the application performance, scalability, or stability issues, then it may a batch process with a strict window of processing? How require a closer look at the application to determine if the critical is this application to the business, and how is it issues have been truly mitigated. Otherwise, performance rated? testing is required. Similarly, resource utilization patterns 3. Application technology and trends may be used to assess the need for further risk mitigation. The state of the application technology stack can be a significant risk factor. Generally, the technology platform does not change from release to release. If the underlying technology is stable and is well known to the application development team, a performance test might not be required. However, if a new technology is being introduced 3 COLLABORATIVE WHITE PAPER SERIES: Identifying Application Performance Risk 7. The schedule of performance tests uses this application? What is the behavior of the users’ workload? In an analysis, we may ask: What would be the Applications will undergo performance testing at different business impact if this application exhibited performance times during their lifetime. The application can be tested problems? Is your business in a highly competitive market, before it is ever released into production; performance where dissatisfied users can switch to you competitor? testing can be scheduled for every major release; or For instance, if the response time doubled or tripled from performance testing can be scheduled based on the normal, how would the user experience degrade, and what extent of the application’s changes. would be the financial impact of service calls and shopping cart abandonment? How long would this condition be III. Risk factors elaborated tolerated and under what conditions would it be tolerated? 1. User population The matrices shown below may be used to identify and The key information to capture is: How many people use assess potential performance engineering risks, using a the application on average and during peak time? Who simple 1-2-3 scoring system: Risk Level User Population Characteristics 1: requires a performance check Figure 1: Figure title 2: depends on other extenuating factors 3: does not require a performance test Business internal: Senior management or 2 board level, enterprise-wide Business external 2 Consumers 1 1: Thousands Average number of concurrent users 2: Hundreds Peak number of concurrent users relative to 1: Application peak TPS is more than five normal business load based on month-end or times the typical TPS for the business periodic event that occurs regularly 1: If population growth doubles in less than nine months 1: A significant increase is expected due to Rate of user population growth the business winning a new contract or large client 3: No user population growth Flash or sudden spikes in concurrent users, above and beyond your current peaking factor: • Based on a new business model of broadcasting messages to your client base telling them to all come to your website 1: Upcoming flash or spike events will add to existing application workload • Currently a novelty event for many businesses, not accounted for in the system design • Eventually folded into the traditional peaking factor 4 If you have a large number of consumers that is growing, you would require performance and scalability testing. For example, flash events are a key risk area because they are occurring more frequently for businesses on application platforms that were neither designed nor tested for sudden workload bursts. 2. Application type Most businesses will rate their