Load testing fundamentals Colin LEVERGER, E.N.S.S.A.T. LANNION [email protected] his paper will provide an overview of users can be simulated by computers to browse ser- one of the main test specialties, known vices and to follow classical user paths. The goal is Tas the load testing. It will give a pre- to observe the behavior of the servers under this ar- view of the methodologies and tools used by tificial generated load, and to ensure that the service professionals of this domain. will not shut down under any circumstances. There are several ways to test a system. Seven of the mains will be listed below [Smar]. 1 Introduction 1. Unitary Test: validation of all functional aspects Load testing is becoming a major preoccupation for of the website (∼ 5 mins) big companies, since the huge expansion of the web and the evolution of uses. Everyone wants to be 2. Component Test: validation of precise con- connected at any time, to use the internet for various text/use cases (∼ 25 mins). reasons (buy, sell, communicate. ). As a company, it is important to maintain a high quality of service 3. Capacity Test: search of the limits of the system, if you do not want to lose any client. A recent report in terms of the number of simultaneous users, says that 40% of shoppers abandon a website that network use, etc. (few hours). takes more than 3 seconds to load [Kiss]. Even more 4. Soak Test: validation of performance during important, a one second page delay in a website very long lapse of time (from 4 to +12 hours). making $100,000 a day could potentially cost $2.5 million in lost sales every year [Kiss]. 5. Stress regime Test: validation of websites with The load testing field represents all the methods several load peaks in a relatively short lapse of used to benchmark a website/software. The goal is time (few hours). to be sure that it will work properly under a high load (with a lot of people browsing the service at the 6. Redundancy Test: validation of load reparti- same time). tion between several equipment (from 2 to +12 We will discuss here about context, methods and hours). tools used to do a load test. 7. Fail-over Test: validation of service behaviors in case of crashes (from 2 to +12 hours). 2 What is load testing? Note that the tests listed above have to be exe- When we talk about load testing, we talk about meth- cuted in theory in this exact order. Indeed, if the ods to simulate load on a service/website. Virtual Unitary Tests are not validated, it is not wise to Page 1 of4 go ahead|you should probably adapt your product 4 Tools before continuing. There are plenty of tools to execute load tests, and choosing one could be difficult. There are some points 3 Methodology Followed by Load to considerate before making a choice: Testers • Budgets given to test team, There are quite a few steps before the tests execution • Prices of tools, themselves. Most of the time, in big companies, the need is written in a specification paper, given by • Number of virtual users to simulate, the project's owners. This one can be very precise, or not: anyway, it should be before starting the • Centralized/decentralized architecture, realization because it will be the guideline during the • Duration of the test campaign, entire process. If it is not precise enough, the tester himself should imagine practical solutions to test the • Special needs for tests. considered system, by developing the given drafts. When the specifications are clarified, it is time to 4.1 Features of tools think about the test platform. As a matter of fact, it is forbidden to execute tests on the production The tools used by load testers are basically simulat- platform, because there are big chances to break ing load on servers. It is possible to program tests, the service (and to lose clients!). The test platform configure details, and to launch the virtual users will be an exact replica of the production platform browsing with them. (machines, infrastructure, filer. ). The market of load testing tools is huge, and plenty Then, it is important to choose and install the of products are available. Most of them offer basic right adapted tools. To learn more about tools, see features, such as the possibility to choose the num- section4. ber of virtual users to simulate, to give a target of After all these previous preparation steps, the tests several different servers, to code different paths for should now begin. In this part, it is important to authorized/unauthorized users, to monitor the test follow the guidelines defined in the first step. These and create graphs. Almost every load testing tool guidelines could contain: allows to use injectors; the injectors are separated machines which are slaves of the main test computer. • Use cases, They are used to simulate more virtual users. In- deed, load testing is consuming a lot of memory and • User paths to follow, we often need more than 50 machines to simulate thousands of users1. • Credentials, In another hand, some products offer exotic fea- tures. It can be possible to watch answers time for • Details about sensible application parts. specific language (Java, PHP, . ), to chain different tests together, etc. Theses features are sometime The tester should be aware of all these details, offered by free and open source projects, but most execute the tests and store results to analyze them of the time, if the need is very specific, it could be carefully. Of course, the final step of the test cam- found on expensive solutions. paign consists in writing a report, which will explain Some projects are using different tools for the same the problem(s) found on the application, the proce- tests campaign, but it is definitely better to try to dure followed during tests to the client and the rec- fit all the needs with only one product. ommendations to improve the production platform performances according to the clients' use. This doc- 4.2 Record the test scenarios ument is critical, because it will be the entry point to improve the application. Scenarios have to be recorded in order to replay Note that the entire process described above is them with thousands of virtual users. To do so, the iterative, and could be executed more than once for 1For example, with a computer with only 8 gigs of memory, a more complex campaign. It is still a good habit to it is barely possible to simulate 100 virtual users before improve the quality of the tests iteratively. having a problem on the computer. Page 2 of4 tools chosen will in most of the cases be used as a • Gatling is not distributed; it is possible to lunch proxy through which all the requests will pass and multiple Gatling on different hosts but it is fully be analyzed. manual. In other words, no injectors on Gatling! A typical scenario will be a user path; this user can do actions such authentification, page browsing, • Gatling is totally asynchronous when JMeter is etc. All the actions done by this particular user will synchronous. For JMeter, the paradigm One be recorded, with the exact tempo included. user = one thread is applied, which is not the It is possible to record as many scenarios as there case with Gatling. is user paths. It is then possible to assign a certain • Gatling produces a fancy report at the end number of users to a specific scenario, with a bunch of every test, and this dynamic report (full of of possible customizations. HTML/JavaScript) is easy to read. JMeter only produces graphs (note that it could be possible 4.3 Open Source to find a module for that but it is not native). The main benefit in using an Open Source product is that it is often totally costless. The high community 4.3.3 What about cloud solutions? around this kind of product is usually reliable, and it can be a good help if there are problems with the The two solutions cited above are software that software. should be installed on physical machines. We are I will now introduce some of the greatest free and living on a Cloud era, and a lot of people have un- open source load testing tools. derstood that it is interesting to have access to load testing functionalities on demand via a web service. Most of the Cloud solutions are based on JMeter; 4.3.1 JMeter they are just simplifying the process of installing First, one of the worldwide most used tools is JMeter. injectors, configuring JMeter by providing beautiful This tool is a pure java application designed to load web interfaces. Most of theses solutions are commer- test functional behavior and measure performance. It cial. was originally designed for testing Web Applications We can mention BlazeMeter or Octoperf which but has since expanded to other test functions [Apac]. are ambassadors of this kind of solution. JMeter is able to test a lot of different servers/protocol types, is multi-threaded, and has a 4.4 Commercial Applications java GUI which facilitates the configuration of the tests and is highly extensible. Indeed, it is possible The use of commercial solutions is sometimes com- to enrich it with modules, developed by you or by the pulsory, because these solutions are often proposing community. JMeter is distributed, which means that a lot more parameters than open source solutions. it can be used with multiple injectors.