WALTy: A User Behavior Tailored Tool for Evaluating Web Application Performance
G. Ruffo, R. Schifanella, and M. Sereno R. Politi Dipartimento di Informatica CSP Sc.a.r.l. c/o Villa Gualino Universita` degli Studi di Torino Viale Settimio Severo, 65 - 10133 Torino (ITALY) Corso Svizzera, 185 - 10149 Torino (ITALY)
Abstract ticular, stressing tools make what-if analysis practical in real systems, because workload intensity can be scaled to In this paper we present WALTy (Web Application Load- analyst hypothesis, i.e., a workload emulating the activity based Testing tool), a set of tools that allows the perfor- of N users with pre-defined behaviors can be replicated, mance analysis of web applications by means of a scalable in order to monitor a set of performance parameters dur- what-if analysis on the test bed. The proposed approach is ing test. Such workload is based on sequences of object- based on a workload characterization generated from infor- requests and/or analytical characterization, but sometimes mation extracted from log files. The workload is generated they are poorly scalable by the analyst; in fact, in many by using of Customer Behavior Model Graphs (CBMG), stressing framework, we can just replicate a (set of) man- that are derived by extracting information from the web ap- ually or randomly generated sessions, losing in objectivity plication log files. In this manner the synthetic workload and representativeness. used to evaluate the web application under test is repre- The main scope of this paper is to present a methodol- sentative of the real traffic that the web application has to ogy based on the generation of a representative traffic. To serve. One of the most common critics to this approach is address this task, we implemented WALTy, a set of tools that synthetic workload produced by web stressing tools is using Customer Behavior Model Graphs for characterizing far from being realistic. The use of the CBMGs might be web user profiles and for generating virtual users web ses- useful to overcome this critic. sions.
1.1 Related Works
1 Introduction The literature on workload characterization of web ap- plication is quite vast and therefore it is difficult to provide One of the main important steps in Capacity Planning here a comprehensive summary of previous contributions. is performance prediction: the goal is to estimate perfor- In this section we only summarize some of the approaches mance measures of a web farm under test for a given set that have been successfully used so far in this field and that of parameters (i.e., response time, throughput, cpu and ram are most closely related to our work. utilization, number of I/O disk accesses, and so on). There Benchmarking tools are intended to evaluate the perfor- are two approaches to predict performance: it is possible to mance of a given web server by using a well-defined and use a benchmarking suite to perform load and stress tests, often standardized workload model when performance ob- and/or use a performance model. A performance model pre- jectives and workloads are measurable and repeatable on dicts the performance of a web system as function of sys- different system infrastructures. The most important exam- tem description and workload parameters. The models can ples are SpecWeb99 [3], Surge [10], TPC-W [16], WebStone be either simulative or analytical. By using these models, [5] and WebBench [4]. performance measures such as response times, throughput, Load testing tools, instead, evaluate the performance of a disk storage and computational resource consumption, can specific web site on the basis of actual users behavior, they be derived. These measures can be used to plan an adequate capture real user requests for further replay where a set of capacity for the web system. On the other side, benchmark- test parameters can be modified. Httperf [18] developed at ing and stressing suites are largely used in the industry for Hewlett-Packard, WebAppLoader [21], openSTA [2], Mer- testing existing architectures with expected traffic. In par- cury Load Runner [1], S-Client [9] and Geist [14] are some
Proceedings of the Third IEEE International Symposium on Network Computing and Applications (NCA’04) 0-7695-2242-4/04 $ 20.00 IEEE interesting proposals. Entry main entrances of the site Browse static (sometimes with volatile content) All applications mentioned above may differ on the char- pages containing advertising information, acterization of the request stream: SpecWeb99, WebStone, on-line catalogues, latest news, . . . Registration user enters his own reserved area or he WebBench, TPC-W use a file-list based approach, Surge is registers herself by means of dynamic an analytical distribution driven tool where mathematical pages (e.g., cgi-bin, php scripts, and so on) Download registered user consults the list distributions represent the main characteristics of the work- of available resources and downloads them load. Finally, httperf, Geist and OpenSTA, use a trace-based Exit user leaves the site, navigating elsewhere or closing the connection; or an hybrid approach (see Section 3 for an approach clas- sification). Table 1. CBMG states of a generic service Some stressing tools, e.g. S-Client, do not aim to provide a realistic workload characterization. Moreover, bench- marking tools like SPECWeb99, WebStone, WebBench pro- duced in Section 3, and WALTy tool is shown as composed vide a limited characterization and others attempt to pro- of two main modules: CBMGBuilder aims at the extraction vide a realistic workload at least for a specific class of web of CBMGs from log files; CBMG2Traffic is the traffic gen- site. For instance, TPC-W specifies an e-commerce work- eration module, that produces a trace-based synthetic work- load that simulates customers browsing the site and buy- load on the basis of the given CBMG descriptions of the ing products. Finally, characterization in some tools, e.g. user sessions and using a modified version of httperf tool. httperf or Geist, greatly depends on a set of configuration This module implements and extends the original proposal parameters. described in [8], where a given CBMG was used to assist The goal of improving web performance can be achieved the analyst to select an existing session. More realistically, if web workload is well understood (see [20] for a survey WALTy uses a CBMG to generate many different sessions of WWW characterizations). The nature of web traffic has characterized by the same profile. In Section 4, a set of ex- been deeply investigated. The first set of invariants is pro- periments will be presented. Section 5 concludes the paper posed in [7], and in [13] a discussion about the self-similar outlining some future developments. nature of the web traffic is presented. In [17] two important models for the characterization of 2 The Customer Behavior Model Graph user sessions are introduced: the Customer Behavior Model Graph (CBMG) and the Customer Visit Model (CVM).Un- The CBMG is a state transition graph proposed for work- like traditional workload characterizations, these graph- load characterization of e-Commerce sites. In this paper we based models are specifically introduced for e-commerce show that such a model may have a wider field of applica- sites, where user sessions are represented in terms of nav- tions. We will use a CBMG to represent every session, and igational patterns. These models can be obtained from the in a second phase a clustering algorithm is used to gather http logs available at the server side. together identified sessions. This is useful for obtaining a One of the novelties of the tool presented in this paper is representation of a user profile, corresponding to group of that it uses a characterization of user behaviors, i.e., the users with similar navigational patterns. Such models can CBMGs not only for e-commerce sites but also for other be used for analyzing e-business services [17], for capacity types of http servers. In this case the main problem that planning tasks, or also for replicating representative work- we need to address is the extraction and/or the classification load for the given site, when a stressing test is going to be of information from those that can be collected from http performed [8]. logs. Moreover, CBMG states are clearly defined for an e- A CBMG is a graph where each node represents a possi- commerce site (e.g., Home, Exit, Browse, Add to Basket, ble state. For the sake of simplicity, a state can be defined Pay, as discussed in [15]). On the contrary, for a generic as a collection of (static or dynamic) pages, having similar service, definition of states is somehow not trivially gener- functionalities. For example, if the site under test provides a alizable. As a consequence, CBMG states must be defined public area about the company’s activities, and a download ad hoc when needed. area is reserved to registered users, each (static or dynamic) page can be assigned to one of the states in Table 1. 1.2 Outline and Contribution of the paper Such states must be defined by the analyst, because sites may have different purposes, and a functional categoriza- In this paper, we focus on a particular application field tion of different pages must be performed by an human ex- of web mining [11], which is Capacity Planning and Web pert. Of course, the number of states can degenerate in the Performance. In particular, in the next section, we present number of existing pages (i.e., a state for each page), but, in how a CBMG for each user profile can be extracted from log general, the number od states should be limited. files and other available data. The whole framework is intro- Existing links between pages belonging to different
Proceedings of the Third IEEE International Symposium on Network Computing and Applications (NCA’04) 0-7695-2242-4/04 $ 20.00 IEEE states, are represented like graph transitions. The weight into a session-oriented format. If we have informa- associated to each arc in the graph represents the transition tion extracted from log files in the Common Log For- probability (Figure 1). mat, a session can be only characterized by a sequence Formally, a CBMG is represented by a pair (P, Z) of n× of requests from the same IP-address. If there is a n matrices, where n is the number of considered states, P = pause longer than a certain amount of time (on aver- 3 [pij ] contains the transition probabilities between states, and age 30 minutes , depending upon the web site) since Z =[zij ] represents the average server side think times a request from the same IP-address and the next, then between states1. A CBMG model is highly scalable: the the session is considered closed. After this process, a Request Log File can be generated, where each session 0.75 0.1 is identified by an unique ID and it is represented by a Browse Browse sequence of page-requests. This session identification 0.8 0.2 0.15 0.3 0.2 0.1 0.7 is, of course, not really representative, and a succes- Registration/ 0.7Registration/ 0.2 Entry 0.2 0.8 Exit Entry Exit sive analysis is awkward. But if session information Login Login 0.7 is available, session tracking techniques can be used as (a) Download 0.6 Download 0.4 (b) well. In Section 3.1 our session identification approach is explained in more detail. Figure 1. CBMGs representing two typical 3. Transforming Sessions: Let us assume that n is the users of the given service: (a) an occasional number of different states. For each session S,wecre- visitor, (b) a registered user ate a point XS =(CS,WS),whereCS =[cij ] is a n × n matrix of transition counts between states i and j,andWS =[wij ] is a n × n matrix of accumulated number of states changes from site to site and it can be fixed server-side think times; before generating the model from log files. In principle, the
number of states can change also when a site is modified, as 4. CMBGs clustering: from all XS points, a number of well as the definition of the states. In these cases, the model CBMGs is generated. A K-means clustering algorithm can be regenerated; however, this happens only when the is adopted in order to find a relatively small number site is strongly modified and this modification influences the of CMBGs that models different sessions with similar entire site structure. Normally, modifications affect only a navigational patterns4. (set of) page(s): we need to upgrade only the corresponding state definition, which is made of a list of pages. As pointed out in [15] (where algorithm seen above are This workload characterization is built by using two explained in detail), the CBMG gives other useful informa- (Web Mining) phases: (1) pre-processing, and (2) cluster- tion. One of the most important is the average number of ing. In the following, we use a fine-grained description, fo- visits Vj to each state j. This value can be calculated by cusing on the pre-processing and the clustering phase. As a solving the following equations: consequence, four different sub-processes are described: V1 =1 1. Merging and filtering web logs: a Web site may be n composed of several servers. All the log files in the Vj = Vk × pkj cluster are merged together. Moreover, log entries not k=1 influencing the identification of user sessions, can be −→ V filtered out (for instance, requests of embedded ob- Other parameters can be derived from vector : average 2 session length, is the average number of states visited by an jects, as images, video clip, and so on) . n−1 user for each visit: k=1 Vk. 2. Getting sessions: log files contain request-oriented in- Moreover, each Vi is an interesting metric, because it repre- formation. During this phase, data are transformed sents the ratio between the number of visitors that entered the i state, w.r.t. the total number of visits. For example, 1In a server log file, a transition can be detected as two successive page-requests of the same user session. The server think time is defined 3According to IFABC Global Web Standards (www.ifabc.org) an User as the difference between corresponding consecutive arrival times, minus Session (or a Visit) is a “series of one or more Page Impressions, served to the server execution time. one User, which ends when there is a gap of 30 minutes or more between 2Resulting CBMGs will be used in WALTy to emulate representative successive Page Impressions for that User.”. traffic. A stressing tool should generate synthetic load due to images, video 4In literature there are many Pattern Recognition and Clustering algo- clips and other embedded objects requests. However, embedded objects rithms that can be used for such goal, and a further research study should can be filtered out during data cleaning. In fact, our tool will make a request be conducted in order to select the solution that best fits this domain. In for each embedded object, when the traffic generator asks for a given web our tool, we implemented a basic K-means methodology as proposed in page. [15].
Proceedings of the Third IEEE International Symposium on Network Computing and Applications (NCA’04) 0-7695-2242-4/04 $ 20.00 IEEE in a e-Commerce site, the buy to visit ratio is equal to Vp, In general, stressing clients replicate a (set of) artificial ses- 5 where p is the given Pay state ; i.e., Vp is the ratio between sion(s) or navigational patterns. Depending on the given the number of customers who effectively buy from the Web stressing tool, virtual users are assigned to: store and the total number of visits. 1. randomly generated user sessions; 2. manually generated user sessions; 3 A Framework based on Representative Workload Generation 3. user sessions extracted from log files. Relevant drawbacks to these approaches are, respectively, Many methodologies, techniques and tools have been de- (1) impossibility to generate a realistic workload, (2) sub- veloped to support capacity planning and performance pre- jectivity, (3) lack of scalability. An acceptable solution diction. The given approach involves performing load, sta- might be to provide every single VU a profile. Such a pro- bility and stress test using workload generators to provide file, can be represented by means of a CBMG. a synthetic workload: web stressing tools use scripts de- In a common trace-based stressing framework, scripts scribing user sessions to simulate so called Virtual Users are generated to describe the behavior of each VU. These (VU) browsing the web site. Four metrics are used to eval- scripts can be used to emulate a browser, sending a server uate the web site performance (requests served per second, sequences of GET (or POST) requests for pages and embed- throughput in bytes per second, round-trip time, errors) and ded objects. Between successive requests, the virtual user many parameters can be measured to detect system bottle- can be configured to wait a given interval of time (client necks (e.g., CPU and RAM utilization, I/O Disks access, think time). When scripts (i.e., virtual users navigational Network traffic). There are two kind of benchmarking ap- patterns) are defined, the given stressing tool activates a load proaches: the absolute benchmarking (or just benchmark- generator module. It provides means to generate a synthetic ing) approach differs from application benchmarking (or workload according to the recorded scripts. just load testing [6]) in the workload model used to test A stressing client is a load generator that runs a test; the architecture: since the scope is to provide comparable during a test session, stressing clients execute one or more results for different platforms (hardware, operating system, scripts by means of each VU. We can assign many VUs to supporting middleware) the generated workload will always each script, batch them in time and number according to be the same, provided it is scalable enough to cope with our needs of analysis and measurement. Moreover, each systems ranging from single processor machines to cluster stressing client includes a performance collector for gath- of servers (i.e., Web Farm). Conversely, the aim of load ering the client’s performance parameters and it can be in- testing is to highlight bottlenecks and points of failure and tegrated with a SNMP agent for sending measures to the the workload should be generated accordingly. One of the master stressing client. In other words, each stressing client most common objections to the approaches developed, is outputs a synthetic http workload which reflects the given that synthetic workload produced by web stressing tools is scripts. This workload is replicated for each VU assigned far from being representative of the real traffic. Three main to the given stressing client. approaches can be identified for the characterization of the The test can be distributed to a plurality of stressing request stream: clients, that under the master stressing client control, can launch generated http Workload against the targeted web • trace based: the characteristics of the Web workload farm via the Internet or a LAN. is based on pre-recorded trace logs (e.g., LoadRunner The master stressing client usually includes an SNMP by Mercury Interactive, OpenSta [2], httperf [18], We- manager, and it collects the performance measurements of bAppLoader [21]); the system under test and of the other stressing clients via SNMP protocol. Finally, it produces a report of perfor- • file list based: the tools provides a list of Web objects mances observed during the test. with their access frequencies. During workload gener- Httperf [18] can be used in such a scenario, and we inte- ation, the next object to be retrieved is chosen on the grated it in our tool. In the following, we describe the fun- basis of its access frequency (e.g., SpecWeb suite [3]); damental steps of our tool: (1) How CBMG are extracted from log files and (2) How CBMGs are used to generate • analytical distribution driven: the Web workload char- httperf virtual users behaviors. acteristics are specified by means of mathematical dis- tributions (e.g., Surge [28]). 3.1 CBMGBuilder: from log files to CBMGs
5In an e-Commerce domain, possible states are: Entry, Browse, Search, In this section we focus on the first component of Add to Cart, Select, Pay and Exit, as described in [15]. WALTy: the creation of CBMGs from input data. Because
Proceedings of the Third IEEE International Symposium on Network Computing and Applications (NCA’04) 0-7695-2242-4/04 $ 20.00 IEEE a Customer Behavior Model Graph is a session-based rep- string that uniquely identifies a session. If the source resentation of a user navigational pattern, session identifi- is a log file in the standard Common Log Format, this cation from data is a central topic, because web logs format field is not directly available; instead, we have the IP is inherently hits-oriented. address and, as previously discussed, the application In general, server side logs include client IP address (ma- of a session-identification heuristic is necessary for chine originating the request, maybe a proxy), user ID (if reproducing a realistic situation. On the other hand, authentication is needed), time/date, request (URI, method if the source is a servlet filter, the session can be and protocol), status (action performed by the server), bytes identified correctly without further analysis. The transferred, referrer and user agent (operative system and timestamp relates to the moment when the request browser used by the client). As this information is uncom- is received by the web server, whereas the request plete (hidden request parameters using the POST method) contains the required resource. Finally, the execu- and not entirely reliable, these information should be inte- tionTime is an estimation of the time that the server grated using packet sniffers and, where available, applica- spends to complete the request; if this information is tion server log files. Depending on the data actually avail- not available, (i.e., when the source is in the standard able for the analysis, typical arising problems in user behav- Common Log Format), we assume a null value. ior reconstruction include: multiple server sessions associ- ated to a single IP client address (as in the case of users ac- • cessing a site through a proxy); multiple IP addresses asso- Kernel-Module scheme implementation: Such as in the ciated to multiple server sessions (the so called mega-proxy object-oriented paradigm, we need a kernel-module problem); user accessing the web from a plurality of ma- scheme. Let the kernel be a core module implement- chines; a user using a plurality of user agents to access the ing an abstract service (e.g. translation of an undefined Web. Assuming that the user has been identified, the asso- input format to our data model); when we need to deal ciated click-stream has to be divided in sessions. The first with a new instance of the service (e.g. translation of relevant problem is the identification of the session termi- the Common Log Format to our data model), a new nation. Other relevant issues are the need to access appli- module implementing the given feature is just added 6 cation and content server information and the integration to the kernel .
with proxy server logged information relative to cached re- Kernel
sources access. KernelClass
ing, and so on. Unluckily, no one of these proposals is Module clearly superior than the others, and the definition of a new ImplModule ImplModule2 standard is missing. Moreover, logging procedures are not + translate(): void + translate(): void uniformly defined. In such a domain, it is important to de- fine a scalable and integrated methodology to session iden- tification. This methodology should be scalable in order to Figure 2. Kernel-Module scheme handle with any new data source without modifying the core application or rewriting and recompiling the code. As a sec- ond important feature, this approach should integrate many Figure 2 shows how Kernel class manages the abstract different input formats in a common framework. Last, new method myMethod.Itusestranslate method declared schemes and formats should be easily included in this envi- in InterfaceModule interface. But, translate method ronment, when they appear to the analyst. is implemented in different modules, e.g. ImplModule Proposed methodology is composed of the following and ImplModule2, according to the requirements of the steps: specific application. These classes are defined as plug- ins for the application. • Data model definition: an abstract data model is A plug-in performs the simple task of transforming the defined. This includes any specific information format of a generic source file into the data model pre- necessary for a correct execution of the implemented viously defined (e.g., CIF). application. Obviously, it is strongly dependent on • the particular domain of interest. In such context, we Plug-in implementation: when a new file format is en- defined the CBMG Input Format (CIF) that is made countered or a session identification technique is intro- of the following fields: sessionID, timestamp, request, 6For example, the Java language uses this mechanism to implement the executionTime.ThesessionID is an alphanumeric Swing graphic library or all the operations involving Collection objects.
Proceedings of the Third IEEE International Symposium on Network Computing and Applications (NCA’04) 0-7695-2242-4/04 $ 20.00 IEEE duced, a new plug-in must be implemented, in order to 3.2 Generating Representative Web Traffic from handle the conversion to the defined data model. This CBMG plug-in is added to the kernel module as previously dis- cussed. When a CBMG is generated for each user profile, the next step in the emulation process is traffic generation. As This process permits a simple and practical management of in the general trace-based framework previously described, several session identification mechanisms, shifting the for- traffic is generated by means of a sequence of http-requests, mat conversion problem at implemented plug-ins. CBMG- with a think time between two successive requests. This Builder module is implemented merely using input file with Section describes how to generate such a sequence from CIF syntax and allow the creation, installation and deletion CBMGs. of a plug-in. Let us suppose that the clustering phase returned m pro- CBMG building process is made of other fundamental files {Φi},wherei =1, ..., m. Each profile is a CBMG steps, in addition to session identification: defined as a pair (P, Z) of matrices n x n,wheren is the • States definition: as described in Section 2, a state is number of states (see Section 2). Observe that each profile a collection of (static or dynamic) pages, having sim- Φi, corresponds to a set of sessions {Si1,Si2, ..., Sip}.Let ilar functionalities. The problem is to map a physi- us indicate the cardinality of this set of sessions with |Φi|. cal resource into a logical state. CBMGBuilder let Moreover, let us define the Representativeness of profile Φi, the user define a state by means of a set of rules. as the value: Each rule has three main components: a directory,a |Φi| ρ(Φi)= , resource name and an extension. Simple regular ex- m (|Φ |) pressions can be adopted as well. For instance, state i Browse can be defined by means of the following rules: i=1 {information/*.*,*/*.html}, that matches with all the which is the rate of the number of sessions corresponding files contained in directory information, and all the to profile Φi, w.r.t., the total number of sessions. files with extension .html. When generating traffic to our system under test (SUT), pro- files Φi are used to properly set up the behaviors of virtual • Embedded objects definition: in the pre-processing users. In Figure 3, a set of different clients is used to run phase (see Section 2), embedded objects should be fil- several clusters of virtual users, that are defined by means tered out. In WALTy, embedded objects are defined by of different profiles. a set of rules and using the syntax adopted for states { } definition, e.g., */*.gif, */*.jpg, */*.swf . WALTy clienti WALTy clientj
profile Φ profile Φ • General parameters specification: in order to create a i j set of CBMGs that represents the behavior of a user’s web site, WALTy allow the user to specify the follow- ...
ing parameters: ρ Φ ρ Φ N* ( i) sessions N* ( j) sessions – number of clusters: each CBMG models the nav- stats igational pattern of a given session. We need Workload Workload Generator Generator clusters of CBMG to model a generic user pro- HTTP file. The analyst can properly configure the num- statistics requests statistics ber of profiles (i.e., clusters). – Session Time Limit (STL): two successive re- quests from the source, should be lesser than given STL to be labelled with the same Session HTTP requests HTTP requests ID. – Session Entry Limit: it specifies the minimum SUT number of requests in a session. Figure 3. Stressing framework based on It goes without saying that the rules above must be de- WALTy clients scribed by an analyst who studied carefully the site. As a consequence, she can properly set up the number of states and their definitions, the number of clusters (i.e., user pro- Observe that value ρ(Φi) is very important to our analy- files), the rules for filtering out hits concerning embedded sis, because it gives a way to calculate a representative num- objects, and so on. ber of virtual users running with the same profiles. For ex-
Proceedings of the Third IEEE International Symposium on Network Computing and Applications (NCA’04) 0-7695-2242-4/04 $ 20.00 IEEE ample, if we want to start a test made of N virtual users ac- Algorithm: CBMGtoSession cessing the server, we can parallelize stressing clients jobs input : A profile Φi =(P, Z), and resource set L output : A session as it follows: client i, that emulates sessions with profile Φi, N ∗ ρ(Φ ) i =1, ..., m begin runs i virtual users, with . Session ←∅; WALTy allows further scalability: in fact, we can perform State ← Entry; while State = Exit do a fine-grained test, changing relative profiles percentage, ← ( ,L ) Page SelectPage State State ; e.g., we can run experiments responding to questions like PageProperties ← SetPageProperties(Page); NextState ← SelectNextState(State, P); ρ(Φ3) “what does it happens when users with profile grows ThinkTime ← in number w.r.t. other classes of users?”. As a conse- EstimateThinkTime(State, NextState, Z); quence, the framework can be generalized: client i runs Request ← CreateRequest(Page, PageProperties, ThinkTime); N ∗ fi ∗ ρ(Φi) virtual users, where 0
Proceedings of the Third IEEE International Symposium on Network Computing and Applications (NCA’04) 0-7695-2242-4/04 $ 20.00 IEEE corresponding to the transition from “old” state S to IT Area 10 Experimentation and next state N . Institutional Area activities of the Information Society Information about Multimedia Area company structure methodologies and institutional Development of all activities necessary multimedia 4 An Illustrative Case Study components of product creation IP Telephony Area CSP site In this section, we first present (some of) the measures Voice over IP research General Area that can be derived by using the tool. Figure 4 shows General informations the measures that can be derived by using the module News and Press Area about the company News and articles about company CMBG2Traffic. In particular, we can define several time activities intervals: the time required to set up the TCP connec- tion Tconn (interval [t1,tstart]) , the time to receive the first byte of the server response (interval [t2,t1]), the time Figure 5. Logical map of the web server under to receive the last byte of the server response (interval test [tend,t2]). From this we can derive the response time Tresp (interval [tend,t1])andthetransaction time Ttrans (interval [t ,t ] end start ). different CBMGs. The representativeness of these CBMG are: ρ(Φ1)=0.03, ρ(Φ2)=0.68, ρ(Φ3)=0.28,and transaction time (Ttrans) ρ(Φ4)=0.01. Figure 6 graphically shows one of these response time (Tresp) CBMGs and Table 2 reports its transition probabilities (up- per matrix) and the server think time (lower matrix). set up connection(Tconn) first byte(Tfirst) last byte (Tlast) y
t t t t
start 1 2 end lephon e ws T Entry Institutional Ne General IP IT Multimedia Exit Figure 4. Measurement intervals Entry 0 0.02 0.42 0.56 0 0 0 0 Institutional 0 0.53 0.19 0.06 0 0 0.03 0.19 News 0 0.13 0.29 0.04 0 0 0 0.54 The CBMGBuilder module provides the CBMG charac- General 0 0.03 0.9 0.06 0 0 0 0.01 IP Telephony 0 0.04 0.11 0.2 0.59 0.03 0.03 0 terization of the web application under test and it allows to IT 0 0.02 0.06 0.05 0.03 0.8 0 0.04 derive other different measures such as the average number Multimedia 0 0.06 0.03 0.06 0 0 0.72 0.13 of visits for each state of the CBMG, and the average ses- Exit 00000000 sion length expressed in terms of number of states (of the
CBMG) visited during a session (see Section 2). y In the following, we provide an example of the possi- lephon
ble investigations allowed by using the tool WALTy. We e ws T
analyze the web server of the CSP, an italian research and Institutional Ne General IP IT Multimedia development company. A simple logical map of this web Institutional 36.3 93.8 115.8 0 0 33.8 News 49.7 78 229.9 0 0 0 site is depicted in Figure 5. The map identifies in a sim- General 63.9 11.4 144.2 0 0 0 ple manner the “macro-areas” of this web server. The areas IP Telephony 185.6 505 96 28 381.4 251.2 IT 80.7 476.8 176.1 319.9 28.8 0 correspond (in most of the cases) to the different type of Multimedia 89.5 324.4 169 0 0 31.4 activities of this institution. We analyzed the web server log files collected during Φ the period from April 11-st until September 16-th 2003. Table 2. CBMG 3: Transition probabilities During the CBMG derivation phase we identify four dif- and think times matrices ferent profiles11 of web-server users that correspond to four
10Observe that, by default, the web logs give timestamps that allow to We performed a simple experiment to verify that the syn- obtain the distance from two consecutive client requests composed by these thetic workload generated by using the CBMGs matches the quantities: client think time, communication delay of network, and service peculiarities of the original workload. In particular, we col- time of server. If appropriately configured, web logs can give the value of service time. Moreover, the informations stored in web logs are inher- lected the log files obtained by using the synthetic work- ently insufficient to calculate the exact value of client think time without load, and then we compared the CBMG-based measures modifying completely the client implementation to track this value. For this reason, the solution adopted in our tool uses only the informations (i.e. 4, in our set of experiments) can be automatically calculated by al- available from web log files, i.e. the value extracted from matrix Z. gorithms (well known in the Pattern Recognition literature) that are not 11The choice of the number of clusters is up to the analyst. This value included in the present version of our tool.
Proceedings of the Third IEEE International Symposium on Network Computing and Applications (NCA’04) 0-7695-2242-4/04 $ 20.00 IEEE based approach in other cases we obtained greater values). Our experiments allow to stat the following concluding re- marks: the results that can be obtained by using web perfor- mance tools, where workload is composed by user sessions, strongly depend on the chosen virtual user behavior. The choice of the appropriate session description(s) is a delicate task. On the other hand, the approach we propose in this pa- per tries to propose a workload generation that does depend on the “real” user navigational patterns.
5 Conclusion and Future Work
In this paper, we presented WALTy, a set of tools de- signed for testing Web applications by means of user profil- ing, traffic characterization, synthetic workload generation and monitoring agents. WALTy applies the general stress- Figure 6. CBMG Φ3 has been derived from the ing methodology, but differently from other existing tools, analysis of the log file of the web server under it does not replicate to the system under test a (subjective) test randomly or manually selected testing session, but it gen- erates synthetic workload in an objective way, and in terms of the adopted user profile characterization. In other words, such a tool allows the following methodological steps: derived from the original log files with those derived by us- ing the synthetic workload. In all the cases, and for differ- 1. identification of the case study and logical analysis of ent virtual user inter-arrival distributions (exponential, uni- the site under test; form, truncated pareto), the values of the two versions of the CBMG-based measures are very closed (the differences 2. identification of logical states of the site, and conse- are less than 5%). quent mapping of the web objects (e.g., pages, php The aim of the next set of experiments is to provide an scripts, and so on) to the states; example of possible investigations that can be performed 3. analysis of the physical system (e.g., how many servers by using our tool. For each experiment we generated a in the web farm, which connection bandwidth, which CBMG-based workload with a different number of virtual computationalandmemorizationresources,...); users N and a duration of 10 minutes. The average ses- sion length of the CBMG used for the experiment is equal 4. automatic creation of Customer Behavior Model N to 3. In our experiments the number of virtual users Graphs from log files; ranges in the interval [3000-3900]. We summarized in Fig- ure 7 the obtained results. In particular, the figure presents 5. analysis of CBMGs (e.g., average session length, aver- measures Ttrans and Tresp, obtained by using the CBMG- agenumberofvisits,...); based workload. By using the Simple Network Manage- ment Protocol (SNMP) we can also access measures such 6. setting up of the test plant, i.e., replication of the sys- as CPU, memory, and disk usage, that are not reported here tem under test in an experimental environment; for brevity. 7. generation of synthetic workload to replicated system The main question that arises concerns the advantages by means of virtual user sessions extracted from CB- of using the CBMG-based method we proposed. To pro- MGs; vide an answer to this question, we perform several dif- ferent experiments with workload composed of user ses- 8. collection of monitored data and reporting. sions (for instance, sessions recorded by capturing real user navigational patterns). In particular, we used user sessions Several additional features are currently under develop- with a length equal to the average session length of the ment. In particular, we are improving measure collection CBMG used for the experiments (i.e., 3). Although we use modules to allow a more user friendly interface of WALTy. user sessions having the same length, we observed com- Moreover, we are implementing a reporting module that al- pletely different behaviors (i.e., in some cases we obtained lows the derivation of performance measures via web inter- measures smaller than those obtained by using the CMBG- face. We are also implementing a module that allows to help
Proceedings of the Third IEEE International Symposium on Network Computing and Applications (NCA’04) 0-7695-2242-4/04 $ 20.00 IEEE 7000 350 Ttrans
6000 300
5000 250
4000 200 Time (msec) Time (msec) 3000 150
2000 100
1000 50 Tfirst
0 0 3000 3200 3400 3600 3800 4000 3000 3200 3400 3600 3800 4000 N N
Figure 7. Numerical results derived for different values of N virtual users and for different measures: Ttrans (left plot), and Tresp (right plot)
the preparation of experiments that use multiple client ma- [10] P. Barford and M. E. Crovella. Generating Representative chines. Another development concerns the integration into Web Workloads for Network and Server Performance Eval- WALTy gui of a module that uses the SNMP protocol to uation. In Proc. of ACM SIGMETRICS ’98, pages 151–160, obtain measures related with the usage of server resources. 1998. The tool is made available under General Purpose Li- [11] R. Cooley, B. Mobasher, and J. Srivastava. Web Mining: Information and Pattern Discovery on the Word Wide Web. cence, and can be asked to the authors of this paper. Technical report, Dept. of Computer Science, University of Minnesota, Minneapolis, MN, USA, 1997. Acknowledgments [12] R. Cooley, B. Mobasher, and J. Srivastava. Data preparation for mining world wide web browsing patterns. Knowledge This work has been supported by the Italian Minister of and Information Systems, 1(1):5–32, 1999. Research (MIUR), within the project WebMinds (FIRB). [13] M. Crovella and A. Bestavros. Self-Similarity in World Wide Web Traffic: Evidence and Possible Causes. IEEE/ACM Transaction on Networking, 5(6), Dec 1997. References [14] K. Kant, V. Tewari, and R. Iyer. Geist: A generator for e- commerce internet server traffic. [1] LoadRunner by Mercury Interactive. [15] D. Menasce,´ V. A. Almeida, R. Fonseca, and M. A. http://www.mercuryinteractive.com/products/loadrunner/. Mendes. A methodology for workload characterization of [2] OpenSTA (Open System Testing Architecture). e-commerce sites. In Proc. of ACM Conf. on E-Commerce, http://www.opensta.org. Nov 1999. [3] Standard Performance Evaluation Corporation. [16] D. A. Menasce. Tpc-w: a benchmark for e-commerce. IEEE http://www.specbench.org. Internet Computing, May-June 2002. [4] Webbench. www.veritest.com/benchmarks/webbench/ [17] D. A. Menasce,´ A. V. Almeida, R. Fonseca, and M. A. webbench.asp. Mendes. Scaling for E-business: Technologies, Models and [5] Webstone. http://www.mindcraft.com/webstone/. Performance and Capacity Planning. Prentice Hall, 2000. [6] M. Andreolini, V. Cardellini, and M. Colajanni. Benchmark- [18] D. Mosberger and T. Jin. httperf: A Tool for Measuring ing models and tools for distributed web-server systems. In Web Server Performance. In In Proc. of First Workshop on Performance Eval- M. C. Calzarossa and S. T. eds., editors, Internet Server Performance, pages 59–67. ACM, 1998. uation of Complex Systems: Techniques and Tools, volume [19] J. Pitkow. In search of reliable usage data on the WWW. LNCS 2459, pages 208–235. Springer-Verlag, September Computer Networks and ISDN Systems, 29(8–13):1343– 2002. 1355, 1997. [7] M. F. Arlitt and C. L. Williamson. Web Server Workload [20] J. E. Pitkow. Summary of WWW Characterizations. Com- Characterization: The Search for Invariants. In Proc. of puter Networks and ISDN Systems, 30(1–7):551–558, 1998. ACM SIGMETRICS ’96, pages 126–137, 1996. [21] K. Wolter and K. Kasprowicz. WebAppLoader: A Simula- [8] G. Ballocca, P. Politi, G. Ruffo, and V. Russo. Benchmark- tion Tool Set for Evaluating Web Application Performance. ing a site with realistic workload. In Proc. of the 5th IEEE In In Proc. of TOOLS 2003, volume LNCS 2794, pages 47– Workshop on Workload Characterization (WWC-5). IEEE 62, Urbana, Illinois, USA, 2003. Springer-Verlag. Press, November 2002. [9] G. Banga and P. Druschel. Measuring the capacity of a web server. In USENIX Symposium on Internet Technologies and Systems, 1997.
Proceedings of the Third IEEE International Symposium on Network Computing and Applications (NCA’04) 0-7695-2242-4/04 $ 20.00 IEEE