Adaptive Control of Apache Web Server Erik Reed Abe Ishihara Ole J. Mengshoel Carnegie Mellon University Carnegie Mellon University Carnegie Mellon University Moffett Field, CA 94035 Moffett Field, CA 94035 Moffett Field, CA 94035 [email protected] [email protected] [email protected] Abstract least load. Typical implementations of load balancing deal with round robin balancing for domain name service Traffic to a Web site can vary dramatically. At the same (DNS) [7]. Much of existing frameworks and previous time it is highly desirable that a Web site is reactive. To research deal with inter-server connections and balanc- provide crisp interaction on thin clients, 150 milliseconds ing, with little or no focus on dynamic parameter adjust- has been suggested as an upper bound on response time. ment of individual servers. Unfortunately, the popular Apache Web server is lim- Focusing on an individual Web server rather than a ited in its capabilities to be reactive under varying traf- cluster of machines, we investigate in this paper an alter- fic. To address this problem, we design in this paper an native approach to ensuring quick and predictable user adaptive controller for the Apache Web server. A modi- response, namely feedback control and specifically Min- fied recursive least squares algorithm is used to identify imum Degree Pole Placement control [5]. This can be in system dynamics and a minimum degree pole placement an environment where inter-server load balancing is al- controller is implemented to adjust the maximum num- ready in place, or when a single server is handling Web ber of concurrent connections. Experimentally, we show traffic. Our interest is in the ability to manipulate param- that the controller effectively regulates the reply time of eters online to control server load in order to maximize HTTP connection requests, and hence provides reactive the efficiency of each server individually under varied response, by limiting the maximum number of connec- and changing conditions. This paper uses the Apache tions accepted by an Apache Web server. Web server,1 a popular open source package [13]. Apache consists of a structured pool of workers, each 1 Background its own process.2 A master process, the Apache daemon, listens for requests and delegates HTTP communications Abrupt spikes in the usage of certain phrases (“lipstick to the worker processes. To control the rate of incoming on a pig,” “our entire economy is in danger,” “who is the connections, an Apache server administrator can adjust real Barack Obama”) in the daily news cycle have been two primary settings: MaxClients (MC) and KeepAlive observed [14]. Corresponding to such spikes, as well as (KA). Depending on server load and Web activity, the other time-varying phenomena, traffic to a Web site can Apache daemon maintains up to MC worker processes, vary dramatically. In spite of such varying traffic, it is which can in one of three states at any time: busy, idle, highly desirable that a Web site is reactive. To provide or thinking. Busy workers are currently processing a re- a crisp interactive experience on thin Web clients, 150 quest and awaiting a reply; thinking workers are wait- milliseconds has been recommended as an upper bound ing for an HTTP request after an established connection; on response time [22]. In fact, quick and reasonably pre- and idle workers are waiting for a new connection re- dictable user response is essential in all computer sys- quest. The KA parameter determines how long work- tems, not only Web services. ers maintain their current connection with a client be- One way to ensure quick and predictable user response fore terminating the connection (i.e. changing their state is through load balancing [7]. A single server’s ability from thinking to idle). Apache 2.2 has default settings of to service clients is bound by multiple factors, includ- MC = 150 worker processes and KA = 5 seconds. ing CPU utilization, memory capacity, bandwidth capac- 1http://httpd.apache.org/ ity, and I/O rate. Load balancing at the cluster level 2This is on a Unix-based system. Apache on Windows, which we allows connections to be distributed to servers with the do not investigate here, is different. 1 This paper uses performance from the user’s perspec- WAGON (Web Traffic Generator and Benchmark) [15] tive as a metric and input to the controller, rather than was used to simulate user generated traffic and vary- CPU- and memory-based metrics. In contrast to previ- ing amounts of I/O, memory, and CPU intensive traffic ous research (Section 2), our experiments include metrics [8]. Open source Web traffic generators include Apache such as time to connect, reply time, and requests per sec- Benchmark and Httperf. Httperf, which we adopt, has ond. With these in mind, we derive a model in real-time been shown to be effective in measuring Web server per- and adaptive controller (Section 3). We then perform pre- formance as well as mimicking user behavior [18]. liminary server benchmarking, an open loop simulation, and lastly test the controller using mean HTTP reply time as a setpoint (Section 4). While we use reply time as the 3 Algorithms and Analysis output modeled by the system, this approach can be ap- The underlying control approach taken is the minimum plied using any other metric collected by the server, such degree pole placement (MDPP) control design proposed as throughput, connection error frequency, or mean time by Astrom [5]. In this approach, feed-forward and feed- until a connection is established. back controllers are designed to force the open-loop sys- tem to follow a reference model. In the ideal case, 2 Related Work model following is achieved perfectly. However, due to parametric uncertainty, this is not achieved in practice. Apache Web server load has previously been modeled This is the motivation to introduce reconfiguration to the as a linear multiple-input multiple-output (MIMO) sys- control problem. During a change in plant dynamics, tem [8], using input parameters KA and MC and output the control effectiveness changes, requiring a larger or parameters memory and CPU usage. This MIMO model smaller actuator signal to maintain desired performance. expands on previous work of single-input single-output (SISO) controllers in computing systems – in particular 3.1 Plant Modeling for congestion control [10] and controlling Web server delay [16]. Robertsson et al. used a PI controller and There are several approaches to the modeling of com- nonlinear model with simulations using Matlab [20]. Ab- putation for control applications. Here, we use linear delzaher et al. explored average requests delay as a set- Auto-Regressive modeling with eXogenous input or lin- point using a feedforward predictor [1–3]. ear ARX. Nonlinear approaches, such as discrete time Another SISO controller using KA and MC as parame- neural networks, may also be used. Nonlinear modeling ters has been researched [11]. Notably, the authors con- is more complex yet may be able to capture inherent non- cluded that SISO is insufficient to obtain accurate mod- linear behavior otherwise unaccounted for in ARX mod- els for CPU, as KA and MC are not independent. The use eling. On the other hand, linear modeling is generally of CPU, memory, KA , and MC was further investigated simpler to understand and implement. by Hellerstein and Diao, along with a queueing model We will assume a finite class of plants P(i) for i 2 to predict server response time [9, 12]. They proposed a [1;M]; the integer i denotes a specific plant or device, MIMO implementation able to capture the interactions of such as a particular server type or configuration. Sup- KA and MC for modeling memory and CPU usage [11]. pose the plant is described by an ARX model with an To control KA and MC, multiple SISO controllers (with additive noise term. For simplicity, we assume Gaussian separate controls loops for both KA and MC) were very noise. Denote P(i), y(i)(k), u(k), and h(k), the plant op- effective when compared to a single MIMO controller. erator, scalar output, input, and noise at sample time k, Also of interest is their averaging of data points over time respectively. before performing controller calculations. The relationship between these quantities is given by: A simple, first order linear time invariant MIMO model was created [8] using inputs KA, MC and outputs y(i)(k) = P(i)u(k) = f T (k − d)q (i) + h(k) (2) CPU, MEM: where f T (k − d) denotes the regression vector and con- CPUk+1 CPUk KAk sists of a tapped delay line of input and output measure- = A + B ; (1) (i) MEMk+1 MEMk MCk ments, and q denotes a vector of parameters corre- sponding to the ith plant. A number of methods exist × where A;B 2 R2 2 were estimated via least squares re- to estimate q (i) in both batch and on-line modes. Similar gression and k represents a discrete time interval. ARX models have been used in modeling a number of Various methods of traffic generation for testing per- digital processes [11]. In the following we will drop the formance of Web servers have also been previously superscript (i) from the plant parameter vector. The pur- researched. A closed source traffic generator termed pose of the superscript was to indicate that there exists 2 a family of (unknown) parameter vectors that adequately A recursive version, derived by Bodson [6], is given as model various plant scenarios described above. The pur- T pose of the system identification algorithm described be- q(t + 1) = q(t) + P(t + 1)x(t + 1) y(t + 1) − x (t + 1)q(t) low is to track and identify these parameter vectors that +mlP(t + 1)(q(t) − q(t − 1)): may change gradually or abruptly during online control.