Shutdown Policies with Power Capping for Large Scale Computing Systems Anne Benoit, Laurent Lefèvre, Anne-Cécile Orgerie, Issam Raïs To cite this version: Anne Benoit, Laurent Lefèvre, Anne-Cécile Orgerie, Issam Raïs. Shutdown Policies with Power Capping for Large Scale Computing Systems. Euro-Par: International European Conference on Parallel and Distributed Computing, Aug 2017, Santiago de Compostela, Spain. pp.134 - 146, 10.1109/COMST.2016.2545109. hal-01589555 HAL Id: hal-01589555 https://hal.archives-ouvertes.fr/hal-01589555 Submitted on 18 Sep 2017 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Shutdown policies with power capping for large scale computing systems Anne Benoit1, Laurent Lef`evre1, Anne-C´ecileOrgerie2, and Issam Ra¨ıs1 1 Univ. Lyon, Inria, CNRS, ENS de Lyon, Univ. Claude-Bernard Lyon 1, LIP 2 CNRS, IRISA, Rennes, France Abstract Large scale distributed systems are expected to consume huge amounts of energy. To solve this issue, shutdown policies constitute an appealing approach able to dynamically adapt the resource set to the actual workload. However, multiple constraints have to be taken into account for such policies to be applied on real infrastructures, in partic- ular the time and energy cost of shutting down and waking up nodes, and power capping to avoid disruption of the system. In this paper, we propose models translating these various constraints into different shut- down policies that can be combined. Our models are validated through simulations on real workload traces and power measurements on real testbeds.3 Keywords: Large scale distributed systems, energy models, shutdown policies, simulations. 1 Introduction Reducing the energy consumption of large scale distributed systems (high perfor- mance computing centers, networks, datacenters) is a mandatory step to address, in order to build a sustainable digital society. Since more than a decade, several technological solutions have been made available by systems designers to help reducing power, like shutdown and slowdown approaches. The first and most explored solution consists in shutting down and waking up some resources de- pending on platform usage. In this paper, the question on how resource providers and managers can be helped to validate their constraints while reducing the en- ergy consumption using only the shutdown and wake-up of large amount of resources is addressed. Resource providers and managers can be human who are responsible of the administration of large supercomputers, but they can also be software com- ponents that deal with resources (schedulers, resource management frameworks, etc.). Nowadays, hardware components of a datacenter or supercomputer (servers, network switches, data storage, etc.) are not yet energy proportional. In fact, 3 This work is integrated and supported by the ELCI project, a French FSN ("Fond pour la Soci´et´eNum´erique") project that associates academic and industrial part- ners to design and provide software environment for high performance computing. the static part (i.e., the part that does not vary with workload) of the energy consumed for example by computing units, represents a high part of the overall energy consumed by the node. Therefore, shutting unused nodes or routers, that are idle and not expected to be used in a predicted duration, could lead to non negligible energy savings. This paper focuses on shutting down and waking up any kind of resources like servers, network devices, memory banks, cores, etc. For clarity's sake, here, the proposed models and validations focus on servers (called nodes). Off-the-shelf software eco-systems are nowadays integrating (mainly basic) shutdown policies. Data center resource managers propose techniques or hooks to configure such capabilities. For example, Slurm [16], an open-source cluster management system, introduces a SuspendTime4 that represents the minimum idle time after which it allows the node to be switched off. Then, the resource manager is responsible for deciding when to switch on and off servers. It takes decisions either based on pre-determined policy [16], on workload predictions [8], on queuing models [5] or on control theory approach [15]. Overall, shutdown seems to be an interesting leverage to save energy (referred to as OnOff leverage). But this technique cannot be applied at large scale if no constraint is respected on the target system. This is especially true if the resource providers take into account several types of constraints, such as the cost of shutdown and wake-up (in time and energy), or power-capping constraints imposed to the whole system. In particular, shutting down too many nodes could cause the power consumption to be under the minimum power capping decided with the electricity provider. Likewise, if too many nodes are waked-up, and if providers take into account the energy consumed during shutdown and wake-up sequences (which is far from being free), limits fixed by the electricity provider can be greatly exceeded. If providers do not take into account such constraints, they can put into danger machines composing the studied computing facility. In this paper, we propose several models of shutdown that can be used under actual and future supercomputer constraints, and that takes into account the impact of shutting down and waking up nodes (time, power and energy) and the Idle and Off states observed after such actions as they impact the power usage. Our formalization allows for a mono or combined usage of models in order to help resource managers and providers respect several constraints at the same time. Several shutdown models that can be handled by resource providers and that deal with infrastructure constraints are explored: • The basic models allow comparisons with several related works where shut- ting down and waking up nodes can either be free and immediate, or not allowed. • The sequence-aware models account for the cost of shutting down or waking up nodes, in terms of time or energy. • The power-capping models aim at respecting power capping requirements. 4 http://slurm.schedmd.com/power save.html The models are used as follows: knowing that there is an idle interval of length Tgap on a given node, the model decides whether the node should be shut down, given the enforced constraints. The paper is organized as follows. Section 2 presents the modeling of the various shutdown (OnOff) policies for basic models, sequence-aware models, and finally models dealing with power-capping. It also deals with the usage and combination of these models. The experimental setup is described in Section 3 and experimental results are analyzed in Section 4. Section 5 presents related work on shutdown techniques for large scale systems. Section 6 concludes and presents future work. 2 Modeling shutdown policies This section presents our characterization of the impact of shutting down and waking-up a node in terms of time and power consumption. It also introduces models acting on the OnOff leverage. 2.1 Model inputs To monitor nodes' wake-up and shutdown sequences, an external power moni- toring allowing us to trace power consumption of nodes is used. It has a rate of one power value per second. The sequences have been monitored to detect when every event happened. For the wake-up sequence, unfortunately, no information could be extracted between BIOS (Basic Input Output System) bootstrap and GRUB (Grand Unified Bootloader) loading. The first monitorable event in this sequence is the Kernel launch; this is displayed on Figure 1, which shows how the power evolves with time during a monitored boot sequence on a node. The time where kernel starts has been recovered with the dmesg tool (which is a logging of what happened during the launch of the kernel). The INIT monitoring is made by modifying the runlevel scripts. These monitored profiles are modeled by a sequence for each node. For node i, Seqi = f(t0; AvrgP0);:::; (tn; AvrgPn)g is the set of timestamps and average power consumption measurements of a wake-up (or Off!On sequence) or shut- down (or On!Off sequence), where t0 and tn represent the starting and ending time respectively of sequence Seq on node i. The length of the sequence is there- fore tn − t0, and at time-step tk (1 ≤ k ≤ n), AvrgPk is the average power consumption of node i. 2.2 Model definitions Basic models. Two basic models are used by most papers in the literature: either the nodes are never shut down (No-OnOff model), or there is no cost (time, energy, power) to wake-up or shutdown a node (LB-ZeroCost-OnOff model: Lower-Bound Zero-Cost OnOff Model), making it very simple to shut- down a node (but very far from reality). In this context, the node consumes 200 150 100 Watt 50 0 0 50 100 150 Time Figure 1: Monitored boot sequence of a node running Linux: BIOS-MBR-GRUB period in red; Kernel in green; Init in gray. nothing when executing an On!Off or Off!On sequence. Thus, there is no cost nor time spent to switch state, and no power peak observed during the sequence. In this context, no influence could be derived from waking-up or shutting down nodes. This LB-ZeroCost-OnOff model therefore provides a theoretical up- per bound on the gains that can be achieved by shutting down nodes. Sequence-aware models. The sequence-aware models make sure that the sequence observed on a node or set of nodes during On!Off or Off!On se- quences fits in time or are beneficial in energy.