Green Computing and Power Saving in HPC Data Centers

Home , Green computing

Research Papers Issue 2011 Green Computing and power saving December 2011 in HPC data centers Scientiﬁc Computing and Operation (SCO)

By Osvaldo Marra SUMMARY Data centers now play an important role in modern IT CMCC University of Salento, Italy infrastructures. Much research effort has been made in the ﬁeld of green [email protected] data center computing in order to save energy, thus making data centers 2 Maria Mirto environmentally friendly, for example by reducing CO emission. The CMCC Scientiﬁc Computing and Operations (SCO) division of the CMCC has [email protected] started a research activity exploiting proactive and reactive monitoring Massimo Cafaro techniques in order to reduce the power consumption of data centers by CMCC using Green Computing technologies. The goal of this research paper is to University of Salento, Italy [email protected] provide an overview about the main issues and challenges to improve HPC resources management in terms of power consumption. A feasibility study and Giovanni Aloisio CMCC on the possibility to introduce a system to measure the power consumption University of Salento, Italy for the CMCC data center, and in particular for the Calypso cluster, is [email protected] described. CMCC Research Papers

INTRODUCTION as to reduce the environmental impact. Techni- cal issues in high-performance green comput- 02 High performance computing (HPC) - using span the spectrum from green infrastruc- ing commodity clusters, special-purpose mul- ture (energy-efficient buildings, intelligent cool- tiprocessors, massive data stores and high- ing systems, green/renewable power sources) bandwidth interconnection networks - is rapidly to green hardware (multi-core computing sys- becoming an indispensable research tool in tems, energy-efficient server design, energy- science and engineering. In studies on cli- efficient solid-state storage) to green software mate changes, HPC is fundamental for mod- and applications (parallelizing computational eling and predicting climate changes and com- science algorithms to run on modern energy- putation is taking an increasingly important role efficient multi-core clusters, intelligent load dis- beside theory and experimentation as a tool tribution and CPU switch-off). The SCO divi- for inquiry and discovery. Beyond science and sion of the CMCC has started a research activ- engineering, large compute clusters, massive ity about the use of proactive and reactive mon- storage, and high-bandwidth connectivity have itoring techniques in order to reduce the power also become mission-critical infrastructure in consumption of data centers by using Green industrial and academic settings. While the Computing technologies. The goal of this re- costs of computers, storage, and networking search paper is to provide an overview about have fallen dramatically over time, the costs of the main issues and challenges to improve the the buildings, power, and cooling that support HPC resources management in terms of power this equipment have risen dramatically. Indeed, consumption. In particular, a feasibility study

Centro Euro-Mediterraneo per i Cambiamenti Climatici the costs for power and infrastructure exceed on the possibility to introduce a system to mea- the cost of the computing equipment they sup- sure the power consumption for the CMCC data port. The environmental costs of information center, and in particular for the Calypso cluster and communication technologies are also in- is given. The research paper is articled as fol- creasing; recent studies estimate that comput- lows. Starting from an overview of data centers, ing and communication technology sectors are several issues related to their management are responsible for 2% of global carbon emissions discussed. In particular several metrics have and these emissions are increasing at 6% an- been introduced in order to measure the power nually [1]. Green computing (also known as consumption (e.g., PUE). Moreover, several al- sustainable computing) can be broadly deﬁned gorithms and techniques are described in order as the problem of reducing the overall carbon to balance the workload on the cluster. Hence footprint (emissions) of computing and com- several benchmarking tools are introduced for munication infrastructure, such as data cen- evaluating the system performances by mea- ters, by using energy-efﬁcient design and op- suring the power consumption when varying the erations. The area has garnered increasing at- workload. Finally, a feasibility study on the pos- tention in recent years, both from a technology sibility to introduce a system to measure the and societal standpoint, due to the increased power consumption for the CMCC data center, focus on environmental and climate change is- and in particular for the Calypso cluster, is pro- sues. Hence, there is a need to balance the vided. dramatic growth of high-performance computing clusters and data centers in the computational sciences with green design and use so Green Computing and power saving in HPC data centers

TOWARDS “GREEN” DATA CENTERS U.S. Environmental Protection Agency (EPA) reported that 61 billion KWh, 1.5% of US elec- A data center hosts computational power, stor- tricity consumption, is used for data center com- 03 age and applications required to support an en- puting [18]. Additionally, the energy consump- terprise business. A data center is central to tion in data centers doubled between 2000 and modern IT infrastructure, as all enterprise con- 2006. It is reported that power and cooling tent is sourced from or passes through it. Data costs are the most dominant costs in data cen- centers can be broadly classiﬁed, on the basis ters [7]. Other studies ([4], [40]) have reported of power and cooling layout, into one of the 4 the global electricity costs for data centers run- tiers: ning into billions of dollars. Thus there is a growing pressure from the general public for Tier 1: Single path for power and cooling; the IT society to offer green data center infras- no redundant components; tructures and services [17]. There are two ma- Tier 2: Redundancy added to Tier 1, jor and complementary methods [38] to build a thereby improving availability; green data center: (1) utilize green elements in the design and building process of a data Tier 3: Multiple power and cooling distri- center, (2) greenify the process of running and bution paths, of which one is active; operating a data center in everyday usage. Var- Tier 4: Two active power and cooling ious research activities have been carried out paths, and redundant components on to manage data centers in a green mode (the each path. latter method), such as reducing data center

temperature [36], [42], [43], increasing server Centro Euro-Mediterraneo per i Cambiamenti Climatici This classification however, is not precise and utilization [33], [35], [41], and decreasing power commercial data centers typically fall between consumption of computing resources [22], [23], Tiers 3 and 4 [21]. A higher tier implies an [27], [32]. A fundamental research topic for the improvement in resource availability and reli- above study is how we can define performance ability, but it comes at the expense of an in- metrics to identify how green a data center is. crease in power consumption. Data centers Green performance metrics, for data centers, host services that require high availability, close are a set of measurements that can qualita- to 99.99%. Fault tolerance, therefore, becomes tively or quantitatively evaluate the environmen- imperative. The loss of one or more compo- tal impact of a data center. Nowadays, the term nents must not cause the data center to termi- green computing requires clear scientific defini- nate its services to clients. Consequently, data tions and it is more or less a marketing concept. centers feature hardware redundancy. Further- Research on developing green data center met- more, data centers are designed for a peak rics brings solid specifications and definitions load, which might be observed only occasion- for green data centers in the following aspects: ally, and for short bursts of time. This con- identify and specify clearly how green a servative design results in over-provisioning of data center is, for example by calculating hardware in a data center. All these factors its energy efficiency or greenhouse gas combined together contribute to the high power emission per time unit; consumption of data centers. So the electricity usage is the most expensive portion of a evaluate data center products and com- data center’s operational costs. In fact, the pare similar data centers; CMCC Research Papers

track green performance to increase a data center’s green efﬁciency, and pro- 04 vide guidance to engineers, manufactur- ers, and service providers;

research and development on future green data center technologies.

In the power/energy metrics Section, several performance metrics are introduced.

ENERGY USE IN A DATA CENTER

Making data centers more energy efﬁcient is a multidimensional challenge that requires a concerted effort to optimize power distribution, cooling infrastructure, IT equipment and IT out- put. A data center is composed of several components involved in the power consumption (Figure 1):

cooling equipment (e.g., computer room

Centro Euro-Mediterraneo per i Cambiamenti Climatici air-conditioner units);

power equipment (e.g., uninterruptible power supplies and power distributions units);

IT equipment (e.g., rack optimized and non-rack optimized enterprise servers, blade servers, storage and networking equipment);

miscellaneous equipment (e.g., lighting).

The cooling system occupies a signiﬁcant part, Figure 1: in terms of energy consumption, in a data cen- Components of a Data Center ter. The IT components present inside a data center when running produce a high heat. To avoid overheating and consequent damage of IT components, it is necessary to cool them and keep them continuously at a constant temperature. For this purpose, a cooling system which includes chillers, pumps and fans is used. Fig- ure 2 shows a typical cooling system, with modular conditioning units and a raised ﬂoor, which Green Computing and power saving in HPC data centers

a constant power even in case of interruptions both from the mains and the internal generator and to compensate for fluctuations of the 05 network or temporary loss of power. The energy is distributed to the computing resources of the data center through the UPS (Uninter- ruptible Power Supply), it is properly adapted to the right voltage of the IT components, using the PDU (Power Distribution Unit). A double connection to two different PDUs may also be available, to improve the reliability of the sys- Figure 2: tem. The degree of redundancy in the system, A cooling system therefore, is a very important consideration in the infrastructure. The data center should not run out of energy so for this there are a number allows the circulation of cold air. The size and of sources of supply and battery backup; the power of the system is based on the expected more the system is redundant, the greater the load of the servers and is chosen during the fault tolerance, but at the same time, the costs design phase of the data center. of installation and maintenance of this system The air flow is evenly distributed through will be higher. The IT equipment is the main the CRAC (Computer Room Air Conditioning), part of a data center, which makes possible its which controls the temperature through a sen- basic operations, and contains the electronics Centro Euro-Mediterraneo per i Cambiamenti Climatici sor typically set at 20°C, which measures the used for computing (servers), storing data (data internal temperature. The cold air is distributed storage) and communicating (network). Col- through the floor of the room with the use of lectively, these components process, store and fans and pumps, cooling the servers from be- transmit digital information. They require a suit- low; the hot air, which goes up, goes back in able environment, where they are positioned the air conditioner to be cooled again and dis- and arranged in an optimal manner, to reduce tributed in the data center. The combination of overheating and to prevent accidents to the ma- these components ensures optimal longevity of chines and the support staff. The servers, typ- the equipment. At the same time, however, this ically, are located in 1U or blade units, rack- ecosystem has a significant cost in the econ- mounted and interconnected via local Ethernet omy of the data center. The power equipment switches, these switches can be used in rack- includes an energy supply system for the data level connections at 1 or 10 Gbps and have a center. Typically, the infrastructure includes the number of active links to one or more Ether- connection to the electricity grid, generators, net switch at cluster level (or data center), this backup batteries and the energy for the cool- second level of switching can potentially cover ing system. The energy is recovered exter- more than ten thousand of individual servers. nally from the power supply. In the event of a The IT equipment must be decided at design power failure occurrence within the data center, time. It is very important to consider the ini- a generator will supply the necessary energy to tial power and predict the future need, to make the system. Electrical device backup batter- room for new updates or inclusion of new com- ies are recharged, with the aim of maintaining ponents and at the same time, to avoid exces- CMCC Research Papers

sive or unnecessary spending. This is just one the things that must be taken into account, the 06 greatest impact on data center costs regarding maintenance is the electricity consumption, including the one due to the cooling system: the bigger the size of the IT equipment, the greater the power required to cool the components. A power and cooling analysis, also referred to as a thermal assessment, measures the relative temperatures in specific areas of a data center, as well as the ability of the data center to tolerate specific temperatures. Among other things, a power and cooling analysis can help identify hot spots, over-cooled areas that can Figure 3: handle greater power use density, the break- Power Consumption in a data center point of equipment loading, the effectiveness of a raised-floor strategy, and optimal equipment positioning (such as AC units) to balance tem- summary of popular power related performance peratures across the data center. The energy metrics. The Data Center Infrastructure Effi- consumption of IT equipment, which perform ciency, DCiE or DCE [5], [6], is an industry ac- useful work in a computer center, represent cepted metric. It is defined as:

Centro Euro-Mediterraneo per i Cambiamenti Climatici on average only 50% of the total consumption, DCiE = IT EquipmentP ower since the remainder is divided between the con- TotalFacilityPower ditioning system (30%) and all the power supply where IT Equipment Power includes the load (Figure 3). For energy is therefore critical that associated with all of the IT equipment, i.e., the share of consumption related to power and computation, storage and network equipment, conditioning is the lowest possible, since the along with supplemental equipment such as activity of the computer center is represented switches, monitors, and workstations/laptops by electronic devices. It is very important to un- used to monitor or otherwise control the data derstand the values and distribution of energy center. Total Facility Power includes all IT Equip- ﬂows within the computer center. The mea- ment power as described above plus everything sured values of the power required for various that supports the IT equipment load such as: equipments are essential for assessing the en- power delivery components i.e., UPS, switch ergy efﬁciency of data centers. gear, generators, PDUs, batteries and distribution losses external to the IT equipment; Cool- ing system components i.e., chillers, computer POWER/ENERGY METRICS room air conditioning units, direct expansion As discussed in the previous section, current air handler units, pumps, and cooling towers. data centers consume a tremendous amount of Other miscellaneous component loads such as power. Power/energy metrics of various scales data center lighting are also included. and various components for data center com- A DCiE value of about 0.5 is considered typical puting have been proposed and applied in data practice and 0.7 and above is better practice. center computing practice. Figure 4 shows a Some data centers are capable of achieving 0.9 Green Computing and power saving in HPC data centers

Chilled Water is the annual District Chilled Wa- ter Energy Use. The HVAC System Effectiveness denotes the 07 overall efficiency potential for HVAC systems. A higher value of this metric means higher po- Figure 4: tential to reduce HVAC energy use [13]. Power consumption metrics taxonomy The metric of SWaP (Space, Watts and Perfor- mance) [20] characterizes a data center’s en- or higher [29]. An organization for green com- ergy efficiency by introducing three parameters puting provides the following standard metric of space, energy and performance together. for calculating the energy efficiency of the data The SWaP is calculated as follows: centers, Power Usage Effectiveness (PUE) [5]: = Performance SWaP (Space∗P owerConsumption) P UE = 1 = TotalFacilityPower DCiE IT EquipmentP ower where Performance is computed using industry- PUE shows the relation between the energy standard benchmarks, Space measures the used by IT equipments and energy used by height of the server in rack units (RUs) and other facilities, such as cooling needed for oper- Power Determines the watts consumed by the ating the IT equipment. For example, a PUE of system, using data from actual benchmark runs 2.0 indicates that for every watt of IT power, or vendor site planning guides. an additional watt is consumed to cool and The SWaP metric gives users an effective distribute power to the IT equipment. At the Centro Euro-Mediterraneo per i Cambiamenti Climatici cross-comparison and total view of a server’s present time, the PUE of a typical enterprise overall efficiency. Users are able to accurately data center falls between the range [1 - 3]. compare the performance of different servers The HVAC (Heating, Ventilation, and Air Con- and determine which ones deliver the optimum ditioning) system of a data center typically in- performance for their needs. The SWaP can cludes the computer room air conditioning and help users better plan for current and future ventilation, a large central cooling plant, and needs and control their data center costs. lighting and other minor loads. The HVAC Sys- The DCeP [14], [10] is proposed to characterize tem Effectiveness is the fraction of the IT equip- the resource consumed for useful work in a data ment energy to the HVAC system energy. The center. The DCeP is calculated as follows: HVAC system energy is the sum of the electrical energy for cooling, fan movement, and any = Useful work produced DCeP Total energy consumed to produce that work other HVAC energy use like steam or chilled Useful work is the tasks performed by the hard- water. The HVAC System Effectiveness is cal- ware within an assessment window. The calcu- culated as follows: lation of total energy consumed is the kWh of HV AC Effectiveness = the hardware times the PUE of the facility. IT HV AC+(F uel+Steam+ChilledW ater)∗293 where IT is the annual IT Electrical Energy Use, MEASURING POWER CONSUMPTION HVAC is the annual HVAC Electrical Energy Use, Fuel is the annual Fuel Energy Use, Steam There are a variety of computer power con- is the annual District Steam Energy Use and sumption benchmarking techniques in practice CMCC Research Papers

today. Even though many techniques are avail- der to get a better idea of how much power a able, there is no universal one size fits all tech- refrigerator consumes, it is important to mea- 08 nique that is appropriate for every benchmark- sure the load over some time period that rep- ing situation. Different methods of benchmark- resents typical refrigerator use patterns. In the ing must be completed for different usage pat- case of a refrigerator it may be necessary to terns. Additionally, measuring computer power measure a weeks worth of use in order to get consumption is different than general computer a clear picture of how much power the refrig- benchmarking because a tool is usually re- erator consumes on average. Similarly com- quired to measure how much electricity is be- puters fluctuate in their power consumption as ing consumed by a running machine. General they carry out various tasks. When comparing computer benchmarks such as CPU or video a computer’s overall power usage, it is crucial benchmarks do not require any special tools. to compare power usage over time [9]. Sim- Since power consumption benchmarks require ply measuring a computer’s power consump- electricity measuring devices, it makes it harder tion when it is idling will not factor in how much for people to participate in the power consump- power may be consumed while it has a load tion benchmarking area. The requirement and placed on it. A good way of comparing two cost of a tool adds a burden to those who wish systems is to compare real tasks on each sys- to run their own computer power consumption tem identically from start until finish [9]. This benchmarks. Here two electricity consump- reveals actual power requirements for tasks. tion meters are described, the affordable KILL It also exposes differences between systems. A WATT meter and the more expensive Watts Here is an example: “Let’s say you are com-

Centro Euro-Mediterraneo per i Cambiamenti Climatici up? meter. There are two fundamental ways paring system A that uses 45 watts of power of measuring power consumption: measuring under load and system B that uses 55 watts power consumption at one moment in time and of power under load. If system B finishes the measuring power consumption over time. Each same task as system A in half time (because method has its pros and cons. Measuring it has a better performing CPU), system B is power consumption over one moment in time actually a more power efficient system for the is useful when measuring a device that is us- benchmarked task than system A even though ing a constant amount of power. Less time is it uses more power under load. Since it takes needed to take a sample when measuring a a lot less time to finish, less power is spent device just in a time instant, rather than when [9]”. Although Idle benchmarks can be com- measuring a device over some period of time. pared between different reviewers and mag- A disadvantage of measuring over one moment azines for identical systems, the load bench- in time is that if the device being measured has marks depend on the specific tests conducted power fluctuations, it is not possible to get an by the reviewers. If two reviewers have identical accurate average of how much power the de- systems but have different benchmarking and vice uses. For example a refrigerator may use measuring schemes, the load benchmarks can- a lot of power when the compressor is activated not be compared. Luckily Idle should be com- but it may use very little power when the com- parable because in order to reach an idle state, pressor is turned off. It may use a little bit more the system has to be left doing effectively noth- power when the refrigerator door is opened and ing or very little. The systems being compared the light turns on. The power consumption of should come installed with the same hardware, a refrigerator fluctuates over the day. In or- operating system, service packs, patches, soft- Green Computing and power saving in HPC data centers

appliances. Some heavy appliances like electric stoves and electric dryers use 240 volts of alternating current (VAC). KILL A WATT is not 09 able to be used with 240 VAC wall plugs. The Watt mode displays how many watts are being consumed through the device at the moment of time the wattage value is recorded. This value tends to ﬂuctuate with most devices as the electrical loads vary. The wattage is what we were mostly concerned with in this work. The Hz mode measures hertz or how many times the alternating current cycles per second (usually 60 times). The PF mode displays Power Factor. The KWH mode displays the amount of kilowatt hours consumed since the KILL A WATT meter was connected to a power source. This mode is very useful in determining how much power Figure 5: a device or appliance uses over some time pe- P3 International KILL A WATT riod. The last mode is the “Hour” mode, which displays the amount of time the KILL A WATT meter has been connected to a power source. ware, and background processes. This would

Measuring power use over time is preferred be- Centro Euro-Mediterraneo per i Cambiamenti Climatici ensure that comparisons could be made because this type of measurement will capture all tween benchmarks performed on two different the ﬂuctuations of power usage as the electri- systems. cal load varies. Measuring power consump- KILL A WATT Power Consumption Meter tion by looking at the amounts of watts used at KILL A WATT (Figure 5) is a simple meter made any given moment in time is not representative by P3 International [11]. In order to use KILL of true power consumption of devices whose A WATT, it is required to plug it into the wall loads vary, such as computers. and then plug a device into KILL A WATT. It can be connected directly to the wall plug or Watts Up? Power Consumption Meter The can be connected to an extension cable or Watts up? power consumption meter (Figure power strip which is then connected to the wall 6) features the same functionality as the KILL plug. It is easier to work with KILL A WATT A WATT meter and more. In addition to the with an AC extension cable so that it could be KILL A WATT features, the Watts up? meter positioned closer to the experimenter for eas- can record power measurements over time in ier viewing. KILL A WATT has several modes watt hours sensitively enough to be used for including Volt (voltage), Amp (amperes), Watt computer power consumption measurements. (wattage), VA (volt-amperes), Hz (hertz), PF In addition to recording the amount of power (power factor), KWH (kilowatt hours), and Hour consumed over time, the meter comes with (the length of time the device has been con- memory which saves the data that is recorded nected to a power source). The voltage tends during measurements. The data can then be to approximately 120 volts for most devices and accessed by loading software from the watts CMCC Research Papers

ter, around 90-220 dollars as of the time of this writing. The Watts Up? meters are more com- 10 plicated and might be harder for some people to learn how to use. Despite these drawbacks the watts up? meter proved to be a better tool for our research in order to carry out serious computer power consumption benchmarks. With- out a watt hour measuring option on a power meter, it is not possible to accurately measure how much power a computer uses when its power ﬂuctuates unless the benchmarks lasts for hours or days.

COMPUTER POWER CONSUMPTION BENCHMARKING

One area of the computing sciences that is be- Figure 6: Watts up? Power Consumption Meter coming more and more important is computer power consumption benchmarking. Bench- marking is a general and widely known approach where a standard load, or benchmark,

Centro Euro-Mediterraneo per i Cambiamenti Climatici up? website [12] and connecting the meter to is used to measure some desired aspects, be- a Windows computer with an included USB ca- havior or other characteristics of the thing being ble. There are several Watts up? meter models observed. In computing, benchmarks are typi- available. Among them the Watts up? PRO es cally computer programs that are run on a sys- model, which includes extra memory for saving tem, enabling accurate and repeatable mea- measurements that are taken over long periods surement of some specific characteristic of in- of time. The Watts up? meter has several ad- terest. A performance metric is typically as- vantages over the KILL A WATT meter. It has sociated with a benchmark or a workload, as more complex features than the KILL A WATT well as a well-defined execution environment. A meter such as more sensitive measurements benchmark is used to emulate real-world com- and memory for data storage. It can connect puting tasks. A data center benchmark is the to a Windows machine with a USB cable and act of running a set of programs, or other op- transfer data which can then be graphed and erations, in order to assess the relative per- used for analysis. It has an extension cable formance of data center performance metrics. that gives the user a lot of leeway to position Typically data center benchmarking is carried the meter in a viewable spot. The watts up? out under a certain workload, which is artifi- power consumption meter’s functionality made cially generated or produced in real life, accord- it a great tool for professional power consump- ing to the following workflow: real or artificial tion benchmarking. There are also several im- workload → benchmark → performance met- portant disadvantages to note about the Watts rics. Although some benchmark workloads are up? meter. The Watts up? meter costs several available, for example, server-side Java under times more money than the KILL A WATT me- various loads for SPECpower [34], JouleSort Green Computing and power saving in HPC data centers

[39], data center benchmarking is generally the performance measured at each target load taken under normal data center practice and level divided by the sum of the average power workload. In this section, two types of bench- (in watts) at each target load. The Green500 11 marks are described: server level benchmarks project [26] aims at increasing awareness about and data center level benchmarks. Server level the environmental impact and long-term sus- benchmarks, like GCPI, Green500 and Joule- tainability of high-end supercomputing by pro- Sort, normally measure energy consumption in viding a ranking of the most energy-efficient su- a compute server or a cluster. Data center level percomputers in the world. The Green500 list benchmarks, such as, HVAC Effectiveness In- uses performance per watt (PPW) as its met- dex and Computer Power Consumption Index ric to rank the energy efficiency of supercom- from LBNL, present a measurement for the en- puters. The performance is defined as the tire data center. achieved maximum GFLOPS (Giga FLoating- point OPerations per Second) performance, by Computer server level benchmark This sec- the Linpack [16] benchmark on the entire sys- tion introduces several performance metrics for tem. The power is defined as the average sys- computer server level performance evaluation: tem power consumption during the execution GCPI, SPECpower, Green500 project, and of Linpack with a defined problem size. Joule- JouleSort. SiCortex [19] proposes an inclu- Sort [39] is an I/O-centric benchmark that mea- sive metric for comparing energy efficiency in sures the energy efficiency of systems at peak the High-Productivity Computing segment. The use. It is an extension of the sort benchmark, Green Computing Performance Index (GCPI) which is used to measure the performance and

[15] is an inclusive metric for comparing energy cost-performance of computer systems. The Centro Euro-Mediterraneo per i Cambiamenti Climatici efficiency in High-Productivity Computing seg- JouleSort benchmark measures the cost of do- ment proposed by SiCortex [19]. The GCPI ing some amount of work, which reflects some analyzes computing performance per kWatt measure of power use, e.g., average power, across a spectrum of industry-standard bench- peak power, total energy, and energy-delay. marks, providing organizations with much- needed guidance in the era of out-of-control Data center level benchmarking Data center data center energy consumption. It has been level benchmarks are used to evaluate how declared that the GCPI is the first index to green a data center is and are also used to com- measure, analyze and rank computers on a pare similar data centers. Lawrence Berkeley broad range of performance metrics relative to National Laboratory (LBNL) releases a set of ef- energy consumed. In 2006, the SPEC com- forts and practices for data center benchmark- munity started to establish SPECpower [34], ing [3]. These practices include improved air an initiative to augment existing SPEC bench- management, emphasizing control and isola- marks with a power/energy measurements. tion of hot and cold air streams; rightsizing cen- SPECpower’s workload is a Java application tral plants and ventilation systems to operate that generates and completes a mix of trans- efficiently both at inception and as the data cen- actions; the reported throughput is the num- ter load increases over time; optimizing central ber of transactions completed per second over chiller plants designed and controlled to maxi- a fixed period. SPECpower reports the en- mize overall cooling plant efficiency, central air- ergy efficiency in terms of overall operations handling units in lieu of distributed units. Figure per watt. This metric represents the sum of 7 shows the formal process for benchmarking CMCC Research Papers

technologies exist within the CPU market such as Intel’s SpeedStep and AMD’s PowerNow! 12 technologies. These dynamically raise and lower both frequency and CPU voltage using ACPI P-states [2]. In [25], DVFS techniques are Figure 7: Data center benchmarking process used to scale down the frequency by 400Mhz while sustaining only a 5% performance loss, resulting in a 20% reduction in power. A power- a data center [13]. Main software packages aware cluster supports multiple power and perfor benchmarking and designing energy effi- formance modes, allowing for the creation of cient buildings and data centers are: Calarch, an efficient scheduling system that minimizes Comis, DoE-2, EnergyPlus, Genopt, home en- power consumption of a system while attempt- ergy saver. ing to maximize performance. When looking to create a DVFS scheduling system for a data CLUSTER POWER MANAGEMENT center, there are a few rules of thumb to build ALGORITHMS a scheduling algorithm which schedules virtual machines in a cluster while minimizing the Two basic power management mechanisms power consumption: are dynamic voltage scaling and node vary- on/vary-off. In the following these mechanisms 1. Minimize the processor supply voltage by are briefly described. Also hybrid approaches scaling down the processor frequency;

Centro Euro-Mediterraneo per i Cambiamenti Climatici that combine these techniques have been in- 2. Schedule virtual machines to processing vestigated. elements with low voltages and try not to Dynamic Voltage and Frequency Scaling One scale Processing Elements (PEs) to high technique being explored is the use of Dynamic voltages. Voltage and Frequency Scaling (DVFS) within Clusters and Supercomputers [30], [31]. By Based on the performance model defined using DVFS one can lower the operating fre- above, Rule 1 is obvious as the power con- quency and voltage, which results in decreased sumption could be reduced when supplied volt- power consumption of a given computing re- ages are minimized. Then Rule 2 is applied: source considerably. High-end computing com- by scheduling virtual machines to processing munities such as cluster computing and super- elements with low voltages and trying not to computing in large data centers, have applied operate PEs with high voltages to support vir- DVFS techniques to reduce power consumption tual machines. A scheduling algorithm for vir- and achieve high reliability and availability [28], tual machines in a DVFS-enabled cluster [45] [24], [8]. A power-aware cluster is defined as is shown in Figure 8 where incoming virtual a compute cluster where compute nodes sup- machine requests arrive at the cluster and are port multiple power/performance modes, for ex- placed in a sorted queue. This system has ample, processors with frequencies that can been modeled, created and described in [44]; be turned up or down. This technique was in this Section we will explain in detail that originally used in portable and laptop systems scheduling mechanism. The scheduling algo- to conserve battery power, and has since mi- rithm runs as a daemon in a cluster with a pre- grated to the latest server chipsets. Current defined schedule interval, INTERVAL. During Green Computing and power saving in HPC data centers

This policy, originally proposed by Pinheiro et. al. [37], turns off server nodes so that only the minimum number of servers required to 13 support the workload are kept active. Nodes are brought online as and when required. VOVO does not use any intra-node voltage scaling, and can therefore be implemented in a cluster that uses standard high-performance processors without dynamic voltage scaling. However, some hardware support, such as a Wake-On-LAN network interface, is needed Figure 8: to signal a server to transition from inactive Working scenario of a DVFS-enabled cluster scheduling to active state. Node vary-on/vary-off can be implemented as a software service running on one of the cluster nodes (or a separate the scheduling interval, incoming virtual ma- support server) to determine whether nodes chines arrive at the scheduler and will be sched- must be taken offline or brought online and uled at the next schedule round. For each vir- requires software probes running on each of tual machine (VM), the algorithm checks the PE the cluster nodes to provide information on the operating point set from low voltage level to high utilization of that node. The load distribution voltage level. The PE with lowest possible volt- mechanism must be aware of the state of the age level is found; if this PE can fulfill the virtual servers in the cluster so that it does not direct Centro Euro-Mediterraneo per i Cambiamenti Climatici machine requirement, the VM can be sched- requests to inactive nodes, or to nodes that uled on this PE. If no PE can schedule the VM, have been selected for vary-off but have not a PE must be selected to operate with higher yet completed all received requests. voltage. This is done using the processor with the highest potential processor speed, which is Given the features of the Euro-Mediterranean then adjusted to run at the lowest speed that Centre for Climate Change (CMCC) clusters, fulfills the VM requirement, and the VM is then at the moment it is not possible to apply DVFS scheduled on that PE. After one schedule inter- algorithms. So, several Vary-on/Vary-off mech- val elapses, some virtual machines may have anisms will be investigated. finished their execution; thus this algorithm at- tempts to reduce a number of PE’s supply volt- A PROPOSAL FOR A METERING ages if they are not fully utilized. SYSTEM FOR THE POWER CONSUMPTION OF CALYPSO Vary-on/Vary-off Node vary-on/vary-off (VOVO) takes whole nodes offline when the The Supercomputing infrastructure provided by workload can be adequately served by a CMCC consists of two cluster systems, which subset of the nodes in the cluster. Machines together reach a theoretical peak performance that are taken off-line may be powered off or of more than 30 TFlops. These systems are placed in a low power state. Machines that complemented by an high-capacity and high- are off-lined are placed back online should the performance storage infrastructure that, over- workload increase. Node varyon/vary-off is an all, provides a useful storage space, which inter-node power management mechanism. amounts to more than 450 Tbytes. The two CMCC Research Papers

supercomputing clusters differ in the type of architecture: the first one is a vector system con- 14 sisting of NEC SX-8R series nodes and SX-9 nodes, while the second one is a scalar system consisting of 30 IBM POWER 6 nodes divided into 3 frames. Each node contains 16 Power 6 dual-core processors, operating at a frequency of 4.7 GHz. Therefore, the number of physical cores amounts to 960 cores, capable of providing a total computing power (theoretical peak performance) of about 18 TFLOPS. To ensure the continuity of the power to such systems - and the consequent preservation of data - the CMCC has a 500KVA Static UPS power system. Figure 9: Diagram of the electrical connection between the ENEL cabinet and CMCC through UPS The power panel is already prepared for a second parallel UPS able to provide total protection for 1000kVA for future additions (Figure 9). The CMCC has a dedicated transformer sub- station from which three lines of service reach the CMCC. A 30 kW line is dedicated to the supply line of IBM systems and, finally, a sup- Centro Euro-Mediterraneo per i Cambiamenti Climatici ordinary user, another of 350 kW is dedicated ply line dedicated to services. The main control to users of air conditioning, and the third one - panels of each line are equipped with meters the most important - is the preferred line ded- which record the total consumption. Regarding icated to computer systems. The presence of Calypso the power of each individual frame is a single supply line and the presence of some managed by four three-phase lines (12 in total) components of redundant cooling allow classi- (Figure 10). fying the CMCC Supercomputing Center as a Tier II data center. Inside the CMCC there are two main areas (Figure 10) well separated and Since we intend to analyze the energy con- with well defined roles. sumption of Calypso, a possibility is to perform In the first environment called Control Room, all the metering of the consumption of each IBM the hosted electrical panels supply power to (frame) rack by measuring the three-phase sup- various power lines of computer systems, stor- ply. Every IBM frame provides 4-phase electri- age and services. The system cooling is local connections that feed to 12 PDUs. These cated in the second environment (greater than PDUs in turn feed the IBM P575 computing 200 square meters), called Computing Room. In- nodes in addition to two control units. Hence, it side the Control Room there are several electri- is possible to place a power panel in the Com- cal panels whose main task is to distribute elec- puting Room for measuring the energy con- trical power to computer systems and storage. sumption of each of the 4 lines per frame, thus There are three main power supply lines: the obtaining a measurement of the total consump- power supply line of NEC systems, the power tion of each IBM Calypso rack (Figure 11). Green Computing and power saving in HPC data centers 15

Uninterruptible Power Supply IBM Panel

Figure 10: Scheme of electrical connection between the control room and Calypso Centro Euro-Mediterraneo per i Cambiamenti Climatici

Uninterruptible IBM Panel Power Supply

Figure 11: Scheme of electrical connection between the control room and Calypso with insertion of a panel dedicated to measure the power consumption of each frame CMCC Research Papers

Bibliography [12] Watts Up? Products: Meters, Electronic Educational Devices, 2008. 16 [1] High Performance Green Comput- [13] Self-benchmarking guide for data centers: ing, A Proposal for New Investments metrics, benchmarks, actions, 2009. Tech- in Faculty Hiring in the Departments nical report, Lawrence Berkeley National of Astronomy and Computer Science. Laboratory, Berkeley, California. http://www.umass.edu/oapa/oapa/proposa [14] 42U DCeP: Data Center energy Produc- ls/high performance green computing.pdf. tivity, 2011. [2] Data Center Power Management and Ben- [15] Green Computing Per- efits to Modular Computing, 2003. Intel formance Index, 2011. Developer Forum. http://www.sicortex.com/green index. [3] Best practices for data centers: lessons [16] Green Computing Performance Index, learned from benchmarking 22 data cen- 2011. http://www.netlib.org/linpack/. ters, 2006. Technical report, Lawrence Berkeley National Laboratory. [17] IEEE TCSC technical area of green computing, 2011. [4] Estimating total power consumption by servers in the US and the world, 2007. [18] Report to Congress on Server and Tech. rep., Lawrence Berkeley National Data Center Energy Efficiency, 2011. Laboratory. www.energystar.gov/ia/partners/prod deve lopment/downloads/EPA Datacenter Repo [5] The green grid data center efficiency met- rt Congress Final1.pdf. Centro Euro-Mediterraneo per i Cambiamenti Climatici rics: PUE and DCIE, 2007. Technical report, The Green Grid. [19] SiCortex, 2011. http://www.sicortex.com/. [6] The Green Grid metrics: data center [20] SWaP (Space, Watts and Performance) infrastructure efficiency (DCiE) detailed Metric, 2011. analysis, 2007. Technical report, The [21] L. A. Barroso and U. Holzle. The Dat- Green Grid. acenter as a Computer: An Introduction [7] The green grids opportunity: decreasing to the Design of Warehouse-Scale Ma- datacenter and other IT energy usage pat- chines. Synthesis Lectures on Computer Ar- terns, 2007. Technical report, the green chitecture, 1(1):1–108, 2009. grid. [22] Y Chen, A Das, W Qin, A Sivasubrama- [8] The Green500 List: Encouraging Sustain- niam, Q Wang, and N Gautam. Manag- able Supercomputing, 2007. IEEE Com- ing server energy and operational costs in puter Society. hosting centers. In SIGMETRICS’05: pro- [9] The Truth About PC Power Consumption, ceedingsofthe2005 ACMSIGMETRICS inter- 2007. Tom’s Hardware, Bestofmedia Net- national conferenceon measurement and mod- work. eling of computer systems, pages 303–314, 2005. [10] A framework for data center energy productivity, 2008. Technical report, The [23] R Das, JO Kephart, C Lefurgy, G Tesauro, GreenGrid. DW Levine, and H Chan. Autonomic multiagent management of power and perfor- [11] P3 - Kill A Watt, 2008. mance in data centers. In AAMAS O08:˜ Green Computing and power saving in HPC data centers

proceedings of the 7th international joint con- proceedings of the 2009 IEEE international ference on autonomous agents and multiagent symposium on parallel & distributed process- systems, pages 107–114, 2008. ing, pages 1–8, 2009. 17 [24] W. Feng, A. Ching, C.H. Hsu, and V. Tech. [33] G Jung, KR Joshi, MA Hiltunen, Green Supercomputing in a Desktop Box. RD Schlichting, and C Pu. A cost- In IIn IEEE International Parallel and Dis- sensitive adaptation engine for server con- tributed Processing Symposium, pages 1–8, solidation of multitier applications. In 2007. ACM/IFIP/USENIX 10th international mid- [25] W. Feng, X. Feng, and R. Ge. Green Su- dleware conference, pages 163–183, 2009. percomputing Comes of Age. IT PROFES- [34] KD Lange. Identifying shades of green: SIONA, 10:17, 2008. the specpower benchmarks. Computer, [26] WC Feng and KW Cameron. The 42:95–97, 2009. Green500 list: encouraging sustainable [35] S Mehta and A Neogi. Recon: a tool supercomputing. Computer, 40:50–55, to recommend dynamic server consoli- 2007. dation in multi-cluster data centers. In [27] VW Freeh and DK Lowenthal. Using mul- IEEE/IFIP network operations and manage- tiple energy gears in mpi programs on a ment symposium: pervasive management for power-scalable cluster. In Pingali K, Yelick ubioquitous networks and services, pages KA, Grimshaw AS (eds) PPOPP. ACM Press, 363–370, 2008. New York, pages 164–173, 2005. [36] CD Patel, R Sharma, CE Bash, and A Beit-

[28] I. Gorton, P Greeeld, A Szalay, and elmal. Thermal considerations in cooling Centro Euro-Mediterraneo per i Cambiamenti Climatici R. Williams. Data-Intensive Computing large scale high compute density data cen- in the 21st Century. Computer, 41:30–32, ters. In Proceedins of Thermal and thermo- 2008. mechanical phenomena in electronic systems - ITHERM 2002, pages 767–776, 2002. [29] S Greenberg, A. Khanna, and W Tschudi. High performance computing with high ef- [37] E. Pinheiro, R. Bianchini, E.V.Carrera, and ficiency. AmSoc Heat Refrig Air-cond Eng T. Heath. Load Balancing and Unbalanc- (ASHRAE) Trans, page 179, 2009. ing for Power and Performance in Cluster- Based Systems. In In Workshop on Com- [30] Chung hsing Hsu and Wu chun Feng. A pilers and Operating Systems for Low Power, feasibility analysis of power awareness in pages 180–195, 2001. commodity-based high-performance clusters. In In Proceedings of IEEE International [38] S. Rivoire, MA Shah, P Ranganathan, and Conference on Cluster Computing, pages 1– J Meza C. Kozyrakis. Models and metrics 10, 2005. to enable energy-efficiency optimizations. Computer, 40(12):39–48, 2007. [31] C. Hsu and W. Feng. A power-aware run- time system for high-performance comput- [39] S Rivoire, MA Shah, P Ranganathan, ing. In In Proceedings of the 2005 ACM/IEEE and C Kozyrakisr. Joulesort: a bal- conference on Supercomputing, 2005. anced energy-efficiency benchmark. In SIGMOD’07: proceedings of the ACM SIG- [32] N Jiang and M Parashar. Enabling au- MOD international conference on manage- tonomic power-aware management of in- ment of data, pages 365–376, 2007. strumented data centers. In IPDPS O09:˜ CMCC Research Papers

[40] A Shah and N Krishnan. Optimization [43] DC Vanderster, A Baniasadi, and NJ Di- 18 of global data center thermal manage- mopoulos. Exploiting task tempera- ment workload for minimal environmental ture profiling in temperature-aware task and economic burden. IEEE Transactions scheduling for computational clusters. In on Components and Packaging Technologies, Asia-Pacific computer systems architecture 31(1):39–45, 2008. conference, pages 175–185, 2007. [41] A Singh, MR Korupolu, and A Mohapa- [44] Gregor von Laszewski, Lizhe Wang, An- tra. Server-storage virtualization: integra- drew J. Younge, and Xi He. Power-Aware tion and load balancing in data centers. In Scheduling of Virtual Machines in DVFS- Proceedings of the ACM/IEEE conference on enabled Clusters. In IEEE Cluster 2009, high performance computing, page 53, 2008. 2009. [42] Q Tang, SKS Gupta, and G Varsamopou- los. Energy-efficient thermal-aware [45] Andrew J Younge, Gregor Von Laszewski, task scheduling for homogeneous high- Lizhe Wang, and Geoffrey C Fox. Provid- performance computing data centers: a ing a Green Framework for Cloud Based cyber-physical approach. IEEE Trans Par- Data Centers. In The Handbook of Ener- allel Distrib Syst, 19(11):1458–1472, 2008. gyAware Green Computing, 2011. Centro Euro-Mediterraneo per i Cambiamenti Climatici

The Euro-Mediteranean Centre for Climate Change is a Ltd Company with its registered office and administration in Lecce and local units in Bologna, Venice, Capua, Sassari and Milan. The society doesn’t pursue profitable ends and aims to realize and manage the Centre, its promotion, and research coordination and different scientific and applied activities in the field of climate change study.