ISS Data Centre Energy Efficiency

ISS Data Centre Energy Efficiency - Research Assessment Report: January 2012

National University of Ireland, Galway

ISS Data Centre

Energy Efficiency Research

Initial Assessment

January 2012

1 ISS Data Centre Energy Efficiency - Research Assessment Report: January 2012

Introduction

“Better energy data collection would not only help to quantify the energy load of data centre operations, thus highlighting the importance of energy-efficiency improvements and facilitating right-sizing of equipment to match the energy load, it would also allow data centre operators to monitor and evaluate the energy savings resulting from specific energy-efficiency improvements. Seeing concrete savings may also spur data centre operators to implement further efficiency measures.”

Energy Star Program - U.S. Environmental Protection Agency, “EPA Report to Congress on Server and Data Center Energy Efficiency”, EPA, Aug 2007.

An exploratory Energy Efficiency (EE) analysis of the ISS Data Centre (DC) in NUI Galway was carried out in the last 2 weeks of July 2011. The analysis was the culmination of 10 months research for 2 full time postgraduate students in the Discipline of IT:

a) Jonathan Hanley – MSc. Software Design & Development - Year 2

b) Mark White – PhD. Energy Informatics - Year 1 (Principal Author of Report)

Others involved in the research were:

 Dr. Hugh Melvin*

 Dr. Michael Schukat*

 Dr. Marcus Keane**

 Maria Linnane***

 Wesley Reilly***

 Dr. Edward Curry****

*Discipline of IT **Discipline of Civil Engineering ***Information Solutions & Services (ISS) ****Digital Enterprise Research Institute (DERI)

The objectives of the research were to:

1. a) Perform data capture in a DC test bed environment – including all values required to perform an EE analysis which is as close to industry standards as practicable.

b) It was important to all stakeholders in the research that the data capture process would not intrude upon the normal operations of the DC.

2 Design/Implement a support ICT infrastructure to store, analyse and visualise the data for all stakeholders (Facilities Management, Operational Staff and other researchers). A website was designed and used as the front end of this objective. It is located at: http://lugh2.it.nuigalway.ie/enformatics.

Note: Due to the sensitive nature of the data captured during the research, log in credentials were provided for those involved. Readers of this report may log in using the following credentials: username = guest & password = guest

Figure 1: ISS Room 140 floor plan – used as an interactive map on the website developed by the research team

3 Identify/assess opportunities for EE gains specific to the ISS test bed

4 Develop analysis tools that could be applied more generally to other DCs

Application of Power Usage Effectiveness to ISS

The industry standard for measuring Energy Efficiency in Data Centres is Power Usage Effectiveness (PUE). It is defined as:

When the equation is broken down into its constituent parts for ISS, it can be seen which values within the DC environment are most relevant for calculating the PUE:

UPS = Uninterruptible Power Supply, AHUs = Air Handling Unit(s) While server virtualisation, hardware placement, UPS/PDU efficiency improvement efforts continue in industry, most current research suggests the greatest single efficiency gain can be achieved in the cooling system – where up to 50% of all DC power consumption may originate1. Using the PUE equation as a starting point the research team designed the architecture in Figure 2 to measure and collect:

a) rack input temperatures

b) rack output temperatures

These values (a, b) were collected using a WSN (Wireless Sensor Network – supplied by The Tyndall Institute) and support ICT infrastructure. Temperature values for 18 wireless sensors placed high, middle and low on both the front and rear of 3 racks (Room 140 - Row 1: Cabinet 1, Row 3: Cabinet 1 & Row 3: Cabinet 2 – ref: figure 1 above) were polled every 30 seconds before being stored in a SQL Server database.

c) AHU (address @ *.170) inlet / outlet temperatures

d) AHU (address @ *.191) inlet / outlet temperatures

e) UPS (address @*.180) power consumption

f) Server CPU loads

These values (c - f) were collected using the Simple Network Management Protocol (SNMP). SNMP is a connectionless messaging system which uses the User Datagram Protocol (UDP), minimising the traffic impact on a network. SNMP Values were polled at 60 second intervals and stored in the database. The 60 second interval was chosen so as not to overload the network with additional traffic volumes – partly satisfying objective 1b.

Figure 2: Data collection architecture

Unfortunately, real time power values for lighting were not available but it was agreed with ISS staff that this figure could be considered negligible (probably less than 1% of total consumption). The IRUSE2 group provided clamp-on power data loggers and a 7 day test of the AHU @ address .191 yielded an average power consumption of 19.9kW (Figure 3).

Figure 3: Excerpt from 7 day power consumption test on AHU @ address .191

One of the external condensers was due to be clamped for 7 days but the test would have caused disruption to the normal work flow of ISS so a static value, collected in a prior test by ISS staff, was used instead.

The reading had been taken from Condenser 1 outside ISS with results as follows:

 Phase R: 0.82A

 Phase S: 0.81A

 Phase T: 0.72A

Taking an average of 0.783 across the 3 phases and assuming a power factor of 0.8 over 220 Volts, the consumption of condenser 1 is: 238.683456 Watts. With 2 condensers deployed for the system in Room 140 the total consumption is approximately 0.477kW.

Including outliers (spikes and temporary outages), the UPS average across the recorded period was calculated at: 61.759kW. Comparing power input and output (Figure 4 below) it can be seen that the UPS system in ISS is operating at close to 92% efficiency.

Figure 4: Input and output power values collected during the test period

The full dataset at the end of the 2 week test included:

 700,000+ values from the wireless temperature sensors

 360,000+ values from the 2 AHUs in Room 140

 280,000+ values from the UPS in Room 139

With other values polled (e.g. CPU loads, AHU fan speeds and AHU filter pressures) the total number of values collected over the 2 week test period exceeded 1.5 million. Additional temperature sensor tests brought the total dataset to approximately 3.5 million values.

Functional Dependencies

There are a number of critical functional dependencies which exist within the DC environment. A functional dependency is a data value which is dependent on some other data value within the same environment. Of most interest in the initial assessment of the results from this test were the input temperatures of the AHUs – which are functionally dependent on the output temperature from the previous cooling cycle and the additional heat generated across the racks during the current cycle. For the purposes of this report the room itself is considered to be a black box - the relationship between the temperatures collected & recognised industry standards is the main focus.

However, before a more in-depth analysis of the data could begin, large discrepancies between the WSN temperatures and AHU sensors were observed (Figure 5).

Figure 5: Differences between AHU and Tyndall temperature sensors

The blue plots are the supply and return values from the AHU tested. It can be seen that there is an average difference of approximately 4°C on the supply side and an average of approximately 1.5°C on the return side. The WSN sensors are higher in both instances. To examine a dataset confidently the data must first be robust and reliable. The team decided to re-test, placing the WSN sensors as close as possible to the AHU sensors (Figure 6).

Figure 6: Retest of sensor differences & calibration results

Having repositioned the WSN sensors as close as possible (there remains some doubt as to the exact positioning of the AHU’s supply sensor) to the AHU sensors, the results for the return sensors (both WSN and AHUs) were close enough (0.5°C on the AHU at address .191 and 1.91°C on the AHU @ address .170) to accept. Having checked the WSN sensors against a calibrated temperature logger, the discrepancy for the supply side was concluded to be as a result of poor positioning.

The WSN temperature results (in isolation from the AHU temperatures) are also outlined below (Figure 7 below):

Figure 7: WSN Temperatures in isolation from AHU results (all values averaged)

The American Society of Heating, Refrigeration and Air Conditioning Engineers (ASHRAE) is the de facto industry leader for data centre EE standards. In 2008 they published the second edition of Thermal Guidelines for Data Processing Environments3 which includes recommendations for temperature and humidity set points. These set points are categorised according to the type of DC in question. ISS is considered to be a legacy DC, in that the rooms containing the IT equipment were retrofitted for purpose. The ASHRAE set point recommendation for the input to racks of a DC such as ISS falls within the range of 18 – 27 ⁰C (64.4 – 80.6⁰F).

The temperature observed at the output of the AHU (14⁰C) was approximately 0.5⁰C below the temperature (14.5⁰C) at the rack inlet closest to the floor. The air pressure in the sub-floor plenum should be maximised by ensuring that there is as little impedance as practicable. Allowing for observed characteristics such as:

 non-uniform (non-parallel) cabling runs

 unsealed cable outlets in the floor

 AHU placement not perpendicular (ideal orientation) to aisles there is an acceptable difference between the supply-side sensors. The difference between the return-side temperatures was more defined. Rack output averages were 26⁰C but AHU return averages were 23.5⁰C. At the time of writing this discrepancy has not been investigated further and remains unexplained. Putting these concerns aside momentarily however, we posed the question: Is ISS operating within the recommended ASHRAE ranges for rack input temperature? The answer, clearly, is - NO.

Measurement Errors

A crucial issue relating to ISS is where the temperature (upon which the AHUs are operated) is being measured.

As far as could be ascertained, ISS is operating its cooling policy with a set point of 21⁰C - measured at the AHU return). ASHRAE guidelines, however, apply to temperatures measured at the upper rack input. The AHUs in ISS are controlled on the basis of the temperature of their return (warm) air – whereas ASHRAE recommends measuring the cold air at the server inlet. This erroneous configuration is widely encountered by analysts in industry. The remedy as suggested by The Green Grid in its June 2011 White Paper #393:

“Reposition the AHU temperature/humidity sensors from the input of the AHU to the front of the IT equipment racks“

PUE Calculation

Leaving the temperature measurements aside we proceeded to calculate the PUE for the DC at ISS.

 UPS = 61.759kW

 AHU = 19.9kW * 2 (there are 2 AHUs in the cooling system for Room 140) = 39.8kW

 Condensers * 2 (2 of the external condensers supply Room 140) = 0.477kW

 Lighting = negligible

PUE = (61.759 + 39.8 + 0.477) / 61.759 = 1.6521

If a 10% discount is factored into the UPS value (allowing for the load in Room 139 which was omitted from this study) then the calculation becomes:

PUE = (55.5831 + 39.8 + 0.477) / 55.5831 = 1.7246

Conclusions

The DC industry agrees that a saving of 2 – 5% on the data centre utility bill can be achieved for each half degree Celsius the temperature in a data centre is raised4. With an increase in working temperature the AHUs are working less to cool the air being sent into the room, reducing the overall power consumption. While there are provisos when considering raising the temperature of the data centre (increased server fan speeds, lower mean time to recovery [MTTR], weakest link equipment operating temperatures), the analysis is more often than not worth the effort.

On initial analysis there appears to be scope to raise the temperature in ISS by up to 10⁰C – but an increase of 5 - 8⁰C is probably a more conservative estimate. If the effort to raise the

7 ISS Data Centre Energy Efficiency - Research Assessment Report: January 2012 data centre temperature is based on ASHRAE guidelines then the input sensor of the AHUs should be repositioned to read the rack inlet temperature – providing readings which relate more directly to the ASHRAE standards.

Another (relatively inexpensive) improvement that should be considered is aisle containment using strip curtains or other similar strategy. Aisle containment reduces cross flows within the DC environment resulting in a range of efficiencies including more efficient AHU performance and reduced internal CPU fan load. Needless to say, the benefits of raising AHU temperatures to within ASHRAE levels would best be realised if done in conjunction with appropriate aisle containment.

While it is understood that the ISS datacentre was retrofit for purpose it is clear that a number of optimisation strategies could be implemented which typically have low capital costs associated and reflect well on the operating budget over the medium term.

References & Notes:

1: Data Center Efficiency Task Force, “Recommendations for Measuring and Reporting Version 2 – Measuring PUE for Data Centers”, 17th May 2011

2: IRUSE (Informatics Research Unit for Sustainable Engineering)

3: Green Grid, “The ROI of Cooling System Energy Efficiency Upgrades” Green Grid, White Paper #39, 2007

4: ASHRAE, “Thermal Guidelines for Data Processing Environments – Expanded Data Center Classes and Usage Guidance”, ASHRAE INC, August 201. ASHRAE (American Society of Heating, Refrigeration and Air Conditioning Engineers) will publish the 4th edition of their guidelines before the end of this year (2011). It is expected they will raise the temperature and humidity recommendations even further

Additional background information is available in Mark White’s literature review (‘A Dynamic Control System for Energy Efficient Cooling of Data Centres’) which examines the current ‘state of the art’ and future trends in the data centre energy efficiency industry. The review can be downloaded from: http://lugh2.it.nuigalway.ie/markwhite/papers/LiteratureReview.pdf