Power Efficiency and Performance with ORNL's XK7 Titan

Work by Jim Rogers Presented by Arthur Bland Director of Operations National Center for Computational Sciences Oak Ridge National Laboratory ORNL’s “Titan” Hybrid System: Cray XK7 with AMD and Tesla processors

SYSTEM SPECIFICATIONS: • Peak performance of 27.1 PF • 24.5 GPU + 2.6 CPU • 18,688 Compute Nodes each with: • 16-Core AMD Opteron CPU • “K20x” GPU • 32 + 6 GB memory • 512 Service and I/O nodes 2 4,352 ft • 200 Cabinets 2 404 m • 710 TB total system memory • Cray Gemini 3D Torus Interconnect • 8.9 MW peak power

2 ORNL ISC’13 ORNL's Cray XK7 Titan | Sample Run: HPL Consumption MW, Instantaneous kW-hours (Cumulative)

10 8,000 7,545.56kW-hr 9 7,000

8 RmaxPower = 8,296.53kW 6,000 7 Instantaneous Measurements 5,000 6 8.93 MW 21.42 PF 5 2,397.14 MF/Watt 4,000

4 3,000 kW, Instantaneous 3 2,000 RmaxPower 2 kW-hours, cumulative 1,000 1

Run Time Duration (hh:mm:ss) - -

3 0:00:53 0:01:53 0:02:53 0:03:53 0:04:53 0:05:53 0:06:53 0:07:53 0:08:53 0:09:53 0:10:53 0:11:53 0:12:53 0:13:53 0:14:53 0:15:53 0:16:53 0:17:53 0:18:53 0:19:53 0:20:53 0:21:53 0:22:53 0:23:53 0:24:53 0:25:53 0:26:53 0:27:53 0:28:53 0:29:53 0:30:53 0:31:53 0:32:53 0:33:53 0:34:53 0:35:53 0:36:53 0:37:53 0:38:53 0:39:53 0:40:53 0:41:53 0:42:53 0:43:53 0:44:53 ORNL0:45:53 0:46:53 0:47:53 ISC’130:48:53 0:49:53 0:50:53 0:51:53 0:52:53 0:53:53 0:54:53 Revised Metering Capabilities for HPC Systems at ORNL

• Each of the seven (7) 2.5 MVA or 3.0 MVA Cabinet/rack transformers designated for HPC use is

Chassis/crate now metered by a separate Schneider Electric CM4000 PowerLogic Circuit blade Monitor CPU • “F” Position within the EEHPCWG A Aspect 4 Category Power • Highly accurate power quality monitor Conv. PDU E Distribution for critical energy systems. Panel Substantially higher B D performance/capability than original equipment. F • Adds very accurate voltage transient Power Power Conv. Conv. and flicker analysis features. Building Transformer 4 ORNL ISC’13 Energy Efficient HPC System Workload Power Measurement Methodology

• ORNL was an early adopter/participant ✓ • The Nov’12 Titan measurement met ✓ Level 2 Aspect 1a/1b and Level 3 ✓ aspect 2/3/4 requirements ✓ • New meters support all Level 3 ✓ requirements for all HPC systems.

✓

5 ORNL ISC’13