Power Efficiency and Performance with ORNL's Cray XK7 Titan Work by Jim Rogers Presented by Arthur Bland Director of Operations National Center for Computational Sciences Oak Ridge National Laboratory ORNL’s “Titan” Hybrid System: Cray XK7 with AMD Opteron and NVIDIA Tesla processors SYSTEM SPECIFICATIONS: • Peak performance of 27.1 PF • 24.5 GPU + 2.6 CPU • 18,688 Compute Nodes each with: • 16-Core AMD Opteron CPU • NVIDIA Tesla “K20x” GPU • 32 + 6 GB memory • 512 Service and I/O nodes 2 4,352 ft • 200 Cabinets 2 404 m • 710 TB total system memory • Cray Gemini 3D Torus Interconnect • 8.9 MW peak power 2 ORNL ISC’13 ORNL's Cray XK7 Titan | Sample Run: HPL Consumption MW, Instantaneous kW-hours (Cumulative) 10 8,000 7,545.56kW-hr 9 7,000 8 RmaxPower = 8,296.53kW 6,000 7 Instantaneous Measurements 5,000 6 8.93 MW 21.42 PF 5 2,397.14 MF/Watt 4,000 4 3,000 kW, Instantaneous 3 2,000 RmaxPower 2 kW-hours, cumulative 1,000 1 Run Time Duration (hh:mm:ss) - - 3 0:00:53 0:01:53 0:02:53 0:03:53 0:04:53 0:05:53 0:06:53 0:07:53 0:08:53 0:09:53 0:10:53 0:11:53 0:12:53 0:13:53 0:14:53 0:15:53 0:16:53 0:17:53 0:18:53 0:19:53 0:20:53 0:21:53 0:22:53 0:23:53 0:24:53 0:25:53 0:26:53 0:27:53 0:28:53 0:29:53 0:30:53 0:31:53 0:32:53 0:33:53 0:34:53 0:35:53 0:36:53 0:37:53 0:38:53 0:39:53 0:40:53 0:41:53 0:42:53 0:43:53 0:44:53 ORNL0:45:53 0:46:53 0:47:53 ISC’130:48:53 0:49:53 0:50:53 0:51:53 0:52:53 0:53:53 0:54:53 Revised Metering Capabilities for HPC Systems at ORNL • Each of the seven (7) 2.5 MVA or 3.0 MVA Cabinet/rack transformers designated for HPC use is Chassis/crate now metered by a separate Schneider Electric CM4000 PowerLogic Circuit blade Monitor CPU • “F” Position within the EEHPCWG A Aspect 4 Category Power • Highly accurate power quality monitor Conv. PDU E Distribution for critical energy systems. Panel Substantially higher B D performance/capability than original equipment. F • Adds very accurate voltage transient Power Power Conv. C Conv. and flicker analysis features. Building Transformer 4 ORNL ISC’13 Energy Efficient HPC System Workload Power Measurement Methodology • ORNL was an early adopter/participant ✓ • The Nov’12 Titan measurement met ✓ Level 2 Aspect 1a/1b and Level 3 ✓ aspect 2/3/4 requirements ✓ • New meters support all Level 3 ✓ requirements for all HPC systems. ✓ 5 ORNL ISC’13 .