Power Efficiency and Performance with ORNL's Cray XK7 Titan

Total Page：16

File Type：pdf, Size：1020Kb

Power Efficiency and Performance with ORNL's Cray XK7 Titan Work by Jim Rogers Presented by Arthur Bland Director of Operations National Center for Computational Sciences Oak Ridge National Laboratory ORNL’s “Titan” Hybrid System: Cray XK7 with AMD Opteron and NVIDIA Tesla processors SYSTEM SPECIFICATIONS: • Peak performance of 27.1 PF • 24.5 GPU + 2.6 CPU • 18,688 Compute Nodes each with: • 16-Core AMD Opteron CPU • NVIDIA Tesla “K20x” GPU • 32 + 6 GB memory • 512 Service and I/O nodes 2 4,352 ft • 200 Cabinets 2 404 m • 710 TB total system memory • Cray Gemini 3D Torus Interconnect • 8.9 MW peak power 2 ORNL ISC’13 ORNL's Cray XK7 Titan | Sample Run: HPL Consumption MW, Instantaneous kW-hours (Cumulative) 10 8,000 7,545.56kW-hr 9 7,000 8 RmaxPower = 8,296.53kW 6,000 7 Instantaneous Measurements 5,000 6 8.93 MW 21.42 PF 5 2,397.14 MF/Watt 4,000 4 3,000 kW, Instantaneous 3 2,000 RmaxPower 2 kW-hours, cumulative 1,000 1 Run Time Duration (hh:mm:ss) - - 3 0:00:53 0:01:53 0:02:53 0:03:53 0:04:53 0:05:53 0:06:53 0:07:53 0:08:53 0:09:53 0:10:53 0:11:53 0:12:53 0:13:53 0:14:53 0:15:53 0:16:53 0:17:53 0:18:53 0:19:53 0:20:53 0:21:53 0:22:53 0:23:53 0:24:53 0:25:53 0:26:53 0:27:53 0:28:53 0:29:53 0:30:53 0:31:53 0:32:53 0:33:53 0:34:53 0:35:53 0:36:53 0:37:53 0:38:53 0:39:53 0:40:53 0:41:53 0:42:53 0:43:53 0:44:53 ORNL0:45:53 0:46:53 0:47:53 ISC’130:48:53 0:49:53 0:50:53 0:51:53 0:52:53 0:53:53 0:54:53 Revised Metering Capabilities for HPC Systems at ORNL • Each of the seven (7) 2.5 MVA or 3.0 MVA Cabinet/rack transformers designated for HPC use is Chassis/crate now metered by a separate Schneider Electric CM4000 PowerLogic Circuit blade Monitor CPU • “F” Position within the EEHPCWG A Aspect 4 Category Power • Highly accurate power quality monitor Conv. PDU E Distribution for critical energy systems. Panel Substantially higher B D performance/capability than original equipment. F • Adds very accurate voltage transient Power Power Conv. C Conv. and flicker analysis features. Building Transformer 4 ORNL ISC’13 Energy Efficient HPC System Workload Power Measurement Methodology • ORNL was an early adopter/participant ✓ • The Nov’12 Titan measurement met ✓ Level 2 Aspect 1a/1b and Level 3 ✓ aspect 2/3/4 requirements ✓ • New meters support all Level 3 ✓ requirements for all HPC systems. ✓ 5 ORNL ISC’13 .

Copy Link

Recommended publications

ORNL Debuts Titan Supercomputer Table of Contents
ReporterRetiree Newsletter December 2012/January 2013 SCIENCE ORNL debuts Titan supercomputer ORNL has completed the installation of Titan, a supercomputer capable of churning through more than 20,000 trillion calculations each second—or 20 petaflops—by employing a family of processors called graphic processing units first created for computer gaming. Titan will be 10 times more powerful than ORNL’s last world-leading system, Jaguar, while overcoming power and space limitations inherent in the previous generation of high- performance computers. ORNL is now home to Titan, the world’s most powerful supercomputer for open science Titan, which is supported by the DOE, with a theoretical peak performance exceeding 20 petaflops (quadrillion calculations per second). (Image: Jason Richards) will provide unprecedented computing power for research in energy, climate change, efficient engines, materials and other disciplines and pave the way for a wide range of achievements in “Titan will provide science and technology. unprecedented computing Table of Contents The Cray XK7 system contains 18,688 nodes, power for research in energy, with each holding a 16-core AMD Opteron ORNL debuts Titan climate change, materials 6274 processor and an NVIDIA Tesla K20 supercomputer ............1 graphics processing unit (GPU) accelerator. and other disciplines to Titan also has more than 700 terabytes of enable scientific leadership.” Betty Matthews memory. The combination of central processing loves to travel .............2 units, the traditional foundation of high- performance computers, and more recent GPUs will allow Titan to occupy the same space as Service anniversaries ......3 its Jaguar predecessor while using only marginally more electricity. “One challenge in supercomputers today is power consumption,” said Jeff Nichols, Benefits ..................4 associate laboratory director for computing and computational sciences.
[Show full text]
Safety and Security Challenge
SAFETY AND SECURITY CHALLENGE TOP SUPERCOMPUTERS IN THE WORLD - FEATURING TWO of DOE’S!! Summary: The U.S. Department of Energy (DOE) plays a very special role in In fields where scientists deal with issues from disaster relief to the keeping you safe. DOE has two supercomputers in the top ten supercomputers in electric grid, simulations provide real-time situational awareness to the whole world. Titan is the name of the supercomputer at the Oak Ridge inform decisions. DOE supercomputers have helped the Federal National Laboratory (ORNL) in Oak Ridge, Tennessee. Sequoia is the name of Bureau of Investigation find criminals, and the Department of the supercomputer at Lawrence Livermore National Laboratory (LLNL) in Defense assess terrorist threats. Currently, ORNL is building a Livermore, California. How do supercomputers keep us safe and what makes computing infrastructure to help the Centers for Medicare and them in the Top Ten in the world? Medicaid Services combat fraud. An important focus lab-wide is managing the tsunamis of data generated by supercomputers and facilities like ORNL’s Spallation Neutron Source. In terms of national security, ORNL plays an important role in national and global security due to its expertise in advanced materials, nuclear science, supercomputing and other scientific specialties. Discovery and innovation in these areas are essential for protecting US citizens and advancing national and global security priorities. Titan Supercomputer at Oak Ridge National Laboratory Background: ORNL is using computing to tackle national challenges such as safe nuclear energy systems and running simulations for lower costs for vehicle Lawrence Livermore's Sequoia ranked No.
[Show full text]
Titan: a New Leadership Computer for Science
Titan: A New Leadership Computer for Science Presented to: DOE Advanced Scientific Computing Advisory Committee November 1, 2011 Arthur S. Bland OLCF Project Director Office of Science Statement of Mission Need • Increase the computational resources of the Leadership Computing Facilities by 20-40 petaflops • INCITE program is oversubscribed • Programmatic requirements for leadership computing continue to grow • Needed to avoid an unacceptable gap between the needs of the science programs and the available resources • Approved: Raymond Orbach January 9, 2009 • The OLCF-3 project comes out of this requirement 2 ASCAC – Nov. 1, 2011 Arthur Bland INCITE is 2.5 to 3.5 times oversubscribed 2007 2008 2009 2010 2011 2012 3 ASCAC – Nov. 1, 2011 Arthur Bland What is OLCF-3 • The next phase of the Leadership Computing Facility program at ORNL • An upgrade of Jaguar from 2.3 Petaflops (peak) today to between 10 and 20 PF by the end of 2012 with operations in 2013 • Built with Cray’s newest XK6 compute blades • When completed, the new system will be called Titan 4 ASCAC – Nov. 1, 2011 Arthur Bland Cray XK6 Compute Node XK6 Compute Node Characteristics AMD Opteron 6200 “Interlagos” 16 core processor @ 2.2GHz Tesla M2090 “Fermi” @ 665 GF with 6GB GDDR5 memory Host Memory 32GB 1600 MHz DDR3 Gemini High Speed Interconnect Upgradeable to NVIDIA’s next generation “Kepler” processor in 2012 Four compute nodes per XK6 blade. 24 blades per rack 5 ASCAC – Nov. 1, 2011 Arthur Bland ORNL’s “Titan” System • Upgrade of existing Jaguar Cray XT5 • Cray Linux Environment
[Show full text]
EVGA Geforce GTX TITAN X Superclocked
EVGA GeForce GTX TITAN X Superclocked Part Number: 12G-P4-2992-KR The EVGA GeForce GTX TITAN X combines the technologies and performance of the new NVIDIA Maxwell architecture in the fastest and most advanced graphics card on the planet. This incredible GPU delivers unrivaled graphics, acoustic, thermal and power-efficient performance. The most demanding enthusiast can now experience extreme resolutions up to 4K-and beyond. Enjoy hyper-realistic, real-time lighting with advanced NVIDIA VXGI, as well as NVIDIA G-SYNC display technology for smooth, tear-free gaming. Plus, you get DSR technology that delivers a brilliant 4K experience, even on a 1080p display. SPECIFICATIONS KEY FEATURES RESOLUTION & REFRESH Base Clock: 1127 MHZ NVIDIA Dynamic Super Resolution Technology Max Monitors Supported: 4 Boost Clock: 1216 MHz NVIDIA MFAA Technology 240Hz Max Refresh Rate Memory Clock: 7010 MHz Effective NVIDIA GameWorks Technology Max Analog: 2048x1536 CUDA Cores: 3072 NVIDIA GameStream Technology Max Digital: 4096x2160 Bus Type: PCI-E 3.0 NVIDIA G-SYNC Ready Memory Detail: 12288MB GDDR5 Microsoft DirectX 12 REQUIREMENTS Memory Bit Width: 384 Bit NVIDIA GPU Boost 2.0 600 Watt or greater power supply.**** Memory Speed: 0.28ns NVIDIA Adaptive Vertical Sync PCI Express, PCI Express 2.0 or PCI Express Memory Bandwidth: 336.5 GB/s NVIDIA Surround Technology 3.0 compliant motherboard with one graphics NVIDIA SLI Ready slot. DIMENSIONS NVIDIA CUDA Technology An available 6-pin PCI-E power connector and Height: 4.376in - 111.15mm OpenGL 4.4 Support an available 8 pin PCI-E power connector Length: 10.5in - 266.7mm OpenCL Support Windows 8 32/64bit, Windows 7 32/64bit, Windows Vista 32/64bit Width: Dual Slot HDMI 2.0, DisplayPort 1.2 and Dual-link DVI PCI Express 3.0 **Support for HDMI includes GPU-accelerated Blu-ray 3D support (Blu-ray 3D playback requires the purchase of a compatible software player from CyberLink, ArcSoft, Corel, or Sonic), x.v.Color, HDMI Deep Color, and 7.1 digital surround sound.
[Show full text]
A Comprehensive Performance Comparison of Dedicated and Embedded GPU Systems
DUJE (Dicle University Journal of Engineering) 11:3 (2020) Page 1011-1020 Research Article A Comprehensive Performance Comparison of Dedicated and Embedded GPU Systems Adnan Ozsoy 1,* 1 Department of Computer Engineering, Hacettepe University, [email protected] (ORCID: https://orcid.org/0000-0002-0302-3721) ARTICLE INFO ABSTRACT Article history: General purpose usage of graphics processing units (GPGPU) is becoming increasingly important as Received 26 May 2020 Received in revised form 29 June 2020 graphics processing units (GPUs) get more powerful and their widespread usage in performance-oriented Accepted 9 July 2020 computing. GPGPUs are mainstream performance hardware in workstation and cluster environments and Available online 30 September 2020 their behavior in such setups are highly analyzed. Recently, NVIDIA, the leader hardware and software vendor in GPGPU computing, started to produce more energy efficient embedded GPGPU systems, Jetson Keywords: NVIDIA Jetson, series GPUs, to make GPGPU computing more applicable in domains where energy and space are limited. Embedded GPGPU, CUDA Although, the architecture of the GPUs in Jetson systems is the same as the traditional dedicated desktop graphic cards, the interaction between the GPU and the other components of the system such as main memory, central processing unit (CPU), and hard disk, is a lot different than traditional desktop solutions. To fully understand the capabilities of the Jetson series embedded solutions, in this paper we run several applications from many different domains and compare the performance characteristics of these applications on both embedded and dedicated desktop GPUs. After analyzing the collected data, we have identified certain application domains and program behaviors that Jetson series can deliver performance comparable to dedicated GPU performance.
[Show full text]
Harnessing the Power of Titan with the Uintah Computational Framework
Alan Humphrey, Qingyu Meng, Brad Peterson, Martin Berzins Scientific Computing and Imaging Institute, University of Utah I. Uintah Framework – Overview II. Extending Uintah to Leverage GPUs III. Target Application – DOE NNSA PSAAP II Multidisciplinary Simulation Center IV. A Developing GPU-based Radiation Model V. Summary and Questions Central Theme: Shielding developers from complexities inherent in heterogeneous systems like Titan & Keeneland Thanks to: John Schmidt, Todd Harman, Jeremy Thornock, J. Davison de St. Germain Justin Luitjens and Steve Parker, NVIDIA DOE Titan – 20 Petaflops DOE for funding the CSAFE project from 1997-2010, 18,688 GPUs DOE NETL, DOE NNSA, INCITE, ALCC NSF for funding via SDCI and PetaApps, XSEDE NSF Keeneland Keeneland Computing Facility 792 GPUs Oak Ridge Leadership Computing Facility for access to Titan DOE NNSA PSAPP II (March 2014) Parallel, adaptive multi-physics framework Fluid-structure interaction problems Patch-based AMR: Particle system and mesh-based fluid solve Plume Fires Chemical/Gas Mixing Foam Explosions Angiogenesis Compaction Sandstone Compaction Industrial Flares Shaped Charges MD – Multiscale Materials Design Patch-based domain decomposition Asynchronous ALCF Mira task-based paradigm Task - serial code on generic “patch” OLCF Titan Task specifies desired halo region Clear separation of user code from parallelism/runtime Strong Scaling: Uintah infrastructure provides: Fluid-structure interaction problem using MPMICE algorithm w/ AMR • automatic MPI message generation • load balancing •
[Show full text]
Digital Signal Processor Supercomputing
Digital Signal Processor Supercomputing ENCM 515: Individual Report Prepared by Steven Rahn Submitted: November 29, 2013 Abstract: Analyzing the history of supercomputers: how the industry arrived to where it is in its current form and what technological advancements enabled the movement. Explore the applications of Digital Signal Processor in high performance computing and whether or not tests have already taken place. Examine whether Digital Signal Processors have a future with supercomputers. i Table of Contents Table of Contents ................................................................................................................................... ii List of Figures ........................................................................................................................................ iii Introduction ........................................................................................................................................... 1 What is a supercomputer? ................................................................................................................. 1 What are supercomputers used for? ................................................................................................. 1 History of Supercomputing .................................................................................................................... 2 The Past (1964 – 1985) ....................................................................................................................... 2 The Present
[Show full text]
AI Chips: What They Are and Why They Matter
APRIL 2020 AI Chips: What They Are and Why They Matter An AI Chips Reference AUTHORS Saif M. Khan Alexander Mann Table of Contents Introduction and Summary 3 The Laws of Chip Innovation 7 Transistor Shrinkage: Moore’s Law 7 Efficiency and Speed Improvements 8 Increasing Transistor Density Unlocks Improved Designs for Efficiency and Speed 9 Transistor Design is Reaching Fundamental Size Limits 10 The Slowing of Moore’s Law and the Decline of General-Purpose Chips 10 The Economies of Scale of General-Purpose Chips 10 Costs are Increasing Faster than the Semiconductor Market 11 The Semiconductor Industry’s Growth Rate is Unlikely to Increase 14 Chip Improvements as Moore’s Law Slows 15 Transistor Improvements Continue, but are Slowing 16 Improved Transistor Density Enables Specialization 18 The AI Chip Zoo 19 AI Chip Types 20 AI Chip Benchmarks 22 The Value of State-of-the-Art AI Chips 23 The Efficiency of State-of-the-Art AI Chips Translates into Cost-Effectiveness 23 Compute-Intensive AI Algorithms are Bottlenecked by Chip Costs and Speed 26 U.S. and Chinese AI Chips and Implications for National Competitiveness 27 Appendix A: Basics of Semiconductors and Chips 31 Appendix B: How AI Chips Work 33 Parallel Computing 33 Low-Precision Computing 34 Memory Optimization 35 Domain-Specific Languages 36 Appendix C: AI Chip Benchmarking Studies 37 Appendix D: Chip Economics Model 39 Chip Transistor Density, Design Costs, and Energy Costs 40 Foundry, Assembly, Test and Packaging Costs 41 Acknowledgments 44 Center for Security and Emerging Technology | 2 Introduction and Summary Artificial intelligence will play an important role in national and international security in the years to come.
[Show full text]
Supercomputer Titan to Get World's Fastest Storage System 17 April 2013, by Bob Yirka
Supercomputer Titan to get world's fastest storage system 17 April 2013, by Bob Yirka $60 million. New additions, including the beefed up storage system, have raised that price to nearly $100. It's a big machine—it takes up 404 square meters of floor space, and consumes 8.2 MW of electricity. At peak times it can perform 27,000 trillion calculations per second courtesy of 18,688 AMD Opteron CPUs and 18,688 Nvidia Tesla K20X GPUs—it's ten times faster than any prior computer at ORNL. It also has 710TB of memory at its disposal. Credit: Oak Ridge National Laboratory The new upgrade will help scientists conduct research efforts that ORNL reps say either cannot be done any other way, or are too dangerous or (Phys.org) —Officials at Oak Ridge National costly. Expected uses for the computer will likely Laboratory (ORNL) have announced the selection involve running applications that typically involve an of the Spider II data storage and retrieval system enormous amount of data crunching such as from DataDirect Networks (DDN) to replace the developing models for weather or economic existing system on the Titan supercomputer. They forecasting, searching for oil deposits or putting say it will give Titan the fastest such system in the together information from millions of sources to world. help track and stop terrorist activity. Currently, ORNL has chosen 31 projects to run this year. The Titan supercomputer (built by Cray Inc.) on the ORNL campus was named the fastest in the world © 2013 Phys.org in November of last year, and currently still holds that title.
[Show full text]
An Analysis of System Balance and Architectural Trends Based on Top500 Supercomputers
ORNL/TM-2020/1561 An Analysis of System Balance and Architectural Trends Based on Top500 Supercomputers Hyogi Sim Awais Khan Sudharshan S. Vazhkudai Approved for public release. Distribution is unlimited. August 11, 2020 DOCUMENT AVAILABILITY Reports produced after January 1, 1996, are generally available free via US Department of Energy (DOE) SciTech Connect. Website: www.osti.gov/ Reports produced before January 1, 1996, may be purchased by members of the public from the following source: National Technical Information Service 5285 Port Royal Road Springfield, VA 22161 Telephone: 703-605-6000 (1-800-553-6847) TDD: 703-487-4639 Fax: 703-605-6900 E-mail: [email protected] Website: http://classic.ntis.gov/ Reports are available to DOE employees, DOE contractors, Energy Technology Data Ex- change representatives, and International Nuclear Information System representatives from the following source: Office of Scientific and Technical Information PO Box 62 Oak Ridge, TN 37831 Telephone: 865-576-8401 Fax: 865-576-5728 E-mail: [email protected] Website: http://www.osti.gov/contact.html This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal lia- bility or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or rep- resents that its use would not infringe privately owned rights. Refer- ence herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not nec- essarily constitute or imply its endorsement, recommendation, or fa- voring by the United States Government or any agency thereof.
[Show full text]
Titan Is Also a Green Powerhouse 14 November 2012
Titan is also a green powerhouse 14 November 2012 modestly more power than a CPU-only system. "With Titan, ORNL provides an exponential increase in scientific computing power toward predictive capabilities to address complex challenges without significantly increasing energy consumption," noted Jeff Nichols, ORNL associate laboratory director for computing and computational sciences. The Cray XK7 system contains 18,688 nodes, each with a 16-core AMD Opteron 6274 processor and an NVIDIA Tesla K20X graphics processing unit (GPU) accelerator. Titan also has more than 700 terabytes of memory. Not only is Oak Ridge National Laboratory's Titan the Because they handle hundreds of calculations world's fastest supercomputer, it is also ranked third in energy efficiency. simultaneously, GPUs can go through many more than CPUs in a given time. By relying on its 299,008 CPU cores to guide simulations and allowing its new NVIDIA GPUs to do the heavy Not only is Oak Ridge National Laboratory's Titan lifting, Titan will enable researchers to run scientific the world's most powerful supercomputer, it is also calculations with greater speed and accuracy. one of the most energy-efficient. Titan came in at number three on the Green500 Provided by Oak Ridge National Laboratory list. Organized by Virginia Tech's Wu-chun Feng and Kirk Cameron, the list takes the world's 500 most powerful supercomputers—as ranked by the Top500 list—and reorders them according to how many calculations they can get per watt of electricity. The Green500 list was announced Wednesday during the SC12 supercomputing conference in Salt Lake City.
[Show full text]
What Is a Core Hour on Titan?
What is a Core Hour on Titan? or 2013 the Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program awarded 1.84 billion coreF hours on Titan, a hybrid-architecture high-performance computing (HPC) system that sports central processing units (CPUs) and graphics processing units (GPUs), which are radically different, in the same machine. So what exactly does it mean to allocate a core hour on Titan? The Oak Ridge Leadership Computing Facility (OLCF) operates Titan based on nodes, or functional units in a message-passing network that can be assigned to work together on a specific task, such as calculating how the Earth’s climate might change over a decade. Each node on Titan runs a single copy of the Cray Linux operating system. Likewise, a node’s cores can work together to run a simulation. Because communication is faster within the components of a single node than between nodes, data is structured so calculations The OLCF is home to Titan, the world’s most powerful stay within nodes as much as possible. For individual projects running on Titan, different nodes can be supercomputer for open assigned to run concurrently. science with a theoretical peak performance exceeding Whereas nodes are the functional units of supercomputers, cores are used to explain how users get 20 petaflops (quadrillion “charged” for time on Titan. For each hour a node runs calculations on Titan, the OLCF assesses users calculations per second). an accounting charge of 30 core hours. The number 30 is the sum of the 16 x86 CPU cores on each That kind of computational capability—almost node’s CPU microprocessor and the 14 streaming multiprocessors on each node’s GPU processor.
[Show full text]