High Performance Computing : Concepts, Methods, & Means An Introduction Prof. Thomas Sterling Department of Computer Science Louisiana State University January 16, 2007 The Hammer of the Mind • The Hammer – Mankind’s 1 st tool – In the most general case: applies a directed force to a concentrated point in our physical world to affect a desired change of state – Many implements of the physical world • Conventional means of inserting nails to wood • Includes knives, spears, arrows, screwdrivers, sledge-hammers, axes, clubs, etc. • Understanding – The “force” that drives our abstract world – Historically, 2 means by which the mind applies understanding • Empiricism – acquiring knowledge through experience • Theory – project beyond immediate experience to new knowledge • Supercomputing – The 3 rd hammer of the mind for applying understanding – Explain the past – Predict the future – Control the present 2 Topics • Supercomputing – the big picture • What is a supercomputer? • Supercomputing as a multidisciplinary field • Challenges and Opportunities • A brief history of supercomputing • Overview of Course • Segment 1 knowledge factors & skills • Resources and rules of engagement 3 Topics • Supercomputing – the big picture • What is a supercomputer? • Supercomputing as a multidisciplinary field • Challenges and Opportunities • A brief history of supercomputing • Overview of Course • Segment 1 knowledge factors & skills • Resources and rules of engagement 4 Applying the Force of Understanding through the Power of Supercomputing 5 Addressing the Big Questions • How to integrate technology into computing engines? • How to push the performance to extremes? – What are the enabling conditions? – What are the inhibiting factors? • How to manage supercomputer resources to deliver useful computing capabilities? – What are the hardware mechanisms? – What are the software policies? • How do users program such systems? – What languages and in what environments? – What are the semantics and strategies? • What grand challenge applications demand these capabilities? • What are the computational models and algorithms that can map the innate application properties to the physical medium of the machine? 6 Challenges in the Physical World Command Our Abilities in the Abstract • Physical Sciences • Technology • Biology and Medical Science • Energy • Meteorology and Climate • Materials and Nanotechnology • National Security 7 A Growth-Factor of a Billion in Performance in a Single Lifetime 1959 IBM 7094 1976 1991 1996 2003 1949 Cray 1 Intel Delta T3E Cray X1 1Edsac 10 3 10 6 10 9 10 12 10 15 One OPS KiloOPS MegaOPS GigaOPS TeraOPS PetaOPS 1823 2001 1943 1951 1964 1982 1988 1997 Babbage Difference Univac 1 CDC 6600 Cray XMP Cray YMP ASCI Red Earth Engine Harvard Simulator Mark 1 8 Performance: a cross-cutting issue the Top-500 list of supercomputers 2.3 PF/s 1 Pflop/s 280.6 TF/s 100 Tflop/s SUM IBM BlueGene/L 10 Tflop/s NEC 1.167 TF/s N=1 Earth Simulator IBM ASCI White 1.646 TF/s 1 Tflop/s LLNL 59.7 GF/s Intel ASCI Red Sandia 100 Gflop/s Fujitsu 'NWT' NAL 10 Gflop/s N=500 0.4 GF/s 1 Gflop/s 100 Mflop/s 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 9 Topics • Supercomputing – the big picture • What is a supercomputer? • Supercomputing as a multidisciplinary field • Challenges and Opportunities • A brief history of supercomputing • Overview of Course • Segment 1 knowledge factors & skills • Resources and rules of engagement 10 Definitions: “supercomputer” Supercomputer : A computing system exhibiting high-end performance capabilities and resource capacities within practical constraints of technology, cost, power, and reliability. Thomas Sterling, 2007 Supercomputer: a large very fast mainframe used especially for scientific computations. Merriam-Webster Online Supercomputer : any of a class of extremely powerful computers. The term is commonly applied to the fastest high-performance systems available at any given time. Such computers are used primarily for scientific and engineering work requiring exceedingly high-speed computations. Encyclopedia Britannica Online 11 Performance • Performance : – A quantifiable measure of rate of doing (computational) work – Multiple such measures of performance • Delineated at the level of the basic operation – ops – operations per second – ips – instructions per second – flops – floating operations per second • Rate at which a benchmark program takes to execute – A carefully crafted and controlled code used to compare systems – Linpack Rmax (Linpack flops) – gups (billion updates per second) – others • Two perspectives on performance – Peak performance • Maximum theoretical performance possible for a system – Sustained performance • Observed performance for a particular workload and run • Varies across workloads and possibly between runs 12 Key Parameters • Peak floating point performance • Main memory capacity • Bi-section bandwidth • I/O bandwidth • Secondary storage capacity • Organization – Class of system – # nodes – # processors per node – Accelerators – Network topology • Control strategy – MIMD – Vector, PVP – SIMD – SPMD 13 Scalability • The ability to deliver proportionally greater sustained performance through increased system resources • Strict Scaling – Fixed size application problem – Application size remains constant with increase in system size • Weak Scaling – Variable size application problem – Application size scales proportionally with system size • Capability computing – in most pure form: strict scaling – Marketing claims tend toward this class • Capacity computing – Throughput computing • Includes job-stream workloads – In most simple form: weak scaling • Cooperative computing – Interacting and coordinating concurrent processes – Not a widely used term – Also: coordinated computing 14 Practical Constraints and Limitations • Cost – Deployment – Operational support • Power – Energy required to run the computer – Energy for support facilities – Energy for cooling (remove heat from machine) • Size – Floor space – Access way for power and signal cabling • Reliability – One factor of availability • Generality – How good is it across a range of problems • Usability – How hard is it to program and manage 15 Productivity – a computing metric of merit • A rich measure of merit for computing – Captures key factors that determine overall impact – Exposes relationship between program development and program execution – Supercedes mere scalar parameters of assumed performance • Focuses attention on all (most) important contributors to overall effectiveness • Permits cogent comparative assessment of alternative system classes • Devised as part of DARPA HPCS Program Phase 1 – T. Sterling – M. Snir – B. Smith – and others 16 Productivity Factors Directed Graph Peak Performance (S P, C M) Performance Efficiency (E) Programmability Application Productivity Portability Construction (Ψ) (C ) S Maintainability Availability Reliability (A) Accessibility 17 General Model of Productivity th NR Ri ≡ i result product TR = Ti Ti ≡ time to compute result Ri ∑ i TL ≡ total lifetime of machine TL = TR +TV +TQ TV ≡ total overhead time of machine NR TQ ≡ quiescent time of machine RL = ∑ Ri TR ≡ working time of machine i NR TL ≡ total number of result products during CL = CLS + CM + CLO CL ≡ all costs associated with machine during TL NR CLS ≡ application software costs during TL CLS = ∑CSi CLO ≡ costs of ownership during TL i CM ≡ cost of procurement and initial installation RL Ψ ≡ CSi ≡ cost of application software for result Ri CL ×TL Ψ ≡ productivity 18 Topics • Supercomputing – the big picture • What is a supercomputer? • Supercomputing: a multidisciplinary field • Challenges and Opportunities • A brief history of supercomputing • Overview of Course • Segment 1 knowledge factors & skills • Resources and rules of engagement 19 Related Fields • Hardware – Device technologies – Logic circuit designs – Architecture • Software – System software – Programming methodologies • End user application problems – Problem area disciplines – Computational algorithms • Cross-cutting issues – Performance – Products and Market drivers – People – Packaging: cost, space, power, reliability 20 Supercomputing: A Discipline of Disciplines in this course • Device technologies – Enabling technologies for logic, memory, & communication – Circuit design • Computer architecture – semantics and structures • Programming – languages, tools, & environments • Models of computation – governing principles • Compilers and runtime software – Maps application program to system resources, mechanisms, and semantics • Operating systems – Manages resources and provides virtual machine • Performance – modeling, measurement, benchmarking, and debugging • Algorithms – Numerical techniques – Means of exposing parallelism • Applications – End user problems, often in sciences and technology 21 Topics • Supercomputing – the big picture • What is a supercomputer? • Supercomputing as a multidisciplinary field • Challenges and Opportunities • A brief history of supercomputing • Overview of Course • Segment 1 knowledge factors & skills • Resources and rules of engagement 22 Where Does Performance Come From? • Device Technology – Logic switching speed and device density – Memory capacity and access time – Communications bandwidth and latency • Computer Architecture – Instruction issue rate • Execution pipelining • Reservation stations • Branch prediction • Cache management – Parallelism • Parallelism – number of operations per cycle per processor – Instruction