History of Supercomputing

Parallel Computing (MC-8836)

Esteban Meneses, PhD School of Computing Costa Rica Institute of Technology [email protected]

I semester, 2021 Marenostrum Supercomputer, Barcelona Supercomputing Center (BSC) What is a Supercomputer? Elusive definition

a computer at the frontline of contemporary processing capacity - particularly speed of calculation (Wikipedia)

a large and very fast computer (Merriam-Webster)

a machine that hierarchically integrates many components to accelerate the execution of particular programs (Esteban Meneses)

a device for turning compute-bound problems into I/O problems

(Ken Batcher)

Parallel Computing (Meneses) History 3 Supercomputer Features

• They are expensive • They represent a major enhancement in computational power • They have a short life (approximately 5 years) • Their performance is measured in FLOPS.

Parallel Computing (Meneses) History 4 First Digital Computers Are they really super-computers?

Colossus (1944) ENIAC (1946) MARK 1 (1948)

First Electronic Digital First Electronic Stored-program Computer Programmable Computer, General-purpose Computer University of Manchester, Bletchley Park, United University of Pennsylvania, United Kingdom Kingdom United States

Parallel Computing (Meneses) History 5 CDC 6600 Generally regarded as the first supercomputer

• Designed by Seymour Cray (the father of supercomputing) • Built at Control Data Corporation • Single CPU, RISC-like system • Memory was faster than 1964 CPU 3 megaFLOPS • Used Minnesota $60 million today FORTRAN

Parallel Computing (Meneses) History 6 ILLIAC IV First massively parallel computer

• It featured 64 processing entities • Designed at the University of Illinois • Built by Burroughs • Parallelism based on vector processing 1971 • 64-bit registers with 64, 200 megaFLOPS 32, and 8-bit formats $O(10) million today • Development of parallelizing compilers for FORTRAN

Parallel Computing (Meneses) History 7 Bell’s Law Corollary of Moore’s Law

Roughly every decade a new, lower priced computer class forms based on a new programming platform, network, and interface resulting in new usage and the establishment of a new industry

Gordon Bell, 1972

Computer classes: • 1960’s: Mainframes • 1970’s: Minicomputers • 1980’s: LAN workstations • 1990’s: Internet web-based computing • 2000’s: Cloud computing • 2010’s: Mobile computing • 2020’s: Virtual (augmented) reality Gordon Bell

Parallel Computing (Meneses) History 8 Cray 1 Most successful supercomputer

• Designed by Seymour Cray at Cray Research • Closely packed integrated circuits • Liquid cooling • Instruction pipelines, vectorization • Cray FORTRAN 1976 parallelizing compiler 136 megaFLOPS $25 million today

Parallel Computing (Meneses) History 9 Cray Research Supercomputers First successful supercomputing company

Cray X-MP Cray 2

1982 1985 200 megaFLOPS 1.9 gigaFLOPS $32 million today $35 million today

First multiprocessor supercomputer First gigascale supercomputer with (4 processors) with shared memory foreground (4) and background (4) processors using mainstream software (UniCOS)

Parallel Computing (Meneses) History 10 The Cambrian Explosion Supercomputer Galore Supercomputing companies exploded in early 80s

Several different designs were explored, two of them persist today: • Symmetric Multiprocessor System (SMP) • Massively Parallel Processor (MPP)

Parallel Computing (Meneses) History 12 Top500 Ranks supercomputers since 1993

• Released twice per year (June and November) at ISC and SC conferences, respectively • Performance measured in FLOPS • LINPACK, dense linear algebra benchmark: Ax = b

• Other rankings: Green500, Graph500, GreenGraph500, HPCG, HPL-AI

Parallel Computing (Meneses) History 13 Japanese Supercomputers The supercomputing rising sun

Numerical Wind Tunnel Earth Simulator

Top 1 from 1993 to 1996 Top 1 from 2002 to 2004 170.4 gigaFLOPS 35.82 teraFLOPS

Built by Fujitsu for the National Aerospace Built by NEC for Earth Simulation Center, Laboratory, with 166 superscalar vector with 5120 vector processors processors

Parallel Computing (Meneses) History 14 ASCI Red First terascale supercomputer

• First supercomputer made with off-the-shelf processors • Developed entirely by Intel for Advanced Simulation and Computer Initiative 1996 1.3 teraFLOPS • Based on Intel Paragon $67 million today • It assembled 6,000 Intel Pentium Pro processors at 200MHz

Parallel Computing (Meneses) History 15 Beowulf Clusters Built with commodity hardware

• Beowulf is a hero in an epic English poem • Thomas Sterling and Donald Becker at NASA built the first prototype in 1994 • Key technologies: • Ethernet interconnect • Linux operating system Democratization of • Message-passing Interface supercomputing (MPI) programming • Beowulf manual, 1998

Parallel Computing (Meneses) History 16 IBM Supercomputers The successful Blue Gene family

BG/L BG/P BG/Q

• Low-power consumption • Slow RISC processors integrated in huge quantities • First supercomputer to use more than 100,000 sockets • No use of accelerators

Parallel Computing (Meneses) History 17 IBM Blue Gene Architecture Hierarchical design

Parallel Computing (Meneses) History 18 Roadrunner First petascale supercomputer

• Built by IBM and installed at Los Alamos National Laboratory • First hybrid supercomputer • 6,480 AMD Opteron processors • 12,960 IBM PowerXCell 8i 2008 accelerators 1.456 petaFLOPS • Used to simulate aging of $100 million nuclear weapons and ensure its safety and reliability

Parallel Computing (Meneses) History 19 Blue Waters First sustained-petascale supercomputer

• Powerful machine owned by an academic institution • Petascale performance while executing a combination of applications • Built by Cray at the 2012 University of Illinois 13.3 petaFLOPS • Never appeared on the $188 million Top500 list • 15% of the machine is hybrid: AMD processors and NVIDIA GPUs

Parallel Computing (Meneses) History 20 Chinese Supercomputers The new kid on the block of supercomputing

Tianhe-1A Tianhe-2 Sunway TaihuLight

Top 1 in 2010 Top 1 2013-2015 Top 1 2016-2018 2.5 petaFLOPS 33.86 petaFLOPS 93.01 petaFLOPS Hybrid architecture (Intel Hybrid architecture (Intel Uses homebrewed Chinese Xeon processors and processor with 260 cores NVIDIA M2050 GPUs) Xeon processors and Intel Xeon Phi co-processors) at $390 million cost

Parallel Computing (Meneses) History 21 Summit An IBM accelerator-based system

Each of its 4608 nodes has: • Two POWER9 22-core processors • Six NVIDIA V100 cards • 600 GB coherent memory accesible by both CPU and GPU • 800 GB of non-volatile RAM 2019 • 200 Gbps InfiniBand connection 148.6 petaFLOPS $325 million program

Parallel Computing (Meneses) History 22 Fugaku Knocking at exascale’s door

• Claimed to be first exascale supercomputer, although using mixed precision • Built at RIKEN Center for Computational Science • Fujitsu A64FX microprocessor 2019 based on ARM version 8.2A 442.01 petaFLOPS • Tofu interconnect with $1 billion program six-dimensional torus topology

Parallel Computing (Meneses) History 23 Top Supercomputers As of November 2020

Parallel Computing (Meneses) History 24 Supercomputing Growth Using Top500 historical data

Parallel Computing (Meneses) History 25 Supercomputing in Costa Rica Co-evolution of usage and procurement

Universidad de Costa Rica

• Matilde (IBM 16-20, 1968, $70 thousand)

• Clotilde (IBM 360, 1974, $1 million)

• School of Physics (circa 2000, beowulf)

• CIGEFI supercomputer (400-core cluster)

Centro Nacional de Alta Tecnolog´ıa

• Kabr´esupercomputer

• Multiple areas: simulation, data science, bioinformatics, machine learning

• Diverse uses: research, training, academic courses

Parallel Computing (Meneses) History 26 Future Supercomputers Technologies in the post-Moore era

Neuromorphic Quantum Computing Graphene Chips Computing

Parallel Computing (Meneses) History 27 Acknowledgments

• Inspired by the Computer History Museum, Mountain View, California. • Most images were extracted from www.wikipedia.org or www.top500.org • Some data from lectures notes by Dr. Albert DeFusco.

Parallel Computing (Meneses) History 28