MIMD Overview! Intel Paragon XP/S Overview!

!! MIMDs in the 1980s and 1990s" !! Distributed-memory MIMD multicomputer"

!! Distributed-memory multicomputers" !! 2D array of nodes" !! Intel Paragon XP/S" !! Thinking Machines CM-5" !! Main memory physically distributed !! IBM SP2" among nodes (16-64 MB / node)"

!! Distributed-memory multicomputers with !! Each node contains two Intel i860 XP hardware to look like shared-memory" processors: application processor to run

!! nCUBE 3" user program, and message processor for inter-node communication" !! Kendall Square Research KSR1"

!! NUMA shared-memory multiprocessors" !! Cray T3D" !! Convex Exemplar SPP-1000" !! Silicon Graphics POWER & Origin"

!! General characteristics"

!! 100s of powerful commercial RISC PEs"

!! Wide variation in PE interconnect network"

!! Broadcast / reduction / synch network"

1! Fall 2008, MIMD! 2! Fall 2008, MIMD!

XP/S Nodes and Interconnection! XP/S Usage!

!! Node composition" !! System OS is based on UNIX, provides distributed system services and full !! 16–64 MB of memory" UNIX to every node" !! Application processor" !! System is divided into partitions, some for !! Intel i860 XP processor (42 MIPS, 50 MHz clock) to execute user programs" I/O, some for system services, rest for user applications" !! Message processor" !! Intel i860 XP processor" !! Users have client/server access, can !! Handles details of sending / receiving a submit jobs over a network, or login message between nodes, including directly to any node" protocols, packetization, etc." !! Supports broadcast, synchronization, and reduction (sum, min, and, or, etc.)" !! System has a MIMD architecture, but supports various programming models: !! 2D mesh interconnection between nodes" SPMD, SIMD, MIMD, shared memory, vector shared memory " !! Paragon Mesh Routing Chip (PMRC) / iMRC routes traffic in the mesh" !! Applications can run on arbitrary number !! 0.75 µm, triple-metal CMOS" of nodes without change" !! Routes traffic in four directions and to and from attached node at > 200 MB/s" !! Run on more nodes for large data sets or to get higher performance"

3! Fall 2008, MIMD! 4! Fall 2008, MIMD! Thinking Machines CM-5 Overview! CM-5 Partitions / Control Processors!

!! Distributed-memory MIMD multicomputer" !! Processing nodes may be divided into (communicating) partitions, and are !! SIMD or MIMD operation" supervised by a control processor"

!! Configurable with up to 16,384 !! Control processor broadcasts blocks of processing nodes and 512 GB of instructions to the processing nodes" memory" !! SIMD operation: control processor broadcasts instructions and nodes are !! Divided into partitions, each managed by closely synchronized" a control processor" !! MIMD operation: nodes fetch instructions !! Processing nodes use SPARC CPUs" independently and synchronize only as required by the algorithm"

!! Control processors in general"

!! Schedule user tasks, allocate resources, service I/O requests, accounting, etc."

!! In a small system, one control processor may play a number of roles"

!! In a large system, control processors are often dedicated to particular tasks (partition manager, I/O cont. proc., etc.)"

5! Fall 2008, MIMD! 6! Fall 2008, MIMD!

CM-5 Nodes and Interconnection! Tree Networks (Reference Material)!

!! Processing nodes" !! Binary Tree" k !! SPARC CPU (running at 22 MIPS)" !! 2 –1 nodes arranged into complete binary tree of depth k–1" !! 8-32 MB of memory" !! Diameter is 2(k–1)" !! (Optional) 4 vector processing units" !! Bisection width is 1" !! Each control processor and processing node connects to two networks" !! Hypertree"

!! Control Network — for operations that !! Low diameter of a binary tree plus involve all nodes at once" improved bisection width" !! Broadcast, reduction (including parallel !! Hypertree of degree k and depth d" prefix), barrier synchronization" !! From “front”, looks like k-ary tree of height d" !! Optimized for fast response & low latency" !! From “side”, looks like upside-down binary !! Data Network — for bulk data transfers tree of height d" between specific source and destination" !! Join both views to get complete network" !! 4-ary hypertree" !! 4-ary hypertree of depth d" !! Provides point-to-point communication for d d d+1 !! 4 leaves and 2 (2 –1) nodes" tens of thousands of items simultaneously" !! Diameter is 2d" !! Special cases for nearest neighbor" d+1 !! Bisection width is 2 " !! Optimized for high bandwidth"

7! Fall 2008, MIMD! 8! Fall 2008, MIMD! IBM SP2 Overview! SP2 System Architecture!

!! Distributed-memory MIMD multicomputer" !! RS/6000 as system console"

!! Scalable POWERparallel 1 (SP1)" !! SP2 runs various combinations of serial, parallel, interactive, and batch jobs" !! Scalable POWERparallel 2 (SP2)" !! Partition between types can be changed" !! RS/6000# !! High nodes — interactive nodes for code workstation# development and job submission" plus 4–128# POWER2# !! Thin nodes — compute nodes" processors" !! Wide nodes — configured as servers, !! POWER2# with extra memory, storage devices, etc." processors# used IBM$s# !! A system “frame” contains 16 thin in RS 6000# processor or 8 wide processor nodes" workstations,# compatible# !! Includes redundant power supplies, with existing# nodes are hot swappable within frame" software" !! Includes a high-performance switch for low-latency, high-bandwidth communication"

9! Fall 2008, MIMD! 10! Fall 2008, MIMD!

SP2 Processors and Interconnection! nCUBE 3 Overview!

!! POWER2 processor" !! Distributed-memory MIMD multicomputer (with hardware to make it look like !! RISC processor, load-store architecture, various versions from 20 to 62.5 MHz" shared-memory multiprocessor)" !! !! Comprised of 8 semi-custom chips: If access is attempted to a virtual memory Instruction Cache, 4 Data Cache,# page marked as “non-resident”, the Fixed-Point Unit, Floating-Point Unit,# system will generate messages to and Storage Control Unit" transfer that page to the local node"

!! !! Interconnection network" nCUBE 3 could have 8–65,536 processors and up to 65 TB memory" !! Routing" !! Can be partitioned into “subcubes”" !! Packet switched = each packet may take a different route" !! Cut-through = if output is free, starts !! Multiple programming paradigms: sending without buffering first" SPMD, inter-subcube processing, client !! Wormhole routing = buffer on subpacket /server" basis if buffering is necessary"

!! Multistage High Performance Switch (HPS) network, scalable via extra stages to keep bw to each processor constant"

!! Guaranteed fairness of message delivery" 11! Fall 2008, MIMD! 12! Fall 2008, MIMD! nCUBE 3 Processor and Interconnect! nCUBE 3 I/O!

!! Processor" !! ParaChannel I/O array"

!! 64-bit custom processor" !! Separate network of nCUBE processors" !! 0.6 µm, 3-layer CMOS, 2.7 million !! 8 computational nodes connect directly to transistors, 50 MHz, 16 KB data cache, one ParaChannel node" 16 KB instruction cache, 100 MFLOPS" !! ALU, FPU, virtual memory management !! ParaChannel nodes can connect to RAID unit, caches, SDRAM controller, 18-port mass storage, SCSI disks, etc." message router, and 16 DMA channels" !! One I/O array can be connected to more –! ALU for integer operations, FPU for than 400 disks" floating point operations"

!! Argument against off-the-shelf processor: shared memory, vector floating-point units, aggressive caches are necessary in MediaCUBE Overview! workstation market but superfluous here" !! For delivery of interactive video to client !! Interconnect" devices over a network (from LAN -based training to video-on-demand to !! Hypercube interconnect" homes)" !! Wormhole routing + adaptive routing around blocked or faulty nodes" !! MediaCUBE 30 = 270 1.5 Mbps data streams, 750 hours of content"

!! MediaCUBE 3000 = 20,000 & 55,000"

13! Fall 2008, MIMD! 14! Fall 2008, MIMD!

Kendall Square Research KSR1 KSR1 System Architecture! Overview and Processor!

!! COMA distributed-memory MIMD !! The ALLCACHE system moves an multicomputer (with hardware to make it address set requested by a processor to look like shared-memory multiprocessor)" the Local Cache on that processor"

!! Provides the illusion of a single !! Multiple variations" sequentially-consistent shared memory" !! 8 cells ($500K): 320 MFLOPS, 256 MB memory, 210 GB disk, 210 MB/s I/O" !! Memory space consists of all the 32 KB local caches" !! 1088 cells ($30M): 43 GFLOPS, 34 GB memory, 15 TB disk, 15 GB/s I/O" !! No permanent location for an “address”"

!! Addresses are distributed and based on !! Each APRD (ALLCACHE Processor, processor need and usage patterns" Router, and Directory) Cell contains:" !! Each processor is attached to a Search !! Custom 64-bit integer and floating-point Engine, which finds addresses and their processors (1.2 µm, 20 MHz, 450,000 contents and moves them to the local transistors, on a 8x13 printed circuit cache, while maintaining cache board)" coherence throughout the system" !! 32 MB of local cache" !! 2 levels of search groups for scalability"

!! Support chips for cache, I/O, etc."

15! Fall 2008, MIMD! 16! Fall 2008, MIMD! Cray T3D Overview! T3D Nodes and Interconnection!

!! NUMA shared-memory MIMD !! Node contains 2 PEs; each PE contains:" multiprocessor" !! DEC Alpha 21064 microprocessor" !! Each processor has a local memory, but !! 150 MHz, 64 bits, 8 KB L1 I&D caches" the memory is globally addressable" !! Support for L2 cache, not used in favor of improving latency to main memory" ! ! DEC Alpha 21064 processors arranged !! 16–64 MB of local DRAM" into a virtual 3D torus (hence the name)" !! Access local memory: latency 87–253ns" !! 32–2048 processors, 512MB–128GB of !! Access remote memory: 1–2µs (~8x)" memory" !! Alpha has 43 bits of virtual address !! Parallel vector# space, only 32 bits for physical address processor (Cray# space — external registers in node Y-MP / C90) used# provide 5 more bits for 37 bit phys. addr." as host computer,# runs the scalar # !! 3D torus connections PE nodes and I/O / vector parts# gateways" of the program" !! Dimension-order routing: when a !! 3D torus is# message leaves a node, it first travels in virtual, includes# the X dimension, then Y, then Z" redundant nodes"

17! Fall 2008, MIMD! 18! Fall 2008, MIMD!

Cray T3E Overview! Convex Exemplar SPP-1000 Overview!

!! T3D = 1993,# !! ccNUMA shared-memory MIMD" T3E = 1995 successor (300 MHz, $1M),# !! 4–128 HP PA 7100 RISC processors, 256 T3E-900 = 1996 model (450 MHz, $.5M)" MB –%32 GB memory"

! !! T3E system = 6–2048 processors, 3.6 ! Hardware support for remote memory –1228 GFLOPS,1–4096 GB memory" access"

!! PE = DEC Alpha 21164 processor# !! System is comprised of up to 16 (300 MHz, 600 MFLOPS, quad issue),# “hypernodes”, each of which contains# local memory, control chip, router chip" 8 processors and# !! L2 cache is on-chip so can$t be eliminated, 4 cache memories# but off-chip L3 can and is" (each 64–512MB)# !! 512 external registers per process" connected by a# !! GigaRing Channel attached to each node crossbar switch " and to I/O devices and other networks" !! Hypernodes# !! T3E-900 = same w/ faster processors,# are connected# up to 1843 GFLOPS" in a ring"

!! Hardware keeps# !! Ohio Supercomputer Center (OSC) had caches consistent# a T3E with 128 PEs (300 MHz),# with each other" 76.8 GFLOPS, 128 MB memory / PE" 19! Fall 2008, MIMD! 20! Fall 2008, MIMD! Silicon Graphics" Silicon Graphics" POWER CHALLENGEarray Overview! Origin 2000 Overview!

!! ccNUMA shared-memory MIMD" !! ccNUMA shared-memory MIMD"

!! SGI says they supply 95% of ccNUMA !! “Small” supercomputers" systems worldwide" !! POWER CHALLENGE — up to 144 MIPS R8000 processors or 288 MISP R1000 !! Various models, 2–128 MIPS R10000 processors, with up to 128 GB memory processors, 16 GB – 1 TB memor" and 28 TB of disk" !! Processing node board contains two !! POWERnode system — shared-memory R10000 processors, part of the shared multiprocessor of up to 18 MIPS R8000 memory, directory for cache coherence, processors or 36 MIPS R1000 plus node # processors, with up to 16 GB of memory" and I/O# interface" !! POWER CHALLENGEarray consists of up to 8 POWER CHALLENGE or !! File serving,# POWERnode systems" data mining,# media serving,# !! Programs that fit within a POWERnode can use the shared-memory model" high-# performance# !! Larger program can span POWERnodes" computing"

21! Fall 2008, MIMD! 22! Fall 2008, MIMD!