Cluster Computing: The Cluster Computing Commodity Supercomputing Lecture 3 By Mark Baker and Rajkumar Buyya Software - Practice and Experience vol. 1, no. 1, Jan 1988 Clusters Clusters Commodity supercomputers Important factor making the use of Built using commodity HW and SW workstations practical: components ØStandardization of tools and utilities Playing a major role in redefining the § MPI and HPF § Allows applications to be developed and concept of supercomputing tested in a cluster and ported to a parallel platform when ready Clusters Clusters Why are clusters preferred over MPPs? Basic definition Ø Workstations are becoming powerful Ø A cluster is a collection of workstations or PCs Ø Bandwidth between workstations is increasing that are interconnected via some network Ø Workstation clusters are easier to integrate technology into existing networks Likely scenario Ø Typical lower user utilization Ø Computers will be state-of-the-art Ø Development tools are more mature Ø Network will be high-bandwidth and low-latency Ø Workstation clusters are cheap and readily available Such a cluster can provide fast and reliable Ø Clusters can be enlarged and individual nodes services to computationally intensive can have their capabilities extended applications 1 Clusters Clusters Topics to be discussed Clusters can be classified as ØComponents ØDedicated cluster ØTools ØNon-dedicated clusters ØTechniques ØMethodologies Clusters Clusters Dedicated clusters Non-Dedicated clusters ØA particular individual does not own a ØIndividuals own workstations workstation ØApplications are executed by stealing ØResources are shared cycles ØParallel computing can be performed ØTension between owner and remote user across the entire cluster ØImportant issues: migration and load balance Clusters Commodity Components Suitable for applications that are not Processors and Memory communication intensive Disk I/O ØTypically low bandwidth and high latency Cluster Interconnects Workstations Operating Systems ØSome sort of Unix platform Windows of Opportunity ØLately, PCs running Linux or NT 2 Processors Processors Today, single-chip CPUs are almost as Related projects powerful as processors used in ØDigital Alpha 21364 processor supercomputers of the recent past § Integrate processing, memory controller, and network interface into a single chip Processors Processors Related projects Processors used in Clusters ØThe Berkeley Intelligent Ram project ØDigital Alpha – Alpha Farm § Exploring the entire spectrum of issues ØIBM Power PC – IBM SP involved in a general-purpose computer ØSun SPARC – Berkeley NOW system that integrates a processor and DRAM onto a single chip ØSGI MIPS ØHP PA Memory The System Bus Amount of memory required varies Needs to match the system’s clock with the target application speed Parallel programs distribute data Intel PCI Bus: 133MBps throughout the nodes ØUsed in Pentium based PCs There should be enough memory to ØDigital AlphaServers avoid constant swapping Decreasing the distinction between Caches are key: 8KB to 2MB PCs and workstations 3 Commodity Components Disk I/O Processors and Memory Disk density is increasing 60-80% Disk I/O every year Disk access time have not kept pace Cluster Interconnects with microprocessor performance Operating Systems Parallel/Grand Challenge applications Windows of Opportunity need to process large amount of data Necessary to improve I/O performance Disk I/O Commodity Components Way of improving Processors and Memory ØCarry out I/O operations concurrently Disk I/O with the support of a parallel file system Cluster Interconnects ØCan be constructed by using the disks associated with each workstation in the Operating Systems cluster Windows of Opportunity Cluster Interconnects Cluster Interconnects Individual nodes in a cluster are Requirements to balance the usually connected with a high-speed computational power of the low-latency high-bandwidth network workstations available Communication uses ØBandwidth: more than 10MBps ØStandard Network protocol: TCP/IP ØLatency: at most 100us ØLow level protocol: Active Messages or Fast Messages 4 Cluster Interconnects Cluster Interconnects Network Technologies Ethernet ØFast Ethernet ØCheap and widely used to form clusters ØATM Standard Ethernet ØMyrinet Ø10Mbps – not enough Fast Ethernet Ø100Mbps – meets the requirement Cluster Interconnects Cluster Interconnects ATM (Asynchronous Transfer Mode) ATM (Asynchronous Transfer Mode) ØSwitched virtual circuit technology ØUsually, no optical fiber in desktops ØDeveloped for telecommunication ØATM on CAT-5 ØIntended to be used for LAN and WAN § 15.5 MBps § Presents an unified approach to both § Allows upgrades of existing networks ØBased around small-fixed size packets without replacing cabling ØDesigned for a number of media § Example: copper wire and optic fiber § Performance varies with the hardware Cluster Interconnects Cluster Interconnects SCI – Scalable Coherent Interface SCI – Scalable Coherent Interface ØIEEE 1596 Standard Ø Point-to-point architecture Ø Directory-based cache coherent ØLow-latency high-bandwidth distributed Ø shared memory access across a network Faster than any network technology available Ø Scalability depends on switches ØProvides a scalable architecture that Ø Expensive allows large systems to be built out of Ø Produced for SPARC Sbus and PCI Based inexpensive mass produced components systems 5 Cluster Interconnects Commodity Components Myrinet Processors and Memory Ø1.28 Gbps full duplex LAN supplied by Disk I/O Myricom Cluster Interconnects ØBased on cut-through switches Operating Systems ØProprietary and high-performance § Low latency and high bandwidth Windows of Opportunity ØUsed in expensive clusters Operating Systems Operating Systems Modern operating systems Solaris ØMultitask ØFrom Sun ØMultithreading at kernel level ØUnix-based multithreaded and multi-user ØUser-level high-performance system multithreading without kernel Ø intervention Supports Intel x86 and SPARC platforms ØNetwork support Ø ØMost popular Network support includes TCP/IP stack, § Solaris, Linux, and Windows NT RPC and NFS Operating Systems Operating Systems Solaris Linux ØProgramming environment includes: Ø Unix-like operating system Ø § ANSI compliant C and C++ compilers Developed by Linus Torvalds, a Finnish undergraduate student in 91-92 § Tools to profile and debug multithreaded Ø programs Open source and free § Later, lots of contribution from other programmers § Wide range of SW tools, libraries and utilities Ø Robust, reliable, POSIX compliant 6 Operating Systems Operating Systems Linux Linux: Why is it so popular? Ø FREE! Available from the Internet and can be ØPre-emptive multi-tasking downloaded without cost ØDemand-paged virtual memory Ø Runs on cheap x86 platforms, yet offers the ØMulti-user support power and flexibility of Unix Ø Easy to fix debugs and improve system ØMulti-processor support performance Ø Users can develop or fine-tune HW drivers and these can be made easily available to other users Ø Applications and system software are freely available (for example: GNU software) Operating Systems Operating Systems NT NT Ø 32-bit pre-emptive multitasking and multi-user ØMicrosoft Corporation is the dominant operating system provider of SW in the personal Ø Fault tolerant: each 32-bit application operates computing market place in its own virtual memory address space ØIN 1996, NT and Windows 95 had Ø Complete OS together 66% of the desktop OS market Ø Supports most CPU architectures share Ø Supports multiprocessor machines through the use of threads Ø Network protocols and services are integrated with the base OS Commodity Components Windows of Opportunity Processors and Memory The resources available in the Disk I/O average NOW offer a number of Cluster Interconnects research opportunities: ØParallel processing Operating Systems ØNetwork RAM for virtual memory Windows of Opportunity ØSoftware RAID ØMulti-path communication 7 Programming Tools Message Passing Systems For HPC on Clusters Message Passing Libraries allow ØMessage Passing Systems efficient parallel programs to be § PVM, MPI written for distributed memory systems ØDistributed Shared Memory Systems ØParallel Debuggers and Profilers Provide routines to initiate and configure the messaging environment ØPerformance Analysis Tools Provide functions for sending and Ø Cluster Monitoring receiving data Message Passing Systems Message Passing Systems Two most popular high-level message- PVM passing systems for scientific and ØEnvironment and message-passing library engineering applications ØDesigned to run parallel applications on ØPVM systems ranging from high-end ØMPI defined by the MPI forum supercomputers to clusters of workstations Message Passing Systems Message Passing Systems MPI MPICH Ø Specification for message-passing ØMost popular of the current free Ø Designed to be standard for distributed implementations of MPI memory parallel computing using explicit message passing ØDeveloped at Argonne National Ø Attempts to establish a practical, portable, Laboratory and Mississipi State efficient, and flexible standard for message passing Ø Available on most of the HPC systems including SMP machines 8 Message Passing Systems Message Passing Systems MPICH MPICH ØPortable, built on top of a restricted ØADI, basic point-to-point message number of HW-independent low-level passing functions, which form the ADI ØRemaining MPICH, management of ØADI, Abstract Device Interface communicators, derived data types, contains 25 functions collective