Parallel and Distributed Computing Chapter 1: Introduction to Parallel Computing Jun Zhang Department of Computer Science University of Kentucky Lexington, KY 40506 Chapter 1: CS621 1 1.1a: von Neumann Architecture Common machine model for over 70 years Stored-program concept CPU executes a stored program A sequence of read and write operations on the memory Order of operations is sequential Chapter 1: CS621 2 1.1b: A More Detailed Architecture based on von Neumann Model Chapter 1: CS621 3 1.1c: Old von Neumann Computer Chapter 1: CS621 4 1.1d: CISC von Neumann Computer CISC stands for Complex Instruction Set Computer with a single bus system Harvard (RISC) architecture utilizes two buses, a separate data bus and an address bus RISC stands for Reduced Instruction Set Computer They are SISD machines – Single Instruction Stream on Single Data Stream Chapter 1: CS621 5 1.1e: Personal Computer Chapter 1: CS621 6 1.1f: John von Neumann December 28, 1903 – February 8, 1957 Hungarian mathematician Mastered calculus at 8 Graduate level math at 12 Got his Ph.D. at 23 His proposal to his 1st wife, “You and I might be able to have some fun together, seeing as how we both like to drink." Chapter 1: CS621 7 1.2a: Motivations for Parallel Computing Fundamental limits on single processor speed Heat dissipation from CPU chips Disparity between CPU & memory speeds Distributed data communications Need for very large scale computing platforms Chapter 1: CS621 8 1.2b: Fundamental Limits – Cycle Speed Intel 8080 2MHz 1974 ARM 2 8MHz 1986 Intel Pentium Pro 200MHz 1996 AMD Athlon 1.2GHz 2000 Intel QX6700 2.66GHz 2006 Intel Core i7 3770k 3.9GHz 2013 Speed of light: 30cm in 1ns Signal travels about 10 times slower Chapter 1: CS621 9 1.2c: High-End CPU is Expensive Price for high-end CPU rises sharply Intel processor price/performance Chapter 1: CS621 10 1.2d: Moore’s Law Moore’s observation in 1965: the number of transistors per square inch on integrated circuits had doubled every year since the integrated circuit was invented Moore’s revised observation in 1975: the pace was slowed down a bit, but data density had doubled approximately every 18 months How about the future? (price of computing power falls by a half every 18 months?) Chapter 1: CS621 11 1.2e: Moore’s Law – Held for Now Chapter 1: CS621 12 1.3: Power Wall Effect in Computer Architecture Too many transistors in a given chip die area Tremendous increase in power density Increased chip temperature High temperature slows down the transistor switching rate and the overall speed of the computer Chip may melt down if not cooled properly Efficient cooling systems are expensive Chapter 1: CS621 13 1.3: Cooling Computer Chips Some people suggest to put computer chips in liquid nitrogen to cool them Chapter 1: CS621 14 1.3: Solutions Use multiple inexpensive processors A processor with multiple cores Chapter 1: CS621 15 1.3: A Multi-core Processor Chapter 1: CS621 16 1.3a: CPU and Memory Speeds In 20 years, CPU speed (clock rate) has increased by a factor of 1000 DRAM speed has increased only by a factor of smaller than 4 How to feed data faster enough to keep CPU busy? CPU speed: 1-2 ns DRAM speed: 50-60 ns Cache: 10 ns Chapter 1: CS621 17 1.3b: Memory Access and CPU Speed Chapter 1: CS621 18 1.3b: CPU, Memory, and Disk Speed Chapter 1: CS621 19 1.3c: Possible Solutions A hierarchy of successively fast memory devices (multilevel caches) Location of data reference (code) Efficient programming can be an issue Parallel systems may provide 1.) larger aggregate cache 2.) higher aggregate bandwidth to the memory system Chapter 1: CS621 20 1.3f: Multilevel Hierarchical Cache Chapter 1: CS621 21 1.4a: Distributed Data Communications Data may be collected and stored at different locations It is expensive to bring them to a central location for processing Many computing assignments may be inherently parallel Privacy issues in data mining and other large scale commercial database manipulations Chapter 1: CS621 22 1.4b: Distributed Data Communications Chapter 1: CS621 23 1.4c: Move Computation to Data (CS626: Large Scale Data Science) Chapter 1: CS621 24 1.5a: Why Use Parallel Computing Save time – reduce wall clock time – many processors work together Solve larger problems – larger than one processor’s CPU and memory can handle Provide concurrency – do multiple things at the same time: online access to databases, search engine Google’s 4,000 PC servers were one of the largest clusters in the world (server farm) Chapter 1: CS621 25 1.4b: Google’s Data Center Chapter 1: CS621 26 1.5b: Other Reasons for Parallel Computing Taking advantages of non-local resources – using computing resources on a wide area network, or even internet (grid or cloud computing) Cost savings – using multiple “cheap” computing resources instead of a high-end CPU Overcoming memory constraints – for large problems, using memories of multiple computers may overcome the memory constraint obstacle Chapter 1: CS621 27 1.6a: Need for Large Scale Modeling Long term weather forecasting Large scale ocean modeling Oil reservoir simulations Car and airplane manufacturing Semiconductor simulation Pollution tracking Large scale commercial databases Aerospace (NASA microgravity modeling) Chapter 1: CS621 28 1.6b: Semiconductor Simulation Before 1975, an engineer had to make several runs through the fabrication line until a successful device was fabricated Device dimensions shrink below 0.1 micro-meter A fabrication line costs 1.0 billion dollars to build A design must be thoroughly verified before it is committed to silicon A realistic simulation for one diffusion process may take days or months to run on a workstation Chip price drops quickly after entering the market Chapter 1: CS621 29 1.4b: Semiconductor Diffusion Process Chapter 1: CS621 30 1.6c: Drug Design Most drugs work by binding to a specific site, called a receptor, on a protein A central problem is to find molecules (ligands) with high binding affinity Need to accurately and efficiently estimate electrostatic forces in molecular and atomic interactions Calculate drug-protein binding energies from quantum mechanics, statistical mechanics and simulation techniques Chapter 1: CS621 31 1.6d: Computing Protein Binding Chapter 1: CS621 32 1.6d: Computer Aided Drug Design Chapter 1: CS621 33 1.7: Issues in Parallel Computing Design of parallel computers Design of efficient parallel algorithms Methods for evaluating parallel algorithms Parallel computer languages Parallel programming tools Portable parallel programs Automatic programming of parallel computers Education of parallel computing philosophy Chapter 1: CS621 34 1.8 Eclipse Parallel Tools Platform A standard, portable parallel integrated development environment that supports a wide range of parallel architectures and run time systems (IBM) A scalable parallel debugger Support for the integration of a wide range of parallel tools An environment that simplifies the end-user interaction with parallel systems Chapter 1: CS621 35 1.9 Message Passing Interface (MPI) We will use MPI for hands-on experience in this class Message Passing Interface can be downloaded from an online website (MPICH) Parallel computing can be simulated on your own computers (Microsoft MPI) UK can provide distributed computing services for research purposes, for free Chapter 1: CS621 36 1.10 Cloud Computing Cloud Computing is a style of computing in which dynamically scalable and often virtualized resources are provided over the Internet Users need not have knowledge of, expertise in, or control over the technology infrastructure in the “cloud” that supports them Compared to: Grid Computing (cluster of networked, loosely coupled computers) Utility Computing (packaging of computing resources as a metered service) and Autonomic Computing (computer systems capable of self-management) Chapter 1: CS621 37 1.11 Cloud Computing By Sam Johnston in Wikimedia Commons Chapter 1: CS621 38 1.12 Cloud Computing A means to increase computing capacity or add computing capabilities at any time without investing in new infrastructure, training new personnel, or licensing new software Chapter 1: CS621 39 1.13 Cloud Computing Chapter 1: CS621 40 1.14 Cloud Computing Chapter 1: CS621 41 1.15 Cloud Computing Chapter 1: CS621 42 1.16 Top Players in Cloud Computing Amazon.com (IaaS, most comprehesive) Vmware (vCloud, software to build cloud) Microsoft (PaaS, IaaS) Salesforce.com (SaaS) Google (IaaS, PaaS, Google App Engine) Rackspace (IaaS) IBM (Openstack) Citrix (compete with Vmware, free cloud operating system) Joyent (compete with Vmware, OpenStack, Citrix) SoftLayer (web hosting service provider) Chapter 1: CS621 43 1.17 Parallel and Distributed Computing (CS621) This class is about parallel and distributed computing algorithms and applications Algorithms for communications between processors Algorithms to solve scientific computing problems Hands-on experience with message-passing interface (MPI) to write parallel programs to solve some problems This class is NOT about parallel data processing (CS626) Chapter 1: CS621 44 *** Hands-on Experience You can down and install a copy of Microsoft MPI (Message Passing Interface) at https://www.microsoft.com/en- us/download/details.aspx?id=57467 You also need MS Visual Studio to work with it. https://visualstudio.microsoft.com/ Chapter 1: CS621 45 *** Hands-on Experience Here is a video demonstrating how to set up MS Visual Studio for MS MPI programming https://www.youtube.com/watch?v=IW3SKDM_yEs&t=330s Please make sure that you have installed the MS Visual Studio and MS MPI on your computer Please run the sample MPI “Hello World” program https://mpitutorial.com/tutorials/mpi-hello-world/ Chapter 1: CS621 46 *** Hands-on Experience If you have serious research projects that need a lot of computing resources, you can also request an account on a supercomputer at the UK Center for Computational Science Go to the Center for Computational Science and get a form to fill out and ask your advisor to sign it https://www.ccs.uky.edu/ Chapter 1: CS621 47.