DOCSLIB.ORG

A Survey of Cache Coherence Schemes for Multiprocessors

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
A Survey of Cache Coherence Schemes for Multiprocessors A Survey of Cache Coherence-Schemes for Multiprocessors Per Stenstrom Lund University hared-memory multiprocessors as opposed to shared, because each cache have emerged as an especially cost- is private to one or a few of the total effective way to provide increased number of processors. computing power and speed, mainly be- Cache coherence The private cache organization appears cause they use low-cost microprocessors in a number of multiprocessors, including economically interconnected with shared schemes tackle the Encore Computer’s Multimax, Sequent memory modules. problem Computer Systems’ Symmetry, and Digi- Figure 1 shows a shared-memory multi- of tal Equipment’s Firefly multiprocessor processor consisting of processors con- maintaining data workstation. These systems use a common nected with the shared memory modules bus as the interconnection network. Com- by an interconnection network. This sys- consistency in munication contention therefore becomes tem organization has three problems1: shared-memory a primary concem, and the cache serves mainly to reduce bus contention. (1) Memory contention. Since a mem- multiprocessors. Other systems worth mentioning are ory module can handle only one memory RP3 from IBM, Cedar from the University request at a time, several requests from They rely on of Illinois at Urbana-Champaign, and different processors will be serialized. software, hardware, Butterfly from BBN Laboratories. These (2) Communication contention. Con- systems contain about 100 processors tention for individual links in the intercon- or a combination connected to the memory modules by a nection network can result even if requests multistage interconnection network with a are directed to different memory modules. of both. considerable latency. RP3 and Cedar also (3) Latency time. Multiprocessors with use caches to reduce the average memory a large number of processors tend to have access time. complex interconnection networks. The la- Shared-memory multiprocessors have tency time for such networks (that is, the an advantage: the simplicity of sharing time a memory request takes to traverse the code and data structures among the pro- network) is long. lows the cache to perform a vast majority cesses comprising the parallel application. of all memory requests (typically more Process communication, for instance, can These problems all contribute to increased than 95 percent); memory handles only a be implemented by exchanging informa- memory access times and hence slow down small fraction. It is therefore not surprising tion through shared variables. This sharing the processors’ execution speeds. that multiprocessor architects also have can result in several copies of a shared Cache memories have served as an im- employed cache techniques to address the block in one or more caches at the same portant way to reduce the average memory problems pointed out above. Figure 2 time. To maintain a coherent view of the access time in uniprocessors. The locality shows a multiprocessor organization with memory, these copies must be consistent. of memory references over time (temporal caches attached to all processors. This This is the cache coherenceproblem or the locality) and space (spatial locality) al- cache organization is often called private, cache consistency problem. A large num- 12 WIB-9162/wRMn-W12$01.W 0 1990 IEEE COMPUTER ber of solutions to this problem have been proposed. This article surveys schemes for cache coherence. These schemes exhibit various degrees of hardware complexity, ranging from protocols that maintain coherence in hardware to software policies that prevent the existence of copies of shared, writable data. First we’ll look at some examples of how shared data is used. These examples help point out a number of performance issues. Then we’ll look at hardware proto- cols. We’ll see that consistency can be maintained efficiently, but in some cases with considerable hardware complexity, especially for multiprocessors with many processors. We’ll investigate software schemes as an altemative capable of reduc- ing the hardware cost. Figure 1. An example of a shared memory multiprocessor.’ Example of algorithms with data sharing Cache coherence poses a problem mainly for shared, read-write data struc- tures. Read-only data structures (such as shared code) can be safely replicated with- out cache coherence enforcement mecha- nisms. Private, read-write data structures might impose a cache coherence problem Interconnection network if we allow processes to migrate from one processor to another. Many commercial multiprocessors help increase throughput for multiuser operating systems where user processes execute independently with no pq pq (or little) data sharing. In this case, we need ... to efficiently maintain consistency for pri- Ipi vate, read-write data in the context of proc- IPI ess migration. We will concentrate on the behavior of Figure 2. An example of a multiprocessor with private caches.’ cache coherence schemes for parallel ap- plications using shared, read-write data structures. To understand how the schemes work and how they perform for different uses of shared data structures, we will Producer: Consumer: investigate two parallel applications that if count <= N then if count <> 0 then use shared data structures differently. mutexbegin mutexbegin These examples highlight a number of buffer[ in ] := item; item := buffer[ out 1; performance issues. in := in + 1 mod N; out := out + 1 mod N count :=count 1; We can find the first example - the count := count + 1; - well-known bounded-buffer producer and mutexend mutexend consumer problem - in any ordinary text on operating systems. Figure 3 shows it in IFigure 3. Pascal-like code for the bounded-buffer problem. a Pascal-like notation. The producer in- serts a data item in the shared buffer if the buffer is not full. The buffer can store N items. The consumer removes an item if the buffer is not empty. We can choose the three shared variables: in, out, and count, and mutexend) protect buffer operations, number of producers and consumers arbi- which keep track of the next item and the which means that one process at most can trarily. The buffer is managed by a shared number of items stored in the buffer. enter the critical section at a time. array, which implements the buffer, and Semaphores (implemented by mutexbegin The second example to consider is a June 1990 13 Cache coherence policies. Hardware- repeat based protocols for maintaining cache par-for J := 1 to N do coherence guarantee memory system co- begin herence without software-implemented xtemp[ J ] := b[ J 1; mechanisms. Typically, hardware mecha- for K := 1 to N do nisms detect inconsistency conditions and xtemp[ J 1 := xtemp[ J 1 + A[ J,K ] * x[ K 1; perform actions according to a hardware-. end; implemented protocol. barrier-sync; Data is decomposed into a number of par-for J := 1 to N do equally sized blocks. A block is the unit of x[ J I := xtemp[ J I; transfer between memory and caches. barrier-sync; Hardware protocols allow an arbitrary until false; number of copies of a block to exist at the same time. There are two policies for Figure 4. Pascal-like code for one iteration of the parallel algorithm for solving a maintaining cache consistency: write- linear system of equations by iteration. invalidate and write-update. The write-invalidate policy maintains consistency of multiple copies in the fol- lowing way: Read requests are carried out locally if a copy of the block exists. When parallel algorithm for solving a linear sys- (3) All elements of vector x are updated a processor updates a block, however, all tem of equations by iteration. It takes the in each iteration. other copies are invalidated. How this is form done depends on the interconnection net- With these examples in mind, we will work used. (Ignore it for the moment.) A x,+]= Ax, + b consider how the proposed schemes for subsequent update by the same processor cache coherence manage copies of the data can then be performed locally in the cache, where x,+!, x,, and b are vectors of size N structures. since copies no longer exist. Figure 5 and A is amatrix of size NxN. Suppose that Proposed solutions range from hard- shows how this policy works. In Figure 5a, each iteration (the calculation of vector ware-implemented cache consistency four copies of block X are present in the x<+Jis performed by N processes, where protocols, which give software a coherent system (the memory copy and three cached each process calculates one vector ele- view of the memory system, to schemes copies). In Figure 5b, processor 1 has ment. The code for this algorithm appears providing varied hardware support but updated an item in block X (the updated in Figure 4. The termination condition does with cache coherence enforcement poli- block is denotedx’) and all other copies are not concern us here. Therefore, we assume cies implemented in software. We will invalidated (denoted I). If processor 2 is- that it never terminates. focus on the implementation cost and per- sues a read request to an item in block X‘, The par-for statement initiates N pro- formance issues of the surveyed schemes. then the cache attached to processor 1 cesses. Each process calculates a new supplies it. value, which is stored in xtemp. The last The write-update policy maintains con- parallel loop in the iteration copies back Hardware-based sistency differently. Instead of invalidat- the elements of xtemp to vector x. This protocols ing all copies, it updates them as shown in requires a barrier synchronization. The Figure 5c. Whether the memory copy is most important observations are Hardware-based protocols include updated or not depends on how this proto- snoopy cache protocols, directory col is implemented. We will look at that (1) Vector b and matrix A are read- schemes, and cache-coherent network later. shared and can be safely cached.
Load more

Recommended publications
  • Comparative Performance Evaluation of Cache-Coherent NUMA and COMA Architectures Abstract 1 Introduction
    Comparative Performance Evaluation of Cache-Coherent NUMA and COMA Architectures Per Stenstromt, Truman Joe, and Anoop Gupta Computer Systems Laboratory Stanford University, CA 94305 Abstract machine. Currently, many research groups are pursuing the design and construction of such multiprocessors [12, 1, 10]. Two interesting variations of large-scale shared-memory ma- As research has progressed in this area, two interesting vari- chines that have recently emerged are cache-coherent mm- ants have emerged, namely CC-NUMA (cache-coherent non- umform-memory-access machines (CC-NUMA) and cache- uniform memory access machines) and COMA (cache-only only memory architectures (COMA). They both have dis- memory architectures). Examples of the CC-NUMA ma- tributed main memory and use directory-based cache coher- chines are the Stanford DASH multiprocessor [12] and the MIT ence. Unlike CC-NUMA, however, COMA machines auto- Alewife machine [1], while examples of COMA machines are matically migrate and replicate data at the main-memoty level the Swedish Institute of Computer Science’s Data Diffusion in cache-line sized chunks. This paper compares the perfor- Machine (DDM) [10] and Kendall Square Research’s KSR1 mance of these two classes of machines. We first present a machine [4]. qualitative model that shows that the relative performance is Common to both CC-NUMA and COMA machines are the primarily determined by two factors: the relative magnitude features of distributed main memory, scalable interconnection of capacity misses versus coherence misses, and the gramr- network, and directory-based cache coherence. Distributed hirity of data partitions in the application. We then present main memory and scalable interconnection networks are es- quantitative results using simulation studies for eight prtraUeI sential in providing the required scalable memory bandwidth, applications (including all six applications from the SPLASH while directory-based schemes provide cache coherence with- benchmark suite).
    [Show full text]
  • Detailed Cache Coherence Characterization for Openmp Benchmarks
    1 Detailed Cache Coherence Characterization for OpenMP Benchmarks Anita Nagarajan, Jaydeep Marathe, Frank Mueller Dept. of Computer Science, North Carolina State University, Raleigh, NC 27695-7534 [email protected], phone: (919) 515-7889 Abstract models in their implementation (e.g., [6], [13], [26], [24], [2], [7]), and they operate at different abstraction levels Past work on studying cache coherence in shared- ranging from cycle-accuracy over instruction-level to the memory symmetric multiprocessors (SMPs) concentrates on operating system interface. On the performance tuning end, studying aggregate events, often from an architecture point work mostly concentrates on program analysis to derive op- of view. However, this approach provides insufficient infor- timized code (e.g., [15], [27]). Recent processor support for mation about the exact sources of inefficiencies in paral- performance counters opens new opportunities to study the lel applications. For SMPs in contemporary clusters, ap- effect of applications on architectures with the potential to plication performance is impacted by the pattern of shared complement them with per-reference statistics obtained by memory usage, and it becomes essential to understand co- simulation. herence behavior in terms of the application program con- In this paper, we concentrate on cache coherence simu- structs — such as data structures and source code lines. lation without cycle accuracy or even instruction-level sim- The technical contributions of this work are as follows. ulation. We constrain ourselves to an SPMD programming We introduce ccSIM, a cache-coherent memory simulator paradigm on dedicated SMPs. Specifically, we assume the fed by data traces obtained through on-the-fly dynamic absence of workload sharing, i.e., only one application runs binary rewriting of OpenMP benchmarks executing on a on a node, and we enforce a one-to-one mapping between Power3 SMP node.
    [Show full text]
  • Analysis and Optimization of I/O Cache Coherency Strategies for Soc-FPGA Device
    Analysis and Optimization of I/O Cache Coherency Strategies for SoC-FPGA Device Seung Won Min Sitao Huang Mohamed El-Hadedy Electrical and Computer Engineering Electrical and Computer Engineering Electrical and Computer Engineering University of Illinois University of Illinois California State Polytechnic University Urbana, USA Urbana, USA Ponoma, USA [email protected] [email protected] [email protected] Jinjun Xiong Deming Chen Wen-mei Hwu IBM T.J. Watson Research Center Electrical and Computer Engineering Electrical and Computer Engineering Yorktown Heights, USA University of Illinois University of Illinois [email protected] Urbana, USA Urbana, USA [email protected] [email protected] Abstract—Unlike traditional PCIe-based FPGA accelerators, However, choosing the right I/O cache coherence method heterogeneous SoC-FPGA devices provide tighter integrations for different applications is a challenging task because of between software running on CPUs and hardware accelerators. it’s versatility. By choosing different methods, they can intro- Modern heterogeneous SoC-FPGA platforms support multiple I/O cache coherence options between CPUs and FPGAs, but these duce different types of overheads. Depending on data access options can have inadvertent effects on the achieved bandwidths patterns, those overheads can be amplified or diminished. In depending on applications and data access patterns. To provide our experiments, we find using different I/O cache coherence the most efficient communications between CPUs and accelera- methods can vary overall application execution times at most tors, understanding the data transaction behaviors and selecting 3.39×. This versatility not only makes designers hard to decide the right I/O cache coherence method is essential.
    [Show full text]
  • Chapter 5 Thread-Level Parallelism
    Chapter 5 Thread-Level Parallelism Instructor: Josep Torrellas CS433 Copyright Josep Torrellas 1999, 2001, 2002, 2013 1 Progress Towards Multiprocessors + Rate of speed growth in uniprocessors saturated + Wide-issue processors are very complex + Wide-issue processors consume a lot of power + Steady progress in parallel software : the major obstacle to parallel processing 2 Flynn’s Classification of Parallel Architectures According to the parallelism in I and D stream • Single I stream , single D stream (SISD): uniprocessor • Single I stream , multiple D streams(SIMD) : same I executed by multiple processors using diff D – Each processor has its own data memory – There is a single control processor that sends the same I to all processors – These processors are usually special purpose 3 • Multiple I streams, single D stream (MISD) : no commercial machine • Multiple I streams, multiple D streams (MIMD) – Each processor fetches its own instructions and operates on its own data – Architecture of choice for general purpose mps – Flexible: can be used in single user mode or multiprogrammed – Use of the shelf µprocessors 4 MIMD Machines 1. Centralized shared memory architectures – Small #’s of processors (≈ up to 16-32) – Processors share a centralized memory – Usually connected in a bus – Also called UMA machines ( Uniform Memory Access) 2. Machines w/physically distributed memory – Support many processors – Memory distributed among processors – Scales the mem bandwidth if most of the accesses are to local mem 5 Figure 5.1 6 Figure 5.2 7 2. Machines w/physically distributed memory (cont) – Also reduces the memory latency – Of course interprocessor communication is more costly and complex – Often each node is a cluster (bus based multiprocessor) – 2 types, depending on method used for interprocessor communication: 1.
    [Show full text]
  • 2.2 Adaptive Routing Algorithms and Router Design 20
    https://theses.gla.ac.uk/ Theses Digitisation: https://www.gla.ac.uk/myglasgow/research/enlighten/theses/digitisation/ This is a digitised version of the original print thesis. Copyright and moral rights for this work are retained by the author A copy can be downloaded for personal non-commercial research or study, without prior permission or charge This work cannot be reproduced or quoted extensively from without first obtaining permission in writing from the author The content must not be changed in any way or sold commercially in any format or medium without the formal permission of the author When referring to this work, full bibliographic details including the author, title, awarding institution and date of the thesis must be given Enlighten: Theses https://theses.gla.ac.uk/ [email protected] Performance Evaluation of Distributed Crossbar Switch Hypermesh Sarnia Loucif Dissertation Submitted for the Degree of Doctor of Philosophy to the Faculty of Science, Glasgow University. ©Sarnia Loucif, May 1999. ProQuest Number: 10391444 All rights reserved INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted. In the unlikely event that the author did not send a com plete manuscript and there are missing pages, these will be noted. Also, if material had to be removed, a note will indicate the deletion. uest ProQuest 10391444 Published by ProQuest LLO (2017). Copyright of the Dissertation is held by the Author. All rights reserved. This work is protected against unauthorized copying under Title 17, United States C ode Microform Edition © ProQuest LLO. ProQuest LLO.
    [Show full text]
  • Adaptive Parallelism for Coupled, Multithreaded Message-Passing Programs Samuel K
    University of New Mexico UNM Digital Repository Computer Science ETDs Engineering ETDs Fall 12-1-2018 Adaptive Parallelism for Coupled, Multithreaded Message-Passing Programs Samuel K. Gutiérrez Follow this and additional works at: https://digitalrepository.unm.edu/cs_etds Part of the Numerical Analysis and Scientific omputC ing Commons, OS and Networks Commons, Software Engineering Commons, and the Systems Architecture Commons Recommended Citation Gutiérrez, Samuel K.. "Adaptive Parallelism for Coupled, Multithreaded Message-Passing Programs." (2018). https://digitalrepository.unm.edu/cs_etds/95 This Dissertation is brought to you for free and open access by the Engineering ETDs at UNM Digital Repository. It has been accepted for inclusion in Computer Science ETDs by an authorized administrator of UNM Digital Repository. For more information, please contact [email protected]. Samuel Keith Guti´errez Candidate Computer Science Department This dissertation is approved, and it is acceptable in quality and form for publication: Approved by the Dissertation Committee: Professor Dorian C. Arnold, Chair Professor Patrick G. Bridges Professor Darko Stefanovic Professor Alexander S. Aiken Patrick S. McCormick Adaptive Parallelism for Coupled, Multithreaded Message-Passing Programs by Samuel Keith Guti´errez B.S., Computer Science, New Mexico Highlands University, 2006 M.S., Computer Science, University of New Mexico, 2009 DISSERTATION Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy Computer Science The University of New Mexico Albuquerque, New Mexico December 2018 ii ©2018, Samuel Keith Guti´errez iii Dedication To my beloved family iv \A Dios rogando y con el martillo dando." Unknown v Acknowledgments Words cannot adequately express my feelings of gratitude for the people that made this possible.
    [Show full text]
  • Detecting False Sharing Efficiently and Effectively
    W&M ScholarWorks Arts & Sciences Articles Arts and Sciences 2016 Cheetah: Detecting False Sharing Efficiently andff E ectively Tongping Liu Univ Texas San Antonio, Dept Comp Sci, San Antonio, TX 78249 USA; Xu Liu Coll William & Mary, Dept Comp Sci, Williamsburg, VA 23185 USA Follow this and additional works at: https://scholarworks.wm.edu/aspubs Recommended Citation Liu, T., & Liu, X. (2016, March). Cheetah: Detecting false sharing efficiently and effectively. In 2016 IEEE/ ACM International Symposium on Code Generation and Optimization (CGO) (pp. 1-11). IEEE. This Article is brought to you for free and open access by the Arts and Sciences at W&M ScholarWorks. It has been accepted for inclusion in Arts & Sciences Articles by an authorized administrator of W&M ScholarWorks. For more information, please contact [email protected]. Cheetah: Detecting False Sharing Efficiently and Effectively Tongping Liu ∗ Xu Liu ∗ Department of Computer Science Department of Computer Science University of Texas at San Antonio College of William and Mary San Antonio, TX 78249 USA Williamsburg, VA 23185 USA [email protected] [email protected] Abstract 1. Introduction False sharing is a notorious performance problem that may Multicore processors are ubiquitous in the computing spec- occur in multithreaded programs when they are running on trum: from smart phones, personal desktops, to high-end ubiquitous multicore hardware. It can dramatically degrade servers. Multithreading is the de-facto programming model the performance by up to an order of magnitude, significantly to exploit the massive parallelism of modern multicore archi- hurting the scalability. Identifying false sharing in complex tectures.
    [Show full text]
  • Reducing Cache Coherence Traffic with a NUMA-Aware Runtime Approach
    This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TPDS.2017.2787123, IEEE Transactions on Parallel and Distributed Systems IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. XX, NO. X, MONTH YEAR 1 Reducing Cache Coherence Traffic with a NUMA-Aware Runtime Approach Paul Caheny, Lluc Alvarez, Said Derradji, Mateo Valero, Fellow, IEEE, Miquel Moreto,´ Marc Casas Abstract—Cache Coherent NUMA (ccNUMA) architectures are a widespread paradigm due to the benefits they provide for scaling core count and memory capacity. Also, the flat memory address space they offer considerably improves programmability. However, ccNUMA architectures require sophisticated and expensive cache coherence protocols to enforce correctness during parallel executions, which trigger a significant amount of on- and off-chip traffic in the system. This paper analyses how coherence traffic may be best constrained in a large, real ccNUMA platform comprising 288 cores through the use of a joint hardware/software approach. For several benchmarks, we study coherence traffic in detail under the influence of an added hierarchical cache layer in the directory protocol combined with runtime managed NUMA-aware scheduling and data allocation techniques to make most efficient use of the added hardware. The effectiveness of this joint approach is demonstrated by speedups of 3.14x to 9.97x and coherence traffic reductions of up to 99% in comparison to NUMA-oblivious scheduling and data allocation. Index Terms—Cache Coherence, NUMA, Task-based programming models F 1 INTRODUCTION HE ccNUMA approach to memory system architecture the data is allocated within the NUMA regions of the system T has become a ubiquitous choice in the design-space [10], [28].
    [Show full text]
  • Leaper: a Learned Prefetcher for Cache Invalidation in LSM-Tree Based Storage Engines
    Leaper: A Learned Prefetcher for Cache Invalidation in LSM-tree based Storage Engines ∗ Lei Yang1 , Hong Wu2, Tieying Zhang2, Xuntao Cheng2, Feifei Li2, Lei Zou1, Yujie Wang2, Rongyao Chen2, Jianying Wang2, and Gui Huang2 fyang lei, [email protected] fhong.wu, tieying.zhang, xuntao.cxt, lifeifei, zhencheng.wyj, rongyao.cry, beilou.wjy, [email protected] Peking University1 Alibaba Group2 ABSTRACT 101 Hit ratio Frequency-based cache replacement policies that work well QPS on page-based database storage engines are no longer suffi- Latency cient for the emerging LSM-tree (Log-Structure Merge-tree) based storage engines. Due to the append-only and copy- 100 on-write techniques applied to accelerate writes, the state- of-the-art LSM-tree adopts mutable record blocks and issues Normalized value frequent background operations (i.e., compaction, flush) to reorganize records in possibly every block. As a side-effect, 0 50 100 150 200 such operations invalidate the corresponding entries in the Time (s) Figure 1: Cache hit ratio and system performance churn (QPS cache for each involved record, causing sudden drops on the and latency of 95th percentile) caused by cache invalidations. cache hit rates and spikes on access latency. Given the ob- with notable examples including LevelDB [10], HBase [2], servation that existing methods cannot address this cache RocksDB [7] and X-Engine [14] for its superior write perfor- invalidation problem, we propose Leaper, a machine learn- mance. These storage engines usually come with row-level ing method to predict hot records in an LSM-tree storage and block-level caches to buffer hot records in main mem- engine and prefetch them into the cache without being dis- ory.
    [Show full text]
  • Parallel Processors and Cache Coherency
    Intro Cache Coherency Notes Computer Architecture II Parallel Processors and Cache Coherency Syed Asad Alam School of Computer Science and Statistics 1/31 Intro Cache Coherency Notes Introduction 2/31 Intro Cache Coherency Notes Flynn’s Taxonomy Introduced by M. J. Flynn SISD → Single processor with a single instruction stream and a single memory (Uniprocessors) SIMD → Single machine instruction with multiple processing elements operating on independent set of data MISD → Multiple processors, each of which execute different instruction sequence on a single sequence of data MIMD → Multiple processors execute different instruction sequences on different data sets 3/31 Intro Cache Coherency Notes Flynn’s Taxonomy Introduced by M. J. Flynn SISD → Single processor with a single instruction stream and a single memory (Uniprocessors) SIMD → Single machine instruction with multiple processing elements operating on independent set of data MISD → Multiple processors, each of which execute different instruction sequence on a single sequence of data MIMD → Multiple processors execute different instruction sequences on different data sets Text sourced from: Computer Organization and Architecture Designing for Performance, William Stallings, Chapter 17 & 18 3/31 Intro Cache Coherency Notes Flynn’s Taxonomy1 1 W. Stallings 4/31 Intro Cache Coherency Notes Flynn’s Taxonomy2 2 W. Stallings 5/31 Intro Cache Coherency Notes Multi-Computer/Cluster – Loosely Coupled 6/31 Intro Cache Coherency Notes Multi-Computer/Cluster – Loosely Coupled 6/31 Intro Cache Coherency Notes Cluster Computer Architecture3 3 W. Stallings 7/31 Intro Cache Coherency Notes Example IITAC Cluster [in Lloyd building] 346 x IBM e326 compute node each with 2 x 2.4GHz 64bit AMD Opteron 250 CPUs, 4GB RAM, ..
    [Show full text]
  • Comparative Modeling and Evaluation of CC-NUMA and COMA on Hierarchical Ring Rchitectures
    1316 IEEE TRANSACTIONSON PARALLEL AND DISTRIBUTEDSYSTEMS, VOL. 6, NO. 12, DECEMBER 1995 , Comparative Modeling and Evaluation of CC-NUMA and COMA on Hierarchical Ring rchitectures Xiaodong Zhang, Senior Member, IEEE, and Yong Yan Abstract-Parallel computing performance on scalable share& large NUMA architectures, the memory system organization memory architectures is affected by the structure of the intercon- also affects communication latency. With respect to the kinds nection networks linking processors to memory modules and on of memory organizations utilized, NUMA memory systems the efficiency of the memory/cache management systems. Cache Coherence Nonuniform Memory Access (CC-NUMA) and Cache can be classified into the following three types in terms of data Only Memory Access (COMA) are two effective memory systems, migration and coherence: and the hierarchical ring structure is an efficient interco~ection Non-CC-NUMA stands for. non-cache-coherentNUMA. network in hardware. This paper focuses on comparative per- formance modeling and evaluation of CC-NUMA and COMA on This type of architecture either supports no local caches a hierarchical ring shared-memory architecture. Analytical mod- at all (e.g., the BBN GPlOOO [I]), or provides a local. els for the two memory systems for comparative evaluation are cache that disallows caching of shared data in order to presented. Intensive performance measurements on data migra- avoid the cache coherence problem (e.g., the BBN tions have been conducted on the KSR-1, a COMA hlerarcbicaI TC2cKm[ 11). ring shared-memory machine. Experimental results support the analytical models, and we present practical observations and CCWUMA stands for cache-coherent NUMA, where comparisons of the two cache coherence memory systems.
    [Show full text]
  • CSC 256/456: Operating Systems
    CSC 256/456: Operating Systems Multiprocessor John Criswell Support University of Rochester 1 Outline ❖ Multiprocessor hardware ❖ Types of multi-processor workloads ❖ Operating system issues ❖ Where to run the kernel ❖ Synchronization ❖ Where to run processes 2 Multiprocessor Hardware 3 Multiprocessor Hardware ❖ System in which two or more CPUs share full access to the main memory ❖ Each CPU might have its own cache and the coherence among multiple caches is maintained CPU CPU … … … CPU Cache Cache Cache Memory bus Memory 4 Multi-core Processor ❖ Multiple processors on “chip” ❖ Some caches shared CPU CPU ❖ Some not shared Cache Cache Shared Cache Memory Bus 5 Hyper-Threading ❖ Replicate parts of processor; share other parts ❖ Create illusion that one core is two cores CPU Core Fetch Unit Decode ALU MemUnit Fetch Unit Decode 6 Cache Coherency ❖ Ensure processors not operating with stale memory data ❖ Writes send out cache invalidation messages CPU CPU … … … CPU Cache Cache Cache Memory bus Memory 7 Non Uniform Memory Access (NUMA) ❖ Memory clustered around CPUs ❖ For a given CPU ❖ Some memory is nearby (and fast) ❖ Other memory is far away (and slow) 8 Multiprocessor Workloads 9 Multiprogramming ❖ Non-cooperating processes with no communication ❖ Examples ❖ Time-sharing systems ❖ Multi-tasking single-user operating systems ❖ make -j<very large number here> 10 Concurrent Servers ❖ Minimal communication between processes and threads ❖ Throughput usually the goal ❖ Examples ❖ Web servers ❖ Database servers 11 Parallel Programs ❖ Use parallelism
    [Show full text]