Comparing Cache Architectures and Coherency Protocols on x86-64 Multicore SMP Systems Daniel Hackenberg Daniel Molka Wolfgang E. Nagel Center for Information Services and High Performance Computing (ZIH) Technische Universität Dresden, 01062 Dresden, Germany {daniel.hackenberg, daniel.molka, wolfgang.nagel}@tu-dresden.de ABSTRACT quad-core designs for dual-socket systems. They integrate Across a broad range of applications, multicore technol- on-chip memory controllers that reduce the memory latency ogy is the most important factor that drives today's mi- and allow the memory bandwidth to scale with the processor croprocessor performance improvements. Closely coupled is count in an SMP system. Serial point-to-point links provide a growing complexity of the memory subsystems with sev- cache-coherent inter-processor connections, thus creating a eral cache levels that need to be exploited efficiently to gain cache-coherent non-uniform memory access (ccNUMA) ar- optimal application performance. Many important imple- chitecture. Both processor families feature a three level mentation details of these memory subsystems are undocu- cache hierarchy and shared last level caches. mented. We therefore present a set of sophisticated bench- The evolution of processor technology towards multicore marks for latency and bandwidth measurements to arbitrary designs drives software technology as well. Parallel program- locations in the memory subsystem. We consider the co- ming paradigms become increasingly attractive to leverage herency state of cache lines to analyze the cache coherency all available resources. This development goes along with a protocols and their performance impact. The potential of change of memory access patterns: While independent pro- our approach is demonstrated with an in-depth compari- cesses typically work on disjoint memory, parallel applica- son of ccNUMA multiprocessor systems with AMD (Shang- tions often use multiple threads to work on a shared data set. hai) and Intel (Nehalem-EP) quad-core x86-64 processors Such parallelism, e.g. producer-consumer problems as well that both feature integrated memory controllers and coher- as synchronization directives inherently require data move- ent point-to-point interconnects. Using our benchmarks we ment between processor cores that is preferably kept on- present fundamental memory performance data and archi- chip. Common memory benchmarks such as STREAM [11] tectural properties of both processors. Our comparison re- are no longer sufficient to adequately assess the memory per- veals in detail how the microarchitectural differences tremen- formance of state-of-the-art processors as they do not cover dously affect the performance of the memory subsystem. important aspects of multicore architectures such as band- width and latency between different processor cores. To improve this situation, we present a set of benchmarks Categories and Subject Descriptors that can be used to determine performance characteristics of C.4 [Performance of Systems]: Measurement techniques memory accesses to arbitrary locations in multicore, multi- processor ccNUMA systems. This includes on- and off-chip General Terms cache-to-cache transfers that we consider to be of growing importance. We also analyze the influence of the cache co- Performance Measurement herency protocol by controlling the coherency state of the cache lines that are accessed by our benchmarks. Keywords Based on these benchmarks we compare the performance Nehalem, Shanghai, Multi-core, Coherency, Benchmark of the memory subsystem in two socket SMP systems with AMD Shanghai and Intel Nehalem processors. Both proces- sors strongly differ in the design of their cache architecture, 1. INTRODUCTION e.g. the last level cache implementation (non-inclusive vs. Multicore technology is established in desktop and server inclusive) and the coherency protocol (MOESI vs. MESIF). systems as well as in high performance computing. The Our results demonstrate how these differences affect the AMD Opteron 2300 family (Shanghai/Barcelona) and In- memory latency and bandwidth. We reveal undocumented tel's Xeon 5500 processors (Nehalem-EP) are native x86-64 architectural properties and memory performance charac- teristics of both processors. While a high-level analysis of application performance is beyond the scope of this paper, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are we present fundamental data to facilitate such research. not made or distributed for profit or commercial advantage and that copies The paper is organized as follows: Section 2 outlines char- bear this notice and the full citation on the first page. To copy otherwise, to acteristics of our test systems. Section 3 introduces the republish, to post on servers or to redistribute to lists, requires prior specific benchmarks that we use to generate the results in Sections 4 permission and/or a fee. and 5. Section 6 presents related work. Section 7 concludes MICRO’09, December 12–16, 2009, New York, NY, USA. this paper and sketches future work. Copyright 2009 ACM 978-1-60558-798-1/09/12 ...$10.00. 413 Shanghai Shanghai Nehalem-EP Nehalem-EP Core 0 Core 1 Core 2 Core 3 Core 4 Core 5 Core 6 Core 7 Core 0 Core 1 Core 2 Core 3 Core 4 Core 5 Core 6 Core 7 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L1 L2 L2 L2 L2 L2 L2 L2 L2 L2 L2 L2 L2 L2 L2 L2 L2 Shared L3 Cache (non-inclusive) Shared L3 Cache (non-inclusive) Shared L3 Cache (inclusive) Shared L3 Cache (inclusive) IMC HT HT IMC IMC QPI QPI IMC F A B E A B C D C D 3 3 3 3 2 2 3 3 I/O I/O 2 2 I/O I/O R R R R R R R R R R D D D D D D D D D D D D D D D D D D D D Figure 1: Block diagram of the AMD (left) and Intel (right) system architecture 2. BACKGROUND AND TEST SYSTEMS AMD's last level cache is non-inclusive [6], i.e neither ex- Dual-socket SMP systems based on AMD Opteron 23** clusive nor inclusive. If a cache line is transferred from the (Shanghai) and Intel Xeon 55** (Nehalem-EP) processors L3 cache into the L1 of any core the line can be removed from have a similar high level design as depicted in Figure 1. the L3. According to AMD this happens if it is \likely" [3] The L1 and L2 caches are implemented per core, while the (further details are undisclosed) that the line is only used L3 cache is shared among all cores of one processor. The by one core, otherwise a copy can be kept in the L3. Both serial point-to-point links HyperTransport (HT) and Quick processors use extended versions of the well-known MESI [7] Path Interconnect (QPI) are used for inter-processor and protocol to ensure cache coherency. AMD Opteron proces- chipset communication. Moreover, each processor contains sors implement the MOESI protocol [2, 5]. The additional its own integrated memory controller (IMC). state owned (O) allows to share modified data without a Although the number of cores, clockrates, and cache sizes write-back to main memory. Nehalem processors implement are similar, benchmark results can differ significantly. For the MESIF protocol [9] and use the forward (F) state to en- example in SPEC's CPU2006 benchmark, the Nehalem typ- sure that shared unmodified data is forwarded only once. ically outperforms AMD's Shanghai [1]. This is a result The configuration of both test systems is detailed in Ta- of multiple aspects such as different instruction-level par- ble 1. The listing shows a major disparity with respect to allelism, Simultaneous Multi-Threading (SMT), and Intel's the main memory configuration. We can assume that Ne- Turbo Boost Technology. Another important factor is the halem's three DDR3-1333 channels outperform Shanghai's architecture of the memory subsystem in conjunction with two DDR2-667 channels (DDR2-800 is supported by the the cache coherency protocol [12]. CPU but not by our test system). However, the main mem- While the basic memory hierarchy structure is similar for ory performance of AMD processors will improve by switch- Nehalem and Shanghai systems, the implementation details ing to new sockets with more memory channels and DDR3. differ significantly. Intel implements an inclusive last level We disable Turbo Boost in our Intel test system as it in- cache in order to filter unnecessary snoop traffic. Core valid troduces result perturbations that are often unpredictable. bits within the L3 cache indicate that a cache line may be Our benchmarks require only one thread per core to access present in a certain core. If a bit is not set, the associated all caches and we therefore disable the potentially disadvan- core certainly does not hold a copy of the cache line, thus tageous SMT feature. We disable the hardware prefetchers reducing snoop traffic to that core. However, unmodified for all latency measurements as they introduce result varia- cache lines may be evicted from a core's cache without noti- tions that distract from the actual hardware properties. The fication of the L3 cache. Therefore, a set core valid bit does bandwidth measurements are more robust and we enable the not guarantee a cache line's presence in a higher level cache. hardware prefetchers unless noted otherwise. Test system Sun Fire X4140 Intel Evaluation Platform Processors 2x AMD Opteron 2384 2x Intel Xeon X5570 Codename Shanghai Nehalem-EP Core/Uncore frequency 2.7 GHz / 2.2 GHz 2.933 GHz / 2.666 GHz Processor Interconnect HyperTransport, 8 GB/s QuickPath Interconnect, 25.6 GB/s Cache line size 64 Bytes L1 cache 64 KiB/64 KiB (per core) 32 KiB/32 KiB (per core) L2 cache 512 KiB (per core), exclusive of L1 256 KiB (per core), non-inclusive L3 cache 6 MiB (shared), non-inclusive 8 MiB (shared), inclusive of L1 and L2 Cache coherency protocol MOESI MESIF Integrated memory controller yes, 2 channel yes, 3 channel 8 x 4 GiB DDR2-667, registered, ECC 6 x 2 GiB DDR3-1333, registered, ECC Main memory (4 DIMMS per processor) (3 DIMMS per processor) Operating system Debian 5.0, Kernel 2.6.28.1 Table 1: Configuration of the test systems 414 3.