A Case for NUMA-aware Contention Management on Multicore Systems
Sergey Blagodurov Sergey Zhuravlev Mohammad Dashti Simon Fraser University Simon Fraser University Simon Fraser University Alexandra Fedorova Simon Fraser University
Abstract performance of individual applications or threads by as much as 80% and the overall workload performance by On multicore systems, contention for shared resources as much as 12% [23]. occurs when memory-intensive threads are co-scheduled Unfortunately studies of contention-aware algorithms on cores that share parts of the memory hierarchy, such focused primarily on UMA (Uniform Memory Access) as last-level caches and memory controllers. Previous systems, where there are multiple shared LLCs, but only work investigated how contention could be addressed a single memory node equipped with the single memory via scheduling. A contention-aware scheduler separates controller, and memory can be accessed with the same competing threads onto separate memory hierarchy do- latency from any core. However, new multicore sys- mains to eliminate resource sharing and, as a conse- tems increasingly use the Non-Uniform Memory Access quence, to mitigate contention. However, all previous (NUMA) architecture, due to its decentralized and scal- work on contention-aware scheduling assumed that the able nature. In modern NUMA systems, there are mul- underlying system is UMA (uniform memory access la- tiple memory nodes, one per memory domain (see Fig- tencies, single memory controller). Modern multicore ure 1). Local nodes can be accessed in less time than re- systems, however, are NUMA, which means that they mote ones, and each node has its own memory controller. feature non-uniform memory access latencies and multi- When we ran the best known contention-aware sched- ple memory controllers. ulers on a NUMA system, we discovered that not only do We discovered that state-of-the-art contention man- they not manage contention effectively, but they some- agement algorithms fail to be effective on NUMA sys- times even hurt performance when compared to a de- tems and may even hurt performance relative to a default fault contention-unaware scheduler (on our experimental OS scheduler. In this paper we investigate the causes for setup we observed as much as 30% performance degra- this behavior and design the first contention-aware algo- dation caused by a NUMA-agnostic contention-aware al- rithm for NUMA systems. gorithm relative to the default Linux scheduler). The focus of our study is to investigate (1) why contention- 1 Introduction management schedulers that targeted UMA systems fail to work on NUMA systems and (2) devise an algorithm Contention for shared resources on multicore proces- that would work effectively on NUMA systems. sors is a well-known problem. Consider a typical mul- Why existing contention-aware algorithms may hurt ticore system, schematically depicted in Figure 1, where performance on NUMA systems: Existing state-of- cores share parts of the memory hierarchy, which we the-art contention-aware algorithms work as follows on term memory domains, and compete for resources such NUMA systems. They identify threads that are sharing as last-level caches (LLC), system request queues and a memory domain and hurting each other’s performance memory controllers. Several studies investigated ways of and migrate one of the threads to a different domain. This reducing resource contention and one of the promising may lead to a situation where a thread’s memory is lo- approaches that emerged recently is contention-aware cated in a different domain than that in which the thread scheduling [23, 10, 16]. A contention-aware scheduler is running. (E.g., consider a thread being migrated from identifies threads that compete for shared resources of a core C1 to core C5 in Figure 1, with its memory being lo- memory domain and places them into different domains. cated in Memory Node #1). We refer to migrations that In doing so the scheduler can improve the worst-case may place a thread into a domain remote from its mem- known to work well on UMA systems may actually C1 C2 C5 C6 hurt performance on NUMA systems. L3 cache L3 cache C3 C4 C7 C8 • We identify NUMA-agnostic migration as the cause for this phenomenon and identify the reasons why Memory Node #1 Memory Node #2 performance degrades. We also show that remote access latency is not the key reason why NUMA- Domain 1 Domain 2 agnostic migration hurt performance. • We design and implement Distributed Intensity NUMA Online (DINO), a new contention-aware al- Memory Node #3 Memory Node #4 gorithm for NUMA systems. DINO prevents super- fluous thread migrations, but when it does perform C9 C10 C13 C14 L3 cache L3 cache migrations, it moves the memory of the threads C11 C12 C15 C16 along with the threads themselves. DINO performs up to 20% better than the default Linux scheduler Domain 3 Domain 4 and up to 50% better than Distributed Intensity, which is the best contention-aware scheduler known Figure 1: A schematic view of a system with four mem- to us [23]. ory domains and four cores per domain. There are 16 • We devise a page migration strategy that works on- cores in total, and a shared L3 cache per domain. line, uses Instruction-Based Sampling, and elimi- nates on average 75% of remote accesses. ory NUMA-agnostic migrations. Our algorithms were implemented at user-level, since NUMA-agnostic migrations create several problems, modern operating systems typically export the interfaces an obvious one being that the thread now incurs a higher for implementing the desired functionality. If needed, the latency when accessing its memory. However, contrary algorithms can also be moved into the kernel itself. to a commonly held belief that remote access latency – The rest of this paper is organized as follows. Sec- i.e., the higher latency incurred when accessing a remote tion 2 demonstrates why existing contention-aware al- domain relative to accessing a local one – would be the gorithms fail to work on NUMA systems. Section 3 key concern in this scenario, we discovered that NUMA- presents and evaluates DINO. Section 4 analyzes mem- agnostic migrations create other problems, which are far ory migration strategies. Section 5 provides the exper- more serious than remote access latency. In particular, imental results. Section 6 discusses related work, and NUMA-agnostic migrations fail to eliminate contention Section 7 summarizes our findings. for some of the key hardware resources on multicore systems and create contention for additional resources. That is why existing contention-aware algorithms that 2 Why existing algorithms do not work on perform NUMA-agnostic migrations not only fail to be NUMA systems effective, but can substantially hurt performance on mod- ern multicore systems. As we explained in the introduction, existing contention- Challenges in designing contention-aware algo- aware algorithms perform NUMA-agnostic migration, rithms for NUMA systems: To address this problem, a and so a thread may end up running on a node remote contention-aware algorithm on a NUMA system must from its memory. This creates additional problems be- migrate the memory of the thread to the same domain sides introducing remote latency overhead. In particu- memory where it migrates the thread itself. However, the need to lar, NUMA-agnostic migrations fail to eliminate controller contention interconnect move memory along with the thread makes thread mi- , and create additional contention grations costly. So the algorithm must minimize thread . The focus of this section is to experimentally migrations, performing them only when they are likely to demonstrate why this is the case. significantly increase performance, and when migrating To this end, in Section 2.1, we quantify how con- memory it must carefully decide which pages are most tention for various shared resources contributes to per- profitable to migrate. Our work addresses these chal- formance degradation that an application may experi- lenges. ence as it shares the hardware with other applications. The contributions of our work can be summarized as We show that memory controller contention and inter- follows: connect contention are the most important causes of per- formance degradation when an application is running re- • We discover that contention-aware algorithms motely from its memory. Then, in Section 2.2 we use
2 four AMD Barcelona processors running at 2.3GHz, and Core 1 Core 2 Core 3 Core 4 64GB of RAM, 16GB per domain. The operating system is Linux 2.6.29.6. Figure 2 schematically represents the L2 cache L2 cache L2 cache L2 cache architecture of each processor in this system. We identify four sources of performance degradation Shared L3 Cache that can occur on modern NUMA systems, such as those shown in Figures 1 and 2:
System Request Interface • Contention for the shared last-level cache (CA). Crossbar switch This also includes contention for the system request queue and the crossbar. Memory HyperTransport™ Controller • Contention for the memory controller (MC). This also includes contention for the DRAM prefetching unit.
• Contention for the inter-domain interconnect (IC). to DRAM to other chips Figure 2: A schematic view of a system used in this • Remote access latency, occurring when a thread’s study. A single domain is shown. memory is placed in a remote node (RL).
Base Case Cache Contention To quantify the effects of performance degradation caused by these factors we use the methodology depicted 0) T C MT MC NONE 4) TC in Figure 3. We run a target application, denoted as MT MC CACACA T with a set of three competing applications, denoted No Cache Contention as C. The memory of the target application is denoted MT , and the memory of the competing applications is 1) T C 5) TC CACACA denoted MC. We vary (1) how the target application is MT, MC MCMCMC MC MT RLRLRL placed with respect to its memory, (2) how it is placed with respect to the competing applications, and (3) how 2) T C 6) TC ICICIC CACACA the memory of the target is placed with respect to the MC MT RLRLRL MT,MC MCMCMC memory of the competing applications. Exploring per- CA, formance in these various scenarios allows us to quantify 3) T C MC 7) TC MCMC IC,MC the effects of NUMA-agnostic thread placement. MT,MC RLRLRL MT,MC RLRLRL Figure 3 summarizes the relative placement of mem- ory and applications that we used in our experiments. CPU running Memory node holding T MT application T T’s memory Next to each scenario we show factors affecting the performance of the target application: CA, IC, MC or CPU running Memory node holding C MC RL. For example, in Scenario 0, an application runs applications C memory of apps. C contention-free with its memory on a local node, so no performance-degrading factors are present. We term this Figure 3: Placement of threads and memory in all exper- the base case and compare to it the performance in other imental configurations. cases. The scenarios where there is cache contention are shown on the right and the scenarios where there is no this finding to explain why NUMA-agnostic migrations cache contention are shown on the left. can be detrimental to performance. We used two types of target and competing appli- cations, classified according to their memory intensity: devil and turtle. The terminology is borrowed from an 2.1 Quantifying causes of contention earlier study on application classification [21]. Devils are memory intensive: they generate a large number of In this section we quantify the effects of performance memory requests. We classify an application as a devil degradation on multicore NUMA systems depending on if it generates more than two misses per 1000 instruc- how threads and their memory are placed in memory do- tions (MPI). Otherwise, an application is deemed a turtle. mains. For this part of the study, we use benchmarks We further divide devils into two subcategories: regular from the SPEC CPU2006 benchmark suite. We perform devils and soft-devils. Regular devils have a miss rate experiments on a Dell PowerEdge server equipped with that exceeds 15 misses per 1000 instructions. Soft-devils
3 have an MPI between two and 15. Solo miss rates, ob- the memory is not migrated with the thread, several tained when an application runs on a machine alone, are memory-intensive threads could still have their mem- used for classification. ory placed in the same memory node and so they would We experimented with nine different target applica- compete for the memory controller when accessing their tions: three devils (mcf, omnetpp and milc), three soft- memory. Furthermore, a migrated thread could be sub- devils, (gcc, bwaves and bzip) and three turtles (povray, ject to the remote access latency, and because a thread calculix and h264). would use the inter-node interconnect to access its mem- Figure 4 shows how an application’s performance de- ory, it would be subject to the interconnect contention. grades in Scenarios 1-7 from Figure 3 relative to Sce- In summary, NUMA-agnostic migrations fail to elimi- nario 0. Performance degradation, shown on the y-axis, nate or even exacerbate the most crucial performance- is measured as the increase in completion time relative degrading factors: MC, IC, RL. to Scenario 0. The x-axis shows the type of competing applications that were running concurrently to generate 2.2 Why existing contention management contention: devil, soft-devil, or turtle. algorithms hurt performance These results demonstrate a very important point ex- hibited in Scenario 3: when a thread runs alone on a Now that we are familiar with causes of performance memory node (i.e., there is no contention for cache), degradation on NUMA systems, we are ready to explain but its memory is remote and is in the same domain as why existing contention management algorithms fail to the memory of another memory-intensive thread, perfor- work on NUMA systems. Consider the following ex- mance degradation can be very severe, reaching 110% ample. Suppose that two competing threads A and B (see MILC, Scenario 3). One of the reasons is that the run on cores C1 and C2 on a system shown in Figure 1. threads are still competing for the memory controller of A contention-aware scheduler would detect that A and the node that holds their memory. But this is exactly B compete and migrate one of the threads, for example the scenario that can be created by a NUMA-agnostic thread B, to a core in a different memory domain, for ex- migration, which migrates a thread to a different node ample core C5. Now A and B are not competing for the without migrating its memory. This is the first piece of last-level (L3) cache, and on UMA systems this would be evidence showing why NUMA-agnostic migrations will sufficient to eliminate contention. But on a NUMA sys- cause problems. tem shown in Figure 1, A and B are still competing for We now present further evidence. Using the data in the memory controller at Memory Node #1 (MC in Fig- these experiments, we are able to estimate how much ure 5), assuming that their memory is physically located each of the four factors (CA, MC, IC, and RL) con- in Node #1. So by simply migrating thread B to another tributes to the overall performance degradation in Sce- memory domain, the scheduler does not eliminate one of nario 7 – the one where performance degradation is the the most significant sources of contention – contention worst. For that, we compare experiments that differ from for the memory controller. each other precisely by one degradation factor involved. Furthermore, the migration of thread B to a different This allows us to single out the influence of this differ- memory domain creates two additional problems, which entiating factor on the application performance. Figure 5 degrade thread B’s performance. Assuming that thread shows the breakdown for the devil and soft-devil applica- B’s memory is physically located in Memory Node #1 tions. Turtles are not shown, because their performance (all operating systems of which we are aware would al- degradation is negligible. The overall degradation for locate B’s memory on Node #1 if B is running on a core each application relative to the base case is shown at the attached to Node #1 and then leave the memory on Node top of the corresponding bar. The y-axis shows the frac- #1 even after thread migration), B is now suffering from tion of the total performance degradation that each factor two additional sources of overhead: interconnect con- causes. Since contention causing factors on a real system tention and remote latency (labeled IC and RL respec- overlap in complex and integrated ways, it is not possi- tively in Figure 5). Although remote latency is not a ble to obtain a precise separation. These results are an crucially important factor, interconnect contention could approximation that is intended to direct attention to the hurt performance quite significantly. true bottlenecks in the system. To summarize, NUMA-agnostic migrations in the ex- The results show that of all performance-degrading isting contention management algorithms cause the fol- factors contention for cache constitutes only a very small lowing problems, listed in the order of severity according part, contributing at most 20% to the overall degrada- to their effect on performance: (1) They fail to eliminate tion. And yet, NUMA-agnostic migrations eliminate memory-controller contention; (2) They may create ad- only contention for the shared cache (CA), leaving the ditional interconnect contention; (3) They introduce re- more important factors (MC, IC, RL) unaddressed! Since mote latency overhead.
4 160 NO CACHE CONTENTION Scenario 1 200 CACHE CONTENTION Scenario 4 140 Scenario 6 Scenario 3 MCF MILC OMNETPP Scenario 7 120 Scenario 2 150 OMNETPP MCF MILC GCC Scenario 5 100 GCC BWAVES 80 BWAVES POVRAY 100 BZIP POVRAY 60 CALCULIX CALCULIX 40 BZIP H264 H264 50 20
0 0 Degradationdue to contention (%) Degradationdue to contention (%) Degradationdue to contention (%) -20 Degradationdue to contention (%) DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS ------DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS DEVILS TURTLES TURTLES TURTLES TURTLES TURTLES TURTLES TURTLES TURTLES TURTLES S S S S S S S S S ------TURTLES TURTLES TURTLES TURTLES TURTLES TURTLES TURTLES TURTLES TURTLES -40 S S S S S S S S S -50 Type of Competing Application Type of Competing Application (a) No cache contention (b) Cache contention
Figure 4: Performance degradation due to contention, cases 1-7 from Figure 3 relative to running contention free (case 0).
IC MC CACHE RL or DINO, that in addition to migrating thread memory 138% 182% 153% 113% 79% 75% 1 along with the thread eliminates superfluous migrations 0.9 and unlike other algorithms improves performance on 0.8 NUMA systems. 0.7 0.6 0.5 0.4 3.1 DI-Plain 0.3 0.2 DI-Plain works by predicting which threads will inter- 0.1 fere if co-scheduled on the same memory domain and Degradationfraction due to each factor Degradationfraction due to each factor 0 placing those threads on separate domains. Prediction MCF OMNETPP MILC GCC BWAVES BZIP is performed online, based on performance characteris- tics of threads measured via hardware counters. To pre- Figure 5: Contribution of each factor to the worst-case dict interference, DI uses the miss-rate heuristic – a mea- performance degradation. sure of last-level cache misses per thousand instructions, which includes the misses resulting from hardware pre- fetch requests. As we and other researchers showed in 3 A Contention-Aware Scheduling Algo- earlier work the miss-rate heuristic is a good approxima- rithm for NUMA Systems tion of contention: if two threads have a high LLC miss rate they are likely to compete for shared CPU resources We design a new contention-aware scheduling algo- and degrade each other’s performance [23, 2, 10, 16]. rithm for NUMA systems. We borrow the contention- Even though the miss rate does not capture the full modeling heuristic from the Distributed Intensity (DI) complexity of thread interactions on modern multicore algorithm, because it was shown to perform within 3% systems, it is an excellent predictor of contention for 1 percent of optimal on non-NUMA systems [23]. Other memory controllers and interconnects – key resource contention aware algorithms use similar principles as bottlenecks on these systems – because it reflects how DI [10, 16]. intensely threads use these resources. Detailed study We begin by explaining how the original DI algorithm showing why the miss rate heuristic works well and works (Section 3.1). For clarity we will refer to it from how it compares to other modeling heuristics is reported now on as DI-Plain. We proceed to show that simply in [23, 2]. extending DI-Plain to migrate memory – this version of DI-Plain continuously monitors the miss rates of run- the algorithm is called DI-Migrate – is not sufficient to ning threads. Once in a while (every second in the orig- achieve good performance on NUMA systems. We con- inal implementation), it sorts the threads according to clude with the description of our new DI-NUMA Online, their miss rates, and assigns them to memory domains so as to co-schedule low-miss-rate threads with high-miss- 1Although some experiments with DI reported in [23] were per- formed on a NUMA machine, the experimental environment was con- rate threads. It does so by first iterating over the sorted figured so as to eliminate any effects of NUMA. threads starting from the most memory-intensive (the one
5 with the highest miss rate) and placing each thread in a by physicists involved in ATLAS, an international par- separate domain, iterating over domains consecutively. ticle physics experiment at the Large Hadron Collider This way it separates memory-intensive threads. Then it at CERN. The WestGrid site at our university is inter- iterates over the array from the other end, starting from ested in deploying contention management algorithms on the least memory-intensive thread, placing each on an their clusters. Prospect of adoption of contention man- unused core in consecutive domains. Then it iterates agement algorithms in a real setting also motivated their from the other end of the array again, and continues alter- user-level implementation – not requiring a custom ker- nating iterations until all threads have been placed. This nel makes the adoption less risky. Our algorithms are im- strategy results in balancing the memory intensity across plemented on Linux as user-level daemons that measure domains. DI-Plain performs no memory migration when threads’ miss rates using perfmon, migrate threads us- it migrates the threads. ing scheduling affinity system calls, and move memory Existing operating systems (Linux, Solaris) would not using the numa migrate pages system call. move the thread’s memory to another node when a thread For SPEC CPU we show one workload for brevity; is moved to a new domain. Linux performs new memory complete results are presented in Section 5. All bench- allocations in the new domain, but will leave the mem- marks in the workload are launched simultaneously and ory allocated before migration in the old one. Solaris if one benchmark terminates it is restarted until each will act similarly2. So on either of these systems, if the benchmark completes three times. We use the result of thread after migration keeps accessing the memory that the second execution for each benchmark, and perform was allocated on another domain, it will cause negative the experiment ten times, reporting the average of these performance effects described in Section 2. runs. For SPEC MPI we show results for eleven different MPI jobs. In each experiment we run a single job, each 3.2 DI-Migrate comprised of 16 processes. We perform ten runs of each Our first (and obvious) attempt to make DI-Plain job and present the average completion times. NUMA-aware was to make it migrate the thread’s mem- We compare performance under DI-Plain and DI- ory along with the thread. We refer to this “intermedi- Migrate relative to the default Linux Completely Fair ate” algorithm in our design exploration as DI-Migrate. Scheduler, to which we refer as Default. Standard de- The description of the memory migration algorithm is viation across the runs is under 6% for the DI algo- deferred until Section 4, but the general idea is that it rithms. Deviation under Default is necessarily high, be- detects which pages are actively accessed and migrates cause being unaware of resource contention it may force them to the new node along with a chunk of surrounding a low-contention thread placement in one run and a high- pages. For now we present a few experiments comparing contention mapping in another. Detailed comparison of DI-Plain with DI-Migrate. Our experiments will reveal deviations under different schedulers is also presented in that memory migration is insufficient to make DI-Plain Section 5. work well on NUMA systems, and this will motivate the Figures 6 and 7 show the average completion time im- design of DINO. provement for the SPEC CPU and SPEC MPI workloads Our experiments were performed on the same system respectively (higher numbers are better) under DI algo- as described in Section 2.1. rithms relative to Default. We draw two important con- The benchmarks shown in this section are scientific clusions. First of all, DI-Plain often hurts performance applications from SPEC CPU2006 and SPEC MPI2007 on NUMA systems, sometimes by as much as 36%. Sec- suites with reference sets in both cases. (In a later ond, while DI-Migrate eliminates performance loss and section we also show results for the multithreaded even improves it for SPEC CPU workloads, it fails to ex- Apache/MySQL workload.) We evaluated scientific ap- cel with SPEC MPI workloads, hurting performance by plications for two reasons. First, they are CPU-intensive as much as 25% for GAPgeofem. and often suffer from contention. Second, they were Our investigation revealed DI-Migrate migrated pro- of interest for our partner Western Canadian Research cesses a lot more frequently in the SPEC MPI work- Grid (WestGrid) – a network of compute clusters used load than in the SPEC CPU workload. While fewer than by scientists at Canadian universities and in particular 50 migrations per process per hour were performed for SPEC CPU workloads, but as many as 400 (per process) 2Solaris will perform new allocations in the new domain if a were performed for SPEC MPI! DI-Migrate will migrate thread’s home lgroup – a representation of a thread’s home memory a thread to a different core any time its miss rate (and its domain – is reassigned upon migration, but will not move the mem- position in the array sorted by miss rates) changes. For ory allocated prior to home lgroup reassignment. If the lgroup is un- changed, even new memory allocations will be performed in the old the dynamic SPEC MPI workload this happened rather domain. frequently and led to frequent migrations.
6 30 DI‐Plain ative contention at coarse granularity (and that is why it 20 DI‐Migrate was shown to perform within 3% of the optimal oracular 10 0 scheduler in DI) it does not perfectly predict how con- ‐10 tention is affected by small changes in the miss rate. ‐20 Figure 8 illustrates this point. It shows on the x-axis % improvement ‐30 SPEC CPU 2006 applications sorted in the decreasing or- ‐40 der by their performance degradation when co-scheduled on the same domain with three instances of itself, relative to running solo. The bars show the miss rates and the line shows the degradations3. In general, with the exception Figure 6: Improvement of completion time under DI- of one outlier mcf, if one application has a much higher Plain and DI-Migrate relative to the Default for a SPEC miss rate than another, it will have a much higher degra- CPU 2006 workload. dation. But if the difference in the miss rates is small, it is difficult to predict the relative difference in degradations. 10 5 What this means is that it is not necessary for the 0 scheduler to migrate threads upon small changes in the ‐5 ‐10 miss rate, only upon the large ones. ‐15 ‐20 DI‐Plain % improvement ‐25 DI‐Migrate ‐30 200% 40 180% MISSES per 1000 instr. 35 lu
fds 160% pop
milc degradation 30
leslie 140% 25 socorro zeusmp 120% lammps tachyon 100% 20 GemsFDTD GAPgeofem 80% 15 60% 10 40% Degradation(%) Figure 7: Improvement of completion time under DI- Degradation(%) 20% 5 0% 0 Plain and DI-Migrate relative to Default for eleven SPEC misses LLC per 1000 instructions MPI 2007 jobs.
Unlike on UMA systems, thread migrations are not cheap on NUMA systems, because you also have to Figure 8: Performance degradation due to contention and move the memory of the thread. No matter how efficient miss rates for SPEC CPU2006 applications. memory migrations are, they will never be completely free, so it is always worth reducing the number of migra- tions to the minimum, performing them only when they 3.3.2 Thread classification in DINO and multi- are likely to result in improved performance. Our analy- threaded support sis of DI-Migrate behaviour for the SPEC MPI workload To build upon this insight, we design DINO to organize revealed that oftentimes migrations resulted in a thread threads into broad classes according to their miss rates, placement that was not better in terms of contention than and to perform migrations only when threads change the placement prior to migration. This invited opportuni- their class, while trying to preserve thread-core affinities ties for improvement, which we used in design of DINO. whenever possible. Classes are defined as follows (again, we borrow the animalistic classification from previous 3.3 DINO work): 3.3.1 Motivation Class 1: turtles – fewer than two LLC misses per 1000 instructions. DINO’s key novelty is in eliminating superfluous thread Class 2: devils – 2-100 LLC misses per 1000 instruc- migrations – those that are not likely to reduce con- tions. tention. Recall that DI-Plain (Section 3.1) triggers mi- Class 3: super-devils – more than 100 LLC misses per grations when threads change their miss rates and their 1000 instructions. relative positions in the sorted array. Miss rates may change rather often, but we found that it is not necessary Threshold values for classes were chosen for our tar- to respond to every change in order to reduce contention. get architecture. Values for other architectures should be This insight comes from the observation that while 3We omit several benchmarks whose counters failed to record dur- the miss rate is an excellent heuristic for predicting rel- ing the experiment.
7 chosen by examining the relationship between the miss DINO then proceeds with the computation of the rates and degradations on that architecture. placement layout for the next interval. The placement Before we describe DINO in detail, we explain the layout defines how threads are placed on cores. It is special new features in DINO to deal with multithreaded computed by taking the most aggressive class instance applications. (a super devil in our example) and placing it on a core in First of all, DINO tries to co-schedule threads of the the first memory domain dom0, then the second aggres- same application on the same memory domain, provided sive (also a super devil) – on a core in the second domain that this does not conflict with DINO’s contention-aware and so on until we reach the last domain. Then we iter- assignment (described below). This assumes that per- ate from the opposite end of the array (starting with the formance improvement from co-operative data sharing least memory-intensive instance) and spread them across when threads are co-scheduled on the same domain are domains starting with dom3. We continue alternating be- much smaller than the negative effects of contention. tween two ends of the array until all class instances have This is true for many applications [22]. However, when been placed on cores. In our example, for the NUMA this assumption does not hold, DINO can be extended to machine with four memory domains and two cores per predict when co-scheduling threads on the same domain domain, the layout will be computed as follows: is more beneficial than separating them, using techniques domain: dom0 dom1 dom2 dom3 described in [9] or [19]. new_core: 0 1 2 3 4 5 6 7 When it is not possible to co-schedule all threads in an layout: D t D t d t d d application on the same domain, and if threads actively Although this example assumes that the number of share data, they will put pressure on memory controller threads equals the number of cores, the algorithm gen- and interconnects. While there is not much the sched- eralizes for scenarios when the number of threads is uler can do in this situation (re-designing the application smaller or greater than the number of cores. In the lat- is the best alternative), it must at least avoid migrating ter case, each core will have T “slots” that can be filled the memory back and forth, so as not to make the perfor- with threads, where T = num threads/num cores, mance worse. Therefore, DINO detects when the mem- and instead of taking one class-instance from the array ory is being “ping-ponged” between nodes and discon- at a time, DINO will take T . tinues memory migration in that case. Now that we determined the layout for class-instances, we are yet to decide which thread will fill each core-class 3.3.3 DINO algorithm description slot – any thread of the given class can potentially fill the slot corresponding to the class. In making this deci- We now explain how DINO works using an example. sion, we would like to match threads to class instances In every rebalancing interval, set to one second in our so as to minimize the number of migrations. And to implementation, DINO reads the miss rate of each thread achieve that, we refer to the matching solution for the from hardware counters. It then determines each thread’s old rebalancing interval, saved in the form of a tuple ar- class based on its miss rate. To reduce the influence ray:
4DINO assigns a unique thread ID to each thread in the workload. domain: dom0 dom1 dom2 dom3
8 old_core: 0 1 2 3 4 5 6 7 3.3.4 DINO’s Effect on Migration Frequency old_class: D t d t d t d t old_processID: 0 1 2 0 0 0 3 4 We conclude this section by demonstrating how DINO old_threadID: 0 1 2 3 4 5 6 7 is able to reduce migration frequency relative to DI- then the initial shape of the new solution would be: Migrate. Table 1 shows the average number of memory migrations per hour of execution under DI-Migrate and domain: dom0 dom1 dom2 dom3 new_core: 0 1 2 3 4 5 6 7 DINO for different applications from the workloads eval- new_class: D t D t d t d d uated in Section 3.2. The results for MPI jobs are given new_processID: 0 1 0 0 0 3 for one of its processes and not for the whole job. Due new_threadID: 0 1 3 4 5 6 to space limitations, we show the numbers for selected Then, the threads whose placement was not deter- applications that are representative of the overall trend. mined in the previous step – i.e., those whose old class is The numbers show that DINO significantly reduces the not the same as their current core’s new class, as deter- number of migrations. As will be shown in Section 5, mined by the new placement, will fill the unused cores this results in up to 30% performance improvements for according to their new class: jobs in the MPI workload. domain: dom0 dom1 dom2 dom3 new_core: 0 1 2 3 4 5 6 7 new_class: D t D t d t d d 4 Memory migration new_processID: 0 1 4 0 0 0 3 2 new_threadID: 0 1 7 3 4 5 6 2 The straightforward solution to implement memory mi- gration is to migrate the entire resident set of the thread Now that the thread placement is determined, DINO when the thread is moved to another domain. This does makes the final pass over the thread tuples to take not work for the following reasons. First of all, for mul- care of multithreaded applications. For each thread tithreaded applications, even those where data sharing is A it checks if there is another thread B of the same rare, it is difficult to determine how the resident set is multithreaded application (new processID(A) = partitioned among the threads. Second, even if the appli- new processID(B)) among the thread tuples not cation is single-threaded, if its resident set is large it will yet iterated so that B is not placed in the same memory not fit into a single memory domain, so it is not possi- domain with A. If there is one, we check the threads ble to migrate it in its entirety. Finally, we experimen- that are placed in the same memory domain with A. If tally found that even in cases where it is possible to mi- there is a thread C in the same domain with A, such that grate the entire resident set of a process, this can hurt per- new processID(A) != new processID(C) formance of applications with large memory footprints. and new class(B) = new class(C) then we So in this section we describe how we designed and im- switch tuples B and C in the new solution. In our plemented a memory migration strategy that determines example this would result in the following assignment: which of the thread’s pages are most profitable to migrate domain: dom0 dom1 dom2 dom3 when the thread is moved to a new core. new_core: 0 1 2 3 4 5 6 7 new_class: D t D t d t d d new_processID: 0 0 4 1 0 0 3 2 new_threadID: 0 3 7 1 4 5 6 2 4.1 Designing the migration strategy
DINO has complexity of O(N) in the number of In order to rapidly evaluate various memory migration threads. Since the algorithm runs at most once a sec- strategies, we designed a simulator based on a widely ond, this has little overhead even for a large number of used binary instrumentation tool for x86 binaries called threads. We found that more frequent thread rebalanc- Pin [15]. Using Pin, we collected memory access traces ing did not yield better performance. Relatively infre- of all SPEC CPU2006 benchmarks and then used a cache quent changes of thread affinities mean that the algo- simulator on top of Pin to determine which of those ac- rithm is best suited for long-lived applications, such as cesses would be LLC misses, and so require an access to the scientific applications we target in our study, data memory. analytics (e.g., MapReduce), or servers. When there’s To evaluate memory migration strategies we used a more threads than cores coarse-grained rebalancing is metric called Saved Remote Accesses (SRA). SRA is the performed by DINO, but fine-grained time sharing of percent of the remote memory accesses that were elimi- cores between threads is performed by the kernel sched- nated using a particular memory migration strategy (af- uler. If threads are I/O- or synchronization-intensive and ter the thread was migrated) relative to not migrating have unequal sleep-awake periods, any resulting load im- the memory at all. For example, if we detect every re- balance must be corrected, e.g., as in [16]. mote access and migrate the corresponding page to the
9 Table 1: Average number of memory migrations per hour of execution under DI-Migrate and DINO for applications evaluated in Section 3.2. SPEC CPU2006 SPEC MPI2007 soplex milc lbm gamess namd leslie lamps GAPgeofem socorro lu DI-Migrate 36 22 11 47 41 381 135 237 340 256 DINO 8 6 5 7 6 2 1 3 2 1
thread’s new memory node, we are eliminating all re- pages enables us to achieve the SRA as high as 74.9%. mote accesses, so the SRA would be 100%. We also confirmed experimentally that this was a good Each strategy that we evaluated detects when a thread value for K (results shown later). is about to perform an access to a remote domain, and migrates one or more memory pages from the thread’s virtual address space associated with the requested ad- 4.2 Implementation of the memory migra- dress. We tried the following strategies: sequential- tion algorithm forward where K pages including and following the Our memory migration algorithm is implemented for one corresponding to the requested address are mi- AMD systems, and so we use IBS, which we access via grated; sequential-forward-backward where K/2 pages Linux performance-monitoring tool perfmon [5]. sequentially preceding and K/2 pages sequentially fol- Migration in DINO is performed in a user-level dae- lowing the requested address are migrated; random mon running separately from the scheduling daemon. where randomly chosen K pages are migrated; pattern- The daemon wakes up every ten milliseconds, sets up based where we detect a thread’s memory-access pat- IBS to perform sampling, reads the next sample and mi- tern by monitoring its previous accesses, similarly to how grates the page containing the memory address in the hardware pre-fetchers do this, and migrate K pages that sample (if the sampled instruction was a load or a store) match the pattern. We found that sequential-forward- along with K pages in the application address space that backward was the most effective migration policy in sequentially precede and follow the accessed page. Page terms of SRA. migration is effected using the numa move pages system Another challenge in designing a memory migration call. strategy is minimizing the overhead of detecting which of the remote memory addresses are actually being ac- cessed. Ideally, we want to be able to detect every re- 5 Evaluation mote access and migrate the associated pages. However, on modern hardware this would require unmapping ad- 5.1 Workloads dress translations on a remote domain and handling a page fault every time a remote access occurs. This re- In this section we evaluate DINO implemented using the sults in frequent interrupts and is therefore expensive. migration strategy described in the previous section. We After analyzing our options we decided to use hard- evaluate three workload types: SPEC CPU2006 applica- ware counter sampling available on modern x86 systems: tions, SPEC MPI2007 applications, and LAMP – Lin- PEBS (Precise Event-Based Sampling) on Intel proces- ux/Apache/MySQL/PHP. sors and IBS (Instruction-Based Sampling) on AMD pro- We used two experimental systems for evaluation. cessors. These mechanisms tag a sample of instruction One was described in Section 2.1. Another one is a Dell with various pieces of information; load and store in- PowerEdge server equipped with two AMD Barcelona structions are annotated with the memory address. processors running at 2GHz, and 8GB of RAM, 4GB per While hardware-based event sampling has low over- domain. The operating system is Linux 2.6.29.6. The head, it also provides relatively low sampling accuracy – experimental design for SPEC CPU and MPI workloads on our system it samples less than one percent of instruc- was described in Section 3.2. The LAMP workload is tions. So we also analysed how SRA is affected depend- described below. ing on the sampling accuracy as well as the number of The LAMP acronym is used to describe the applica- pages that are being migrated. The lower the accuracy, tion environment consisting of Linux, Apache, MySQL the higher the value of K (pages to be migrated) needs and PHP. The main data processing in LAMP is done to be to achieve a high SRA. For the hardware sampling by the Apache HTTP server and the MySQL database accuracy that was acceptable in terms of CPU overhead engine. The server management daemons apache2 and (less than 1% per core), we found that migrating 4096 mysqld are responsible for arranging access to the web-
10 site scripts and database files and performing the ac- resident-set migration when K reaches 4096. Whole- tual work of data storage and retrieval. We use Apache resident set migration actually works quite well for 2.2.14 with PHP version 5.2.12 and MySQL 5.0.84. Both this workload, because migrations are performed infre- apache2 and mysqld are multithreaded applications that quently and the resident set is small. spawn one new distinct thread for each new client con- However upon experimenting with the LAMP work- nection. This client thread within a daemon is then re- load we found that whole-resident set migration was sponsible for executing the client‘s request. detrimental to performance, most likely because the resi- In our experiment, clients continuously retrieve from dent sets were much larger and also because this is a mul- the Apache server various statistics about website activ- tithreaded workload where threads share data. Figure 10 ity. Our database is populated with the data gathered by shows performance and deviation improvement when the web statistics system for five real commercial web- K = 4096 relative to whole-resident-set migration. Per- sites. This data includes the information about website‘s formance is substantially improved when K = 4096. audience activity (what pages on what website were ac- We experimented with smaller values of K, but found cessed, in what order, etc.) as well as the information no substantial differences on performance. about visitors themselves (client OS, user agent informa- We conclude that migrating very large chunks of mem- tion, browser settings, session id retrieved from the cook- ory is acceptable for processes with small resident sets, ies, etc.). The total number of records in the database is but not advisable for multithreaded applications and/or more than 3 million. We have four Apache daemons, applications with large resident sets. DINO migrates each responsible for handling a different type of request. threads infrequently, so a relatively large value of K re- There are also four MySQL daemons that perform main- sults in good performance. tenance of the website database. K 16 average me improvement (%) We further demonstrate the effect that the choice of 14 worst me improvement (%) (the number of pages that are moved on every migration) 12 me devia on improvement (%) 10 has on performance of DINO. Then we compare DINO 8 to DI-Plain, DI-Migrate and Default. 6 4 2 % improvement 0 ‐2 5.2 Effect of K ‐4 Two of our workloads, SPEC CPU and LAMP demon- strate the key insights, and so we focus on those work- loads. We show how performance changes as we vary the value of K. We compare to the scenario where DINO Figure 10: Performance improvement with DINO for migrates the thread’s entire resident set upon migrating K = 4096 relative to whole-resident-set migration for the thread itself. The per-process resident sets of the two LAMP. chosen workloads could actually fit in a single memory node on our system (it had 4GB per node), so whole- resident-set migration was possible. For SPEC CPU 5.3 DINO vs. other algorithms applications, resident sets vary from under a megabyte to 1.6GB for mcf. In general, they are in hundreds of We compare performance under DINO, DI-Plain and DI- megabytes for memory-intensive applications and much Migrate relative to Default, and similarly to the previous smaller for others. In LAMP, MySQL’s resident set was section, report completion time improvement, worst-case about 400MB and Apache’s was 120MB. execution time improvement and deviation improvement. We show average completion time improvement (for Figures 11-13 show the results for the three work- Apache/MySQL this is average completion time per re- load types, SPEC CPU, SPEC MPI and LAMP respec- quest), worst-case execution time improvement, and de- tively. For SPEC CPU, DI-Plain hurts completion time viation improvement. Completion time improvement is for many applications, but both DI-Migrate and DINO the average over ten runs. To compute the worst-case ex- improve, with DINO performing slightly better than DI- ecution time we run each workload ten times and record Migrate for most applications. Worst-case improvement the longest completion time. Improvement in deviation numbers show a similar trend, although DI-Plain does is the percent reduction in standard deviation of the aver- not perform as poorly here. Improvements in the worst- age completion time. case execution time indicate that a scheduler is able to Figure 9 shows the results for the SPEC CPU work- avoid pathological thread assignments that create espe- loads. Performance is hurt when we migrate a small cially high contention, and produce more stable perfor- number of pages, but becomes comparable to whole- mance. Deviation of running times is improved by all
11 5 5 5 0 0 0 ‐5 ‐5 ‐5 ‐10 ‐10 ‐10 ‐15 ‐15 ‐15 average me improvement (%) ‐20 ‐20 ‐20 worst me improvement (%)
average me improvement (%) % improvement % improvement
‐25 % improvement ‐25 ‐25 me devia on improvement (%) worst me improvement (%) ‐30 average me improvement (%) ‐30 me devia on improvement (%) ‐30 ‐35 worst me improvement (%) ‐35 ‐35 me devia on improvement (%)
(a) 1 page(4KB) (b) 256 pages (1MB) (c) 4096 pages (16MB)
Figure 9: Performance improvement with DINO as K is varied relative to whole-resident-set migration for SPEC CPU.
20 DI‐Plain 30 DI‐Plain 40 25 DI‐Plain 20 DI‐Migrate DI‐Migrate 20 DI‐Migrate 30 15 DINO DINO 15 DINO 20 10 10 10 5 0 10 5 0 ‐10 0 ‐5 0 ‐10 ‐10 ‐20 DI‐Plain ‐15 ‐20 DI‐Migrate ‐5 ‐30 ‐20 ‐30 DINO ‐10 ‐40 % average me improvement % average me improvement % average me improvement % average me improvement
(a) Average execution time improvement of DINO (IBS), DI-Migrate (IBS) and DI-Plain over Default Linux scheduler.
35 DI‐Plain 70 70 DI‐Plain 45 DI‐Plain 30 DI‐Migrate 60 40 DI‐Migrate 60 DI‐Migrate 35 25 DINO 50 DINO 40 50 DINO 30 20 25 30 40 20 15 20 30 15 10 10 20 10 5 0 5 10 0 0 ‐10 ‐20 DI‐Plain 0 ‐5 ‐5 DI‐Migrate ‐10 ‐30 ‐10 % worst me improvement % worst me improvement ‐10 % worst me improvement ‐40 DINO % worst me improvement
(b) Worst-case execution time improvement of DINO (IBS), DI-Migrate (IBS) and DI-Plain over Default Linux scheduler.
8 25 25 DI‐Plain 14 DI‐Plain DI‐Plain DI‐Migrate 12 DI‐Migrate DI‐Migrate 6 20 20 DINO DINO DINO 10 4 15 8 15 10 6 2 4 10 0 5 2 0 5 0 ‐2 DI‐Plain ‐2 % devia on improvement % devia on improvement % devia on improvement % devia on improvement ‐5 ‐4 DI‐Migrate DINO 0
(c) Deviation improvement of DINO (IBS), DI-Migrate (IBS) and DI-Plain over Default Linux scheduler.
Figure 11: DINO, DI-Migrate and DI-Plain relative to Default for SPEC CPU 2006 workloads. three schedulers relative to Default. they still share data, putting pressure on interconnects. As to SPEC MPI workloads (Figure 12) only DINO is Nevertheless, DINO still manages to improve comple- able to improve completion times across the board, by as tion time and worst-case execution time in some cases, much as 30% for some jobs. DI-Plain and DI-Migrate, to a larger extent than the other two algorithms. on the other hand, can hurt performance by as much as 20%. Worst-case execution time also consistently im- 5.4 Discussion proves under DINO, while sometimes degrading under DI-Plain and DI-Migrate. Our evaluation demonstrates that DINO is significantly LAMP is a tough workload for DINO or any scheduler better at managing contention on NUMA systems than that optimizes memory placement, because the workload the DI algorithm designed without NUMA awareness or is multithreaded and no matter how you place threads DI that was simply extended with memory migration.
12 40 DI‐Plain 40 DI‐Plain 10 DI‐Plain 30 DI‐Migrate 30 DI‐Migrate 8 DI‐Migrate DINO 6 DINO 20 DINO 20 4 10 10 2 0 0 0 ‐10 ‐10 ‐2 ‐4 ‐20 ‐20 ‐6 ‐30 ‐30 ‐8 % devia on improvement lu lu lu % worst me improvement fds fds fds pop pop pop milc milc milc % average me improvement leslie leslie leslie socorro socorro socorro zeusmp zeusmp zeusmp lammps lammps lammps tachyon tachyon tachyon GemsFDTD GemsFDTD GemsFDTD GAPgeofem GAPgeofem GAPgeofem (a) Completion time improvement (b) Worst-case time improvement (c) Deviation improvement
Figure 12: DINO, DI-Migrate and DI-Plain relative to Default for SPEC MPI 2007.
16 DI‐Plain 20 DI‐Plain 3 DI‐Plain 14 DI‐Migrate DI‐Migrate 15 2 DI‐Migrate 12 DINO DINO 10 DINO 8 10 1 6 4 5 0 2 0 ‐1 0 ‐2 ‐5 ‐2 ‐4
‐6 ‐10 % devia on improvement ‐3 % worst me improvement % average me improvement
(a) Completion time improvement (b) Worst-case time improvement (c) Deviation improvement
Figure 13: DINO, DI-Migrate and DI-Plain relative to Default for LAMP.
Multiprocess workloads representative of scientific Grid as well as surrounding pages to the new domain. clusters show excellent performance under DINO. Im- LaRowe et al. [12] presented a dynamic multiple-copy provements for the challenging multithreaded workloads policy placement and migration policy for NUMA sys- are less significant as expected, and wherever degrada- tems. The policy periodically reevaluates its memory tion occurs for some threads it is outweighed by perfor- placement decisions and allows multiple physical copies mance improvements for other threads. of a single virtual page. It supports both migration and replication with the choice between the two operations 6 Related Work based on reference history. A directory-based invalida- tion scheme is used to ensure the coherence of replicated Research on NUMA-related optimizations to systems is pages. The policy applies a freeze/defrost strategy: to de- rich and dates back many years. Many research efforts termine when to defrost a frozen page and trigger reeval- addressed efficient co-location of the computation and uation of its placement is based on both time and refer- related memory on the same node [14, 3, 12, 19, 1, 4]. ence history of the page. The authors evaluate various More ambitious proposals aimed to holistically redesign fine-grained page migration and/or replication strategies, the operating system to dovetail with NUMA architec- however, since their test machine only has one processor tures [7, 17, 6, 20, 11]. None of the previous efforts, how- per NUMA node, they do not address contention. The ever, addressed shared resource contention in the context strategies developed in this work could have been very of NUMA systems and the associated challenges. useful for our contention aware scheduler if the inexpen- Li et al. in [14] introduced AMPS, an operating sys- sive mechanisms that the authors used for detecting page tem scheduler for asymmetric multicore systems that accesses were available to us. Detailed page reference supports NUMA architectures. AMPS implemented a history is difficult to obtain without hardware support; NUMA-aware migration policy that can allow or deny obtaining it in software may cause overhead for some thread migration requested by the scheduler. The authors workloads. used the resident set size of a thread in deciding whether Goglin et al. [8] developed an effective implementa- or not the OS schedule is allowed to migrate thread to tion of the move pages system call in Linux, which al- a different domain. If the migration overhead were ex- lows the dynamic migration of large memory areas to pected to be high the migration would be disallowed. be significantly faster than in previous versions of the Our scheduler, instead of prohibiting migrations, detects OS. This work is integrated in Linux kernel 2.6.29 [8], which pages are being actively accessed and moves them which we use for our experiments. The Next-touch pol-
13 icy, also introduced in the paper to facilitate thread-data degradation versus the benefits from co-scheduling, we affinity, works as follows: the application marks pages would need stronger hardware support, such as that avail- that it will likely access in the future as Migrate-on-next- able on IBM Open-Power 720 PCs or on the newest Ne- touch using a new parameter to the madvise() system halem systems (as demonstrated by the member of our call. The Linux kernel then ensures that the next access team [9]). to these pages causes a special page fault resulting in the The VMware ESX hypervisor supports NUMA load pages being migrated to their threads. The work pro- balancing and automatic page migration for its vir- vides developers with an opportunity to improve mem- tual machines (VMs) in commercial systems [1]. ESX ory proximity for their programs. Our work, on the other Server 2 assigns each virtual machine a home node on hand, improves memory proximity by using hardware whose processors a VM is allowed to run and its newly- counters data available on every modern machine. No allocated memory comes from the home node as well. involvement from the developer is needed. Periodically, a special rebalancer module selects a VM Linux kernel since 2.6.12 supports the cpusets mech- and changes its home node to the least-loaded node. In anism and its ability to migrate the memory of the ap- our work we do not consider load balancing. Instead, plications confined to the cpuset along with their threads we make thread migration decisions based on shared re- to the new nodes if the parameters of a cpuset change. source contention. To eliminate possible remote access Schermerhorn et al. further extended the cpuset function- penalties associated with accessing the memory on the ality by adding an automatic page migration mechanism old node, ESX Server 2 performs page migration from to it [18]: if enabled, it migrates the memory of a thread the virtual machine’s original node to its new home node. within the cpuset nodes whenever the thread migrates to ESX selects migration candidates based on finding hot a core adjacent to a different node. Two options for the remotely-accessed memory from page faults. The DINO memory migration are possible. The first is a lazy mi- scheduler, on the other hand, identifies hot pages using gration, when the kernel attempts to unmap any anony- Instruction-Based Sampling. No modification to the OS mous pages in the process’s page table. When the pro- is required. cess subsequently touches any of these unmapped pages, The SGI Origin 2000 system [4] implemented the the swap fault handler will use the ”migrate-on-fault” following hardware-supported [13] mechanism for co- mechanism to migrate the misplaced pages to the correct location of computation and memory. When the dif- node. Lazy migration may be disabled, in which case, ference between remote and local accesses for a given automigration will use direct, synchronous migration to memory page is greater than a tunable threshold, an in- pull all anonymous pages mapped by the process to new terrupt is generated to inform the operating system that node. The efficiency of lazy automigration is compara- the physical memory page is suffering an excessive num- ble to our memory migration solution based on IBS (we ber of remote references and hence has to be migrated. performed experiments to verify). However, automigra- Our solution to page migration is different in that it tion requires kernel modification (it is implemented as a detects ”hot” remotely accessed pages via Instruction- collection of kernel patches), while our solution is imple- Based Sampling, and performs migration in the context mented on user level. Cpuset mechanism needs explicit of a contention-aware scheduler. configuration from the system administrator and it does In a series of papers [7] [17] [6] [20] the authors not perform contention management. describe a novel operating system Tornado specifically In [19] the authors group threads of the same applica- designed for NUMA machines. The goal of this new tion that are likely to share data onto neighbouring cores OS is to provide data locality and application indepen- to minimize the costs of data sharing between them. dence for OS objects thus minimizing penalties due to re- They rely on several features of Performance Monitor- mote memory access in a NUMA system. The K42 [11] ing Unit unique to IBM Open-Power 720 PCs: the abil- project, which is based on Tornado, is an open-source ity to monitor CPU stall breakdown charged to different research operating system kernel that incorporates such microprocessor components and using the data sampling innovative design principles like structuring the system to track the sharing pattern between threads. The DINO using modular, object-oriented code (originally demon- algorithm introduced in our work complements [19] as it strated in Tornado), designing the system to scale to very is designed to mitigate contention between applications. large shared-memory multiprocessors, avoiding central- DINO provides sharing support by attempting to group ized code paths and global locks and data structures and threads of the same application and their memory on the many more. K42 keeps physical memory close to where same NUMA node, but as long as co-scheduling multiple it is accessed. It uses large pages to reduce hardware threads of the same application does not contradict with and software costs of virtual memory. K42 project has a contention-aware schedule. In order to develop a more resulted in many important contributions to Linux, on precise metric that assesses the effects of performance which our work relies. As a result, we were able to avoid
14 deleterious effects of remote memory accesses without [8] GOGLIN,B., AND FURMENTO, N. Enabling High-Performance requiring changes to the applications or the operating Memory Migration for Multithreaded Applications on Linux. In system. We believe that our NUMA contention-aware Proceedings of IPDPS (2009). scheduling approach that was demonstrated to work ef- [9] KAMALI, A. Sharing Aware Scheduling on Multicore Systems. Master’s thesis, Simon Fraser University, 2010. fectively in Linux can also be easily implemented in K42 with its inherent user-level implementation of ker- [10] KNAUERHASE,R.,BRETT, P., HOHLT,B.,LI, T., AND HAHN, S. Using OS Observations to Improve Performance in Multicore nel functionality and native performance monitoring in- Systems. IEEE Micro 28, 3 (2008), pp. 54–66. frastructure. [11] KRIEGER,O.,AUSLANDER,M.,ROSENBURG,B.,WIS- NIEWSKI, R. W., XENIDIS,J.,DA SILVA,D.,OSTROWSKI, M.,APPAVOO,J.,BUTRICO,M.,MERGEN,M.,WATERLAND, 7 Conclusions A., AND UHLIG, V. K42: Building a Complete Operating Sys- tem. In Proceedings of EuroSys (2006).
We discovered that contention-aware algorithms de- [12] LAROWE, R. P., JR.,ELLIS,C.S., AND HOLLIDAY,M.A. signed for UMA systems may hurt performance on sys- Evaluation of NUMA Memory Management Through Modeling tems that are NUMA. We found that contention for mem- and Measurements. IEEE Transactions on Parallel and Dis- ory controllers and interconnects occurring when thread tributed Systems 3 (1991), 686–701. runs remotely from its memory are the key causes. To ad- [13] LAUDON,J., AND LENOSKI, D. The SGI Origin: a ccNUMA highly scalable server. In Proceedings of ISCA (1997). dress this problem we presented DINO: a new contention management algorithm for NUMA systems. While de- [14] LI, T., BAUMBERGER,D.,KOUFATY,D.A., AND HAHN, S. Efficient Operating System Scheduling for Performance- signing DINO we found that simply migrating a thread’s Asymmetric Multi-core Architectures. In Proceedings of Super- memory when the thread is moved to a new node is not a computing (2007). sufficient solution; it is also important to eliminate super- [15] LUK,C.-K.,COHN,R.,MUTH,R.,PATIL,H.,KLAUSER,A., fluous migrations: those that add to migration cost with- LOWNEY,G.,WALLACE,S.,REDDI, V. J., AND HAZELWOOD, out providing the benefit. The goals for our future work K. Pin: Building Customized Program Analysis Tools with Dy- namic Instrumentation. In Proceedings of PLDI (2005). are (1) devising metric for predicting a trade-off between performance degradation and benefits from thread shar- [16] MERKEL,A.,STOESS,J., AND BELLOSA, F. Resource- Conscious Scheduling for Energy Efficiency on Multicore Pro- ing and (2) investigate the impact of using small versus cessors. In Proceedings of EuroSys (2010). large memory pages during migration. [17] PARSONS,E.,GAMSA,B.,KRIEGER,O., AND STUMM,M. (De-)Clustering Objects for Multiprocessor System Software. In References Proceedings of IWOOS (1995). [18] SCHERMERHORN, L. T. Automatic Page Migration for Linux. [1] VMware ESX Server 2 NUMA Support. White paper. [Online] Available: http://www.vmware.com/pdf/esx2 NUMA.pdf . [19] TAM,D.,AZIMI,R., AND STUMM, M. Thread Clustering: Sharing-Aware Scheduling on SMP-CMP-SMT Multiprocessors. [2] BLAGODUROV,S.,ZHURAVLEV,S., AND FEDOROVA,A. In Proceedings of EuroSys (2007). Contention-aware scheduling on multicore systems. ACM Trans. Comput. Syst. 28 (December 2010), 8:1–8:45. [20] UNRAU,R.C.,KRIEGER,O.,GAMSA,B., AND STUMM,M. [3] BRECHT, T. On the Importance of Parallel Application Place- Hierarchical Clustering: a Structure for Scalable Multiprocessor ment in NUMA Multiprocessors. In USENIX SEDMS (1993). Operating System Design. J. Supercomput. 9, 1-2 (1995), 105– 134. [4] CORBALAN,J.,MARTORELL,X., AND LABARTA, J. Eval- uation of the Memory Page Migration Influence in the System [21] XIE, Y., AND LOH, G. Dynamic Classification of Program Performance: the Case of the SGI O2000. In Proceedings of Su- Memory Behaviors in CMPs. In Proceedings of CMP-MSI percomputing (2003), pp. 121–129. (2008). [5] ERANIAN, S. What can performance counters do for memory [22] ZHANG,E.Z.,JIANG, Y., AND SHEN, X. Does cache shar- subsystem analysis? In Proceedings of MSPC (2008). ing on modern cmp matter to the performance of contemporary [6] GAMSA,B.,KRIEGER,O., AND STUMM, M. Optimizing IPC multithreaded programs? In Proceedings of PPOPP (2010). Performance for Shared-Memory Multiprocessors. In Proceed- [23] ZHURAVLEV,S.,BLAGODUROV,S., AND FEDOROVA, A. Ad- ings of ICPP (1994). dressing Contention on Multicore Processors via Scheduling. In [7] GAMSA,B.,KRIEGER,O., AND STUMM, M. Tornado: Maxi- Proceedings of ASPLOS (2010). mizing Locality and Concurrency in a Shared Memory Multipro- cessor Operating System. In Proceedings of OSDI (1999).
15