Understanding the Spatial Characteristics of DRAM Errors in HPC Clusters∗ Ayush Patwari Ignacio Laguna Purdue University Lawrence Livermore National Laboratory [email protected] [email protected] Martin Schulz Saurabh Bagchi Lawrence Livermore National Laboratory Purdue University [email protected] [email protected] ABSTRACT as DRAM, this problem will only continue to grow. We therefore Understanding DRAM errors in high-performance computing (HPC) need a thorough understanding of DRAM errors in HPC systems clusters is paramount to address future HPC resilience challenges. to address the resilience challenges of future exascale systems. While there have been studies on this topic, previous work has While previous studies on DRAM errors in HPC systems exist, focused on on-node and single-rack characteristics of errors; con- most of this work has been focused on on-node (e.g., chip-level) [7, versely, few studies have presented insights into the spatial behavior 9] and single-rack error characteristics (e.g., top/bottom of the of DRAM errors across an entire cluster. Understanding spatial pe- rack) [10]. However, no previous work has provided insights into culiarities of DRAM errors through an entire cluster is crucial for DRAM error characteristics with respect to the physical layout cluster temperature management, job allocation, and failure pre- of affected nodes and DRAM chips. Understanding such spatial diction. In this paper, we study the spatial nature of DRAM errors characteristics of DRAM errors is of great value to HPC centers on data gathered in a large production HPC cluster. Our analysis for room temperature management, job scheduling and allocation shows that nodes with high degree of errors are grouped in spatial policies, failure prediction, and overall system resilience. regions for time periods, suggesting that these “susceptible” regions In this paper, we present a study of the spatial and temporal are collectively more vulnerable to errors than other regions. We behavior of DRAM errors in a production cluster at the Lawrence then use our observations to build a predictor, which identifies such Livermore National Laboratory (LLNL). The goal of this study is regions given prior neighboring regions patterns. to provide insights into the spatial correlation of errors in DRAM modules and nodes across the entire cluster and across individual CCS CONCEPTS racks. In particular, we employ statistical and data analytics models that help us answer the following research questions: (a) given • Computer systems organization → Reliability; • Hardware a node n that experienced a large number of DRAM errors in a → Transient errors and upsets; given period of time t, what is the probability that another neighbor ACM Reference format: node m, at distance d from n, will also experience a large number Ayush Patwari, Ignacio Laguna, Martin Schulz, and Saurabh Bagchi. 2017. of errors? and (b) can we identify spatial groups of nodes that Understanding the Spatial Characteristics of DRAM Errors in HPC Clusters. experience a high degree of errors versus groups of nodes that In Proceedings of FTXS’17, Washington , DC, USA, June 26, 2017, 6 pages. do not experience such behavior? Our data comprises records of https://doi.org/http://dx.doi.org/10.1145/3086157.3086164 correctable DRAM errors of a 1296-node cluster at LLNL (Cab), which was gathered for period of 14 months (May/2013–July/2014). 1 INTRODUCTION In summary, our main contributions are: Dynamic Random Access Memory (DRAM) errors are a common • We develop a spatial analysis framework to visualize the source of failures in High-Performance Computing (HPC) clus- errors as seen in the physical layout of the cluster. We ters [5, 7–9]. With exascale machines estimated to have up to show that several nodes, which report errors in the same hundreds of petabytes of main memory, typically implemented time-frame, or epoch, are spatially grouped together. This ∗This work was performed under the auspices of the U.S. Department of Energy by phenomenon can be seen in several epochs. Lawrence Livermore National Laboratory under contract DEAC52-07NA27344 (LLNL- • We introduced metrics for the number of neighboring er- CONF-727473), and supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, program manager Lucy Nowell. roneous nodes and the minimum distance from a neighbor erroneous node. A statistical analysis of these metrics sug- Permission to make digital or hard copies of all or part of this work for personal or gests that error grouping is not due to a random occurrence, classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation but rather due to external factors. This also reveals inter- on the first page. Copyrights for components of this work owned by others than ACM esting insights into the probability of a node having errors must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, given prior error patterns in neighboring regions. to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. • We design a classification model to predict whether a node FTXS’17, June 26, 2017, Washington , DC, USA will experience errors at any given point given prior his- © 2017 Association for Computing Machinery. tory and build a model based on the above observations ACM ISBN 978-1-4503-5001-3/17/06...$15.00 https://doi.org/http://dx.doi.org/10.1145/3086157.3086164 to classify data samples as to whether they would report a FTXS’17, June 26, 2017, Washington , DC, USA Ayush Patwari, Ignacio Laguna, Martin Schulz, and Saurabh Bagchi A rack of A colored square Black color is empty nodes represents a node space in the cluster Racks row 1 The cluster is composed of two rows of racks Aisle that separates the rows of racks Racks row 2 Figure 1: Snapshot of memory errors recorded between 5/20/2013 and 6/19/2013 on the Cab cluster displayed on the physical layout of the system. Nodes without errors are shown in white, while nodes with at least one error in the current epoch are colored. The black regions are either non-compute nodes or switches for the IB interconnect. The central horizontal gray block represents a hot aisle separating the two rows of racks. new error on a node or not. This model is able to achieve latent, i.e., they are not detected until the affected cell in the DIMM a high F1-score (a metric that combines recall and preci- is accessed. Because of this, many accesses of an affected cell may sion) on a highly-unbalanced dataset (only 650 out of 251k generate a large number of errors in a short period of time. Since we samples report a new error). do not have memory access data for the nodes we cannot distinguish an erroneous node with few errors from one with many errors. 2 TERMINOLOGY Number of errors. The total number of correctable errors ob- served on a node for the given epoch. We first introduce some definitions and notations used in the paper. Timestamp. Record of the time at which a sample was collected. Correctable Error (CE). Errors on a DRAM DIMM (Dual In-line Time since last reset. Uncorrectable errors are normally fol- Memory Module) in a node, which are caused by transient, hard, or lowed by a kernel panic and the node is rebooted. When this occurs, intermittent faults, but which can be corrected transparently using the error counters for the node are reset and error data between techniques like ECC. the beginning of the epoch and the hard reset is lost. This metric Uncorrectable Error (UE). Errors on a DRAM DIMM in a node, measures the time elapsed since the last time a reset occurred. which cannot be corrected by the memory system and typically lead to application aborts or node crashes. Epoch. A parameter of our study that defines a period of time 3 EXPERIMENTAL SETUP in which errors are analyzed. For all our analysis in this paper, an We describe the experimental setup for this work and introduce epoch is taken to be one month. the notations used in the following sections. Erroneous Node. A node on which at least one correctable error was observed during the epoch being considered. Note that if a node had errors in a previous epoch, it will not be considered an 3.1 Data Gathering erroneous node in the current epoch if it does not have reported As mentioned earlier, we use DRAM error data collected from the new errors. Also note that we do not make a distinction between Cab cluster at LLNL over a period of 14 months from May 2013 two nodes, one with a single error and one with many errors in to July 2014. Cab has a total of 1296 nodes connected with an an epoch—both would be considered erroneous nodes. The reason InfiniBand (IB) network and each node has two Intel 8-Core Xeon for this is that DRAM errors have the characteristic that they are E5-2670 processors. The physical layout can be seen in the Fig. 1. Understanding the Spatial Characteristics of DRAM Errors in HPC Clusters FTXS’17, June 26, 2017, Washington , DC, USA The cluster uses the TLCC operating system, which is a derivate of After experimenting with different epoch time scales, we observed RHEL, and has x4 DDR3 1600MHz memory running in S4ECD4ED. the following significant behaviors. EDAC (Error Detection and Correction). We used EDAC [1] Spatial Node Grouping. By studying the erroneous nodes in to collect error data on the cluster.