GILES-DOCUMENT-2019.Pdf
Total Page:16
File Type:pdf, Size:1020Kb
ABSTRACT Hardware Transactional Persistent Memory by Ellis Giles Recent years have witnessed a sharp shift towards real-time data-driven and high- throughput applications, impelled by pervasive multi-core architectures and paral- lel programming models. This shift has spurred a broad adoption of in-memory databases and massively-parallel transaction processing across scientific, business, and industrial application domains. However, these applications are severely hand- icapped by the difficulties in maintaining persistence on typical durable media like hard-disk drives (HDDs) and solid-state drives (SSDs) without sacrificing either per- formance or reliability. Two emerging hardware developments hold enormous promise for transformative gains in both speed and scalability of concurrent data-intensive applications. The first is the arrival of Persistent Memory, or PM, a generic term for byte-addressable non- volatile memories, such as Intel's 3D XPointTM technology. The second is the avail- ability of CPU-based transaction support known as Hardware Transactional Mem- ory, or HTM, which makes it easier for applications to exploit multi-core concurrency without the need for expensive lock-based software. This thesis introduces Hardware Transactional Persistent Memory, the first union of HTM with PM without any changes to known processor designs or protocols, al- lowing for high-performance, concurrent, and durable transactions. The techniques presented are supported on three pillars: handling uncontrolled cache evictions from the processor cache hierarchy, logging to resist failure during persistent memory up- dates, and transaction ordering to permit consistent recovery from a machine crash. We develop pure software solutions that work with existing processor architectures as well as software-assisted solutions that exploit external memory controller hardware support. The thesis also introduces the notion of relaxed versus strict durability, allowing individual applications to tradeoff performance against robustness, while guaranteeing recovery to a consistent system state. Dedication This thesis is dedicated in memory of my father Vick Forrest Giles and as an inspi- ration to my children Alden Samuel Giles and Meredith Elise Giles. Acknowledgments First and foremost, I would like to thank my advisor Dr. Peter Varman for the many meetings and countless hours helping me develop research ideas and pointing out interesting areas for more exploration. Only with his guidance and support has this research, including this thesis and the many papers, been possible. I would also like to thank Dr. Kshitij Doshi from Intel Corporation for all of the guidance, insight, and assistance into the area of research along with the hours of help spent with papers and research ideas. Additionally, I would like to thank Dr. Joseph Cavallaro, Dr. Chris Jermaine, and Dr. Eugene Ng for providing valuable feedback and taking the time to serve on my thesis committee. Finally, I would like to thank all of my family. Without the continuing support from the entire family, I would not have been able to attend graduate school or immerse myself in research. My wife Mary-Helen was invaluable at providing support and helping with the children when graduate school made demands on my time. May this thesis inspire my children Alden Samuel Giles and Meredith Elise Giles to pursue higher education. My father Vick F. Giles also provided daily support for the children and guidance on my education. He would have wished to have seen the completion of the work and attended graduation. My mother and stepfather, Linda and Douglas Rowlett, also provided invaluable help and guidance along the way as well as inspiration with their Ph.D.'s from Rice University. I would also like to acknowledge the financial support from the National Science Foundation and the Software and Services Group at Intel Corporation. Contents Abstract ii Dedication iv Acknowledgments v List of Illustrations xi List of Tables xvii List of Algorithms xviii List of Theorems xix 1 Introduction and Overview 1 1.1 Introduction . .1 1.2 In-Memory Data Management Challenges . .2 1.3 Persistent Memory Overview . .4 1.3.1 Persistent Memory and Big Data . .6 1.4 Hardware Transactional Memory Overview . .7 1.5 HTM + PM Overview . 10 1.6 System Architecture Overview . 12 1.7 Contributions . 15 1.8 Thesis Organization . 18 2 Persistent Memory Transactions 20 2.1 Persistent Memory Access . 20 2.1.1 Persistent Memory Access Transaction Properties . 22 2.2 Durability for Persistent Memory . 24 2.2.1 Persistent Memory Models . 26 vii 2.3 Atomicity for Persistent Memory Transactions . 27 2.3.1 Undo Logging . 29 2.3.2 Block Copy-on-Write . 31 2.3.3 Redo Logging . 31 2.3.4 Versioning . 33 2.3.5 Summary . 34 2.4 Isolation for Concurrent Persistent Memory Transactions . 35 2.4.1 Two-Phase Locking . 36 2.4.2 Software Transactional Memory . 37 2.4.3 Hardware Transactional Memory . 39 2.4.4 Checkpoints . 41 2.4.5 Summary . 42 2.5 Related Persistent Memory Transaction Work . 43 2.5.1 Hardware Supported Transaction Work . 43 2.5.2 File System Work . 48 2.5.3 Software and Concurrent Transaction Work . 49 2.6 Summary . 53 3 Hardware Transactional Memory + Persistent Memory 54 3.1 HTM + PM Basics . 54 3.1.1 Fallback Paths . 56 3.2 Challenges Persisting HTM Transactions . 57 3.2.1 Logging Challenges . 60 3.2.2 Ordering Challenges . 64 3.3 Related Work . 66 3.4 Hardware Transactional Persistent Memory Model . 69 3.4.1 Ordering HTM Transactions . 74 3.4.2 System Checkpointing Model . 75 viii 3.4.3 Transaction Durability Model . 78 3.4.4 Transactional Durability Extensions . 79 3.4.5 Logging and Recovery . 80 3.4.6 Composability . 81 3.4.7 Example . 82 3.4.8 HTM Performance Tuning for Durability . 84 3.5 Summary . 92 4 Continuous Checkpointing HTM for PM 93 4.1 Background on Variable Aliasing for PM Transactions . 93 4.2 Continuous Checkpointing HTM for PM Approach . 95 4.2.1 Alias Table Model . 99 4.3 Algorithm and Implementation . 102 4.3.1 Blue and Red Queues . 103 4.3.2 Logging and Recovery . 104 4.3.3 Alias Table Implementation . 105 4.3.4 Lock Free Algorithm . 107 4.3.5 Data Structure Properties . 113 4.4 Evaluation . 114 4.4.1 Benchmarks . 116 4.4.2 Hash Table . 121 4.4.3 Red-Black Tree . 125 4.5 Summary . 128 5 Hardware Transactional Persistent Memory Controller 130 5.1 Background on WrAP Controller . 130 5.2 Persistent Memory Controller Approach . 132 5.2.1 Transaction Lifecycle . 133 5.2.2 Persistent Memory Controller . 136 ix 5.2.3 Recovery . 139 5.2.4 Protocol Properties . 140 5.3 Algorithm and Implementation . 141 5.3.1 Software Library . 142 5.3.2 PM Controller Implementation . 145 5.3.3 Example . 149 5.4 Evaluation . 150 5.4.1 Benchmarks . 152 5.4.2 Hash Table . 156 5.4.3 Red-Black Tree . 159 5.4.4 Persistent Memory Controller Analysis . 160 5.5 Summary . 165 6 Future Work and Conclusions 166 6.1 Future Work . 166 6.1.1 Future Software Work . 166 6.1.2 Future Persistent Memory Controller Work . 169 6.2 Conclusions . 170 A Persisting In-Memory Databases Using PM 173 A.1 Overview . 173 A.2 Redis Enhancements . 175 A.2.1 Local Alias Table Batched . 177 A.3 Evaluation . 179 A.4 Summary . 181 B Persistent Memory Containers 182 B.1 Virtualization Overview . 182 B.2 Containerized Persistent Memory Approach . 185 x B.3 Evaluation . 188 B.4 Summary . 190 C Evaluation on Intel Optane DC PM DIMMs 192 C.1 Persistent Memory Configuration . 192 C.2 Evaluation . 194 C.2.1 Benchmarks . 196 C.2.2 Hash Table . 200 C.2.3 Red-Black Tree . 205 C.3 Summary . 207 Bibliography 208 Illustrations 1.1 Persistent Memory provides a large quantity of fast, byte-addressable persistent memory alongside DRAM on the main memory bus. .5 1.2 Logical layout of an Intel E5 Series High-Core Count processor that supports HTM transactions with 18 cores and two memory controllers with both DRAM and PM . 13 1.3 Logical layout of two Intel Xeon processors on a motherboard with a total of four memory controllers accessing both DRAM and PM with HTM support. Processors and cores have shared and cache-coherent access to all memory locations. 14 2.1 Persistent Memory can be accessed directly using normal loads and stores, bypassing the file system interface and page caches. 21 2.2 Writes to persistent memory from a core require moving through multiple buffers, complicating persistence. 24 2.3 Writes to PM may be evicted out of the cache hierarchy in any order. 29 2.4 Undo Logging approach for transactional atomicity of persistent memory . 30 2.5 Redo Logging approach for transactional atomicity of persistent memory . ..