Efficient Synchronization for a Large-Scale Multi-Core Chip Architecture
Total Page:16
File Type:pdf, Size:1020Kb
EFFICIENT SYNCHRONIZATION FOR A LARGE-SCALE MULTI-CORE CHIP ARCHITECTURE by Weirong Zhu A dissertation submitted to the Faculty of the University of Delaware in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Electrical and Computer Engineering Spring 2007 c 2007 Weirong Zhu All Rights Reserved EFFICIENT SYNCHRONIZATION FOR A LARGE-SCALE MULTI-CORE CHIP ARCHITECTURE by Weirong Zhu Approved: Gonzalo R. Arce, Ph.D. Chairperson of the Department of Electrical and Computer Engineering Approved: Eric W. Kaler, Ph.D. Dean of the College of Engineering Approved: Carolyn A. Thoroughgood, Ph.D. Vice Provost for Research and Graduate Studies I certify that I have read this dissertation and that in my opinion it meets the academic and professional standard required by the University as a dissertation for the degree of Doctor of Philosophy. Signed: Guang R. Gao, Ph.D. Professor in charge of dissertation I certify that I have read this dissertation and that in my opinion it meets the academic and professional standard required by the University as a dissertation for the degree of Doctor of Philosophy. Signed: Fouad Kiamilev, Ph.D. Member of dissertation committee I certify that I have read this dissertation and that in my opinion it meets the academic and professional standard required by the University as a dissertation for the degree of Doctor of Philosophy. Signed: Stephan Bohacek, Ph.D. Member of dissertation committee I certify that I have read this dissertation and that in my opinion it meets the academic and professional standard required by the University as a dissertation for the degree of Doctor of Philosophy. Signed: Li Liao, Ph.D. Member of dissertation committee ACKNOWLEDGEMENTS First of all, I would like to thank my advisor, Prof. Guang R. Gao, for his advise- ment, support, understanding, concern, and help through out all these years for my Ph.D. program. His vision, insight, attitude, and methodology on the research problems always teach me a lot. His enthusiasm, belief, and perseverance on the research of parallel pro- cessing and multithreading always inspire and encourage me to continue my work every time when I feel frustrated, confused, or desperate. In my academia life, he is my advisor for guiding me through the graduate study and research. In my personal life, he and his wife Peggy take good care of me in many ways. I would also like to thank Dr. Kevin Theobold, who introduced me to the new world of parallel system and computer architecture and taught me lots of computing skills, which benefit my whole Ph.D. study. I thank Dr. Fouad Kiamilev, Dr. Stephan Bohacek, and Dr. Li Liao, who served in my dissertation committee. Their comments and feedback on my dissertation are invalu- able. I want to acknowledge many present members of our group at CAPSL, University of Delaware. They have helped and contributed in many ways to the work of this disser- tation. In particular, I would like to thanks Ziang Hu, Juan del Cuvillo, Yuan Zhang, and Ioannis E. Venetis for many useful and insightful discussions. I also got many help and guidance from some former CAPSL and ACAPS members. I would like to thanks Vu- granam C. Sreedhar, and Parimala Thulasiraman for their selfless help, encouragement, and suggestions. iv I would also like to thank Yanwei Niu, Chuan Shen, and Jizhu Lu for the col- laboration during the course of the EARTH project, which lays the foundation for my understanding on fine-grain multithreading techniques. Cyclops-64 is a big and exciting project with the collaboration of almost the whole CAPSL group. I am very lucky to be one of them. I would like to thank all my teammates, including Ziang Hu, Juan del Cuvillo, Fei Chen, Guangming Tang, Ioannis E. Venetis, Yanwei Niu, Long Chen, and many others. The team provides a stimulus environment for a lot of interesting scientific discussions and collaborations. My internship in ETI is a valuable asset for gaining experience and understanding on Cyclops-64 architecture and its system software. I would like to thank my colleagues, Geoff Gerfin, Dobry Brice, John Tully, and Wesley Toland, for all the work we spent together on building up the Cyclops-64 system software toolchain, without which my dissertation can not be completed. I would like to acknowledge the support from IBM, in particular, Monty Denneau, who is the architect of the Cyclops-64 architecture. This work would not have been possible without the generous support of the re- search grants by IBM, ETI, DoD, NSF (CNS-0509332), DoE (DE-FC02-01ER25503), and other government sponsors. Xinyi Zhang, gave me all of her love, understanding, and patience for all these years. She colored my life. Finally, my parents and my sister, deserve the most credits for this accomplish- ment. Their love, support, and patience are the source of everything I could achieve. They gave me far more than what I could ever give back. v Dedicated to my parents, Tianxiu Li and Peifu Zhu. vi TABLE OF CONTENTS LIST OF FIGURES ................................ xii LIST OF TABLES ................................. xvii ABSTRACT ..................................... xviii Chapter 1 INTRODUCTION ............................... 1 1.1 Towards Multi-Core Chip Architectures and Beyond ........... 1 1.2 Problem Description ............................ 3 1.3 Contributions ............................... 6 1.4 Synopsis .................................. 8 2 BACKGROUND AND PREVIOUS WORK .................. 9 2.1 Synopsis .................................. 9 2.2 The Limits of Single Processor Chip Architecture ............. 9 2.3 Multi-Core Chip Architectures ....................... 12 2.3.1 Two Types of Multi-Core Architecture Designs .......... 13 2.3.2 Large-Scale Multi-Core Chip Architecture ............ 15 2.3.3 Cellular Architecture ....................... 15 2.3.4 IBM Cyclops-64 Cellular Architecture .............. 16 2.4 Synchronization .............................. 23 2.5 Atomic Instructions ............................ 23 2.6 Spin Lock ................................. 27 2.6.1 Software Based Locking Algorithms ............... 27 2.6.1.1 Test and Set Lock .................... 27 vii 2.6.1.2 Ticket Lock ....................... 28 2.6.1.3 Array Based Locks ................... 29 2.6.1.4 Linked-List Based Lock ................. 29 2.6.1.5 Reactive Lock ...................... 29 2.6.2 Hardware Based Locking Mechanisms .............. 30 2.6.2.1 QOLB .......................... 30 2.6.2.2 SMT Lock-Box ..................... 31 2.6.2.3 SoCLC: System-on-a-Chip Lock Cache ......... 32 2.7 Non-Blocking Synchronization ...................... 33 2.7.1 Lock-Free Concurrent Data Structures ............... 34 2.7.2 Transactional Memory ....................... 36 2.7.2.1 Hardware Transactional Memory ............ 37 2.7.2.2 Software Transactional Memory ............ 40 2.8 Data-Level Fine-Grain Synchronization .................. 44 2.8.1 Full/Empty Bits .......................... 45 2.8.2 Register-Register Communication ................. 46 2.8.3 I-Structure and M-Structure .................... 46 2.9 Synchronization Optimization with Compiler ............... 46 2.9.1 Synchronization Optimization for DOACROSS Loops ...... 46 2.9.2 Compiler-Automated Lock Assignment .............. 47 2.10 Synchronization on Cyclops-64 ...................... 47 3 PROBLEM FORMATION AND EXPERIMENTAL INFRASTRUCTURE 49 3.1 Synopsis .................................. 49 3.2 Problem Formation ............................ 49 3.2.1 Performance Characteristics of Synchronization Mechanisms on Large-Scale Multi-Core Architectures ............... 50 3.2.2 Performance Characteristics of OpenMP Language Construct on Large-Scale Multi-Core Architectures ............... 51 viii 3.2.3 Fine-Grain Synchronization on Large-Scale Multi-Core Architectures ........................... 52 3.3 Experimental Infrastructure ........................ 54 3.3.1 Cyclops-64 System Software Toolchain .............. 54 3.3.1.1 TiNy Threads ...................... 57 3.3.1.1.1 TNT Thread model and API ......... 58 3.3.1.1.2 TNT Memory model ............. 59 3.3.1.1.3 TNT Synchronization model ......... 61 3.3.2 FAST Simulator .......................... 62 3.3.3 Omni OpenMP Compiler ..................... 65 4 EVALUATION OF SYNCHRONIZATION MECHANISMS ON CYCLOPS-64 .................................. 69 4.1 Evaluation of Spin Lock Algorithms on C64 ................ 70 4.1.1 Spin Lock Algorithms ....................... 70 4.1.2 Microbenchmarks ......................... 71 4.1.3 Evaluation ............................. 72 4.1.4 Customizing MCS for C64 .................... 76 4.2 Lock-based and Lock-Free Concurrent Data Structures on C64 ...... 80 4.3 Evaluation of Language Constructs in OpenMP on C64 .......... 89 4.3.1 EPCC OpenMP Microbenchmark ................. 90 4.3.2 Synchronization Constructs .................... 90 4.3.3 Scheduling Policies ........................ 93 4.3.4 Array Clause for PARALLEL ................... 97 4.3.5 Related Work and Summary .................... 102 4.4 Summary .................................. 103 5 SSB: SYNCHRONIZATION STATE BUFFER ................ 106 5.1 Motivation ................................. 106 5.2 Challenges ................................. 108 ix 5.3 SSB: A Novel Fine-Grain Synchronization Solution ............ 109 5.4 Design Principles of Synchronization State Buffer ............. 113 5.4.1 Large-Scale Multi-Core Architecture ............... 113 5.4.2 Formalization of the Key Observation ............... 115 5.5 Design of Synchronization