VERIFICATION OF HIERARCHICAL CACHE COHERENCE PROTOCOLS FOR FUTURISTIC PROCESSORS by Xiaofang Chen A dissertation submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Computer Science School of Computing The University of Utah December 2008 Copyright °c Xiaofang Chen 2008 All Rights Reserved THE UNIVERSITY OF UTAH GRADUATE SCHOOL SUPERVISORY COMMITTEE APPROVAL of a dissertation submitted by Xiaofang Chen This dissertation has been read by each member of the following supervisory committee and by majority vote has been found to be satisfactory. Chair: Ganesh L. Gopalakrishnan Steven M. German Ching-Tsun Chou John B. Carter Rajeev Balasubramonian THE UNIVERSITY OF UTAH GRADUATE SCHOOL FINAL READING APPROVAL To the Graduate Council of the University of Utah: I have read the dissertation of Xiaofang Chen in its final form and have found that (1) its format, citations, and bibliographic style are consistent and acceptable; (2) its illustrative materials including figures, tables, and charts are in place; and (3) the final manuscript is satisfactory to the Supervisory Committee and is ready for submission to The Graduate School. Date Ganesh L. Gopalakrishnan Chair: Supervisory Committee Approved for the Major Department Martin Berzins Chair/Director Approved for the Graduate Council David S. Chapman Dean of The Graduate School ABSTRACT Multicore architectures are considered inevitable, given that sequential processing hardware has hit various limits. Unfortunately, the memory system of multicore pro- cessors is a huge bottleneck, as distant memory accesses cost thousands of cycles. To combat this problem, one must design aggressively optimized cache coherence protocols. This introduces two problems for futuristic cache coherence protocols which will be hierarchically organized for scalable designs: design correctness and hardware imple- mentation correctness. Experiences show that monolithic verification will not scale to hierarchical designs and implementations. Hence there exist two unsolved problems for futuristic cache coherence protocols: (i) handle the complexity of several coherence protocols running concurrently, i.e., hierarchical protocols, and (ii) verify that the RTL implementations correctly implement the specifications. Our thesis is that formal methods based on model checking and assume guarantee verification methods can be developed to substantially ameliorate these problems faced by designers. More specifically, to solve the first problem, we develop assume guarantee reasoning to decompose a hierarchical coherence protocol into a set of abstract protocols. The approach is conservative, in the sense that by verifying these abstract protocols, the original hierarchical protocol is guaranteed to be correct with respect to its properties. For the second problem, we develop a formal theory to check the refinement relation- ship, i.e., under which conditions can we claim that an RTL implementation correctly implements a high level specification. We also develop a compositional approach using abstraction and assume guarantee reasoning to reduce the verification complexity of refinement check. Finally, we evaluate the solutions in collaboration with industry and partly mechanize the refinement check. We propose research that will lead to a characterization of the proposed mechanisms through several verification tools and protocol benchmarks. For high level modeling and verification, we show that for three hierarchical protocols with different features which we developed for multiple chip-multiprocessors, more than a 20-fold improvement in terms of the number of states visited can be achieved. For refinement check, we show that for a driving coherence protocol example with realistic hardware features, refinement check can find subtle bugs which are easy to miss by checking coherence properties alone. Furthermore, we show that for the protocol example, our compositional approach can finish the refinement check within 30 minutes while a state-of-art verification tool in industry cannot finish in over a day. Finally, we extend a hardware language and a tool, to mechanize the refinement check process. In summary the research accomplished in this work substantiates our thesis statement with a body of results generated using tools that were constructed during this research. v CONTENTS ABSTRACT ................................................... iv LIST OF FIGURES ............................................. ix LIST OF TABLES .............................................. x CHAPTERS 1. INTRODUCTION .......................................... 1 1.1 Verification of Cache Coherence Protocols . ......... 3 1.2 ProblemStatement ................................ ........ 4 1.3 ThesisStatement................................. ......... 7 1.4 Contributions.................................... ......... 8 1.5 Outline ......................................... ........ 9 2. BACKGROUND AND RELATED WORK ........................ 11 2.1 CacheCoherenceStates.............................. ....... 11 2.1.1 TheMSIProtocol................................ ..... 11 2.1.2 TheMESIProtocol............................... ..... 12 2.2 Common Cache Coherence Protocols . 13 2.2.1 SnoopingProtocols............................. ....... 14 2.2.2 Directory Based Coherence Protocols . ...... 15 2.2.3 Token Coherence Protocols . ...... 16 2.3 CachingHierarchies ............................... ........ 17 2.4 Common Coherence Properties to Be Verified . ..... 19 2.5 RelatedWork ...................................... ...... 20 2.5.1 Verifying Hierarchical Protocol Specifications . ............ 20 2.5.2 CheckingRefinement ............................... 23 2.5.2.1 The Approach by Burch and Dill . 23 2.5.2.2 The Approach of Aggregation of Distributed Actions ...... 23 2.5.2.3 TheBluespecApproach .......................... 24 2.5.2.4 The Approach of Architecture Description Languages ..... 24 2.5.2.5 The Approach of OneSpin Solutions . 25 2.5.2.6 TheApproachofSystemC ........................ 25 2.5.2.7 Compositional Verification . ..... 26 3. VERIFICATION OF HIERARCHICAL COHERENCE PROTOCOLS IN THE SPECIFICATIONS ................................... 27 3.1 Benchmarking Hierarchical Coherence Protocols . .......... 27 3.1.1 TheFirstBenchmarkProtocol...................... ...... 27 3.1.2 TheSecondBenchmarkProtocol ..................... 31 3.1.3 The Third Benchmark Snooping . 34 3.2 A Compositional Framework . ....... 35 3.2.1 Abstraction................................... ....... 36 3.2.2 AssumeGuaranteeReasoning....................... ..... 43 3.2.3 ExperimentalResults ............................ ...... 47 3.2.4 SoundnessProof................................ ...... 48 3.2.4.1 SomePreliminaries ........................... 48 3.2.4.2 A Theory Justifying Circular Reasoning . 48 3.3 VerificationOneLevelataTime ...................... ........ 51 3.3.1 Abstraction One Level at a Time . ...... 52 3.3.2 Using “Per-Level” Abstraction . ........ 56 3.3.3 Using History Variables . ....... 58 3.3.4 ExperimentalResults ............................ ...... 61 3.3.5 SoundnessProof................................ ...... 62 3.4 Error-traceJustification ......................... ............ 63 3.4.1 Spurious/Genuine Error Identification . .......... 64 3.4.2 Guided Search for Stuttering Equivalent Execution . .......... 65 3.4.3 Interface Aware Guided Search . ....... 67 3.4.4 Interface Aware Bounded Search . ...... 69 3.4.5 Implementation and Experimental Results . ........ 71 3.5 Summary......................................... ....... 77 4. REFINEMENT CHECK BETWEEN PROTOCOL SPECIFICATIONS AND IMPLEMENTATIONS .................................. 78 4.1 SomePreliminaries ............................... ......... 80 4.1.1 States, Transitions, Models . ......... 80 4.1.2 Executions .................................... ...... 80 4.1.3 Annotations ................................... ...... 81 4.1.4 Transactions .................................. ....... 82 4.1.5 ImplementationModel ........................... ...... 84 4.2 CheckingRefinement ................................. ..... 84 4.3 MonolithicModelChecking .......................... ....... 86 4.3.1 Checking Correct Implementation . ...... 88 4.4 Compositional Model Checking . ....... 91 4.4.1 Abstraction................................... ....... 93 4.4.1.1 ASimpleExample.............................. 99 4.4.2 AssumeGuaranteeReasoning....................... 100 4.4.2.1 ASimpleExample.............................. 106 4.5 Implementing the Compositional Approach . ......... 107 4.6 ATransactionBasedRefinementChecker .................. 108 4.6.1 HardwareMurphi ................................ 109 vii 4.6.2 AnOverviewofMuv ............................... 112 4.6.3 Assertion Generation for Refinement Check . 112 4.6.3.1 Checking Write-Write Conflicts . 113 4.6.3.2 Computing Read and Write Sets . 115 4.6.3.3 Checking Serializability . 116 4.6.3.4 Checking Variable Equivalence . 118 4.6.4 ExperimentalBenchmarks ......................... 118 4.6.5 ExperimentalResults ............................ 120 4.6.5.1 TransactionDetails .......................... 121 4.7 Summary......................................... ....... 121 5. CONCLUSIONS AND FUTURE DIRECTIONS ................... 123 5.1 Predicate Abstraction for Murphi . ........... 125 5.1.1 LearningfromPAM ............................... 127 5.2 BoundedTransactionBasedTesting.................... ........ 128 5.2.1 Possible Future Directions for BT