Reining in the Functional Verification of Complex Processor Designs with Automation, Prioritization, and Approximation by Biruk Wendimagegn Mammo A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Computer Science and Engineering) in The University of Michigan 2017 Doctoral Committee: Professor Valeria M. Bertacco, Chair Professor Todd M. Austin Associate Professor Heath Hofmann Professor Scott Mahlke c Biruk Wendimagegn Mammo All Rights Reserved 2017 To my family ii Acknowledgments I would first like to thank my advisor, Professor Valeria Bertacco, for her constant sup- port and mentorship, which started even before I joined the University of Michigan. This thesis would not have been possible without her guidance and hard work. I would also like to thank my committee members, Professors Todd Austin, Heath Hofmann, and Scott Mahlke, for their support and valuable feedback. I have benefited greatly from the excellent faculty at the University of Michigan. I am very fortunate to have worked closely with Doctor Mark Brehob, Professors Todd Austin, Scott Mahlke, and Tom Wenisch, who are all excellent teachers, mentors, and collabora- tors. I would like to give special thanks to Professor Austin for being a source of inspiration and good humor. Thank you for always making time to discuss ideas with me and to help me improve my papers and presentations. I am fortunate to have been part of a dynamic group of graduate students. I would like to thank Andrew DeOrio, Joseph Greathouse, Jason Clemons, Andrea Pellegrini, and Debapriya Chatterjee for showing me the ropes and guiding me through life in graduate school. I am grateful for the innumerable discussions I have had with Ritesh Parikh and Rawan Abdel-Khalek that helped me make progress in my research. This work would not have been possible without the contributions of my collaborators: Doowon Lee, Harrison Davis, Yijun Hou, Milind Furia, Daya Khudia, Andrew DeOrio, and Debapriya Chatterjee. I would also like to thank the many friends and colleagues that provided an intellectually stimulating and enjoyable environment. This includes Aasheesh Kolli, Ankit Sethia, Gau- rav Chada, Misiker Aga, Abraham Addisie, Zelalem Aweke, Shamik Ganguly, Ricardo Rodriguez, and many others I do not have space to mention here. I am especially grateful to my dear friends and colleagues Salessawi Ferede, William Arthur, and Animesh Jain, for supporting me through all the trials and tribulations of graduate school. I would also like to express my gratitude to the people who worked behind the scenes to make this work possible. Thank you Laura Fink Coutcher, Kyle Banas, Don Winsor, and other DCO staff for keeping the computers running. I will never forget how Don once went beyond the call of duty to come to work on a Sunday evening to fix a problem so iii that I can meet a deadline. Thank you Dawn Freysinger, Denise Duprie, Karen Liska, Anne Rhoades, Lauri Johnson-Rafalski, Stephen Reger, Ashley Andreae, Tracie Straub, and Christine Boltz for your support and for being there when I needed help with admin- istrative issues. I would also like to thank the National Science Foundation (NSF) and the Center for Future Computer Architecture Research (CFAR) for funding my research. Finally, I would like to thank my family. I am indebted to my wife, Yodit Beyene, for her friendship and love during the challenging last years of graduate school. I would like to express my deepest gratitude to my parents and my siblings, for their support and encouragement. Even though they live half a world away, they have been with me in spirit through every moment of this journey. iv Preface Modern processor designs are among humankind’s most complex creations. A processor design today is so complex that it is impossible to guarantee its complete correctness. Pro- cessor companies spend a significant amount of time and money to obtain what can only be described as an acceptable risk of inaccuracy. Add a highly competitive market with ever-shrinking time-to-market demands, and it is common for processors to be released with design defects that may have devastating consequences to consumers and compa- nies. These challenges put undue pressure on the functional verification effort to ensure the correctness of a design, which today surpasses the effort to actually implement the design. Today’s designs cram several interacting components into small spaces. The functional verification burden is only expected to worsen as more and more custom designs with higher levels of integration emerge. We are witnessing the proliferation of custom comput- ing systems, ranging from custom systems-on-chip (SoCs) for servers to custom nodes in an internet-of-things (IoT) environment. Emerging applications, such as machine learning, big data, and virtual reality, have massive computing demands that are going to drive even more complexity into processor designs. For these and other unforeseen design efforts to be successful, it is imperative that the cost of design and, ultimately, the time and resources spent on functional verification go down. Simulation-based methods carry a bulk of the functional verification effort. In these methods, tests are executed on an implementation of the design while the design’s behav- ior is monitored and checked for correctness. The simulation-based functional verification effort today is mostly a best-effort endeavor limited by design complexity and the amount of time available to execute tests. A typical simulation-based functional verification effort today is focused on executing tests to meet a set of verification goals defined at the begin- ning of the design project. Even though the goals are an incomplete set of design behaviors to target for verification, the verification process is practiced with an expectation of com- pleteness: engineers strive to accurately and completely meet an inherently incomplete set of goals. In this dissertation, we recognize an opportunity to reduce the cost of verification by v removing the expectation of completeness, not just at the beginning of a design project, but across all functional verification activities. To this end, we introduce a new perspective that we refer to as the automation, prioritization, and approximation (APA) approach. In this approach, we perceive functional verification as multiple endeavors that may sometimes be at odds with each other. These endeavors pertain to handling design complexity, devel- oping mechanisms to detect functional errors, undertaking tasks to execute a verification plan, and managing engineer effort. Our APA approach gives us the means to strategically leverage incompleteness for these endeavors. We first prioritize certain aspects of an en- deavor, and then automate activities pertaining to the prioritized aspects while exploiting approximations that trade-off accuracy for large gains in efficiency. In this dissertation, we partition the functional verification effort into three main activ- ities – planning & test generation, test execution & bug detection, and bug diagnosis – and present our solutions in the context of these activities. We present an automatic mechanism for prioritizing design behaviors while planning the verification of modern SoCs. This au- tomatic prioritization is achieved by analyzing the execution of software packages being developed for the SoC. Our mechanism is also able to generate regression suites that run much faster than the original software packages. We also develop automatic bug detection mechanisms that unshackle high-performance validation platforms, allowing them to ex- ecute tests at high speeds while looking for high-priority bugs in the design. Finally, we develop an automatic solution for diagnosing bugs, which can shave off weeks of engineer effort. vi Table of Contents Dedication ....................................... ii Acknowledgments ................................... iii Preface ......................................... v List of Figures ..................................... x List of Tables ...................................... xii Abstract .........................................xiii Chapter 1 Introduction ............................... 1 1.1 Designcomplexityandtheverificationchallenge . ........ 2 1.2 TheAPAapproach .............................. 5 1.3 Functionalverificationactivities . ...... 6 1.3.1 Planning&testgeneration . 6 1.3.2 Testexecution&bugdetection. 7 1.3.3 Bugdiagnosis............................. 9 1.4 Dissertationoverview. 10 1.5 Dissertationorganization . ... 11 Chapter 2 An Introduction to Simulation-based Functional Verification .... 12 2.1 Overviewofthefunctionalverificationprocess . ........ 12 2.2 Simulation-basedverificationtestbench . ....... 14 2.2.1 Stimulusgenerators. 14 2.2.2 Checkers ............................... 16 2.2.3 Coverage ............................... 17 2.3 The role of high-leveldesignmodels inverification . ......... 19 2.4 Summary ................................... 20 Chapter 3 Addressing Planning and Test Generation ............... 22 3.1 Background on accelerator-rich SoCs . .... 23 3.2 APA:benefitsandaccuracytradeoffs . .... 24 vii 3.3 Prioritizing system-level interactions with AGARSoC . ........... 26 3.3.1 Analyzingmemorytraces . 26 3.3.2 Identifyinginteractionscenarios . .... 27 3.3.3 Abstractrepresentation . 28 3.4 Coveragemodelgenerationandanalysis . ..... 30 3.5 Testgeneration ................................ 31 3.5.1 Generatedtestprogramstructure. .. 32 3.5.2 Schedulegenerationalgorithm . 32