Practical Concurrency Testing or: How I Learned to Stop Worrying and Love the Exponential Explosion Ben Blum CMU-CS-18-128 December 2018 School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee: Garth Gibson, Chair David A. Eckhardt Brandon Lucia Haryadi Gunawi, University of Chicago Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy. Copyright © 2018 Ben Blum This research was sponsored by the U.S. Army Research Office under grant number W911NF0910273 and by Intel ISTC-CC. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of any sponsoring institution, the U.S. government, or any other entity. Keywords: concurrency, testing, debugging, verification, model checking, data races, education, transactional memory For my family, my teachers, and my students. iv Abstract Concurrent programming presents a challenge to students and experts alike because of the complexity of multithreaded interactions and the diffi- culty of reproducing and reasoning about bugs. Stateless model checking is a testing approach which forces a program to interleave its threads in many different ways, checking for bugs each time. This technique is powerful, in principle capable of finding any nondeterministic bug in finite time, but suffers from exponential explosion as program size increases. Checking an exponen- tial number of thread interleavings is not a practical or predictable approach for programmers to find concurrency bugs before their project deadlines. In this thesis, I develop several new techniques to make stateless model checking more practical for human use. I have built Landslide, a stateless model checker specializing in undergraduate operating systems class projects. Landslide extends the traditional model checking algorithm with a new frame- work for automatically managing multiple state spaces according to their esti- mated completion times, which I show quickly finds bugs should they exist and also quickly verifies correctness otherwise. I evaluate Landslide’s suitability for inexpert use by presenting the results of many semesters providing it to stu- dents in 15-410, CMU’s Operating System Design and Implementation class, and more recently, students in similar classes at the University of Chicago and Penn State University. Finally, I extend Landslide with a new concurrency model for hardware transactional memory, and evaluate several real-world transactional benchmarks to show that stateless model checking can keep up with the developing concurrency demands of real-world programs. vi Acknowledgments I honor here the many who supported me during these 7 long years and made this journey possible. Thesis Professor Garth Gibson is largely responsible for shaping me into the researcher I am today. Despite having primary research interests in other fields, Garth enthusiastically took on the project, and always pushed me to explore new problem domains, to design scientifically thorough experiments, to explain difficult concepts approachably, to appreciate related work as charitably as possible, and to seek guidance from industry people with relevant experience, despite my often stubborn refusal to leave my own little comfort zone. So, despite the times I needed to send 10+ email reminders to get your attention: truly, thanks for everything, Garth. Professor David A. Eckhardt provided unending guidance on education, writing, career, and life in general, helped revise each semester’s recruiting lecture with a keen eye for the student mind, contributed several clever survey questions, and supplied constant encouragement about Landslide’s value to 15-410, not to mention the red-ink-encrusted printed draft of this document I received immediately after my defense. My debt of thanks to Dave is of such magnitude that I may never hope to pay it back, but perhaps forward instead. Jiří Šimša, another former student of Garth’s, gave invaluable support dur- ing my first few years. Jiří’s mentorship oriented me in the world of research, from teaching me how DPOR works to helping revise my paper drafts to offer- ing the kind of candid career advice you can’t get from a professor. Had I not met Jiří and learned about dBug during grad school applications way back, none of this would have happened. Many have collaborated with me directly on the research itself, whom I thank here in approximate chapter order. Michael J. Sullivan and Nathaniel Wesley Filardo were Landslide’s first other users besides myself back in its pro- totype days. Joshua Wise, Miriam Zimmerman, and Glenn Willen helped re- vise my explanation of DPOR in Chapter 3. Professor Brandon Lucia provided valuable revisions to Chapter 4’s content that ultimately led to its publication at OOPSLA 2016. Michael J. Sullivan double-checked the proofs therein. Pro- fessor Timothy Zhu from Penn State University and Professor Haryadi Gu- nawi and TA Kevin Zhao from the University of Chicago graciously allowed me to use their students as guinea pigs and helped to improve Landslide’s stability and robustness for use beyond CMU’s walls, greatly enriching Chap- ter 5’s evaluation. David Blum and Skye Toor provided invaluable advice on statistical analysis that helped me evaluate the student grade distribu- tions therein. Mario Dehesa-Azuara generously contributed the transactional memory data structures that make up about half of Chapter 6’s evaluation suite. Carlo Angiuli, Evan Cavallo, Ziv Scully, Jim McCann, Matthew Maurer, Stefan Muller, Guillaume Didier, Sol Boucher, and Michael J. Sullivan helped me rehearse and polish the snake-fight talk. Most importantly of all, I thank every student who ever used Landslide. Though the introduction section may say differently, their smiles and gratitude were the true motivation of this thesis. Academia The community of 15-410 TAs has nurtured my love of teaching since well be- fore I took the class myself. 15-410 could not be the world’s best OS class without course staff’s unwavering dedication to thorough grading, the So- cratic method, and empathy to struggling students. It was a blessing to serve, and later to do research, among you all. Thanks especially to Stanley Zhang, the 2018 TA who who worked with me to ensure 15-410 could keep grading projects with Landslide even after I leave. Thanks to Wind River for gener- ously providing Simics educational licenses. Thanks also to Adam Blank for founding and organizing the class Great Practical Ideas, which teaches oft- neglected programming skills such as version control and debugging, which I had the opportunity to teach as head instructor one semester. I thank the Parallel Data Lab, my research group at CMU, for supporting me at the PDL retreat every fall, pushing me to refine my presentation skills and to network with strangers (now familiar faces) from industry. I also thank the industry attendees for their interest and enlightening conversations. Joan Digney and Karen Lindenfelser make a lot of gears turn behind the scenes at PDL, and Deb Cavlovich and Catherine Copetas do likewise for CSD as a whole. Thanks to Nicholas D. Matsakis for the mentorship during my two summers as a Rust research intern at Mozilla, and to the rest of the Rust community as well for bringing better concurrent programs into the world. And to everyone at large who insisted how cool it was that my research helps students: thank you for helping me keep my head up during darker times. The ThursDz Council, my group of now-former grad student, Shadyside- bound (in spirit if not in person) restaurant-hoppers, surrounded me with both friendship and mentorship over the years. In particular, Chris Martens, jcreed, Jim McCann, Rob Simmons, Tom 7, and William “wjl” Lovas: you each have been role models to me far more than you know. Without SIGBOVIK’s unique brand of humorous, self-aware, and definitely legitimate research, I probably would not have been interested in grad school in the first place, or perhaps would have grown too frustrated along the way for lack of a cathartic outlet for conference paper woes. I thank all who’ve worn the mantle of Harry Bovik over the years, from the conference’s original founders, to every paper author, to all future organizers who’ll keep those “mainstream” conferences on their toes well into the future. viii Life My parents, Eve and David Blum, and my grandparents, Margie Granach and Elsie Blum, provided bottomless emotional support throughout my career as a student. I appreciate beyond words for the privileged life they provided that allowed me the opportunity to pursue higher education. Looking back on my 11 years at CMU, I am especially grateful for the early friendships that continued well past first graduation. From those who stayed in Pittsburgh to those who kept in touch, from regular tabletop, board, and/or dance gaming partners to those who offered company and a couch to sleep on while I traveled, I thank 8 Gianfortoni, Adam Blank, Alan Vangpat, Alex Yuschik, Anand Subramanian, Andrew Krieger, car bauer, Carolyn Sawyer, Elizabeth “Kempy” Kemp, Elly Fong-Jones, E. Forney, Emily J Leathers, Emma Cating-Subramanian, Eric Faust, Gabriel Mengde Routh, Greg Hanneman, hazel virdó, Jack Ferris, Jake Lengyel, Jason “Wyrm” Deng, Joshua Keller, Joshua Wise, Josiah Boning, Julia Tuttle, Kartik Null Cating-Subramanian, Kellie “K2” Medlin, Laura Abbott, Maija E. Mednieks, Margaret Meyerhofer, Michael Arntzenius, Michael “Sully” Sullivan, Miriam Zimmerman, Naomi Saphra, Nathaniel Wesley Filardo, Rhett Lauffenburger, Richard Ha, rglenn, Ryan Pearl, Todd Eisenberger, and Zachary McCord for their wonderful friend- ship. I am equally grateful to the new friends I made during my graduate years, whether through board/card/dance gaming, tea drinking, movie watch- ing, and/or as more permanent Pittsburgh residents that I am extremely sad to move away from: Alexandra Lee Falk, Alexis Dyer, Barbara Jensen, Brendan McShane, Brian E.