Low-level Concurrent Programming Using the Relaxed Memory Calculus Michael J. Sullivan CMU-CS-17-126 November 2017 School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 Thesis Committee: Karl Crary, Chair Kayvon Fatahalian Todd Mowry Paul McKenney, IBM Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy. Copyright c 2017 Michael J. Sullivan This research was sponsored in part by the Carnegie Mellon University Presidential Fellowship. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of any sponsoring institution, the U.S. government or any other entity. Keywords: programming languages, compilers, concurrency, memory models, relaxed memory Abstract The Relaxed Memory Calculus (RMC) is a novel approach for portable low- level concurrent programming in the presence of the the relaxed memory behavior caused by modern hardware architectures and optimizing compilers. RMC takes a declarative approach to programming with relaxed memory: programmers explicitly specify constraints on execution order and on the visibility of writes. This differs from other low-level programming language memory models, which—when they exist—are usually based on ordering annotations attached to synchronization opera- tions and/or explicit memory barriers. In this thesis, we argue that this declarative approach based on explicit programmer-specified constraints is a practical approach for implementing low-level concurrent algorithms. To this end, we present RMC-C++, an extension of C++ with RMC constraints, and rmc-compiler, an LLVM based compiler for it. We live on a placid island of ignorance in the midst of black seas of infinity, and it was not meant that we should voyage far. — “The Call of Cthulhu”, H.P. Lovecraft And all I ask is a tall ship and a star to steer her by; — “Sea Fever”, John Masefield iv Acknowledgments It’s difficult to overstate how much thanks is owed to my parents, Jack and Mary Sullivan. They’ve provided nearly three decades of consistent encouragement, support (emotional, moral, logistical, financial, and more), and advice (“Put ‘write thesis proposal’ on your To-Do List, then write the proposal and cross it off!”). I know I wasn’t the easiest kid to raise, but they managed to mostly salvage the situation. Thank you—I wouldn’t be here without you. Thanks to my brother, Joey, who my interactions with over the years have shaped me in more ways that I understand. And I’m very grateful that—at this point in our lives—most of those interactions are now positive. Immeasurable thanks to my advisor, Karl Crary, without whom this never would have hap- pened. Karl is an incredible researcher and it was a pleasure to spend the last six years collabo- rating with him. Karl also—when I asked him about the CMU 5th Year Masters—was the one to convince me to apply to Ph.D. programs instead. Whether that is quite worth a “thanks” is a hard question to answer with anything approaching certainty, but I believe it is. I also owe a great debt to Dave Eckhardt. Operating Systems Design and Implementation (15-410) is the best class that I have ever taken and I haven’t been able to get it out of my head. The work I did over seven semesters as one of Dave’s TAs for 410—teaching students about concurrency, operating systems, and, most of all, giving thoughtful and detailed feedback about software architecture and design—is the work I am most proud of, and I am supremely thankful for being able to be a part of it. While much of my work is likely to take me far afield, I try to carry a little of the kernel hacker ethic in my heart wherever I go. On top of that, Dave has been a consistent source of advice, support, and guidance without which I’d never have made it through grad school. I owe much to a long string of excellent teachers I had in the Mequon-Thiensville School Dis- trict, beginning with my third-grade teacher Mrs. Movell, who—in response to me asking if there was a less tedious way to draw some particular shape in LOGO—arranged for a math teacher to come over from the high school to teach me about the wonders of variables, if statements, and GOTOs. I owe a particularly large debt to Robin Schlei, who provided years of support, and to Kathleen Connelly, who first put me in touch with Mark Stehlik. Thank you! On that note, thanks to Mark Stehlik, who was as good an undergraduate advisor as could be asked for, even if he wouldn’t let me count HOT Compilation as an “applications” class. And thanks to all the great teachers I had, especially Bob Harper, who kindled an interest in programming languages. Grad school was a long and sometimes painful road, and I’d have never made it through without my hobbies. Chief among them, my regular tabletop role-playing game group, who sac- rificed countless Saturday nights to join me in liberating Polonia, exploring the Ironian Wastes for the Ambrosia Trading Company, and conclusively demonstrating, via their work for the Kro- mian Mages Guild, that not all wizards are subtle: Rhett Lauffenburger (Roger Kazynski), Gabe Routh (Hrothgar Boarbuggerer ne´ Hrunting), Ben Blum (Yolanda Swaggins-Moonflute), Matt Maurer (Jack Silver), Michael Arntzenius (Jakub the Half-Giant), and Will Hagen (Daema). A GM couldn’t ask for a better group of players. I’d also like to thank my other major escape: the EVE Online alliance Of Sound Mind, and especially the CEOs who have built SOUND into such a great group to fly with—XavierVE, June Ting, Ronnie Cordova, and Jacob Matthew Jansen. v Fly safe! The SCS Musical provided another consistently valuable form of stress in my life, and I’d like to thank everybody involved in helping to revive it and keep it running. I had allowed myself to forget about the giant technical-theatre-shaped hole in my heart, and I don’t think I’ll be able to again. Thanks to the old gang from Homestead—Becki Johnson, Gina Buzzanca, and Michael and Andrew Bartlein. Though we’ve been scattered far and wide from our parents’ basements in Wisconsin—and though the Bartleins are rubbish at keeping in touch—you’ve remained some of my truest friends. A lot of people provided—in their own ways—important emotional support at different points in the process. I owe particular thanks to Jack Ferris, Joshua Keller, Margaret Meyerhofer, Amanda Watson, and Sarah Schultz. Salil Joshi and Joe Tassarotti contributed in important ways to the design of RMC and its compiler. Ben Segall, Paul Dagnelie, and Rick Benua provided helpful suggestions about bench- marking woes. Paul and Rick also provided useful comments on the draft. Matt Maurer provided some much needed statistical help. I’d like to thank Shaked Flur et al. for providing me with the ARMv8 version of the ppcmem tool, which proved invaluable for exploring the possible behav- iors of ARMv8s. A heartfelt thanks to everyone on my committee. Paul McKenney provided early and in- teresting feedback on this document (and a quote from “Sea Fever”), Kayvon Fatahalian gave excellent advice on what he saw as the key arguments of my work, and Todd Mowry helped me understand the computer architect’s view of the world a little better. I’ve had a lot of fantastic colleagues during the decade I spent at CMU, both as an undergrad and as a grad student. I learned more about computer science and about programming languages in conversation with my colleagues than I have from any books or papers. Thanks to Joshua Wise, Jacob Potter, Ben Blum, Ben Segall, Glenn Willen, Nathaniel Wesley Filardo, Matthew Wright, Elly Fong-Jones, Philip Gianfortoni, Car Bauer, Laura Abbott, Chris Lu, Eric Faust, Andrew Drake, Amanda Watson, Robert Marsh, Jack Ferris, Joshua Keller, Joshua Watzman, Cassandra Sparks, Ryan Pearl, Kelly Hope Harrington, Margaret Meyerhofer, Rick Benua, Paul Dagnelie, Michael Arntzenius, Carlo Anguili, Matthew Maurer, Joe Tassarotti, Rob Simmons, Chris Martens, Nico Feltman, Favonia, Danny Gratzer, Stefan Mueller, Anna Gommerstadt, and many many others. Thanks also to all the wonderful colleagues I had working at Mozilla and Facebook on the Rust and Hack programming languages. The most stressful and miserable part of my grad school process was the semester I spent preparing my thesis proposal. I was nearly burnt out after it and I don’t think I could have kept on going without the ten weeks I spent “Working From Tahoe” (WFT) from The Fairway House in Lake Tahoe. Thanks to Vail Resorts and everybody who split the ski cabin with me for making it possible—especially to Jack Ferris and Kelly Hope Harrington for bullying me into it. And thanks to the Northstar Ski Patrol, the paramedics of the Truckee Fire Protection District, and the staff at the Tahoe Forest Hospital and the University of Pittsburgh Medical Center for all the excellent care I received after I skied into a tree. Thanks to Aaron Rodgers and Mike McCarthy, but not to Dom Capers. Go Pack Go! NVIDIA Corporation donated a Jetson TK1 development kit which was used for testing vi and benchmarking. The IBM Power Systems Academic Initiative team provided access to a POWER8 machine which was used for testing and benchmarking. To all of the undoubtedly countless people who I really really ought to have thanked here but who slipped my mind: I’m sorry, and thank you! And last, but certainly not least, I’d like to extend a heartfelt thanks to Phil, Madz, Manny, and all the rest of the gang down at Emarhavil Heavy Industries.