;login SPRING 2020 VOL. 45, NO. 1 : & Ceph BlueStore Runs on Raw Storage Abutalib Aghayev, Sage Weil, Michael Kuchnik, Mark Nelson, Greg Ganger, and George Amvrosiadis & Notary: Rethinking eWallets Anish Athalye, Adam Belay, M. Frans Kaashoek, Robert Morris, and Nickolai Zeldovich & Choosing the Right SLIs Jaime Woo and Emil Stolarsky & Getting Things Done by Organizing Your Work Todd Palino & Ethics for AI Researchers Jessica Cussins Newman and Rajvardhan Oak Columns Constraint and Controls: The STAMP Approach Laura Nolan Garbage Collection in Python Peter Norton Collecting Configuration in Golang Chris “Mac” McEniry Using eBPF Dave Josephsen Examining IT Job Growth Dan Geer UPCOMING EVENTS SREcon20 Americas West 29th USENIX Security Symposium March 24–26, 2020, Santa Clara, CA, USA August 12–14, 2020, Boston, MA, USA www.usenix.org/srecon20americaswest Co-located with SOUPS 2020 www.usenix.org/sec20 HotEdge ’20: 3rd USENIX Workshop on Hot Co-located with USENIX Security ’20 Topics in Edge Computing WOOT ’20: 14th USENIX Workshop on April 30, 2020, Santa Clara, CA, USA Offensive Technologies www.usenix.org/hotedge20 August 10–11, 2020 www.usenix.org/woot20 OpML ’20: 2020 USENIX Conference on Operational Machine Learning CSET ’20: 13th USENIX Workshop on Cyber Security Experimentation and Test May 1, 2020, Santa Clara, CA, USA August 10, 2020 www.usenix.org/opml20 www.usenix.org/cset20 PEPR ’20: 2020 USENIX Conference on Privacy ScAINet ’20: 2020 USENIX Security and AI Engineering Practice and Respect Networking Summit May 11–12, 2020, Santa Clara, CA, USA August 10, 2020 www.usenix.org/scainet20 www.usenix.org/pepr20 FOCI ’20: 10th USENIX Workshop on Free and SREcon20 Asia/Pacific Open Communications on the Internet June 15–17, 2020, Sydney, Australia August 11, 2020 www.usenix.org/srecon20apac www.usenix.org/foci20 HotSec ’20: 2020 USENIX Summit on Hot Topics 2020 USENIX Annual Technical Conference in Security July 15–17, 2020, Boston, MA, USA August 11, 2020 www.usenix.org/atc20 www.usenix.org/hotsec20 Co-located with USENIX ATC ’20 SREcon20 Europe/Middle East/Africa HotCloud ’20: 12th USENIX Workshop on October 27–29, 2020, Amsterdam, Netherlands Hot Topics in Cloud Computing www.usenix.org/srecon20emea July 13, 2020 www.usenix.org/hotcloud20 OSDI ’20: 14th USENIX Symposium on HotStorage ’20: 12th USENIX Workshop on Operating Systems Design and Hot Topics in Storage and File Systems Implementation July 13, 2020 November 4–6, 2020, Banff, Alberta, Canada www.usenix.org/hotstorage20 Sponsored by USENIX in cooperation with ACM SIGOPS SOUPS 2020: Sixteenth Symposium on Usable Abstract registrations due May 5, 2020 Privacy and Security www.usenix.org/osdi20 August 9–11, 2020, Boston, MA, USA Co-located with USENIX Security ’20 LISA20 www.usenix.org/soups2020 December 7–9, 2020, Boston, MA, USA SREcon20 Americas East December 7–9, 2020, Boston, MA, USA USENIX Open Access Policy USENIX is the first computing association tooffer free and open access to all of our conference proceedings and videos. We stand by our mission to foster excellence and innovation while supporting research with a practical bias. Please help us support open access by becoming a USENIX member and asking your colleagues to do the same! www.usenix.org/membership www.usenix.org/facebook twitter.com/usenix www.usenix.org/linkedin www.usenix.org/youtube SPRING 2020 VOL. 45, NO. 1 EDITOR Rik Farrow EDITORIAL MANAGING EDITOR Michele Nelson 2 Musings Rik Farrow COPY EDITORS SYSTEMS Steve Gilmartin Amber Ankerholz 6 File Systems Unfit as Distributed Storage Back Ends: Lessons from 10 Years of Ceph Evolution PRODUCTION Abutalib Aghayev, Sage Weil, Michael Kuchnik, Mark Nelson, Arnold Gatilao Ann Heron Greg Ganger, and George Amvrosiadis Jasmine Murcia 12 Notary: A Device for Secure Transaction Approval Olivia Vernetti Anish Athalye, Adam Belay, M. Frans Kaashoek, Robert Morris, TYPESETTER and Nickolai Zeldovich Star Type USENIX ASSOCIATION AI/ML 2560 Ninth Street, Suite 215 17 Artificial Intelligence: Ethics in Practice Berkeley, California 94710, USA Jessica Cussins Newman and Rajvardhan Oak Phone: +1 510.528.8649 [email protected] 22 The Emerging Practice of Operational ML: USENIX OpML www.usenix.org Conference Nisha Talagala and Joel Young ;login: is the official magazine of the USENIX SRE AND SYSADMIN Association. ;login: (ISSN 1044-6397) is published quarterly by the USENIX 25 Are We All on the Same Page? Let’s Fix That Luis Mineiro Association, 2560 Ninth Street, Suite 215, Berkeley, CA 94710, USA. 31 Getting Things Done Todd Palino A one-year print subscription to ;login: 35 It’s an SLO World: What Theme Parks Can Teach Us about magazine is available only to USENIX User-First Reliability Jaime Woo and Emil Stolarsky Association members at the Sustainer level and higher. Periodicals postage is PROGRAMMING paid at Berkeley, CA, USA, and additional 39 Interview with Mary Ann Horton Rik Farrow mailing offices. POSTMASTER: Send address changes to COLUMNS ;login:, USENIX Association, 2560 Ninth Street, Suite 215, Berkeley, CA 94710, USA. 44 Constraints and Controls: The Sociotechnical Model of Site Reliability Engineering Laura Nolan ©2020 USENIX Association USENIX is a registered trademark of the 49 Python and Memory Peter Norton USENIX Association. Many of the designa- tions used by manufacturers and sellers 53 iVoyeur—eBPF Tools: What’s in a Name? Dave Josephsen to distinguish their products are claimed Simplifying Repetitive Command Line Flags with viper as trademarks. USENIX acknowledges all 56 trademarks herein. Where those desig na tions Chris “Mac” McEniry appear in this publication and USENIX is 59 For Good Measure: Cyberjobsecurity Dan Geer aware of a trademark claim, the designations have been printed in caps or initial caps. 62 /dev/random: Artificial Ethics Robert G. Ferrell BOOKS 64 Book Reviews Mark Lamourine and Rik Farrow USENIX NOTES 68 The Year of Engagement Liz Markel, Community Engagement Manager 71 USENIX Association Financial Statements for 2018 Musings RIK FARROWEDITORIAL Rik is the editor of ;login:. read Brian Kernighan’s latest book recently, and many things in it struck [email protected] chords with me. While the book was mostly about UNIX history, it was I what Brian wrote about the influence that UNIX had on the development of computers, programming, and even printing that grabbed my attention. For example, Brian pointed out that computers had been highly customizable machines that generally ran programs that were terribly inflexible. If you wrote files to disk using one program, you could only use that program, or a closely related one, to manipulate those files. UNIX, by comparison, uses byte streams for files, ones that can be opened by any program, even ones that can’t do anything sensible with the bytes, but that flexibility is enormous. Think of the old version of spell; that was a pipeline that converted a text file to a list of words, sorted those words, ran uniq on them, and then compared the results to a dictionary. None of those tools is unique to a spell-checking program. Today, spell is a binary, not a shell script, but you can find a version of the original script in Brian’s book. Computer Architecture Another idea that caught my attention appears in the book at the bottom of page 128: UNIX and C had a large impact on computing hardware in the 1980s and 1990s. Most successful instruction set architectures were well matched to C and UNIX. I certainly never really thought about that back when I was working with UNIX workstations in the late 1980s. I know I worked on at least a dozen different workstations in that period, mostly various RISC architectures as that was the hotness of the day. A key feature of all of those instruction set architectures (ISAs) was that instead of being word oriented, all were byte oriented. That may seem too simple, but consider the types that popular programming languages use: very few are tied to a memory-length word. Some types, like float and double, are closely related to actual hardware in the CPU, but things like arrays, strings, different flavors of integers, structures, are all byte, or multiples of bytes, oriented. I am not certain that UNIX is responsible for this, but UNIX certainly was a huge influence. I was talking about this with Jon Callas, who worked for DEC in the ’90s, and he pointed out that DEC’s Alpha CPU worked equally well running Ulrix/64, DEC’s VMS, and Windows NT (Windows these days). None of these operating systems and their underlying languages were word oriented, although I do wonder about VMS, which was still written in assembler in the 1990s. CISC vs. RISC Today, most servers run variants of Intel’s ISA, while the world of the small is mainly RISC. That Intel is byte oriented is no mystery: the Intel 8080 CPU had eight-bit registers and a 16-bit address space. Most registers were paired, so could appear as 16 bits wide, but the only register capable of integer arithmetic was the A register, and that was eight bits in width. 2 SPRING 2020 VOL. 45, NO. 1 www.usenix.org EDITORIAL Musings In the ’80s, we thought that RISC was the way of the future, Switching to SRE/Sysadmin, Luis Mineiro explains how as RISC allowed CPU designs to be simpler. What happened Zalando, Europe’s largest online fashion platform, has learned instead was that Intel kept making up for the weakness of CISC to deal with paging. In an age of distributed systems, when SLIs through hardware tricks, including converting their CISC ISA (service level indictors) show something has gone wrong, you into an internal RISC-like ISA. These tricks require more only want to page the people responsible for the sub-system transistors and more energy, meaning there is still a chance that causing the slowdown or outage.