;login WINTER 2020 VOL. 45, NO. 4 : & Characterizations of Cloud Functions Workloads Mohammad Shahrad, Rodrigo Fonseca, Íñigo Goiri, Gohar Chaudhry, and Ricardo Bianchini & RLBox: Simplifying In-Process Sandboxing Tal Garfinkel, Shravan Narayan, Craig Disselkoen, Hovav Shacham, and Deian Stefan & BIBIFI Contests: Motivated Developers Still Make Security Mistakes Daniel Votipka, Kelsey R. Fulton, James Parker, Matthew Hou, Michelle L. Mazurek, and Michael Hicks & SRE Best Practices for Capacity Final Print Issue Specials Management Favorite Articles Luis Quesada Torres and Doug Colish Rik Farrow, Laura Nolan, and Arvind Krishnamurthy Interview with USENIX Member #7 Rik Farrow Interview with Kirk McKusick Columns Rik Farrow Review of Alex Hidalgo’s Book about SLOs Open Access Laura Nolan Laura Nolan Understanding Linux Containers Corey Lueninghoener BPF and Histograms Dave Josephsen Cryptographic Hash Functions Simson L. Garfinkel Software Supply Chain Security Dan Geer, Bentz Tozer, and John Speed Meyers Thanks to our USENIX Supporters! USENIX appreciates the financial assistance our Supporters provide to subsidize our day-to-day operations and to continue our non-profit mission. Our supporters help ensure: • Free and open access to technical information • Student Grants and Diversity Grants to participate in USENIX conferences • The nexus between academic research and industry practice • Diversity and representation in the technical workplace We need you now more than ever! Contact us at [email protected]. USENIX PATRONS USENIX BENEFACTORS USENIX PARTNERS We offer our heartfelt appreciation to the following sponsors and champions of conference diversity, open access, and our SREcon communities via their sponsorship of multiple conferences: Ethyca Equinix Metal Microsoft Azure Goldman Sachs LinkedIn Salesforce More information at www.usenix.org/supporters WINTER 2020 VOL. 45, NO. 4 EDITORIAL EDITOR 2 Musings Rik Farrow Rik Farrow OPINION MANAGING EDITOR 6 Video Conferencing Must Evolve Michael Mattioli Michele Nelson S E C U R I T Y COPY EDITORS Steve Gilmartin Build It, Break It, Fix It Contests: Motivated Developers Still 9 Amber Ankerholz Make Security Mistakes Daniel Votipka, Kelsey R. Fulton, James Parker, Matthew Hou, Michelle L. Mazurek, and Michael Hicks PRODUCTION 15 The Road to Less Trusted Code: Lowering the Barrier to Arnold Gatilao In-Process Sandboxing Tal Garfinkel, Shravan Narayan, Craig Disselkoen, Ann Heron Hovav Shacham, and Deian Stefan Jasmine Murcia Olivia Vernetti 23 Using Safety Properties to Generate Vulnerability Patches Zhen Huang, David Lie, Gang Tan, and Trent Jaeger TYPESETTER Linda Davis 29 Interview with Sergey Bratus Rik Farrow USENIX ASSOCIATION SYSTEMS 2560 Ninth Street, Suite 215 35 Characterization and Optimization of the Serverless Workload Berkeley, California 94710, USA at a Large Cloud Provider Mohammad Shahrad, Rodrigo Fonseca, Phone: +1 510.528.8649 Íñigo Goiri, Gohar Chaudhry, and Ricardo Bianchini [email protected] 40 Posh: A Data-Aware Shell Deepti Raghavan, Sadjad Fouladi, www.usenix.org Philip Levis, and Matei Zaharia POSTMASTER: Send address changes to Rik Farrow 46 Interview with Margo Seltzer ;login:, USENIX Association, 2560 Ninth Street, SRE Suite 215, Berkeley, CA 94710, USA. 49 SRE Best Practices for Capacity Management Luis Quesada Torres ©2020 USENIX Association and Doug Colish USENIX is a registered trademark of the 57 The Case for CS Knowledge in SRE Adam McKaig USENIX Association. Many of the designa- tions used by manufacturers and sellers COLUMNS to distinguish their products are claimed 61 Book Review: Implementing Service Level Objectives as trademarks. USENIX acknowledges all by Alex Hidalgo Laura Nolan trademarks herein. Where those desig na tions appear in this publication and USENIX is 64 Systems Notebook: What’s in That Container? Cory Lueninghoener aware of a trademark claim, the designations 68 iVoyeur: BPF and Histograms Dave Josephsen have been printed in caps or initial caps. 71 SIGINFO: The Tricky Cryptographic Hash Function Simson L. Garfinkel Cover Image by BUMIPUTRA, distributed by Pixabay. 76 Programming Workbench: Compressed Sparse Row Format for Representing Graphs Terence Kelly 83 For Good Measure—Counting Broken Links: A Quant’s View of Software Supply Chain Security Dan Geer, Bentz Tozer, and John Speed Meyers 87 /dev/random: Discontent Creator Robert G. Ferrell BOOKS 89 Book Reviews Mark Lamourine and Rik Farrow USENIX NOTES 95 ;login: Enters a New Phase of Its Evolution Cat Allman, Rik Farrow, Casey Henderson, Arvind Krishnamurthy, and Laura Nolan 95 Interview with Clem Cole Rik Farrow 98 Interview with Kirk McKusick Rik Farrow 99 ;login: and Open Access Laura Nolan 100 Our Favorite ;login: Articles, 2005-2019 Rik Farrow, Laura Nolan, and Arvind Krishnamurthy Musings RIK FARROWEDITORIAL Rik is the editor of ;login:. ’ve sometimes been asked why computers are still so insecure, so emi- [email protected] nently hackable. Didn’t Bill Gates once shut down development at Micro- Isoft so they could improve the security of Windows decades ago? While not quite decades ago, Gates really did shut down Windows development in 2002 and sent 7,000 systems programmers to special security training with the goal of “Trustworthy Computing.” It didn’t work. While some things got better, and the rampage of worms slowed down, admin- istrators and users continued to have to install patches frequently. In 2003, Patch Tuesday became a regular feature, followed by Exploit Wednesday for those who had ignored the rou- tine of installing patches on the second Tuesday of the month. Part of Microsoft’s problem was a matter of programming culture, with a focus on new features. Exchange server, the email server product, actually had a good record for security, while the IIS web server certainly did not. Two distinct groups, with a different culture, worked on these products, resulting in very different security outcomes. But the problem of insecurity is not unique to Microsoft. Sun Microsystems started deliver- ing insecure workstations in the early 1980s, and continued to do so through the ’90s. Sun employees announced at USENIX Security that they had a program for securing SunOS, but it was for internal use only. Dan Farmer, Brad Powell, and Matt Archibald released Titan for Solaris in 1998 as a public solution to tightening and securing Solaris. Linux was having severe issues with security at the end of the ’90s but quickly improved over the next couple of years. But today, people build malware specifically for Linux, as Linux servers and desktops have become important targets for invading networks. So far, all I’ve done is write about how the struggle to defend software against exploits has been a failure, but not why. The answer lies partially in the nature of software and largely because of our hardware designs. First, programming is hard. I am constantly amazed at people announcing that they intend to turn everyone into a programmer. Perhaps these well-meaning projects can turn some people into middling programmers, but not ones who will be writing the next generation of services. I have had the misfortune of consulting in IT shops and have seen the carnage firsthand. On the plus side, when I turned out a handful of lines of shell script that did what they had failed to do in weeks, it made me look like a wizard. I have said this before: most programmers, by defini- tion, have an average skill level, and half are below average. This is hard to remember when you work in Silicon Valley or at a top-ten university and all of your coworkers are geniuses. Second, our computer systems were not designed for security. They were designed to be flex- ible. There are hardware security mechanisms that are important to security, such as the so- called rings, with the lowest numbered ring having the most access to hardware, and higher rings being reserved for “untrusted” code. Yet the largest and most complex programs run on most systems are the operating systems, and these run at the innermost ring. That makes the operating system the most important target for any attacker. Microsoft has taken advantage of the ring added to support virtualization, called ADM-V or Intel VT, in Windows 10. They load kernel modules using Virtual Secure Mode, where the 2 WINTER 2020 VOL. 45, NO. 4 www.usenix.org EDITORIAL Musings operating system and critical system modules get executed in Listing 1 depicts the main() function for a “Hello World!” pro- virtual containers. This beats the pants off the Linux model, gram. Compilers produce assembler as an intermediate format, where the kernel resides in a single address space, but still hasn’t and that’s what appears in Listing 1. You can learn to program prevented bootkits from being installed in Windows 10 systems. in assembler, but you have to handle things that compilers make This is supposed to be prevented by UEFI, but this can be worked easy to do, like choosing the register to use (anything beginning around using firmware rootkits and on many motherboards with %), managing the stack, managing memory. Each CPU archi- because of the wrong settings being used. tecture has a different set of registers and assembly instructions, although assemblers themselves, like as, work the same. You still Memory management is the next level of protection, but it have comments, but assembler is hard to read and is not portable was designed to protect programs running in one process from between CPU architectures. programs running in another process. Through