J U N E 2 0 0 8 V O L U M E 3 3 N U M B E R 3 OPINION Musings 2 Rik Farrow SYSADMin Data Corruption in the Storage Stack: A Closer Look 6 Lakshmi BaiR avasundaR a m , THE USENIX MAGAZINE GaRth Goodson, Bianca schRoedeR, andRea Arpaci-dusseau, and Remzi aRpaci-dusseau Pergamum: Energy-efficient Archival Storage with Disk Instead of Tape 15 m aRk w. stoReR, kevin m . GReenan, ethan L . milleR, and k aL adhaR voRuGanti Don’t Blame Disks for Every Storage Subsystem Failure 22 weihanG JianG, chongfenG hu, YuanYuan zhou, and Ark adY k anevsk Y TierStore: A Distributed File System for Challenged Networks in Developing Regions 32 michaeL demmeR, Bowei du, and eRic BReweR The Present and Future of SAN/NAS: Interview with Dave Hitz and Brian Pawlowsky of NetApp 39 inteRview by m argo seLtzeR PROGRAMMinG Driving the Evolution of Software Languages to a Concurrent Future 45 andRew BRownswoRd The Murky Issue of Changing Process Identity: Revising “Setuid Demystified” 55 dan tsafriR, diL m a da siLva , and david waGneR COLUMns Practical Perl Tools: A Little Place for Your Stuff 67 david n. BL ank-edeL m an Pete’s All Things Sun (PATS): The State of ZFS 72 peteR BaeR GaLvin iVoyeur: Admin, Root Thyself. 77 david Josephsen /dev/random 83 RoBeRt G. Ferrell STANDARDS Update on Standards: Undue Influence? 85 nick stouGhton BooK REVIEWS Book Reviews 88 ElizaBeth zwick Y et aL . USEniX NOTES Election Results 91 Notice of Annual Meeting 91 ConFEREncES FAST ’08 Reports 93 LSF ’08 Reports 107 The Advanced Computing Systems Association June08_login_covers.indd 1 5/13/08 4:44:56 PM Upcoming Events The SixTh inTernaTional ConferenCe on WorkShop on Cyber SecuriTy Mobile Systems, appliCaTionS, and ServiCeS experiMenTaTion and Test (CSeT ’08) (MobiSyS 2008) Co-located with USENIX Security ’08 Jointly sponsored by ACM SIGMOBILE and USENIX July 28, 2008, SAn Jose, cA, uSA June 17–20, 2008, Breckenridge, cO, uSA http://www.usenix.org/cset08 http://www.sigmobile.org/mobisys/2008/ 17Th USENIX SecuriTy SyMpoSium 2008 USENIX annUal TeChniCal ConferenCe July 28–AuguSt 1, 2008, SAn Jose, cA, uSA June 22–27, 2008, BostOn, MA, uSA http://www.usenix.org/sec08 http://www.usenix.org/usenix08 3rd USENIX WorkShop on hoT Topics in first USENIX WorkShop on large-SCale SecuriTy (hoTSeC ’08) CoMputing (LASCo ’08) Co-located with USENIX Security ’08 Co-located with USENIX ’08 July 29, 2008, SAn Jose, cA, uSA June 23, 2008, BostOn, MA, uSA http://www.usenix.org/hotsec08 http://www.usenix.org/lasco08 Third WorkShop on SecuriTy MeTrics xen SummiT norTh aMeriCa 2008 (MeTriCon 3.0) Co-located with USENIX ’08 Co-located with USENIX Security ’08 June 23–24, 2008, BostOn, MA, uSA July 29, 2008, SAn Jose, cA, uSA http://xen.org/xensummit/ http://www.securitymetrics.org/content/Wiki. jsp?page=Metricon3.0 2nd inTernaTional ConferenCe on distributed evenT-baSed Systems (DEBS 2008) 22nd large installaTion SysteM adMinistraTion Organized in cooperation with USENIX, the IEEE and IEEE ConferenCe (LISa ’08) Computer Society, ACM SIGSOFT, and ACM SIGMOD Sponsored by USENIX and SAGE July 2–4, 2008, Rome, itAly November 9–14, 2008, SAn diegO, cA, uSA http://debs08.dis.uniroma1.it/ http://www.usenix.org/lisa08 2008 USENIX/aCCURATe electroniC SyMpoSium on CoMputer human inTeraction voTing TeChnology WorkShop (EVT ’08) for ManageMenT of inforMaTion TeChnology Co-located with USENIX Security ’08 (CHIMiT ’08) July 28–29, 2008, SAn Jose, cA, uSA Sponsored by ACM in association with USENIX http://www.usenix.org/evt08 November 14–15, 2008, SAn diegO, cA, uSA http://www.chimit08.org 2nd USENIX WorkShop on offenSive TeChnologieS (WOOT ’08) aCM/IFIP/USENIX 9Th inTernaTional Co-located with USENIX Security ’08 MiddleWare ConferenCe (MiddleWare 2008) July 28, 2008, SAn Jose, cA, uSA december 1–5, 2008, leuven, BelgiuM http://www.usenix.org/woot08 http://middleware2008.cs.kuleuven.be Submissions due: June 1, 2008 For a complete list of all USENIX & USENIX co-sponsored events, see http://www.usenix.org/events. June08_login_covers.indd 2 5/13/08 4:44:58 PM OPINION Musings 2 RIk Farrow SYSADMin Data Corruption in the Storage Stack: A Closer Look 6 L AkshmI BairavAsundara m , Garth Goodson, Bianca Schroeder, ANdrea ARpaci-Dusseau, and REmzI ARpaci-Dusseau Pergamum: Energy-efficient Archival Storage with Disk Instead of Tape 15 contents m ARk w. Storer, Kevin m . Greenan, EThan L . Miller, and k AL Adhar Voruganti Don’t Blame Disks for Every Storage Subsystem Failure 22 Weihang Jiang, Chongfeng Hu, Yuanyuan zhou, and ARk AdY k ANEvsk Y TierStore: A Distributed File System for Challenged Networks in Developing Regions 32 Michael Demmer, BOwei Du, and Eric Brewer The Present and Future of SAN/NAS: Interview with Dave Hitz and Brian Pawlowsky of NetApp 39 Interview by m ARGO Seltzer VOL. 33, #3, June 2008 PROGRAMMinG Driving the Evolution of Software Languages Editor ;login: is the official to a Concurrent Future 45 Rik Farrow magazine of the ANdrew BrowNsword [email protected] USENIX Association. The Murky Issue of Changing Process ;login: (ISSN 1044-6397) is Managing Editor Identity: Revising “Setuid Demystified” 55 Jane-Ellen Long published bi-monthly by the [email protected] USENIX Association, 2560 Dan Tsafrir, Dil m A Da SilvA , and Ninth Street, Suite 215, DavId Wagner Copy Editor Berkeley, CA 94710. David Couzens [email protected] $90 of each member’s annual COLUMns Practical Perl Tools: A Little Place for dues is for an annual sub- Your Stuff 67 produCtion scription to ;login:. Subscrip- Casey Henderson tions for nonmembers are DavId N. Blank-Edelm an Jane-Ellen Long $120 per year. Michele Nelson Pete’s All Things Sun (PATS): Periodicals postage paid at The State of ZFS 72 t ypEsEt tEr Berkeley, CA, and additional Peter Baer Galvin Star Type offices. [email protected] iVoyeur: Admin, Root Thyself. 77 POSTMASTER: Send address DavId JOsEphsen USENIX assoCiation changes to ;login:, 2560 Ninth Street, USENIX Association, /dev/random 83 Suite 215, Berkeley, 2560 Ninth Street, Robert G. Ferrell California 94710 Suite 215, Berkeley, Phone: (510) 528-8649 CA 94710. FAX: (510) 548-5738 ©2008 USENIX Association STANDARDS Update on Standards: Undue Influence? 85 http://www.usenix.org http://www.sage.org USENIX is a registered trade- Nick Stoughton mark of the USENIX Associa- tion. Many of the designations used by manufacturers and BooK REVIEWS Book Reviews 88 sellers to distinguish their Elizabeth zwick Y et al . products are claimed as trade- marks. USENIX acknowledges all trademarks herein. Where USEniX NOTES those designations appear in Election Results 91 this publication and USENIX Notice of Annual Meeting 91 is aware of a trademark claim, the designations have been printed in caps or initial caps. ConFEREncES FAST ’08 Reports 93 LSF ’08 Reports 107 ;LOGIN: JUNE 2008 ARTICLE TITLE login_articles_JUNE08.indd 1 5/13/08 4:50:47 PM We are all accustomed to our software having bugs. We would be sur- R i k Fa rr o w prised if our software actually worked perfectly, with no glitches or gaping secu- rity holes. Surprised? More like flummoxed. Oddly, we seem to have a much stronger belief about our hardware being perfect, that processors can perform 128-bit float- musings ing-point multiplies accurately, and that disks actually store what we ask them to. [email protected] Well, guess again. During FAST ’08, I was treated to what has become a yearly spectacle: researchers digging into massive disk-error databases to pick apart what goes wrong with disks while in production. At FAST ’07, the big news was the failure curve for hard drives not being bathtub-shaped. In 2008, researchers looked for, and found, other problems with using disks that are certainly surprising. As one researcher pointed out, the typical hard drive includes 300,000 lines of code in its firm- ware: room enough for errors, eh? And we thought we were using hardware, but like many hardware devices, even hard drives are software-controlled. As Goodson and his researchers write in the lead article in this issue, silent write errors are actu- ally a big problem. Their findings certainly have me longing for new filesystem designs, such as ZFS and the under-development BTRFS, that include checksums with the data they store. Even enter- prise-level drives have a problem with silent write errors, which is something you might not expect, as these drives write checksums and error-cor- recting code into each sector. But when the wrong data gets written, or data gets written to the wrong physical block, error-correction code just isn’t going to help you. Buggy Hardware There was once a time when I relied on hardware failure. I’ve built a lot of my desktop systems over the years and would use them until they were ob- solete (for three to five years). Over time, I would use my own personal method for organizing direc- tories, resulting in a pretty incredible mess. I could still find things, of course, and rarely lost things. But my file hierarchies began to look more and more like the amazing Winchester Mystery House in San Jose. ; LOGIN : vOL. 33, NO. 3 login_articles_JUNE08.indd 2 5/13/08 4:50:48 PM Then a miracle would happen. The hard drive would cease working, and only the directories I had deemed important enough to back up could be re- stored. I would start out with a brand new, empty filesystem, on a hard disk that was generally twice as big as the previous one. Backup media evolved from floppy disks (really) to quarter-inch tape cartridges (a whopping 45 MBs), then to CDs (700 MB!), and finally to the plug-in USB drive.