DOCSLIB.ORG

Roll Your Own? a Neer Klaus Knopper

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Roll Your Own? a Neer Klaus Knopper Build your own Distro Intro COVER STORY ROLLTools and techniques forYOUR creating custom Linux systemsOWN Tools such as Linux From Scratch, NimbleX Live CD Generator, Kiwi image system, and the Ubuntu Custom- ization Kit can help you create your own custom Linux system. BY KLAUS KNOPPER f you can’t find a Linux that strikes by Knoppix founder and Live-distro pio- these CDs were merely intended as huge your fancy, why not roll your own? A neer Klaus Knopper. bootable floppies with a command shell Inew generation of Linux tools helps and some troubleshooting tools, I started you build a custom Linux system. You Why Create a Live CD? thinking about the possibility of creating can use these distro-building tools to In 1999, collections of system repair a CD that was not just a “set of tools,” create Live CDs – bootable discs suitable tools on bootable business-card-sized but a full Linux desktop system on a CD. for troubleshooting, mass installation, CDs became a If I could put a and even ordinary end-user tasks. Add a popular mar- whole Linux sys- missing driver to the base system and re- keting give- tem on a portable master a modified DVD, then start your away at CD, I wouldn’t familiar home environment from any- Linux-related have to carry a where – without disturbing the contents expos. laptop with me to of the hard drive. Although foreign IT environ- Today’s Live systems aren’t confined ments. I could just to the CD tray. Live-Linux users also boot from USB sticks and other portable media. And not all custom distros fall in the Live niche. If you’re really ambi- tious, you can build a conventional hard-disk system file by file. This month, we examine tools for creating Linux systems. We start with a look at the doc- uments and utilities of the Linux From Scratch project. We also take you to the NimbleX website, where you can create a custom bootable CD online. And we examine a pair of competing distribution tools from a pair of major vendors: SUSE’s Kiwi and the Ubuntu Customization Kit (UCK). But first, we start with an intro- duction to the challenges of Live Linux COVER STORY Linux From Scratch .......................... 31 NimbleX Live CD Generator ....... 36 Kiwi .............................................................. 40 Ubuntu Customization Kit .......... 43 MARCH 2008 ISSUE 88 21 021-025_coverintro.indd 21 17.01.2008 18:54:23 Uhr COVER STORY Build your own Distro Intro isolinux.bin is a small program with the same purpose as lilo or grub. This mini-bootloader loads an operating sys- tem’s kernel and accepts options typed 3. Read file again from within an interactive boot shell. User Process 2. Write/modify The isolinux.bin boot loader also allows same file loading of additional menus. Some dis- tributions offer a fancy interface with graphics, interactive elements, and mouse support, but so far, the plain iso- linux bootloader has shown the best 2. Reading file /ramdisk (read-write) compatibility with the full range of BIOS NOT in ramdisk yet types. With isolinux.bin, cases in which the CD won’t boot because of problems with the BIOS/ bootloader combination are very rare. /KNOPPIX on CD (read-write) isolinux allows boot options and ker- nel choices, but you have to type them on the boot command line. If you just hit mount - t aufs -o br:/ramdisk=rw:/KNOPPIX=ro none /UNIONFS Return without entering the options, iso- linux continues to load the kernel with Figure 1: A union filesystem invisibly integrates a writable ramdisk with a read-only medium. default options set in isolinux.cfg (List- ing 2). Because isolinux is capable of slip my portable system into any avail- some of which you’ll learn about in this reading files directly from the ISO file- able computer and boot my own familiar issue. Behind the scenes, though, the de- system structure of a CD (hence the work environment. The concept of boot- velopers of these tools face the same name), no additional drivers or mounts ing from an easily transportable medium problems I faced when creating the first are necessary in order to access the ker- was a very promising idea. versions of Knoppix. No matter what nel and the initial ramdisk. Some Linux distributors offered Live special features you put inside, a good isolinux accesses the CD using BIOS demos and semi-Live bootable CDs (CDs Live system has to boot Linux, support calls. As soon as the operating system that offered some basic tools as a front write access to important files, and suc- kernel starts and switches to protected end for a hard-disk installation), but cessfully detect the system hardware. mode, everything about the CD – includ- these tools were intended for demonstra- ing its location – is “forgotten.” At that tion purposes – to get people to buy the Booting from a CD or DVD point, the operating system must find “real” product in a shrink-wrapped box. Although hard-disk-like devices (includ- where the CD is located, mount its file- I wanted a Live system for practical ing USB sticks, Zip media, and even work: software tests, presentations, and floppy disks) read a boot record at de- Listing 2: isolinux.cfg play. I did not spend much time on find- vice or partition start, the El Torito BIOS 01 DEFAULT linux ing a fancy name for this experiment; I standard for CD-ROMs and DVDs takes a 02 APPEND initrd=minirt.gz just called it Knoppix [1]. different approach to loading an OS. The Affordable USB memory sticks and developer has three options for design- 03 TIMEOUT 300 USB hard disks were not available then, ing a bootable CD or DVD: 04 PROMPT 1 so I decided to take a look at the techni- • Put a bootable floppy disk image 05 cal aspects of booting and running soft- (which cannot be larger than 2.88MB) 06 LABEL knoppix ware directly from a read-only CD. This on the CD and set floppy emulation 07 KERNEL linux task posed a unique set of problems, mode. which I will describe in this article. • Use a (very small) hard-disk image 08 APPEND initrd=minirt.gz Live systems have grown much more (a seldom-used option). sophisticated since the early days. • Start a bootloader program present on SysLinux Knoppix now comes as a full DVD the CD in no emulation mode. (See An equivalent to isolinux for DOS- image. Several tools are available for Listing 1 for a command that creates a formatted hard-disk-like devices (such creating custom Live CDs and DVDs – bootable CD ISO image.) as USB sticks and flash memory) is called SysLinux. It is quite easy to create Listing 1: Creating a Bootable CD ISO Image a bootable memory stick from a CD by just installing SysLinux and renaming 01 mkisofs -pad -l -r -J \ boot/isolinux/isolinux.cfg to syslinux. 02 -no-emul-boot -boot-load-size 4 -boot-info-table \ cfg. Knoppix versions from 5.0 and up 03 -b boot/isolinux/isolinux.bin -c boot/isolinux/boot.cat \ include a small utility called mkbootdev 04 -o cd-image.iso cd-data-directory for this task. 22 ISSUE 88 MARCH 2008 021-025_coverintro.indd 22 17.01.2008 18:54:59 Uhr Upcoming Conferences 2008 LINUX STORAGE & FILESYSTEM WORKSHOP THE SIXTH INTERNATIONAL CONFERENCE ON Co-located with FAST ’08 MOBILE SYSTEMS, APPLICATIONS, AND SERVICES FEBRUARY 25–26, 2008, SAN JOSE, CA, USA (MOBISYS 2008) http://www.usenix.org/lsf08 Jointly sponsored by ACM SIGMOBILE and USENIX JUNE 10–13, 2008, BRECKENRIDGE, CO, USA http://www.sigmobile.org/mobisys/2008/ 6TH USENIX CONFERENCE ON FILE AND STORAGE TECHNOLOGIES (FAST ’08) Sponsored by USENIX in cooperation with ACM SIGOPS, IEEE Mass 2008 USENIX ANNUAL TECHNICAL CONFERENCE Storage Systems Technical Committee (MSSTC), and IEEE TCOS JUNE 22–27, 2008, BOSTON, MA, USA FEBRUARY 26–29, 2008, SAN JOSE, CA, USA http://www.usenix.org/usenix08 http://www.usenix.org/fast08 2008 USENIX/ACCURATE ELECTRONIC 2008 ACM INTERNATIONAL CONFERENCE ON VIRTUAL VOTING TECHNOLOGY WORKSHOP (EVT ’08) EXECUTION ENVIRONMENTS (VEE ’08) Co-located with USENIX Security ’08 Sponsored by ACM SIGPLAN in cooperation with USENIX JULY 28–29, 2008, SAN JOSE, CA, USA MARCH 5–7, 2008, SEATTLE, WA, USA http://vee08.cs.tcd.ie 3RD USENIX WORKSHOP ON HOT TOPICS IN SECURITY (HOTSEC ’08) USABILITY, PSYCHOLOGY, AND SECURITY 2008 Co-located with USENIX Security ’08 Co-located with NSDI ’08 JULY 29, 2008, SAN JOSE, CA, USA APRIL 14, 2008, SAN FRANCISCO, CA, USA http://www.usenix.org/upsec08 17TH USENIX SECURITY SYM POSIUM JULY 28–AUGUST 1, 2008, SAN JOSE, CA, USA 1ST USENIX WORKSHOP ON LARGE-SCALE http://www.usenix.org/sec08 EXPLOITS AND EMERGENT THREATS (LEET ’08) BOTNETS, SPYWARE, WORMS, AND MORE Co-located with NSDI ’08 22ND LARGE INSTALLATION SYSTEM ADMINISTRATION APRIL 15, 2008, SAN FRANCISCO, CA, USA CONFERENCE (LISA ’08) http://www.usenix.org/leet08 Sponsored by USENIX and SAGE Paper submissions due: February 11, 2008 NOVEMBER 9–14, 2008, SAN DIEGO, CA, USA 5TH USENIX SYMPOSIUM ON NETWORKED 8TH USENIX SYMPOSIUM ON OPERATING SYSTEMS SYSTEMS DESIGN AND IMPLEMENTATION DESIGN AND IMPLEMENTATION (OSDI ’08) (NSDI ’08) DECEMBER 8–10, 2008, SAN DIEGO, CA, USA Sponsored by USENIX in cooperation with ACM SIGCOMM http://www.usenix.org/osdi08 and ACM SIGOPS Paper submissions due: May 8, 2008 APRIL 16–18, 2008, SAN FRANCISCO, CA, USA http://www.usenix.org/nsdi08 USENIX: THE ADVANCED COMPUTING SYSTEMS ASSOCIATION TECHNICAL SESSIONS AND TRAINING PROGRAM INFORMATION AND HOW TO REGISTER ARE AVAILABLE ONLINE AND FROM THE USENIX OFFICE: http://www.usenix.org/events | Email: [email protected] | Tel: +1.510.528.8649 | Fax: +1.510.548.5738 021-025_coverintro.indd 23 17.01.2008 18:55:17 Uhr COVER STORY Build your own Distro Intro system, and continue to start the system.
Load more

Recommended publications
  • Ubuntu Kung Fu
    Prepared exclusively for Alison Tyler Download at Boykma.Com What readers are saying about Ubuntu Kung Fu Ubuntu Kung Fu is excellent. The tips are fun and the hope of discov- ering hidden gems makes it a worthwhile task. John Southern Former editor of Linux Magazine I enjoyed Ubuntu Kung Fu and learned some new things. I would rec- ommend this book—nice tips and a lot of fun to be had. Carthik Sharma Creator of the Ubuntu Blog (http://ubuntu.wordpress.com) Wow! There are some great tips here! I have used Ubuntu since April 2005, starting with version 5.04. I found much in this book to inspire me and to teach me, and it answered lingering questions I didn’t know I had. The book is a good resource that I will gladly recommend to both newcomers and veteran users. Matthew Helmke Administrator, Ubuntu Forums Ubuntu Kung Fu is a fantastic compendium of useful, uncommon Ubuntu knowledge. Eric Hewitt Consultant, LiveLogic, LLC Prepared exclusively for Alison Tyler Download at Boykma.Com Ubuntu Kung Fu Tips, Tricks, Hints, and Hacks Keir Thomas The Pragmatic Bookshelf Raleigh, North Carolina Dallas, Texas Prepared exclusively for Alison Tyler Download at Boykma.Com Many of the designations used by manufacturers and sellers to distinguish their prod- ucts are claimed as trademarks. Where those designations appear in this book, and The Pragmatic Programmers, LLC was aware of a trademark claim, the designations have been printed in initial capital letters or in all capitals. The Pragmatic Starter Kit, The Pragmatic Programmer, Pragmatic Programming, Pragmatic Bookshelf and the linking g device are trademarks of The Pragmatic Programmers, LLC.
    [Show full text]
  • CERIAS Tech Report 2017-5 Deceptive Memory Systems by Christopher N
    CERIAS Tech Report 2017-5 Deceptive Memory Systems by Christopher N. Gutierrez Center for Education and Research Information Assurance and Security Purdue University, West Lafayette, IN 47907-2086 DECEPTIVE MEMORY SYSTEMS ADissertation Submitted to the Faculty of Purdue University by Christopher N. Gutierrez In Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy December 2017 Purdue University West Lafayette, Indiana ii THE PURDUE UNIVERSITY GRADUATE SCHOOL STATEMENT OF DISSERTATION APPROVAL Dr. Eugene H. Spa↵ord, Co-Chair Department of Computer Science Dr. Saurabh Bagchi, Co-Chair Department of Computer Science Dr. Dongyan Xu Department of Computer Science Dr. Mathias Payer Department of Computer Science Approved by: Dr. Voicu Popescu by Dr. William J. Gorman Head of the Graduate Program iii This work is dedicated to my wife, Gina. Thank you for all of your love and support. The moon awaits us. iv ACKNOWLEDGMENTS Iwould liketothank ProfessorsEugeneSpa↵ord and SaurabhBagchi for their guidance, support, and advice throughout my time at Purdue. Both have been instru­ mental in my development as a computer scientist, and I am forever grateful. I would also like to thank the Center for Education and Research in Information Assurance and Security (CERIAS) for fostering a multidisciplinary security culture in which I had the privilege to be part of. Special thanks to Adam Hammer and Ronald Cas­ tongia for their technical support and Thomas Yurek for his programming assistance for the experimental evaluation. I am grateful for the valuable feedback provided by the members of my thesis committee, Professor Dongyen Xu, and Professor Math­ ias Payer.
    [Show full text]
  • HTTP-FUSE Xenoppix
    HTTP-FUSE Xenoppix Kuniyasu Suzaki† Toshiki Yagi† Kengo Iijima† Kenji Kitagawa†† Shuichi Tashiro††† National Institute of Advanced Industrial Science and Technology† Alpha Systems Inc.†† Information-Technology Promotion Agency, Japan††† {k.suzaki,yagi-toshiki,k-iijima}@aist.go.jp [email protected], [email protected] Abstract a CD-ROM. Furthermore it requires remaking the entire CD-ROM when a bit of data is up- dated. The other solution is a Virtual Machine We developed “HTTP-FUSE Xenoppix” which which enables us to install many OSes and ap- boots Linux, Plan9, and NetBSD on Virtual plications easily. However, that requires in- Machine Monitor “Xen” with a small bootable stalling virtual machine software. (6.5MB) CD-ROM. The bootable CD-ROM in- cludes boot loader, kernel, and miniroot only We have developed “Xenoppix” [1], which and most part of files are obtained via Internet is a combination of CD/DVD bootable Linux with network loopback device HTTP-FUSE “KNOPPIX” [2] and Virtual Machine Monitor CLOOP. It is made from cloop (Compressed “Xen” [3, 4]. Xenoppix boots Linux (KNOP- Loopback block device) and FUSE (Filesys- PIX) as Host OS and NetBSD or Plan9 as Guest tem USErspace). HTTP-FUSE CLOOP can re- OS with a bootable DVD only. KNOPPIX construct a block device from many small block is advanced in automatic device detection and files of HTTP servers. In this paper we describe driver integration. It prepares the Xen environ- the detail of the implementation and its perfor- ment and Guest OSes don’t need to worry about mance. lack of device drivers.
    [Show full text]
  • Open Source Projects As Incubators of Innovation
    RESEARCH CONTRIBUTIONS TO ORGANIZATIONAL SOCIOLOGY AND INNOVATION STUDIES / STUTTGARTER BEITRÄGE ZUR ORGANISATIONS- UND INNOVATIONSSOZIOLOGIE SOI Discussion Paper 2017-03 Open Source Projects as Incubators of Innovation From Niche Phenomenon to Integral Part of the Software Industry Jan-Felix Schrape Institute for Social Sciences Organizational Sociology and Innovation Studies Jan-Felix Schrape Open Source Projects as Incubators of Innovation. From Niche Phenomenon to Integral Part of the Software Industry. SOI Discussion Paper 2017-03 University of Stuttgart Institute for Social Sciences Department of Organizational Sociology and Innovation Studies Seidenstr. 36 D-70174 Stuttgart Editor Prof. Dr. Ulrich Dolata Tel.: +49 711 / 685-81001 [email protected] Managing Editor Dr. Jan-Felix Schrape Tel.: +49 711 / 685-81004 [email protected] Research Contributions to Organizational Sociology and Innovation Studies Discussion Paper 2017-03 (May 2017) ISSN 2191-4990 © 2017 by the author(s) Jan-Felix Schrape is senior researcher at the Department of Organizational Sociology and Innovation Studies, University of Stuttgart (Germany). [email protected] Additional downloads from the Department of Organizational Sociology and Innovation Studies at the Institute for Social Sciences (University of Stuttgart) are filed under: http://www.uni-stuttgart.de/soz/oi/publikationen/ Abstract Over the last 20 years, open source development has become an integral part of the software industry and a key component of the innovation strategies of all major IT providers. Against this backdrop, this paper seeks to develop a systematic overview of open source communities and their socio-economic contexts. I begin with a recon- struction of the genesis of open source software projects and their changing relation- ships to established IT companies.
    [Show full text]
  • Software Wars, Business Strategies and IP Litigation
    Software Wars, Business Strategies and IP Litigation Immediately after Oracle America filed a patent infringement suit against Google Inc. in August 2010 the trade press labelled this a “software war.” Their interest was a trial featuring Oracle’s “star” counsel David Boies. In mid-January often cited Florian Mueller of NoSoftwarePatents, wrote “Google is patently too weak to protect Android—Google’s cell phone software.”1 (On April 4 Lead bidder Google offered $900 million for Nortel’s 6,000 patents and applications. A decision is expected in June}. By the end of March he had counted 39 patent infringement suits against Google.2 Whether Oracle wins Oracle v Google or not, Oracle and IBM may become losers. It is important that Oracle not alienate the software development community or its customers as it attempts to monetise the assets from the Sun Microsystems—now Oracle America— acquisition. The Oracle v. Google complaint was carefully targeted at Google’s Android software, avoiding the appearance of monetising the Java language and platform itself. The Android software was developed as open source in conjunction with the Open Handset Alliance. Android's mobile operating system is based on a modified version of the Linux kernel, subject to Linux patents. Google published all of the computer code under an Apache Software Foundation License.3 This license is royalty-free and places no restrictions on derivative products or commercial use. Based on IDC market share estimates by the fourth quarter of 2010, Android phones had 39% of the market. An Oracle asset gained through the acquisition of Sun was the OpenOffice open-source software—one of the only remaining substantial “competitors” to Microsoft’s document processing applications.
    [Show full text]
  • DMFS - a Data Migration File System for Netbsd
    DMFS - A Data Migration File System for NetBSD William Studenmund Veridian MRJ Technology Solutions NASAAmes Research Center" Abstract It was designed to support the mass storage systems de- ployed here at NAS under the NAStore 2 system. That system supported a total of twenty StorageTek NearLine ! have recently developed DMFS, a Data Migration File tape silos at two locations, each with up to four tape System, for NetBSD[I]. This file system provides ker- drives each. Each silo contained upwards of 5000 tapes, nel support for the data migration system being devel- and had robotic pass-throughs to adjoining silos. oped by my research group at NASA/Ames. The file system utilizes an underlying file store to provide the file The volman system is designed using a client-server backing, and coordinates user and system access to the model, and consists of three main components: the vol- files. It stores its internal metadata in a flat file, which man master, possibly multiple volman servers, and vol- resides on a separate file system. This paper will first man clients. The volman servers connect to each tape describe our data migration system to provide a context silo, mount and unmount tapes at the direction of the for DMFS, then it will describe DMFS. It also will de- volman master, and provide tape services to clients. The scribe the changes to NetBSD needed to make DMFS volman master maintains a database of known tapes and work. Then it will give an overview of the file archival locations, and directs the tape servers to move and mount and restoration procedures, and describe how some typi- tapes to service client requests.
    [Show full text]
  • Unionfs: User- and Community-Oriented Development of a Unification File System
    Unionfs: User- and Community-Oriented Development of a Unification File System David Quigley, Josef Sipek, Charles P. Wright, and Erez Zadok Stony Brook University {dquigley,jsipek,cwright,ezk}@cs.sunysb.edu Abstract If a file exists in multiple branches, the user sees only the copy in the higher-priority branch. Unionfs allows some branches to be read-only, Unionfs is a stackable file system that virtually but as long as the highest-priority branch is merges a set of directories (called branches) read-write, Unionfs uses copy-on-write seman- into a single logical view. Each branch is as- tics to provide an illusion that all branches are signed a priority and may be either read-only writable. This feature allows Live-CD develop- or read-write. When the highest priority branch ers to give their users a writable system based is writable, Unionfs provides copy-on-write se- on read-only media. mantics for read-only branches. These copy- on-write semantics have lead to widespread There are many uses for namespace unifica- use of Unionfs by LiveCD projects including tion. The two most common uses are Live- Knoppix and SLAX. In this paper we describe CDs and diskless/NFS-root clients. On Live- our experiences distributing and maintaining CDs, by definition, the data is stored on a read- an out-of-kernel module since November 2004. only medium. However, it is very convenient As of March 2006 Unionfs has been down- for users to be able to modify the data. Uni- loaded by over 6,700 unique users and is used fying the read-only CD with a writable RAM by over two dozen other projects.
    [Show full text]
  • We Get Letters Sept/Oct 2018
    SEE TEXT ONLY WeGetletters by Michael W Lucas letters@ freebsdjournal.org tmpfs, or be careful to monitor tmpfs space use. Hey, FJ Letters Dude, Not that you’ll configure your monitoring system Which filesystem should I use? to watch tmpfs, because it’s temporary. And no matter what, one day you’ll forget —FreeBSD Newbie that you used memory space as a filesystem. You’ll stash something vital in that temporary space, then reboot. And get really annoyed Dear FreeBSD Newbie, when that vital data vanishes into the ether. First off, welcome to FreeBSD. The wider com- Some other filesystems aren’t actively terrible. munity is glad to help you. The device filesystem devfs(5) provides device Second, please let me know who told you to nodes. Filesystems that can’t store user data are start off by writing me. I need to properly… the best filesystems. But then some clever sysad- “thank” them. min decides to hack on /etc/devfs.rules to Filesystems? Sure, let’s talk filesystems. change the standard device nodes for their spe- Discussing which filesystem is the worst is like cial application, or /etc/devd.conf to create or debating the merits of two-handed swords as reconfigure device nodes, and the whole system compared to lumberjack-grade chainsaws and goes down the tubes. industrial tulip presses. While every one of them Speaking of clever sysadmins, now and then has perfectly legitimate uses, in the hands of the people decide that they want to optimize disk novice they’re far more likely to maim everyone space or cut down how many copies of a file involved.
    [Show full text]
  • 60 External USB Hard Drive KNOW-HOW 60
    KNOW-HOW Ask Klaus! ASK KLAUS! during unmounting, and lost data results I edit an existing script that already on a frequent basis when I disconnect comes with Fedora 7 (as used by the the device. desktop), to resolve the problem? Or is The drive heads are not stopped ei- this behavior compiled in? Klaus Knopper is the creator of ther, and a strange whirring chirp sound I believe the “eject” feature used by Knoppix and co-founder of the occurs when unplugging the drive while Gnome is only meant for flash memory the platters are still spinning. drives such as USB keys, where there is LinuxTag expo. He currently I searched Google and found a shell no concept of flushing the data in RAM script to properly unmount the drive, to the magnetic storage medium and works as a teacher, program- and it uses a tool called sdparm to sync stopping the heads, and hence it is a mer, and consultant. If you have and stop the drive heads before physi- rather simple solution for unmounting cally disconnecting the USB hard drive. I that is not suitable for external portable a configuration problem, or if did not have sdparm installed by default hard drives. from my Fedora 7 distribution (obtained Thanks in advance for considering this you just want to learn more through a Linux Magazine DVD) and Linux hardware challenge. about how Linux works, send had to search and install the tool myself using the package manager. (Zenity mes- The quite long script you sent your questions to: sage boxes also had to be installed for can be reduced to (more or less) this script.) this skeleton: klaus@linux-magazine.
    [Show full text]
  • LSF ’07: 2007 Linux Storage & Filesystem Workshop Namically Adjusted According to the Current Popularity of Its Hot Zone
    June07login1summaries_press.qxd:login summaries 5/27/07 10:27 AM Page 84 PRO: A Popularity-Based Multi-Threaded Reconstruction overhead of PRO is O(n), although if a priority queue is Optimization for RAID-Structured Storage Systems used in the PRO algorithm the computation overhead can Lei Tian and Dan Feng, Huazhong University of Science and be reduced to O(log n). The entire PRO implementation in Technology; Hong Jiang, University of Nebraska—Lincoln; Ke the RAIDFrame software only added 686 lines of code. Zhou, Lingfang Zeng, Jianxi Chen, and Zhikun Wang, Hua- Work on PRO is ongoing. Future work includes optimiz - zhong University of Science and Technology and Wuhan ing the time slice, scheduling strategies, and hot zone National Laboratory for Optoelectronics; Zhenlei Song, length. Currently, PRO is being ported into the Linux soft - Huazhong University of Science and Technology ware RAID. Finally, the authors plan on further investigat - ing use of access patterns to help predict user accesses and Hong Jiang began his talk by discussing the importance of of filesystem semantic knowledge to explore accurate re - data recovery. Disk failures have become more common in construction. RAID-structured storage systems. The improvement in disk capacity has far outpaced improvements in disk band - The first questioner asked about the average rate of recov - width, lengthening the overall RAID recovery time. Also, ery for PRO. Hong answered that the average reconstruc - disk drive reliability has improved slowly, resulting in a tion time is several hundred seconds in the experimental very high overall failure rate in a large-scale RAID storage setup.
    [Show full text]
  • Modular Data Storage with Anvil
    Modular Data Storage with Anvil Mike Mammarella Shant Hovsepian Eddie Kohler UCLA UCLA UCLA/Meraki [email protected] [email protected] [email protected] http://www.read.cs.ucla.edu/anvil/ ABSTRACT age strategies and behaviors. We intend Anvil configura- Databases have achieved orders-of-magnitude performance tions to serve as single-machine back-end storage layers for improvements by changing the layout of stored data – for databases and other structured data management systems. instance, by arranging data in columns or compressing it be- The basic Anvil abstraction is the dTable, an abstract key- fore storage. These improvements have been implemented value store. Some dTables communicate directly with sta- in monolithic new engines, however, making it difficult to ble storage, while others layer above storage dTables, trans- experiment with feature combinations or extensions. We forming their contents. dTables can represent row stores present Anvil, a modular and extensible toolkit for build- and column stores, but their fine-grained modularity of- ing database back ends. Anvil’s storage modules, called dTa- fers database designers more possibilities. For example, a bles, have much finer granularity than prior work. For ex- typical Anvil configuration splits a single “table” into sev- ample, some dTables specialize in writing data, while oth- eral distinct dTables, including a log to absorb writes and ers provide optimized read-only formats. This specialization read-optimized structures to satisfy uncached queries. This makes both kinds of dTable simple to write and understand. split introduces opportunities for clean extensibility – for Unifying dTables implement more comprehensive function- example, we present a Bloom filter dTable that can slot ality by layering over other dTables – for instance, building a above read-optimized stores and improve the performance read/write store from read-only tables and a writable journal, of nonexistent key lookup.
    [Show full text]
  • Updates on CVMFS and Container Integration
    Updates on CVMFS and container integration EP-SFT weekly meeting, 14 - Sept - 2020 Simone Mosciatti 1 Background on CVMFS ● FUSE filesystem targeted to software distribution ● Works over HTTP ● Lazy pull of files based on actual file-system calls ● Great bandwidth efficiency ● Aggressive use of caches + CAS for improving latency ● The file-system can grow indefinitely in size keeping good performances as long as the subcatalogs are managed correctly. ● Widely used and deployed inside CERN, and great interest also outside (EESI, Microsoft, HFT firms, other scientific experiments). 2 Background on containers ● Use linux namespace to create isolated environment where to run computation ● Distribute the container root file-system as hash-verified tarballs ● In a nutshell a. Create the root filesystem by stacking the content of hash-verified tarballs on top of each other b. Create an isolated environment using namespaces c. Run (reproducible) computation ● Different implementations (docker, singularity, podman, k8s-crio). ● Underneath they use all the same pieces a. containerd b. containers/storage ● Moving towards rootless implementations, simplifying deployment on GRID, data centers and supercomputer where CVMFS is already present. 3 How containers can be consumed (1) Unpacked file-system a. The whole file-system of the container is provided as a directory b. The container runtime pick up the directory filesystem and creates all the infrastructure c. Works out of the box with Singularity d. Works with podman with some tricks (specifying the runtime engine) singularity exec /cvmfs/unpacked.cern.ch/registry.hub.docker.com/atlas/athena:21.0.77/ All the file-system of the container is already in the directory 4 How containers can be consumed (2) Layers a.
    [Show full text]