DOCSLIB.ORG

PERFORMANCE ANALYSIS of CONTAINERIZED APPLICATIONS on LOCAL and REMOTE STORAGE Qiumin Xu1, Manu Awasthi2, Krishna T

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
PERFORMANCE ANALYSIS of CONTAINERIZED APPLICATIONS on LOCAL and REMOTE STORAGE Qiumin Xu1, Manu Awasthi2, Krishna T PERFORMANCE ANALYSIS OF CONTAINERIZED APPLICATIONS ON LOCAL AND REMOTE STORAGE Qiumin Xu1, Manu Awasthi2, Krishna T. Malladi3, Janki Bhimani4, Jingpei Yang3, Murali Annavaram1 1USC, 2IIT Gandhinagar 3Samsung 4Northeastern 1 Docker Becomes Very Popular Software container platform with many desirable features Ease of deployment, developer friendliness and light virtualization Mainstay in cloud platforms Google Cloud Platform, Amazon EC2, Microsoft Azure Storage Hierarchy is the key component High Performance SSDs NVMe, NVMe over Fabrics 2 Agenda Docker, NVMe and NVMe over Fabrics (NVMf) How to best utilize NVMe SSDs for single container? Best configuration performs similar to raw performance Where do the performance anomalies come from? Do Docker containers scale well on NVMe SSDs? Exemplify using Cassandra Best strategy to divide the resources Scaling Docker containers on NVMe-over-Fabrics 3 What is Docker Container? Each virtualized application includes an entire OS (~10s of GB) Docker container comprises just application and bins/libs Shares the kernel with other container Much more portable and efficient figure from https://docs.docker.com 4 Non-Volatile Memory Express (NVMe) A storage protocol standard on top of PCIe NVMe SSDs connect through PCIe and support the standard Since 2014 (Intel, Samsung) Enterprise and consumer variants NVMe SSDs leverage the interface to deliver superior perf 5X to 10X over SATA SSD[1] [1] Qiumin Xu et al. “Performance analysis of NVMe SSDs and their implication on real world databases.” SYSTOR’15 5 Why NVMe over Fabrics (NVMf)? Retains NVMe performance over network fabrics Eliminate unnecessary protocol translations Enables low latency and high IOPS remote storage J. M. Dave Minturn, “Under the Hood with NVMe over Fabrics,”, SINA Ethernet Storage Forum 6 Storage Architecture in Docker Storage Options: 11. Through Docker Filesystem (Aufs, Btrfs, Overlayfs) 22. Through Virtual Block Devices (2.a Loop-lvm, 2.b Direct-lvm) 33. Through Docker Data Volume (-v) Container Read / Write Operations 1 -g option 2 3 -v option Storage Driver Aufs, Btrfs, Devicemapper Data Volume Overlayfs (Loop-lvm, direct-lvm) 2.a 2.b Host Backing File system (EXT4, XFS, etc.) Base Device Thin Pool Sparse Files NVMe SSDs 7 Optimize Storage Configuration for Single Container Experimental Environment Dual-socket, 12 HT cores Xeon E5-2670 V3 enterprise-class NVMe SSD Docker Samsung XS1715 FIO kernel v4.6.0 Docker v1.11.2 ? fio used for traffic generation Asynchronous IO engine, libaio NVMe SSD 32 concurrent jobs and iodepth is 32 (XS1715) Measure steady state performance 8 Performance Comparison — Host Backing Filesystems 800 K RAW 3000 RAW 600 EXT4 EXT4 2500 XFS XFS 400 2000 (MB/s) 200 1500 Average BW BW Average Average IOPS Average 0 1000 RR RW SR SW EXT4 performs 25% worse for RR XFS performs closely resembles RAW for all but RW 9 Tuning the Performance Gap —Random Reads 700K IOPS 800K 600 RR 400 200 IOPS of IOPS 0 Default dioread_nolock EXT4 XFS allows multiple processes to read a file at once Uses allocation groups which can be accessed independently EXT4 requires mutex locks even for read operations 10 Tuning the Performance Gap —Random Writes 250K RAW EXT4 XFS 200 RW 150 100 IOPS of IOPS 50 0 1 2 4 8 16 24 28 32 48 64 # of Jobs XFS performs poorly with high thread count Contention in exclusive locking kills the write performance Used by extent look up and write checks Patch available but not for Linux 4.6 [1] [1] https://www.percona.com/blog/2012/03/15/ext4-vs-xfs-on-ssd/ 11 Storage Architecture in Docker Storage Options: 11. Through Docker Filesystem (Aufs, Btrfs, Overlayfs) 22. Through Virtual Block Devices (2.a Loop-lvm, 2.b Direct-lvm) 33. Through Docker Data Volume (-v) Container Read / Write Operations 1 -g option 2 3 -v option Storage Driver Aufs, Btrfs, Devicemapper Data Volume Overlayfs (Loop-lvm, direct-lvm) 2.a 2.b Host Backing File system (EXT4, XFS, etc.) Base Device Thin Pool Sparse Files NVMe SSDs 12 Docker Storage Options Option 1: Through Docker File System Aufs (Advanced multi-layered Unification FileSystem): A fast reliable unification file system Btrfs (B-tree file system): A modern CoW file system which implements many advanced features for fault tolerance, repair and easy administration Overlayfs: Another modern unification file system which has simpler design and potentially faster than Aufs 13 Performance Comparison Option 1: Through Docker File System 800K Raw Raw 3000 Aufs 600 Aufs Btrfs 2500 Btrfs Overlay Overlay 400 2000 (MB/s) 200 1500 Average BW BW Average Average IOPS Average 0 1000 RR RW SR SW Aufs and Overlayfs performs close to raw block device for most cases Btrfs has the worst performance for random workloads 14 Tuning the Performance Gap of Btrfs —Random Reads 4000 R RAW EXT4 Btrfs 3000 2000 1000 BW (MB/s) of R BW (MB/s) 0 Block Size Btrfs doesn’t work well for small block size yet Btrfs must read the file extent before reading the file data. Large block size reduces the frequency of reading metadata 15 Tuning the Performance Gap of Btrfs —Random Reads 200K 150 100 IOPS of RW IOPS 50 0 Default nodatacow Btrfs Btrfs doesn’t work well for random writes due to CoW overhead 16 Storage Architecture in Docker Storage Options: 11. Through Docker Filesystem (Aufs, Btrfs, Overlayfs) 22. Through Virtual Block Devices (2.a Loop-lvm, 2.b Direct-lvm) 33. Through Docker Data Volume (-v) Container Read / Write Operations 1 -g option 2 3 -v option Storage Driver Aufs, Btrfs, Devicemapper Data Volume Overlayfs (Loop-lvm, direct-lvm) 2.a 2.b Host Backing File system (EXT4, XFS, etc.) Base Device Thin Pool Sparse Files NVMe SSDs 17 Docker Storage Configurations Option 2: Through Virtual Block Device Devicemapper storage driver leverages the thin provisioning and snapshotting capabilities of the kernel based Device Mapper Framework Loop-lvm uses sparse files to build the thin-provisioned pools Direct-lvm uses block device to directly create the thin pools (Recommended by Docker) 18 Docker Storage Configurations Option 3: Through Docker Data Volume (-v) Data persists beyond the lifetime of the container and can be shared and accessed from other containers * figure from https://github.com/libopenstorage/openstorage 19 Performance Comparison Option 2 & Option 3 800K RAW RAW 3000 600 Direct-lvm Direct-lvm Loop-lvm -v 2500 Loop-lvm -v 400 Aufs -v Aufs -v 2000 Overlay -v (MB/s) Overlay -v 200 Average BW BW Average 1500 Average IOPS Average 0 1000 RR RW SR SW Direct-lvm has worse performance for RR/RW LVM, device mapper, and the dm-thinp kernel module introduced additional code paths and overhead may not suit IO intensive workloads 20 Application Performance Cassandra Database NoSQL database Scale linearly to the number of nodes in the cluster (theoretically) [1] Requires data persistence uses docker volume to store data [1] Rabl, Tilmann et al. "Solving Big Data Challenges for Enterprise Application Performance Management”, VLDB’13 21 Scaling Docker Containers on NVMe multiple containerized Cassandra Databases Experiment Setup Dual socket, Xeon E5 server, 10Gb ethernet N = 1, 2, 3, … 8 containers Each container is driven by a YCSB client Record Count: 100M records, 100GB in each DB Client thread count: 16 Workloads Workload A, 50% read, 50% update, Zipfian distribution Workload D, 95% read, 5% insert, normal distribution 22 Results-Throughput Workload D, directly attached SSD C1 C2 C3 C4 C5 C6 C7 C8 Cgroups 50000 40000 30000 20000 Throughput (ops/sec) Throughput 10000 0 1 2 3 4 5 6 7 8 # of Cassandra Containers Aggregated throughput peaks at 4 containers Cgroups: 6 CPU cores, 6GB memory, 400MB/s bandwidth 23 Strategies for Dividing Resources CPU MEM CPU+MEM BW All Uncontrolled 50000 40000 30000 20000 10000 0 Throughput (ops/sec) Throughput 0 1 2 3 4 5 6 7 8 9 # of Cassandra Containers MEM has the most significant impact on throughput Best strategy for dividing resources using cgroups Assign 6 CPU cores for each container, leave other resource uncontrolled 24 Scaling Containerized Cassandra using NVMf Experiment Setup Cassandra + Docker 10Gbe 40Gbe Application NVMf Target YCSB Clients Server Storage Server 25 Results-Throughput DAS_A NVMf_A DAS_D NVMf_D 4 3 2 RelativeTPS 1 0 1 2 3 4 5 6 7 8 # of Cassandra Instances The throughput of NVMf is within 6% - 12% compared to directly attached SSDs 26 Results-Latency DAS_A NVMf_A DAS_D NVMf_D 8 6 Latency 4 2 Relative 0 1 2 3 4 5 6 7 8 # of Cassandra Instances NVMF incurs only 2% - 15% longer latency than direct attached SSD. 27 Results-CPU Utilization NVMF incurs less than 1.8% CPU Utilization on Target Machine 28 SUMMARY Best Option in Docker for NVMe Drive Performance Overlay FS + XFS + Data Volume Best Strategy for Dividing Resources using Cgroups Control only the CPU resources Scaling Docker Containers on NVMf Throughput: within 6% - 12% vs. DAS Latency: 2% - 15% longer than DAS THANK YOU! [email protected] 29.
Load more

Recommended publications
  • The Kernel Report
    The kernel report (ELC 2012 edition) Jonathan Corbet LWN.net [email protected] The Plan Look at a year's worth of kernel work ...with an eye toward the future Starting off 2011 2.6.37 released - January 4, 2011 11,446 changes, 1,276 developers VFS scalability work (inode_lock removal) Block I/O bandwidth controller PPTP support Basic pNFS support Wakeup sources What have we done since then? Since 2.6.37: Five kernel releases have been made 59,000 changes have been merged 3069 developers have contributed to the kernel 416 companies have supported kernel development February As you can see in these posts, Ralink is sending patches for the upstream rt2x00 driver for their new chipsets, and not just dumping a huge, stand-alone tarball driver on the community, as they have done in the past. This shows a huge willingness to learn how to deal with the kernel community, and they should be strongly encouraged and praised for this major change in attitude. – Greg Kroah-Hartman, February 9 Employer contributions 2.6.38-3.2 Volunteers 13.9% Wolfson Micro 1.7% Red Hat 10.9% Samsung 1.6% Intel 7.3% Google 1.6% unknown 6.9% Oracle 1.5% Novell 4.0% Microsoft 1.4% IBM 3.6% AMD 1.3% TI 3.4% Freescale 1.3% Broadcom 3.1% Fujitsu 1.1% consultants 2.2% Atheros 1.1% Nokia 1.8% Wind River 1.0% Also in February Red Hat stops releasing individual kernel patches March 2.6.38 released – March 14, 2011 (9,577 changes from 1198 developers) Per-session group scheduling dcache scalability patch set Transmit packet steering Transparent huge pages Hierarchical block I/O bandwidth controller Somebody needs to get a grip in the ARM community.
    [Show full text]
  • Storage Administration Guide Storage Administration Guide SUSE Linux Enterprise Server 12 SP4
    SUSE Linux Enterprise Server 12 SP4 Storage Administration Guide Storage Administration Guide SUSE Linux Enterprise Server 12 SP4 Provides information about how to manage storage devices on a SUSE Linux Enterprise Server. Publication Date: September 24, 2021 SUSE LLC 1800 South Novell Place Provo, UT 84606 USA https://documentation.suse.com Copyright © 2006– 2021 SUSE LLC and contributors. All rights reserved. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”. For SUSE trademarks, see https://www.suse.com/company/legal/ . All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its aliates. Asterisks (*) denote third-party trademarks. All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its aliates, the authors nor the translators shall be held liable for possible errors or the consequences thereof. Contents About This Guide xii 1 Available Documentation xii 2 Giving Feedback xiv 3 Documentation Conventions xiv 4 Product Life Cycle and Support xvi Support Statement for SUSE Linux Enterprise Server xvii • Technology Previews xviii I FILE SYSTEMS AND MOUNTING 1 1 Overview
    [Show full text]
  • Lecture 9: Data Storage and IO Models Lecture 9
    Lecture 9 Lecture 9: Data Storage and IO Models Lecture 9 Announcements • Submission Project Part 1 tonight • Instructions on Piazza! • PS2 due on Friday at 11:59 pm • Questions? Easier than PS1. • Badgers Rule! Lecture 9 Today’s Lecture 1. Data Storage 2. Disk and Files 3. Buffer Manager - Prelims 3 Lecture 9 > Section 1 1. Data Storage 4 Lecture 9 > Section 1 What you will learn about in this section 1. Life cycle of a query 2. Architecture of a DBMS 3. Memory Hierarchy 5 Lecture 9 > Section 1 Life cycle of a query Query Result Query Database Server Query Execute Parser Optimizer Select R.text from |…|……|………..|………..| Report R, Weather W |…|……|………..|………..| where W.image.rain() Scheduler Operators |…|……|………..|………..| and W.city = R.city |…|……|………..|………..| and W.date = R.date |…|……|………..|………..| and |…|……|………..|………..| R.text. |…|……|………..|………..| matches(“insurance claims”) |…|……|………..|………..| |…|……|………..|………..| |…|……|………..|………..| |…|……|………..|………..| Query Query Syntax Tree Query Plan Result Segments 6 Lecture 9 > Section 1 > Architecture of a DBMS Internal Architecture of a DBMS query Query Execution data access Storage Manager I/O access 7 Lecture 9 > Section 1 > Storage Manager Architecture of a Storage Manager Access Methods Sorted File Hash Index B+-tree Heap File Index Manager Buffer Manager Recovery Manager Concurrency Control I/O Manager IO Accesses In Systems, IO cost matters a ton! 8 Lecture 9 > Section 1 > Data Storage Data Storage • How does a DBMS store and access data? • main memory (fast, temporary) • disk (slow, permanent) • How
    [Show full text]
  • Rootless Containers with Podman and Fuse-Overlayfs
    CernVM Workshop 2019 (4th June 2019) Rootless containers with Podman and fuse-overlayfs Giuseppe Scrivano @gscrivano Introduction 2 Rootless Containers • “Rootless containers refers to the ability for an unprivileged user (i.e. non-root user) to create, run and otherwise manage containers.” (https://rootlesscontaine.rs/ ) • Not just about running the container payload as an unprivileged user • Container runtime runs also as an unprivileged user 3 Don’t confuse with... • sudo podman run --user foo – Executes the process in the container as non-root – Podman and the OCI runtime still running as root • USER instruction in Dockerfile – same as above – Notably you can’t RUN dnf install ... 4 Don’t confuse with... • podman run --uidmap – Execute containers as a non-root user, using user namespaces – Most similar to rootless containers, but still requires podman and runc to run as root 5 Motivation of Rootless Containers • To mitigate potential vulnerability of container runtimes • To allow users of shared machines (e.g. HPC) to run containers without the risk of breaking other users environments • To isolate nested containers 6 Caveat: Not a panacea • Although rootless containers could mitigate these vulnerabilities, it is not a panacea , especially it is powerless against kernel (and hardware) vulnerabilities – CVE 2013-1858, CVE-2015-1328, CVE-2018-18955 • Castle approach : it should be used in conjunction with other security layers such as seccomp and SELinux 7 Podman 8 Rootless Podman Podman is a daemon-less alternative to Docker • $ alias
    [Show full text]
  • System Calls System Calls
    System calls We will investigate several issues related to system calls. Read chapter 12 of the book Linux system call categories file management process management error handling note that these categories are loosely defined and much is behind included, e.g. communication. Why? 1 System calls File management system call hierarchy you may not see some topics as part of “file management”, e.g., sockets 2 System calls Process management system call hierarchy 3 System calls Error handling hierarchy 4 Error Handling Anything can fail! System calls are no exception Try to read a file that does not exist! Error number: errno every process contains a global variable errno errno is set to 0 when process is created when error occurs errno is set to a specific code associated with the error cause trying to open file that does not exist sets errno to 2 5 Error Handling error constants are defined in errno.h here are the first few of errno.h on OS X 10.6.4 #define EPERM 1 /* Operation not permitted */ #define ENOENT 2 /* No such file or directory */ #define ESRCH 3 /* No such process */ #define EINTR 4 /* Interrupted system call */ #define EIO 5 /* Input/output error */ #define ENXIO 6 /* Device not configured */ #define E2BIG 7 /* Argument list too long */ #define ENOEXEC 8 /* Exec format error */ #define EBADF 9 /* Bad file descriptor */ #define ECHILD 10 /* No child processes */ #define EDEADLK 11 /* Resource deadlock avoided */ 6 Error Handling common mistake for displaying errno from Linux errno man page: 7 Error Handling Description of the perror () system call.
    [Show full text]
  • Serverless Network File Systems
    Serverless Network File Systems Thomas E. Anderson, Michael D. Dahlin, Jeanna M. Neefe, David A. Patterson, Drew S. Roselli, and Randolph Y. Wang Computer Science Division University of California at Berkeley Abstract In this paper, we propose a new paradigm for network file system design, serverless network file systems. While traditional network file systems rely on a central server machine, a serverless system utilizes workstations cooperating as peers to provide all file system services. Any machine in the system can store, cache, or control any block of data. Our approach uses this location independence, in combination with fast local area networks, to provide better performance and scalability than traditional file systems. Further, because any machine in the system can assume the responsibilities of a failed component, our serverless design also provides high availability via redundant data storage. To demonstrate our approach, we have implemented a prototype serverless network file system called xFS. Preliminary performance measurements suggest that our architecture achieves its goal of scalability. For instance, in a 32-node xFS system with 32 active clients, each client receives nearly as much read or write throughput as it would see if it were the only active client. 1. Introduction A serverless network file system distributes storage, cache, and control over cooperating workstations. This approach contrasts with traditional file systems such as Netware [Majo94], NFS [Sand85], Andrew [Howa88], and Sprite [Nels88] where a central server machine stores all data and satisfies all client cache misses. Such a central server is both a performance and reliability bottleneck. A serverless system, on the other hand, distributes control processing and data storage to achieve scalable high performance, migrates the responsibilities of failed components to the remaining machines to provide high availability, and scales gracefully to simplify system management.
    [Show full text]
  • Dm-X: Protecting Volume-Level Integrity for Cloud Volumes and Local
    dm-x: Protecting Volume-level Integrity for Cloud Volumes and Local Block Devices Anrin Chakraborti Bhushan Jain Jan Kasiak Stony Brook University Stony Brook University & Stony Brook University UNC, Chapel Hill Tao Zhang Donald Porter Radu Sion Stony Brook University & Stony Brook University & Stony Brook University UNC, Chapel Hill UNC, Chapel Hill ABSTRACT on Amazon’s Virtual Private Cloud (VPC) [2] (which provides The verified boot feature in recent Android devices, which strong security guarantees through network isolation) store deploys dm-verity, has been overwhelmingly successful in elim- data on untrusted storage devices residing in the public cloud. inating the extremely popular Android smart phone rooting For example, Amazon VPC file systems, object stores, and movement [25]. Unfortunately, dm-verity integrity guarantees databases reside on virtual block devices in the Amazon are read-only and do not extend to writable volumes. Elastic Block Storage (EBS). This paper introduces a new device mapper, dm-x, that This allows numerous attack vectors for a malicious cloud efficiently (fast) and reliably (metadata journaling) assures provider/untrusted software running on the server. For in- volume-level integrity for entire, writable volumes. In a direct stance, to ensure SEC-mandated assurances of end-to-end disk setup, dm-x overheads are around 6-35% over ext4 on the integrity, banks need to guarantee that the external cloud raw volume while offering additional integrity guarantees. For storage service is unable to remove log entries documenting cloud storage (Amazon EBS), dm-x overheads are negligible. financial transactions. Yet, without integrity of the storage device, a malicious cloud storage service could remove logs of a coordinated attack.
    [Show full text]
  • CS 5600 Computer Systems
    CS 5600 Computer Systems Lecture 10: File Systems What are We Doing Today? • Last week we talked extensively about hard drives and SSDs – How they work – Performance characterisEcs • This week is all about managing storage – Disks/SSDs offer a blank slate of empty blocks – How do we store files on these devices, and keep track of them? – How do we maintain high performance? – How do we maintain consistency in the face of random crashes? 2 • ParEEons and MounEng • Basics (FAT) • inodes and Blocks (ext) • Block Groups (ext2) • Journaling (ext3) • Extents and B-Trees (ext4) • Log-based File Systems 3 Building the Root File System • One of the first tasks of an OS during bootup is to build the root file system 1. Locate all bootable media – Internal and external hard disks – SSDs – Floppy disks, CDs, DVDs, USB scks 2. Locate all the parEEons on each media – Read MBR(s), extended parEEon tables, etc. 3. Mount one or more parEEons – Makes the file system(s) available for access 4 The Master Boot Record Address Size Descripon Hex Dec. (Bytes) Includes the starEng 0x000 0 Bootstrap code area 446 LBA and length of 0x1BE 446 ParEEon Entry #1 16 the parEEon 0x1CE 462 ParEEon Entry #2 16 0x1DE 478 ParEEon Entry #3 16 0x1EE 494 ParEEon Entry #4 16 0x1FE 510 Magic Number 2 Total: 512 ParEEon 1 ParEEon 2 ParEEon 3 ParEEon 4 MBR (ext3) (swap) (NTFS) (FAT32) Disk 1 ParEEon 1 MBR (NTFS) 5 Disk 2 Extended ParEEons • In some cases, you may want >4 parEEons • Modern OSes support extended parEEons Logical Logical ParEEon 1 ParEEon 2 Ext.
    [Show full text]
  • Ext4 File System and Crash Consistency
    1 Ext4 file system and crash consistency Changwoo Min 2 Summary of last lectures • Tools: building, exploring, and debugging Linux kernel • Core kernel infrastructure • Process management & scheduling • Interrupt & interrupt handler • Kernel synchronization • Memory management • Virtual file system • Page cache and page fault 3 Today: ext4 file system and crash consistency • File system in Linux kernel • Design considerations of a file system • History of file system • On-disk structure of Ext4 • File operations • Crash consistency 4 File system in Linux kernel User space application (ex: cp) User-space Syscalls: open, read, write, etc. Kernel-space VFS: Virtual File System Filesystems ext4 FAT32 JFFS2 Block layer Hardware Embedded Hard disk USB drive flash 5 What is a file system fundamentally? int main(int argc, char *argv[]) { int fd; char buffer[4096]; struct stat_buf; DIR *dir; struct dirent *entry; /* 1. Path name -> inode mapping */ fd = open("/home/lkp/hello.c" , O_RDONLY); /* 2. File offset -> disk block address mapping */ pread(fd, buffer, sizeof(buffer), 0); /* 3. File meta data operation */ fstat(fd, &stat_buf); printf("file size = %d\n", stat_buf.st_size); /* 4. Directory operation */ dir = opendir("/home"); entry = readdir(dir); printf("dir = %s\n", entry->d_name); return 0; } 6 Why do we care EXT4 file system? • Most widely-deployed file system • Default file system of major Linux distributions • File system used in Google data center • Default file system of Android kernel • Follows the traditional file system design 7 History of file system design 8 UFS (Unix File System) • The original UNIX file system • Design by Dennis Ritche and Ken Thompson (1974) • The first Linux file system (ext) and Minix FS has a similar layout 9 UFS (Unix File System) • Performance problem of UFS (and the first Linux file system) • Especially, long seek time between an inode and data block 10 FFS (Fast File System) • The file system of BSD UNIX • Designed by Marshall Kirk McKusick, et al.
    [Show full text]
  • How to Create a Custom Live CD for Secure Remote Incident Handling in the Enterprise
    How to Create a Custom Live CD for Secure Remote Incident Handling in the Enterprise Abstract This paper will document a process to create a custom Live CD for secure remote incident handling on Windows and Linux systems. The process will include how to configure SSH for remote access to the Live CD even when running behind a NAT device. The combination of customization and secure remote access will make this process valuable to incident handlers working in enterprise environments with limited remote IT support. Bert Hayes, [email protected] How to Create a Custom Live CD for Remote Incident Handling 2 Table of Contents Abstract ...........................................................................................................................................1 1. Introduction ............................................................................................................................5 2. Making Your Own Customized Debian GNU/Linux Based System........................................7 2.1. The Development Environment ......................................................................................7 2.2. Making Your Dream Incident Handling System...............................................................9 2.3. Hardening the Base Install.............................................................................................11 2.3.1. Managing Root Access with Sudo..........................................................................11 2.4. Randomizing the Handler Password at Boot Time ........................................................12
    [Show full text]
  • File Handling in Python
    hapter C File Handling in 2 Python There are many ways of trying to understand programs. People often rely too much on one way, which is called "debugging" and consists of running a partly- understood program to see if it does what you expected. Another way, which ML advocates, is to install some means of understanding in the very programs themselves. — Robin Milner In this Chapter » Introduction to Files » Types of Files » Opening and Closing a 2.1 INTRODUCTION TO FILES Text File We have so far created programs in Python that » Writing to a Text File accept the input, manipulate it and display the » Reading from a Text File output. But that output is available only during » Setting Offsets in a File execution of the program and input is to be entered through the keyboard. This is because the » Creating and Traversing a variables used in a program have a lifetime that Text File lasts till the time the program is under execution. » The Pickle Module What if we want to store the data that were input as well as the generated output permanently so that we can reuse it later? Usually, organisations would want to permanently store information about employees, inventory, sales, etc. to avoid repetitive tasks of entering the same data. Hence, data are stored permanently on secondary storage devices for reusability. We store Python programs written in script mode with a .py extension. Each program is stored on the secondary device as a file. Likewise, the data entered, and the output can be stored permanently into a file.
    [Show full text]
  • In Search of the Ideal Storage Configuration for Docker Containers
    In Search of the Ideal Storage Configuration for Docker Containers Vasily Tarasov1, Lukas Rupprecht1, Dimitris Skourtis1, Amit Warke1, Dean Hildebrand1 Mohamed Mohamed1, Nagapramod Mandagere1, Wenji Li2, Raju Rangaswami3, Ming Zhao2 1IBM Research—Almaden 2Arizona State University 3Florida International University Abstract—Containers are a widely successful technology today every running container. This would cause a great burden on popularized by Docker. Containers improve system utilization by the I/O subsystem and make container start time unacceptably increasing workload density. Docker containers enable seamless high for many workloads. As a result, copy-on-write (CoW) deployment of workloads across development, test, and produc- tion environments. Docker’s unique approach to data manage- storage and storage snapshots are popularly used and images ment, which involves frequent snapshot creation and removal, are structured in layers. A layer consists of a set of files and presents a new set of exciting challenges for storage systems. At layers with the same content can be shared across images, the same time, storage management for Docker containers has reducing the amount of storage required to run containers. remained largely unexplored with a dizzying array of solution With Docker, one can choose Aufs [6], Overlay2 [7], choices and configuration options. In this paper we unravel the multi-faceted nature of Docker storage and demonstrate its Btrfs [8], or device-mapper (dm) [9] as storage drivers which impact on system and workload performance. As we uncover provide the required snapshotting and CoW capabilities for new properties of the popular Docker storage drivers, this is a images. None of these solutions, however, were designed with sobering reminder that widespread use of new technologies can Docker in mind and their effectiveness for Docker has not been often precede their careful evaluation.
    [Show full text]