DOCSLIB.ORG

Verifying a High-Performance Crash-Safe File System Using a Tree Specification

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Verifying a High-Performance Crash-Safe File System Using a Tree Specification Verifying a high-performance crash-safe file system using a tree specification Haogang Chen,y Tej Chajed, Alex Konradi,z Stephanie Wang,x Atalay İleri, Adam Chlipala, M. Frans Kaashoek, Nickolai Zeldovich MIT CSAIL ABSTRACT 1 INTRODUCTION File systems achieve high I/O performance and crash safety DFSCQ is the first file system that (1) provides a precise by implementing sophisticated optimizations to increase disk fsync fdatasync specification for and , which allow appli- throughput. These optimizations include deferring writing cations to achieve high performance and crash safety, and buffered data to persistent storage, grouping many trans- (2) provides a machine-checked proof that its implementa- actions into a single I/O operation, checksumming journal tion meets this specification. DFSCQ’s specification captures entries, and bypassing the write-ahead log when writing to the behavior of sophisticated optimizations, including log- file data blocks. The widely used Linux ext4 is an example bypass writes, and DFSCQ’s proof rules out some of the of an I/O-efficient file system; the above optimizations allow common bugs in file-system implementations despite the it to batch many writes into a single I/O operation and to complex optimizations. reduce the number of disk-write barriers that flush data to The key challenge in building DFSCQ is to write a speci- disk [33, 56]. Unfortunately, these optimizations complicate fication for the file system and its internal implementation a file system’s implementation. For example, it took 6 years without exposing internal file-system details. DFSCQ in- for ext4 developers to realize that two optimizations (data troduces a metadata-prefix specification that captures the writes that bypass the journal and journal checksumming) properties of fsync and fdatasync, which roughly follows taken together can lead to disclosure of previously deleted the behavior of Linux ext4. This specification uses a no- data after a crash [30]. This bug was fixed in November of tion of tree sequences—logical sequences of file-system tree 2014 by forbidding users from mounting an ext4 file sys- states—for succinct description of the possible states after tem with both journal bypassing for data writes and journal a crash and to describe how data writes can be reordered checksumming. A comprehensive study of several file sys- with respect to metadata updates. This helps application tems in Linux also found a range of other bugs [34: §6]. developers prove the crash safety of their own applications, avoiding application-level bugs such as forgetting to invoke Somewhat surprisingly, there exists no precise specifi- fsync on both the file and the containing directory. cation that would allow proving the correctness of a high- An evaluation shows that DFSCQ achieves 103 MB/s on performance file system, ruling out bugs like the ones de- large file writes to an SSD and durably creates small files at scribed above. For example, the POSIX standard is noto- a rate of 1,618 files per second. This is slower than Linux riously vague on what crash-safety guarantees file-system ext4 (which achieves 295 MB/s for large file writes and 4,977 operations provide [44]. Of particular concern are the guar- files/s for small file creation) but much faster than twore- antees provided by fsync and fdatasync, which give appli- cent verified file systems, Yggdrasil and FSCQ. Evaluation cations precise control over what data the file system flushes results from application-level benchmarks, including TPC-C to persistent storage. Unfortunately, file systems provide on SQLite, mirror these microbenchmarks. imprecise promises on exactly what data is flushed, and, in fact, for the Linux ext4 file system, it depends on the op- tions that an administrator specifies when mounting the file system [40]. Because of this lack of precision, applications Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies such as databases and mail servers, which try hard to make are not made or distributed for profit or commercial advantage and that sequences of file creates, writes, and renames crash-safe by copies bear this notice and the full citation on the first page. Copyrights for issuing fsyncs and fdatasyncs, may still lose data if the file third-party components of this work must be honored. For all other uses, contact the owner/author(s). system crashes at an inopportune time [40]. For example, SQLite and LightningDB, two popular databases, improperly SOSP’17, October 28–31, 2017, Shanghai, China. used fsync and fdatasync, leading to possible data loss [65]. © 2017 Copyright is held by the owner/author(s). ACM ISBN 978-1-4503-5085-3/17/10. https://doi.org/10.1145/3132747.3132776 y Now at Databricks. z Now at Google. x Now at UC Berkeley. 1 A particularly challenging behavior to specify is log-bypass flushed. This paper uses the notion of tree sequences to cap- writes1. A file system typically uses a write-ahead logto ture the metadata-prefix specification. Abstractly, the state guarantee that changes are flushed to disk atomically and in of the file system is a sequence of trees (directories and files), order. The one exception is data writes: to avoid writing data each corresponding to a metadata update performed by the blocks to disk twice (once to the log and once to the file’s application, in the order the updates were issued; the se- blocks), file systems bypass the log when writing file data. quence captures the metadata order. Data writes apply to Since data blocks are not written to the log, they may be every tree in the sequence, reflecting the fact that they can re-ordered with respect to logged metadata changes, and this be reordered with respect to metadata. After a crash, the file reordering must be precisely captured in the specification. A system can end up in any tree in the sequence. Calling fsync further complication arises from the fact that this reordering logically truncates the sequence of trees to just the last one, can lead to corruption in the presence of block reuse. For and fdatasync makes file data writes durable in every tree example, a bypass write to file f can modify block b on in the sequence. disk, but the metadata update allocating b to f has not been (4) To demonstrate that the metadata-prefix specification flushed to disk yet; b may still be in use by another file or is easy to use, we implemented, specified, and proved the cor- directory, which might be corrupted by the bypass write. rectness of a core pattern used by applications to implement The specification must exclude this possibility. crash-safe updates. This code pattern modifies a destina- This paper makes several contributions: tion file atomically, even if the file system crashes during (1) An approach to specifying file-system behavior purely the update. That is, if a crash happens, either the destina- in terms of abstract file-system trees, as opposed to expos- tion file exists with a copy of all the new bytes that should ing the internal details of file-system optimizations, such as have been written to it, or the destination file is not modi- log-bypass writes directly updating disk blocks. A purely fied. This pattern captures the core goal of many crash-safe tree-based specification shields application developers from applications—ensuring that some sequence of file creates, having to understand and reason about low-level file-system writes, renames, fsyncs, and fdatasyncs is crash-safe. If an implementation details. application’s use of fsync and fdatasync does not ensure (2) A precise metadata-prefix specification that describes crash safety, the application developer is unable to prove the the file system’s behavior in the presence of crashes (in addi- application correct. tion to describing non-crash behavior), including the fsync (5) To demonstrate that the metadata-prefix specification and fdatasync system calls and sophisticated optimizations allows proving the correctness of a high-performance file such as log-bypass writes, using the tree-based specification system, we built DFSCQ (Deferred-write FSCQ), based on approach. Intuitively, the specification says that if an ap- FSCQ [10] and its Crash-Hoare Logic infrastructure.3 DFSCQ plication makes a sequence of metadata changes, the file implements standard optimizations such as deferring writes, system will persist these changes to disk in order, meaning group commit, non-journaled writes for file data, check- that the state after a crash will correspond to a prefix of the summed log commit, and so on. We report the results of application’s changes. On the other hand, data updates need several benchmarks on top of DFSCQ, compared to Linux not be ordered; thus, the name metadata-prefix. Our specifi- ext4 and two recent verified file systems. cation is inspired by the observed behavior of the Linux ext4 DFSCQ has several limitations. First, DFSCQ (and the file system,2 as well as other prior work [11, 35]. metadata-prefix specification) does not capture concurrency; (3) A formalization of this metadata-prefix specification. system calls are executed one at a time. Second, DFSCQ A precise formal specification is important for two reasons. runs as a Haskell program, which incurs a large trusted First, it allows developers of a mission-critical application computing base (the Haskell compiler and runtime) as well such as a database to prove that they are using fsync cor- as CPU performance overhead. Addressing these limitations rectly and that the database doesn’t run the risk of losing remains an interesting open problem for future work. data. Second, it allows file-system developers to prove that the optimizations that they implement will still provide the 2 RELATED WORK same guarantees, embodied in the metadata-prefix specifica- DFSCQ is the first file system with a machine-checked proof tion, that applications may rely on.
Load more

Recommended publications
  • Trusted Docker Containers and Trusted Vms in Openstack
    Trusted Docker Containers and Trusted VMs in OpenStack Raghu Yeluri Abhishek Gupta Outline o Context: Docker Security – Top Customer Asks o Intel’s Focus: Trusted Docker Containers o Who Verifies Trust ? o Reference Architecture with OpenStack o Demo o Availability o Call to Action Docker Overview in a Slide.. Docker Hub Lightweight, open source engine for creating, deploying containers Provides work flow for running, building and containerizing apps. Separates apps from where they run.; Enables Micro-services; scale by composition. Underlying building blocks: Linux kernel's namespaces (isolation) + cgroups (resource control) + .. Components of Docker Docker Engine – Runtime for running, building Docker containers. Docker Repositories(Hub) - SaaS service for sharing/managing images Docker Images (layers) Images hold Apps. Shareable snapshot of software. Container is a running instance of image. Orchestration: OpenStack, Docker Swarm, Kubernetes, Mesos, Fleet, Project Docker Layers Atomic, Lattice… Docker Security – 5 key Customer Asks 1. How do you know that the Docker Host Integrity is there? o Do you trust the Docker daemon? o Do you trust the Docker host has booted with Integrity? 2. How do you verify Docker Container Integrity o Who wrote the Docker image? Do you trust the image? Did the right Image get launched? 3. Runtime Protection of Docker Engine & Enhanced Isolation o How can Intel help with runtime Integrity? 4. Enterprise Security Features – Compliance, Manageability, Identity authentication.. Etc. 5. OpenStack as a single Control Plane for Trusted VMs and Trusted Docker Containers.. Intel’s Focus: Enable Hardware-based Integrity Assurance for Docker Containers – Trusted Docker Containers Trusted Docker Containers – 3 focus areas o Launch Integrity of Docker Host o Runtime Integrity of Docker Host o Integrity of Docker Images Today’s Focus: Integrity of Docker Host, and how to use it in OpenStack.
    [Show full text]
  • Rootless Containers with Podman and Fuse-Overlayfs
    CernVM Workshop 2019 (4th June 2019) Rootless containers with Podman and fuse-overlayfs Giuseppe Scrivano @gscrivano Introduction 2 Rootless Containers • “Rootless containers refers to the ability for an unprivileged user (i.e. non-root user) to create, run and otherwise manage containers.” (https://rootlesscontaine.rs/ ) • Not just about running the container payload as an unprivileged user • Container runtime runs also as an unprivileged user 3 Don’t confuse with... • sudo podman run --user foo – Executes the process in the container as non-root – Podman and the OCI runtime still running as root • USER instruction in Dockerfile – same as above – Notably you can’t RUN dnf install ... 4 Don’t confuse with... • podman run --uidmap – Execute containers as a non-root user, using user namespaces – Most similar to rootless containers, but still requires podman and runc to run as root 5 Motivation of Rootless Containers • To mitigate potential vulnerability of container runtimes • To allow users of shared machines (e.g. HPC) to run containers without the risk of breaking other users environments • To isolate nested containers 6 Caveat: Not a panacea • Although rootless containers could mitigate these vulnerabilities, it is not a panacea , especially it is powerless against kernel (and hardware) vulnerabilities – CVE 2013-1858, CVE-2015-1328, CVE-2018-18955 • Castle approach : it should be used in conjunction with other security layers such as seccomp and SELinux 7 Podman 8 Rootless Podman Podman is a daemon-less alternative to Docker • $ alias
    [Show full text]
  • Providing User Security Guarantees in Public Infrastructure Clouds
    1 Providing User Security Guarantees in Public Infrastructure Clouds Nicolae Paladi, Christian Gehrmann, and Antonis Michalas Abstract—The infrastructure cloud (IaaS) service model offers improved resource flexibility and availability, where tenants – insulated from the minutiae of hardware maintenance – rent computing resources to deploy and operate complex systems. Large-scale services running on IaaS platforms demonstrate the viability of this model; nevertheless, many organizations operating on sensitive data avoid migrating operations to IaaS platforms due to security concerns. In this paper, we describe a framework for data and operation security in IaaS, consisting of protocols for a trusted launch of virtual machines and domain-based storage protection. We continue with an extensive theoretical analysis with proofs about protocol resistance against attacks in the defined threat model. The protocols allow trust to be established by remotely attesting host platform configuration prior to launching guest virtual machines and ensure confidentiality of data in remote storage, with encryption keys maintained outside of the IaaS domain. Presented experimental results demonstrate the validity and efficiency of the proposed protocols. The framework prototype was implemented on a test bed operating a public electronic health record system, showing that the proposed protocols can be integrated into existing cloud environments. Index Terms—Security; Cloud Computing; Storage Protection; Trusted Computing F 1 INTRODUCTION host level. While support data encryption at rest is offered by several cloud providers and can be configured by tenants Cloud computing has progressed from a bold vision to mas- in their VM instances, functionality and migration capabil- sive deployments in various application domains. However, ities of such solutions are severely restricted.
    [Show full text]
  • 11.7 the Windows 2000 File System
    830 CASE STUDY 2: WINDOWS 2000 CHAP. 11 11.7 THE WINDOWS 2000 FILE SYSTEM Windows 2000 supports several file systems, the most important of which are FAT-16, FAT-32, and NTFS (NT File System). FAT-16 is the old MS-DOS file system. It uses 16-bit disk addresses, which limits it to disk partitions no larger than 2 GB. FAT-32 uses 32-bit disk addresses and supports disk partitions up to 2 TB. NTFS is a new file system developed specifically for Windows NT and car- ried over to Windows 2000. It uses 64-bit disk addresses and can (theoretically) support disk partitions up to 264 bytes, although other considerations limit it to smaller sizes. Windows 2000 also supports read-only file systems for CD-ROMs and DVDs. It is possible (even common) to have the same running system have access to multiple file system types available at the same time. In this chapter we will treat the NTFS file system because it is a modern file system unencumbered by the need to be fully compatible with the MS-DOS file system, which was based on the CP/M file system designed for 8-inch floppy disks more than 20 years ago. Times have changed and 8-inch floppy disks are not quite state of the art any more. Neither are their file systems. Also, NTFS differs both in user interface and implementation in a number of ways from the UNIX file system, which makes it a good second example to study. NTFS is a large and complex system and space limitations prevent us from covering all of its features, but the material presented below should give a reasonable impression of it.
    [Show full text]
  • In Search of the Ideal Storage Configuration for Docker Containers
    In Search of the Ideal Storage Configuration for Docker Containers Vasily Tarasov1, Lukas Rupprecht1, Dimitris Skourtis1, Amit Warke1, Dean Hildebrand1 Mohamed Mohamed1, Nagapramod Mandagere1, Wenji Li2, Raju Rangaswami3, Ming Zhao2 1IBM Research—Almaden 2Arizona State University 3Florida International University Abstract—Containers are a widely successful technology today every running container. This would cause a great burden on popularized by Docker. Containers improve system utilization by the I/O subsystem and make container start time unacceptably increasing workload density. Docker containers enable seamless high for many workloads. As a result, copy-on-write (CoW) deployment of workloads across development, test, and produc- tion environments. Docker’s unique approach to data manage- storage and storage snapshots are popularly used and images ment, which involves frequent snapshot creation and removal, are structured in layers. A layer consists of a set of files and presents a new set of exciting challenges for storage systems. At layers with the same content can be shared across images, the same time, storage management for Docker containers has reducing the amount of storage required to run containers. remained largely unexplored with a dizzying array of solution With Docker, one can choose Aufs [6], Overlay2 [7], choices and configuration options. In this paper we unravel the multi-faceted nature of Docker storage and demonstrate its Btrfs [8], or device-mapper (dm) [9] as storage drivers which impact on system and workload performance. As we uncover provide the required snapshotting and CoW capabilities for new properties of the popular Docker storage drivers, this is a images. None of these solutions, however, were designed with sobering reminder that widespread use of new technologies can Docker in mind and their effectiveness for Docker has not been often precede their careful evaluation.
    [Show full text]
  • MINCS - the Container in the Shell (Script)
    MINCS - The Container in the Shell (script) - Masami Hiramatsu <[email protected]> Tech Lead, Linaro Ltd. Open Source Summit Japan 2017 LEADING COLLABORATION IN THE ARM ECOSYSTEM Who am I... Masami Hiramatsu - Linux kernel kprobes maintainer - Working for Linaro as a Tech Lead LEADING COLLABORATION IN THE ARM ECOSYSTEM Demo # minc top # minc -r /opt/debian/x86_64 # minc -r /opt/debian/arm64 --arch arm64 LEADING COLLABORATION IN THE ARM ECOSYSTEM What Is MINCS? My Personal Fun Project to learn how linux containers work :-) LEADING COLLABORATION IN THE ARM ECOSYSTEM What Is MINCS? Mini Container Shell Scripts (pronounced ‘minks’) - Container engine implementation using POSIX shell scripts - It is small (~60KB, ~2KLOC) (~20KB in minimum) - It can run on busybox - No architecture dependency (* except for qemu/um mode) - No need for special binaries (* except for libcap, just for capsh --exec) - Main Features - Namespaces (Mount, PID, User, UTS, Net*) - Cgroups (CPU, Memory) - Capabilities - Overlay filesystem - Qemu cross-arch/system emulation - User-mode-linux - Image importing from dockerhub And all are done by CLI commands :-) LEADING COLLABORATION IN THE ARM ECOSYSTEM Why Shell Script? That is my favorite language :-) - Easy to understand for *nix administrators - Just a bunch of commands - Easy to modify - Good for prototyping - Easy to deploy - No architecture dependencies - Very small - Able to run on busybox (+ libcap is perfect) LEADING COLLABORATION IN THE ARM ECOSYSTEM MINCS Use-Cases For Learning - Understand how containers work For Development - Prepare isolated (cross-)build environment For Testing - Test new applications in isolated environment - Test new kernel features on qemu using local tools For products? - Maybe good for embedded devices which has small resources LEADING COLLABORATION IN THE ARM ECOSYSTEM What Is A Linux Container? There are many linux container engines - Docker, LXC, rkt, runc, ..
    [Show full text]
  • “Application - File System” Divide with Promises
    Bridging the “Application - File System” divide with promises Raja Bala Computer Sciences Department University of Wisconsin, Madison, WI [email protected] Abstract that hook into the file system and the belief that the underlying file system is the best judge File systems today implement a limited set of when it comes to operations with files. Unfor- abstractions and semantics wherein applications tunately, the latter isn’t true, since applications don’t really have much of a say. The generality know more about their behavior and what they of these abstractions tends to curb the application need or do not need from the file system. Cur- performance. In the global world we live in, it seems rently, there is no real mechanism that allows reasonable that applications are treated as first-class the applications to communicate this informa- citizens by the file system layer. tion to the file system and thus have some degree In this project, we take a first step towards that goal of control over the file system functionality. by leveraging promises that applications make to the file system. The promises are then utilized to deliver For example, an application that never ap- a better-tuned and more application-oriented file pends to any of the files it creates has no means system. A very simple promise, called unique-create of conveying this information to the file sys- was implemented, wherein the application vows tem. Most file systems inherently assume that never to create a file with an existing name (in a it is good to preallocate extra blocks to a file, directory) which is then used by the file system so that when it expands, the preallocated blocks to speedup creation time.
    [Show full text]
  • NOVA: a Log-Structured File System for Hybrid Volatile/Non
    NOVA: A Log-structured File System for Hybrid Volatile/Non-volatile Main Memories Jian Xu and Steven Swanson, University of California, San Diego https://www.usenix.org/conference/fast16/technical-sessions/presentation/xu This paper is included in the Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST ’16). February 22–25, 2016 • Santa Clara, CA, USA ISBN 978-1-931971-28-7 Open access to the Proceedings of the 14th USENIX Conference on File and Storage Technologies is sponsored by USENIX NOVA: A Log-structured File System for Hybrid Volatile/Non-volatile Main Memories Jian Xu Steven Swanson University of California, San Diego Abstract Hybrid DRAM/NVMM storage systems present a host of opportunities and challenges for system designers. These sys- Fast non-volatile memories (NVMs) will soon appear on tems need to minimize software overhead if they are to fully the processor memory bus alongside DRAM. The result- exploit NVMM’s high performance and efficiently support ing hybrid memory systems will provide software with sub- more flexible access patterns, and at the same time they must microsecond, high-bandwidth access to persistent data, but provide the strong consistency guarantees that applications managing, accessing, and maintaining consistency for data require and respect the limitations of emerging memories stored in NVM raises a host of challenges. Existing file sys- (e.g., limited program cycles). tems built for spinning or solid-state disks introduce software Conventional file systems are not suitable for hybrid mem- overheads that would obscure the performance that NVMs ory systems because they are built for the performance char- should provide, but proposed file systems for NVMs either in- acteristics of disks (spinning or solid state) and rely on disks’ cur similar overheads or fail to provide the strong consistency consistency guarantees (e.g., that sector updates are atomic) guarantees that applications require.
    [Show full text]
  • High Velocity Kernel File Systems with Bento
    High Velocity Kernel File Systems with Bento Samantha Miller, Kaiyuan Zhang, Mengqi Chen, and Ryan Jennings, University of Washington; Ang Chen, Rice University; Danyang Zhuo, Duke University; Thomas Anderson, University of Washington https://www.usenix.org/conference/fast21/presentation/miller This paper is included in the Proceedings of the 19th USENIX Conference on File and Storage Technologies. February 23–25, 2021 978-1-939133-20-5 Open access to the Proceedings of the 19th USENIX Conference on File and Storage Technologies is sponsored by USENIX. High Velocity Kernel File Systems with Bento Samantha Miller Kaiyuan Zhang Mengqi Chen Ryan Jennings Ang Chen‡ Danyang Zhuo† Thomas Anderson University of Washington †Duke University ‡Rice University Abstract kernel-level debuggers and kernel testing frameworks makes this worse. The restricted and different kernel programming High development velocity is critical for modern systems. environment also limits the number of trained developers. This is especially true for Linux file systems which are seeing Finally, upgrading a kernel module requires either rebooting increased pressure from new storage devices and new demands the machine or restarting the relevant module, either way on storage systems. However, high velocity Linux kernel rendering the machine unavailable during the upgrade. In the development is challenging due to the ease of introducing cloud setting, this forces kernel upgrades to be batched to meet bugs, the difficulty of testing and debugging, and the lack of cloud-level availability goals. support for redeployment without service disruption. Existing Slow development cycles are a particular problem for file approaches to high-velocity development of file systems for systems.
    [Show full text]
  • Singularityce User Guide Release 3.8
    SingularityCE User Guide Release 3.8 SingularityCE Project Contributors Aug 16, 2021 CONTENTS 1 Getting Started & Background Information3 1.1 Introduction to SingularityCE......................................3 1.2 Quick Start................................................5 1.3 Security in SingularityCE........................................ 15 2 Building Containers 19 2.1 Build a Container............................................. 19 2.2 Definition Files.............................................. 24 2.3 Build Environment............................................ 35 2.4 Support for Docker and OCI....................................... 39 2.5 Fakeroot feature............................................. 79 3 Signing & Encryption 83 3.1 Signing and Verifying Containers.................................... 83 3.2 Key commands.............................................. 88 3.3 Encrypted Containers.......................................... 90 4 Sharing & Online Services 95 4.1 Remote Endpoints............................................ 95 4.2 Cloud Library.............................................. 103 5 Advanced Usage 109 5.1 Bind Paths and Mounts.......................................... 109 5.2 Persistent Overlays............................................ 115 5.3 Running Services............................................. 118 5.4 Environment and Metadata........................................ 129 5.5 OCI Runtime Support.......................................... 140 5.6 Plugins.................................................
    [Show full text]
  • Firecracker: Lightweight Virtualization for Serverless Applications
    Firecracker: Lightweight Virtualization for Serverless Applications Alexandru Agache, Marc Brooker, Andreea Florescu, Alexandra Iordache, Anthony Liguori, Rolf Neugebauer, Phil Piwonka, and Diana-Maria Popa, Amazon Web Services https://www.usenix.org/conference/nsdi20/presentation/agache This paper is included in the Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’20) February 25–27, 2020 • Santa Clara, CA, USA 978-1-939133-13-7 Open access to the Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’20) is sponsored by Firecracker: Lightweight Virtualization for Serverless Applications Alexandru Agache Marc Brooker Andreea Florescu Amazon Web Services Amazon Web Services Amazon Web Services Alexandra Iordache Anthony Liguori Rolf Neugebauer Amazon Web Services Amazon Web Services Amazon Web Services Phil Piwonka Diana-Maria Popa Amazon Web Services Amazon Web Services Abstract vantage over traditional server provisioning processes: mul- titenancy allows servers to be shared across a large num- Serverless containers and functions are widely used for de- ber of workloads, and the ability to provision new func- ploying and managing software in the cloud. Their popularity tions and containers in milliseconds allows capacity to be is due to reduced cost of operations, improved utilization of switched between workloads quickly as demand changes. hardware, and faster scaling than traditional deployment meth- Serverless is also attracting the attention of the research com- ods. The economics and scale of serverless applications de- munity [21,26,27,44,47], including work on scaling out video mand that workloads from multiple customers run on the same encoding [13], linear algebra [20, 53] and parallel compila- hardware with minimal overhead, while preserving strong se- tion [12].
    [Show full text]
  • Container-Based Virtualization for Byte-Addressable NVM Data Storage
    2016 IEEE International Conference on Big Data (Big Data) Container-Based Virtualization for Byte-Addressable NVM Data Storage Ellis R. Giles Rice University Houston, Texas [email protected] Abstract—Container based virtualization is rapidly growing Storage Class Memory, or SCM, is an exciting new in popularity for cloud deployments and applications as a memory technology with the potential of replacing hard virtualization alternative due to the ease of deployment cou- drives and SSDs as it offers high-speed, byte-addressable pled with high-performance. Emerging byte-addressable, non- volatile memories, commonly called Storage Class Memory or persistence on the main memory bus. Several technologies SCM, technologies are promising both byte-addressability and are currently under research and development, each with dif- persistence near DRAM speeds operating on the main memory ferent performance, durability, and capacity characteristics. bus. These new memory alternatives open up a new realm of These include a ReRAM by Micron and Sony, a slower, but applications that no longer have to rely on slow, block-based very large capacity Phase Change Memory or PCM by Mi- persistence, but can rather operate directly on persistent data using ordinary loads and stores through the cache hierarchy cron and others, and a fast, smaller spin-torque ST-MRAM coupled with transaction techniques. by Everspin. High-speed, byte-addressable persistence will However, SCM presents a new challenge for container-based give rise to new applications that no longer have to rely on applications, which typically access persistent data through slow, block based storage devices and to serialize data for layers of block based file isolation.
    [Show full text]