DOCSLIB.ORG

The Design, Implementation, and Deployment of a System

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
The Design, Implementation, and Deployment of a System The Design, Implementation, and Deployment of a System to Transparently Compress Hundreds of Petabytes of Image Files for a File-Storage Service Daniel Reiter Horn, Ken Elkabany, and Chris Lesniewski-Lass, Dropbox; Keith Winstein, Stanford University https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/horn This paper is included in the Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI ’17). March 27–29, 2017 • Boston, MA, USA ISBN 978-1-931971-37-9 Open access to the Proceedings of the 14th USENIX Symposium on Networked Systems Design and Implementation is sponsored by USENIX. The Design, Implementation, and Deployment of a System to Transparently Compress Hundreds of Petabytes of Image Files For a File-Storage Service Daniel Reiter Horn Ken Elkabany Chris Lesniewski-Laas Keith Winstein Dropbox Dropbox Dropbox Stanford University Abstract 200 150 We report the design, implementation, and deployment JPEGrescan (progressive) Lepton of Lepton, a fault-tolerant system that losslessly com- (this work) presses JPEG images to 77% of their original size on av- 100 erage. Lepton replaces the lowest layer of baseline JPEG compression—a Huffman code—with a parallelized arith- MozJPEG metic code, so that the exact bytes of the original JPEG 50 (arithmetic) file can be recovered quickly. Lepton matches the com- 40 pression efficiency of the best prior work, while decoding 30 more than nine times faster and in a streaming manner. Decompression speed (Mbits/s) Better Lepton has been released as open-source software and 20 packjpg has been deployed for a year on the Dropbox file-storage backend. As of February 2017, it had compressed more 15 6 7 8 9 10 15 20 25 than 203 PiB of user JPEG files, saving more than 46 PiB. Compression savings (percent) Figure 1: Compression savings and decompression speed (time- 1 Introduction to-last-byte) of four lossless JPEG compression tools. Diamonds In the last decade, centrally hosted network filesystems show 25th, 50th, and 75th percentiles across 200,000 JPEGs have grown to serve hundreds of millions of users. These between 100 KiB and 4 MiB (x 4). services include Amazon Cloud Drive, Box, Dropbox, compresses an average JPEG image by about 23%. This Google Drive, Microsoft OneDrive, and SugarSync. is expected to save Dropbox more than 8% of its overall Commercially, these systems typically offer users a backend file storage. Lepton introduces new techniques storage quota in exchange for a flat monthly fee, or no that allow it to match the compression savings of prior fee at all. Meanwhile, the cost to operate such a system work (PackJPG) while decoding more than nine times increases with the amount of data actually stored. There- faster and in a streaming manner (Figure 1). fore, operators benefit from anything that reduces the net amount of data they store. Some of the challenges in building Lepton included: These filesystems have become gargantuan. After less • Round-trip transparency. Lepton needs to deter- than ten years in operation, the Dropbox system contains ministically recover the exact bytes of the original roughly one exabyte (± 50%) of data, even after applying file, even for intentionally malformed files, and even techniques such as deduplication and zlib compression. after updates to the Lepton software since the file We report on our experience with a different technique: was originally compressed. format-specific transparent file compression, based on a • Distribution across independent chunks. The statistical model tuned to perform well on a large corpus. Dropbox back-end stores files in independent 4-MiB In operating Dropbox, we observed that JPEG im- chunks across many servers. Lepton must be able ages [10] make up roughly 35% of the bytes stored. We to decompress any substring of a JPEG file, without suspected that most of these images were compressed in- access to other substrings. efficiently, either because they were limited to “baseline” • Low latency and streaming. Because users are methods that were royalty-free in the 1990s, or because sensitive to file download latency, we must opti- they were encoded by fixed-function compression chips. mize decompression—from Lepton’s format back In response, we built a compression tool, called Lepton, into Huffman-coded JPEG output—for both time-to- that replaces the lowest layer of baseline JPEG images— first-byte and time-to-last-byte. To achieve this, the the lossless Huffman coding—with a custom statistical Lepton format includes “Huffman handover words” model that we tuned to perform well across a broad corpus that enable the decoder to be multithreaded and to of JPEG images stored in Dropbox. Lepton losslessly start transmitting bytes soon after a request. USENIX Association 14th USENIX Symposium on Networked Systems Design and Implementation 1 • Security. Before reading input data, Lepton enters a efficient compression without affecting the decoded restricted environment where the only allowed sys- image. These tools include JPEGrescan [13] and tem calls are read, write, exit, and sigreturn. MozJPEG [2], both based on the jpegtran tool written Lepton must pre-spawn threads and allocate all mem- by the Independent JPEG Group [9]. The tools employ ory before it sees any input. two key techniques: replacing Huffman coding with arith- • Memory. To preserve server resources, Lepton must metic coding (which is more efficient but was patent- work row-by-row on a JPEG file, instead of decoding encumbered when JPEG was formalized and is not part the entire file into RAM. of the baseline JPEG standard), and rewriting the file We deployed Lepton in April 2016 on the production in “progressive” order, which can group similar values Dropbox service. Compression is applied immediately together and result in more efficient coding. Xu et al. de- to all uploads of new JPEG files. We are gradually com- veloped a related algorithm which uses a large number of pressing files in existing storage and have ramped up to context-dependent Huffman tables to encode JPEG im- a rate of roughly a petabyte per day of input, consuming ages, reporting average compression savings of 15% in about 300 kilowatts continuously. As of February 2017, 30-50ms [22]. These tools preserve the exact pixels of Lepton had been run on more than 150 billion user files, the decoded image, but do not allow bit-exact round-trip accounting for more than 203 PiB of input. It reduced recovery of the original file. their size by a total of more than 46 PiB. We have released Format-aware, file-preserving recompression. A fi- Lepton as open-source software [1]. nal set of tools can re-compress a JPEG file with round- We report here on Lepton’s design (x 3), evaluation trip recovery of the exact bytes of the original file. These (x 4), and production deployment (x 5), and share a num- include PackJPG [17, 11] and PAQ8PX [12]. These tools ber of case studies and anomalies (x 6) encountered in use different techniques, but in our evaluations, achieved operating the system at a large scale. roughly the same compression savings (23%) on average. 2 Related Work These tools use techniques unavailable in a real-time setting. For example, one of PackJPG’s compression tech- In network filesystems and storage services, four general niques requires re-arranging all of the compressed pixel categories of approaches are used to compress user files. values in the file in a globally sorted order. This means Generic entropy compression. Many filesystems com- that decompression is single-threaded, requires access to press files using generic techniques, such as the Deflate the entire file, and requires decoding the image into RAM algorithm [6], which combines LZ77 compression [23] before any byte can be output. The time-to-first-byte and and Huffman coding [8] and is used by software such as time-to-last-byte are too high to satisfy the service goals zlib, pkzip, and gzip. More recent algorithms such as for the Dropbox service. PAQ8PX is considerably slower. LZMA [15], Brotli [3], and Zstandard [5] achieve a range Lepton was inspired by PackJPG and the algorithms of tradeoffs of compression savings and speed. developed by Lakhani [11], and Lepton uses the same In practice, none of these algorithms achieve much JPEG-parsing routines that PackJPG uses (uncmpjpg). compression on “already compressed” files, including However, unlike PackJPG, Lepton only utilizes compres- JPEGs. In our evaluation on 200,000 randomly selected sion techniques that can be implemented without “global” user JPEG images, each of these algorithms achieved operations, so that decoding can be distributed across in- savings of 1% or less (x 4). dependent chunks and multithreaded within each chunk. Lossy image compression. A second category of ap- 3 Lepton: Design and Implementation proach involves decoding user images and re-compressing At its core, Lepton is a stand-alone tool that performs them into a more-efficient format, at the cost of some vi- round-trip compression and decompression of baseline sual fidelity. The “more-efficient format” may simply JPEG files. We have released Lepton as open-source soft- be a lower-resolution or lower-quality JPEG. The JPEG ware [1]; it builds and runs on Linux, MacOS, Windows, files produced by fixed-function hardware encoders in iOS, Android, and Emscripten (JavaScript). cellular-phone cameras can typically be reduced in size In its current deployment at Dropbox, Lepton is exe- by 70% before users see a perceptual difference [19]. The cuted directly by a back-end file server or, when a file re-compression may also use more sophisticated tech- server is under high load, execution is “outsourced” from niques such as WebP or single-frame H.265. With this the file server to a cluster of machines dedicated to Lep- approach, storage savings for the provider can be as high ton only (x 5.5).
Load more

Recommended publications
  • ROOT I/O Compression Improvements for HEP Analysis
    EPJ Web of Conferences 245, 02017 (2020) https://doi.org/10.1051/epjconf/202024502017 CHEP 2019 ROOT I/O compression improvements for HEP analysis Oksana Shadura1;∗ Brian Paul Bockelman2;∗∗ Philippe Canal3;∗∗∗ Danilo Piparo4;∗∗∗∗ and Zhe Zhang1;y 1University of Nebraska-Lincoln, 1400 R St, Lincoln, NE 68588, United States 2Morgridge Institute for Research, 330 N Orchard St, Madison, WI 53715, United States 3Fermilab, Kirk Road and Pine St, Batavia, IL 60510, United States 4CERN, Meyrin 1211, Geneve, Switzerland Abstract. We overview recent changes in the ROOT I/O system, enhancing it by improving its performance and interaction with other data analysis ecosys- tems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly improve experiment’s software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically in- creased over the last couple of years. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly, because there are significant trade-offs between the increased CPU cost for reading and writing files and the reduces storage space. 1 Introduction In the past years, Large Hadron Collider (LHC) experiments are managing about an exabyte of storage for analysis purposes, approximately half of which is stored on tape storages for archival purposes, and half is used for traditional disk storage. Meanwhile for High Lumi- nosity Large Hadron Collider (HL-LHC) storage requirements per year are expected to be increased by a factor of 10 [1].
    [Show full text]
  • Arxiv:2004.10531V1 [Cs.OH] 8 Apr 2020
    ROOT I/O compression improvements for HEP analysis Oksana Shadura1;∗ Brian Paul Bockelman2;∗∗ Philippe Canal3;∗∗∗ Danilo Piparo4;∗∗∗∗ and Zhe Zhang1;y 1University of Nebraska-Lincoln, 1400 R St, Lincoln, NE 68588, United States 2Morgridge Institute for Research, 330 N Orchard St, Madison, WI 53715, United States 3Fermilab, Kirk Road and Pine St, Batavia, IL 60510, United States 4CERN, Meyrin 1211, Geneve, Switzerland Abstract. We overview recent changes in the ROOT I/O system, increasing per- formance and enhancing it and improving its interaction with other data analy- sis ecosystems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly to improve experiment’s software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically in- creased during the LHC era. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly: there are sig- nificant trade-offs between the increased CPU cost for reading and writing files and the reduce storage space. 1 Introduction In the past years LHC experiments are commissioned and now manages about an exabyte of storage for analysis purposes, approximately half of which is used for archival purposes, and half is used for traditional disk storage. Meanwhile for HL-LHC storage requirements per year are expected to be increased by factor 10 [1]. arXiv:2004.10531v1 [cs.OH] 8 Apr 2020 Looking at these predictions, we would like to state that storage will remain one of the major cost drivers and at the same time the bottlenecks for HEP computing.
    [Show full text]
  • Unfoldr Dstep
    Asymmetric Numeral Systems Jeremy Gibbons WG2.11#19 Salem ANS 2 1. Coding Huffman coding (HC) • efficient; optimally effective for bit-sequence-per-symbol arithmetic coding (AC) • Shannon-optimal (fractional entropy); but computationally expensive asymmetric numeral systems (ANS) • efficiency of Huffman, effectiveness of arithmetic coding applications of streaming (another story) • ANS introduced by Jarek Duda (2006–2013). Now: Facebook (Zstandard), Apple (LZFSE), Google (Draco), Dropbox (DivANS). ANS 3 2. Intervals Pairs of rationals type Interval (Rational, Rational) = with operations unit (0, 1) = weight (l, r) x l (r l) x = + − ⇥ narrow i (p, q) (weight i p, weight i q) = scale (l, r) x (x l)/(r l) = − − widen i (p, q) (scale i p, scale i q) = so that narrow and unit form a monoid, and inverse relationships: weight i x i x unit 2 () 2 weight i x y scale i y x = () = narrow i j k widen i k j = () = ANS 4 3. Models Given counts :: [(Symbol, Integer)] get encodeSym :: Symbol Interval ! decodeSym :: Rational Symbol ! such that decodeSym x s x encodeSym s = () 2 1 1 1 1 Eg alphabet ‘a’, ‘b’, ‘c’ with counts 2, 3, 5 encoded as (0, /5), ( /5, /2), and ( /2, 1). { } ANS 5 4. Arithmetic coding encode1 :: [Symbol ] Rational ! encode1 pick foldl estep unit where = ◦ 1 estep :: Interval Symbol Interval 1 ! ! estep is narrow i (encodeSym s) 1 = decode1 :: Rational [Symbol ] ! decode1 unfoldr dstep where = 1 dstep :: Rational Maybe (Symbol, Rational) 1 ! dstep x let s decodeSym x in Just (s, scale (encodeSym s) x) 1 = = where pick :: Interval Rational satisfies pick i i.
    [Show full text]
  • Compresso: Efficient Compression of Segmentation Data for Connectomics
    Compresso: Efficient Compression of Segmentation Data For Connectomics Brian Matejek, Daniel Haehn, Fritz Lekschas, Michael Mitzenmacher, Hanspeter Pfister Harvard University, Cambridge, MA 02138, USA bmatejek,haehn,lekschas,michaelm,[email protected] Abstract. Recent advances in segmentation methods for connectomics and biomedical imaging produce very large datasets with labels that assign object classes to image pixels. The resulting label volumes are bigger than the raw image data and need compression for efficient stor- age and transfer. General-purpose compression methods are less effective because the label data consists of large low-frequency regions with struc- tured boundaries unlike natural image data. We present Compresso, a new compression scheme for label data that outperforms existing ap- proaches by using a sliding window to exploit redundancy across border regions in 2D and 3D. We compare our method to existing compression schemes and provide a detailed evaluation on eleven biomedical and im- age segmentation datasets. Our method provides a factor of 600-2200x compression for label volumes, with running times suitable for practice. Keywords: compression, encoding, segmentation, connectomics 1 Introduction Connectomics|reconstructing the wiring diagram of a mammalian brain at nanometer resolution|results in datasets at the scale of petabytes [21,8]. Ma- chine learning methods find cell membranes and create cell body labelings for every neuron [18,12,14] (Fig. 1). These segmentations are stored as label volumes that are typically encoded in 32 bits or 64 bits per voxel to support labeling of millions of different nerve cells (neurons). Storing such data is expensive and transferring the data is slow. To cut costs and delays, we need compression methods to reduce data sizes.
    [Show full text]
  • Forcepoint DLP Supported File Formats and Size Limits
    Forcepoint DLP Supported File Formats and Size Limits Supported File Formats and Size Limits | Forcepoint DLP | v8.8.1 This article provides a list of the file formats that can be analyzed by Forcepoint DLP, file formats from which content and meta data can be extracted, and the file size limits for network, endpoint, and discovery functions. See: ● Supported File Formats ● File Size Limits © 2021 Forcepoint LLC Supported File Formats Supported File Formats and Size Limits | Forcepoint DLP | v8.8.1 The following tables lists the file formats supported by Forcepoint DLP. File formats are in alphabetical order by format group. ● Archive For mats, page 3 ● Backup Formats, page 7 ● Business Intelligence (BI) and Analysis Formats, page 8 ● Computer-Aided Design Formats, page 9 ● Cryptography Formats, page 12 ● Database Formats, page 14 ● Desktop publishing formats, page 16 ● eBook/Audio book formats, page 17 ● Executable formats, page 18 ● Font formats, page 20 ● Graphics formats - general, page 21 ● Graphics formats - vector graphics, page 26 ● Library formats, page 29 ● Log formats, page 30 ● Mail formats, page 31 ● Multimedia formats, page 32 ● Object formats, page 37 ● Presentation formats, page 38 ● Project management formats, page 40 ● Spreadsheet formats, page 41 ● Text and markup formats, page 43 ● Word processing formats, page 45 ● Miscellaneous formats, page 53 Supported file formats are added and updated frequently. Key to support tables Symbol Description Y The format is supported N The format is not supported P Partial metadata
    [Show full text]
  • Compression: Putting the Squeeze on Storage
    Compression: Putting the Squeeze on Storage Live Webcast September 2, 2020 11:00 am PT 1 | ©2020 Storage Networking Industry Association. All Rights Reserved. Today’s Presenters Ilker Cebeli John Kim Brian Will Moderator Chair, SNIA Networking Storage Forum Intel® QuickAssist Technology Samsung NVIDIA Software Architect Intel 2 | ©2020 Storage Networking Industry Association. All Rights Reserved. SNIA-At-A-Glance 3 3 | ©2020 Storage Networking Industry Association. All Rights Reserved. NSF Technologies 4 4 | ©2020 Storage Networking Industry Association. All Rights Reserved. SNIA Legal Notice § The material contained in this presentation is copyrighted by the SNIA unless otherwise noted. § Member companies and individual members may use this material in presentations and literature under the following conditions: § Any slide or slides used must be reproduced in their entirety without modification § The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations. § This presentation is a project of the SNIA. § Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. § The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK. 5 | ©2020 Storage Networking Industry Association.
    [Show full text]
  • RFC 8878: Zstandard Compression and the 'Application/Zstd'
    Stream: Internet Engineering Task Force (IETF) RFC: 8878 Obsoletes: 8478 Category: Informational Published: February 2021 ISSN: 2070-1721 Authors: Y. Collet M. Kucherawy, Ed. Facebook Facebook RFC 8878 Zstandard Compression and the 'application/zstd' Media Type Abstract Zstandard, or "zstd" (pronounced "zee standard"), is a lossless data compression mechanism. This document describes the mechanism and registers a media type, content encoding, and a structured syntax suffix to be used when transporting zstd-compressed content via MIME. Despite use of the word "standard" as part of Zstandard, readers are advised that this document is not an Internet Standards Track specification; it is being published for informational purposes only. This document replaces and obsoletes RFC 8478. Status of This Memo This document is not an Internet Standards Track specification; it is published for informational purposes. This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are candidates for any level of Internet Standard; see Section 2 of RFC 7841. Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc8878. Copyright Notice Copyright (c) 2021 IETF Trust and the persons identified as the document authors. All rights reserved. Collet & Kucherawy Informational Page 1 RFC 8878 application/zstd February 2021 This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document.
    [Show full text]
  • A Novel Coding Architecture for Lidar Point Cloud Sequence
    IEEE Robotics and Automation Letters (RAL) paper presented at the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) October 25-29, 2020, Las Vegas, NV, USA (Virtual) A Novel Coding Architecture for LiDAR Point Cloud Sequence Xuebin Sun1*, Sukai Wang2*, Miaohui Wang3, Zheng Wang4 and Ming Liu2, Senior Member, IEEE Abstract— In this paper, we propose a novel coding architec- the point cloud data. However, these methods are unsuitable ture for LiDAR point cloud sequences based on clustering and for unmanned vehicles. Traditional image or video encoding prediction neural networks. LiDAR point clouds are structured, algorithms, such as JPEG2000 , JPEG-LS [3], and HEVC which provides an opportunity to convert the 3D data to a 2D array, represented as range images. Thus, we cast the [4], were designed mostly for encoding integer pixel values, LiDAR point clouds compression as a range images coding and using them to encode floating-point LiDAR data will problem. Inspired by the high efficiency video coding (HEVC) cause significant distortion. Furthermore, the range image is algorithm, we design a novel coding architecture for the point characterized by sharp edges and homogeneous regions with cloud sequence. The scans are divided into two categories: nearly constant values, which is quite different from textured intra-frames and inter-frames. For intra-frames, a cluster-based intra-prediction technique is utilized to remove the spatial video. Thus, coding the range image with traditional tools redundancy. For inter-frames, we design a prediction network such as the block-based discrete cosine transform (DCT) model using convolutional LSTM cells, which is capable of followed by coarse quantization can result in significant predicting future inter-frames according to the encoded intra- coding errors at sharp edges, causing a safety hazard in frames.
    [Show full text]
  • Massively-Parallel Lossless Data Decompression
    Massively-Parallel Lossless Data Decompression Evangelia Sitaridi∗, Rene Muellery, Tim Kaldeweyz, Guy Lohmany and Kenneth A. Ross∗ ∗Columbia University: feva, [email protected] yIBM Almaden Research: fmuellerr, [email protected] zIBM Watson: [email protected] Abstract—Today’s exponentially increasing data volumes and returns caused by per-block overheads. In order to exploit the high cost of storage make compression essential for the Big the high degree of parallelism of GPUs, with potentially Data industry. Although research has concentrated on efficient thousands of concurrent threads, our implementation needs compression, fast decompression is critical for analytics queries that repeatedly read compressed data. While decompression can to take advantage of both intra-block parallelism and inter- be parallelized somewhat by assigning each data block to a block parallelism. For intra-block parallelism, a group of different process, break-through speed-ups require exploiting the GPU threads decompresses the same data block concurrently. massive parallelism of modern multi-core processors and GPUs Achieving this parallelism is challenging due to the inherent for data decompression within a block. We propose two new data dependencies among the threads that collaborate on techniques to increase the degree of parallelism during decom- pression. The first technique exploits the massive parallelism decompressing that block. of GPU and SIMD architectures. The second sacrifices some In this paper, we propose and evaluate two approaches to compression efficiency to eliminate data dependencies that limit address this intra-block decompression challenge. The first parallelism during decompression. We evaluate these techniques technique exploits the SIMD-like execution model of GPUs on the decompressor of the DEFLATE scheme, called Inflate, which is based on LZ77 compression and Huffman encoding.
    [Show full text]
  • Zstandard Compression in Openzfs
    1 of 6 Zstandard Compression in OpenZFS BY ALLAN JUDE ZFS is a highly advanced filesystem with integrated volume manager that was added to FreeBSD in 2007 and has since become a major part of the operating system. ZFS includes a transparent and adjustable compression feature that can seamlessly compress data before stor- ing it and decompress it before returning it for the application’s use. Because the compression is managed by ZFS, applications need not be aware of it. Filesystem compression not only saves space, but in many circumstances, can even lower read and write latency by reducing the total volume of data that needs to be stored or retrieved. Originally ZFS supported a small number of compression algorithms: LZJB (an improved Lem- pel–Ziv variant created by Jeff Bonwick, one of the co-creators of ZFS, it is moderately fast but only offers low compression ratios), ZLE (Zero Length Encoding, which only compresses runs of zeros), and the nine levels of gzip (the familiar slow, but moderately high-compression algorithm). Users could thus choose In 2015, LZ4 replaced LZJB as between no compression, fast but modest compression, or slow but higher compression. Unsurprisingly, these the default when users enable same users often went to great lengths to separate out compression without specifying data that should be compressed from data that was already compressed in order to avoid ZFS trying to re- an algorithm. compress it and wasting time to no benefit. For various historical reasons, compression still defaults to “off” in newly created ZFS storage pools. In 2013, ZFS added a new compression algorithm, LZ4, which offered both higher speed and better compression ratios than LZJB.
    [Show full text]
  • GROOT: a Real-Time Streaming System of High-Fidelity Volumetric Videos
    GROOT: A Real-time Streaming System of High-Fidelity Volumetric Videos Kyungjin Lee Juheon Yi Youngki Lee Seoul National University Seoul National University Seoul National University [email protected] [email protected] [email protected] Sunghyun Choi Young Min Kim Samsung Research Seoul National University [email protected] [email protected] Abstract ACM Reference Format: We present GROOT, a mobile volumetric video streaming system Kyungjin Lee, Juheon Yi, Youngki Lee, Sunghyun Choi, and Young Min Kim. 2020. GROOT: A Real-time Streaming System of High-Fidelity Volumetric that delivers three-dimensional data to mobile devices for a fully Videos . In The 26th Annual International Conference on Mobile Computing immersive virtual and augmented reality experience. The system and Networking (MobiCom ’20), September 21–25, 2020, London, United King- design for streaming volumetric videos should be fundamentally dom. ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3372224. different from conventional 2D video streaming systems. First, the 3419214 amount of data required to deliver the 3D volume is considerably larger than conventional videos with frames of 2D images, even 1 Introduction compared to high-resolution 2D or 360◦ videos. Second, the 3D data Volumetric video is an emerging media that provides a highly immer- representation, which encodes the surface of objects within the sive and interactive user experience. Different from 2D videos and volume, is a sparse and unorganized data structure with varying 360◦ videos, the volumetric video consists of 3D data, enabling users scales, whereas a conventional video is composed of a sequence of to watch the video with six-degrees-of-freedom (6DoF).
    [Show full text]
  • State of the Art and Future Trends in Data Reduction for High-Performance Computing
    DOI: 10.14529/jsfi200101 State of the Art and Future Trends in Data Reduction for High-Performance Computing Kira Duwe1, Jakob L¨uttgau1, Georgiana Mania2, Jannek Squar1, Anna Fuchs1, Michael Kuhn1, Eugen Betke3, Thomas Ludwig3 c The Authors 2020. This paper is published with open access at SuperFri.org Research into data reduction techniques has gained popularity in recent years as storage ca- pacity and performance become a growing concern. This survey paper provides an overview of leveraging points found in high-performance computing (HPC) systems and suitable mechanisms to reduce data volumes. We present the underlying theories and their application throughout the HPC stack and also discuss related hardware acceleration and reduction approaches. After intro- ducing relevant use-cases, an overview of modern lossless and lossy compression algorithms and their respective usage at the application and file system layer is given. In anticipation of their increasing relevance for adaptive and in situ approaches, dimensionality reduction techniques are summarized with a focus on non-linear feature extraction. Adaptive approaches and in situ com- pression algorithms and frameworks follow. The key stages and new opportunities to deduplication are covered next. An unconventional but promising method is recomputation, which is proposed at last. We conclude the survey with an outlook on future developments. Keywords: data reduction, lossless compression, lossy compression, dimensionality reduction, adaptive approaches, deduplication, in situ, recomputation, scientific data set. Introduction Breakthroughs in science are increasingly enabled by supercomputers and large scale data collection operations. Applications span the spectrum of scientific domains from fluid-dynamics in climate simulations and engineering, to particle simulations in astrophysics, quantum mechan- ics and molecular dynamics, to high-throughput computing in biology for genome sequencing and protein-folding.
    [Show full text]