PROC. OF THE 6th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2013) 3 Bloscpack: a compressed lightweight serialization format for numerical data Valentin Haenel∗† F Abstract—This paper introduces the Bloscpack file format and the accompany- compressor. The block size is chosen such that it either fits ing Python reference implementation. Bloscpack is a lightweight, compressed into a typical L1 cache (for compression levels up to 6) or L2 binary file-format based on the Blosc codec and is designed for lightweight, cache (for compression levels larger than 6). In modern CPUs fast serialization of numerical data. This article presents the features of the L1 and L2 are typically non-shared between other cores, and file-format and some some API aspects of the reference implementation, in particular the ability to handle Numpy ndarrays. Furthermore, in order to demon- so this choice of block size leads to an optimal performance strate its utility, the format is compared both feature- and performance-wise to during multi-thread operation. a few alternative lightweight serialization solutions for Numpy ndarrays. The Also, Blosc features a shuffle filter [Alted2009] (p.71) which performance comparisons take the form of some comprehensive benchmarks may reshuffle multi-byte elements, e.g. 8 byte doubles, by over a range of different artificial datasets with varying size and complexity, the significance. The net result for series of numerical elements results of which are presented as the last section of this article. with little difference between elements that are close, is Index Terms—applied information theory, compression/decompression, that similar bytes are placed closer together and can thus python, numpy, file format, serialization, blosc be better compressed (this is specially true on time series datasets). Internally, Blosc uses its own codec, blosclz, which is a derivative of FastLZ [FastLZ] and implements the LZ77 1 INTRODUCTION [LZ77] scheme. The reason for Blosc to introduce its own When using compression during storage of numerical data codec is mainly the desire for simplicity (blosclz is a highly there are two potential improvements one can make. First, streamlined version of FastLZ), as well as providing a better by using compression, naturally one can save storage space. interaction with Blosc infrastructure. Secondly—and this is often overlooked—one can save time. Moreover, Blosc is designed to be extensible, and allows When using compression during serialization, the total com- other codecs than blosclz to be used in it. In other words, one pression time is the sum of the time taken to perform the can consider Blosc as a meta-compressor, in that it handles compression and the time taken to write the compressed data the splitting of the data into blocks, optionally applying the to the storage medium. Depending on the compression speed shuffle filter (or other future filters), while being responsible and the compression ratio, this sum maybe less than the of coordinating the individual threads during operation. Blosc time taken to serialize the data in uncompressed format i.e. then relies on a "real" codec to perform that actual compres- writeuncompressed > writecompressed +timecompress sion of the data blocks. As such, one can think of Blosc as The Bloscpack file format and Python reference implemen- a way to parallelize existing codecs, while allowing to apply tation aims to achieve exactly this by leveraging the fast, filters (also called pre-conditioners). In fact, at the time when multithreaded, blocking and shuffling Blosc codec. arXiv:1404.6383v2 [cs.MS] 29 Apr 2014 the research presented in this paper was conducted (Summer 2013), a proof-of-concept implementation existed to integrate 2 BLOSC the well known Snappy codec [Snappy] as well as LZ4 [LZ4] Blosc [Blosc] is a fast, multitreaded, blocking and shuffling into the Blosc framework. As of January 2014 this proof of compressor designed initially for in-memory compression. concept has matured and as of version 1.3.0 Blosc comes Contrary to many other available compressors which oper- equipped with support for Snappy [Snappy], LZ4 [LZ4] and ate sequentially on a data buffer, Blosc uses the blocking even Zlib [zlib]. technique [Alted2009], [Alted2010] to split the dataset into Blosc was initially developed to support in-memory com- individual blocks. It can then operate on each block using pression in order to mitigate the effects of the memory hierar- a different thread which effectively leads to a multithreaded chy [Jacob2009]. More specifically, to mitigate the effects of memory latency, i.e. the ever growing divide between the CPU * Corresponding author: [email protected] speed and the memory access speed–which is also known as † Independent the problem of the starving CPUs [Alted2009]. Copyright © 2014 Valentin Haenel. This is an open-access article dis- The goal of in-memory compression techniques is to have tributed under the terms of the Creative Commons Attribution Li- a numerical container which keeps all data as in-memory cense, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. compressed blocks. If the data needs to be operated on, it http://creativecommons.org/licenses/by/3.0/ is decompressed only in the caches of the CPU. Hence, data 4 PROC. OF THE 6th EUR. CONF. ON PYTHON IN SCIENCE (EUROSCIPY 2013) can be moved faster from memory to CPU and the net result 4.3 ZFile is faster computation, since less CPU cycles are wasted while ZFile is the native serialization format that ships with the waiting for data. Similar techniques are applied successfully Joblib [Joblib] framework. Joblib is equipped with a caching in other settings. Imagine for example, one wishes to transfer mechanism that supports caching input and output arguments binary files over the internet. In this case the transfer time to functions and can thus avoid running heavy computations can be significantly improved by compressing the data before if the input has not changed. When serializing ndarrays with transferring it and decompressing it after having received it. Joblib, a special subclass of the Pickler is used to store the As a result the total compressed transfer time, which is taken metadata whereas the datablock is serialized as a ZFile. ZFile to be the sum of the compression and decompression process uses zlib [zlib] internally and simply runs zlib on the entire and the time taken to transfer the compressed file, is less data buffer. zlib is also an implementation of the DEFLATE than the time taken to transfer the plain file. For example algorithm. One drawback of the current ZFile implementation the well known UNIX tool rsync [rsync] implements a is that no chunking scheme is employed. This means that the -z switch which performs compression of the data before memory requirements might be twice that of the original input. sending it and decompression after receiving it. The same Imagine trying to compress an incompressible buffer of 1GB: basic principle applies to in-memory compression, except in this case the memory requirement would be 2GB, since that we are transferring data from memory to CPU. Initial the entire buffer must be copied in memory as part of the implementations based on Blosc exist, c.f. Blaze [Blaze] and compression process before it can be written out to disk. carray [CArray], and have been shown to yield favourable results [Personal communication with Francesc Alted]. 5 BLOSCPACK FORMAT 3 NUMPY The Bloscpack format and reference implementation builds a The Numpy [VanDerWalt2011], [Numpy] ndarray is the serialization format around the Blosc codec. It is a simple de-facto multidimensional numerical container for scientific chunked file-format well suited for the storage of numerical python applications. It is probably the most fundamental data. As described in the Bloscpack format description, the package of the scientific python ecosystem and widely used big-picture of the file-format is as follows: and relied upon by third-party libraries and applications. It |-header-|-meta-|-offsets-| consists of the N-dimensional array class, various different |-chunk-|-checksum-|-chunk-|-checksum-|...| initialization routines and many different ways to operate on the data efficiently. The format contains a 32 byte header which contains various options and settings for the file, for example a magic string, 4 EXISTING LIGHTWEIGHT SOLUTIONS the format version number and the total number of chunks. There are a number of other plain (uncompressed) and com- The meta section is of variable size and can contain any pressed lightweight serialization formats for Numpy arrays metadata that needs to be saved alongside the data. An that we can compare Bloscpack to. We specifically ignore optional offsets section is provided to allow for partial more heavyweight solutions, such as HDF5, in this compari- decompression of the file in the future. This is followed by a son. series of chunks, each of which is a blosc compressed buffer. • NPY Each chunk can be optionally followed by a checksum of • NPZ the compressed data which can help to protect against silent • ZFile data corruption. The chunked format was initially chosen to circumvent a 4.1 NPY 2GB limitation of the Blosc codec. In fact, the ZFile format NPY [NPY] is a simple plain serialization format for numpy. It suffers from this exact limitation since zlib—at least the is considered somewhat of a gold standard for the serialization. Python bindings—is also limited to buffers of 2GB in size. The One of its advantages is that it is very, very lightweight. The limitation stems from the fact that int32 are used internally format specification is simple and can easily be digested within by the algorithms to store the size of the buffer and the an hour. In essence it simply contains the ndarray metadata maximum value of an int32 is indeed 2GB.