sign in sign up
<<
Home , Brotli, LZFSE, Lzip

FLOW CYTOMETRY –

A.E. Bras PhD Student

Erasmus University, Rotterdam, the Netherlands Flow Cytometry Flow Cytometry Flow Cytometry Flow Cytometry Flow Cytometry Flow Cytometry Older Systems Older Systems Newer Systems Newer Systems Newer Systems Newer Systems

FLOATING POINT DATA ------Newer Systems

FLOATING POINT DATA ------

MILLIONS ------Storage

FLOATING POINT DATA ------

MILLIONS ------Storage

.FCS

FLOATING POINT DATA ------

MILLIONS ------Storage

FLOW CYTOMETRY STRANDARD .FCS

FLOATING POINT DATA ------

MILLIONS ------Storage

FLOW CYTOMETRY STRANDARD .FCS

PRO WIDELEY USED FLOATING POINT DATA ------

MILLIONS ------Storage

FLOW CYTOMETRY STRANDARD .FCS

PRO WIDELEY USED FLOATING POINT DATA ------CON ------NO COMPRESSION ------

MILLIONS ------

.FCS Lossless Compression

.FCS

≈ 70 % . Lossless Compression

.FCS

≈ 70 % .ZIP

Better alternative? Lossless Compression - Benchmark Lossless Compression - Benchmark

.FCS

167.131 Lossless Compression Compression Lossless - Benchmark 167.131 .FCS RANDOM Lossless Compression Compression Lossless - Benchmark 167.131 .FCS RANDOM Lossless Compression Compression Lossless ... LZ4HC LZO1B LZO2A DENSITY LIZARD LZSSE2 LZO1Y LZSSE8 LZO1Z LZO1X UCL_NRV2B UCL_NRV2E UCL_NRV2D CRUSH LZFSE LIBDEFLATE ZLING XPACK ZSTD TORNADO CSC LZHAM GLZA XZ LZMA 0.695 0.692 0.690 0.676 0.665 0.637 0.626 0.622 0.598 0.590 0.568 0.558 0.533 0.533 0.533 ... 0.777 0.775 0.774 0.771 0.770 0.768 0.768 0.758 0.757 0.755 0.734 0.725 0.725 RATIO - Benchmark 167.131 .FCS RANDOM ... ZLIBH FASTAC SUBOTIN LZG LZSSE4 LZ4 LZSSE2 BSCQLFC LZSSE8 DOBOZ BRIEFLZ ZLIB CRUSH LZFSE LIBDEFLATE ZSTD XPACK BALZ CSC BROTLI LZHAM FLZMA2 LZMA LZIP BCM ZPAQ CODEC ... 0.823 0.819 0.819 0.803 0.781 0.777 0.768 0.763 0.758 0.750 0.727 0.695 0.692 0.690 0.676 0.674 0.625 0.612 0.598 0.590 0.580 0.571 0.569 0.539 0.533 0.533 0.510 0.460 RATIO Lossless Compression Compression Lossless ... LZ4HC LZO1B LZO2A DENSITY LIZARD LZSSE2 LZO1Y LZSSE8 LZO1Z LZO1X UCL_NRV2B UCL_NRV2E UCL_NRV2D ZLIB CRUSH LZFSE LIBDEFLATE ZLING XPACK ZSTD TORNADO BROTLI CSC LZHAM GLZA XZ LZMA LZIP CODEC 0.695 0.692 0.690 0.676 0.665 0.637 0.626 0.622 0.598 0.590 0.568 0.558 0.533 0.533 0.533 ... 0.777 0.775 0.774 0.771 0.770 0.768 0.768 0.758 0.757 0.755 0.734 0.725 0.725 RATIO - Benchmark 167.131 .FCS RANDOM ... ZLIBH FASTAC SUBOTIN LZG LZSSE4 LZ4 LZSSE2 BSCQLFC LZSSE8 DOBOZ BRIEFLZ ZLIB CRUSH LZFSE LIBDEFLATE ZOPFLI ZSTD XPACK BALZ CSC BZIP2 BROTLI LZHAM FLZMA2 LZMA LZIP BCM ZPAQ CODEC ... 0.823 0.819 0.819 0.803 0.781 0.777 0.768 0.763 0.758 0.750 0.727 0.695 0.692 0.690 0.676 0.674 0.625 0.612 0.598 0.590 0.580 0.571 0.569 0.539 0.533 0.533 0.510 0.460 RATIO Lossless Compression - Benchmark

CODEC RATIO CODEC RATIO LZIP 0.533 ZPAQ 0.460 LZMA 0.533 BCM 0.510 XZ 0.533 LZIP 0.533 GLZA 0.558 LZMA 0.533 LZHAM 0.568 FLZMA2 0.539 CSC 0.590 LZHAM 0.569 BROTLI 0.598 BROTLI 0.571 TORNADO 0.622 BZIP2 0.580 ZSTD 0.626 CSC 0.590 XPACK 0.637 BALZ 0.598 ZLING 0.665 XPACK 0.612 LIBDEFLATE 0.676 ZSTD 0.625 LZFSE 0.690 ZOPFLI 0.674 CRUSH 0.692 LIBDEFLATE 0.676 ZLIB 0.695 LZFSE 0.690 UCL_NRV2D 0.725 CRUSH 0.692 UCL_NRV2E 0.725 ZLIB 0.695 UCL_NRV2B 0.734 BRIEFLZ 0.727 LZO1X 0.755 DOBOZ 0.750 LZO1Z 0.757 LZSSE8 0.758 LZSSE8 0.758 BSCQLFC 0.763 LZO1Y 0.768 LZSSE2 0.768 LZSSE2 0.768 LZ4 0.777 LIZARD 0.770 LZSSE4 0.781 DENSITY 0.771 LZG 0.803 LZO2A 0.774 .FCS SUBOTIN 0.819 LZO1B 0.775 FASTAC 0.819 LZ4HC 0.777 ZLIBH 0.823 ...... 167.131 ...... Lossless Compression - Benchmark Lossless Compression - Benchmark

≈ 0.45 Implementation in R Implementation in R

base::memCompress(type="XZ") Implementation in R

base::memCompress(type="XZ") Bioconductor - flowCore Bioconductor - flowCore Bioconductor - flowCore

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs Bioconductor - flowCore

base::readBin C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

C:\output.fcs Bioconductor - flowCore

base::readBin C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

base::writeBin C:\output.fcs Bioconductor - flowCore

base::readBin C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

base::memCompress base::writeBin C:\output.fcs Bioconductor - flowCore

base::readBin C:\input.fcs base::memDecompress

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS

base::memCompress base::writeBin C:\output.fcs Bioconductor - flowCore

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS ..GZ

C:\output.fcs Bioconductor - flowCore

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS .TAR.GZ

C:\output.fcs Bioconductor - flowCore

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS .TAR.GZ

C:\output.fcs Bioconductor - flowCore

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS .FCS.XZ

C:\output.fcs Bioconductor - flowCore

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS .FCS.XZ

C:\output.fcs Bioconductor - flowCore

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS .FCS.XZ

C:\output.fcs Bioconductor - flowCore

C:\input.fcs

flowCore::read.FCS flowSom::FlowSOM flowCore::write.FCS .FCS.XZ

C:\output.fcs Bioconductor – flowCore - Pipelines Bioconductor – flowCore - Pipeline

C:\input\01.fcs C:\input\02.fcs

flowCore::read.flowSet flowCore::write.flowSet

C:\output\01.fcs C:\output\02.fcs Bioconductor – flowCore - Pipeline C:\input.zip base::unzip

C:\input\01.fcs C:\input\02.fcs

flowCore::read.flowSet flowCore::write.flowSet

C:\output\01.fcs C:\output\02.fcs Bioconductor – flowCore - Pipeline C:\input.zip base::unzip

C:\input\01.fcs C:\input\02.fcs

flowCore::read.flowSet flowCore::write.flowSet

C:\output\01.fcs C:\output\02.fcs

base::zip

C:\output.zip Bioconductor – flowCore - Pipeline C:\input.zip base::unzip

C:\input\01.fcs C:\input\02.fcs

flowCore::read.flowSet flowCore::write.flowSet

C:\output\01.fcs C:\output\02.fcs

base::zip

C:\output.zip Bioconductor – flowCore - Pipeline C:\input.zip base::unzip

C:\input\01.fcs C:\input\02.fcs

flowCore::read.flowSet ARCHIVAL flowCore::write.flowSet CYTOMETRY STANDARD C:\output\01.fcs C:\output\02.fcs .ACS base::zip C:\output.zip Bioconductor – flowCore - Pipeline

ARCHIVAL CYTOMETRY = STANDARD .ACS .ZIP Bioconductor – flowCore - Pipeline

ARCHIVAL CYTOMETRY = STANDARD .FCS .FCS .FCS .ACS .ZIP Bioconductor – flowCore - Pipeline

.TXT ARCHIVAL CYTOMETRY = STANDARD .FCS .FCS .FCS .ACS .ZIP Bioconductor – flowCore - Pipeline C:\input.zip base::unzip

C:\input\01.fcs C:\input\01.fcs

flowCore::read.flowSet flowCore::write.flowSet

C:\output\01.fcs C:\output\01.fcs

base::zip

C:\output.zip Bioconductor – flowCore - Pipeline

C:\input.acs

flowCore::read.flowSet flowCore::write.flowSet

C:\output.acs Bioconductor – flowCore - Pipeline

base::unzip C:\input.acs flowCore::read.flowSet flowCore::write.flowSet

C:\output.acs Bioconductor – flowCore - Pipeline

base::unzip C:\input.acs flowCore::read.flowSet flowCore::write.flowSet

base::zip C:\output.acs C:\input.fcs.xz flowCore::read.FCS flowCore::write.FCS

C:\output.fcs.xz

C:\input.acs flowCore::read.flowSet flowCore::write.flowSet

C:\output.acs C:\input.fcs.xz flowCore::read.FCS PRO flowCore::write.FCS PERFORMANCE

C:\output.fcs.xz

C:\input.acs flowCore::read.flowSet flowCore::write.flowSet

C:\output.acs C:\input.fcs.xz flowCore::read.FCS PRO CON flowCore::write.FCS PERFORMANCE COMPATIBILITY

C:\output.fcs.xz

C:\input.acs flowCore::read.flowSet flowCore::write.flowSet

C:\output.acs C:\input.fcs.xz flowCore::read.FCS PRO CON flowCore::write.FCS PERFORMANCE COMPATIBILITY

C:\output.fcs.xz

C:\input.acs flowCore::read.flowSet PRO flowCore::write.flowSet COMPATIBILITY

C:\output.acs C:\input.fcs.xz flowCore::read.FCS PRO CON flowCore::write.FCS PERFORMANCE COMPATIBILITY

C:\output.fcs.xz

C:\input.acs flowCore::read.flowSet PRO CON flowCore::write.flowSet COMPATIBILITY PERFORMANCE

C:\output.acs C:\input.fcs.xz flowCore::read.FCS PRO CON flowCore::write.FCS PERFORMANCE COMPATIBILITY EASY C:\output.fcs.xz

C:\input.acs flowCore::read.flowSet PRO CON flowCore::write.flowSet COMPATIBILITY PERFORMANCE EASY C:\output.acs C:\input.fcs.xz flowCore::read.FCS PRO CON flowCore::write.FCS PERFORMANCE COMPATIBILITY EASY C:\output.fcs.xz BENIFIT

C:\input.acs flowCore::read.flowSet PRO CON flowCore::write.flowSet COMPATIBILITY PERFORMANCE EASY C:\output.acs BENIFIT Questions?

Lossless Compression of Cytometric Data Anne E. Bras & Vincent H. J. van der Velden