Recode: a Data Reduction and Compression Description for High Throughput Time-Resolved Electron Microscopy
ReCoDe: A Data Reduction and Compression Description for High Throughput Time-Resolved Electron Microscopy 1,2 1,2 1,2 3 1,2,4 1,2 1,2 Abhik Datta , Kian Fong Ng , Balakrishnan Deepan , Melissa Ding , See Wee Chee , Yvonne Ban , Jian Shi , Duane Loh1,2,4 1 Centre for BioImaging Sciences, National University of Singapore, Singapore 117557. 2 Department of Biological Sciences, National University of Singapore, Singapore 117557. 3 Department of Computer Science and Engineering, Ohio State University, Columbus, OH 43210, USA. 4 Department of Physics, National University of Singapore, Singapore 117551. Abstract Fast, direct electron detectors have significantly improved the spatio-temporal resolution of electron microscopy movies. Preserving both spatial and temporal resolution in extended observations, however, requires storing prohibitively large amounts of data. Here, we describe an efficient and flexible data reduction and compression scheme (ReCoDe) that retains both spatial and temporal resolution by preserving individual electron events. Running ReCoDe on a workstation we demonstrate on-the-fly reduction and compression of raw data streaming off a detector at 3 GB/s, for hours of uninterrupted data collection. The output was 100-fold smaller than the raw data and saved directly onto network-attached storage drives over a 10 GbE connection. We discuss calibration techniques that support electron detection and counting (e.g. estimate electron backscattering rates, false positive rates, and data compressibility), and novel data analysis methods enabled by ReCoDe (e.g. recalibration of data post acquisition, and accurate estimation of coincidence loss). Introduction Fast, back-thinned direct electron detectors are rapidly transforming electron microscopy.
[Show full text]