• Abstract and Figures
• Public Full-text

A GPU-based Architecture for Real-Time Data Assessment at Synchrotron Experiments Suren Chilingaryan, Andreas Kopmann, Alessandro Mirone, Tomy dos Santos Rolo Abstract—Current imaging experiments at synchrotron beam to understand functionality of devices and organisms and to lines often lack a real-time data assessment. X-ray imaging optimize technological processes [1], [2]. cameras installed at synchrotron facilities like ANKA provide In recent years, synchrotron tomography has seen a tremen- millions of pixels, each with a resolution of 12 bits or more, and take up to several thousand frames per second. A given dous decrease of total acquisition time [3]. Based on available experiment can produce data sets of multiple gigabytes in a photon flux densities at modern synchrotron sources ultra-fast few seconds. Up to now the data is stored in local memory, X-ray imaging could enable investigation of the dynamics of transferred to mass storage, and then processed and analyzed off- technological and biological processes with a time scale down line. The data quality and thus the success of the experiment, can, to the milliseconds range in the 3D. Using modern CMOS therefore, only be judged with a substantial delay, which makes an immediate monitoring of the results impossible. To optimize based detectors it is possible to reach up to several thousand the usage of the micro-tomography beam-line at ANKA we have frames per second. For example, frame rates of 5000 images ported the reconstruction software to modern graphic adapters per second were achieved using a filtered white beam from which offer an enormous amount of calculation power. We were ESRF’s ID19 wiggler source [4]. Using a larger effective able to reduce the reconstruction time from multiple hours to just pixel size the frame rates of 40000 images per second were a few minutes with a sample dataset of 20 GB. Using the new reconstruction software it is possible to provide a near real-time reported [5]. As a result of improved image acquisition, a given visualization and significantly reduce the time needed for the first experiment can produce data sets of multiple gigabytes in a evaluation of the reconstructed sample. The main paradigm of few seconds. There is a major challenge to process these data our approach is 100% utilization of all system resources. The in reasonable amount of time facilitating on-line reconstruction compute intensive parts are offloaded to the GPU. While the and fast feedback. GPU is reconstructing one slice, the CPUs are used to prepare Two main approaches are used at the moment to handle the the next one. A special attention is devoted to minimize data transfers between the host and GPU memory and to execute I/O huge data-sets produced at the tomographic setups. operations in parallel with the computations. It could be shown • At TopoTomo beam-line of ANKA (synchrotron facility that for our application not the computational part but the data at KIT [5]) up to now the data was stored in local transfers are now limiting the speed of the reconstruction. Several memory, transferred to mass storage, and then processed changes in the architecture of the DAQ system are proposed to overcome this second bottleneck. The article will introduce the and analyzed off-line. The data quality and thus the system architecture, describe the hardware platform in details, success of the experiment can only be judged with a and analyze performance gains during the first half year of substantial delay, which makes an immediate monitoring operation. of the results impossible. • Second alternative is supercomputer-based processing, expensive in terms of money and power consumption. For I. INTRODUCTION example, a pipelined data acquisition system combining a RIVEN by substantial developments in digital detector fast detector system, high speed data networks, and mas- D technology there is presently a tremendous progress sively parallel computers was employed at synchrotron at going on in X-ray technology offering broad applications in Aragonne National Laboratory to acquire and reconstruct the fields of medical diagnostics, homeland security, non- a full tomogram in tens of minutes [3]. destructive testing, materials research and others. X-ray imag- Instead we decided to use computation power of modern ing permits spatially resolved visualization of the 2D and graphic adapters provided by NVIDIA and AMD. Including 3D structures in materials and organisms which is crucial hundreds of simple processors used to transform vertexes in for the understanding of their properties. Further it allows 3D space they offer a way to speed up the process of more than defect recognition in devices in a broad resolution range one order of magnitude at low cost and with good scalability. A from the macro down to the nano-scale. Additional resolution peak performance of fastest graphic cards exceeds one teraflop in the time domain gives insight in the temporal structure (for single precision numbers). Top gaming desktops include evolution and thus access to dynamics of processes allowing up to four of such cards and, thus, producing about 5 teraflops of computational power. Compared to 100 gigaflops provided S. Chilingaryan and A. Kopmann are with Institute of Data Processing and Electronics, Karlsruhe Institute of Technology, Karlsruhe, Germany by commonly used servers, this gives a potential speedup of (telephone: +49 724 7826579, e-mail: [email protected]) 50 times [6], [7]. A. Mirone is with European Synchrotron Radiation Facility, Grenoble, To provide this enormous computational power to devel- France T. dos Santos Rolo is with Institute for Synchrotron Radiation, Karlsruhe opers NVIDIA released a CUDA (Common Unified Device Institute of Technology, Karlsruhe, Germany Architecture [8]) toolkit. It extends C language with few 978-1-4244-7110-2/10/$26.00 ©2010 IEEE syntactical constructs used for GPU programming. CUDA details about PyHST and our optimizations are presented. The exploits data parallelism: basically the architecture allows to performance evaluation, a discussion of I/O bottleneck, and a execute a sequence of mathematical operations, so called ker- review of available hardware platforms are provided in fifth nels, simultaneously over a multi-dimensional set of data. API section. Finally, conclusion summarizes presented information. includes functions to transfer the data from the system memory to GPU memory, execute kernels to process these data, and II. IMAGING AT SYNCHROTRON LIGHT SOURCES transfer the results back to the system memory. For developer convenience NVIDIA and community have released a number Fundamentally, the imaging at synchrotron light sources of libraries implementing many standard algorithms. NVIDIA is working as follows. The sample is placed at horizontal CUDA SDK includes cuFFT - an interface to perform multi- platform and evenly rotated in front of a pixel detector. The dimensional FFT (Fast Fourier Transformations), cuBLAS - a parallel X-rays are penetrating the sample and the pixel de- BLAS (Basic Linear Algebra Subprograms) implementation, tector registers series of parallel 2D projections of the sample and cudpp - a collection of parallel reduction algorithms. There density at different angles. These projections along with the are community implementations of LAPACK (Linear Algebra coordinates of the rotation axis and the angle between slices PACKage) and quantity of computer vision algorithms [9], are the input data for PyHST. To reconstruct a 3D image from [10], [11]. projection PyHST employs a FBP algorithm. All projections In addition to vendor-dependent programming toolkits, like are replicated in 3D space across the X-rays direction and CUDA, Khronos Group have introduced OpenCL (Open Com- integrated, see Fig. 1. Since the sample is rotating round the puting Language [12]), a new industry standard for par- strictly vertical axis, the sample can be seen as a sequence of allel programming. The architecture of OpenCL is similar horizontal slices which can be reconstructed separately. If the to CUDA, but, unlike the proprietary NVIDIA technology, center of rotation is located at a center of coordinate system it allows to execute developed application on both AMD then to find a value of point with coordinates (x, y) in slice z, it and NVIDIA GPUs as well as on general-purpose multi- is necessary to compute the following sum over all projections: P − core/multiprocessor CPUs. Pp=1 Ip(xcos(pα) ysin(pα), z) where P is a number of FBP (Filtered Back Projection) is a standard method to projections, α is an angle between projections, and Ip is a p- reconstruct 3D image of object from multiplicity of tomo- th projection. Since the projections are digital images of finite graphic projections [13]. PyHST (High Speed Tomography resolutions, the linear interpolation is performed to get density in Python [14]) is a fast implementation of filtered back- value at position xcos(pα) − ysin(pα) [13], [20]. projection developed at ESRF (European Synchrotron Radi- ation Facility) and used currently at synchrotrons at ESRF and KIT [5]. While it uses heavily optimized C-code for performance intensive parts of algorithm, the processing of typical data set with nowadays Xeon servers is still requiring about one hour of computations. In order to increase speed of processing and achieve a near real-time performance we accelerated reconstruction by offloading most of the computa- tions to GPU with the help of CUDA toolkit. The optimized version of PyHST is able to reconstruct a typical 3D image (3 gigavoxel image is reconstructed from 2000 projections) in only 40 seconds using a GPU server equipped with 4 graphic cards, I/O time excluded. It is approximately 30 times faster Fig. 1. Image reconstruction using back-projection algorithm. The sample is evenly rotating and pixel detectors registers series of parallel projections at compared to 8-core Xeon server used before. different angles (left). All registered projections are replicated across direction There are a few other research projects aiming to GPU- of incidence and integrated (right).

Load more

  8

Copy Link