4.45 Pflops Astrophysical N-Body Simulation on K computer - The Gravitational Trillion-Body Problem Tomoaki Ishiyama Keigo Nitadori Junichiro Makino Center for Computational Science Center for Computational Science Graduate School of Science and Engineering University of Tsukuba University of Tsukuba Tokyo Institute of Technology [email protected] [email protected] [email protected] Abstract—As an entry for the 2012 Gordon-Bell performance prize, we report performance results of astrophysical N-body simulations of one trillion particles performed on the full system of K computer. This is the first gravitational trillion-body simulation in the world. We describe the scientific motivation, the numerical algorithm, the parallelization strategy, and the performance analysis. Unlike many previous Gordon-Bell prize winners that used the tree algorithm for astrophysical N-body simulations, we used the hybrid TreePM method, for similar level of accuracy in which the short-range force is calculated by the tree algorithm, and the long-range force is solved by the particle- mesh algorithm. We developed a highly-tuned gravity kernel for Fig. 1. The hierarchical tree algorithm. White circles represent particles. short-range forces, and a novel communication algorithm for Blue circles are the multipole expansions of tree nodes. Red solid arrows long-range forces. The average performance on 24576 and 82944 and blue dotted arrows show the particle-particle and the particle-multipole nodes of K computer are 1.53 and 4.45 Pflops, which correspond interactions, respectively. to 49% and 42% of the peak speed. I. INTRODUCTION and the total calculation cost per timestep becomes (N log N), achieving drastic reduction from the (N 2) Astrophysical N-body simulations have been widely used O O to study the nonlinear structure formation in the Universe. cost of the direct summation method. Such simulations are usually called as cosmological N-body Thus, the tree algorithm have been used for most of practical simulations. In these simulations, a particle moves according large cosmological calculations in the last two decades, and to the gravitational forces from all the other particles in the many Gordon-Bell prizes have been given to such simulations system. (1992 [3], 1995-2001 [4]–[10], 2003 [11], and 2009-2010 Thus, the most straightforward algorithm to calculate the [12] [13]). In order to accelerate the calculation of gravity, acceleration of a particle is to calculate the N 1 forces some of them used GRAPEs [4] [5] [8]–[11] , which are from the rest of the system, where N is the total− number of special-purpose computers [14]–[16], or graphics processing particles in the system. This method is usually called the direct units (GPUs) [12] [13]. summation. This method is unpractical for large N (N > 106), There is one difference between large cosmological simula- since the calculation cost is proportional to N 2. Therefore, tions in the literature and those awarded Gordon-Bell prizes so faster algorithms with some approximation are usually used far. In practically all recent large calculations, except for those arXiv:1211.4406v5 [astro-ph.CO] 13 Apr 2015 in cosmological N-body simulations. for Gordon-Bell prizes, the periodic boundary condition is The tree algorithm [1] [2] is the most widely used used. With the periodic boundary condition, the computational algorithm for cosmological N-body simulations. The basic domain is a cube, and we assume that there are infinite copies idea of the tree algorithm is to use a hierarchical oct-tree of them which fill the infinite space. This condition is used to structure to represent an N-body system (figure 1). The force model the Universe which is uniform in very large scale. On from particles in one box to one particle can be calculated the other hand, in simulations for past Gordon-Bell prizes, the by evaluating the multipole expansion, if the error is small open boundary condition, in which the calculation domain is enough (if the box and the particles are well separated). If initially a sphere, was used. not, the force is evaluated as the sum of forces from eight From theoretical point of view, in both cases one can only subboxes. By recursively applying this procedure, one can analyze structures of the size sufficiently small compared calculate the total force on a particle with (log N) cost, to the size of the computational domain. However, periodic O boundary is computationally more efficient. The reason is that SC12, November 10-16, 2012, Salt Lake City, Utah, USA with open boundary, only the structures near the center of the 978-1-4673-0806-9/12$31.00 c 2012 IEEE sphere are reliable. Structures near the boundary are affected radius with n particles is (n2). Thus, for a cell with 1000 times more particles than average,O the cost is 106 times more expensive. The TreePM algorithm can solve this problem, since the calculation cost of such cell is (n log n). Thus, the TreePM algorithm is very efficient and isO used in a number of recent large cosmological N-body simulations (e.g. [25] [26]). The TreePM algorithm is relatively new. It became popular around 2000. Thus, it is not surprising that Gordon-Bell winners in 1990’s used the pure tree algorithm. Advantage of the TreePM algorithm over the pure tree algorithm is twofold. The first one is, as we already noted, we can use entire volume Fig. 2. The schematic view of the P3M and the TreePM algorithm. Large red and blue dashed circles show the cutoff radius of each algorithm. Within for data analysis. The second one is that for the same level these radii, the short-range forces are calculated by the direct method or the of accuracy, the TreePM algorithm requires significantly less tree algorithm. White circles represent particles. Blue circles are the multipole operations. With the tree algorithm, the contributions of distant expansions of tree nodes. Red solid arrows and blue dotted arrows show the particle-particle and the particle-multipole interactions, respectively. The (large) cells dominate the error in the calculated force. With residual force is calculated by the PM algorithm. the TreePM algorithm, the contributions of distant particles are calculated using FFT. Thus, we can allow relatively moderate accuracy parameter for the tree part, resulting in considerable by the presence of the boundary to the vacuum. Thus, only a reduction in the computational cost. small fraction of the total computational volume is useful for In this paper, we describe our MPI/OpenMP hybrid TreePM the analysis of the structure formed. On the other hand, with implementation GreeM [24], which is a massively parallel the periodic boundary, everywhere is equally reliable and can TreePM code based on the implementation of Yoshikawa be used for the analysis. & Fukushige (2005) [23] for large cosmological N-body The PM (Particle Mesh) algorithm has been widely used in simulations, and present performance results of the largest cosmological N-body simulations with the periodic boundary gravitational N-body simulation that has ever been done in condition since 1980’s. The PM algorithm can obtain the the world. We use one trillion dark matter particles to study gravitational potential on a regular grid. The mass density at the nonlinear evolution of the first dark matter structures in the a grid point is calculated by assigning the masses of nearby early Universe. The numerical simulations were carried out on particles by some kernel function. Then, the Poisson equation K computer at the RIKEN Advanced Institute for Computa- is solved using FFT. Finally, the gravitational force on a tional Science, which is the world’s fastest supercomputer at particle position is obtained by differentiating and interpolating the time this paper is submitted. It consists of 82944 SPARC64 the potential on the mesh. For details, see Eastwood (1981) VIIIfx oct-core processors with the clock speed of 2.0 GHz [17]. (the total number of core is 663552) and 1.3PB of memory. In general, the PM algorithm is much faster but less accurate The peak performance is 10.6 Pflops. than the tree algorithm since the spatial force resolution is The scientific motivation of this simulation is to study the limited by the size of the mesh. In order to overcome this nature of dark matter particles, which is a long-standing prob- 3 problem, hybrid algorithm such as the P M (Particle-Particle lem in both astrophysics and particle physics. One candidate Particle-Mesh) and the TreePM (Tree Particle-Mesh) algorithm of the dark matter particle is the lightest supersymmetric have been developed. The main idea of these algorithm is that particle, the neutralino. Since the neutralino is itself its anti- the gravitational force is split into two components, short- and particle, it self-annihilates and produces new particles and long-range forces. The short-range force decreases rapidly at gamma-rays. Indirect detection experiments to detect these large distance, and drops zero at a finite distance. This part productions are the important way to study the nature of dark with the cutoff function on the force shape is calculated by matter. Therefore, such indirect detection experiments are one a high resolution algorithm, such as the direct summation of the key projects of the current generation gamma-ray space 3 (P M) or the tree algorithm (TreePM) (e.g. [18]–[24]). The telescope Fermi [27] and the next generation ground-based long-range force drops at large wavenumbers in the frequency chelenkov telescope array CTA1. space, and is calculated by the PM algorithm. Figure 2 shows The Milky Way is in the bottom of the potential of a 3 a schematic view of the P M and the TreePM algorithm.