COSMOLOGY WITH THE JUBILEE AND MULTIDARK SIMULATIONS

S. GOTTLOBER1, I. ILIEV2, G. YEPES3 1 Leibniz Institute for Astrophysics Potsdam, An der Sternwarte 16, 14482 Potsdam, Germany 2 Astronomy Centre, Department of Physics Astronomy, Pevensey II Building, University of Sussex, fj Falmer, Brighton BNl 9QH, United Kingdom 3 Grupo de Astrofisica, Universidad Aut6noma de Madrid, Madrid E-28049, Sp ain

We review a series of only simulations with up to 216 and 57 billion particles, performed within the Jubilee and MultiDark projects, respectively. All the data products from these simulations are made available to the community through interactive public databases.

1 Introduction

It is almost 50 years since the cosmic microwave background (CMB) radiation was first detected. This radiation was imprinted on the sky just a few hundred thousand years after the Big Bang. The measured temperature fluctuations of the CMB radiation tell us that shortly after the Big Bang the Universe was almost homogeneous with tiny density fluctuations of the order of 10-5. Starting from these tiny fluctuations generated duringthe early inflationary phase all the observed structures in the universe have been formed: huge clusters of with masses up to a few 1015 solar masses in the knots of the cosmic web which comprise galaxies in a wide range of masses fromdwarfs (109 solar masses) to massive elliptical galaxies (1013 solar masses). Comparing the power spectrum of measured temperature fluctuations with theoretical mod­ els, cosmologists conclude that the Universe is spatially flat and consists at present of about 693 of some unknown , 263 of also unknown Dark Matter and 53 of baryonS'. In the evolved universe one can directly observe the distribution of baryons and indirectly deduce (e.g. by gravitational lensing, velocity measurements, X-ray emission) the distribution of Dark Matter. The formation of structureon large scales is well understood. The initial small perturbations grow by gravitational instability and form bound objects called halos which decouple from the expansion of the universe. These bound objects grow hierarchically by accretion of matter and merging with smaller halos. The gravitational clustering becomes increasingly non-linear. Dark Matter is more abundant than baryons and hence becomes most important for formation of large scale structures where gravity dominates. The vastness of scales and the non-linearity of gravitational clustering are the reasons why numerical simulations and the intensive use of the largest supercomputers are the only methods suited to study the gravitationally driven growth of structures down to the high over densities in halos and the formationof galaxies therein. Cos­ mological simulations follow the clustering of matter by numerically solving the gravitational interaction based on an N-body approach. Additionally one would need to model hydrodynam­ ical processes, radiative cooling, star formation and the feedback of stars, in order to simulate in detail the formation ofgalaxies in their different environments. However, in large volumes and "These are the recent Planck values, the WMAP 5-year values differ slightly, see Table 1 for a large number of galaxiesit is practically impossible to followthe hydrodynamical processes. Therefore, Dark Matter only simulations are performed with very high resolution so that the evolution of halos and subhalos can be followed numerically. These halos and sub-halos host the galaxies. In a second (postprocessing) step the properties of the galaxies can be determined by semi-analytical or abundance matching algorithms. Semi analytical methods are based on the formationhistory of the dark matter halos and on recipes which predict the properties of galaxies formed in these halos. Abundance matching algorithms compare observed luminosity functions with halo mass or velocity functions deduced from dark matter only N-body simulations. In case of subhalos one needs also some information about the evolution, namely the properties of the subhalo at infall. In order to simulate the formation of structure in the universe fulfilling the above require­ ments, a very high mass and spatial resolutions are necessary which imposes a strong challenge for present day computational algorithms. Moreover, such simulations need a large amount of computational time and require huge amounts of available memory as well as large data storage facilities for further analyses of results. Therefore, they can be performed only at the largest su­ percomputer centres. In the followingtwo sections we review briefly two projects of Large Scale Structure formation in Gpc size cosmological volumes, the Jubilee and the MultiDark projects.

2 The Jubilee simulation The Jubilee simulation 7 is a N-body simulation with 60003 (216 billion) particles in a volume of (6 h-1Gpc)3 performed with the CUBEP3M N-body code, a P3M (particle-particle-particle­ mesh) code 2. CUBEP3M calculates the long-range gravity forces on a 2-level mesh and short­ range forces exactiy, by direct summation over local particles. The code is massively-parallel, using hybrid (combining MPI and OpenMP) parallelization and has been shown to scale well up to tens of thousands of computing cores. The cosmological and simulation parameters used in the Jubilee simulation are listed in § 2.1. The minimum resolved halo mass (with 20 particles) 12 -1 is 1.49 x 10 h M0, corresponding to galaxies slightly more massive than the Milky Way. Due to the large volume of the simulation, very massive objects exist so that the high end of the mass function can be determined with unprecedented accuracy. For instance, when comparing with the predictions of the widely-used Tinker mass function, we showed that there is an over prediction of these extreme objects G,s . During the last decade the precision in the determination of the Hubble parameter has reached the per cent level. Within the Jubilee simulation we studied statistical properties of the local Hubble parameter as measured by local observers and showed that the distribution of the local Hubble parameter depends not only on the scale of inhomogeneities, but also on how one defines the positions of observers in the cosmic web and what referenceframe is used 9. The predictions for the local Hubble parameter from the simulation can be compared with observational constraints based on Type Ia supernova (SNia) and CMB observations. Due to cosmic variance, for observers located in random haloes the Hubble constant determined from nearby SNia may differ from that measured from the CMB by ±0.8 per cent at lo- statistical significance. This scatter is too small to significantly alleviate a recently claimed discrepancy between current measurements assuming a fiat ACDM model. However, for observers located in the centres of the largest voids permitted by the standard ACDM model the local Hubble constant measurements could differ from SNia as high as 5 per cent. In future the ISW signal will be used to help to discriminate between cosmological models. At present the ISW effect does not constrain the ACDM model to anything like the precision of the standard datasets (CMB and BAOs). However, for a universe containing an amount of or one with a temporally varying dark energy component, the ISW effect will be an aid in constraining the theoretical models and predictions. For alternative cosmological models a variety of expectations of ISW signal arise. The ISW signal from the Jubilee simulation has been cross-correlated with the catalogue of mock Luminous Red Galaxies (LRGs) The ISW-LRG cross-correlation signal for a full-sky survey and £< 30 is strongest for lowerredshift 8. > bins (z � 0.2 to 0.5), whereas for £ 30 the signal is best observed with surveys covering z � 0.6 - 1.0. 2.1 The Jubilee database The Jubilee database http ://www jubilee-proj. ect . org contains data produced by two sim­ ulations, namely the Small Jubilee simulation ((3.072 h Gpc)3 cubic box with 30723 particles) and the Big Jubilee simulation ((6.0 h-1Gpc)3 cubic box-1 with 60003 particles). Both simulations were done with the CUBEP3M parallel N-body code in the Juropa Supercomputer at the Juelich 0.73, Supercomputer Center (Germany). The simulation parameters are !1m = 0.27, !1A = h = 0.70, !1b = 0.044, as = 0.80, n8 = 0.96, gravitational smoothing 50 h-1kpc comoving, mass resolution 7.49 1010 M , starting redshift 100. x h z The simulations have-1 been0 post-processed to extract= with spherical overdensity halos (AHF and SO) as well as Friends-of-Friends (FOF) algorithms catalogs of halos. These catalogs are very large, containing up to 400 million halos per snapshot. To avoid huge data transfers the user can choose which specific properties she/he wants to download.

3 The MultiDark simulations

The MultiDark simulations are a series of large Dark Matter only simulations summarized in Table 1. These simulations have been performed with different codes: the ART code 1 (first two lines in Table 1) and the Gadget code 5 (all remaining simulations in Table 1). The first MultiDark database4 has been built with the two ART simulations. Accurate predictions of the abundance and clustering of dark matter haloes play a key role in testing the standard cosmological model. However, it poses significant challenges when it comes to testing its predictions forthe distribution and properties of galaxies. The simulations provide very detailed predictions on the distribution of dark matter, but connecting the luminous galaxies with their dark matter haloes is a much more difficult task. There are different possibilities to make this -halo connection. Halo Abundance Matching (HAM) is a simple and yet realistic way to bridge the gap between dark matter haloes and galaxies. One key ingredient of these models are the subhaloes of more massive halos. They must be included in the abundance matching prescription because each lump of dark matter with enough mass and concentration should host a galaxy regardless whether that is the central object or a satellite. However a simple abundance matching model is not applicable because subhalos may loss a substantial fraction of mass due to tidal stripping while the stellar mass hosted by these subhalos is not expected to be reduced by stripping. Therefore, using the properties of the subhalos at infall to the host halo is more appropriate. In the large volume cosmological simulations of the MultiDark project convergence at the 10% accuracy for the abundance of haloes and subhaloes, and the correlation 3 functions can be achieved with � 150 particles per halo • 3.1 The MultiDark database database (www .cosmosim. org) The CosmoSim is the successor of the MultiDark database www. multidark .org). It currently provides access to six cosmological simulations - including( a high-resolution resimulation of selected regions with hydrodynamics and star formation - which originate from the MultiDark and CLUES (www .clues-project . org) projects. Outputs of large cosmological simulations are typically stored at supercomputer centres with restricted access and encompass terabytes of data - too much to be downloaded by everyone. By providing the data via a database everybody can easily access the data, filter or combine the results directly on the server and use them for his own research. Table 1: Numerical and cosmological parameters for the simulations. The columns give the box size, the number of particles, the particle mass, the adopted values for flA the clustering at 1 Mpc, the spectral index and the Hubble parameter nMatter, ilBaryon, ' 8h- us, n, h. box [Mpc/h] particles mp [h-1 Md as ns h :> \J M 1000 2048 8.6 x 10 0.270, 0.047ns 0.730,A 0.82 0.95 0.7 250 20483 1.4 x 108 0.27 0.047 0.73 0.82 0.95 0.7 2500 38403 2.1 x1010 0.27 0.047 0.73 0.82 0.95 0.7 2500 38403 2.2 x 1010 0.29 0.047 0.71 0.82 0.95 0.7 2500 38403 2.2 1010 0.29 0.047 0.71 0.90 0.95 0.7 3 x 10 2500 3840 2.4 x 10 0.31 0.047 0.69 0.82 0.95 0.7 2500 38403 2.4 x 1010 0.31 0.047 0.69 0.82 0.96 0.678 1000 38403 1.5 x 109 0.31 0.047 0.69 0.82 0.96 0.678 400 38403 9.7 x 107 0.31 0.047 0.69 0.82 0.96 0.678

The available data products include catalogues of dark matter halos, their inner properties, merging histories, information about the cosmic web and for selected snapshots even the raw particle distributions allowing for much deeper studies of dark matter halos and their envi­ ronment. All the simulations and database tables are made available through a web interface including an extensive documentation. The increase in resolution for cosmological simulations has led to larger data volumes, with individual tables reaching sizes in the terabyte range. By exploring a new database technol­ ogy, it is now possible to store and analyse snapshots from simulations with nearly 60 billion particles directly in the CosmoSim database. The new database technology, the Spider engine for MariaDB/MySQL, allows to spread the data over many server nodes, and one head node, resulting in a distribution of the computational task over many server nodes.

Acknowledgments

The simulations described here have been performed at the JSC Juelich, the NAS Ames and the LRZ Munich. Publishing the simulations in databases would be impossible without the permanent help of the IT groups in Potsdam and Madrid, special thanks to Kristin Riebe, Harry Anke, Adrian Partl and Fernando Campos.

References

1. S. Gottlober, A. Klypin, "The ART of Cosmological Simulations ", in: High Performance Computing in Science and Engineering, Springer-Verlag, 2008, p.29 2. J. Harnois-Deraps, U.-L. Pen, I. T. Iliev , H. Merz, J. D. Emberson, V. Desjacques, MNRAS 436, 540 (2013) 3. A. Klypin, F. Prada, G. Yepes, S. Hess, S. Gottlober, 2013, arXiv, arXiv:1310.3740 4. K. Riebe, et al. AN 334, 691 (2013) 5. V. Springel, MNRAS 364, 1105 (2005) 6. W. A. Watson, I. T. Iliev, A. D'Aloisio, A. Knebe, P. R. Shapiro, G. Yepes, MNRAS 433, 1230 (2013) 7. W. A. Watson, I. T. Iliev, J. M. Diego, S. Gottliiber, A. Knebe, E. Martinez-Gonzalez, G. Yepes, MNRAS 437, 3776 (2014) 8. W. A. Watson, et al. MNRAS 438, 412 (2014) 9. R. Wojtak, A. Knebe, W. A. Watson, I. T. Iliev, S. HeB, D. Rapetti, G. Yepes, S. Gottliiber, MNRAS 438, 1805 (2014)