German Contributions to the CMS Computing Infrastructure Armin Scheurer, Institut für Experimentelle Kernphysik on behalf of the German CMS community

The Large Hadron Collider & the CMS Experiment CMS Computing, Service and Analysis Challenges

The CMS computing model anticipates various Prompt Reconstruction Calibration hierarchically linked Tier centres to counter the Express-Stream challenges provided by the enormous amounts of Analysis data which will be collected by the CMS detector at RAW Input Data CMS Tier 0 the Large Hadron Collider, LHC, at CERN. During the Storage CAF past years, various computing exercises were T0 – T1: 600 MB/sec LHCb performed to test the readiness of the computing infrastructure, the Grid middleware and the Re-Reco experiments’ software for the startup of the LHC Tier 1 Tier 1 Tier 1 Tier 1 Skims 100 MB/sec Tier 1 Tier 1 which took place in September 2008. Tier 1 Tier 1 ALICE ATLAS T1 – T2: 50 - 500 MB/sec MC Upload In 2006, 2007 and 2008, three of those Computing, (20 MB/sec) Software and Analysis (CSA) Challenges have been Simulation Tier 2 Tier 2 Tier 2 Tier 2 Analysis T 3 T 3 performed. The latter one, CSA 2008, was operated T 3 T 3 User Aerial view of the LHC and its 4 main experiments The Compact Muon Solenoid experiment at a level of 100% of the computing and network resources which are required for the startup of the Schematic view of the CMS computing workflow. The Proton – Proton Collider: 2 General Purpose • Total Weight: 12 500 t • Nations: 38 LHC and the CMS experiment. numbers correspond to the metrics of CSA 2008. • Circumference: 27 km Experiments: • Overall Diameter: 15 m • Institutions: 182 • Beam Energy: 7 TeV • CMS & ATLAS • Overall Length: 21.5 m • Scientists & The whole CMS computing workflow was tested during CSA • Below Surface: 100 m • Readout Channels: > 100 Mio. Engineers: > 2000 2008 simultaneously. The following steps were performed: 2 PB • Temperature: -271 °C 4 Dedicated Experiments: • Magnetic Field: • Production/Simulation at the Tier 2s as input for the Tier 0 • Energy Use: 1 TWh/a • ALICE, LHCb, LHCf, Totem • Solenoid: ~ 4 Tesla • Prompt reconstruction at the Tier 0 • Yoke: ~ 2 Tesla • Re-reconstruction at the Tier 1s CMS Trigger and Event Rate: • Data distribution to the CMS WLCG Tier 1s and Tier 2s Collision Rate: 40 MHz Level 1 Trigger: Hardware • Data analyses Tier Structure Tier at the Tier 2s Tier 2/3 2/3 Tier 2/3 2/3 Cumulative amount of data transferred 60 TB/sec Tier 1 during the 4 weeks of CSA 2008 operation Tier Tier 1 CC-IN2P3 Tier 1 Tier 2/3 France 2/3 CSA Achievements Event Size: 1.5 MB PIC ASGC Goal Status Goal Status Goal Status Service Tier Spain Taiwan Tier 2008 2008 2007 2007 2006 2006 Tier 0 150-300 100 Only at 50 Achieved Achieved 2/3 2/3 Reco Rate Hz Hz bursts Hz Tier 0  Tier 1 600 Achieved 300 Only at 150 High Level Trigger: Achieved Tier 1 Tier 1 Transfer Rate MB/sec partially MB/sec bursts MB/sec Software Tier 1  Tier 2 50-500 20-200 Achieved 10-100 Achieved Achieved FNAL CNAF Transfer Rate MB/sec MB/sec partially MB/sec Tier 1  Tier 1 100 50 Achieved Tier Tier Achieved N/A - United States Italy Transfer Rate MB/sec MB/sec partially Tier 1 Job 50 000 25 000 12 000 3 000 2/3 2/3 Achieved Achieved Tier 0 Submission jobs/day jobs/day jobs/day jobs/day Tier 2 Job 150 000 75 000 20 000 48 000 Tier 1 Achieved Achieved Tier 1 Submission jobs/day jobs/day jobs/day jobs/day Tape & HDD Storage Tier Tier Monte Carlo 1.5 x 109 6 Achieved 50 x 10 Achieved N/A - ~ 1.5 PB/year (CMS) 2/3 RAL Tier 1 NDGF 2/3 Simulation events/year events/month Nordic European United Kingdom Achievements of CMS Computing, Software and Analysis chall- Recorded Events: 150 Hz Countries enges performed in 2008, 2007 and 2006 at levels of 100%, 50% Tier GridKa Tier Germany and 25% of the required resources for the LHC startup, respectively. 2/3 Tier 2 Tier 2 2/3 Aachen DESY Tier 3 German Tier Sites & Resources Karlsruhe Performance During the last years, CMS has tested intensively the computing and network Tier 1: infrastructure of the experiment in order to cope with the challenges provided The German Tier 1 Centre GridKa is located at Forschungszentrum Karlsruhe by the LHC after its startup. It can be seen that all German Tier sites and (KIT, Campus Nord) and operated by the Steinbuch Centre for Computing (SCC). especially the Tier 1 centre GridKa and the Tier 2 at DESY and Aachen GridKa is a multi-VO Tier 1 which supports in total 8 large HEP (and to a lesser performed very well during these tests. Additionally, the daily business of each extent Astroparticle experiments): tier site was covered excellently during a long and stable period of running in 2008 and 2009. 4 LHC experiments: • ALICE, ATLAS, CMS & LHCb German CMS Members Data Distribution: Transfer of cosmic muon datasets from the Tier 0 at CERN to GridKa over 2 4 non-LHC experiments: weeks of stable operation at rates of 10 • BaBar, CDF, Compass & D0 to 20 TB/day.

Network: Storage: CPU: • 10 Gbit: GridKa – CERN • Based on dCache (developed • CMS Resources: Grid-Job Performance: • 3 x 10 Gbit: GridKa – T1 at DESY and FNAL) ~ 800 cores, Cropped ranking-list of the CMS Grid-job • 10 Gbit: GridKa – T2 • CMS Disk: ~ 650 TB ~ 2 GB RAM per core performance from May to August 2008. • CMS Tape: ~ 900 TB The German Tier 1 and Tier 2 centres are Uni Hamburg CMS GridKa resource pledges for the performing well and are ready for incoming collission data at the end of 2009. next three years. In addition, the DESY Hamburg/ German D-Grid initiative provides Zeuthen resources for e.g. the German HEP Monte Carlo & User Analyses: community. Number of running jobs at the German Tier 2 during the last year (March 2008 to February 2009). The plot illustrates different Tier 2: RWTH Aachen FSP-CMS (BMBF) stages of extension of the computing The Tier 2 centre in Germany is a federation of installations at the University of infrastructure during the last year. Aachen and the research centre DESY in Hamburg. The Tier 2 part at DESY is First Real Data: especially built to provide the infrastructure for storage-intense user analyses, Research Centre The first real beam event which was whereas the Aachen part is meant mainly for e.g. Monte Carlo production or Karlsruhe detector calibration tasks which rely on a fast and potent computing recorded by CMS in September 2008. During the 10 days of LHC operation and infrastructure. CMS data taking, a lot of data was Resources of e.g. the Aachen part: Uni Karlsruhe recorded and distributed among the • Based on dCache, about 100 TB disk space different tier sites. GridKa was the first • ~ 360 cores (+ substantial T3 resources integrated centre outside CERN which obtained first coherently) beam data. Share of the Cluster Hardware 2008/2009: • 90% CMS, 5% Auger, 5% IceCube Tier 3s & National Analysis Facility: German CMS Member Institutions: At all CMS member sites in Germany, Resources of e.g Tier 3 Karlsruhe Selected Publications varying amounts of computing and • I. Physikalisches Institut B, RWTH Aachen storage resources are grouped together to Storage: • III. Physikalisches Institut A, RWTH Aachen • Experience Building and Operating the CMS Tier-1 Computing Centers form several Tier 3 centres. They offer • 750 GB HDD space per node • III. Physikalisches Institut B, RWTH Aachen C. Grandi et al., 2009, CHEP 09 • 370 TB Lustre space (accessible via • Algorithms for the Identification of b-Quark Jets with First Data at CMS some of their resources to the whole CMS • Deutsches Elektronen-Synchrotron DESY, Hamburg and Zeuthen community, but are especially built to Infiniband) A. Scheurer, 2008, IEKP-KA/2008-19 provide reliable and fast access to the • Institut für Experimentalphysik, Universität Hamburg • Challenges of the LHC Computing Grid by the CMS experiment huge amount of CMS data for local users CPU: • Institut für Experimentelle Kernphysik, Universität Karlsruhe A. Scheurer, M. Ernst, A. Flossdorf, C. Hof, T. Kress, K. Rabbertz, G. Quast and German CMS members. In addition, a • 200 worker nodes (1600 cores) German e-Science, 2007, ID: 316623.0 • 2 GB RAM per core so-called National Analysis Facility (NAF) • CMS Computing, Software and Analysis Challenge in 2006 (CSA06) Summary was set-up at the research centre DESY. The CMS Collaboration, 2007, CERN/LHCC 2007-010