ASPERA Meeting, Paris 8 Feb 2008 Structure

MARENOSTRUM at BSC (Barcelona SuperComputing Center) & RES (The Spanish Supercomputing Network)

Prepared by Sergi Girona ASPERA Director Operaciones BSC-CNS Presented by Jesus Marco Meeting, Paris RES Universidad de Cantabria Centro Nacional de Supercomputación

● BSC-CNS mission: ● Research, develop & manage technology to promote science advance

● BSC-CNS objectives: ● R&D in: ● Computational Sciences ● Life Sciences ● Earth Sciences. ● Supercomputing Support to external research.

● BSC-CNS is a Consortium with the following partners: ● Spanish Ministry of Education & Science (MEC) – 51% ● Government of Catalonia (Generalitat, DIUE) – 37% ● Universitat Politècnica de Catalunya (UPC) – 12%

ASPERA Meeting, Paris 8 Feb 2008 Structure

ASPERA Meeting, Paris 8 Feb 2008 Computer Sciences Department

Computer architecture: Benchmarking, analysis and prediction tools: • Highly o-o-o architectures • Tracing scalability • Hardware multithreading • Pattern and structure identification • Design space exploration for multicore • Visualization and analysis chips and Hw accelerators • Processor, memory, network, system • Transactional memory (Hw, Hw-assisted) • SIMD and vector extensions/units The Grid

Programming models: • Scalability of MPI and UPC • OpenMP for multicore, SMP Future Petaflop and ccNUMA systems • DSM for clusters Large cluster • CellSs systems Grid and cluster computing: • Transactional Memory • Programming models • Embedded architectures • Resource management Small DMM • I/O for Grid

Operating environments:

cc-NUMA • Autonomic application servers Chip On-board SMP • Resource management for heterogeneous workloads • Coordinated scheduling and resource management • Parallel file system scalability

ASPERA Meeting, Paris 8 Feb 2008 Life Sciences Department

THE ENCODE THE EARTH PROJECT PROTEOME

● Genome all organisms + all metagenomes ● 1.4x106 sequences ● 32 Tb of RAM used ● 4000 processors ● 38 CPU years ● 9 Tb of output data ● G-Superscalar ● 3.5 days job! ● 5 DAYS lagtime!!!

ASPERA Meeting, Paris 8 Feb 2008 Earth Sciences Department

Resolution improved to 12 km for all Europe, 4 km for the Iberian peninsula, and 1 km for hot spot regions within MareNostrum Supercomputer

Surface tropospheric ozone, 18-21 Jun 2007 ASPERA Meeting, Paris 8 Feb 2008 CASE Department

● Aero-acoustic coupling ● 10,000 processors; 95% parallel efficiency

ASPERA Meeting, Paris 8 Feb 2008 A unique place …………

ASPERA Meeting, Paris 8 Feb 2008 MareNostrum

“It is not the most powerful supercomputer in the world, but it is the most beautiful” (Fortune, Sept. 2006)

ASPERA Meeting, Paris 8 Feb 2008 MareNostrum

● Processors ● 2560 JS21, 2.3GHz ● 4 cores per board ● 8 Gbytes RAM ● 36 Gbytes SAS disk ● Networking ● Myrinet ● 2 Spine 1280

● 10 Clos256 Blade centers Storage servers Gigabit switch ● 2560 Myrinet cards Myrinet racks Operations rack 10/100 switches ● Gigabit ● 10/100 Ethernet 94,20 TFlops ● Disk 10240 processors ● GPFS over 20 nodes 20 TBytes memory 280 + 90 TBytes disk ● OS ● Linux

ASPERA Meeting, Paris 8 Feb 2008 Processor, Blade

4 Processors

8 Gbytes Memory

Daughter card Myrinet

36 Gbytes Disk

ASPERA Meeting, Paris 8 Feb 2008 Blades, blade center and blade center racks

Blade Center • 14 blades per chassis (7U) • 56 processors JS21 Processor Blade • 112GB memory • 2-way 2.3 GHz Power PC 970 MP • Gigabit ethernet switch • 8GB memory (512KB L2 cache) • Local SAS drive (36 GB) • 2x1Gb Ethernet on board • Myrinet daughter card

6 chassis in a rack (42U) • 336 processors • 672GB memory

ASPERA Meeting, Paris 8 Feb 2008 Hardware/Myrinet

Bandwidth: 250 Mbytes Latency: 4 micros

ASPERA Meeting, Paris 8 Feb 2008 Operation: money saving

ASPERA Meeting, Paris 8 Feb 2008 MareNostrum

● MareNostrum Evolution:

World European List rank rank November 2004 4 1 June 2005 5 1 November 2005 8 1 June 2006 11 3 November 2006 5 1 June 2007 9 1 November 2007 13 3

ASPERA Meeting, Paris 8 Feb 2008 MareNostrum, a live Mare

● MareNostrum 20042006 ● 481210240 PowerPCPowerPC 970 970 cores cores ● 24062560 JS20JS21 2.22.3 GHz ● 1020 TB of Memory ● 48 GB per node ● 270380 TB Storage Capacity ● 3 networks ● Myrinet ● Gigabit ● 10/100 Ethernet ● Operating System ● Linux 2.6 (SuSE)

ASPERA Meeting, Paris 8 Feb 2008 Spanish Supercomputing Network / Red Española de Supercomputación RES

MareNostrum Processor: 10240 PowerPC 970 2.3 GHz Memory: 20 TBytes Disk: 280 + 90 TBytes Network: Myrinet, Gigabit, 10/100 System: Linux

UPM Processor: 2408 PowerPC 970 2.2 GHz Memory: 4.7 TBytes Disk: 63 + 47 TBytes Network: Myrinet, Gigabit, 10/100 System: Linux

IAC, UMA, UC, UZ, UV Process: 512 PowerPC 970 2.2 GHz Memory: 1 TByte Disk: 14 + 10 TBytes Network: Myrinet, Gigabit, 10/100 System: Linux

ASPERA Meeting, Paris 8 Feb 2008 Magerit Altamira

Picasso CesarAugusta LaPalma Tirant

ASPERA Meeting, Paris 8 Feb 2008 Red Española de Supercomputación

● Spanish Ministry of Education & Science (MEC) has created a distributed network of supercomputers to support the supercomputing needs of the Spanish research groups. ● Initial nodes of the network are placed at BSC-CNS, at Universidad Politécnica de Madrid (UPM), at IAC (Instituto de Astrofísica de Canarias) and at the Universities of Cantabria, Málaga, Valencia y Zaragoza. ● Access to the resources of the Red Española de Supercomputación is granted to a research group by a Single Access Committee formed by independent prestigious spanish researchers. ● BSC-CNS coordinates the joint technical management of the network.

ASPERA Meeting, Paris 8 Feb 2008 Red Española de Supercomputación

● Sharing computing time ● 20% managed by the hosting institution ● 80% assigned through the Access Committee ● Single Access Committee ● 44 Spanish researchers ● 4 panels ● Astronomy, Space & Earth Sciences ● Biomedicine & Life Sciences ● Physics & Engineering ● Chemistry & Materials ● Resources assigned in four months periods

ASPERA Meeting, Paris 8 Feb 2008 Access Committee

QUALITY OF THE PROJECT (>30%) MPACT OF COMPUTING AT MN HPC EXPERIENCE NEED OF MN (>30%) SUITS THE ARCHITECTURE (>30%)

DETAILED

PROJECT EXPERTS PANEL CAN ASK FOR A PEER REVIEW THROUGH ANEP GENERAL PROJECT EXPERTS PANEL

PROJECT ALREADY EVALUATED POSITIVELY SUPPORT PRIORITIZATION (WRITTEN FORM)

APLICATION

FORM PRIORITIZATION LISTING DECISION ACCESS BSC RESEARCHER BSC IT ANEP COMMITTEE DIRECTION (ACCEPTANCE)

PEER REVIEW

ACUSE DE RECIB COMPROMISO TIEMPO RESPUESTA DEFECTO DE FORMA

QUALITY OF THE SCIENTIFIC PROJECT (>30%) IMPACT OF COMPUTING AT MN GROUP LEVEL HPC EXPERIENCE

ASPERA Meeting, Paris 8 Feb 2008 Evolution of requested/offered hours

Solicitado 30 Ofertado

Millones de horas de Millones 20

Tirant2007 10 CesarAugusta2007 Altamira 2007 Picaso2007 MareNostrum Magerit 2006 2007 LaPalma2007 0 2006Q2 2006Q3 2006Q4 2007P1 2007P2 2007P3 2008P1

ASPERA Meeting, Paris 8 Feb 2008 Research users at MN:

● MN has provided support to more than 300 external projects ● Distribution by scientific area: 35

25 Astronomy, Space 20 and Earth Science Biomedicine and Life 15 Science Physics and 10 Engineering Porcentaje uso Chemistry and Material Science 5

2006 06 2_ P 20 P3_ P4 2006 007 P1 2007 2 2 P ASPERA Meeting, Paris 8 Feb 2008 PRACE: an Strategic European Answer ir1 tier

GENCI Ecosystem ir0 tier

Principal Partners

General Partners

ASPERA Meeting, Paris 8 Feb 2008 European Projects - DEISA

• Consortium of leading national supercomputing centres that currently deploys and operates a persistent, production quality, distributed supercomputing environment with continental scope •Objectives: • To deploy and operate a persistent, production quality, distributed supercomputing environment with continental scope • To enable scientific discovery across a broad spectrum of science and technology. Scientific impact is the only criterion for sucess • Partners: IDRIS – CNRS, Forschungszentrum Jülich, Rechenzentrum Garching of the Max Planck Society, CINECA, CSC, SARA, European Center for Medium – Range Weather Forecasts, Leibniz computing Centre of the Bavarian Academy of Sciences and Humanities, HRLS, Barcelona Supercomputing Center.

ASPERA Meeting, Paris 8 Feb 2008 Mobility Programs: HPC - Europe

• Consortium of six leading High Performance Computing infrastructures and six centers of excellence: BSC - CNS (Barcelona), CINECA (Bologna), EPCC (Edinburgh), HLRS (Stuttgart), IDRIS (Paris) and SARA (Amsterdam).

• Objectives: aims to provide advanced computational services in an integrated manner to the European research community working at the frontiers of science

• Services: It enables researchers working in any eligible country in Europe to visit a participating research institute to carry out a collaborative visit of up to 3 months’ duration, and to gain access to some of the most powerful High Performance Computing facilities in Europe

• Supported Projects: in any field of computational science, including theoretical physics, condensed matter physics, material sciences, computational fluid dynamics, astrophysics, meteo- climatology, computer science, numerical methods in engineering, bio-engineering, computational chemistry, molecular modeling, genomics and proteomics, computational biology and life sciences www.hpc-europa.org

ASPERA Meeting, Paris 8 Feb 2008 What can we offer?

● Access to MareNostrum: Access Committee ● Support on usage ● Support on parallelization

● Mobility programs for doctoral candidates, postdoctoral and senior researchers ● As indicated, through HPC-Europa project (funded by EC) ● As an “Infraestructura Científica y Tecnológica Singular del Estado Español”, MEC offers support to ● Get familiar with the infrastructure ● Perform a scientific experiment

ASPERA Meeting, Paris 8 Feb 2008 Research leaders with access to MareNostrum

ASPERA Meeting, Paris 8 Feb 2008 Thanks ! http://www.bsc.es

ASPERA Meeting, Paris 8 Feb 2008