DOCSLIB.ORG

Potential Use of Supercomputing Resources in Europe and in Spain for CMS (Including Some Technical Points)

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Potential use of supercomputing resources in Europe and in Spain for CMS (including some technical points) Presented by Jesus Marco (IFCA, CSIC-UC, Santander, Spain) on behalf of IFCA team, And with tech contribution from Jorge Gomes, LIP, Lisbon, Portugal @ CMS First Open Resources Computing Workshop CERN 21 June 2016 Some background… The supercomputing node at the University of Cantabria, named ALTAMIRA, is hosted and operated by IFCA It is not large (2500 cores) but it is included in the Supercomputing Network in Spain, that has quite significant resources (more than 80K cores) that will be doubled along next year. As these resources are granted on the basis of the scientific interest of the request, CMS teams in Spain could directly benefit of their use "for free". However the request needs to show the need for HPC resources (multicore architecture in CMS case) and the possibility to run in a “existing" framework (OS, username/passw. access, etc.). My presentation aims to cover these points and ask for experience from other teams/experts on exploiting this possibility. The EU PRACE initiative joins many different supercomputers at different levels (Tier-0, Tier-1) and could also be explored. Evolution of the Spanish Supercomputing Network (RES) See https://www.bsc.es/marenostrum-support-services/res Initially (2005) most of the resources were based on IBM-Power +Myrinet Since 2012 many of the centers evolved to use Intel x86 + Infiniband • First one was ALTAMIRA@UC: – 2.500 Xeon E5 cores with FDR IB to 1PB on GPFS • Next: MareNostrum 3 @ BSC – >50.000 cores Xeon E5 with FDR • Latest one: FinisTerrae II @ CESGA – >7.000 cores Xeon E5 2680v3 + FDR) NEXT YEAR: MareNostrum4 @ BSC • x4 current capacity • MN3 nodes will remain available! Access to the Spanish Supercomputing Network (RES) See https://www.bsc.es/marenostrum-support-services/res The RES consists of a distributed virtual infrastructure of supercomputers located in different sites, each of which contributes to the total processing power available to users of different R&D groups in Spain or based in another country but developed by with participation of Spanish researchers RES Call for Projects: Participants are encouraged to submit proposed activities where RES resources and expertise help to accelerate relevant contributions to research areas where intensive computing is an enabling technology Evaluation by an Access Committee, advised by a Technical Experts Panel and a Scientific Experts Panel, of the activity, which must provide detailed objectives and an approximate time line for completion. Access to the Spanish Supercomputing Network (RES) See https://www.bsc.es/marenostrum-support-services/res Activity call is continuously open and the evaluation is every four months. Those activities, whose computing needs require access to resources during more than a period, can request access for up to two periods, reporting to the Access Committee the intermediate results obtained and then, so the Committee could confirm the access the second period. In the activities were access is granted, RES will provide CPU hours in the supercomputers (hardware and basic system software) and know-how of the Operations team (Systems and User Support team) to ensure the correct performance of the user applications. Interest for CMS Addressing peaks of demand on simulation Easiest Use Case? Can be planned in advance and submitted to calls with details Opportunistic use for high granularity jobs In these busy systems, where large parallel jobs take up to 3 days, there is a significant “waiting” time for free Experience in Altamira: we use this time to provide opportunistic resources to long tail of users requesting jobs with 1-16 cores (Initial support by Regional/University funds: “USO OPORTUNISTA DE RECURSOS DE SUPERCOMPUTACION EN SIMULACION OPORTUSIM” “Incentivación de Proyectos de I+D, SODERCAN “) Large Data “fast processing” Exploit Access to storage via Infiniband FDR (for example at IFCA we share GPFS volumes with the Tier-2 system) Most nodes are linked to Geant via dark fiber, data can be moved fast Can be linked to “hot” analysis with high impact if needed Potential Issues POLITICAL/STRATEGY (but likely not to be discussed now…) Resources are limited RES calls are over subscribed Arguments: Scientific Impact, detailed Use Cases Next year MN node renewal will provide at least 2x resources Experience could be extended to other countries, or towards PRACE Local support As stated in the RES call Can be complemented by WLCG experts close to RES Key word: Cost / Efficiency of using Supercomputing resources In many cases the cost of a supercomputing node is similar (IB=10GB Eth) Scale factor is key (operating 50K cores is << 50x operating 1K cores) Need to adapt to a “Fixed” Supercomputing Environment Use a single group for “production” (ssh, not certificates but username/passw) “Encapsulate” (“virtualize”) over existing platform Start a “bridging” project based on Cloud technology How? See next… udocker ([email protected]) • tool to execute content of docker containers in user space when docker is not available • enables download of docker containers from dockerhub • enables execution of docker containers by non‐privileged users • can be used to execute the content of docker containers in Linux batch systems and interactive clusters managed by others • wrapper around other tools to mimic docker capabilities • current version uses proot to provide a chroot like environment without privileges (runs on CentOS 6, CentOS 7, Fedora, Ubuntu) • More info at: • https://www.gitbook.com/book/indigo‐dc/udocker/details • https://indigo‐dc.gitbooks.io/udocker/content/doc/user_manual.html Example of (already proven) path to integration into a “supercomputing framework” using “a la cloud” techniques being explored in the INDIGO‐DataCloud project (contribution from Jorge Gomes, LIP team, Lisbon, Portugal) udocker • Examples: # download, but could also import or load a container exported/save by docker $ udocker.py pull ubuntu:latest $ udocker.py create --name=myubuntu ubuntu:latest # make the host homedir visible inside the container and execute something $ udocker.py run -v $HOME myubuntu /bin/bash <<EOF cat /etc/lsb-release ls -l $HOME EOF Example of (already proven) path to integration into a “supercomputing framework” using “a la cloud” techniques being explored in the INDIGO‐DataCloud project (contribution from Jorge Gomes, LIP team, Lisbon, Portugal) udocker • Examples: # download, but could also import or load a container exported/save by docker $ udocker.py pull fedora:latest $ udocker.py create --name=myfed fedora:latest # install something and run $ udocker.py run myfed /bin/bash yum install –y firefox $ udocker.py run --bindhome --hostauth –hostenv -v /sys -v /proc \ -v /var/run -v /dev --user=myusername --dri myfed firefox Example of (already proven) path to integration into a “supercomputing framework” using “a la cloud” techniques being explored in the INDIGO‐DataCloud project (contribution from Jorge Gomes, LIP team, Lisbon, Portugal) udocker • Everything is stored in the user home dir or some other location • Container layers are download to the user home • Directory trees can be created/extracted from these container layers • proot uses the debugger ptrace mechanism to change pathnames and execute transparently inside a directory tree • No impact on read/write or execution, only impact on system calls using pathnames (ex. open, chdir, etc) • Does not require installation of software in the host system: • udocker is a python script • proot is statically compiled Example of (already proven) path to integration into a “supercomputing framework” using “a la cloud” techniques being explored in the INDIGO‐DataCloud project (contribution from Jorge Gomes, LIP team, Lisbon, Portugal) And then…PRACE ? PRACE Research Infrastructure (http://www.prace-ri.eu/ ) is the top level of the European HPC ecosystem Again, focus is on HPC for very large parallel applications But again, one could try to exploit similar ideas More HPC-oriented architectures (GPU, Power, Blue Gene) Calls are regularly launched There are two levels of resources: Tier-0 systems (>1Petaflop = >50K cores approx.) Also Tier-1 systems Initial Focus: opportunistic use? THANKS FOR YOU INTEREST! 12.
Recommended publications
  • National HPC Task-Force Modeling and Organizational Guidelines

    National HPC Task-Force Modeling and Organizational Guidelines

    FP7-INFRASTRUCTURES-2010-2 HP-SEE High-Performance Computing Infrastructure for South East Europe’s Research Communities Deliverable 2.2 National HPC task-force modeling and organizational guidelines Author(s): HP-SEE Consortium Status –Version: Final Date: June 14, 2011 Distribution - Type: Public Code: HPSEE-D2.2-e-2010-05-28 Abstract: Deliverable D2.2 defines the guidelines for set-up of national HPC task forces and their organisational model. It identifies and addresses all topics that have to be considered, collects and analyzes experiences from several European and SEE countries, and use them to provide guidelines to the countries from the region in setting up national HPC initiatives. Copyright by the HP-SEE Consortium The HP-SEE Consortium consists of: GRNET Coordinating Contractor Greece IPP-BAS Contractor Bulgaria IFIN-HH Contractor Romania TUBITAK ULAKBIM Contractor Turkey NIIFI Contractor Hungary IPB Contractor Serbia UPT Contractor Albania UOBL ETF Contractor Bosnia-Herzegovina UKIM Contractor FYR of Macedonia UOM Contractor Montenegro RENAM Contractor Moldova (Republic of) IIAP NAS RA Contractor Armenia GRENA Contractor Georgia AZRENA Contractor Azerbaijan D2.2 National HPC task-force modeling and organizational guidelines Page 2 of 53 This document contains material, which is the copyright of certain HP-SEE beneficiaries and the EC, may not be reproduced or copied without permission. The commercial use of any information contained in this document may require a license from the proprietor of that information. The beneficiaries do not warrant that the information contained in the report is capable of use, or that use of the information is free from risk, and accept no liability for loss or damage suffered by any person using this information.
  • Soluciones Para Entornos HPC

    Soluciones Para Entornos HPC

    Soluciones para entornos HPC Dr. Abel Francisco Paz Gallardo. IT Manager / Project Leader @ CETA-Ciemat [email protected] V Jornadas de Supercomputación y Avances en Tecnología CETA-Ciemat/ Noviembre 2012 Soluciones para entornos HPC Abel Francisco Paz Gallardo EXTREMADURA RESEARCH CENTER FOR ADVANCED TECHNOLOGIES INDICE 1 HPC… ¿Qué? ¿Cómo? . 2 Computación. (GPGPU,. UMA/NUMA,. etc.) 3 Almacenamiento en HPC . 4 Gestión de recursos . Soluciones para entornos HPC CETA-Ciemat/ Noviembre 2012 Soluciones para entornos HPC Abel Francisco Paz Gallardo 2 EXTREMADURA RESEARCH CENTER FOR ADVANCED TECHNOLOGIES 1 HPC… ¿Qué? ¿Cómo? ¿Qué? o HPC – High Performance Computing (Computación de alto rendimiento) o Objetivo principal: Resolución de problemas complejos ¿Cómo? Tres pilares fundamentales: o Procesamiento = Cómputo o Almacenamiento o Gestión de recursos HPC = + + CETA-Ciemat/ Noviembre 2012 Soluciones para entornos HPC Abel Francisco Paz Gallardo EXTREMADURA RESEARCH CENTER FOR ADVANCED TECHNOLOGIES 2 Computación (GPGPU, NUMA, etc.) ¿Qué es una GPU? Primera búsqueda en 2006: - Gas Particulate Unit Unidad de partículas de gas ¿? GPU = Graphics Processing Unit (Unidad de procesamiento gráfico). CETA-Ciemat/ Noviembre 2012 Soluciones para entornos HPC Abel Francisco Paz Gallardo EXTREMADURA RESEARCH CENTER FOR ADVANCED TECHNOLOGIES 2 Computación (GPGPU, NUMA, etc.) La cuestión es… Si una GPU en un videojuego procesa miles de polígonos, texturas y sombras en tiempo real… ¿Por qué no utilizar esta tecnología para procesamiento de datos? CETA-Ciemat/
  • Innovatives Supercomputing in Deutschland

    Innovatives Supercomputing in Deutschland

    inSiDE • Vol. 6 No.1 • Spring 2008 Innovatives Supercomputing in Deutschland Editorialproposals for the development of soft- Contents ware in the field of high performance Welcome to this first issue of inSiDE in computing. The projects within this soft- 2008. German supercomputing is keep- ware framework are expected to start News ing the pace of the last year with the in autumn 2008 and inSiDE will keep Automotive Simulation Centre Stuttgart (ASCS) founded 4 Gauss Centre for Supercomputing taking reporting on this. the lead. The collaboration between the JUGENE officially unveiled 6 three national supercomputing centres Again we have included a section on (HLRS, LRZ and NIC) is making good applications running on one of the three PRACE Project launched 7 progress. At the same time European main German supercomputing systems. supercomputing has seen a boost. In this In this issue the reader will find six issue you will read about the most recent articles about the various fields of appli- Applications developments in the field in Germany. cations and how they exploit the various Drag Reduction by Dimples? architectures made available. The sec- An Investigation Relying Upon HPC 8 In January the European supercomput- tion not only reflects the wide variety of ing infrastructure project PRACE was simulation research in Germany but at Turbulent Convection in large-aspect-ratio Cells 14 started and is making excellent prog- the same time nicely shows how various Editorial ress led by Prof. Achim Bachem from architectures provide users with the Direct Numerical Simulation Contents the Gauss Centre. In February the fast- best hardware technology for all of their of Flame / Acoustic Interactions 18 est civil supercomputer JUGENE was problems.
  • Recent Developments in Supercomputing

    Recent Developments in Supercomputing

    John von Neumann Institute for Computing Recent Developments in Supercomputing Th. Lippert published in NIC Symposium 2008, G. M¨unster, D. Wolf, M. Kremer (Editors), John von Neumann Institute for Computing, J¨ulich, NIC Series, Vol. 39, ISBN 978-3-9810843-5-1, pp. 1-8, 2008. c 2008 by John von Neumann Institute for Computing Permission to make digital or hard copies of portions of this work for personal or classroom use is granted provided that the copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise requires prior specific permission by the publisher mentioned above. http://www.fz-juelich.de/nic-series/volume39 Recent Developments in Supercomputing Thomas Lippert J¨ulich Supercomputing Centre, Forschungszentrum J¨ulich 52425 J¨ulich, Germany E-mail: [email protected] Status and recent developments in the field of supercomputing on the European and German level as well as at the Forschungszentrum J¨ulich are presented. Encouraged by the ESFRI committee, the European PRACE Initiative is going to create a world-leading European tier-0 supercomputer infrastructure. In Germany, the BMBF formed the Gauss Centre for Supercom- puting, the largest national association for supercomputing in Europe. The Gauss Centre is the German partner in PRACE. With its new Blue Gene/P system, the J¨ulich supercomputing centre has realized its vision of a dual system complex and is heading for Petaflop/s already in 2009. In the framework of the JuRoPA-Project, in cooperation with German and European industrial partners, the JSC will build a next generation general purpose system with very good price-performance ratio and energy efficiency.
  • (PDF) Kostenlos

    (PDF) Kostenlos

    David Gugerli | Ricky Wichum Simulation for All David Gugerli Ricky Wichum The Politics of Supercomputing in Stuttgart David Gugerli, Ricky Wichum SIMULATION FOR ALL THE POLITICS OF SUPERCOMPUTING IN STUTTGART Translator: Giselle Weiss Cover image: Installing the Cray-2 in the computing center on 7 October 1983 (Polaroid UASt). Cover design: Thea Sautter, Zürich © 2021 Chronos Verlag, Zürich ISBN 978-3-0340-1621-6 E-Book (PDF): DOI 10.33057/chronos.1621 German edition: ISBN 978-3-0340-1620-9 Contents User’s guide 7 The centrality issue (1972–1987) 13 Gaining dominance 13 Planning crisis and a flood of proposals 15 5 Attempted resuscitation 17 Shaping policy and organizational structure 22 A diversity of machines 27 Shielding users from complexity 31 Communicating to the public 34 The performance gambit (1988–1996) 41 Simulation for all 42 The cost of visualizing computing output 46 The false security of benchmarks 51 Autonomy through regional cooperation 58 Stuttgart’s two-pronged solution 64 Network to the rescue (1997–2005) 75 Feasibility study for a national high-performance computing network 77 Metacomputing – a transatlantic experiment 83 Does the university really need an HLRS? 89 Grid computing extends a lifeline 95 Users at work (2006–2016) 101 6 Taking it easy 102 With Gauss to Europe 106 Virtual users, and users in virtual reality 109 Limits to growth 112 A history of reconfiguration 115 Acknowledgments 117 Notes 119 Bibliography 143 List of Figures 153 User’s guide The history of supercomputing in Stuttgart is fascinating. It is also complex. Relating it necessarily entails finding one’s own way and deciding meaningful turning points.
  • Application-Level Fault Tolerance and Resilience in HPC Applications

    Application-Level Fault Tolerance and Resilience in HPC Applications

    Doctoral Thesis Application-level Fault Tolerance and Resilience in HPC Applications Nuria Losada 2018 Application-level Fault Tolerance and Resilience in HPC Applications Nuria Losada Doctoral Thesis July 2018 PhD Advisors: Mar´ıaJ. Mart´ın Patricia Gonz´alez PhD Program in Information Technology Research Dra. Mar´ıaJos´eMart´ınSantamar´ıa Dra. Patricia Gonz´alezG´omez Profesora Titular de Universidad Profesora Titular de Universidad Dpto. de Ingenier´ıade Computadores Dpto. de Ingenier´ıade Computadores Universidade da Coru~na Universidade da Coru~na CERTIFICAN Que la memoria titulada \Application-level Fault Tolerance and Resilience in HPC Applications" ha sido realizada por D~na.Nuria Losada L´opez-Valc´arcelbajo nuestra direcci´onen el Departamento de Ingenier´ıade Computadores de la Universidade da Coru~na,y concluye la Tesis Doctoral que presenta para optar al grado de Doctora en Ingenier´ıaInform´aticacon la Menci´onde Doctor Internacional. En A Coru~na,a 26 de Julio de 2018 Fdo.: Mar´ıaJos´eMart´ınSantamar´ıa Fdo.: Patricia Gonz´alezG´omez Directora de la Tesis Doctoral Directora de la Tesis Doctoral Fdo.: Nuria Losada L´opez-Valc´arcel Autora de la Tesis Doctoral A todos los que lo hab´eishecho posible. Acknowledgments I would especially like to thank my advisors, Mar´ıaand Patricia, for all their support, hard work, and all the opportunities they've handed me. I consider my- self very lucky to have worked with them during these years. I would also like to thank Gabriel and Basilio for their collaboration and valuable contributions to the development of this work. I would like to say thanks to all my past and present colleagues in the Computer Architecture Group and in the Faculty of Informatics for their fellowship, support, and all the coffee breaks and dinners we held together.
  • Curriculum Vitae

    Curriculum Vitae

    CURRICULUM VITAE Name: Mateo Valero Position: Full Professor (since 1983) Contact: Phone: +34-93- Affiliation: Universitat Politécnica de Catalunya 4016979 / 6986 (UPC) Fax: +34-93-4017055 Computer Architecture Department E-mail: [email protected] Postal Address: Jordi Girona 1-3, Módulo D6 URL 08034 – Barcelona, Spain www.ac.upc.es/homes/mateo 1. Summary Professor Mateo Valero, born in 1952 (Alfamén, Zaragoza), obtained his Telecommunication Engineering Degree from the Technical University of Madrid (UPM) in 1974 and his Ph.D. in Telecommunications from the Technical University of Catalonia (UPC) in 1980. He has been teaching at UPC since 1974; since 1983 he has been a full professor at the Computer Architecture Department. He has also been a visiting professor at ENSIMAG, Grenoble (France) and at the University of California, Los Angeles (UCLA). He has been Chair of the Computer Architecture Department (1983-84; 1986-87; 1989- 90 and 2001-2005) and the Dean of the Computer Engineering School (1984-85). His research is in the area of computer architecture, with special emphasis on high performance computers: processor organization, memory hierarchy, systolic array processors, interconnection networks, numerical algorithms, compilers and performance evaluation. Professor Valero has co-authored over 600 publications: over 450 in Conferences and the rest in Journals and Books Chapters. He was involved in the organization of more than 300 International Conferences as General Chair (11), including ICS95, ISCA98, ICS99 and PACT01, Steering Committee member (85), Program Chair (26) including ISCA06, Micro05, ICS07 and PACT04, Program Committee member (200), Invited Speaker (70), and Session Chair (61). He has given over 400 talks in conferences, universities and companies.
  • Financing the Future of Supercomputing: How to Increase

    Financing the Future of Supercomputing: How to Increase

    INNOVATION FINANCE ADVISORY STUDIES Financing the future of supercomputing How to increase investments in high performance computing in Europe years Financing the future of supercomputing How to increase investment in high performance computing in Europe Prepared for: DG Research and Innovation and DG Connect European Commission By: Innovation Finance Advisory European Investment Bank Advisory Services Authors: Björn-Sören Gigler, Alberto Casorati and Arnold Verbeek Supervisor: Shiva Dustdar Contact: [email protected] Consultancy support: Roland Berger and Fraunhofer SCAI © European Investment Bank, 2018. All rights reserved. All questions on rights and licensing should be addressed to [email protected] Financing the future of supercomputing Foreword “Disruptive technologies are key enablers for economic growth and competitiveness” The Digital Economy is developing rapidly worldwide. It is the single most important driver of innovation, competitiveness and growth. Digital innovations such as supercomputing are an essential driver of innovation and spur the adoption of digital innovations across multiple industries and small and medium-sized enterprises, fostering economic growth and competitiveness. Applying the power of supercomputing combined with Artificial Intelligence and the use of Big Data provide unprecedented opportunities for transforming businesses, public services and societies. High Performance Computers (HPC), also known as supercomputers, are making a difference in the everyday life of citizens by helping to address the critical societal challenges of our times, such as public health, climate change and natural disasters. For instance, the use of supercomputers can help researchers and entrepreneurs to solve complex issues, such as developing new treatments based on personalised medicine, or better predicting and managing the effects of natural disasters through the use of advanced computer simulations.
  • RES Updates, Resources and Access / European HPC Ecosystem

    RES Updates, Resources and Access / European HPC Ecosystem

    RES updates, resources and Access European HPC ecosystem Sergi Girona RES Coordinator RES: HPC Services for Spain •The RES was created in 2006. •It is coordinated by the Barcelona Supercomputing Center (BSC-CNS). •It forms part of the Spanish “Map of Unique Scientific and Technical Infrastructures” (ICTS). RES: HPC Services for Spain RES is made up of 12 institutions and 13 supercomputers. BSC MareNostrum CESGA FinisTerrae CSUC Pirineus & Canigo BSC MinoTauro UV Tirant UC Altamira UPM Magerit CénitS Lusitania & SandyBridge UMA PiCaso IAC La Palma UZ CaesarAugusta SCAYLE Caléndula UAM Cibeles 1 10 100 1000 TFlop/s (logarithmiC sCale) RES: HPC Services for Spain • Objective: coordinate and manage high performance computing services to promote the progress of excellent science and innovation in Spain. • It offers HPC services for non-profit, open R&D purposes. • Since 2006, it has granted more than 1,000 Million CPU hours to 2,473 research activities. Research areas Hours granted per area 23% AECT 30% Mathematics, physics Astronomy, space and engineering and earth sciences BCV FI QCM 19% 28% Life and health Chemistry and sciences materials sciences RES supercomputers BSC (MareNostrum 4) 165888 cores, 11400 Tflops Main processors: Intel(R) Xeon(R) Platinum 8160 Memory: 390 TB Disk: 14 PB UPM (Magerit II) 3920 cores, 103 Tflops Main processors : IBM Power7 3.3 GHz Memory: 7840 GB Disk: 1728 TB UMA (Picasso) 4016 cores, 84Tflops Main processors: Intel SandyBridge-EP E5-2670 Memory: 22400 GB Disk: 720 TB UV (Tirant 3) 5376 cores, 111,8 Tflops
  • Enabling the Deployment of Virtual Clusters on the VCOC Experiment of the Bonfire Federated Cloud

    Enabling the Deployment of Virtual Clusters on the VCOC Experiment of the Bonfire Federated Cloud

    CLOUD COMPUTING 2012 : The Third International Conference on Cloud Computing, GRIDs, and Virtualization Enabling the Deployment of Virtual Clusters on the VCOC Experiment of the BonFIRE Federated Cloud Raul Valin, Luis M. Carril, J. Carlos Mouri˜no, Carmen Cotelo, Andr´es G´omez, and Carlos Fern´andez Supercomputing Centre of Galicia (CESGA) Santiago de Compostela, Spain Email: rvalin,lmcarril,jmourino,carmen,agomez,[email protected] Abstract—The BonFIRE project has developed a federated OpenCirrus. BonFIRE offers an experimenter control of cloud that supports experimentation and testing of innovative available resources. It supports dynamically creating, updat- scenarios from the Internet of Services research community. ing, reading and deleting resources throughout the lifetime Virtual Clusters on federated Cloud sites (VCOC) is one of the supported experiments of the BonFIRE Project whose main of an experiment. Compute resources can be configured with objective is to evaluate the feasibility of using multiple Cloud application-specific contextualisation information that can environments to deploy services which need the allocation provide important configuration information to the virtual of a large pool of CPUs or virtual machines to a single machine (VM); this information is available to software user (as High Throughput Computing or High Performance applications after the machine is started. BonFIRE also Computing). In this work, we describe the experiment agent, a tool developed on the VCOC experiment to facilitate the supports elasticity within an experiment, i.e., dynamically automatic deployment and monitoring of virtual clusters on create, update and destroy resources from a running node of the BonFIRE federated cloud. This tool was employed in the experiment, including cross-testbed elasticity.
  • Efficient Multithreaded Untransposed, Transposed Or Symmetric Sparse

    Efficient Multithreaded Untransposed, Transposed Or Symmetric Sparse

    Efficient Multithreaded Untransposed, Transposed or Symmetric Sparse Matrix-Vector Multiplication with the Recursive Sparse Blocks Format Michele Martonea,1,∗ aMax Planck Society Institute for Plasma Physics, Boltzmannstrasse 2, D-85748 Garching bei M¨unchen,Germany Abstract In earlier work we have introduced the \Recursive Sparse Blocks" (RSB) sparse matrix storage scheme oriented towards cache efficient matrix-vector multiplication (SpMV ) and triangular solution (SpSV ) on cache based shared memory parallel computers. Both the transposed (SpMV T ) and symmetric (SymSpMV ) matrix-vector multiply variants are supported. RSB stands for a meta-format: it recursively partitions a rectangular sparse matrix in quadrants; leaf submatrices are stored in an appropriate traditional format | either Compressed Sparse Rows (CSR) or Coordinate (COO). In this work, we compare the performance of our RSB implementation of SpMV, SpMV T, SymSpMV to that of the state-of-the-art Intel Math Kernel Library (MKL) CSR implementation on the recent Intel's Sandy Bridge processor. Our results with a few dozens of real world large matrices suggest the efficiency of the approach: in all of the cases, RSB's SymSpMV (and in most cases, SpMV T as well) took less than half of MKL CSR's time; SpMV 's advantage was smaller. Furthermore, RSB's SpMV T is more scalable than MKL's CSR, in that it performs almost as well as SpMV. Additionally, we include comparisons to the state-of-the art format Compressed Sparse Blocks (CSB) implementation. We observed RSB to be slightly superior to CSB in SpMV T, slightly inferior in SpMV, and better (in most cases by a factor of two or more) in SymSpMV.
  • Searching for Genetic Interactions in Complex Disease by Using Distance Correlation

    Searching for Genetic Interactions in Complex Disease by Using Distance Correlation

    Searching for genetic interactions in complex disease by using distance correlation Fernando Castro-Prado, University and Health Research Institute of Santiago de Compostela, Spain. E-mail: [email protected] Javier Costas Health Research Institute of Santiago de Compostela, Spain. and Wenceslao González-Manteiga and David R. Penas University of Santiago de Compostela, Spain. Summary. Understanding epistasis (genetic interaction) may shed some light on the ge- nomic basis of common diseases, including disorders of maximum interest due to their high socioeconomic burden, like schizophrenia. Distance correlation is an association measure that characterises general statistical independence between random variables, not only the linear one. Here, we propose distance correlation as a novel tool for the detection of epistasis from case-control data of single nucleotide polymorphisms (SNPs). This approach will be developed both theoretically (mathematical statistics, in a context of high-dimensional statistical inference) and from an applied point of view (simulations and real datasets). Keywords: Association measures; Distance correlation; Epistasis; Genomics; High- dimensional statistical inference; Schizophrenia 1. Introduction The application field that motivates the present article is going to be explained here- inafter. The starting point is a genomic problem, whose importance and interest will be addressed. In addition, the state of the art on this field of knowledge will be sum- marised; underscoring one of the most recent techniques, which has a strong theoretical basis. Upon this, some hypotheses will be made. arXiv:2012.05285v1 [math.ST] 9 Dec 2020 1.1. Epistasis in complex disease The role of heredity in psychiatry has been studied for almost a century, with Pearson (1931) not having “the least hesitation” in asserting its relevance.