Evaluation and Benchmarking of Singularity MPI Containers on EU
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Industrial Control Via Application Containers: Migrating from Bare-Metal to IAAS
Industrial Control via Application Containers: Migrating from Bare-Metal to IAAS Florian Hofer, Student Member, IEEE Martin A. Sehr Antonio Iannopollo, Member, IEEE Faculty of Computer Science Corporate Technology EECS Department Free University of Bolzano-Bozen Siemens Corporation University of California Bolzano, Italy Berkeley, CA 94704, USA Berkeley, CA 94720, USA fl[email protected] [email protected] [email protected] Ines Ugalde Alberto Sangiovanni-Vincentelli, Fellow, IEEE Barbara Russo Corporate Technology EECS Department Faculty of Computer Science Siemens Corporation University of California Free University of Bolzano-Bozen Berkeley, CA 94704, USA Berkeley, CA 94720, USA Bolzano, Italy [email protected] [email protected] [email protected] Abstract—We explore the challenges and opportunities of control design full authority over the environment in which shifting industrial control software from dedicated hardware to its software will run, it is not straightforward to determine bare-metal servers or cloud computing platforms using off the under what conditions the software can be executed on cloud shelf technologies. In particular, we demonstrate that executing time-critical applications on cloud platforms is viable based on computing platforms due to resource virtualization. Yet, we a series of dedicated latency tests targeting relevant real-time believe that the principles of Industry 4.0 present a unique configurations. opportunity to explore complementing traditional automation Index Terms—Industrial Control Systems, Real-Time, IAAS, components with a novel control architecture [3]. Containers, Determinism We believe that modern virtualization techniques such as application containerization [3]–[5] are essential for adequate I. INTRODUCTION utilization of cloud computing resources in industrial con- Emerging technologies such as the Internet of Things and trol systems. -
Soluciones Para Entornos HPC
Soluciones para entornos HPC Dr. Abel Francisco Paz Gallardo. IT Manager / Project Leader @ CETA-Ciemat [email protected] V Jornadas de Supercomputación y Avances en Tecnología CETA-Ciemat/ Noviembre 2012 Soluciones para entornos HPC Abel Francisco Paz Gallardo EXTREMADURA RESEARCH CENTER FOR ADVANCED TECHNOLOGIES INDICE 1 HPC… ¿Qué? ¿Cómo? . 2 Computación. (GPGPU,. UMA/NUMA,. etc.) 3 Almacenamiento en HPC . 4 Gestión de recursos . Soluciones para entornos HPC CETA-Ciemat/ Noviembre 2012 Soluciones para entornos HPC Abel Francisco Paz Gallardo 2 EXTREMADURA RESEARCH CENTER FOR ADVANCED TECHNOLOGIES 1 HPC… ¿Qué? ¿Cómo? ¿Qué? o HPC – High Performance Computing (Computación de alto rendimiento) o Objetivo principal: Resolución de problemas complejos ¿Cómo? Tres pilares fundamentales: o Procesamiento = Cómputo o Almacenamiento o Gestión de recursos HPC = + + CETA-Ciemat/ Noviembre 2012 Soluciones para entornos HPC Abel Francisco Paz Gallardo EXTREMADURA RESEARCH CENTER FOR ADVANCED TECHNOLOGIES 2 Computación (GPGPU, NUMA, etc.) ¿Qué es una GPU? Primera búsqueda en 2006: - Gas Particulate Unit Unidad de partículas de gas ¿? GPU = Graphics Processing Unit (Unidad de procesamiento gráfico). CETA-Ciemat/ Noviembre 2012 Soluciones para entornos HPC Abel Francisco Paz Gallardo EXTREMADURA RESEARCH CENTER FOR ADVANCED TECHNOLOGIES 2 Computación (GPGPU, NUMA, etc.) La cuestión es… Si una GPU en un videojuego procesa miles de polígonos, texturas y sombras en tiempo real… ¿Por qué no utilizar esta tecnología para procesamiento de datos? CETA-Ciemat/ -
Portability: Containers, Cloud
JEDI Portability Across Platforms Containers, Cloud Computing, and HPC Mark Miesch, Rahul Mahajan, Xin Zhang, David Hahn, Francois Vandenberg, Jim Rosinski, Dan Holdaway, Yannick Tremolet, Maryam Abdioskouei, Steve Herbener, Mark Olah, Benjamin Menetrier, Anna Shlyaeva, Clementine Gas Academy website http://academy.jcsda.org/june2019 ‣ Instructions for accessing AWS ‣ Activity instructions ‣ Presentation slides ‣ Doxygen documentation for fv3-bundle We will add further content throughout the week Outline I) JEDI Portability Overview ✦ Unified vision for software development and distribution II) Container Fundamentals ✦ What are they? How do they work? ✦ Docker, Charliecloud, and Singularity III) Using the JEDI Containers ✦ How they are built and deployed ✦ Mac and Windows (Vagrant) IV) HPC and Cloud Computing ✦ Environment modules ✦ Containers in HPC? V) Summary and Outlook JEDI Software Dependencies ‣ Essential ✦ Compilers, MPI ✦ CMake Common versions among users ✦ SZIP, ZLIB and developers minimize ✦ LAPACK / MKL, Eigen 3 stack-related debugging ✦ NetCDF4, HDF5 ✦ udunits ✦ Boost (headers only) ✦ ecbuild, eckit, fckit ‣ Useful ✦ ODB-API, eccodes ✦ PNETCDF ✦ Parallel IO ✦ nccmp, NCO ✦ Python tools (py-ncepbufr, netcdf4, matplotlib…) ✦ NCEP libs ✦ Debuggers & Profilers (ddt/TotalView, kdbg, valgrind, TAU…) The JEDI Portability Vision I want to run JEDI on… Development ‣ My Laptop/Workstation/PC ✦ We provide software containers ✦ Mac & Windows system need to first establish a linux environment (e.g. a Vagrant/VirtualBox virtual machine) Development -
The Component Architecture of Open Mpi: Enabling Third-Party Collective Algorithms∗
THE COMPONENT ARCHITECTURE OF OPEN MPI: ENABLING THIRD-PARTY COLLECTIVE ALGORITHMS∗ Jeffrey M. Squyres and Andrew Lumsdaine Open Systems Laboratory, Indiana University Bloomington, Indiana, USA [email protected] [email protected] Abstract As large-scale clusters become more distributed and heterogeneous, significant research interest has emerged in optimizing MPI collective operations because of the performance gains that can be realized. However, researchers wishing to develop new algorithms for MPI collective operations are typically faced with sig- nificant design, implementation, and logistical challenges. To address a number of needs in the MPI research community, Open MPI has been developed, a new MPI-2 implementation centered around a lightweight component architecture that provides a set of component frameworks for realizing collective algorithms, point-to-point communication, and other aspects of MPI implementations. In this paper, we focus on the collective algorithm component framework. The “coll” framework provides tools for researchers to easily design, implement, and exper- iment with new collective algorithms in the context of a production-quality MPI. Performance results with basic collective operations demonstrate that the com- ponent architecture of Open MPI does not introduce any performance penalty. Keywords: MPI implementation, Parallel computing, Component architecture, Collective algorithms, High performance 1. Introduction Although the performance of the MPI collective operations [6, 17] can be a large factor in the overall run-time of a parallel application, their optimiza- tion has not necessarily been a focus in some MPI implementations until re- ∗This work was supported by a grant from the Lilly Endowment and by National Science Foundation grant 0116050. -
Slicing (Draft)
Handling Parallelism in a Concurrency Model Mischael Schill, Sebastian Nanz, and Bertrand Meyer ETH Zurich, Switzerland [email protected] Abstract. Programming models for concurrency are optimized for deal- ing with nondeterminism, for example to handle asynchronously arriving events. To shield the developer from data race errors effectively, such models may prevent shared access to data altogether. However, this re- striction also makes them unsuitable for applications that require data parallelism. We present a library-based approach for permitting parallel access to arrays while preserving the safety guarantees of the original model. When applied to SCOOP, an object-oriented concurrency model, the approach exhibits a negligible performance overhead compared to or- dinary threaded implementations of two parallel benchmark programs. 1 Introduction Writing a multithreaded program can have a variety of very different motiva- tions [1]. Oftentimes, multithreading is a functional requirement: it enables ap- plications to remain responsive to input, for example when using a graphical user interface. Furthermore, it is also an effective program structuring technique that makes it possible to handle nondeterministic events in a modular way; develop- ers take advantage of this fact when designing reactive and event-based systems. In all these cases, multithreading is said to provide concurrency. In contrast to this, the multicore revolution has accentuated the use of multithreading for im- proving performance when executing programs on a multicore machine. In this case, multithreading is said to provide parallelism. Programming models for multithreaded programming generally support ei- ther concurrency or parallelism. For example, the Actor model [2] or Simple Con- current Object-Oriented Programming (SCOOP) [3,4] are typical concurrency models: they are optimized for coordination and event handling, and provide safety guarantees such as absence of data races. -
Assessing Gains from Parallel Computation on a Supercomputer
Volume 35, Issue 1 Assessing gains from parallel computation on a supercomputer Lilia Maliar Stanford University Abstract We assess gains from parallel computation on Backlight supercomputer. The information transfers are expensive. We find that to make parallel computation efficient, a task per core must be sufficiently large, ranging from few seconds to one minute depending on the number of cores employed. For small problems, the shared memory programming (OpenMP) and a hybrid of shared and distributive memory programming (OpenMP&MPI) leads to a higher efficiency of parallelization than the distributive memory programming (MPI) alone. I acknowledge XSEDE grant TG-ASC120048, and I thank Roberto Gomez, Phillip Blood and Rick Costa, scientific specialists from the Pittsburgh Supercomputing Center, for technical support. I also acknowledge support from the Hoover Institution and Department of Economics at Stanford University, University of Alicante, Ivie, and the Spanish Ministry of Science and Innovation under the grant ECO2012- 36719. I thank the editor, two anonymous referees, and Eric Aldrich, Yongyang Cai, Kenneth L. Judd, Serguei Maliar and Rafael Valero for useful comments. Citation: Lilia Maliar, (2015) ''Assessing gains from parallel computation on a supercomputer'', Economics Bulletin, Volume 35, Issue 1, pages 159-167 Contact: Lilia Maliar - [email protected]. Submitted: September 17, 2014. Published: March 11, 2015. 1 Introduction The speed of processors was steadily growing over the last few decades. However, this growth has a natural limit (because the speed of electricity along the conducting material is limited and because a thickness and length of the conducting material is limited). The recent progress in solving computationally intense problems is related to parallel computation. -
Application-Level Fault Tolerance and Resilience in HPC Applications
Doctoral Thesis Application-level Fault Tolerance and Resilience in HPC Applications Nuria Losada 2018 Application-level Fault Tolerance and Resilience in HPC Applications Nuria Losada Doctoral Thesis July 2018 PhD Advisors: Mar´ıaJ. Mart´ın Patricia Gonz´alez PhD Program in Information Technology Research Dra. Mar´ıaJos´eMart´ınSantamar´ıa Dra. Patricia Gonz´alezG´omez Profesora Titular de Universidad Profesora Titular de Universidad Dpto. de Ingenier´ıade Computadores Dpto. de Ingenier´ıade Computadores Universidade da Coru~na Universidade da Coru~na CERTIFICAN Que la memoria titulada \Application-level Fault Tolerance and Resilience in HPC Applications" ha sido realizada por D~na.Nuria Losada L´opez-Valc´arcelbajo nuestra direcci´onen el Departamento de Ingenier´ıade Computadores de la Universidade da Coru~na,y concluye la Tesis Doctoral que presenta para optar al grado de Doctora en Ingenier´ıaInform´aticacon la Menci´onde Doctor Internacional. En A Coru~na,a 26 de Julio de 2018 Fdo.: Mar´ıaJos´eMart´ınSantamar´ıa Fdo.: Patricia Gonz´alezG´omez Directora de la Tesis Doctoral Directora de la Tesis Doctoral Fdo.: Nuria Losada L´opez-Valc´arcel Autora de la Tesis Doctoral A todos los que lo hab´eishecho posible. Acknowledgments I would especially like to thank my advisors, Mar´ıaand Patricia, for all their support, hard work, and all the opportunities they've handed me. I consider my- self very lucky to have worked with them during these years. I would also like to thank Gabriel and Basilio for their collaboration and valuable contributions to the development of this work. I would like to say thanks to all my past and present colleagues in the Computer Architecture Group and in the Faculty of Informatics for their fellowship, support, and all the coffee breaks and dinners we held together. -
Instruction Level Parallelism Example
Instruction Level Parallelism Example Is Jule peaty or weak-minded when highlighting some heckles foreground thenceforth? Homoerotic and commendatory Shelby still pinks his pronephros inly. Overneat Kermit never beams so quaveringly or fecundated any academicians effectively. Summary of parallelism create readable and as with a bit says if we currently being considered to resolve these two machine of a pretty cool explanation. Once plug, it book the parallel grammatical structure which creates a truly memorable phrase. In order to accomplish whereas, a hybrid approach is designed to whatever advantage of streaming SIMD instructions for each faction the awful that executes in parallel on independent cores. For the ILPA, there is one more type of instruction possible, which is the special instruction type for the dedicated hardware units. Advantages and high for example? Two which is already present data is, to process includes comprehensive career related services that instruction level parallelism example how many diverse influences on. Simple uses of parallelism create readable and understandable passages. Also note that a data dependent elements that has to be imported from another core in another processor is much higher than either of the previous two costs. Why the charge of the proton does not transfer to the neutron in the nuclei? The OPENMP code is implemented in advance way leaving each thread can climb up an element from first vector and compare after all the elements in you second vector and forth thread will appear able to execute simultaneously in parallel. To be ready to instruction level parallelism in this allows enormous reduction in memory. -
Introduction to Containers
Introduction to Containers Martin Čuma Center for High Performance Computing University of Utah [email protected] Overview • Why do we want to use containers? • Containers basics • Run a pre-made container • Build and deploy a container • Containers for complex software 06-Nov-20 http://www.chpc.utah.edu Slide 2 Hands on setup 1. Download the talk slides http://home.chpc.utah.edu/~mcuma/chpc/Containers20s.pdf https://tinyurl.com/yd2xtv5d 2. Using FastX or Putty, ssh to any CHPC Linux machine, e.g. $ ssh [email protected] 3. Load the Singularity and modules $ module load singularity 06-Nov-20 http://www.chpc.utah.edu Slide 3 Hands on setup for building containers 1. Create a GitHub account if you don’t have one https://github.com/join {Remember your username and password!} 2. Go to https://cloud.sylabs.io/home click Remote Builder, then click Sign in to Sylabs and then Sign in with GitHub, using your GitHub account 3. Go to https://cloud.sylabs.io/builder click on your user name (upper right corner), select Access Tokens, write token name, click Create a New Access Token, and copy it 4. In the terminal on frisco, install it to your ~/.singularity/sylabs-token file nano ~/.singularity/sylabs-token, paste, ctrl-x to save 06-Nov-20 http://www.chpc.utah.edu Slide 4 Why to use containers? 06-Nov-20 http://www.chpc.utah.edu Slide 5 Software dependencies • Some programs require complex software environments – OS type and versions – Drivers – Compiler type and versions – Software dependencies • glibc, stdlibc++ versions • Other libraries -
Copy, Rip, Burn : the Politics of Copyleft and Open Source
Copy, Rip, Burn Berry 00 pre i 5/8/08 12:05:39 Berry 00 pre ii 5/8/08 12:05:39 Copy, Rip, Burn The Politics of Copyleft and Open Source DAVID M. BERRY PLUTO PRESS www.plutobooks.com Berry 00 pre iii 5/8/08 12:05:39 First published 2008 by Pluto Press 345 Archway Road, London N6 5AA www.plutobooks.com Copyright © David M. Berry 2008 The right of David M. Berry to be identifi ed as the author of this work has been asserted by him in accordance with the Copyright, Designs and Patents Act 1988. British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 978 0 7453 2415 9 Hardback ISBN 978 0 7453 2414 2 Paperback Library of Congress Cataloging in Publication Data applied for This book is printed on paper suitable for recycling and made from fully managed and sustained forest sources. Logging, pulping and manufacturing processes are expected to conform to the environmental standards of the country of origin. The paper may contain up to 70% post consumer waste. 10 9 8 7 6 5 4 3 2 1 Designed and produced for Pluto Press by Chase Publishing Services Ltd, Sidmouth, EX10 9QG, England Typeset from disk by Stanford DTP Services, Northampton Printed and bound in the European Union by CPI Antony Rowe, Chippenham and Eastbourne Berry 00 pre iv 5/8/08 12:05:41 CONTENTS Acknowledgements ix Preface x 1. The Canary in the Mine 1 2. The Information Society 41 3. -
Enabling the Deployment of Virtual Clusters on the VCOC Experiment of the Bonfire Federated Cloud
CLOUD COMPUTING 2012 : The Third International Conference on Cloud Computing, GRIDs, and Virtualization Enabling the Deployment of Virtual Clusters on the VCOC Experiment of the BonFIRE Federated Cloud Raul Valin, Luis M. Carril, J. Carlos Mouri˜no, Carmen Cotelo, Andr´es G´omez, and Carlos Fern´andez Supercomputing Centre of Galicia (CESGA) Santiago de Compostela, Spain Email: rvalin,lmcarril,jmourino,carmen,agomez,[email protected] Abstract—The BonFIRE project has developed a federated OpenCirrus. BonFIRE offers an experimenter control of cloud that supports experimentation and testing of innovative available resources. It supports dynamically creating, updat- scenarios from the Internet of Services research community. ing, reading and deleting resources throughout the lifetime Virtual Clusters on federated Cloud sites (VCOC) is one of the supported experiments of the BonFIRE Project whose main of an experiment. Compute resources can be configured with objective is to evaluate the feasibility of using multiple Cloud application-specific contextualisation information that can environments to deploy services which need the allocation provide important configuration information to the virtual of a large pool of CPUs or virtual machines to a single machine (VM); this information is available to software user (as High Throughput Computing or High Performance applications after the machine is started. BonFIRE also Computing). In this work, we describe the experiment agent, a tool developed on the VCOC experiment to facilitate the supports elasticity within an experiment, i.e., dynamically automatic deployment and monitoring of virtual clusters on create, update and destroy resources from a running node of the BonFIRE federated cloud. This tool was employed in the experiment, including cross-testbed elasticity. -
Efficient Multithreaded Untransposed, Transposed Or Symmetric Sparse
Efficient Multithreaded Untransposed, Transposed or Symmetric Sparse Matrix-Vector Multiplication with the Recursive Sparse Blocks Format Michele Martonea,1,∗ aMax Planck Society Institute for Plasma Physics, Boltzmannstrasse 2, D-85748 Garching bei M¨unchen,Germany Abstract In earlier work we have introduced the \Recursive Sparse Blocks" (RSB) sparse matrix storage scheme oriented towards cache efficient matrix-vector multiplication (SpMV ) and triangular solution (SpSV ) on cache based shared memory parallel computers. Both the transposed (SpMV T ) and symmetric (SymSpMV ) matrix-vector multiply variants are supported. RSB stands for a meta-format: it recursively partitions a rectangular sparse matrix in quadrants; leaf submatrices are stored in an appropriate traditional format | either Compressed Sparse Rows (CSR) or Coordinate (COO). In this work, we compare the performance of our RSB implementation of SpMV, SpMV T, SymSpMV to that of the state-of-the-art Intel Math Kernel Library (MKL) CSR implementation on the recent Intel's Sandy Bridge processor. Our results with a few dozens of real world large matrices suggest the efficiency of the approach: in all of the cases, RSB's SymSpMV (and in most cases, SpMV T as well) took less than half of MKL CSR's time; SpMV 's advantage was smaller. Furthermore, RSB's SpMV T is more scalable than MKL's CSR, in that it performs almost as well as SpMV. Additionally, we include comparisons to the state-of-the art format Compressed Sparse Blocks (CSB) implementation. We observed RSB to be slightly superior to CSB in SpMV T, slightly inferior in SpMV, and better (in most cases by a factor of two or more) in SymSpMV.