DOCSLIB.ORG

INRIA Paris/Rocquencourt Conda Packages Documentation Release 1.0.0

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

INRIA Paris/Rocquencourt Conda packages Documentation Release 1.0.0 David Froger Jun 05, 2017 Contents 1 User documentation 3 1.1 Getting started..............................................3 1.2 Package listing..............................................4 1.3 Package details..............................................9 1.4 Linking with packages.......................................... 10 2 Packager documentation 13 2.1 External links............................................... 13 2.2 build.sh guidelines............................................. 13 2.3 Vagrant boxes.............................................. 16 2.4 Package testing.............................................. 17 2.5 Documenting............................................... 17 3 Contact 19 i ii INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0 The INRIA Paris-Rocquencourt SED Conda channel aims at easily distributing scientific softwares and libraries using the Conda package manager. Supported platforms are GNU/Linux 64 bits (old system like CentOS 5 are not supported) and OSX from 10.7 to 10.11. Windows is not supported yet (no plan to support it for now). Root privileges are not required, except for installing compilers in the system. Conda recipes are available. Contents 1 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0 2 Contents CHAPTER 1 User documentation Getting started Prerequisites on GNU/Linux Install gcc, g++, and gfortran. For example, using apt-get: sudo apt-get install gcc g++ gfortran or using yum: yum install gcc-c++ gcc-gfortran Prerequisites on OSX Install Xcode Command Line Tools. Download gfortran (select the gfortran-X.X-bin.tar.gz package) from HPC Mac OS X, and install it: sudo tar xvzf gfortran-X.X-bin.tar.gz -C / Conda installation Download the Conda installer, and install conda: On GNU/Linux 64 bits: wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh bash Miniconda-latest-Linux-x86_64.sh or, on OSX: 3 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0 curl -O https://repo.continuum.io/miniconda/Miniconda-latest-MacOSX-x86_64.sh bash Miniconda-latest-MacOSX-x86_64.sh You will be asked for a place to install miniconda. Either choose the default $HOME/miniconda or a custom place, for example /local/<user>/miniconda. At the end of the installation, you will be asked for modifying your Bash configuration file (~/.bashrc on GNU/Linux, or ~/.bash_profile on OSX): answer yes. To take this modification into account, either open a new terminal, or update the one you are using with: source ~/.bashrc # on GNU/Linux source ~/.bash_profile # on OSX Note: the Conda cheat sheet is a usefull ressource to get started with Conda, (managing environments for example is worth to learn). Installing packages Install a package provided by the The INRIA Paris-Rocquencourt SED Conda channel , for example: conda install -c inria-pro-sed petsc Your may also want to add the inria-pro-sed Conda channel to your configuration (this modify your ~/.condarc): conda config --add channels inria-pro-sed in order to omit the -c option: conda install petsc Package listing Linear algebra libraries Available libraries are: 4 Chapter 1. User documentation INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0 Pack- Description Links age METIS Serial Graph Partitioning and Fill-reducing Matrix Ordering recipe, pack- ages Mumps A parallel sparse direct solver recipe, pack- ages Open Open MPI: A high performance message passing library recipe, MPI pack- ages ParMETIS ParMETIS is an MPI-based parallel library that implements a variety of algorithms for recipe, partitioning unstructured graphs, meshes, and for computing fill-reducing orderings of pack- sparse matrices. ages PETSc PETSc: Portable, Extensible Toolkit for Scientific Computation recipe, pack- ages ScaLA- ScaLAPACK is a library of high-performance linear algebra routines for parallel recipe, PACK distributed memory machines. pack- ages Su- SuperLU is a general purpose library for the direct solution of large, sparse, nonsymmetric recipe, perLU_DISTsystems of linear equations on high performance machines. pack- ages Eigen Eigen is a C++ template library for linear algebra. recipe, pack- ages Note: • Open MPI is build with C, C++ and Fortran API. Relationships between main libraries are (blue indicate package provided by continuum.io). 1.2. Package listing 5 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0 PETSc SuperLU_DIST Mumps ScaLAPACK ParMETIS OpenBLAS OpenMPI METIS dolfin boost qt vtk Eigen3 ffc ply OpenMPI ufl fiat instant six sympy numpy SWIG CMake 6 Chapter 1. User documentation INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0 Image processing libraries (not yet available) Pack- Description Links age Open- OpenMVS (Multi-View Stereo) is a library for computer-vision scientists and recipe, MVS especially targeted to the Multi-View Stereo reconstruction community. packages Open- “open Multiple View Geometry” is a library for computer-vision scientists and recipe, MVG especially targeted to the Multiple View Geometry community. packages GLFW GLFW is an Open Source, multi-platform library for creating windows with OpenGL recipe, package contexts and receiving input and events. OpenExifOpenExif is an object-oriented library for accessing Exif formatted JPEG image files. recipe, package OSI Osi (Open Solver Interface) provides an abstract base class to a generic linear recipe, package programming (LP) solver, along with derived classes for specific solvers. CLP Clp (Coin-or linear programming) is an open-source linear programming solver recipe, package written in C++. Mesa Mesa is an open-source implementation of the OpenGL specification - a system for recipe not rendering interactive 3D graphics. available, package OpenMVS OpenMVG GLFW OpenEXIF CLP OSI Mesa Software engineering tools Available tools: 1.2. Package listing 7 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0 Package Description Links buildbot- Buildbot is an open-source framework for automating software build, test, and recipe, master release processes. packages buildbot- Buildbot is an open-source framework for automating software build, test, and recipe, slave release processes. packages klink A Simple and Clean Sphinx Docs Theme recipe, packages nbtools Tools for managing IPython notebooks recipe, packages Relationships between main libraries are (blue indicate package provided by continuum.io). Buildbot Master python2.7 Twisted Jinja2 dateutil SQLAlchemy Migrate SQLAlchemy Decorator Tempita nbtools Python Jinja2 IPython Notebook matplotlib nose ghdiff chardet File format libraries Available libraries: 8 Chapter 1. User documentation INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0 Pack- Description Links age GetPot Powerful Input File and Command Line Parser recipe, packages h5utils h5utils is a set of utilities for visualization and conversion of scientific data in the free, recipe, portable HDF5 format. packages HDF5 HDF5 is a data model, library, and file format for storing and managing data. recipe, packages LM5 Reads and writes mesh files (.mesh) or physical solution files (.sol) recipe, packages yaml- A YAML parser and emitter in C++ recipe, cpp packages Note: • HDF5 is build with C and C++ API. Python bindings Available Python bindings are: Pack- Description Links age h5py The h5py package is a Pythonic interface to the HDF5 binary data format. recipe, packages mpi4py This package provides Python bindings for the Message Passing Interface (MPI) recipe, standard. packages petsc4py PETSc for Python recipe, packages Package details Integer and real types Numerical algebra libraries make use of 32 bits index integers and 64 bits floating point numbers. Note: OpenBLAS is provided by continuum analytics. The package openblas-0.2.14-3 contains the file in- clude/openblas_config.h. This file does not defined OPENBLAS_USE64BITINT, so blasint is a int (32 bits). C++ standard library on OSX On OSX, two implementations of the C++ standard library are available: • The GNU implementation: libstdc++.dylib • The Clang implementation: libc++.dylib The ABI of these two libraries are incompatibles. All packages in the INRIA Paris-Rocquencourt SED Conda channel are built with the Clang implementation. 1.3. Package details 9 INRIA Paris/Rocquencourt Conda packages Documentation, Release 1.0.0 Note: A Python script (for example), may import a C++ extension linked with libc++.dylib and another C++ exten- sion linked with libstdc++.dylib without problems, as long as the two extensions do not directly exchange C++ data structures. glibc and gcc versions Packages are build on Centos 6.6, using glibc version 2.12, and gcc version 4.4.7. As a consequence, packages will work on all GNU/Linux distribution providing glibc version 2.12 or greater. Taken from distrowatch, here are some examples: distribution glibc gcc centos 5.11 2.5 4.1.2 centos 6.6 2.12 4.4.7 centos 7.0 2.17 4.8.3 ubuntu 12.04 2.15 4.6.3 ubuntu 14.04 2.19 4.8.2 debian 7 2.13 4.7.2 fedora 17 2.15 4.7.0 suse 12.1 2.12.1 4.6.2 Instead of using the system gcc, we may in the future use the gcc conda package, which provide a recent gcc version, linked with an old glibc version. But currently (5 October, 2015), gcc Conda package (4.8.5-3, in asmeurer channel), has too much problems: • https://groups.google.com/a/continuum.io/forum/#!search/froger/conda/Lb_Vnz6O85s/20dsvlvlBwAJ • https://github.com/conda/conda-recipes/issues/435 • https://github.com/conda/conda-recipes/issues/431
Recommended publications
  • Creating Rpms Guide

    Creating Rpms Guide

    CREATING RPMS (Student version) v1.0 Featuring 36 pages of lecture and a 48 page lab exercise This docu m e n t serves two purpose s: 1. Representative sample to allow evaluation of our courseware manuals 2. Make available high quality RPM documentation to Linux administrators A bout this m aterial : The blue background you see simulates the custom paper that all Guru Labs course w are is printed on. This student version does not contain the instructor notes and teaching tips present in the instructor version. For more information on all the features of our unique layout, see: http://ww w . g urulabs.co m /courseware/course w are_layout.php For more freely available Guru Labs content (and the latest version of this file), see: http://www.gurulabs.co m/goodies/ This sample validated on: Red Hat Enterprise Linux 4 & Fedora Core v3 SUSE Linux Enterprise Server 9 & SUSE Linux Professional 9.2 About Guru Labs: Guru Labs is a Linux training company started in 199 9 by Linux experts to produce the best Linux training and course w are available. For a complete list, visit our website at: http://www.gurulabs.co m/ This work is copyrighted Guru Labs, L.C. 2005 and is licensed under the Creative Common s Attribution- NonCom mer cial- NoDerivs License. To view a copy of this license, visit http://creativecom m o n s.org/licenses/by- nc- nd/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 943 0 5, USA. Guru Labs 801 N 500 W Ste 202 Bountiful, UT 84010 Ph: 801-298-5227 WWW.GURULABS.COM Objectives: • Understand
  • Buildbot: a Continuous Integration System

    Buildbot: a Continuous Integration System

    Buildbot: A continuous integration system Krzysztof Voss January, 2013 Outline • Testing and Continuous Integration • Introduction to Buildbot • BuildMaster • BuildMaster components • BuildSlave • Installation and Usage 1 Testing and continuous integration Tests: • the best specification • safety-net for refactoring • bug identification Tests are the most effective if we: • run them often • run them on different machines/environments • can easily see their results 2 The most straightforward approach would entail: • logging in to different machines • fetching the newest source code • running tests • analyzing their output In case we want to test a few environments, repeating the above steps is tedious. Developers do not focus on the code, instead they run tests. A continuous integration system performs all of these steps for us, so developers can focus on their code. 3 Introduction to Buildbot: Features • run builds on a variety of BuildSlave platforms • arbitrary build process: handles projects using C, Python, . • minimal host requirements: python and Twisted • BuildSlave can be behind a firewall if they can still do checkout • status delivery through web page, email, IRC, other protocols • track builds in progress, provide estimated completion time • flexible configuration by subclassing generic build process classes 4 • debug tools to force a new build, submit fake Changes, query BuildSlave status • released under the GPL source: http://buildbot.net/buildbot/docs/current/manual/introduction.html 5 Introduction to Buildbot: Overview system overview source: http://buildbot.net/buildbot/docs/0.8.1/full.html 6 BuildMaster BuildMaster components source: http://buildbot.net/buildbot/docs/0.8.1/full.html 7 BuildMaster BuildMaster: • holds the configuration of the entire system.
  • CERN R&D on Spack and the Turnkey Stack

    CERN R&D on Spack and the Turnkey Stack

    CERN R&D on Spack and the Turnkey Stack HSF Software Workshop - 2020/11/19 Valentin Volkl (CERN), Key4hep Software Group Thanks to Andre Sailer, Pere Mato, Ben Morgan for use of Slides Valentin Volkl: Key4HEP & Spack Table of Contents ● Key4HEP - Introduction and motivation ● The Spack package manager ● Technical considerations ● Deployment to CVMFS ● Workflow -Spack for Development Valentin Volkl: Key4HEP & Spack 2 Dependency Graph of Key4HEP Stack Experiment-specific Packages Valentin Volkl: Key4HEP & Spack 3 Dependency Graph of Key4HEP stack Experiment-specific Packages + HEP-specific packages Valentin Volkl: Key4HEP & Spack 4 Dependency Graph of Key4HEP stack Experiment-specific Packages + HEP-specific Packages + General Purpose Libraries Valentin Volkl: Key4HEP & Spack 5 Requirements for a Build System [ ] Need to be able to scale to a typical experiment software stack [ ] Combinatorics of multiple platforms, versions, Release/Debug ... [ ] Easy deployment to CVMFS [ ] Allow local builds independent of central CVMFS installations [ ] Support software development usecases The KEY4HEP stack contains some 300 packages ● 60 Experiment-specific ● 50 HEP-specific ● 200 System/General Purpose 14 GB install size, some 6h to build on single 4-core machine Valentin Volkl: Key4HEP & Spack 6 Key4HEP ● Future detector studies critically rely on well-maintained software stacks to model detector concepts and to understand a detector’s limitations and physics reach ● We have a scattered landscape of specific software tools on the one hand and integrated
  • Analysis of Devops Tools to Predict an Optimized Pipeline by Adding Weightage for Parameters

    Analysis of Devops Tools to Predict an Optimized Pipeline by Adding Weightage for Parameters

    International Journal of Computer Applications (0975 – 8887) Volume 181 – No. 33, December 2018 Analysis of DevOps Tools to Predict an Optimized Pipeline by Adding Weightage for Parameters R. Vaasanthi V. Prasanna Kumari, PhD S. Philip Kingston Research Scholar, HOD, MCA Project Manager SCSVMV University Rajalakshmi Engineering Infosys, Mahindra City, Kanchipuram College, Chennai Chennai ABSTRACT cloud. Now-a-days more than ever, DevOps [Development + Operations] has gained a tremendous amount of attention in 2. SCM software industry. Selecting the tools for building the DevOps Source code management (SCM) is a software tool used for pipeline is not a trivial exercise as there are plethora’s of tools development, versioning and enables team working in available in market. It requires thought, planning, and multiple locations to work together more effectively. This preferably enough time to investigate and consult other plays a vital role in increasing team’s productivity. Some of people. Unfortunately, there isn’t enough time in the day to the SCM tools, considered for this study are GIT, SVN, CVS, dig for top-rated DevOps tools and its compatibility with ClearCase, Mercurial, TFS, Monotone, Bitkeeper, Code co- other tools. Each tool has its own pros/cons and compatibility op, Darcs, Endevor, Fossil, Perforce, Rational Synergy, of integrating with other tools. The objective of this paper is Source Safe, and GNU Bazaar. Table1 consists of SCM tools to propose an approach by adding weightage to each with weightage. parameter for the curated list of the DevOps tools. 3. BUILD Keywords Build is a process that enables source code to be automatically DevOps, SCM, dependencies, compatibility and pipeline compiled into binaries including code level unit testing to ensure individual pieces of code behave as expected [4].
  • Building Embedded Linux Systems ,Roadmap.18084 Page Ii Wednesday, August 6, 2008 9:05 AM

    Building Embedded Linux Systems ,Roadmap.18084 Page Ii Wednesday, August 6, 2008 9:05 AM

    Building Embedded Linux Systems ,roadmap.18084 Page ii Wednesday, August 6, 2008 9:05 AM Other Linux resources from O’Reilly Related titles Designing Embedded Programming Embedded Hardware Systems Linux Device Drivers Running Linux Linux in a Nutshell Understanding the Linux Linux Network Adminis- Kernel trator’s Guide Linux Books linux.oreilly.com is a complete catalog of O’Reilly’s books on Resource Center Linux and Unix and related technologies, including sample chapters and code examples. ONLamp.com is the premier site for the open source web plat- form: Linux, Apache, MySQL, and either Perl, Python, or PHP. Conferences O’Reilly brings diverse innovators together to nurture the ideas that spark revolutionary industries. We specialize in document- ing the latest tools and systems, translating the innovator’s knowledge into useful skills for those in the trenches. Visit con- ferences.oreilly.com for our upcoming events. Safari Bookshelf (safari.oreilly.com) is the premier online refer- ence library for programmers and IT professionals. Conduct searches across more than 1,000 books. Subscribers can zero in on answers to time-critical questions in a matter of seconds. Read the books on your Bookshelf from cover to cover or sim- ply flip to the page you need. Try it today for free. main.title Page iii Monday, May 19, 2008 11:21 AM SECOND EDITION Building Embedded Linux SystemsTomcat ™ The Definitive Guide Karim Yaghmour, JonJason Masters, Brittain Gilad and Ben-Yossef, Ian F. Darwin and Philippe Gerum Beijing • Cambridge • Farnham • Köln • Sebastopol • Taipei • Tokyo Building Embedded Linux Systems, Second Edition by Karim Yaghmour, Jon Masters, Gilad Ben-Yossef, and Philippe Gerum Copyright © 2008 Karim Yaghmour and Jon Masters.
  • Robust Linux Binaries.Pdf

    Robust Linux Binaries.Pdf

    Robust Linux Binaries How to use Portage to provide a solid base stack for HEP G. Amadio https://xkcd.com/1987 Why do we fall into this kind of situation? ● We want software that’s not part of the system ● Use pip, conda, homebrew, etc to get it somewhere ● Not compiled with same compiler → incompatible ABI ● Often requires setting LD_LIBRARY_PATH and/or PYTHONPATH ● Problems if a package is both in the system and in add-ons ● Updates to the system do not take add-ons into account 3 Classic example: ROOT, Python, and Anaconda ● User has Anaconda installation with Python, wants ROOT ● User then tries to build ROOT with system compiler and link with Python from Anaconda installation ● libPyROOT.so has to link against system’s libstdc++.so and anaconda’s libpython2.7.so ● Problem: libpython2.7.so from Anaconda is not guaranteed to be ABI-compatible with system libraries ● Solution: install GCC from anaconda that was used to build Python and build ROOT with that compiler instead 4 Why LD_LIBRARY_PATH should be avoided ● LD_LIBRARY_PATH is commonly used to add directories to the linker’s search path ● Problem: LD_LIBRARY_PATH takes precedence, overrides important system libraries ○ See e.g. https://sft.its.cern.ch/jira/browse/SPI-1083 ● Solution: use a wrapper script or even better, don’t use LD_LIBRARY_PATH $ ssh lxplus7 $ lsb_release -d Description: CentOS Linux release 7.5.1804 (Core) $ source /cvmfs/sft.cern.ch/lcg/views/LCG_latest/x86_64-centos7-gcc7-opt/setup.sh $ ldd /usr/bin/git linux-vdso.so.1 => (0x00007ffdb55b3000) libpcre.so.1
  • Studying Routing Issues in Vanets by Using NS-3 Bachelor Thesis on Informatics

    Studying Routing Issues in Vanets by Using NS-3 Bachelor Thesis on Informatics

    Studying Routing Issues in VANETs by Using NS-3 Bachelor Thesis on Informatics by Christos Profentzas Thesis supervisor Dr. Periklis Chatzimisios Alexander Technological Educational Institute of Thessaloniki Department of Informatics A.T.E.I. of Thessaloniki P.O. Box 141 GR -547 00 Thessaloniki, Macedonia, Greece November 2012 i Acknowledgements This research project would not have been possible without the sup- port of many people. The author wishes to express his gratitude to his supervisor, Assistant Professor Periklis Chatzimisios (Alexan- der TEI of Thessaloniki, Greece) and Assistant Professor Gennaro Boggia (Politecnico di Bari, Italy) who was abundantly helpful and offered invaluable assistance, support and guidance. Deepest grati- tude are also due to the members of the supervisory committee, Assis- tant Professor Luigi Alfredo Grieco and Ph.D Student Giuseppe Piro without whose knowledge and assistance this study would not have been successful. Special thanks also to all group members of Telematics Lab at the Electrical & Electronics Engineering Depart- ment of Politecnico di Bari, for sharing the literature, invaluable assis- tance and laboratory facilities. The author would also like to convey thanks to the Office of Erasmus Program and Faculty of Alexander Technological Educational Institution of Thessaloniki for providing the financial means. Abstract A Vehicular Ad-hoc Network (VANET) is a system of nodes (vehi- cles) that are being connected with each other by wireless technolo- gies. Usually the nodes are moving with very high speeds and, thus, the topology is unpredictable and frequently changing. Such networks can be stand alone and making paths along vehicles or may be con- nected by an infrastructure internet.
  • NVIDIA CUDA Installation Guide for Linux

    NVIDIA CUDA Installation Guide for Linux

    NVIDIA CUDA Installation Guide for Linux Installation and Verification on Linux Systems DU-05347-001_v11.4 | September 2021 Table of Contents Chapter 1. Introduction........................................................................................................ 1 1.1. System Requirements...............................................................................................................1 1.2. About This Document............................................................................................................... 3 Chapter 2. Pre-installation Actions..................................................................................... 4 2.1. Verify You Have a CUDA-Capable GPU....................................................................................4 2.2. Verify You Have a Supported Version of Linux........................................................................ 5 2.3. Verify the System Has gcc Installed........................................................................................5 2.4. Verify the System has the Correct Kernel Headers and Development Packages Installed........................................................................................................................................5 2.5. Install MLNX_OFED.................................................................................................................. 7 2.6. Choose an Installation Method................................................................................................ 7 2.7. Download
  • This Book Doesn't Tell You How to Write Faster Code, Or How to Write Code with Fewer Memory Leaks, Or Even How to Debug Code at All

    This Book Doesn't Tell You How to Write Faster Code, Or How to Write Code with Fewer Memory Leaks, Or Even How to Debug Code at All

    Practical Development Environments By Matthew B. Doar ............................................... Publisher: O'Reilly Pub Date: September 2005 ISBN: 0-596-00796-5 Pages: 328 Table of Contents | Index This book doesn't tell you how to write faster code, or how to write code with fewer memory leaks, or even how to debug code at all. What it does tell you is how to build your product in better ways, how to keep track of the code that you write, and how to track the bugs in your code. Plus some more things you'll wish you had known before starting a project. Practical Development Environments is a guide, a collection of advice about real development environments for small to medium-sized projects and groups. Each of the chapters considers a different kind of tool - tools for tracking versions of files, build tools, testing tools, bug-tracking tools, tools for creating documentation, and tools for creating packaged releases. Each chapter discusses what you should look for in that kind of tool and what to avoid, and also describes some good ideas, bad ideas, and annoying experiences for each area. Specific instances of each type of tool are described in enough detail so that you can decide which ones you want to investigate further. Developers want to write code, not maintain makefiles. Writers want to write content instead of manage templates. IT provides machines, but doesn't have time to maintain all the different tools. Managers want the product to move smoothly from development to release, and are interested in tools to help this happen more often.
  • Expert Python Programming Third Edition

    Expert Python Programming Third Edition

    Expert Python Programming Third Edition Become a master in Python by learning coding best practices and advanced programming concepts in Python 3.7 Michał Jaworski Tarek Ziadé BIRMINGHAM - MUMBAI Expert Python Programming Third Edition Copyright © 2019 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Kunal Chaudhari Acquisition Editor: Chaitanya Nair Content Development Editor: Zeeyan Pinheiro Technical Editor: Ketan Kamble Copy Editor: Safis Editing Project Coordinator: Vaidehi Sawant Proofreader: Safis Editing Indexer: Priyanka Dhadke Graphics: Alishon Mendonsa Production Coordinator: Shraddha Falebhai First published: September 2008 Second edition: May 2016 Third edition: April 2019 Production reference: 1270419 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78980-889-6 www.packtpub.com To my beloved wife, Oliwia, for her love, inspiration, and her endless patience.
  • The Spack Package Manager: Bringing Order to HPC Software Chaos

    The Spack Package Manager: Bringing Order to HPC Software Chaos

    The Spack Package Manager: Bringing Order to HPC Software Chaos Todd Gamblin Matthew LeGendre Michael R. Collette Gregory L. Lee [email protected] [email protected] [email protected] [email protected] Adam Moody Bronis R. de Supinski Scott Futral [email protected] [email protected] [email protected] Lawrence Livermore National Laboratory ABSTRACT Worse, the space of required builds grows combinatorially with Large HPC centers spend considerable time supporting software for each new configuration parameter. As a result, LLNL staff spend thousands of users, but the complexity of HPC software is quickly countless hours dealing with build and deployment issues. outpacing the capabilities of existing software management tools. Existing package management tools automate parts of the build Scientific applications require specific versions of compilers, MPI, process [2, 10, 11, 12, 23, 24, 38, 39, 41]. For the most part, they and other dependency libraries, so using a single, standard software focus on keeping a single, stable set of packages up to date, and stack is infeasible. However, managing many configurations is they do not handle installation of multiple versions or configurations. difficult because the configuration space is combinatorial in size. Those that do handle multiple configurations typically require that We introduce Spack, a tool used at Lawrence Livermore National package files be created for each combination of options [10, 11, Laboratory to manage this complexity. Spack provides a novel, re- 12, 23], leading to a profusion of files and maintenance issues. cursive specification syntax to invoke parametric builds of packages Some allow limited forms of composition [11, 12, 23], but their and dependencies.
  • Cernvm – a Virtual Software Appliance for LHC Applications

    Cernvm – a Virtual Software Appliance for LHC Applications

    17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 042003 doi:10.1088/1742-6596/219/4/042003 CernVM – a virtual software appliance for LHC applications P Buncic1, C Aguado Sanchez1, J Blomer1, L Franco1, A Harutyunian2,3, P Mato1, Y Yao3 1 CERN, 1211 Geneve 23, Geneva, Switzerland 2Armenian e-Science Foundation, Yerevan, Armenia 3Yerevan Physics Institute after A.I. Alikhanyan, Yerevan, Armenia 4 Lawrence Berkeley National Laboratory, 1 Cyclotron Road, CA 94720 Abstract. CernVM is a Virtual Software Appliance capable of running physics applications from the LHC experiments at CERN. It aims to provide a complete and portable environment for developing and running LHC data analysis on any end-user computer (laptop, desktop) as well as on the Grid, independently of Operating System platforms (Linux, Windows, MacOS). The experiment application software and its specific dependencies are built independently from CernVM and delivered to the appliance just in time by means of a CernVM File System (CVMFS) specifically designed for efficient software distribution. The procedures for building, installing and validating software releases remains under the control and responsibility of each user community. We provide a mechanism to publish pre-built and configured experiment software releases to a central distribution point from where it finds its way to the running CernVM instances via the hierarchy of proxy servers or content delivery networks. In this paper, we present current state of CernVM project and compare performance of CVMFS to performance of traditional network file system like AFS and discuss possible scenarios that could further improve its performance and scalability.