DOCSLIB.ORG

Integrationofscientificandenginee

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Integrationofscientificandenginee Integraton of scientfc and engineering applicatons into CERN’s computng resources Markus Jylhänkangas Bachelor’s thesis May 2019 Informaton and communicaton technology Degree Programme in Sofware Engineering Description Author(s) Type of publication Date Jylhänkangas, Markus Bachelor’s thesis May 2019 Language of publication: English Number of pages Permission for web publi- 47 cation: x Title of publication Integration of scientific and engineering applications into CERN’s computing resources Degree programme Software Engineering Supervisor(s) Kokkonen, Tero Rantonen, Mika Assigned by CERN Abstract Following CERN’s strategy to consolidate all computing resources under Linux, it was planned to move the Windows HPC service to Linux-based infrastructure. The scientific and engineering applications that were running under Windows HPC were therefore assigned to be migrated to Linux HPC and HTC environments. The main objective is the migration of engineering applications to Linux-based computing resources. The first objective for the migration was to design a strategy to carry out the mi- gration by identifying potential risks and select target service HPC or HTC, to make efficient use of resources. Second objective was to implement templates, plugins and necessary changes to the HTC and HPC infrastructure. Migration was also going to be tested and vali- dates with users. The migration was implemented by first installing the applications on the Linux resources. Template script were written to submit applications to HPC or HTC. Modification to HPC were made with Puppet to make it possible to run the applications As a result, the applications were migrated to either HPC or HTC resources and the Win- dows HPC was deprecated. Authentication and security inside the Linux HPC cluster was also improved. Running complex scientific and engineering applications on Linux HPC and HTC resources is possible, but there are some compatibility problems with different parallel computing im- plementations. Keywords/tags (subjects) HPC, HTC, SLURM, HTCondor, Batch Miscellaneous (Confidential information) Kuvailulehti Tekijä(t) Julkaisun laji Päivämäärä Jylhänkangas, Markus Opinnäytetyö, AMK Toukokuu 2019 Sivumäärä Julkaisun kieli 47 Englanti Verkkojulkaisulupa myönnetty: x Työn nimi Integration of scientific and engineering applications into CERN’s computing resources Tutkinto-ohjelma Ohjelmistotekniikka Työn ohjaaja(t) Tero Kokkonen Mika Rantonen Toimeksiantaja(t) CERN Tiivistelmä CERNin strategiaa tietojenkäsittelyn vakauttamisesta seuraten Windows HPC-palvelu suun- niteltiin siirrettäväksi Linux-pohjaiseen infrastruktuuriin. Sen vuoksi tehtäväksi annettiin tieteellisten ja teknisten sovellusten siirtäminen Windows HPC-ympäristöstä Linux HPC ja HTC-ympäristöön. Päätehtävänä sovellusten siirrossa oli teknisten sovellusten siirto Linux-pohjaisiin tieto- jenkäsittelyresursseihin. Ensimmäisenä̈ tehtävänä siirrossa oli suunnitella strategia siirrosta ja siirtoon liittyvistä mahdollisista riskeistä sekä valita kohdeympäristö HPC tai HTC. Toi- senä tehtävänä oli toteuttaa ja kehittää sapluunat, lisäosat sekä tarvittavat muutokset HTC- ja HPC-infrastruktuureihin. Siirron lopputulokset testattiin ja vahvistettiin käyttäjien kanssa. Siirto suoritettiin ensin asentamalla sovellukset Linux resursseille. Sovellusten suoritta- mista varten kehitettiin sapluunat, joilla voitiin lähettää töitä Linux-resursseihin. Jotta so- vellukset saatiin toimimaan, muutoksia piti tehdä HPC-ympäristöön. Nämä muutokset suo- ritettiin Puppetilla. Tuloksena sovellukset siirrettiin joko HPC- tai HTC-resursseille ja Windows HPC-palvelu otettiin pois käytöstä. Käyttäjien identiteetin todennusta sekä turvallisuutta parannettiin Linux HPC-klusterissa. Monimutkaisten tieteellisten ja teknisten sovellusten suorittaminen Linux HPC- ja HTC- resursseilla on mahdollista, mutta näiden välillä on pieniä yhteensopivuusongelmia. Avainsanat (asiasanat) HPC, HTC, SLURM, HTCondor, Batch Muut tiedot (salassa pidettävät liitteet) 1 Contents Figures................................................................................................................. 3 Tables ................................................................................................................... 3 1 Introducton...................................................................................................... 5 1.1 Motvaton and background ............................................................. 5 1.2 CERN........................................................................................................... 5 1.3 Scope and objectves ........................................................................... 6 2 Research methodology................................................................................ 8 2.1 Constructve research.......................................................................... 8 2.2 Research process................................................................................... 8 3 Private cloud..................................................................................................... 10 3.1 Agile Infrastructure............................................................................... 10 3.2 OpenStack ................................................................................................ 11 3.3 Confguraton tools............................................................................... 12 3.4 Ceph............................................................................................................ 12 3.5 Version control ....................................................................................... 12 4 Computng systems ....................................................................................... 13 4.1 Computng at CERN.............................................................................. 13 4.2 High-Performance Computng......................................................... 13 4.2.1Windows HPC cluster ................................................................ 14 4.2.2Linux HPC cluster......................................................................... 15 4.3 Slurm .......................................................................................................... 17 4.4 Linux Control Groups........................................................................... 18 2 4.5 High-Throughput Computng ........................................................... 18 4.6 Scientfc and engineering applicatons........................................ 19 5 Migraton Plan ................................................................................................. 21 6 Development workfow............................................................................... 23 7 Migraton............................................................................................................ 25 7.1 HTC and HTCondor ............................................................................... 25 7.2 HPC and Slurm........................................................................................ 25 7.3 Slurm PAM................................................................................................ 27 7.3.1Puppet ............................................................................................. 29 7.4 Applicatons............................................................................................. 30 7.4.1CST..................................................................................................... 30 7.4.2Ansys................................................................................................. 31 7.4.3Ansys RSM HTCondor plugin .................................................. 33 7.5 AUKS ........................................................................................................... 36 7.5.1Puppet module............................................................................. 38 8 Results................................................................................................................. 40 9 Conclusions ....................................................................................................... 43 References ......................................................................................................... 47 3 Figures Figure 1. Agile infrastructure................................................................................ 10 Figure 2. Openstack .............................................................................................. 11 Figure 3. Windows cluster.................................................................................... 15 Figure 4. Linux cluster........................................................................................... 17 Figure 5. HPC or HTC migraton process.............................................................. 21 Figure 6. Development environment .................................................................. 23 Figure 7. Ansys Workbench update..................................................................... 32 Figure 8. Auks ........................................................................................................ 37 Tables Table 1. HPC node types ...................................................................................... 16 Table 2. HTC node types....................................................................................... 18 Table 3. What batch system applicaton should run.........................................
Load more

Recommended publications
  • Version 7.8-Systemd
    Linux From Scratch Version 7.8-systemd Created by Gerard Beekmans Edited by Douglas R. Reno Linux From Scratch: Version 7.8-systemd by Created by Gerard Beekmans and Edited by Douglas R. Reno Copyright © 1999-2015 Gerard Beekmans Copyright © 1999-2015, Gerard Beekmans All rights reserved. This book is licensed under a Creative Commons License. Computer instructions may be extracted from the book under the MIT License. Linux® is a registered trademark of Linus Torvalds. Linux From Scratch - Version 7.8-systemd Table of Contents Preface .......................................................................................................................................................................... vii i. Foreword ............................................................................................................................................................. vii ii. Audience ............................................................................................................................................................ vii iii. LFS Target Architectures ................................................................................................................................ viii iv. LFS and Standards ............................................................................................................................................ ix v. Rationale for Packages in the Book .................................................................................................................... x vi. Prerequisites
    [Show full text]
  • Pluggable Authentication Modules
    Who this book is written for This book is for experienced system administrators and developers working with multiple Linux/UNIX servers or with both UNIX and Pluggable Authentication Windows servers. It assumes a good level of admin knowledge, and that developers are competent in C development on UNIX-based systems. Pluggable Authentication Modules PAM (Pluggable Authentication Modules) is a modular and flexible authentication management layer that sits between Linux applications and the native underlying authentication system. The PAM framework is widely used by most Linux distributions for authentication purposes. Modules Originating from Solaris 2.6 ten years ago, PAM is used today by most proprietary and free UNIX operating systems including GNU/Linux, FreeBSD, and Solaris, following both the design concept and the practical details. PAM is thus a unifying technology for authentication mechanisms in UNIX. This book provides a practical approach to UNIX/Linux authentication. The design principles are thoroughly explained, then illustrated through the examination of popular modules. It is intended as a one-stop introduction and reference to PAM. What you will learn from this book From Technologies to Solutions • Install, compile, and configure Linux-PAM on your system • Download and compile third-party modules • Understand the PAM framework and how it works • Learn to work with PAM’s management groups and control fl ags • Test and debug your PAM confi guration Pluggable Authentication Modules • Install and configure the pamtester utility
    [Show full text]
  • Security Guide
    Fedora 19 Security Guide A Guide to Securing Fedora Linux Johnray Fuller John Ha David O'Brien Scott Radvan Eric Christensen Adam Ligas Murray McAllister Scott Radvan Daniel Walsh Security Guide Dominick Grift Eric Paris James Morris Fedora 19 Security Guide A Guide to Securing Fedora Linux Edition 19.1 Author Johnray Fuller [email protected] Author John Ha [email protected] Author David O'Brien [email protected] Author Scott Radvan [email protected] Author Eric Christensen [email protected] Author Adam Ligas [email protected] Author Murray McAllister [email protected] Author Scott Radvan [email protected] Author Daniel Walsh [email protected] Author Dominick Grift [email protected] Author Eric Paris [email protected] Author James Morris [email protected] Copyright © 2007-2013 Fedora Project Contributors. The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. The original authors of this document, and Red Hat, designate the Fedora Project as the "Attribution Party" for purposes of CC-BY-SA. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
    [Show full text]
  • SUSE Linux Enterprise Server 15 SP2 Security and Hardening Guide Security and Hardening Guide SUSE Linux Enterprise Server 15 SP2
    SUSE Linux Enterprise Server 15 SP2 Security and Hardening Guide Security and Hardening Guide SUSE Linux Enterprise Server 15 SP2 Introduces basic concepts of system security, covering both local and network security aspects. Shows how to use the product inherent security software like AppArmor, SELinux, or the auditing system that reliably collects information about any security-relevant events. Supports the administrator with security-related choices and decisions in installing and setting up a secure SUSE Linux Enterprise Server and additional processes to further secure and harden that installation. Publication Date: September 24, 2021 SUSE LLC 1800 South Novell Place Provo, UT 84606 USA https://documentation.suse.com Copyright © 2006– 2021 SUSE LLC and contributors. All rights reserved. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”. For SUSE trademarks, see https://www.suse.com/company/legal/ . All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its aliates. Asterisks (*) denote third-party trademarks. All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its
    [Show full text]
  • Guide to the Secure Configuration of Red Hat Enterprise Linux 5
    Guide to the Secure Configuration of Red Hat Enterprise Linux 5 Revision 4.2 August 26, 2011 Operating Systems Division Unix Team of the Systems and Network Analysis Center National Security Agency 9800 Savage Rd. Suite 6704 Ft. Meade, MD 20755-6704 2 Warnings Do not attempt to implement any of the recommendations in this guide without first testing in a non- production environment. This document is only a guide containing recommended security settings. It is not meant to replace well- structured policy or sound judgment. Furthermore this guide does not address site-specific configuration concerns. Care must be taken when implementing this guide to address local operational and policy concerns. The security changes described in this document apply only to Red Hat Enterprise Linux 5. They may not translate gracefully to other operating systems. Internet addresses referenced were valid as of 1 Dec 2009. Trademark Information Red Hat is a registered trademark of Red Hat, Inc. Any other trademarks referenced herein are the property of their respective owners. Change Log Revision 4.2 is an update of Revision 4.1 dated February 28, 2011. Added section 2.5.3.1.3, Disable Functionality of IPv6 Kernel Module Through Option. Added discussion to section 2.5.3.1.1, Disable Automatic Loading of IPv6 Kernel Module, indicating that this is no longer the preferred method for disabling IPv6. Added section 2.3.1.9, Set Accounts to Disable After Password Expiration. Revision 4.1 is an update of Revision 4 dated September 14, 2010. Added section 2.2.2.6, Disable All GNOME Thumbnailers if Possible.
    [Show full text]
  • Security Guide Security Guide SUSE Linux Enterprise Server 12 SP4
    SUSE Linux Enterprise Server 12 SP4 Security Guide Security Guide SUSE Linux Enterprise Server 12 SP4 Introduces basic concepts of system security, covering both local and network security aspects. Shows how to use the product inherent security software like AppArmor or the auditing system that reliably collects information about any security-relevant events. Publication Date: September 24, 2021 SUSE LLC 1800 South Novell Place Provo, UT 84606 USA https://documentation.suse.com Copyright © 2006– 2021 SUSE LLC and contributors. All rights reserved. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”. For SUSE trademarks, see https://www.suse.com/company/legal/ . All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its aliates. Asterisks (*) denote third-party trademarks. All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its aliates, the authors nor the translators shall be held liable for possible errors or the consequences thereof. Contents About This Guide xvi 1 Available Documentation xvi 2 Giving Feedback xviii 3 Documentation Conventions xviii 4 Product Life Cycle
    [Show full text]
  • Security and Hardening Guide Security and Hardening Guide SUSE Linux Enterprise Server 15 SP1
    SUSE Linux Enterprise Server 15 SP1 Security and Hardening Guide Security and Hardening Guide SUSE Linux Enterprise Server 15 SP1 Introduces basic concepts of system security, covering both local and network security aspects. Shows how to use the product inherent security software like AppArmor, SELinux, or the auditing system that reliably collects information about any security-relevant events. Supports the administrator with security-related choices and decisions in installing and setting up a secure SUSE Linux Enterprise Server and additional processes to further secure and harden that installation. Publication Date: September 24, 2021 SUSE LLC 1800 South Novell Place Provo, UT 84606 USA https://documentation.suse.com Copyright © 2006– 2021 SUSE LLC and contributors. All rights reserved. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”. For SUSE trademarks, see https://www.suse.com/company/legal/ . All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its aliates. Asterisks (*) denote third-party trademarks. All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its
    [Show full text]
  • Open Source Licenses
    Threat Protection System v5.1.0 Open Source Licenses The TippingPoint Threat Protection System (TPS) devices use open source components. Many open source license agreements require user documentation to contain notification that the open source software is included in the product. For inquiries about acquiring license code, contact support. The following agreements are for software that this product includes or may include: • "BIND License Agreement" on page 9 • "boost License Agreement" on page 10 • "coreutils License Agreement regarding coreutils & libmspack" on page 11 • "corosync License Agreement" on page 20 • "cpputest License Agreement" on page 21 • "License Agreement regarding dbus; ecryptfs-utils; gdb; glibc; gptfdisk; ipset; keyutils; libcgi; libnih; libstatgrab; linux; lm-sensors; lttng; mxml; net-tools; open-vm-tools; pam-tacplus; rng-tools; RRDTool; syslog-ng; upstart & util-linux" on page 22 • "gSoap License Agreement" on page 26 • "host-sflow License Agreement" on page 33 • "ipmitool License Agreement" on page 38 • "jitterentropy-rngd License Agreement" on page 39 • "libpcap License Agreement" on page 40 • "libxml License Agreement" on page 41 • "NTP License Agreement" on page 42 • "OpenSSH License Agreement" on page 43 • "License Agreement regarding OpenSSL & openssl-fips" on page 49 • "qDecoder License Agreement" on page 52 • "shadow License Agreement" on page 53 • "tcpdump License Agreement" on page 55 TippingPoint acknowledges that the following open source components may be used in this product: • Adaptive Public License
    [Show full text]
  • Containers in Controls Workshop Rémi Voirin - CERN Accelerator Controls (BE-CO) [email protected] ICALEPCS 2019 - Brooklyn, NY
    Containers in Controls Workshop Rémi Voirin - CERN Accelerator Controls (BE-CO) [email protected] ICALEPCS 2019 - Brooklyn, NY Sunday 6th October, 2019 Containers in Controls Workshop 2 Outline Just a reminder... Defining containerisation Impacts of containerisation Hands-on Docker Container security Sunday 6th October, 2019 Containers in Controls Workshop 3 Table of Contents Just a reminder... Defining containerisation Impacts of containerisation Hands-on Docker Container security Sunday 6th October, 2019 Containers in Controls Workshop 4 Just a reminder... • This workshop is about sharing knowledge and ideas • Informal discussions • Something unclear or anything to add? Please ask questions during my presentations! Sunday 6th October, 2019 Containers in Controls Workshop 5 Table of Contents Just a reminder... Defining containerisation Impacts of containerisation Hands-on Docker Container security Sunday 6th October, 2019 Containers in Controls Workshop 6 Where we come from: Linux namespaces • Introduced in 2002 in kernel 2.4.19 • Goal: isolate processes - no interference with the rest of the system • The mount namespace came first (think chroot) • There are other namespaces including: • PID (CLONE_NEWPID) • network (CLONE_NEWNET) • cgroups (CLONE_NEWNS) Sunday 6th October, 2019 Containers in Controls Workshop 7 Where we come from: cgroups • Cap resource usage • CPU • Memory • Disk I/O • Create a rule in a config file ("not more than 200 MB of RAM"), and then apply it to a process Sunday 6th October, 2019 Containers in Controls Workshop 8
    [Show full text]
  • Proxmox Virtual Environment
    DATASHEET Proxmox Virtual Environment OVERVIEW AT A GLANCE Proxmox VE is a complete virtualization management solution for Complete virtualization solution servers. You can virtualize even the most demanding application for production environments workloads running on Linux and Windows Servers. It combines the leading Kernel-based Virtual Machine (KVM) hypervisor and container- KVM hypervisor based virtualization on one management platform. Lightweight Linux Containers Thanks to the unique multi-master design there is no need for an (LXC) additional management server. This saves ressources and also allows Web-based Management high availabilty without single point of failures (no SPOF). Interface With the included web-based management you can easily control all Comprehensive management functionality. Full access to all logs from all nodes in a cluster is included, feature set including task logs like running backup/restore processes, live-migration Multi-node High Availability or high availability (HA) triggered activities. Clusters VM Templates and Clones ENTERPRISE-READY Multiple storage types supported Proxmox VE includes all the functionality you need to deploy an like Ceph, NFS, ZFS, Gluster, enterprise-class virtualization environment in your company‘s iSCSI,... datacenter. Multiple authentication sources combined with role based Open source license GNU AGPL, user- and permission management enable full control of your HA v3 virtualization cluster. The RESTful web API enables easy integration for third party management tools like custom hosting environments. With the future-proof open source development model, your full access to the source code as well as maximum flexibility and security are guaranteed. ABOUT PROXMOX Proxmox Server Solutions GmbH is a privately held corporation based in Vienna, Austria.
    [Show full text]
  • Plan 9 Authentication in Linux
    Plan 9 Authentication in Linux ∗ Ashwin Ganti Google Inc. [email protected] ABSTRACT password and various information about the user (such as In Linux, applications like su and login currently run as her home directory and default shell). Applications heavily root in order to access authentication information and set depended on these files which led to less extensible code. or alter the identity of the process. In such cases, if the ap- Moreover if a new authentication mechanism was introduced plication is compromised while running as a privileged user, it required all the applications (like login, su etc.) that use the entire system can become vulnerable. An alternative the authentication information be rewritten to support it. approach is taken by the Plan 9 operating system from Bell Labs, which runs such applications as a non-privileged user Pluggable Authentication Modules (PAM) [10] [2] were cre- and relies on a kernel-based capability device working in co- ated to provide more effective ways to authenticate users. ordination with an authentication server to provide the same PAM provides a generic framework enabling uniform au- services. This avoids the risk of an application vulnerability thentication of user applications. It enables user applica- becoming a system vulnerability. tions to perform authentication without actually knowing the implementation details of the underlying mechanisms. This paper discusses the extension of Linux authentication It is usually done through a password based authentication mechanisms to allow the use of the Plan 9 approach with mechanism but also supports a challenge response interac- existing Linux applications in order to reduce the security tion.
    [Show full text]
  • Proxmox VE Datasheet 1/4 © Proxmox Server Solutions Gmbh | KEY FEATURES
    Datasheet Proxmox Virtual Environment 6 AT A GLANCE OVERVIEW • Complete open-source platform for Proxmox VE is a complete open-source solution for enterprise enterprise virtualization virtualization that tightly integrates KVM hypervisor and LXC containers, software-defined storage and networking functionality on a single • Easy management of compute, platform. With the central user interface you can easily run VMs and network, and storage with the containers, manage software-defined storage resources and networking central web interface functionality, highly available clusters, and multiple out-of-the-box tools • 100% software-defined architecture like backup/restore, live migration, storage replication, or the integrated • Two virtualization technologies firewall. Proxmox VE enables you to virtualize even the most demanding supported: KVM hypervisor & Linux Linux and Windows application workloads. Container (LXC) By combining two virtualization technologies on a single platform, • Web-based UI, CLI, API Proxmox VE provides maximum flexibility to your data center. It includes strong high-availability (HA) support and—thanks to the unique multi- • High-Availability (HA) Cluster master design—you don’t need an additional management server thus Manager saving resources and allowing HA without single point of failure (SPOF). • Live Migration ENTERPRISE-READY • Built-in services: firewall, backup/restore, storage replication, Enterprises use the powerful Proxmox VE platform to easily install, etc. manage and monitor hyperconverged infrastructures (HCI) in their data center. Multiple authentication sources combined with role-based user • Open-source license GNU AGPL, v3 and permission management enable full control of the HA clusters. The • Enterprise support agreements REST API enables easy integration of third-party management tools like for example custom hosting environments.
    [Show full text]