Self-Scaling Clusters and Reproducible Containers to Enable Scientific Computing
Total Page：16
File Type：pdf, Size：1020Kb
Self-Scaling Clusters and Reproducible Containers to Enable Scientific Computing Peter Z. Vaillancourt∗, J. Eric Coultery, Richard Knepperz, Brandon Barkerx ∗Center for Advanced Computing Cornell University, Ithaca, New York, USA Email: [email protected] yCyberinfrastructure Integration Research Center Indiana University, Bloomington, IN, USA Email: [email protected] zCenter for Advanced Computing Cornell University, Ithaca, New York, USA Email: [email protected] xCenter for Advanced Computing Cornell University, Ithaca, New York, USA Email: [email protected] Abstract—Container technologies such as Docker have become cyberinfrastructure at a number of campuses, enabling them a crucial component of many software industry practices espe- to provide computational resources to their faculty members cially those pertaining to reproducibility and portability. The con- [2]. As more organizations utilize cloud resources in order tainerization philosophy has influenced the scientific computing community, which has begun to adopt – and even develop – con- to provide flexible infrastructure for research purposes, CRI tainer technologies (such as Singularity). Leveraging containers has begun providing tools to harness the scalability and utility for scientific software often poses challenges distinct from those of cloud infrastructure, working together with the members encountered in industry, and requires different methodologies. of the Aristotle Cloud Federation Project [3]. Red Cloud, the This is especially true for HPC. With an increasing number Cornell component of the Aristotle Cloud Federation, provides of options for HPC in the cloud (including NSF-funded cloud projects), there is strong motivation to seek solutions that provide an OpenStack cloud services much like the Jetstream resource, flexibility to develop and deploy scientific software on a variety that is available both to Cornell faculty as well as to Aristotle of computational infrastructures in a portable and reproducible members. way. The flexibility offered by cloud services enables virtual CRI focuses its activities on knitting together cyberinfras- HPC clusters that scale on-demand, and the Cyberinfrastructure tructure resources at multiple institutions. While the XSEDE Resource Integration team in the XSEDE project has developed a set of tools which provides scalable infrastructure in the Federation presents the resources of sundry service providers cloud. We now present a solution which uses the Nix package at different levels of interoperability [4], there are many manager in an MPI-capable Docker container that is converted to campus cyberinfrastructure installations that are focused on Singularity. It provides consistent installations, dependencies, and meeting local needs. Expertise in cluster administration is a environments in each image that are reproducible and portable scarce resource, and not all institutions that elect to provide a across scientific computing infrastructures. We demonstrate the utility of these containers with cluster benchmark runs in a centralized computing cluster are immediately able to leverage self-scaling virtual cluster using the Slurm scheduler deployed technical and policy knowledge to implement and maintain it. in the Jetstream and Aristotle Red Cloud OpenStack clouds. The CRI team assists these campuses in set-up, maintenance, arXiv:2006.14784v2 [cs.DC] 3 Aug 2020 We conclude this technique is useful as a template for scientific and policy generation. If the institution has interest, CRI software application containers to be used in the XSEDE compute assists with integration into the larger XSEDE framework. environment, other Singularity HPC environments, and cloud computing environments. CRI has worked with 6 different institutions in 2019 and Index Terms—Containers, Docker, Singularity, Slurm, Cluster, 2020 to provide support for cyberinfrastructure installations, Cloud, Scientific Computing, HPC, MPI, Cyberinfrastructure and has directly assisted in bringing over 1 PetaFLOP of computing capacity online, in addition to advising on other I. INTRODUCTION cyberinfrastructure activities. The Cyberinfrastructure Resource Integration (CRI) group Previously, CRI introduced the virtual cluster toolkit [5], [6], of Extreme Science and Engineering Development (XSEDE) which allows for the creation of cluster computing resources [1] provides software and services to bring best practices from on cloud resources. It was specifically created for the Jetstream the National Supercomputing Centers to university campuses [7] research cloud, but is adaptable to other cloud resources, interested in implementing their own computational infrastruc- as seen in our deployment to Red Cloud in this paper. This ture. By providing software and documentation for campus virtual cluster toolkit provides infrastucture that can be used adopters, CRI has been able to assist in the creation of for a variety of jobs including multi-node jobs, managed by the Slurm scheduler [8]. 1) XCBC: In an effort to help solve the problems facing CRI has also taken efforts to expand software offerings campus resource providers in giving their researchers access which take advantage of the benefits of cloud computing to quality High Performance Computing (HPC) environments, models: reproducible application containers and self-scaling the XSEDE CRI team developed a toolkit called the XSEDE- clusters built on cloud infrastructure. Reproducible containers Compatible Basic Cluster (XCBC) to easily build a local HPC take advantage of container runtime environments which allow system conforming to the basic environment available on most applications to run in lightweight virtual spaces that make use XSEDE resources, modulo differences in specialized hardware of the underlying operating system, while providing reliable or site-specific software. This project evolved [11] from being installation of a set of software and dependencies at a given based on the Rocks [12] cluster management software to the version. Self-scaling clusters rely on common cluster system OpenHPC [13] project with Ansible [14]. The CRI team has images and the Slurm scheduling tool in order to create and performed a dozen in-person site visits to help implement destroy additional virtual machines as jobs are placed into HPC systems using this toolkit, in addition to dozens more the Slurm queue. Reproducible containers provide reliable, remote consultations. Discussions with sites implementing portable applications with the same constituent software at the toolkit [15]–[17] have shown that the local XSEDE-like every instantiation, while self-scaling clusters provide efficient environment does in fact help researchers onboard to XSEDE use of computational infrastructure without incurring more more easily, in addition to providing an environment with low usage than the applications require. wait times compared to the full queues on larger systems, where researchers may wait for days before a particular job II. COMPUTE ENVIRONMENT CHOICES runs. 2) Singularity in HPC: The problem remains, however, that A. XSEDE Compute Environment individual researchers may not have the software they need Researchers at U.S. universities and federally-funded re- available on either a national or a local campus system, and search and development centers (FFRDC’s) consistently report getting new custom software installed is non-trivial for the a lack of sufficient computational capability available to them administrators of those systems. In many cases, this can be [9], [10]. The XSEDE project provides a socio-technical addressed through the use of containers, which have sparked system which facilitates access to resources provided through a revolution in horizontal scaling of software in the enterprise the XSEDE federation, but despite the considerable resources world. In addition, containers provide the opportunity to utilize available, not all requests can be accommodated. XSEDE differing versions of software than those installed, or to allocation proposals outstrip allocable resources by a factor package private or custom software for development purposes. of three. By passing on best practices and interoperability to There are significant challenges to using containers effec- other institutions, the XSEDE project can foster the extension tively on HPC systems, however, due to security concerns, of computational capacity beyond what is available through complexity of installations for scientific software, and lack the project alone, and in addition, can make it easier for of widespread adoption as a distribution method for scientific researchers to move between systems. By providing common software. The popularity and prevelance of Docker [18] out- filesystem layouts, software, and scheduling systems, adopters side of the HPC community notwithstanding, security concerns of XSEDE software capabilities can make it easier for re- around the fact that a user running docker has root access to a searchers to work on XSEDE systems as well as other systems system prevent its use on HPC systems without modifications which have adopted similar affordances, making it easier to to mitigate risk. Newer container runtime environments have move between resources and facilitating scaling up or down been developed with an eye towards solving these security in compute capability as necessary. concerns and even designed specifically for HPC systems, such However, not all XSEDE resources and not all campus as CharlieCloud [19] and Singularity [20],