Building the SLATE Platform Joe Breen Lincoln Bryant Gabriele Carcassi University of Utah University of Chicago University of Michigan [email protected] [email protected] [email protected] Jiahui Chen Robert W. Gardner∗ Ryan Harden University of Utah University of Chicago University of Chicago [email protected] [email protected] [email protected] Martin Izdimirski Robert Killen Ben Kulbertis University of Utah University of Michigan University of Utah [email protected] [email protected] [email protected] Shawn McKee Benedikt Riedel Jason Stidd University of Michigan University of Chicago University of Utah [email protected] [email protected] [email protected] Luan Truong Ilija Vukotic University of Utah University of Chicago [email protected] [email protected]

ABSTRACT KEYWORDS We describe progress on building the SLATE (Services Layer at the Distributed computing, Containerization, Edge computing Edge) platform. The high level goal of SLATE is to facilitate creation ACM Reference format: of multi-institutional science computing systems by augmenting Joe Breen, Lincoln Bryant, Gabriele Carcassi, Jiahui Chen, Robert W. Gard- the canonical Science DMZ pattern with a generic, “programmable", ner, Ryan Harden, Martin Izdimirski, Robert Killen, Ben Kulbertis, Shawn Mc- secure and trusted underlayment platform. This platform permits Kee, Benedikt Riedel, Jason Stidd, Luan Truong, and Ilija Vukotic. 2018. hosting of advanced container-centric services needed for higher- Building the SLATE Platform. In Proceedings of Practice and Experience in level capabilities such as data transfer nodes, software and data Advanced Research Computing, Pittsburgh, PA, USA, July 22–26, 2018 (PEARC caches, workfow services and science gateway components. SLATE ’18), 7 pages. uses best-of-breed data center virtualization and containerization https://doi.org/10.1145/3219104.3219144 components, and where available, software defned networking, to enable distributed automation of deployment and service lifecycle 1 MOTIVATION management tasks by domain experts. As such it will simplify cre- Multi-institutional research collaborations propel much of the sci- ation of scalable platforms that connect research teams, institutions ence today. These collaborations require platforms connecting ex- and resources to accelerate science while reducing operational costs periment facilities, computational resources, and data distributed and development cycle times. among laboratories, research computing centers, and in some cases commercial cloud providers. The scale of the data and complexity of CCS CONCEPTS the science drive this diversity. In this context, research computing • Computer systems organization → Grid computing, Edge teams strive to empower their universities with emergent technolo- Computing; gies which bring new and more powerful computational and data capabilities that foster multi-campus and multi-domain collabora- tions to accelerate research. Recently, many institutions invested ∗Corresponding Author in their campus network infrastructure with these goals in mind. Yet even for the most advanced, well-stafed, and well-equipped campus research computing centers the task is daunting. Acquir- Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed ing and maintaining the full scope of cyber-engineering expertise for proft or commercial advantage and that copies bear this notice and the full citation necessary to meet the complex and expanding demands of data and on the frst page. Copyrights for components of this work owned by others than ACM computationally driven science is too costly and does not scale to must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specifc permission and/or a the full spectrum of science disciplines. The diversity of compu- fee. Request permissions from [email protected]. tation, data and research modalities all but ensures that scientists PEARC ’18, July 22–26, 2018, Pittsburgh, PA, USA spend more time on computation and data management related © 2018 Association for Computing Machinery. ACM ISBN 978-1-4503-6446-1/18/07...$15.00 tasks than on their domain science while research computing staf https://doi.org/10.1145/3219104.3219144 spend more time integrating domain specifc software stacks with PEARC ’18, July 22–26, 2018, Pitsburgh, PA, USA J. Breen, et al.

for high performance, wide area applications. A dedicated SLATE edge cluster will augment the existing Science DMZ by ofering a platform capable of hosting advanced, centrally managed research computing edge services including, where the local infrastructure permits it, the deployment of virtual circuits and other software de- fned networking constructs [2, 37]. For a small institution with lim- ited resources, the SLATE concept may provide a complete Science DMZ infrastructure to more quickly integrate these institutions into centrally managed research platforms. The SLATE concept will allow local resource administrators the ability to simply install the Figure 1: SLATE implemented in the local HPC context infrastructure while ofering central research groups the services needed for mdanagement of software and science tools used on the platform. Thus, a local researcher in a small institution could limited applicability and sustainability beyond the immediate com- focus on the science and connecting any requisite science instru- munities served. Capabilities elsewhere are not available locally, mentation, while the local IT staf would not have the burden of and vice versa. How should campus and HPC resource providers trying to understand the science requirements, application software evolve their cyberinfrastructure to more easily incorporate data dependencies, data and workfow particulars, etc. A good science infrastructure building blocks developed in other contexts? use-case comes from a medium sized collaboration to detect dark matter. 2 APPROACH We address this challenge in complexity and scaling by introducing a Services Layer At The data center “Edge” (SLATE) which enables distributed automation, centralized delivery and operation of data, 2.1 A Platform for Dark Matter Searches software, gateway and workfow infrastructure. Much as Google Observations of the cosmic microwave background fuctuation, re-imagined the data center [41] and initiated a wave of data center large-scale galaxy surveys, and studies of large-scale structure for- virtualization development, we view advanced “cyberinfrastruc- mation indicate that a large fraction of the matter in the universe is ture as code” as an appropriate metaphor for transforming the way not visible. An exotic but as yet undiscovered elementary particle advanced science platforms are built and operated. SLATE will aug- could explain these observations. Several experiments have been ment the Science DMZ pattern [33] by adding a secure and trusted built in last two decades to prove the existence of such elusive “underlayment” platform to host advanced data and software ser- particles but their detection has proved challenging as we do not vices needed for higher-level, connective functions between data have yet a clear picture of what they are and if they really exist. centers. These connective services could include, for example, a The XENON1T [27] experiment, a two-phase xenon Time Projec- domain specifc content delivery network endpoint, a collabora- tion Chamber has been built in the Laboratori Nazionali del Gran tion data cache, a job (workfow) scheduler, a service component Sasso (LNGS) in Italy to study fundamental questions about the ex- of a distributed science gateway, an http-based software cache, istence and make-up of dark matter. Commissioning began during or a resource discovery service (for higher level, meta scheduling the frst few months of 2016, with the frst large scale science run systems). beginning in December 2016. Because of its one ton fducial mass A major focus of SLATE will be development of community and ultra-low background, the XENON1T experiment is probing accepted Science DMZ patterns capable of supporting advanced properties of dark matter in yet unexplored regions. and emerging technologies while respecting local site security poli- A data processing and analysis hub is hosted by the University cies and autonomy. In this project, we are focusing on production of Chicago Research Computing Center to complement processing services for data mobility, management, and access as driven by and analysis facilities in Europe (Stockholm is the European analysis data-intensive science domains. For operators of distributed data hub). A distributed data management service for the collaboration management services, content delivery networks, and science gate- was built so that experiment and simulation data sets at various ways SLATE will closely resemble the NIST defnition [38] of a stages of processing could be distributed and shared easily through- PaaS (Platform-as-a-Service) though the resemblance does not limit out the 22 member institutes. Figure 2 shows the deployment of the underlying infrastructure to a cloud context. We are leveraging the XENON1T Rucio service [35]. The deployment is a network experience and lessons learned in the deployment and operation of of fve storage endpoints (GridFTP-based data transfer nodes) in data systems linking over 60 data centers for the LHC computing Europe and the U.S providing a highly scalable global namespace, a grid. Figure 1 gives a schematic of the SLATE concept in the local reliable and fast transfer service, a subscription (“placement rules”) HPC context. service, and a deletion service for data lifecycle management. The The SLATE concept should accommodate large, well-equipped raw experimental data are uploaded at the experiment lab and au- HPC centers, research computing facilities at institutions with fewer tomatically replicated to specifc HPC centers for processing, and resources, as well as commercial cloud providers. Modern HPC cen- data products from those systems are registered into the system for ters which support data intensive science typically have a Science distribution to the analysis hubs. The Rucio client tools provide a DMZ which hosts dedicated data transfer nodes, perfSONAR [1, 28] uniform data access model for the various endpoints. measurement hosts, and security and enforcement policies needed Building the SLATE Platform PEARC ’18, July 22–26, 2018, Pitsburgh, PA, USA

Figure 2: The multi-institution data management platform for the XENON collaboration A central server for managed fle transfer is hosted at Brookhaven National Laboratory, and central fle and dataset catalogs and service agents are hosted by the University of Chicago. At each of the storage endpoints is a GridFTP service. Each of these services is managed indi- vidually so that the platform itself requires eforts from 10 individual administrators. With a SLATE hosted platform, the services, confgu- ration, monitoring and optimization could be managed by a single operator, requiring only basic server management at the endpoints.

Figure 3: SLATE architecture We envision the SLATE platform as a key component for scal- ing and operating data placement services like Rucio for multi- institution science collaborations like XENON1T, especially where • A high performance networking device, either deploying a local expertise in complex scientifc software stacks may be limited. new device or reusing an existing high performance network device 3 ARCHITECTURE • An out-of-band management server, to allow SLATE plat- SLATE leverages advances made by previous testbeds [12, 29, 30, form administrators to perform system maintenance and for 36, 39] and similar platforms [8, 19] and integrates best-of-breed SLATE central services to stage OS updates and confgura- data center virtualizatio