THE EASTERN REGIONAL NETWORK (ERN): SIMPLIFYING MULTI-CAMPUS RESEARCH COLLABORATIONS

Barr von Oehsen Associate Vice President, Office of Advanced Research Computing Rutgers, The State University of New Jersey TABLE OF CONTENTS

• Eastern Regional Network Vision and Mission • Science Drivers • History • Proof of Concept • Commercial Cloud • perfSONAR • Future Projects • ERN Partners

[ 2 ] Eastern Regional Network (ERN)

• Vision: To simplify multi-campus collaborations and partnerships that advance the frontiers of research, pedagogy, and innovation.

• Mission: Through a partnership of educational institutions, research facilities, regional network providers, and , the ERN will provide layered and transparent access to shared data and computing facilities for research projects located at and between partner sites. The resulting regional research and education platform will support a diverse set of science drivers and emergent educational opportunities and offer the educational research community access to a broad range of collaborative multi-institutional resources that are not available on any one campus alone.

[ 3 ] Solving Challenges Related to:

• Supporting and enabling research collaborations that span multiple universities • Sharing and accessing data (containerized workflows) • Building, managing, supporting, and sharing distributed federated resources/infrastructure that spans multiple campuses and commercial cloud designed to enable research • Creating performant, software-configurable, and intelligent networks to enable access to research resources • Creating testbeds around new and emerging technologies, software stacks, etc., being asked for by the research community • Having a safe space for Central IT to experiment with new and emerging technologies without the worry of breaking enterprise infrastructure • Sharing expertise and building collaborations across both research and IT silos • Workforce development, especially for schools that don’t typically have access to campus cyberinfrastructure, research testbeds, new and emerging technologies, etc. • Writing broader impact statements on grant proposals

[ 4 ] What Are The Science Drivers?

• Cryo-EM • Precision Health • Proteomics • Systems Biology • Genomics • Computational Social Science • Brain Imaging • Computer Science • Climate Research • Network Research • High Energy Physics • Multi-Scale Modeling • Digital Humanities • Chemistry • Social Sciences • Marine and Coastal Science • Medical/Clinical Research

[ 5 ] ERN Origins and Evolution

• Germ of an idea: at 2017 National Research Platform meetings Can the regional research platform idea work in the Northeast? Rutgers, OSHEAN, KINBER • Coalition of the Willing: January 2018 gathering at Rutgers-Newark Can we find something of common interest to work on? KINBER, Rutgers, OSHEAN, MGHPCC, Internet2, NYSERnet • Resource Federation Proof of Concept: Can we work together to prototype a potentially beneficial resource federation idea? Rutgers, MGHPCC, Syracuse, NJ Edge, U Maine, Google, Internet2 “Early if not elegant” approach; emphasis on getting people to work together Used Rutgers Federated Ecosystem as model

[ 6 ] Federated Computing Across New Jersey – ERN Model

State-wide multi-campus, distributed NYSERNet, I2 HPC and storage, fast, low-latency network that is part of global Science Newark HPC DMZs: Local Storage DTN HPC • NSF CC* Funded (OAC-1659232)

Local • SDN Based 100 Gbps Network DTN DTN Storage New Brunswick Core PE I2, NJEdge, MagPi, KINBER Central Storage Repository • Data Transfer Nodes (FIONA) HPC • Advanced Computing and Storage Camden Local DTN Storage • Performance and monitoring support (perfSONAR, XDMoD) • Testbed as a service • Federated across campuses • Policy driven

[ 7 ] Origins and Evolution

• Northeast perfSONAR Mesh: Can we work together toward a regional science DMZ? Federation sites plus • BU, Brown, Colby, CEN, KINBER, Network Maine, OSHEAN, UNH Less-specialized technology allows expanded participation • Growing interest: Maybe we really can do this? [in progress] expanded steering committee; per-project working groups All of the above plus • U Delaware, NJIT, U Buffalo, Bucknell, UMass Amherst, Yale, Princeton, BTAA, U Kentucky, Case Western Reserve

[ 8 ] Resource Federation Proof of Concept

§ Six sites, five states § Ability to launch containerized (Singluarity) jobs as well as traditional HPC jobs, from any site to any other site including commercial cloud, via SLURM scheduler. § Working with Google, Cisco, Internet2, SchedMD (SLURM), and OSG on enhancements § InCommon authentication and authorization § Policy based federation of computing resources § Reserving slices of the network § Cloud bursting § OSG federation with containerized SLURM Recipe for joining: (slurmd) https://github.com/rutgers-oarc/ern-poc § Testing different data sharing approaches as well as filesystems

[ 9 ] Resource Federation Proof of Concept

[ 10 ] Federation with Commercial Cloud

• Partnership with Google Cloud and SchedMD (SLURM) to develop federated hybrid environment • Allows federation across resources (including cloud services) • Allows us to offer one stop shop for users • Allows on demand elastic computing • Initial Proof of Concept • Set up federated environment between Rutgers and Google Cloud • Google Cloud instance is set up similar to local resource, includes SLURM • Connection through Internet2 between both environments • Able to submit jobs directly from Rutgers HPC to Google Cloud using SLURM submit script • Elasticity/bursting capability now being tested • Next steps: Extend to Jetstream, OSG

[ 11 ] ERN perfSONAR Mesh

• 23 nodes, 14 sites, six states • New sites welcome • Still in teething phase • Improves visibility on connectivity/bandwidth/delay • Starting point for connectivity improvements (where they matter) • Prerequisite for a regional data DMZ

Site: http://mesh.hpc.rutgers.edu/maddash-webui/

Recipe for joining: https://github.com/rutgers-oarc/ern-poc

[ 12 ] Community Ownership

• ERN is not a Rutgers project, it is a community project • Loosely based on the condo model • Proof of concepts led by partner campuses, regional network providers, industry, etc. • Test out new ideas to address challenges • Can be funded or lead to funding • For the greater good of the community • Using CaRCC stakeholders document as a guide • Steering Committee (10 members) • ERN Council in process • Leverages other community projects

[ 13 ] Looking ahead – Some Possibilities • Regional technology testbeds • Regional Science DMZ (Data DMZ) • Common interface to resources, independent of provider (Atrio?) • Campus resources, national resources, commercial cloud • Resource bartering system/Marketplace • Advanced network technologies (experiment->pilot->production) • Simplified engagement with national resources and commercial cloud • National Research Platform (NRP) • Open Storage Network (OSN) • Big Data Hubs • Open Science Grid • National Resources (XSEDE, GENI, Cloudlab, Chameleon) • Commercial cloud providers (Azure, AWS) • Regional infrastructure proposals (NSF CC*, CICI, Mid-Scale, etc.) • Improve competitiveness on larger solicitations • Mosaic of smaller proposals that benefit the whole • New security models • Zero trust? Google BeyondCorp [ 14 ] ERN Partners • University of Kentucky • University of Maine-Orono Universities: • University of Massachusetts-Amherst • Brown University • University of • Case Western Reserve • University of New Hampshire • Columbia University Network Providers: • Franklin and Marshall • CAAREN • George Washington University • CEN • Indiana University • Internet2 • NJIT • KINBER • Penn State University • NetworkMaine • Princeton University • NJEdge • Rutgers University • NYSERNET • Syracuse University • OARnet • Yale University • OSHEAN • University of Chicago Organizations: • University of Delaware MGHPCC • University of Northeast Big Data Innovation Hub [ 15 ]