Next Generation Distributed Environments For Global Science

Joe Mambretti, Director, ([email protected]) International Center for Advanced Internet Research (www.icair.org) Northwestern University Director, Metropolitan Research and Education Network (www.mren.org) Director, StarLight, PI StarLight IRNC SDX,Co-PI Chameleon, PI-iGENI, PI- OMNINet (www.startap.net/starlight)

Asia Pacific Advanced Network (APAN) Co-Located With Supercomputing Asia March 26-29, 2018 Singapore

NG Digital Sky Survey

ATLAS

BIRN: Biomedical Informatics Research LHCONE Network CAMERA CineGrid www.lhcone.net ANDRILL: www.nbirn.net metagenomics Carbon Tracker www.cinegrid.org Antarctic camera.calit2.net www.esrl.noaa.gov/ ALMA: Atacama Geological gmd/ccgg/carbontrack Large Millimeter Drilling er Array www.andrill.org www.alma.nrao.edu OOI-CI GEON: Geosciences ci.oceanobservatories.org Network ISS: International Space Station www.geongrid.org GLEON: Global Lake Comprehensive www.nasa.gov/statio Ecological Large-Array n Observatory Stewardship System DØ (DZero) Network www-d0.fnal.gov Pacific Rim www.class.noaa.gov www.gleon.org Applications and Grid Middleware Assembly WLCG www.pragma- LIGO lcg.web.cern.ch/LCG/publi grid.net www.ligo.org TeraGrid c/ IVOA: www.teragrid.org International Virtual Observatory Sloan Digital Sky www.ivoa.net Globus Alliance Survey XSEDE SKA OSG www.globus.org www.sdss.org www.xsede.org www.opensciencegrid.org www.skatelescope.o rg Compilation By Maxine Brown

New Science Communities Using LHCONE

• Belle II Experiment, Particle Physics Experiment Designed To Study Properties of B Mesons (Heavy Particles Containing a Bottom Quark). • Pierre Auger Observatory, Studying Ultra-High Energy Cosmic Rays, the Most Energetic and Rarest of Particles In the Universe. • In August 2017 the PAO, LIGO and Virgo Collaboration Measured a Gravitational Wave Originating From a Binary Neutron Star Merger. • The NOvA Experiment Is Designed To Answer Fundamental questions in neutrino Physics. • The XENON Dark Matter Project Is a Global Collaboration Investing Fundamental Properties of Dark Matter, Largest Component Of The Universe. • ProtoNUMA/NUMA – Collaborative Research On Nutrinos iCAIR Basic Research Topics

• Transition From Legacy Networks To Networks That Take Full Advantage of IT Architecture and Technology • Extremely Large Capacity (Multi-Tbps Streams) • Specialized Network Services, Architecture and Technologies for Data Intensive Science • High Degrees of Communication Services Customization • Highly Programmable Networks • Network Facilities As Enabling Platforms for Any Type of Service • Network Virtualization • Tenet Networks • Network Virtualization • Network Programming Languages (e.g., P4) API (e.g., Jupyter) • Disaggregation • Orchestrators • Highly Distributed Signaling Processes • Network Operations Automation (Including Through AI/) • SDN/SDX/SDI/OCX/SDC/SDE Issues

• 21st Century Scientists Encounter Unprecedented Opportunities As Well As Deeply Complex Challenges By Utilize New Knowledge Discovery Techniques Based On Exploring Hidden Patterns In Exabytes of Globally Distributed Data. • Scientific Data is Growing Exponentially • Growth Will Further Accelerate As New Major Instrumentation Is Implemented. • Also, New, Highly Sophisticated Analytic Techniques For Big Data Based Investigations Are Being Created. • The Challenges That Arise From This Global Scale Research Are Being Addressed Through the Creation of Powerful, Innovative Globally Distributed Computational Science Platforms, Such As the Global Research Platform (GRP). Global Research Platform (GRP).

• The Global Research Platform (GRP) Provide Next Generation Services. Architecture, Techniques, and Technologies For World-Wide Data Intensive Scientific Research. • This Platform Provides Exceptional Support For Capturing, Managing, Indexing, Analyzing, Storing, Sharing, Visualizing and Transporting Exescale Data. • Innovations Being Developed Include Those Related To: – Heterogeneity – Virtualization, – Segmentation – Open Infrastructure Architecture and Components – Orchestration – Sliceability – Multi-level Resource Abstraction and programmability, – Granulated Customization, – Fluid Data Distribution – AI/machine Learning/Deep Learning Emerging Capabilities/Technologies

• Built-In Preconfigured Examples/Templates To Establish Infrastructure Foundation Workflows • Orchestration • Zero-Touch “Playbooks” For Different Segments of Infrastructure Foundation Workflows After Implementing Initial Suites (e.g., Using Jupyter) • Interactive Control Over Running Workflows • Portability for Different Infrastructure Foundation Workflows • Options/Capabilities for Specialized Customization • Options For Real Time Visualization Of Individual Workflows, At Highly Granulated Levels Network Research: Kernel Bypass Networks Drivers: • AI Training/DL • Distributed Storage Systems • Virtual Networking • Programmable Networks/SDN • NFV • In-Memory Caching • Low Latency • High Capacity (e.g., Big Data) • Low Latency and High Capacity Motivation For KBNets & Issues

• Need To Minimize Network CPU Overhead • Sub Performance of Traditional OS Stack

Issues To Be Addressed

• Control Planes • Data Planes (Transport) • Management Plane • Virtualization • Programming Languages • NIC/Backplane/Switch/Chip Architecture

Techniques

• RDMA (Remote Direct Memory Access) • DPDK (Data Plane Development Kit) • SmartNICs • New Backplane/Switch Fabrics • RoCE (RDMA Over Converged Ethernet) • Et Al • New High Performance OS Stacks Programmable Network Techniques and Devices

• SDN/SDX & Network Programming Languages • Programmable Switch ASICs • Programmable Network Processors • FPGAs • Programmable NICs • Ref: Barefoot Tofino, Intel FlexPipe, Cavium XPliant, Netronome Agilio. • P4 Based In-Network Telemetry • AI/ML/DL Integrated With Network Programming • Jupyter • DTNs

iCAIR: Founding Partner of the Global Lambda Integrated Facility Available Advanced Network Resources

Visualization courtesy of Bob Patterson, NCSA; data compilation by Maxine Brown, UIC. www.glif.is AutoGOLE International Multi-Domain Provisioning Using AutoGOLE Based Network Service Interface (NSI 2.0)

* Network Service Interface (NSI 2.0) * An Architectural Standard Developed By the *Open Grid Forum (OGF) * OGF Pioneered Programmable Networking (Initially Termed “Grid Networking”) Techniques That Made Networks ‘First Class Citizens” in Grid Environments – Programmable With Grid Middleware * Currently Being Placed Into Production By R&E Networks Around the World IRNC: RXP: StarLight SDX A Software Defined Networking Exchange for Global Science Research and Education Joe Mambretti, Director, ([email protected]) International Center for Advanced Internet Research (www.icair.org) Northwestern University Director, Metropolitan Research and Education Network (www.mren.org) Co-Director, StarLight (www.startap.net/starlight) PI IRNC: RXP: StarLight SDX Co-PI Tom DeFanti, Research Scientist, ([email protected]) California Institute for Telecommunications and Information Technology (Calit2), University of California, San Diego Co-Director, StarLight Co-PI Maxine Brown, Director, ([email protected]) Electronic Visualization Laboratory, University of Illinois at Chicago Co-Director, StarLight Co-PI Jim Chen, Associate Director, International Center for Advanced Internet Research, Northwestern University

National Science Foundation International Research Network Connections Program Workshop Chicago, Illinois May 15, 2015 Emerging US SDX Interoperable Fabric

IRNC SDX GENI SDX IRNC SDX

IRNC SDX UoM SDX GENI IRNC SDX SDX IRNC SDX Global LambdaGrid Workshop 2017 Demonstrations, Sydney Australia

International Multi-Domain Provisioning Using AutoGOLE Based Network Service Interface (NSI 2.0) Using RNP MEICAN Tools for NSI Provisioning Large Scale Airline Data Transport Over SD-WANs Using NSI and DTNs Large Scale Science Data Transport Over SD-WANs Using NSI and DTNs SDX Interdomain Interoperability At L3 Transferring Large Files E2E Across WANs Enabled By SD-WANs and SDXs

ESA Service Prototype

Source; John Graham UCSD

Implementing a SCinet DTN

Source; Jim Chen, iCAIR SUPERMICRO 24X NVMe SUPER SERVER

NVMe Type A: 8 X Intel P3700 800G NVMe Type B: 8 X SamSung 950 512G + M.2 to U.2 Adopter Dell 14G Solution Configuration (Co-Research & Development in Collaboration with Dell)

DDR4 DDR4 PowerEdge R740XD Server

Intel® Xeon® DDR4 processor DDR4 E5-2600 v4 2 X Intel® Xeon® Gold 6136 QPI 3.0G,12C/24T,10.4GT/s 2UPI,24.75M 2 Channels

® ® Cache,Turbo,HT (150W) Intel Xeon DDR4 processor DDR4 E5-2600 v4

DDR4 DDR4 192G DDR4-2666

PCIe* 3.0, 40 lanes PCI-e Configuration Investigation: 2 X Mellanox ConnectX-5 100GE VPI LAN Up to 4x10GbE 4 X Kingston/Liqid AIC NVMe PCI-e Intel® C610 series chipset X8 SSD Drives

WBG Optional SAS/SATA Drives Recent WAN DTN Testing

• Preparation – Tests Conducted By Se-Young Yu (iCAIR) • DTNs: • @iCAIR : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz, Mellanox ConnectX-5 100G NIC • @PACWAVE - LA : Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz, Mellanox ConnectX-5 100G NIC • @UvA : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz, Mellanox ConnectX-3 40G NIC • @CERN : Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Intel 82599ES 10G NIC • Tuning Parameters : • BIOS, CPU, NIC, TCP Stack, O/S and MTU Tuning Applied Memory-to-Memory Test Results Disk To Memory Test Results Wenji Wu, Phi Demar et al

Source: Wenji Wu Source: Wenji Wu Source: Wenji Wu Source: Wenji Wu Source: Wenji Wu L2 SW Disaggregation: Falconwitch Prototype Configuration NVMe Type A: Intel P3700 800G x 8 NVMe Type B: M.2 to U.2 Adopter with SamSung 950 Pro 512G X 8 SamSung 960 Pro 1T X 8 SamSung 960 Pro 2T X 8 4 X Mellanox ConnectX-5 100GE 2 X PCI-e X16 host adapter to host

Host node: SuperWorkstation 7048GR-TR 2 X E5-2667 V4 2 X PCI-e X16 host adapter Falconwitch PCI-e Switching Fabric Selected ML Frameworks (Of Many):

• Apache Singa • • H2O • MLlib () • Scikit-Learn (Python) • Shogun (C++) • TensorFlow • (Python) • (~ Scientific Computing) • Veles (C++, w/ Some Python) Summary

• Data Intensive Science Can Benefit From A GRP, Including Enhanced Services/Techniques/Technologies For High Performance WAN Data Transport • One Approach: Relying On L2 WAN Transport Channels • A Complementary Enabling Capability: Using DTNs Integrated With Specialized WAN Paths To Optimize E2E Data Flows • Core Components Can Be Supplemented By Enhancing Software Stacks, e.g., Jupyter, NSI, MEICAN, P4 Programs, BDE, AI/ML/DL, etc • Today, Many Components Exist To Create An E2E Services For Data Intensive Science • Major Opportunity: Creating This Service and Placing It Into Production www.startap.net/starlight

Thanks to the NSF, DOE, DARPA, NIH, USGS, NASA, Universities, National Labs, International Partners, and Other Supporters