SC20 Advanced Networking For Data Intensive Science Experiments and Demonstrations

Joe Mambretti, Director, ([email protected]) International Center for Advanced Internet Research (www.icair.org) Northwestern University Director, Metropolitan Research and Education Network (www.mren.org) Co-Director, StarLight (www.startap.net/starlight), Director, StarLight International/National Communications Exchange Facility (www.startap.net/starlight), PI IRNC: RXP: StarLight SDX, Co-PI Chameleon, PI-iGENI, PI-OMNINet

JET Meeting November 17, 2020 International Conference On High Performance Computing Networking, Storage and Analytics November 9-20, 2020

GRP: Services, Architecture, Technology

• Architecture: “Global Science DMZ” • Services Optimized For Science Workflows • High Performance Transport Over WANs For Large Capacity Data Streams E2E • Enhanced Virtualization • Highly Programmable • Specialized Components (Software Stacks, Next Gen DTNs, Advanced APIs, Customized Devices, Federation, INT, P4, etc.) • Interdomain Dynamic Provisioning • Production Resources Adjacent to Testbeds Next Generation Distributed Enviroment For Global Science Global Research Platform Network 100 Gbps

StarLight

AutoGOLE Fabric 2020 (60+ StarLight SDX Overview

• This IRNC StarLight SDX Initiative Is Designing, Implementing, and Operating New Services For Global Data Intensive Sciences, Based On Emerging Next Generation Architecture and Technolgies, Including Virtualization, Orchestration, Segmentation (Slicing), Software Defined Resources, Programmability and Customization. • These Macro Trends Enable Exchanges To Be Agile Platforms For Dynamic Services Provisioning, Real-Time Responsiveness, and Distributed Control Over Core Resources, Including By Applications, Edge Processes and Devices.

This Project Is Transitioning Network Exchanges To Open Innovation Platforms r ICT Testbed ...... and l I OevelopmentPromotJon Cente• I r •

1 1 :~,NET 1-.~ ~ JGN I I I •• I I SINET : (Japan) : : (Japan) 1 (Los Angeles) 1 1 (Los Angeles)

10G

NX100G I l I I I I I (Chicago) (Amsterdan) :-1~t; r~a~i~n~ 1-] 110G 1- -- - -~ ------I SON Networks I I I ""'l ◄.-----)1,..,1 -?- ~:;:::: I & Testbeds .,_ 1 1 2 1 1 1 (China) : ....--... PACIFIC WAVE 1..,.1- x__0_0_G__ _, 1, ___., 1.._____, I I (Los Angeles) I I (Chicago) I I 1______1 I

I l I NORDUnet I S ~ ng/\ReN : ...... ,_ ..... ,.. ., - ..... , I I I : (Singapore) I 2x10G (Nordic) (Los Angeles) 1 (Chicago) I I r I I I__ ...

I 1◄◄1-----, 2x10G 10G 10 : (Honolulu) : ___ _t ___ _ l I l l I 1 (Mauna) I l l ] [ Southern Light ] I I / ~ I -t ltl "t4llill 1 : OpenTransits : CMoo .... AtlanticWave ----i--- RE/\N N Z i I , i--...... 1 1 1 ■ 1 REN 1 : KRt.~ :)-JET : 1 Network 1 r------\ I I I I CSTNET : I I {Miami) ~ : RNP : I {Atlanta) I I I 1~ aarnet1 (New Zealand) : {Taiwan) {Taiwan) I : (S. Korea) : {China) : : {France) : (D.C.) J J {Brazil) I 1 1 (Chicago) I {Chicago) I {Chicago) I {Chicago) I ~ (Auckland) I (Chicago) 1 1 _ _£N~~Y_:> ~ )- _I I_ i5~o!~u~L_ _ I : { Australia) : - - - f ---- I (Seattle) I ◄ - ~------, •◄--• I ANA300 ◄ ---• I Overlay Network I I (Los Angeles) I 1 Federal Agency Networks, Regional Network 1 I I 1 State Wide Networks, University Neworks, Testbed Networks 1 ESNet 1 00G If Not Specified I~------(Chicago) I ◄ .. I PetaTrans: Petascale Sc1e ces Data Ta sfer Singapore Sydney/Brisbane Daejeo rn CERN ,, - ·- ·--·- ·- ·- .I- ..... --,...., I ( I I DTN I I I I I @100G I I I I I I I l,__ _1111 __ j \ _.,,, I Seattle/L. A./Sunr1 yVille

,,,,.. --·----·---, ( \ I rn.,,.~~~~~ I I .---.1-'!l.. F I I I Amsterdam 1, _IJ_RP~Sites / ' I I I' ~------· I I I I f I I ,,,. J I , _CEN______ll/ciana I / I ...... I Ottawa/Hanover I S T •t- R L .I G H T 1 ,. t.-1 OpLlc11l &rl.itiA' T.Ail!' .., I ------·- · _.,.,, Chicago JG DATA

4x OOG OT \ lll ~ net SCinet L1 SCine L1 DIii Z9332F 400G ~~----- I I \\\\ \\\\ ~~ SC1net L2/3 Ill/ Ill! Ill/ Jll !!tr' 2CRSI Booth 1801 ------~~:=JL------:~!.-.,__ ___ ADR FPGA I

Starlight/UIC AS RL Booth 993 Caltech/ESne AmLight/Yale/ ortheastem Booth 543 ~ CENI NRE Map for SC20

Blfll .. H ll[WFOUNDLAN n Ma111, Canada .AND LAIRAOOW

MA OIIA e&nonton•

OUlllC V;inr;OUY f 0

0 IAAIO v,ct10ria Srattlr 0

A,HINGTON 200G ,eoec c,iy Portland 0 0 CENI U OVA S ... 011A

MORTII C-4k0TA oorBON List of NREs:

IDAHQ ·100G SOUlll NRENRLB DA,KOl A SL-3 rr iC SL-4 100G Sr SL-5 San Francisco SL-7 .. EB 10 • SL-9 S.s •Jase NRE-12 ILLIHO S SL-8 r ..I OMN A OlOR 00 Kan,asCny a IC.ANSA. SC20 Experiments/Demonstrations Testbed

08/26/2020

MfieN Cio.. 87IXhl "' ,~ ""'1 S5S 18!0 """1"'2 ,.., 1SQ3 1:!10 ""° - """ ....., xt-2..J.::U.l-ll\tl ..,. 13,12 Ql'U11Sl.g(d,1:ll) l,l lf6, •1:_ • 4, """ 1015 "'-il l"I I ""' 1 on.41~ 1l l~ , 1;, 11, lt lS, ,.,,. -~ MJ\nJ

,,.. ""'25 114 1M 10 t,"3 1 1"2 """ """ ..... ;no 1122 "'" iCAIR 29100 O\omolom 29100

...,, 1121 1115' 117 ms 1,.. 1~ 7 1m ,... 115 1/0 ,o 10 111 °"'" 1862 !J!i(,. 1~11) ·~ 1e&:I '"' !65 ,... ,"""... , 1D1 S ,.., !65 ... 5$ 2040 1866 2113.! ,... ..,._ ,... ,.., UnllQ 204 1 1!167 , , ... ""',.,, "',... 1872 19 11 .. ,.,, 5$ - """ """ """' ""' --- Ti:::.~

I O I ~ ~ ~ ~ lflH!--e- ~=~====t;:;:::_=;-;P.

~ ST RLIGHT SM Design and implementation

SOinet DTN-a~-a-service NUMA~fieme

OS_Conlrolled Send~r:MachlneConf BIND_TO_ UMA BIND_TO_ COflE NtJM!A Jupyt0r controllor Disk ,config CIPU Contaioo:r Type INDIVIDUAL MD_RAID Dis Manager eceiver:Machine<:onf ZFS_RAID· NUMlA Test:ies1Case OPU ::' f S®porled rnod~les / i Container type ~tys \let ., , \ Nat.-nrk~ , u~~ ~ 1 1 Optim zation tesl ! ~~r'5 L : \ ~ .....~~~ / ,.-J,_,_.!bo~--•-,~ .... lL" •' ~.,.... ', • ' "~ J ••• •·· ••' ·.1• :··--- i NIC ·. \ j. ~irt454• \ / ' ii;,srl3 TllSt Loa · r Transfer tesl / ':1,/1. ___ I O?timizer procedure ; , i i , ' OS 'i \ ~ SOX YFIF ~t !_ nuncp Sflav, Op ·mizalion ,'.i[ ', t, ; >---◄ I . ' : ...... N\!McoF' / \ Kuberneli!i •. d'd lsM Network Provilion •.. ,___ -!'" ... OpenNSA Kubernetes _,, Docker ___ExtraCI praw Nuttop

Thre•ad SSH Science Workflow in DTN port Thread VI StJ lze rt Pro111 elhe 1.J s II') UiXDB 1. P~;irQ 2, lw~ Ill< 8 ~alii!ifer Sl!liJp Server ilperf3

Ti meUlll!I Thread SSH dport Thread

Source: Se Young Yu, iCAIR Jupyter Client for NSI OpenNSA Integration Securely allow users to run NSI OpenN SA services (i.e. dynamically sti tch layer 2/3 circuits based on technolog ies e.g. VLAN .) Features: Authentication , Authorization , Accounting Au thentication: SS L/TLS operation authenticate bo h server and clien t. (certifi ca tes need to be exchanged pni or unl!ess using public certificates) Au thorization : server willl allow access us,ers to r,equest servi ces on cert ain ports/VLAN s based on user 'dentification. I.e. Request com mand arguments will be parsed and authorized if allowed. Accounting : for future requirement. Additi onal Feature:Asynchronous(non-blockin g operation allows multiplle users to request services simultaneously)

Server Client Step 1: establish TCP connection witll SSUTLS dient-server authentication ,------f 008 - CTN-as-a-Service Registration Verify clie nt ■ Verify server Remote Jupyter Client certifi cate ■ certificate ■ Open NSA Server - Opennsa Step 2: ctient req uest service and: server Jupyter Client will parse the com mand arguments Se na request TCP Au th orize req uest and AM. Server sena l)ack output/error - Note : certain comma nds willl ■ retu m timel¥ responses ■ OpenNSA ■ Application Close .. Step 3: Finish and clos e the connection Close confi rm

Source: Se Young Yu, iCAIR SCinet DTN-as-a-Service @ SC20

• For SC20, DaaS supports XNET Experiments, 12+ NREs: ROCE over WAN, P4 Experiments/Demonstrations, SENSE/openNSA integration, Kubernetes Federation, PCI-e Gen4 DTNs and Many More, Including The “Bring-Your-Own- Testbed Concept” erv ce: ,,------A 10 9 p rts :' I 'I ' Cl

SL-PetaTrnm Q! X 100G •OSH

. OTN I 1 @10Gl100G

GSFC & '4RL @6 X 100G

s

GSFC .!QOC Nodt

1 ~ ST RLIGHT SM NREOS-GRP Service: International P4 Experimental Networks(iP4EN}!:Cinet

( £ ~!b~ ---1 .. m.. 6.s I l cwna I ..._ __Ottaw a_ )

~~ ---, PACIFIC WAVE ~ etFFEFI I C I a 'f;;; ~ ~ l PP 1 • -·----· r-~~~ ---1 \. Ta oyuan I .....__....,.. UCSanDiego ---✓ I t'it)calit snscl ..._ S__an Die go_ /

~ ST RLIGHT SM NREll-GRP Service: Research Platforms Federation Demonstration S:inet

Goal: Secure multi-domain resource sharing cross regional, national and international research platforms Solution: Admiralty. The software enables users to schedule workloads in a different cluster by federating the source and target clusters. Participants: • Global Research Platform(GRP) • Pacific Research Platform(PRP) • MREN Research platform(MRP) • Towards National Research Platform(TNRP) • PacficWave • KISTI

~ ST RLIGHT SM ~ _· ••e ata. XJ.ll'less~: owa . · re·. 1cta . e?I ·._·c: ie . 1u '. a ... e, c -FL"'i'IT• lilb· H ig'h,... pe :rfonnan:ce 1oa1ta ·wransfer ·..:-- ~,.fBP:G v

I ... -,- ..... t m a i'3 a

.~ s T R LI G H T SM ROBIN - PI Wenji Wu Fermi National Accelerator Laboratory

ROBIN (_BuciQ/.filgData Express/SENSE) A Next Generation High performance Data Service Platform

Scientific Applications (e.g. , LSST, CMS) ,.______, ------········------·-·---·------········· ------Rucio - Smart Namespace - Storage Support - Consistency Data Management Service - Easy Integration - AA Support - Proven Track Record ------.....------, - End-to-end Performance Optimization BigData Express - Peer-to-peer, Scalable, Extensible High-performance Data Transfer Service - High Performance Data Transfer Engine - Cllogon-base Security ,.______, ------·-··------····-···------·-···-·------SENSE - Software Defined-Networking (SDN) for End-to-End Virtual Guaranteed Network Smart Network Service - A Highly Intuitive "lntent"-based lnterfac.e

~ ST RLIGHT SM ROB IN (_BuciQ/.filgData Express/SENSE) A Next Generation High performance Data Service Platform

Rucio Client

( Ruc io CU ) ( Python Client APls )

Rucio Se rver

( Core Services ) ( Web Server ) ( Task Queue )

( Trnnster swm itter) ( Trnnsf er Poller ) ( Trnnsfe r finisher ) ('----- B,igD»-afa--~ Express Plug,~.i_ n______, )

Scheduer

Data Transfer Launching Agent t ( mdlmFTP )

( DTN Agent )

(storage Agen~

DTN

Q ST RLIGHT SM ROB N Cross-Atlantic estbed

. ESnet 'I CERN, Switzer1and r------=-----. i !( SENSE Servas ) ! Head Node \.. :.: .., I BOE Por1ail Starlight site: ♦ ( ) BOE Scheduler • DTN: dtnl 10.sl.startap.net, with several Intel Dote Transfer NVMe drives for data storage, a 100GB Mellanox Launchina Aaent NIC for data transfer, and a lG IC for control. dtn01 .cern.ch • Head node: 165.x.x.157, with a lG NIC for control. UlllOP ( Rudo C ·eot) CERN site: t · Starlight, USA ·1 I • DTN: dtnOJ .cern.ch, with a rotational disk for data

1GE ______j_ 1 storage, a 1OG E Mellanox NIC for data transfer,

I 1 1~157, ' C .. 142 ,------.,. and JG NIC for control. I 'I R Server 'I • Head node: cixp-urfnet.cern. ch, with a 1G NIC for I ' Head No

~ ST RLIGHT SM The Persistent Multiresource Infrastructure ~GNA-G Persistent Multiresource Infrastructure 2020 update 4 SENSE .... -· Orcheltnltor ...... -·········-·...... ·· •· ···-... ··-­ .•. .-·· -- ... --····· MEICAN •······ .. --··· ··• ..... __ -········ ·· -- -· ... ----, ., .. -•·· ------····· ··- •isn [S,,e, 9w1._,.. Ce.1ect1 5El

..... °"'""'" D1Nl 01N2 0lN 1 01111 OTl<2 DTN t DTN2 DlN D1112 0 . IC.AJA RNP R ,_ caN ...... D- """""

SEMSE COfllfOllf

DIN SAND

JCA111•SA

HoaVUN1 HOIIVL.AHs -"""'- JN:r89 lt91$-39!8 39S5·391&

[ht N~'IJOffl o,50Rf

AutoGOLE

~ ST RLIGHTSM StarLight

COIie

LJ [i] - by Tom Lehman ...... e.m~•....,.Df - ~ AutoGOLE-SENSE – Tom Lehman ST RLIGHT SM IRNC-OSG Global Fabric PRP/TNR Co nnects

CITRIS B ATAO < I STITUTE

12 NYC 100G FIONA e 10G FIONAB NYSERNet C 100G 192TB 100G FIONA Clemson U Hawa 40G 160TB 40G 192 100G FIONA Composable Platform as a Service Instrument for Deep Learning & Visua lization (COMPaaS DLV) Using the Liqid Composable Infrastructure System for ...

... on-demand post-processing, ... interoperable Science DMZs and Data Transfer Nodes ... SAGE2™, the de facto operating system for managing Al, deep-leaning, CFD, and (DTNs) at a national/ international scale, using best Big Data on tiled-display walls, ena bling local and visualization. strategies from the Pacific Resea rch Platform (PRP) remote collaborators to access, share and interact with a over the GLIF infrastructure. variety of digital media.

Jevl SFaward CNS-1828265 o UIC/EVl for COMPaaS DLV Liqid: h ps:// li'WW.liqid.com/ Electronic Visualization Laboratory https://www.evl.uic.edu/co mpaas-dlv PRP: https://prp.ucsd.edu G IF: h ps:// vww.glif. is University of at Chicago SAGE2 is trademark of the Universrry of Illinois Board of Trustees https://www.evl.uic.edu SAGE2: http://sage2 .sagecommo s.org/ ul , Da

HI A 1 't cc renc1 Ilya Baldine PI, RENCI: FABRIC Next GRP Workshop

• Date: September 13-17, 2021 • Place: Innsbruck Austria, Co Located With eSCience Conference

• Global Research Platform Website: • www.theglobalresearchplatform.net THANKS!

• Academia Scinica, AmLight, AARnet, Asia-Pacific Research Platform Consortium, AutoGOLE Consortium, CalTech, CANARIE, CENIC, CERN, Ciena Research Lab, Esnet, , Fermi National Accelerator Laboratory, Global Research Platform Consortium, International Center for Advanced Internet Research Northwestern University, KISTI, LHC Networking Consortium, NASA Goddard Space Flight Center, Metropolitan Research and Education Network (MREN), MREN Research Platform Consortium (MRP), Mid-Atlantic Crossroads (MAX), Naval Research Laboratory, National Energy Research Scientific Computing Center (NERSC), National Science Foundation, Pacific Research Platform, Pacific Wave, RNP, SingAREN, StarLight International/National Communications Exchange Facility Consortium, SURFnet, TWAREN, University of San Diego, University of Illinois Chicago, University of Amsterdam, et al www.startap.net/starlight

Thanks to the NSF, DOE, NASA, NIH, DARPA Universities, National Labs, International & Industrial Partners, and Other Supporters

' ST RLIGHT SM

"Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Networking and Information Technology Research and Development Program."

The Networking and Information Technology Research and Development (NITRD) Program

Mailing Address: NCO/NITRD, 2415 Eisenhower Avenue, Alexandria, VA 22314

Physical Address: 490 L'Enfant Plaza SW, Suite 8001, Washington, DC 20024, USA Tel: 202-459-9674, Fax: 202-459-9673, Email: [email protected], Website: https://www.nitrd.gov

NITRD

~