SC20 Advanced Networking For Data Intensive Science Experiments and Demonstrations Joe Mambretti, Director, ([email protected]) International Center for Advanced Internet Research (www.icair.org) Northwestern University Director, Metropolitan Research and Education Network (www.mren.org) Co-Director, StarLight (www.startap.net/starlight), Director, StarLight International/National Communications Exchange Facility (www.startap.net/starlight), PI IRNC: RXP: StarLight SDX, Co-PI Chameleon, PI-iGENI, PI-OMNINet JET Meeting November 17, 2020 International Conference On High Performance Computing Networking, Storage and Analytics November 9-20, 2020 GRP: Services, Architecture, Technology • Architecture: “Global Science DMZ” • Services Optimized For Science Workflows • High Performance Transport Over WANs For Large Capacity Data Streams E2E • Enhanced Virtualization • Highly Programmable • Specialized Components (Software Stacks, Next Gen DTNs, Advanced APIs, Customized Devices, Federation, INT, P4, etc.) • Interdomain Dynamic Provisioning • Production Resources Adjacent to Testbeds Next Generation Distributed Enviroment For Global Science Global Research Platform Network 100 Gbps StarLight AutoGOLE Fabric 2020 (60+ StarLight SDX Overview • This IRNC StarLight SDX Initiative Is Designing, Implementing, and Operating New Services For Global Data Intensive Sciences, Based On Emerging Next Generation Architecture and Technolgies, Including Virtualization, Orchestration, Segmentation (Slicing), Software Defined Resources, Programmability and Customization. • These Macro Trends Enable Exchanges To Be Agile Platforms For Dynamic Services Provisioning, Real-Time Responsiveness, and Distributed Control Over Core Resources, Including By Applications, Edge Processes and Devices. This Project Is Transitioning Network Exchanges To Open Innovation Platforms r ICT Testbed ......... and l I OevelopmentPromotJon Cente• I r • 1 1 :~,NET 1-.~ ~ JGN I I I •• I I SINET : (Japan) : : (Japan) 1 (Los Angeles) 1 1 (Los Angeles) 10G NX100G I l I I I I I (Chicago) (Amsterdan) :-1~t; r~a~i~n~ 1-] 110G 1- -- - -~ - -- ---- I SON Networks I I I ""'l ◄.-----)1,..,1 -?- ~:;:::: I & Testbeds .,_ 1 1 2 1 1 1 (China) : ....--... PACIFIC WAVE 1..,.1- x__0_0_G__ _, 1, ___., 1.._____, I I (Los Angeles) I I (Chicago) I I 1_ _ _ _ _ _ _ _ 1 I I l I NORDUnet I S ~ ng/\ReN : ........ ............... ,_ ..... ,.. ., - ..... , I I I : (Singapore) I 2x10G (Nordic) (Los Angeles) 1 (Chicago) I I r I I I__ ... I 1◄◄1-----, 2x10G 10G 10 : (Honolulu) : ___ _t ___ _ l I l l I 1 (Mauna) I l l ] [ Southern Light ] I I / ~ I -t ltl "t4llill 1 : OpenTransits : CMoo .... AtlanticWave ----i--- RE/\N N Z i I , i--. ...... 1 1 1 ■ 1 REN 1 : KRt.~ :)-JET : 1 Network 1 r- - - - - - - \ I I I I CSTNET : I I {Miami) ~ : RNP : I {Atlanta) I I I 1~ aarnet1 (New Zealand) : {Taiwan) {Taiwan) I : (S. Korea) : {China) : : {France) : (D.C.) J J {Brazil) I 1 1 (Chicago) I {Chicago) I {Chicago) I {Chicago) I ~ (Auckland) I (Chicago) 1 1 _ _£N~~Y_:> ~ )- _I I_ i5~o!~u~L_ _ I : { Australia) : - - - f ---- I (Seattle) I ◄ - ~------------------------------------, •◄--• I ANA300 ◄ ---• I Overlay Network I I (Los Angeles) I 1 Federal Agency Networks, Regional Network 1 I I 1 State Wide Networks, University Neworks, Testbed Networks 1 ESNet 1 00G If Not Specified I~------------------------------------ (Chicago) I ◄ .. I PetaTrans: Petascale Sc1e ces Data Ta sfer Singapore Sydney/Brisbane Daejeo rn CERN ,, - ·- ·--·- ·- ·- .I- ..... --,...., I ( I I DTN I I I I I @100G I I I I I I I l,__ _1111 __ j \ _.,,, I Seattle/L. A./Sunr1 yVille ,,,,.. --·----·---, ( \ I rn.,,.~~~~~ I I .---.1-'!l.. F I I I Amsterdam 1, _IJ_RP~Sites / ' I I I' ~--------· I I I I f I I ,,,. J I , _CEN_________ll/ciana I / I . .... I Ottawa/Hanover I S T •t- R L .I G H T 1 ,. t.-1 OpLlc11l &rl.itiA' T.Ail!' .., I --------·- · _.,.,, Chicago JG DATA 4x OOG OT \ lll ~ net SCinet L1 SCine L1 DIii Z9332F 400G ~~----- I I \\\\ \\\\ ~~ SC1net L2/3 Ill/ Ill! Ill/ Jll !!tr' 2CRSI Booth 1801 ------~~:=JL-------:~!.-.,__ ___ ADR FPGA I Starlight/UIC AS RL Booth 993 Caltech/ESne AmLight/Yale/ ortheastem Booth 543 ~ CENI NRE Map for SC20 Blfll .. H ll[WFOUNDLAN n Ma111, Canada .AND LAIRAOOW MA OIIA e&nonton• OUlllC V;inr;OUY f 0 0 IAAIO v,ct10ria Srattlr 0 A,HINGTON 200G ,eoec c,iy Portland 0 0 CENI U OVA S ... 011A MORTII C-4k0TA oorBON List of NREs: IDAHQ ·100G SOUlll NRENRLB DA,KOl A SL-3 rr iC SL-4 100G Sr SL-5 San Francisco SL-7 .. EB 10 • SL-9 S.s •Jase NRE-12 ILLIHO S SL-8 r ..I OMN A United States OlOR 00 Kan,asCny a IC.ANSA. SC20 Experiments/Demonstrations Testbed 08/26/2020 MfieN Cio.. 87IXhl "' ,~ ""'1 S5S 18!0 """1"'2 ,.., 1SQ3 1:!10 ""° - """ ....., xt-2..J.::U.l-ll\tl ..,. 13,12 Ql'U11Sl.g(d,1:ll) l,l lf6, •1:_ • 4, """ 1015 "'-il l"I I ""' 1 on.41~ 1l l~ , 1;, 11, lt lS, ,.,,. -~ MJ\nJ ,,.. ""'25 114 1M 10 t,"3 1 1"2 """ """ ..... ;no 1122 "'" iCAIR 29100 O\omolom 29100 ...,, 1121 1115' 117 ms 1,.. 1~ 7 1m ,... 115 1/0 ,o 10 111 °"'" 1862 !J!i(,. 1~11) ·~ 1e&:I '"' !65 ,... ,"""... , 1D1 S ,.., !65 ... 5$ 2040 1866 2113.! ,... ..,._ ,... ,.., UnllQ 204 1 1!167 , , ... ""',.,, "',... 1872 19 11 .. ,.,, 5$ - """ """ """' ""' --- Ti:::.~ I O I ~ ~ ~ ~ lflH!--e- ~=~====t;:;:::_=;-;P. ~ ST RLIGHT SM Design and implementation SOinet DTN-a~-a-service NUMA~fieme OS_Conlrolled Send~r:MachlneConf BIND_TO_ UMA BIND_TO_ COflE NtJM!A Jupyt0r controllor Disk ,config CIPU Contaioo:r Type INDIVIDUAL MD_RAID Dis Manager eceiver:Machine<:onf ZFS_RAID· NUMlA Test:ies1Case OPU ::' f S®porled rnod~les / i Container type ~tys \let ., , \ Nat.-nrk~ , u~~ ~ 1 1 Optim zation tesl ! ~~r'5 L : \ ~ .....~~~ / ,.-J,_,_.!bo~--•-,~ .... lL" •' ~.,.... ', • ' "~ J ••• •·· ••' ·.1• :··--- i NIC ·. \ j. ~irt454• \ / ' ii;,srl3 TllSt Loa · r Transfer tesl / ':1,/1. ___ I O?timizer procedure ; , i i , ' OS 'i \ ~ SOX YFIF ~t !_ nuncp Sflav, Op ·mizalion ,'.i[ ', t, ; >---◄ I . ' : ...... N\!McoF' / \ Kuberneli!i •. d'd lsM Network Provilion •.. ,___ -!'" ... OpenNSA Kubernetes _,, Docker ___ExtraCI praw Nuttop Thre•ad SSH Science Workflow in DTN port Thread VI StJ lze rt Pro111 elhe 1.J s II') UiXDB 1. P~;irQ 2, lw~ Ill< 8 ~alii!ifer Sl!liJp Server ilperf3 Ti meUlll!I Thread SSH dport Thread Source: Se Young Yu, iCAIR Jupyter Client for NSI OpenNSA Integration Securely allow users to run NSI OpenN SA services (i.e. dynamically sti tch layer 2/3 circuits based on technolog ies e.g. VLAN .) Features: Authentication , Authorization , Accounting Au thentication: SS L/TLS operation authenticate bo h server and clien t. (certifi ca tes need to be exchanged pni or unl!ess using public certificates) Au thorization : server willl allow access us,ers to r,equest servi ces on cert ain ports/VLAN s based on user 'dentification. I.e. Request com mand arguments will be parsed and authorized if allowed. Accounting : for future requirement. Additi onal Feature:Asynchronous(non-blockin g operation allows multiplle users to request services simultaneously) Server Client Step 1: establish TCP connection witll SSUTLS dient-server authentication ,------f 008 - CTN-as-a-Service Registration Verify clie nt ■ Verify server Remote Jupyter Client certifi cate ■ certificate ■ Open NSA Server - Opennsa Step 2: ctient req uest service and: server Jupyter Client will parse the com mand arguments Se na request TCP Au th orize req uest and AM. Server sena l)ack output/error - Note : certain comma nds willl ■ retu m timel¥ responses ■ OpenNSA ■ Application Close .. Step 3: Finish and clos e the connection Close confi rm Source: Se Young Yu, iCAIR SCinet DTN-as-a-Service @ SC20 • For SC20, DaaS supports XNET Experiments, 12+ NREs: ROCE over WAN, P4 Experiments/Demonstrations, SENSE/openNSA integration, Kubernetes Federation, PCI-e Gen4 DTNs and Many More, Including The “Bring-Your-Own- Testbed Concept” erv ce: ,,---------- A 10 9 p rts :' I 'I ' Cl SL-PetaTrnm Q! X 100G •OSH . OTN I 1 @10Gl100G GSFC & '4RL @6 X 100G s GSFC .!QOC Nodt 1 ~ ST RLIGHT SM NREOS-GRP Service: International P4 Experimental Networks(iP4EN}!:Cinet ( £ ~!b~ ---1 .. m.. 6.s I l cwna I ..._ __Ottaw a_ ) ~~ ---, PACIFIC WAVE ~ etFFEFI I C I a 'f;;; ~ ~ l PP 1 • -·----· r-~~~ ---1 \. Ta oyuan I .....__....,.. UCSanDiego ---✓ I t'it)calit snscl ..._ S__an Die go_ / ~ ST RLIGHT SM NREll-GRP Service: Research Platforms Federation Demonstration S:inet Goal: Secure multi-domain resource sharing cross regional, national and international research platforms Solution: Admiralty. The software enables users to schedule workloads in a different cluster by federating the source and target clusters. Participants: • Global Research Platform(GRP) • Pacific Research Platform(PRP) • MREN Research platform(MRP) • Towards National Research Platform(TNRP) • PacficWave • KISTI ~ ST RLIGHT SM ~ _· ••e ata. XJ.ll'less~: owa . · re·. 1cta . e?I ·._·c: ie . 1u '. a ... e, c -FL"'i'IT• lilb· H ig'h,... pe :rfonnan:ce 1oa1ta ·wransfer ·..:-- ~,.fBP:G v I ... -,- ..... t m a i'3 a .~ s T R LI G H T SM ROBIN - PI Wenji Wu Fermi National Accelerator Laboratory ROBIN (_BuciQ/.filgData Express/SENSE) A Next Generation High performance Data Service Platform Scientific Applications (e.g. , LSST, CMS) ,.__________________ ___, ---- -- --- --- --········---- --- ------- ---------- ----- --·-·---·- ---------- ----- ------------ ········· ----- --- Rucio - Smart Namespace - Storage Support - Consistency Data Management Service - Easy Integration - AA Support - Proven Track Record ---- ------- --- ------------ ------------- ------- ---------- ---------- ----------------------