Hsin-Yen CHEN ASGC GDB @ Taipei 3 Apr. 2019

Development of the LHCONE in Asia

Hsin-Yen CHEN ASGC GDB @ Taipei 3 Apr. 2019 After WWW, CERN decided in 2000 to develop and deploy distributed computing to support a new generation of big data driven research. Middleware had been developed for geographically distributed cloud centers to share resources with applications to particle physics research over the past 15 years.

ASGC was established to join the international collaboration centered at CERN to build a platform for distributed computing to support e-Science.

By collaborating with CERN, we stay in the forefront of distributed computing. WWW was Invented at CERN 5 SEPTEMBER 2008 VOL 321 SCIENCE Building Advanced Distributed Computing Platform Strategy: Collaborations with Research Groups - Applications Drive & Technology Push R&D: DiCOS, System Efficiency & Machine Learning Applications

Application (Science) • Deeper Understanding Natural Disaster • ATLAS, CMS (High Energy Physics) • Soundscape Monitoring Network • Alpha Magnetic Spectrometer (Particle & Astronomy) • Earth Science • KAGRA, VIRGO (Gravitation) • Cryo-EM (2017) • TEXONO (Neutrino) • Computational Biology (2017) • World Wide Grid Computing (CERN) • Advanced Networking (iCAIR) • Bioinformatics (U. Chicago) • Proton Therapy (NCU, CGU/CGMH)

DiCOS: Distributed Cloud Operating System

• Leveraging WLCG technologies and moving from petabyte-scale towards Exabyte scale: • ~100PB in 2018 and expected to have ~1000PB in 2025; • Mobilizing 10PB/month • 2M+ jobs/day • Extending beyond high energy physics • Growing our own R&D Capability 3 ASGC Global Network Sep. 2013 ~ Sep. 2018

10G Amsterdam Chicago 10G Geneva Tokyo Palo Alto New York 1G 10G Hong LA Kong 10G TAIWAN 10G

ASGCNet ASGC Global Network 1 Sep. 2018

Amsterdam 10G Geneva 10G 2x10G

Chicago 10G

TAIWAN 10G 10G 10G

Singapore TAIWAN Global R&E Network

Amsterdam

Chicago 2x10G Geneva Tokyo 1G 2.5G 10G Palo Alto 2.5G 10G New York 2.5G 1G Hong 10G *2 10G LA Kong 2*2.5G TAIWAN 2x10G Singapore 2x10G

ASGCnet (AS 24167) ASNet (AS 9264) TWAREN (AS 7539) TANet (AS 1659) ASGC e-Science Global Network

TANet Internet2 ESNet LHCONE CANARIE LHCONE FNAL 100G NASA LHCONE TWAREN SINET ASNet LHCONE STARLight LHCONE iCAIR U. Chi 10G 10G 20G JGN/NICT 10G APAN-JP 10G LHCONE TW US NORDUNET LHCONE AARNet/AU 10G CERNET/CN 10G 20G 20G CERNLight CSTNET/CN 10G LHCONE ERNET/IN 10G RU KIAE NKN/IN 20G LHCONE HARENET/HK SG NL 20G INHEREN/ID CERN KREONET/KR 10G KOREN/KR LEARN/LX 10G 1G 10G GEANT MYREN/MY LHCONE NREN/NP PERN/PK TEIN AMSIX UNINet/TH SARA VinaRen/VN LHCONE SingaREN/SG SURFNet LHCOPN

Private network connecting Tier0 and Tier1s - Dedicated to LHC data transfers and analysis - Secured: only declared IP prefixes can exchange traffic - Advanced routing: communities for traffic engineering, load balancing.

T1 T1 T0

T1 T1

8 Numbers - 14 Tier1s + 1 Tier0 - 12 countries in 3 continents - Dual stack IPv4- IPv6 - 660 Gbps to the Tier0 - Moved ~160 PB in the last year

https://twiki.cern.ch/twiki/bin/view/LHCOPN/OverallNetworkMaps

9 HL-LHC plan

Here today

10 LHCONE VRF: A global high performance Science DMZ network infrastructure for High Energy Physics data analysis (LHC, Belle II, Pierre Auger Observatory, NOvA, XENON, DUNE) Bold black lines in the following plot are 100Gb/s backbone of LHCONE to support the ExaBytes science data transfer

KIAE PIONIER CSTNet China CANARIE Poland PSNC CERNet2 China JANET SURFsara Cyfronet AGH Kraków, Europe China Next IHEP, Beijing Asia America Canada MOXY Nikhef U.Warsaw ICM Generation CMS ATLAS SINET+ SimFraU TRIUMF-T1 Montreal UK ESnet (Montreal) NORDUnet Internet, IC Netherlands Internet Uvic, Utor, UBC, McGill NL-T1 Nordic KIAE/JINR Exchange, KREONet2 GÉANT, NDGF-T1, MAN LAN, Helsinki Moscow INR Beijing NDGF-T1b Russia CNGI-6IX NDGF-T1c (Troitsk) ICEPP RRC-KI T1 HEPNET U Tokyo SINET JINR T1 JINR, IHEP Protvino, GridPNPI, Japan Sarov KISTI RCC-KI T2, ITEP Hirosh., Tsukuba, KEK T1 Tokyo SANET KREONet2 Tokyo DFN DESY Slovakia Korea Germany URAN FMPh- KISTI –T1 PNU Tokyo Global Research Platform Network (GRPnet) KIT DE-KIT-T1 UNIBA xx CERN SeatleCANARIE

KNU, KCMS Osaka Ukraine IEPSAS- to 2 RWTH, Wupp.U, Kharkov-KIPT

Tokyo PNWG, Kosice KREONet Starlight CANARIE GSI KREONet2, PacWave / SINET, (Chicago) ESnet, CERN, MREN,

CANARIE, PNWG 2 ESnet Internet2ASGC , GÉANT, ESnet (same VRF as CSTNet Osaka (Seattle) GARR, CANARIE, ANL, KREONet KRLight ESnet Europe) CESNET

GÉANT to ESnet USA FNAL, KREONet2, CERN, ht

Orient+) Daejeon via PacificWave, GRPnet, NORDUnet, GÉANT, Czechia to BNL-T1 BNL-T1, FNAL-T1 Chicago

(London via (London Seattle, CANARIE ASGC, KIAE/RU, CSTNET, GÉANT, Caltech, prague_cesnet_lcg2, CANARIE

ESNET NORDUn NetherLig (Amsterdam) Europe

SINET PNNL, SLAC, ANL, ORNL CUDI NL-T1, U. Mich, FZU/praguelcg2 et

, , LA AmsterdamSURFsara, ESnet, Hamburg

PSNC, KREONet2, Poznan (New York)(New

ESnet, ESnet, KIAE/JINR Internet2,

MAN KREONet2

LAN CANARIE via via CANARIE Internet2, CERN,

PacWave Amsterdam NORDUnet MANLAN JGN MIT, UChi, UFla, CERN to ASGC Frankfurt

UOak, Vand, Prague via Orient+ GÉANT GÉANT to

Japan GÉANT (all of Paris Beijing Beijing JGNHong NRENS and sites) Vienna PNWG GEANT

Kong and Brazilian sites), London

exchange) (distributed (distributed

HK PacificWave ESnet, NORDUnet, New York Budapest PacificWave/ KIAE/RU (HEPNET, GÉANT Hong Kong UWisc Geneva London HKIX GRIDPNPI, IHEP),

to Brussels

ESnet CANARIE Internet2

- Singapore Hong Kong AGLT2 Harvard GÉANT to

TEIN MIT TEIN

U Chicago, DePaul U, SINET Bucharest TEINMumbai PacWave UM Lisbon Illinois Century Network (ICN), NET2 (Sunnyvale) P AGLT2 Illinois State U, UIUC, Loyola U, BU Madrid Milan

BTAA NCSA, Northwestern, (Chicago) Internet2, SINET, MSU OmniPo ASGC Southern Ill U, U Minn., Singapore RedIRI

Caltech, UCSD, U Illinois Chicago (UIC) S Taiwan AARNet, UCSD UT WiscNet, U Michigan (UMich), NKN, India to Arlington JGN ASGC-T1 Michigan State U (MSU) GÉANT, TIFR ASGC CANARIE, ASGC2, NCU, UNL UIC NORDUnet, GARR PERN NTU UCSB UChi ThaiREN (MWT2) SINET GÉANT CERNLight ESnet Indiana Korea Pakistan Thailand Open NCP-LCG2 SingAREN MyREN UCSD UIUC PacWave (MWT2) USA London CERN PK-CIIT Singapore Malaysia IU(MWT2), ND, Los Angeles Purdue ESnet Geneva London CERN-T0 SINET, Caltech WIX Geneva Internet2 CERN-T1 CANARIE, Internet2 (Washington) Amsterdam SG ESnet ESnet Internet2 AARNet

NKN NKN USA CERN

India India Australia Vanderbilt, UFlorida, SOX AtlanticWave

(distributed (distributed to Los Angeles Internet 2 exchange) CERN UMel NE, SoW, Harvard VECC, TIFR RENATER Belnet France CC-IN2P3-T1 LPNHE ARNES AMPATH Belgium LPC, APC, NAP of Americas BEgrid-ULB-VUB, Slovenia CPPM, LAL SiGNET xx (Miami) BegridNumbers-UCL, IIHE RoEduNet Ver. 4.31, Oct. 25, 2018 – WEJohnston, ESnet, [email protected] IPHC, IPNHO LLR, Romania GARR CANARIE NREN/site router at exchange point NOTES Subatech LHCONE VRF domain/aggregator CEA- UAIC xx, ISS, Communication links: 1) LHCOPN paths are not shown on this diagram - 20 R&E networks IRFU NIHAM, ANSP Network provider 1/10, 20/30/40, and 100Gb/s 2) The “LHCONE peerings” at the exchange points CUDI FCCN ITIM, NIPNEx3 indicate who has a presence there and not that GARR London PoP router Connection internal to a domain, Mexico Portugal RedIRIS all peer with each other (see PTTA - 14 Tier1s and ~70 Tier2s in 5 and of unspecified bandwidth UNAM Spain CNAF-T1 Italy Collaborating sites not yet https://twiki.cern.ch/twiki/bin/view/LHCONE/Lhc CUDI JGN – NAP do Brasil Underlined link information OneVRF ) RNP/IPĔ continents PIC-T1 INFN GRNET connected to LHCONE SingAREN/NSCC (São Paulo) indicates link provider, not use 3) See http://lhcone.net for more detail. Brazil CIEMAT-LCG2, INFN Bari, Catania, Greece Exchange point/regional R&E communication nexus Frascati, Legnaro, UAM-LCG2 Pisa, Napoli HEPLAB xx Redecomep - ~250 perfSONAR instances Milano, Roma1, w/ switch providing VLAN connections Ioannina ANSP Torino PNU LHC ALICE or LHCb site Sites that are REUNA SAMPA CBPF HEPGrid UNL standalone VRFs SPRACE CNAF-T1 LHC Tier 1 ATLAS and CMS Chile (USP) (UERJ) - 6 collaborations - yellow outline indicates LHC+Belle II site UChi LHC Tier 2/3 ATLAS and CMS LPNHE KEK Belle II Tier 1/2 } - Dashed outline indicates distributed site CEA- LHCONE: A global infrastructure for the LHC Tier1 data center and Tier 2/3 analysis center connectivity LHCONE workshop @ APAN 38 (2014)

SimFraU NDGF-T1 SCINET UVic UAlb CANARIE (UTor) NDGF-T1b NDGF-T1c NL-T1 NORDUnet GLORIAD TRIUMF-T1 Canada McGill (global) SARA Nordic KurchatovT1 Netherlands ICEPP Korea U Tokyo CERN-T1 SINET CERN MREN KISTI Japan (Chicago) Chicago Amsterdam (CERNLight) Korea (StarLight) (NetherLight) Geneva TIFR India Geneva CIC Indiana KNU AGLT2 (3) OmniPoP GigaPoP DESY KERONET2 UIUC (Chicago) RWTH Wup.U DE-KIT-T1 Korea MWT2 SLAC GSI DFN ESnet FNAL-T1 New York Germany CERN USA (MAN LAN) Seattle BNL-T1 India (PNWG)

ASGC-T1+T2 CENIC GÉANT ASGC2 USA Caltech ASGC Europe

Taiwan AGLT2 IU MSU MIT Los UNL AGLT2 AGLT2 UIUC Angeles UM WSU MWT2 NCU NTU CC-IN2P3-T1 UCSD IN2P3 CEA TWAREN Washington UWisc NE (10 sites) RENATER (IRFU) Taiwan UFlorida (WIX) PurU GLakes SoW France Vandebilt UNeb Internet2 Harvard USA PIC-T1 INFN CNAF-T1 RedIRIS (9 sites) GARR Spain Italy

UNAM CUDI End sites – LHC Tier 2/3 LHCONE VRF domain NTU unless indicated as Tier 1 Mexico LHCONE VRF aggregrator networks UNL Sites that are standalone VRFs

Chicago Regional R&E communication nexus Communication links, 10, 20, 30, and 100Gb/s 12 August 2014 See http://lhcone.net for details. 12 Progresses made in Asia since 2015

Asia Asia-Pacific VRFs 2017

Internet2 ESnet CANARIE US AS11537 US AS293 CA

AARnet AU AS7575

MyREN SINET JP AS2907 MY CERNlight CH AS20641 ThaiREN JGN TH AS24475 JP AS17934

ASGCNet TW AS24167 CERnet CN

TEIN CSTnet ASIA AS24490 CN KREONET KR AS17579

Asia-Pacific

NKN Not Asia-Pacifc IN LHCONE peering LHCONE transit TIFR GEANT LHCONE mutual transit IN EU AS20965 Connection in place, but no LHCONE Missing connection Asia-Pacific VRFs 2018

Internet2 ESnet CANARIE RU-VRF US AS11537 US AS293 CA AS 6509 RU AS57484

AARnet AU AS7575 NORDUnet EU AS2603

MyREN SINET MY JP AS2907 CERNlight CH AS20641 ThaiREN TH AS24475 JGN JP AS17934 INHERENT ID AS18007 ASGCNet TW AS24167 CERnet CN AS23911 TEIN TWAREN TW AS7539 CSTnet HK AS24489 SG AS24490 CN AS7497 KREONET KR AS17579

PERN Asia-Pacific PK Not Asia-Pacifc NKN LHCONE peering IN AS55824 LHCONE transit TIFR GEANT LHCONE mutual transit IN AS 2697 EU AS20965 Connection in place, but no LHCONE Missing connection Asia-Pacific VRFs 2019

Internet2 ESnet CANARIE RU-VRF US AS11537 US AS293 CA AS 6509 RU AS57484

AARnet AU AS7575 NORDUnet EU AS2603

Amsterdam

Chicago Daejeon Seoul Seattle Tokyo

LA London Hong Taipei Kong

ASGC JGN/SingAREN KOREN KREONET SINET Singapore SingAREN/I2 TransPAC NORDUnet/GEANT/TEIN/ SingAREN/AARNet

JGN supports LHCONE in Asia • JGN started the operation of KREONET LHCONE router at Hong Kong (KR) SINET (JP) • Big science is a driving force of next- TEIN gen ICT infrastructure (HK) JGN (Tokyo-Hong Kong)

• Under the collaboration with SINET, JGN ASGC (TW) JGN supports LHCONE between (HK) Japanese and Asian academic

organizations JGN • JGN established LHCONE peering with (SG) SINET, KREONET, ASGC, and TEIN-HK 19 LHCONE between ESnet and Asia • JGN also started the support of LHCONE between ESnet and Asia • JGN established LHCONE peering with ESnet, and exchange routing information between ESnet and Asia (TEIN-HK, ASGC, KREONET) • This is under the mutual cooperation with TransPAC • We also thank to WIDE to support the interconnection between JGN and TransPAC/Pacific Wave in Tokyo

KREONET (KR) ESnet (US) TEIN TransPAC/Pacific Wave (HK) JGN (Tokyo-Seattle) (Tokyo-Hong Kong) JGN (HK) ASGC (TW) JGN (SG) 20

LHCONE Connectivity (Draft Plan in 2019) 100G

VRF GEANT/EU SINET KEK AMS VRF 100G SNET Tsukuba U-Tokyo ICEPP VRF VRF backup JGN SNET SINET GEANT/EU Tokyo Tokyo NY

100G CANARIE/CA KREONET2/KR VRF backup JGN 100G 100G HK VRF ASGC/TW 100G SINET Esnet/US LA TEIN

SINET Internet2/US SG 22 LHCONE in China

• IHEP in China connected to LHCONE in Dec. 2017, Thanks to • CERN, CSTNet, CERNet, ASGC, GEANT, …… • VRF Peerings have been set up for IHEP-Eur. /IHEP-ASGC • Peering between CNGI-6IX and TEIN in Beijing • VRF Peering between CNGI-6IX and GEANT (over Orient+) • VRF Peering between TEIN and ASGC • More Peerings should be launched • Internet2、ESNet、 KREONET ……

E-Infrastructure for R&D

Pakistan’s National Education and Research Network, With Other NREN’s through Trans-EurAsia Information Network (TEIN)

• 155 Mbps Bandwidth in 2008

• Upgraded bandwidth 1Gbps in 2015 Digging in the LHCONE routing table

NORDUnet has run an analyses of the routing tables of most of the LHCONE VRFs.

It resulted that reachability is fragmented, especially on IPv6: - Only GEANT has a full view of all LHCONE destinations - Especially the sites behind TEIN cannot reach a fraction of LHCONE

IPv4 reachability map: https://indico.cern.ch/event/725706/contributions/3149436/attachments/1744301/2823447/go IPv6 reachability map: https://indico.cern.ch/event/725706/contributions/3149436/attachments/1744301/2823448/go

The community will work on improving this situation, which will be followed-up in the future meetings

27 ASGC LHCONE IPv4 Reachability ASGC LHCONE IPv6 Reachability LHCONE Looking Glass

At the LHCONE meeting of March 2018 it was decided to implement a tool to analyze the routing tables of the different VRFs For this, a Route Server and a Looking Glass web interface have been set up at CERN The looking glass can show the prefixes known in the different VRFs and the attributes of the routes At the moment the route-server peers with CERNlight and NORDUnet VRFs only. Other peerings will be added in the coming months

The looking glass is accessible at http://lhcone-lg.cern.ch/

30 Comment/Suggestion

• LHCONE Transit between Asia-US and Asia-EU • ASGC, JGN/TransPAC, TEIN/GEANT • Avoid asymmetric route between Asia and US/EU • MTU size 9000 • … etc Following LHCONE/LHCOPN meeting

• 4~5 June 2019, Umeå University, Sweden • https://indico.cern.ch/event/772031/ • January 2020: Workshop at CERN with LHC experiments • March 2020: co-located with ISGC in Taipei Thanks • Tony Cass, Edoardo Martelli (CERN) • William (Bill) Johnston, Michael OConnor (ESnet) • Vincenzo Capone (GÉANT) • Lars Fischer, Magnus Bergroth (NORDUnet) • Andrew Lee (TransPAC) • Hiroshi Sakamoto (U. Tokyo) • Tomoaki Nakamura (KEK) • Eiji Kawai (JGN/NICT) • Motonori Nakamura (SINET/NII) • Patch Lee, Zhonghui Li (Asi@Connect) • Buseung Cho, Sang-Un Ahn (KISTI) • Gang Chen, Qi Fazhi (IHEP) • Vikas Singhal (VECC, Klokata) • Brij Jashal (TIFR) • Rajesh Verma (NKN, India) • Saqib Haleem (NCP) • … etc