Infrastructure SINET 100G Global Ring and Data Exploitation

Sep. 18, 2019 Tomohiro Kudoh Professor, Information Technology Center, The University of Tokyo Cross Appointment Fellow, National Institute of Advanced Industrial Science and Technology (AIST) Visiting Professor, National Institute of Informatics (NII)

Data Exploitation Platform Two topics 1

• Overview of SINET • 100Gbps domestic mesh and international ring • Data Exploitation Platform • PoC environment for everyone/everyday data exploitation applications, leveraged by SINET

Data Exploitation Platform SINET (Science Information Network ) 2

• Japanese academic backbone network • Operated by NII (National Institute of Informatics) • Connects more than 800 universities and research institutions. • Connects many research facilities: seismology, space science, high-energy physics, nuclear fusion, computing science, and so on. • Being used by over 2 million users • Supports international research collaboration through international lines. • More than 100Gbps connection to every prefecture in Japan • 100Gbps international ring • Started mobile connection service

Data Exploitation Platform Slide provided by S. Urushidani @ NII History of SINET

1987 Packet exchange network as predecessor of SINET 1992 SINET as Internet backbone (29 sites, 6-50Mbps) 2002 Super-SINET for leading-edge science (14 sites, up to 10Gbps) in parallel to SINET 2007 SINET3 (34 prefectures, 1Gbps to 40Gbps) as integrated backbone with new services 2011 SINET4 (47 prefectures, 2.4Gbps to 40Gbps) with highly-available nodes and lines 2016 SINET5 (47 prefectures, 100Gbps or more) with 100Gbps International lines and mobile capability

• Expansion of coverage area • 100Gbps line for every prefecture • Placement of SINET nodes at DCs • 100Gbps international lines • Line speed of 2.4Gbps to 40Gbps • Mobile capability Europe : Newly covered prefecture

USA and Europe USA Asia Singapore

 SINET5, which covers all the prefectures with 100-Gbps lines, started its operation in April 2016. The international lines were upgraded to 100Gbps in early 2019.  It is used by 917 universities and research institutions, and by more than 3 million users.

Inter-Univ. National Municipal Private Junior Colleges of Labs and Research Universities Universities Universities Colleges Technology Others Total Institutes

Number of 86 83 399 79 56 16 198 917 Sapporo Organizations (100%) (90%) (66%) (25%) (98%) (100%)

(As of August 2019) *

: SINET node Amsterdam Lab. C : Domestic line (100Gbps or more) Univ. B : International line (100Gbps) Lab. D : Access line of user organization Univ. A

Fukuoka * Osaka

New York Tokyo Los Angeles

 SINET had two direct 10-Gbps lines to London for low latency and high-performance in 2016. SINET also had a 100-Gbps line to Los Angeles in 2016.

SINET4 SINET5 (2016 ~)

: SINET 100-Gbps line : SINET 10-Gbps line

 Japan-Europe line, Japan-New York line (via Los Angeles), and Japan-Singapore line were upgraded to 100 Gbps in February or March 2019.  NII is seeking resilient circuit architecture in collaboration with other NRENs.

SURFnet NORDUnet CANARIE Amsterdam In February 2019 CalREN ESnet

GÉANT Pacific Internet2 MAN Wave LAN Los Angeles New York Asi@Connect Tokyo

Singapore RedCLARA

AARNet REUNA : SINET 100-Gbps line : Other 100-Gbps line Until March 2022

 100Gbps ring connection around the globe is expected to supercharge EU-Japanese science collaborations.

https://www.geant.org/News_and_Events/Pages/100Gbps-ring-connection-around-the-globe-supercharges.aspx

SURFnet SURFnet MAN New York MAN New York London LAN Amsterdam LAN CANARIE CANARIE GÉANT NORDUnet GÉANT NORDUnet ESnet ESnet Pacific CalREN Pacific CalREN Internet2 Wave Internet2 Wave Los Angeles Los Angeles

Asi@Connect Asi@Connect

Tokyo Tokyo

Singapore : 100-Gbps line Singapore : New 100-Gbps line : 10-Gbps line : Replaced 100-Gbps line

AARnet AARnet

Until February 2019 From March 2019

Advanced North Atlantic Collaboration (June 2019)

SURFnet New York MAN : SINET 100-Gbps line Amsterdam LAN : Other 100-Gbps line CANARIE GÉANT NORDUnet ESnet Pacific CalREN Internet2 Wave Los Angeles

Asia Pacific Ring Collaboration （December 2017） Asia-pacific Europe Ring Collaboration (July 2019） Asi@Connect Tokyo

Singapore

AARnet

Belle II (Several 10Gbps) HPC Data Backup (over 90Gbps)

Tsukuba - LA Kobe - Kashiwa

USA Tsukuba Kobe Kashiwa

8K Video Transmission (25Gbps~50Gbps) 231Gbps-Speed File Transfer In November 2017 NII Server Okinawa - Tokyo Sapporo

• Experiment using special protocol (MMCFTP) at TNC19@Estonia in June 2019. • Attained 321Gbps data transfer rate between Japan and Estonia over the following four 100-Gbps lines. • Tokyo – Amsterdam – Estonia • Tokyo – Seattle – Chicago – Montreal – Amsterdam – Estonia • Tokyo – Los Angels – New York – Amsterdam – Estonia • Tokyo – Singapore – London – Estonia

 Secure and high-performance VPN services have been growing, and the number of VPNs exceeds 2,700.

Collaborative Research Environment Use of Cloud Computing

Large Experimental Cloud Data Centers Facilities Direct Connect Research Data

L2VPN L2VPN Univ. E L2VPN/L3VPN Univ. A

Lab. B Univ. Lab. Lab. D Univ. C

Multiple Campus Bandwidth Guarantee e.g. International Olympiad in Informatics 2018 : Logical Router (L2VPN) : Logical Router (Internet) L2VPN with 2Gbps Tokyo Tsukuba Internet Virtual Campus LAN/ VPLS

Communication quality was very Students important ! Campus 1 Campus 2 Campus 3 Server (AWS) Olympiad Site © 2019 National Institute of Informatics 11 Slide provided by S. Urushidani @ NII Usage Examples (1)

1) High-Energy Physics 2) High-Performance Computing

3) Nuclear Fusion Science 4) Seismology

5) Astronomy 6) Geodesy

7) ALMA Telescope 8) Space Observation

Asteroid Explorer

9) Medical Image Data Analysis 10) Remote Diagnosis by 8K Video

Kyoto University The University of Collecting image Hospital Tokyo Hospital data by L2VPN and analyzing 8K Monitor 8K them by AI 8K monitor Camera Encoder L2VPN Decoder

Microscope Microscope operation & Pathologist Specimens camera focus

11) Remote Medical Education 12) Medical Information Backup

46 national univs’ hospitals L2VPN

• Traffic amount between Tokyo and Osaka has been increasing due to encouraged traffic by upgraded internarial lines as well as ever-increasing traffic from existing applications. • We plan to introduce a new 400Gbps optical link between Tokyo and Osaka.

Being Operational in December 2019

400Gbps Optical Link * between Tokyo and Osaka Europe

: SINET Node : 400Gbps : 100Gbps (Domestic) : 100Gbps (International)

United States

Japan and California in the same scale

Corning

Colusa

Sacramento

Emeryville

San Francisco Merced Palo Alto Sunnyvale Fresno Soledad

Bakersfield San Luis Obispo Riverside Los Angeles Palm Desert Tustin

San Diego El Centro Yuma Tijuana

： nodes ： 100 Gbps connections

Source: https://cenic.org/network/network-overview Data Exploitation Platform 16 Slide provided by S. Urushidani @ NII Mobile Platform in SINET5

 NII started to introduce mobile capability into SINET in order to support IoT research and mobile-oriented applications toward 5G era in December 2018.  Mobile virtual network created for SINET over commercial mobile network is directly combined with SINET VPN plane through gateways for secure communication.

Cloud Cooperative Vendors University Compute Compute Compute Various analysis platforms are flexibly available. Storage Storage Storage Data Data Data Data Data Data

L2VPN VPN L2VPN

Gateway Mobile Network Mobile communication (Softbank, KDDI, DoCoMo) Fixed promotes expansion of research activities. Mobile Virtual Network

Local Community Wide Area Restricted Area University Enterprise Local public entity Nature Protection Shore Mobile Area

 31 cloud datacenters (26 service providers) are directly-connected to SINET in order to provide high-performance and secure cloud services for more than 190 universities.

Cloud Datacenters

Direct Connection

Selectable

Universities

NII plans to launch SINET6 in April 2022 and is now exploring its concept.

Today

2016 2017 2018 2019 2020 2021 2022 2023

SINET5 (6 years)

▲Full-scale Operation Start ▲Domestic 100Gbps Backbone ▲Domestic 400Gbps Line

▲100-Gbps US Line ▲100-Gbps US line (LA and NY) ▲20-Gbps Euro Line ▲100-Gbps Euro Line Migration ▲10-Gbps Asia ▲100-Gbps Asia Line

▲Mobile Capability▲5G (Planned)

Basic Detailed Concept deployment Design Design SINET6

© 2019 National Institute of Informatics 19 Data Exploitation Platform (tentative name) project 20 • Will provide a rapid PoC environment for R&D data exploitation activities including industry-academia collaboration projects. • Shared platform for various data exploitation activities • Combine SINET and high performance computing and storage infrastructure • Users can use wide bandwidth low latency “slices” • Wide-area virtual infrastructure isolated from “the internet” • Connect edge devices with high performance computing and storage infrastructure and supports real-time data processing • Will host: • Various data exploiting activities, especially in SMEs, local governments and agriculture / fishing • Key to solve the regional disparity problems • Will provide matching function of: • Those who want to analyze their own data, • Various data and their owners • Researchers who have skills/tools to analyze data, • The data exploitation platform infrastructure

Data Exploitation Platform To realize Knowledge Intensive Society: Platform for Data Exploitation

Input：Events Platform Output：Knowledge

Knowledge Fishing Power Grid Distillation: Data New Exploitation Agriculture Logistics Materials Platform

震度

City Telemedicine Evacuation Planning Mathematical Model Past Events Background Data （Topograpy, Population Etc. ）

Data exploitation is the key to realize highly productive knowledge intensive society. Data exploitation opportunities should be provided to everyone / every region. → SINET is a great infrastructure to provide such opportunities to all over the country.

Data Exploitation Platform Core Pillars of the Data Exploitation platform 22

Dynamic provisioning of real-time data collection / storage / analysis infrastructure leveraged by SINET On-demand construction of virtual infrastructure (slices) including remote censors, storage and compute resources Using the provided virtual infrastructure, users can process data and get analysis result in real-time. Fusion of Data Science and Computational Science First-rate computing infrastructure for both data science and computing science enables highly accurate data analysis

Promote combined use of various kinds of data and knowledge by matching and consulting Build a cross academia / industry community of data owners, data analysis specialists and those who want to utilize data so as to accelerate data exploitation in various area

Data Exploitation Platform On-demand platform 23 Geographically distributed IaaS including network

Disaster protection Medicine

Shared infrastructure Network / Storage / Computers

SINET5

SINET mobile Data

Agriculture / Fishing Smart Cities

Energy / Power Grid

Data Exploitation Platform SINET mobile infrastructure 24

• SINET mobile infrastructure to support data collection and analysis • IoT devices can be connected to SINET L2VPN • Secure: a virtual SINET network is constructed in mobile networks. Closed application designated network isolated from the internet connects edge devices and compute & storage infrastructure

Cloud Cooperative Vendors Data Exploitation Platform Compute Compute Compute Various analysis platforms are flexibly available. Storage Storage Storage Data Data Data Data Data Data

L2VPN VPN L2VPN

Gateway Mobile Network Mobile communication (Softbank, KDDI, DoCoMo) Fixed promotes expansion of research activities. Mobile Virtual Network

Local Community Wide Area Restricted Area University Enterprise Local public entity Nature Protection Shore Mobile Area Data Exploitation Platform Edited based on a slide provided by NII Overview of the Data Exploitation Platform 25

Data

SMEs Local Governments Technologies: Agriculture, Fishing Data Alalysis Matching Security Hub Data Coordinations Etc.

Rapid PoC environment After PoC, services are moved AIST to commercial infrastructure ABCI SINET mobile Clouds Storage

Clouds Storage SINET5 HPC Stream Servers Servers HPC Stream Data Exploitation Platform Servers Servers Infrastructure Commercial Infrastructure Data Exploitation Platform Deployment and operation plan 26

• Small scale testbed will be available in FY2019 • First deployment system will be operational by the end of FY2020 • Computer & storage will be installed in Kashiwa II campus of the University of Tokyo • AIST’s ABCI (AI bridging infrastructure) will also participate in the program • Japan’s fastest supercomputer for AI • The platform will be jointly operated by : • Universities in Japan • NII (National Institute of Informatics) • AIST (National Institute of Advanced Industrial Science and Technology)

Data Exploitation Platform Infrastructure of the Data Exploitation Platform 27

SINET mobile The Internet

Public Cloud Virtual Platform (slice) Shared Object Storage

AIST ABCI

Servers Cold Storage

Secure / High UTokyo Supercomputers speed Storage Data Exploitation Platform 2019/9/6 Slide provided by R. Takano @ AIST ABCI: The World’s First Large-Scale Open AI Infrastructure

28 Slide provided by R. Takano @ AIST ABCI is a powerful infrastructure to compete with hyper giants for AI R&D

29 Two-level Isolation for Security Sensitive Applications 30

• The platform will host security sensitive applications such as medical data, personal data • Those applications require higher security level. Cloud-like environment may not be sufficient • The platform will be used by many different kind of users, and not all of them can be trusted. • Number of IoT devices may be large, and hard to manage. Some IoT devices do not have enough computing power for secure encryption. • In addition to software based security (authentication, authorization and encryption), the data exploitation platform will provide isolation by the network (VLAN) • VLAN are controlled by the resource manager (port-based VLAN) • Even if the hypervisor is compromised, a user on a slice cannot access data of another slice

Data Exploitation Platform Summary 31

• The Data Exploitation Platform is “more for every day applications than big sciences” • Data Exploitation for everyone: SMEs, local governments, agricultures, fishing etc. • Provide PoC environment for commercial applications • A real-time data processing environment. • It is a geographically distributed IaaS, directly connectable to edge devices. • It can work as a streaming data gathering infrastructure for super computers such as ABCI • Leveraging the SINET mobile infrastructure • Supports Two-level isolation (software level and network level) for security sensitive applications • In combination with the SINET VPN (VLAN) service, can support

Data Exploitation Platform