The Energy Sciences Network: Overview, Update, Impact

ASCAC Meeting Gregory Bell, Ph.D. American Geophysical Union Director, Energy Sciences Network (ESnet) Washington, DC Director, Scientific Networking Division March 24, 2015 Lawrence Berkeley National Laboratory Overview

Update

Impact

2 Overview

Update

Impact

3 courtesy LACMA

4 This is not an ISP. It’s a DOE user facility engineered to overcome the constraints of geography.

We do this by offering unique capabilities, and optimizing the facility for data acquisition, data placement, data sharing, data mobility. It’s a DOE user facility engineered to overcome the constraints of geography.

ESnet’s superficial resemblance to an international ISP is deceptive, and arguably a risk. Our vision:

Scientific progress will be completely unconstrained by the

physical location of instruments, people, computational resources, or data.

8 This is not a new way of describing ESnet.

1. “What we can do on LANs today is indicative of what we wish to be able to do on wide area networks .”

2. “Just as we expect a computer to perform as if we are the only user , we expect the network to give that same appearance.” The basic facts (new or notable):

High-speed international networking facility, optimized for DOE science missions: •connecting 50 labs, plants and facilities with >150 networks, universities, research partners globally •340Gbps transatlantic extension in production (Dec 2014) •university connections to better serve LHC science

•$35M in FY15, 42FTE •older than commercial Internet, growing ~twice as fast •the DOE user facility that serves all others $62M ARRA grant funded 100G upgrade in 2011: •fiber assets + access to spectrum, shared with Internet2 •new era of optical networking, abundant capacity •world’s first 100G network at continental scale Culture of urgency: •several recent awards •80% engineers, highly motivated

general Internet. general

different goals than than goals different ESnet is designed for for designed is ESnet Elephant flows require almost lossless networks.

Local (LAN)

International Metro Area Regional

Continental

Measured (TCP Reno) Measured (HTCP) Theoretical (TCP Reno) Measured (no loss) . See Eli Dart, Lauren Rotman, Brian Tierney, Mary Hester, and Jason Zurawski. The Science DMZ: A Network Design Pattern for Data-Intensive 12 Science. In Proceedings of the IEEE/ACM Annual SuperComputing Conference (SC13), Denver CO, 2013.

, sensors and instruments.

16 80% of ESnet traffic originates or terminates outside the DOE complex.

17 In a nutshell:

• Data intensive science inevitably drives network intensity. • DOE traffic continues to grow exponentially. • Data ‘point sources’ are becoming more numerous, less expensive. • DOE data flows typically include universities, global collaborations. – only 20% of ESnet data flows are DOE DOE

18 Now, a few brief examples of ESnet’s role in DOE workflows, starting with LHC science.

19 Evolution of LHC data model:

In chronological order 1. Copy as much data as is feasible to analysis centers worldwide, with hierarchical This evolution distribution scheme (‘Monarc’ model, implies growing deterministic flows). faith in networks, 2. Relax the hierarchy, and rely on optimistic growing caching. opportunity for Software 3. Use ‘federated data stores’ to fetch Defined portions of relevant data sets from remote Networking. storage (anywhere), just before they’re needed.

3/24/2015 To support this evolving data model, research networks have built a vast global overlay.

LHC Open Network Environment (LHCONE ): •dedicated and isolated network overlay for LHC experiments •gives consistent, high-performance access for LHC computing centers – extensive use of virtual circuits •30 networks (with ESnet as core participant), dozens of universities

•an international highway system optimized for LHC flows – custom global instrument, but also a collaboration

21 22 This architecture (instruments and computational resources coupled by networks) now spreading outside HEP: ‘super-facilities.’

Experimental facilities are being transformed by new detectors, advanced mathematics, robotics, automation, advanced networks.

23 Super-facility example #1:

Researchers from Berkeley Lab and SLAC conducted protein crystallography experiments at LCLS to investigate photoexcited states of PSII, with near-real-time computational analysis at NERSC.

“Taking snapshots of photosynthetic water oxidation using femtosecond X-ray diffraction and spectroscopy,” Nature Communications 5, 4371 (9 July 2014) 24 Data flow from single LCLS detector tripled network utilization for major HPC center.

25 Super-facility example #2:

Real-time analysis of ‘slot-die’ technique for printing organic photovoltaics, using ALS + NERSC (SPOT Suite for reduction, remeshing, analysis) + OLCF (HipGISAXS running on Titan w/ 8000 GPUs).

http://www.es.net/news-and-publications/esnet-news/2015/esnet-paves-way-for-hpc- superfacility-real-time-beamline-experiments/ Results presented at March 2015 meeting of American Physical Society by Alex Hexemer. Additional DOE contributions: GLOBUS (ANL), CAMERA (Berkeley Lab) 26 Super-facility-on-demand demo at NSF GENI conference tomorrow .

ESnet, Internet2 ESnet

SPADE instance - server at Argonne

ExoGENI SPADE VM @ ExoGENI SPADE VM @ Starlight, Chicago Oakland, California

• fictional - but realistic - workflow Data from ALS • dedicated systems for data transfer and beamline network circuits created programmatically • future vision: application declares intention for super-facility, network responds Compute Cluster • “Science DMZ as a service” NERSC, LBL

http://portal.nersc.gov/project/als/sc14/ Overview

Update

Impact

28 340Gbps extension. Why?

25% of ESnet traffic

Immediate driver: more capacity for LHC Run 2. But the extension supports all DOE missions , reducing barriers to European collaborators / instruments. How? [short detour]

30 3/24/2015 Internet’s physical substrate, optical fiber…

courtesy thefoa.org

courtesy http://shulihallak.com/

31 …also criss-crosses the ocean floors.

courtesy Telegeography Worldwide submarine cable system, 1901

3/24/201533 courtesy wikipedia 3/24/201534 courtesy Tyco Reliance 3/24/201535 courtesy http://shulihallak.com/

courtesy Rod Wilson, Ciena 36 3/24/2015 courtesy http://www.tstt.co.tt/

courtesy Rod Wilson, Ciena 37 3/24/2015

courtesy Rod Wilson, Ciena 38 3/24/2015 39 3/24/2015 ESnet operations: focus on simplicity, automation, core mission.

Jan 2014 Jan 2015 Change Monitored hosts 433 394 -10% Auto-patched hosts 111 187 44% Auto-configured hosts 111 120 8% Physical hosts 284 203 -40%

Virtual hosts 125 180 31% Hypervisors 11 11 None OS Versions 24 14 -71%

ESnet’s video collaboration and X.509 services for DOE were highly distinctive at one time, but no longer. We have transitioned them to commercial providers, in the spirit of focusing on our core mission.

40 3/24/2015 Our portal (my.es.net) now greatly enhanced.

41 Our portal (my.es.net) now greatly enhanced.

42 Honors for ESnet5 deployment:

FierceGovernment chose the ESnet5 Deployment Team as a recipient of the annual Fierce 15 award, in recognition of “federal employees and teams who have done particularly innovative things.”

Information Week named ESnet as one of the “top 15 innovators”, among government entities at every layer: federal, state and local. It was the second time in four years ESnet received this award.

3/24/2015 More recent honors and awards:

ESnet’s OSCARS software was honored with a 2013 R&D100 Award, and more recently with a 2014 Secretary’s Honor Award from DOE.

ESnet’s Network Research Testbed received CENIC’s 2015 Innovations in Networking award: “ESnet inspires us to do more for our communities, and to do better at what we do.” (CENIC CEO Louis Fox)

Inder Monga, ESnet CTO and Division Deputy, named chair of Research Associate Council for Open Networking Foundation (most important membership org promoting SDN). 3/24/2015 Staffing remains lean.

• Brazil (422) • Greece (68) • UK (180) • France (66) • Czech Republic (142) • Italy (61) • Netherlands (138) • Germany (54)

• Croatia (112) • Ireland (52) • Internet2 (~100) • Slovenia (51) • Hungary (94) • Belgium (50) • Australia (80) • Portugal (47) • Norway (78) • ESnet (42) • Switzerland (76)

Caveats: varying service and business models make comparisons difficult, but the large-scale pattern is instructive. (Headcount numbers interpolated from bar graph on page 80 of the most recent GÉANT Association Compendium , or described elsewhere in that report.)

45 We are working to increase diversity in the field of network engineering.

• Sponsored two early-career women from DOE/SC labs to attend an important annual conference for network engineers (Internet2/ESnet Technology Exchange, Fall 2014). • Co-organizing diversity track and panel at same conference. • Participating in the steering committee of Internet2’s gender diversity initiative for women in IT (across US university space). • Grace Hopper conference, for career development as well as recruitment. • In partnership with FRGP and others: proposal to NSF to fund SCinet participation for young female engineers.

46 Overview

Update

Impact

47 Our vision and strategic goals guide impacts.

48 Our vision and strategic goals guide impacts.

49 Reminder: 80% of ESnet traffic originates or terminates outside the DOE complex.

ESnet’s success is not sufficient, because networks share fate . SC invests nearly $1B/year in university research, and campus networks matter to DOE.

50 Introducing Science DMZ , a network design pattern for data-intensive science (origin: ESnet & NERSC). Three components, all required 1. Friction-free network path: • highly-capable network devices (wire-speed, deep queues) • at or near site perimeter, with option for virtual circuit connections • security policies tailored for science

2. Dedicated, Data Transfer Nodes (DTNs): • hardware, operating system, libraries optimized for data transfer • appropriate tools such as Globus and GridFTP 3. Performance measurement / test nodes: • perfSONAR • testing, assurance, forensics

Much more information: http://fasterdata.es.net/science-dmz/ Science DMZ now recognized as best practice.

NSF is investing $60M to promote adoption by US universities (among other CI goals). Fourth funding round underway.

>120 universities in the US have deployed this DOE architecture.

IN addition: USDA, NIH – with NASA, NOAA investigating.

Australian, Canadian universities following suit.

What’s next? Evolution of Science DMZ as a regional cyberinfrastructure platform.

Pacific Research Platform initiative , lead by Larry Smarr (Calit2/UCSD) •first large-scale effort to coordinate and integrate Science DMZs •participation by all major California R&E institutions, CENIC, ESnet •announced March 6: http://cenic.org/news/item/high-performance-big- science-pacific-research-platform-debuts-at-cenic

53 3/24/2015

Back to impacts: ‘fasterdata’ knowledgebase.

54 55 56 Selected impacts in research and innovation.

57 Ongoing impacts (universities, labs, networks):

•OSCARS for virtual circuits with service guarantees – adopted by >40 networks & universities – production software for almost 10 years – initially funded by ASCR NGNS, LBNL LDRD – Secretary’s Honor Award, April 7

•perfSONAR for network measurement (diagnostics, assurance, forensics) – integral to ScienceDMZ – ESnet, Indiana, Internet2 and GÉANT major collaborators – 1200 deployments worldwide

•Science DMZ – network design pattern for data intensive science – NSF support with >120 campus deployments in the US alone

58 3/24/2015 Ongoing impacts (industry):

•Frequently among the first customers for new technology (100G terrestrial, 100G trans-Atlantic, 400G terrestrial, fastest available NICs, etc). •Deep collaboration with Infinera to demonstrate first SDN for optical transport (October 2012) resulted in new product, with major carrier announcement March 2015. •Ongoing collaboration with different optical vendor will result in significant reduction in product development cycle (to be announced in May). •Frequent engagement with relevant startups, for instance Corsa Technology: – first public customer for disruptive ‘white box’ networking gear – worked closely to develop packet-processing pipeline useful to DOE science missions – Corsa closed $16M series B funding this week

59 3/24/2015 Two sources of future impact:

• ESnet testbed – available to researchers and industry since 2011, now upgraded with low-cost ‘white box’ SDN equipment (Corsa Technology), on an international footprint. Intended uses: –R&D for SDN, NFV, NDN, systems, protocols, security –ESnet6 prototypes –federation with other testbeds –platform for collaborations and demonstrations • SDN innovation, including ESnet Operating System –platform for science apps to express intent , simultaneously, across >1 network –focus on requirements not being met by industry, open-source projects –supported by Berkeley Lab LDRD –multipoint VPN service demonstration in May

60 3/24/2015 ESnet SDN impacts, in a nutshell:

2006 2012

2012 2013

2013 2014 Spotlight on one project: global SDN BGP peering

Inter-operability with routing standards protocol (BGP) using SDN techniques. Physically distinct control plane (using off the rack Unix server) and data plane (using OpenFlow switch) functions. Demonstrated ~40% FIB compression (13,215 -> 7,577 routes) Vandervecken software by Dynamic layer 2 setup (using OSCARS with NSI) for transport of layer 3 BGP protocol messages.

Goals •Explore feasilbility of ‘white box’ designs for core Internet routing. •Backward compatibility with standard protocols (especially BGP) without compromising performance, scalability, operability. •Simplifly control and management through logical centralization. In conclusion, a reminder about our vision:

Scientific progress will be completely unconstrained by the physical location of instruments, people, computational resources, or data.

ESnet is an instrument for discovery, and increasingly the glue for DOE super-facilities.

The new European extension supports LHC Run 2, plus all DOE missions.

ESnet innovation is impacting scientists, researchers, universities, industry – around the world.

64 Thank you.

[email protected]

Additional Slides

66 Broad questions driving ESnet research and development activities:

1. How can we continue to scale up and handle exponential traffic growth with linear budgets?

1. Can we create useful abstractions to enable productive interaction between science applications and the network?

1. Can we transform ESnet into a programmable platform that can be operationally supported ?

, at at ,

, Denver CO, 2013. CO, Denver , Result of 99.9954% water transferred, “line rate.” at Result 100% ofdata100% transferred, slowly <<5% optimal speed. Through careful careful Through we engineering, can minimize packet loss. sign Pattern for Data-Intensive Data-Intensive for Pattern sign

onference (SC13) onference

Assumptions: 10Gbps TCP flow, 80ms RTT. RTT. 80ms flow, TCP 10Gbps Assumptions: r, and Jason Zurawski. The Science DMZ: A Network De Network A DMZ: Science The Zurawski. Jason r, and

C SuperComputing Annual IEEE/ACM the of Proceedings

Science. In In Science.

Network ‘pipe’that Network at drops packets of.0046%. rate Physical pipe that pipe Physical at rate water leaks .0046% by of volume.

See Eli Dart, Lauren Rotman, Brian Tierney, Mary Heste Mary Tierney, Brian Rotman, Lauren Dart, See Eli

speed of light determined by determined Elephant Flows Place Great Demands on Networks on Demands Great Flows Place Elephant essentially fixed essentially

More detailed network diagram.

0 E

G a T

D -

N 8

1 V

- K

perfSONAR

rSonarenabled erfS P N

9 8 Site w/ PerfSonar enabled PerfSonar Site w/ SANFORD ONENET LBN L perfSONAR HOUS (ASGC/KREONET2/TANET2) ASIA-PACIFIC E SNET -HOU SITE S-KA NS-1 00GE- 07831 perfSONAR KANS 1 00 GI1 G 00

AMES

E K

S C

N I

3 T

0 E T

T M 0

- /

H O

T O

O F A

U A RUSSIA & CHINA

IP E 0

S 0 C

- 3

N 8 S H

A 3

0 /

S 0

2 H /

-1 5 3

0 (GLORIAD)

0 0 G /

E E 0 eigCnnections Con eering P 10G KCP

- KCP S

& n nternational R&E and I 0

7 N

I 7 S

9 E

5 U

T -

- 0

C 0

I C ULTRA LIGHT

USLHCNET -

S-0011ISU-

K A

MERIT

1 0

perfSONAR

0 G

INTERNET2 E

FNAL - 1G P eering Connections eering 1G P

R&E andInternational R&E

7 ANL

2 1

1 1

E CANADA 0 0

- 1

(CANARIE)

G 0

0 1 0

8 G

E 0 G

2 0 E

- 1 G 7

0 E

0 - 2

8 E - 0

G - 0

2 0 7

NASH E

6 - 6 7 8 7 5 9 0 1 3 79 9 5 3 5 2 1

ES NE T-C 10 HIC

Capacity waves 44 Capacity 0G -N E- AS

Optical Layer 0 H 780 -

7 STAR CHIC perfSONAR

N A

A S

S H

H 100GE-07617 -

O - O R

R (OPEN TRANSIT) N

N L

L FRANCE -

ORAU 1 - PNNL

OSTI 1

ORO 0 6 BJC Y12

0 E

G 2

S 6 -

A 4

- N A U 0

100G W U

< 1000km -

T - W

W -

A a

a v T STARLIGHT NWU/STARLIGHT/UCHICAGO/ EQX-CHI

ave ave v e L E

e POPGIG A

- INDIANA- A S

- # ICCNUI-

CICNET-OMNIPOP/DREN/ # - N

1 INTERNET2/MREN/NISN/

2 E

I A

P UIOWA/UWMADISON/ I T

P S

- A

H S

1 H

0 B

ORNL ORNL 0 -

G S

WISCNET

E T A

u’ wavemux’d 0 R

4x10G 4x10G -

7 O

perfSONAR

0 T

perfSONAR 0

H 3

ATLA

4 E

E 0

SE-HCWAH10E06 6 ASH-100GE-0761 ESNET-CHIC-W

T 0

A E 0 E-Link10G E

S T

N - L

ET 0 A

1 -AT 6 L 3 00 A- - GE CH 4 -07 AT 7 809 - SCI-ENET-013114 GE-Link1G SOX perfSONAR

SRS E C2Ln(2Mb) OC12-Link(622M S N E T -A T L A -W A S

S REL H -

EM 1 0 0 G E C-i k(155Mb) OC3-Lin

-0 EQX-ASH

7 1

NNSA 6

7 9 0

DOE 8

0 3 2

8 6

0 0 0 - DOE-

GTN E

N LANL-DC E

LNL/L T

- B

-ik(100Mb) E-Link F E

1 S

JLAB 5

T T 1

H -

3 1

9 T 0

0 A 0

1 R

0 7

- -

perfSONAR 1

0 FORR

0 WASH G

- 1 0 2 0 9 3 1 5 1 5 S-ik4Mb) DS3-Link(45M

9 - 0 0 0 1 H T E 6

3 perfSONAR E S

P P PL N E

SNL-DC 1 T 0 -B 0 O G S

E T

1 - - 1 N 0 0 0 E 7 0 G W 5 G E 9 Y E - 8 0 1 - 7 1

0 9 1 7 PU-HEP 5 8 8 5 T GFDL H 2 & NOX 0 1 -1 0 0 G 0 E G - E 0 -

perfSONAR NEWY 7 1 9 5 5 9 6 6 AOFA 6 perfSONAR WIX Redistribute. For copies contact [email protected] contact For copies Redistribute. Not Do Only. Use Business is DOE Mapfor This GIG OP AP perfSONAR BNL MAX BOST 3ROX/PSC MIT perfSONAR INTERNET2/ DREN/ NASA INTERNET2 / / NYSERNET INTERNET2

AMERICA LATIN (AMPATH/CLARA) ASIA-PACIFIC ASIA-PACIFIC NORDUNET) (BINP/HEPNET) MANLAN EUROPE EUROPE CANADA (GEANT/ (CANARIE) (GEANT) (SINET)

U C 7 - U B B W C O C T Q S - N _ 0 A P 2 1 8 M 4 S _ 4 0 G

E _ 5 2

LOND

E A

N M -

O S

N -

perfSONAR

_ L

E O E

X N

_ GEANT

_ E

S E

N E

E X T

_ _

1 E 4

2 S 6 N

4 perfSONAR AMST CERN CERN-5 GEANT

CER N-2

perfSONAR M S

GEANT G E

- E

_ E

4 2 5 perfSONAR

69 The ‘Default Free’ Zone

70 Segmenting the world of scientific collaboration.

100PB A few large collaborations have internal software and networking organizations 10PB Medium collaboration scale, e.g. HPC codes 1PB Small collaboration scale, e.g. light and neutron sources 100TB

10TB Data Scale Large collaboration scale, e.g. LHC 1TB New projects 100GB targeting 100 beamline users, ESnet’s climate historical focus 10GB community.

Collaboration Scale

71 3/24/2015 ESnet / Infinera / Brocade multi-layer packet- optical SDN demonstration (Oct 2013)

Advanced Reservation System (OSCARS) L0/L1 Topology OTS Config Multi-Layer Multi-Layer Multi-Layer Manager Topology App Path Engine Provisioning

Multi-Layer SDN Controller SDN Control Traffic Optimization Floodlight Layer Engine

OpenFlow & OpenFlow REST/JSON

Host A Host B

OTS Presented at Virtualization SDN World Congress

WDM/ OTN/ (Bad Homburg, Packet Infinera DTN-X Germany)