Esnet: Advanced NETWORKING for SCIENCE
Total Page:16
File Type:pdf, Size:1020Kb
ENERGY SCIENCES NETWORK ESnet: Advanced NETWORKING for SCIENCE Researchers around the world using advanced computing for scientific discovery are connected via the DOE-operated Energy Sciences Network (ESnet). By providing a reliable, high-performance communications infrastructure, ESnet facilitates the large-scale, collaborative science endeavors fundamental to Office of Science missions. Energy Sciences Network tive science. These include: sharing of massive In many ways, the dramatic achievements of 21st amounts of data, supporting thousands of collab- century scientific discovery—often involving orators worldwide, distributed data processing enormous data handling and remote collabora- and data management, distributed simulation, tion requirements—have been made possible by visualization, and computational steering, and accompanying accomplishments in high-per- collaboration with the U.S. and international formance networking. As increasingly advanced research and education (R&E) communities. supercomputers and experimental research facil- To ensure that ESnet continues to meet the ities have provided researchers with powerful requirements of the major science disciplines a tools with unprecedented capabilities, advance- new approach and a new architecture are being ments in networks connecting scientists to these developed. This new architecture includes ele- tools have made these research facilities available ments supporting multiple, high-speed national to broader communities and helped build greater backbones with different characteristics—redun- collaboration within these communities. The dancy, quality of service, and circuit oriented DOE Office of Science (SC) operates the Energy services—all the while allowing interoperation of Sciences Network (ESnet). Established in 1985, these elements with the other major national and ESnet currently connects tens of thousands of international networks supporting science. researchers at 27 major DOE research facilities to universities and other research institutions in the Evolving Science Environments To ensure that ESnet U.S. and around the world (figure 1; sidebar Large-scale collaborative science—big facilities, continues to meet the “What is ESnet?,” p50). massive amounts of data, and thousands of col- requirements of the major ESnet’s mission is to provide an interoperable, laborators—is a key element of the DOE SC scien- science disciplines a new effective, reliable, high-performance network tific mission. The science community that approach and a new communications infrastructure, along with participates in DOE’s large collaborations and architecture are being selected leading-edge Grid-related and collabora- facilities is almost equally divided between labo- developed. tion services in support of large-scale, collabora- ratories and universities, and also has a significant 48 S CIDAC REVIEW S UMMER 2007 WWW. SCIDACREVIEW. ORG A. T OVEY ESnet–Connecting DOE Labs to the World of Science Canada Canada Asia-Pacific (CANARIE) Russia and China (CANARIE) (Sinet/Singaren Asia-Pacific (GLORIAD) CERN TWAREN) (ASCC/Kreonet2/Sinet/ (USLHCNet) TWAREN/Singaren) Seattle Asia-Pacific Europe (Sinet) (GÉANT) Boise Australia Boston (AARnet) Chicago Europe (GÉANT) Cleveland New York Sunnyvale Asia-Pacific Denver (ASCC/Sinet/ Kansas City Washington WIDE) ORNL San Diego Los Angeles Albuquerque Australia Atlanta (AARnet) South America (AMPATH) Jacksonville Latin America Houston (CUDI/CLARA) Production IP Core (10 Gbps) MANs (20-30 Gbps) SDN Hubs Future Hubs or Backbone Loops for Site Access Science Data Network (SDN) IP Core Core (20-30-50 Gbps) Major International Connections Figure 1. ESnet connects the DOE laboratories to collaborators at research and education institutions around the world. international component. This component con- to analysis, simulation, and storage facilities. sists of very large international facilities and inter- High network reliability is required for intercon- national teams of collaborators participating in necting components of distributed large-scale U.S.-based experiments, often requiring the move- science computing and data systems and to sup- ment of massive amounts of data between the lab- port various modes of remote instrument oper- oratories and these international facilities and ation. New network services are needed to collaborators. Distributed computing and storage provide bandwidth guarantees for data transfer systems for data analysis, simulations, and instru- deadlines, remote data analysis, real-time inter- ment operation are becoming common; for data action with instruments, and coupled computa- analysis in particular, Grid-style distributed sys- tional simulations. tems predominate. This science environment is very different from Requirements from Data, Instruments, Facilities, that of a few years ago and places substantial new and Science Practice demands on the network. High-speed, highly There are some 20 major instruments and facili- reliable connectivity between laboratories and ties currently operated or being built by the DOE U.S. and international R&E institutions (sidebar SC, in addition to the Large Hadron Collider “U. S. and International Science,” p51) is required (LHC) at CERN (figure 2, p50) and the ITER fusion to support the inherently collaborative, global research project in France (figure 4, p51). To date, nature of large-scale science. Increased capac- ESnet has characterized 14 of these for their ity is needed to accommodate a large and steadily future networking requirements. DOE facilities— increasing amount of data that must traverse the such as the Relativistic Heavy Ion Collider (RHIC) network to get from instruments to scientists and at BNL, the Spallation Neutron Source at ORNL, S CIDAC REVIEW S UMMER 2007 WWW. SCIDACREVIEW. ORG 49 ENERGY SCIENCES NETWORK What Is ESnet? As the primary DOE network providing single largest supporter of basic research in laboratory-to-laboratory and university-to- production Internet service to almost all of the physical sciences in the United States, the university link. The keys to ensuring this are DOE’s laboratories and other sites, ESnet DOE SC supports 25,500 Ph.D.s, postdoctoral engineering, operations, and constant serves an estimated 50,000 to100,000 DOE and graduate students, and 21,500 users of monitoring. ESnet has worked with the users, as well as more than 18,000 non-DOE facilities, half of whom are at universities. Internet2 and the international R&E researchers from universities, other The goal is to provide the same level of community to establish a suite of monitoring government agencies, and private industry, service and connectivity from a DOE tools that can be used to provide a full mesh that use SC facilities. The connections with laboratory to U.S. and European R&E of paths that continuously check all major university researchers are critical. As the institutions as the connectivity between any interconnection points. CERN Figure 2. An aerial view of CERN. The Large Hadron Collider (LHC) ring is 27 km in circumference and provides two counter-rotating, 7 TeV proton beams to collide in the middle of the detectors. the National Energy Research Scientific Comput- geographic footprint are determined by the loca- ing (NERSC) Center at LBNL, and the Leadership tion of the instruments and facilities, and the Computing Centers at ORNL and ANL, as well as locations of the associated collaborative commu- the LHC at CERN—are typical of the hardware nity, including remote and/or distributed com- infrastructure for science in the 21st century. puting and storage used in the analysis systems. These facilities generate four types of network These locations also establish requirements for requirements: bandwidth, connectivity and geo- connectivity to the network infrastructure that graphic footprint, reliability, and network serv- supports the collaborators. Reliability require- ices. Bandwidth needs are determined by the ments are driven by how closely coupled the quantity of data produced and the need to move facility is with remote resources. For example, the data for remote analysis. Connectivity and off-line data analysis—where an experiment 50 S CIDAC REVIEW S UMMER 2007 WWW. SCIDACREVIEW. ORG U.S. and International Science A. T ESnet is a network in the OVEY traditional sense of the word, connecting end user sites to various other networks. A large fraction of all the national data traffic supporting U.S. science is carried by three networks—ESnet, Internet2, and National Lambda Rail. These three entities represent the architectural scope of science-oriented networks. Internet2 is a backbone network connecting U.S. regional networks to each other and to international networks. National Lambda Rail is a collection of light paths, or lambda channels, that are used to construct specialized R&E networks. Figure 3. The large-scale data flows in ESnet reflect the scope of DOE SC collaborations. ESnet’s top 100 data flows generate 50% of all ESnet traffic (ESnet handles about 3 x109 flows/month) Of the top 100 flows, 91 are from the DOE Labs (not shown) to other R&E institutions (shown on the map). runs and generates data, and the data is analyzed ITER after the fact—may be tolerant of some level of network outages. On the other hand, when remote operation or analysis must occur within the operating cycle time of an experiment, or when other critical components depend on the connection, then very little network downtime is acceptable. The reliability issue is critical