The Science DMZ: A Network Design Pattern for Data-Intensive Science Eli Dart Lauren Rotman Brian Tierney Energy Sciences Network Energy Sciences Network Energy Sciences Network Lawrence Berkeley National Lawrence Berkeley National Lawrence Berkeley National Laboratory Laboratory Laboratory Berkeley, CA 94720 Berkeley, CA 94720 Berkeley, CA 94720 [email protected] [email protected] [email protected] Mary Hester Jason Zurawski Energy Sciences Network Internet2 Lawrence Berkeley National Office of the CTO Laboratory Washington DC, 20036 Berkeley, CA 94720 [email protected] [email protected] Abstract Categories and Subject Descriptors The ever-increasing scale of scientific data has become a sig- C.2.1 [Computer{Communication Networks]: Network nificant challenge for researchers that rely on networks to Architecture and Design; C.2.3 [Computer{Communication interact with remote computing systems and transfer re- Networks]: Network Operations|network management, net- sults to collaborators worldwide. Despite the availability work monitoring; C.2.5 [Computer{Communication Net- of high-capacity connections, scientists struggle with inade- works]: Local and Wide-Area Networks|internet quate cyberinfrastructure that cripples data transfer perfor- mance, and impedes scientific progress. The Science DMZ paradigm comprises a proven set of network design patterns General Terms that collectively address these problems for scientists. We Performance, Reliability, Design, Measurement explain the Science DMZ model, including network architec- ture, system configuration, cybersecurity, and performance 1. INTRODUCTION tools, that creates an optimized network environment for science. We describe use cases from universities, supercom- A design pattern is a solution that can be applied to a puting centers and research laboratories, highlighting the general class of problems. This definition, originating in the effectiveness of the Science DMZ model in diverse opera- field of architecture [1,2], has been adopted in computer sci- tional settings. In all, the Science DMZ model is a solid ence, where the idea has been used in software designs [6] platform that supports any science workflow, and flexibly and in our case network designs. The network design pat- accommodates emerging network technologies. As a result, terns we discuss are focused on high end-to-end network the Science DMZ vastly improves collaboration, accelerating performance for data-intensive science applications. These scientific discovery. patterns focus on optimizing the network interactions be- tween wide area networks, campus networks, and computing systems. The Science DMZ model, as a design pattern, can be adapted to solve performance problems on any existing network. Of these performance problems, packet loss has proven to be the most detrimental as it causes an observable This manuscript has been authored by an author at Lawrence Berkeley Na- and dramatic decrease in data throughput for most applica- tional Laboratory under Contract No. DE-AC02-05CH11231 with the U.S. tions. Packet loss can be caused by many factors including: Department of Energy. The U.S. Government retains, and the publisher, by firewalls that cannot effectively process science traffic flows; accepting the article for publication, acknowledges, that the U.S. Govern- routers and switches with inadequate burst capacity; dirty ment retains a non-exclusive, paid-up, irrevocable, world-wide license to optics; and failing network and system components. In ad- publish or reproduce the published form of this manuscript, or allow others to do so, for U.S. Government purposes. dition, another performance problem can be the misconfig- uration of data transfer hosts, which is often a contributing ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As factor in poor network performance. such, the Government retains a nonexclusive, royalty-free right to publish Many of these problems are found on the local area net- or reproduce this article, or to allow others to do so, for Government pur- works, often categorized as \general-purpose" networks, that poses only. Copyright is held by the owner/author(s). Publication rights are not designed to support large science data flows. Today licensed to ACM. many scientists are relying on these network infrastructures SC13 November 17-21, 2013, Denver, CO, USA to share, store, and analyze their data which is often geo- Copyright 2013 ACM 978-1-4503-2378-9/13/11 ...$15.00. http://dx.doi.org/10.1145/2503210.2503245 graphically dispersed. The Science DMZ provides a design pattern developed to procurement systems, web browsing, and so forth. Second, specifically address these local area network issues and offers these general networks must also be built with security that research institutions a framework to support data-intensive protects financial and personnel data. Meanwhile, these net- science. The Science DMZ model has been broadly deployed works are also used for research as scientists depend on this and has already become indispensable to the present and infrastructure to share, store, and analyze data from many future of science workflows. different sources. As scientists attempt to run their appli- The Science DMZ provides: cations over these general-purpose networks, the result is often poor performance, and with the increase of data set • A scalable, extensible network infrastructure free from complexity and size, scientists often wait hours, days, or packet loss that causes poor TCP performance; weeks for their data to arrive. • Appropriate usage policies so that high-performance Since many aspects of general-purpose networks are dif- applications are not hampered by unnecessary con- ficult or impossible to change in the ways necessary to im- straints; prove their performance, the network architecture must be adapted to accommodate the needs of science applications • An effective \on-ramp" for local resources to access without affecting mission critical business and security op- wide area network services; and erations. Some of these aspects that are difficult to change might include the size of the memory buffers for individual • Mechanisms for testing and measuring, thereby ensur- interfaces; mixed traffic patterns between mail and web traf- ing consistent performance. fic that would include science data; and emphasis on avail- This paper will discuss the Science DMZ from its devel- ability vs. performance and what can be counted on over opment to its role in future technologies. First, Section 2 time for network availability. will discuss the Science DMZ's original development in ad- The Science DMZ model has already been implemented at dressing the performance of TCP-based applications. Sec- various institutions to upgrade these general-purpose, insti- ond, Section 3 enumerates the components of the Science tutional networks. The National Science Foundation (NSF) DMZ model and how each component adds to the overall recognized the Science DMZ as a proven operational best paradigm. Next, Sections 4 and 5 offer some sample illustra- practice for university campuses supporting data-intensive tions of networks that vary in size and purpose. Following, science and specifically identified this model as eligible for Section 6 will discuss some examples of Science DMZ imple- funding through the Campus Cyberinfrastructure{Network mentations from the R&E community. And lastly, Section 7 Infrastructure and Engineering Program (CC-NIE).1 This highlights some future technological advancements that will program was created in 2012 and has since been respon- enhance the applicability of the Science DMZ design. sible for implementing approximately 20 Science DMZs at different locations|thereby serving the needs of the science 2. MOTIVATION community. Another NSF solicitation was released in 2013 and awards to fund a similar number of new Science DMZ's When developing the Science DMZ, several key principles are expected. provided the foundation to its design. First, these design patterns are optimized for science. This means the compo- 2.1 TCP Performance nents of the system|including all the equipment, software The Transmission Control Protocol (TCP) [15] of the and associated services|are configured specifically to sup- TCP/IP protocol suite is the primary transport protocol port data-intensive science. Second, the model is designed used for the reliable transfer of data between applications. to be scalable in its ability to serve institutions ranging TCP is used for email, web browsing, and similar applica- from large experimental facilities to supercomputing sites tions. Most science applications are also built on TCP, so it to multi-disciplinary research universities to individual re- is important that the networks are able to work with these search groups or scientists. The model also scales to serve a applications (and TCP) to optimize the network for science. growing number of users at those facilities with an increas- TCP is robust in many respects|in particular it has so- ing and varying amount of data over time. Lastly, the Sci- phisticated capabilities for providing reliable data delivery ence DMZ model was created with future innovation in mind in the face of packet loss, network outages, and network by providing the flexibility to incorporate emerging network congestion. However, the very mechanisms that make TCP services. For instance, advances in virtual circuit services, so reliable also make it perform poorly