doi:10.1145/2076450.2076468 Globus Online manages fire-and-forget file transfers for big-data, high-performance scientific collaborations.
By Bryce Allen, John Bresnahan, Lisa Childers, Ian Foster, Gopi Kandaswamy, Raj Kettimuthu, Jack Kordas, Mike Link, Stuart Martin, Karl Pickett, and Steven Tuecke Software as a Service for Data Scientists
As big data emerges as a force in science,2,3 so, too, do new, onerous tasks for researchers. Data from specialized instrumentation, numerical simulations, and downstream manipulations must be collected, indexed, archived, shared, replicated, and analyzed. These tasks are not new, but the complexities involved in performing them for terabyte or when data volumes were measured larger datasets (increasingly common in kilobytes. The result is a computa- across scientific disciplines) are quite tional crisis in many laboratories and different from those that applied a growing need for far more powerful data-management tools, yet the typi- key insights cal researcher lacks the resources and expertise to operate these tools. The costs of research data life-cycle The answer may be to deliver re- management are growing dramatically as search data-management capabili- data becomes larger and more complex. ties to users as hosted “software as a SaaS approaches are a promising service,” or SaaS,18 a software-delivery solution, outsourcing time-consuming model in which software is hosted research data management tasks to third-party services. centrally and accessed by users using a thin client (such as a Web browser) Globus Online demonstrates the potential over the Internet. As demonstrated in of SaaS for research data management, simplifying data movement for research- many business and consumer tools, ers and research facilities alike. SaaS leverages intuitive Web 2.0 in-
february 2012 | vol. 55 | no. 2 | communications of the acm 81 contributed articles
terfaces, deep domain knowledge, mand is issued; that is, third-party and economies of scale to deliver ca- transfers may be (indeed, frequently pabilities that are easier to use, more are) involved. Our goal in designing capable, and/or more cost-effective GO is a solution that provides extreme than software accessed through other ease-of-use without compromising means. The opportunity for continu- A common question reliability, speed, or security. A 2008 ous improvement via dynamic deploy- about GO is whether report by Childers et al.5 of Argonne ment of new features and bug fixes National Laboratory makes clear the is also significant, as is the potential data can be moved importance of usability. In particu- for expert operators to intervene and more effectively lar, failure recovery is often a human- troubleshoot on the user’s behalf. intensive process, as reported by We report here on a research data- through the Childers et al.5: “The tools that we use management system called Globus to move files typically are the standard Online, or GO, that adopts this ap- physical movement Unix tools included with ssh… it’s just proach, focusing on GO’s data-move- of media rather painful. Painful in the sense of having ment functions (“GO-Transfer”). We to manage the transfers by hand, re- describe how GO leverages modern than through starting transfers when they fail—all Web 2.0 technologies to provide in- communication of this is done by hand.” tuitive interfaces for fire-and-forget In addition, datasets may have file transfers between GridFTP end- over networks. nested structures and contain many points1 while leveraging hosting for files of different size (see the side- automatic fault recovery, high per- bar “Other Approaches”). Source and formance, simplified security con- destination may have different se- figuration, and no client software in- curity requirements and authentica- stallation. We also describe a novel tion interfaces. Networks and storage approach for providing a command- servers may suffer transient failures. line interface (CLI) to SaaS without Transfers must be tuned to exploit distributing client software and Glo- high-speed research networks. Direc- bus Connect to simplify installation tories may have to be mirrored across of a personal GridFTP server for use multiple sites, but only some files dif- with GO. Our experiments show low fer between source and destination. overhead for small transfers and high Firewalls, Network Address Transla- performance for large transfers, rela- tion, and other network complexities tive to conventional tools. may have to be addressed. For these Adoption of this new service has and other reasons, it is not unusual to been notable. One year after product hear of even modest-scale wide-area launch, November 2010, more than data transfers requiring days of care- 3,000 registered users had in aggre- ful “babysitting” or of being aban- gate moved more than two petabytes doned for high-bandwidth but high- (2×1015B) and 150 million files; nu- latency (frequently labor-intensive merous high-performance comput- and error-prone) “sneakernet.”11 ing (HPC) facilities and experimental facilities recommend GO to their us- Why Move Data at All? ers; and several “science gateways” Why not just leave data where it is are integrating GO as a data upload/ created? Such an option is certainly download solution. GO has also been preferred when possible, and we may adopted as a foundational element hope that over time moving computa- of the National Science Foundation’s tion to data rather than the other way new (as of 2011) Extreme Science and round will be more common. How- Engineering Discovery Environment ever, in practice, data scientists often (XSEDE) supercomputer network find data is “in the wrong place” and (http://www.xsede.org/). thus must be moved for a variety of reasons. Data may be produced at a Data Movement location (such as a telescope or sen- Researchers often must copy many sor array) where large-scale storage files with potentially large aggregate cannot be located easily. It may be size among two or more network- desirable to collocate data from many connected locations, or “endpoints,” sources to facilitate analysis—a com- that may or may not include the com- mon requirement in, say, genomics. puter from which the transfer com- Remote copies may be required for
82 communications of the acm | february 2012 | vol. 55 | no. 2 contributed articles disaster recovery. Data analysis may require computing power or special- ized computing systems not avail- able locally. Policy or sociology may Other Approaches require replication of data sets in One alternative for data movement involves running tools on the user’s computer; for example, Rsync,20 scp, file transfer program (FTP), secure FTP, and bbftp13 are all distinct geographical regions; for ex- used to move data between a client computer and a remote location. Other software ample, in high-energy physics, all data (such as globus-url-copy, Reliable File Transfer, File Transfer Service, and Lightweight produced at the Large Hadron Col- Data Replicator) can each manage large numbers of transfers. However, the need to lider, Geneva, Switzerland, must es- download, install, and run software is a significant barrier to use. Users spend much time configuring, operating, and updating such tools though rarely have theIT and networking sentially be replicated in the U.S. and knowledge necessary to fix things when they do not “just work,” which is all too often. elsewhere for independent analysis. Some big-science projects have developed specialized solutions to the problem; for It is also frequently the case that the example, the PhEDEx high-throughput data-transfer-management system9 manages data movement among sites participating in the Compact Muon Solenoid experiment aggregate data-analysis requirements at CERN, and the Laser Interferometer Gravitational Wave Observatory (LIGO) project of a community exceed the analysis developed the LIGO Data Replicator.4 These centrally managed systems allow users capacity of a data provider, in which to hand off data-movement tasks to a third-party service that performs them on their case data must be downloaded for lo- behalf. However, these services require professional operators functioning only among carefully controlled endpoints within these communities. cal analysis. This is the case in, for ex- Managed services (such as YouSendIt and DropBox) also provide data-management ample, the Earth System Grid, which solutions but do not address researchers’ need for high-performance movement of delivers climate simulation output to large quantities of data. BitTorrent8 and Content Distribution Networks21 are good at distributing a relatively stable set of large files (such as movies) but do not address data its 25,000 users worldwide. scientists’ need for many frequently updated files managed in directory hierarchies. Another common question about The Integrated Rule-Oriented Data System17 is often run in hosted configurations, GO is whether data can be moved but, though it performs some data-transfer operations (such as for data import), data more effectively through the physi- transfer is not its primary function or focus. The Kangaroo,19 Stork,14 and CATCH15 systems all manage data movement over cal movement of media rather than wide-area networks using intermediate storage systems where appropriate to optimize through communication over net- end-to-end reliability and/or performance. They are not designed as SaaS data- works. After all, no network can ex- movement solutions, but their methods could be incorporated into GO. ceed the bandwidth of a FedEx truck. Web and REST interfaces to centrally operated services are conventional in business, underpinning such services as Salesforce.com (customer relationship The answer is, again, that while physi- management), Google Docs, Facebook, and Twitter—an approach not yet common in cal shipment has its place (and may science. Two exceptions are the PhEDEx Data Service,9 with both REST and CLIs, and be much cheaper if the alternative is the National Energy Research Supercomputing Center, or NERSC, Web toolkit called NEWT7 that enables RESTful operations against HPC center resources. to pay for a high-speed network con- nection), it is not suitable in all situ- ations. Latency is high, and so is the human overhead associated with the user to provide credentials GO can mands to transfer the requested files loading and unloading media, as well use to interact with those endpoints directly between the endpoints. GO as with keeping track of what has been on the user’s behalf. GO then estab- monitors the transfer progress and shipped. Nevertheless, it could be fea- lishes authenticated GridFTP control updates transfer state in the state da- sible to integrate into GO methods channels with each endpoint and is- tabase. This information can be used for orchestrating physical shipment sues the appropriate GridFTP com- to restart transfers after faults and re- when it is determined to be faster and/or more cost-effective—as with Principal Globus Online data-transfer commands. Cho’s and Gupta’s Pandora (“People and networks moving data around”) system.6 Class Name Description ls List files and directories on an endpoint. Different Interfaces for transfer Request data transfer of one or more files or directories between end- Different Users Create points; support recursive directory transfer and rsync-like synchronization. Transfer The Computation Institute at the scp Request data transfer of a single file or directory; syntax and semantics based on secure copy utility to facilitate retargeting to GO of scripts using University of Chicago and Argonne scp for data movement. National Laboratory operates GO as status List transfers initiated by requesting user, along with summary information a highly available service (http://www. (such as status, start time, and completion time). globusonline.org/) to which users Monitor details Provide details on a transfer (such as number of files transferred submit data-movement and synchro- Transfers and number of faults). nization requests. A typical transfer events List events associated with a specified transfer: start, stop, request proceeds as follows: A user performance, faults. cancel Terminate specified transfer or individual file in a transfer. authenticates with GO and submits a Control wait Wait for specified transfer to complete; show progress bar. request. GO records the request into Transfers its state database, inspects the re- Alter deadline for a transfer. quest to determine what endpoints are involved, and if necessary prompts
february 2012 | vol. 55 | no. 2 | communications of the acm 83 contributed articles
port progress to the user. GO keeps at- through standard-session cookie- joe$ ssh cli.globusonline.org tempting a failed request periodically based client authentication. The Welcome to globusonline.org, ian. until the task deadline is reached or Web GUI supports browsing remote $ scp alcf#dtn:~/myfile nersc#dtn:~/ the user cancels the request. When file systems, as well as submitting, myfile the transfer completes or an unrecov- monitoring, and cancelling transfer Contacting ‘gs1.intrepid.alcf.anl.gov’... erable fault is encountered the user is requests. Enter MyProxy pass phrase: ******** notified via email. A CLI supports client-side script- GO supports a friendly, intuitive ing; for example, a script that, each This example command also illus- Web GUI for ad hoc and less-technical evening, transfers new files created trates how endpoints can define logi- users; a CLI for use by more advanced during the day to a remote reposi- cal names for physical nodes. For users and for scripting; and a Rep- tory or that automatically moves out- example, alcf#dtn denotes the data- resentational State Transfer (REST) put from an analysis job back to a lo- transfer nodes running GridFTP serv- application programming interface cal machine. A CLI typically requires ers at the Argonne Leadership Com- (API) facilitating integration for sys- installation of client-side libraries, puting Facility (ALCF: http://www.alcf. tem builders that also supports the though it is counter to the key SaaS anl.gov/). Sites can define and publish GO Web interface. The table here lists tenet of not requiring client software their own endpoint definitions (such GO’s primary transfer-management installation to use the service. To obvi- as alcf#dtn, nersc#dtn); users are able functions. Additional endpoint-man- ate having to install software, the GO to define custom endpoint definitions agement functions provide for the system provides all GO users with a re- as well (such as mylaptop, myserver). creation, deletion, configuration, ac- stricted shell, to which they can ssh to More than 300 such endpoint defini- tivation, and deactivation of logical execute commands. Thus, a user, Joe, tions have been defined, incorporat- endpoints. Other functions support can write ing many major research computing administrative tasks (such as listing systems in the U.S. and elsewhere available commands, reviewing com- ssh [email protected] \ worldwide. mand history, and obtaining com- scp alcf#dtn:~/myfile nersc#dtn:~/ mand help). myfile User Profile and The REST interface uses HTTP Identity Management GET, PUT, POST, and DELETE opera- to copy myfile from source alcf#dtn to An important GO feature is the abil- tions against a defined set of URLs destination nersc#dtn. The boldface ity to handle transfers across mul- representing GO resources. Thus, to text invokes the GO scp, or secure tiple security domains with multiple create a transfer task, a user issues a copy, command, mirroring the syntax user identities. Unlike many systems, POST to https://transfer.api.globuson- of the popular scp. It supports many including most previous Grid file- line.org/v0.10/transfer with a docu- regular scp options, plus some addi- transfer services, GO does not require ment describing the transfer request, tional features, and is much faster be- a single, common security credential including, for example, source and cause it invokes GridFTP transfers. Al- across all transfer endpoints. Rather, destination endpoints and file paths ternatively, Joe can first ssh to http:// it assumes users have many identities and options; to access the status of a cli.globusonline.org/, then issue a se- for use with different service provid- task, the user issues a GET request to ries of commands directly: ers and that GO’s job is to ensure the https://transfer.api.globusonline.org/ v0.10/task/
ted using JavaScript Object Nota- User tion (JSON) and Extensible Markup Worker gateway User Language (XML). Supported security
mechanisms include HTTPS mutual GridFTP TM User authentication with an X.509 client server certificate and (for Web browsers) Profiles and state HTTPS server authentication with User cookie-based client authentication. Notification The Web interface builds on the target REST interface using standard Asyn- chronous JavaScript (AJAX) and XML techniques. A GO Web page contains standard HTML, CSS, and JavaScript, interacting with the REST interface
84 communications of the acm | february 2012 | vol. 55 | no. 2 contributed articles
right identities are brought to bear at Figure 2. Globus Connect architecture. the right time for any transfer and do so in a way that is easy for users to un- derstand. To this end, all users have GO ac- Globus (1) Globus Connect counts where they easily configure User TM Online registers with (2) User makes request to Globus Online their profile with various identities; Globus Online: e.g., “transfer for example, they can register their data from MyDesktop to SiteA” (3) Globus MyProxy Certification Authority (CA)16 Online forwards requests to identities (such as for NERSC and Globus Connect other computing centers using the ap- GridFTP Globus server Connect proach), OAuth protocol12 identities “SiteA” (4) Globus Connect establishes data channel connection to SiteA and transfers data (such as for ALCF, the NSF XSEDE net- “MyDesktop” work of supercomputer centers, and Facebook), OpenID identities (such as for Google), Shibboleth10 identities (such as for campus credentials), and X.509 identities (such as from the U.S. caching them until they expire or are transaction database, transfer agents, Department of Energy Grids CA and explicitly deleted by the user. GO also history database, transfer REST API International Grid Trust Federation- uses the same user proxy credential server, CLI server, and backup. In ad- certified CA). for endpoints that have the same de- dition, the GO team runs two Nagios Though GO stores identities, it fault MyProxy server, so users need servers on EC2 instances, one in the does not store passwords; rather, it not enter the same password multiple U.S. East region to monitor all other knows only the user name so it can times. If a credential expires before instances, the other in the U.S. West prompt for the appropriate informa- the transfer completes, GO notifies region to monitor the health of the tion when that identity is needed. In the user via email that the user must primary Nagios server. The GO team addition, identities can be configured re-authenticate. Until such time as also uses the Chef configuration- as “federated identities” the user uses the credential is renewed, the transfer management tool for provisioning to authenticate with GO; for example, is suspended. all servers. The vast majority of GO is users who have already authenticated programmed in Python, running on their browser session with an OpenID, Scalable Cloud-based Ubuntu Linux servers, with Cassandra OAuth, or Shibboleth identity can use Implementation and Postgres databases. GO without having to authenticate SaaS requires reliability and scalabil- Figure 1 is a somewhat abstracted further, and X.509 (proxy) identities ity, continuing to operate despite the view of the GO implementation, show- can be used to authenticate with the failure of individual components and ing the user gateway servers support- GO Web site, CLI, and REST API. behaves appropriately as usage grows. ing interaction between users and the GO keeps track of what security cre- To this end, the GO team applies system via Web GUI, CLI, and REST in- dentials are required by the different methods commonly used by SaaS pro- terfaces; the worker processes orches- endpoints with which users may wish viders, running GO on a commercial trating data transfers and other tasks to communicate. Then, where possi- cloud provider, Amazon Web Services (such as notifying users of changes in ble, it caches information it can use to (AWS). The GO implementation uses a state); and the profiles and state da- facilitate access. For example, assume combination of Amazon Elastic Com- tabase storing user profiles, request user U must provide X.509 credential pute Cloud (EC2), Amazon Elastic state, and endpoint information. U-A to access endpoint A and X.509 Load Balancing, and Amazon Simple The authors’ current thinking on credential U-B to access endpoints Storage Service (S3). availability is that that the research B1 and B2. To perform a file transfer The GO implementation involves community needs between three and from A to B1, as requested by the user, platform services, which provide user, four 9s (99.9%–99.99%), correspond- GO requires short-term (typically 12- profile, and group-management func- ing to between one and 10 minutes hour) X.509 proxy credentials22 it can tions, and the file-transfer service downtime per week. Longer than 10 use to authenticate the user request that implements the data-movement minutes lack of availability can be to the GridFTP servers running at functionality that is the focus of this problematic for users employing GO endpoints A and B1. If GO does not article. The GO team runs the plat- as part of time-critical work processes have such credentials, it prompts the form services on a collection of EC2 (such as in astronomy data-process- user for them when the user requests instances across several availability ing pipelines). This requirement is the transfer. Alternatively, a user (or zones in Amazon’s U.S. East region a primary reason the GO team hosts script) can proactively push X.509 (located in Virginia), including Web GO on AWS rather than on a research proxy credentials to GO for use with server, load balancer, database, and computing facility, which, in our expe- specific endpoints. backup. The file-transfer service runs rience, provides closer to two-9s avail- When GO has the needed creden- on a collection of EC2 instances host- ability when occasional maintenance tials it proceeds with the transfer, ed in the U.S. East region, including of shutdowns are taken into account.
february 2012 | vol. 55 | no. 2 | communications of the acm 85 contributed articles
Globus Connect (Multi-User) to/from a Globus Connect instance Globus Connect Multi-User (GCMU), GO users need not install software via the control-channel tunnel. Thus, simplifying the process of connecting to request transfers between remote to request a transfer to/from the com- a shared server or cluster to GO. With GridFTP servers. However, software puter on which they have installed GCMU a resource owner can quickly installation is required if a source or Globus Connect, users interact with set up a GO endpoint on any server destination computer does not have GO just as they would request any that can then be accessed by multi- GridFTP installed, as when, for exam- other transfer (see Figure 2). GO re- ple users for remote data movement. ple, transferring data to/from a user’s lays the request via the tunnel to the GCMU packages a GridFTP server, computer. Globus Connect server, which then MyProxy server, and MyProxy Online To address this need, we introduced executes the transfer. CA pre-configured for GO use, requir- in early 2011 Globus Connect, a one- Globus Connect establishes only ing only a few steps to install and use. click download-and-install applica- outbound connections and thus can A growing number of research facili- tion for Linux, MacOS, and Windows. work behind a firewall or other net- ties users (such as the University of Globus Connect consists of a GridFTP work interface device that does not Colorado, University of Michigan, server that runs as the user (rather allow for inbound connections. The Oak Ridge National Laboratory, and than root from inetd like a typical Globus Connect server is stateless Advanced Photon Source) use GCMU GridFTP server) and a GSI-OpenSSH and thus can be started and stopped to make their resources accessible to client configured to establish an au- at will; all state associated with trans- remote users. thenticated connection to a GO relay fers is maintained by GO. Autoupdate Globus Connect also incorporates server, so as to tunnel GridFTP con- means the user need not maintain the user-friendly methods for automat- trol channel requests from GO. This software over time. ing the process of generating, install- Globus Connect GridFTP server uses The GO team also streamlined the ing, and configuring the certificate only outbound data-channel connec- process of standing up a GridFTP serv- required for a Globus Connect instal- tions. GO can direct transfer requests er as a shared resource by introducing lation. It uses an online private CA in- corporated into GO to generate a new Figure 3. Data-transfer performance between ALCF and NERSC. service certificate when a user adds a Globus Connect endpoint. Users copy go-single-ep scp guc tuned-guc go a secret “setup key” from the GO Web 10,000 site to the Globus Connect setup win- 1,000 dow to securely pair it with their new endpoint definition. Globus Connect 100 uses the setup key as a one-time-use token to download the certificate, pri- 10 vate key, and GridFTP gridmap con-
Rate (Mbits/sec) 1 figuration over a secure GSI-OpenSSH connection. GO can then authenticate 0.1 to the Globus Connect instance and be sure it is talking to the correct one. 0.01
1.E+03 1.E+04 1.E+05 1.E+06 1.E+07 1.E+08 1.E+09 Optimized File Transfers for All File Size (bytes) GridFTP client interfaces allow us- ers to optimize transfer performance by setting parameters (such as TCP Figure 4. Data-transfer performance between two EC2 instances. buffer size, number of outstanding requests, or “pipelining,” number of scp guc tuned-guc go concurrent control channel connec- 10,000 tions, or “concurrency,” and number of TCP channels used for data move- 1,000 ment, or “parallelism”). However, few users have the experience and time
100 needed to apply these settings effec- tively.
Rate (Mbits/sec) GO obviates the need for user tun- 10 ing by applying heuristics to set pa- rameters based on the number and
1 size of files in a transfer. Upon arrival of a recursive transfer request, GO 1.E+03 1.E+04 1.E+05 1.E+06 1.E+07 1.E+08 1.E+09 crawls the directory to find the files File Size (bytes) to transfer, determining file size in the process. It then sorts them by size
86 communications of the acm | february 2012 | vol. 55 | no. 2 contributed articles and performs the transfer in chunks, they may choose to log into the GO setting parameters for each chunk ac- CLI gateway and issue the commands cording to the average size of its files. directly, thus avoiding the per-request If a chunk has more than 100 files and ssh cost. an average file size smaller than 50MB, To evaluate GO’s performance- GO applies small settings, making use Though GO stores optimization logic in practical situ- of pipelining: Specifically, concur- identities, it does ations, we also conducted tests (in rency=2 files in transit at once, paral- 2011) to compare GO performance lelism=2 sockets per file, and pipelin- not store when transferring large quantities ing=20 requests outstanding at once. passwords; of data between pairs of endpoints If all files in a chunk are larger then with that achieved using scp and the 250MB, GO applies large settings that rather, it knows globus-url-copy (GUC) client. As scp is use more parallelism and moderate only the user name known to perform poorly, particularly pipelining: concurrency=2, parallel- over wide-area networks, we included ism=8, and pipelining=5. In all other and how to use it, this option in the test primarily as a cases, the default setting is used: con- sanity check; if GO is not better than currency=2, parallelism=4, and pipe- so it can prompt scp, then something is wrong. GUC, on lining=10. When a Globus Connect or for the appropriate the other hand, drives GridFTP trans- GCMU endpoint is used as the desti- fers and so represents a fairer com- nation in a GO transfer, then stream information parison. However, in its default con- mode (not mode E, which allows for when that identity figuration (which, Globus developers out-of-order transmission) must be tell us, many users use unchanged) used, and concurrency is the only op- is needed. GUC does not employ optimizations timization that can be applied. When used by GO; for example, GUC does using steam mode, then for small file not enable concurrency, parallelism, chunks, GO sets concurrency=8. pipelining, or data channel cach- These simple heuristics have ing. This comparison thus permitted proved effective but can surely be im- evaluation of the performance gains proved; for example, GO could be ex- many users expect from GO. We also tended to manipulate the TCP buffer compared GO against GUC with pa- size (such as on the basis of round- rameters tuned by an expert to maxi- trip-time measurements), select alter- mize performance—tuned-guc in the native transport protocols, or reserve results. the network. Figure 3 charts results of GO-based data transfer in 2011 over a high- Performance and Scalability speed wide-area network—ESNet, Dispatching requests to a hosted ser- the Energy Sciences Network, http:// vice rather than executing them di- www.es.net/—between two high-per- rectly on a user’s computer introduc- formance parallel storage systems, es temporal overhead due to the need and Figure 4 between local-instance to communicate the request to the GO storage of two EC2 instances within user gateway operating on a remote different Amazon Availability Zones computer. To evaluate this overhead, in a single geographic region to ap- we conducted tests (in 2011), issuing proximate a transfer over a campus 100 consecutive requests to transfer network. Figure 3 gives results both a 1B file between two locations, using between a single data-transfer node scp first, then GOscp dispatched to GO (DTN) at ALCF and NERSC (“go-sin- via SSH. We measured total times of 93 gle-ep”) and (the default configura- and 273 seconds, respectively, an av- tion) using the two DTNs supported erage per-request cost of 0.93 seconds by ALCF and NERSC (“go”). Each DTN for scp and 2.73 seconds for GO scp. is a fast server with a 10Gb/s network We concluded that the request-setup to ESnet and a fast local connect to cost associated with the use of GO is each site’s GPFS parallel file system. ~1.8 seconds. This overhead is accept- Meanwhile, scp performs poorly for able for many data-transfer applica- all data transfers, and GUC, with its tions, though certainly not for all. default configuration, performs poor- Note that users who want to request ly for all data transfer sizes over the many transfers will normally do so wide area, as well as for small files in with a single request. If users want to the local area. (The default configura- perform consecutive small requests, tion clearly requires improvement.)
february 2012 | vol. 55 | no. 2 | communications of the acm 87 contributed articles
Fortunately, tuned-guc performs bet- Martin Feller, Daniel Gunter, Thomas R., de Torcy, A., Wan, M., Schroeder, W., Chen, S.-Y., Gilbert, L., Tooby, P., and Zhu, B. iRODS Primer: ter than untuned GUC in almost all Howe, Lukasz Lacinski, Steven Link, Integrated Rule-Oriented Data System. Morgan and cases. In the wide-area case, it does Ravi Madduri, Daniel Morgan, Mi- Claypool Publishers, 2010. 18. sun, W., Zhang, K., Chen, S.-K., Zhang, X., and Liang, less well than GO for smaller files, chael Russell, Eugene Sadhu, Mei-Hui H. Software as a service: An integration perspective. probably because GO drives GridFTP Su, Vas Vasiliadis, Vanamala Venka- In Proceedings of the Fifth International Conference on Service-Oriented Computing, B. Krämer, K.-J. Lin, pipelining more aggressively, due to taswamy, and Andrew Zich for their and P. Narasimhan, Eds. (Vienna, Austria, Sept. 17–20). the improved pipelining support in work on GO, as well as many users for Springer, Berlin/Heidelberg, 2007, 558–569. 19. thain, D., Basney, J., Son, S.-C., and Livny, M. The GO’s GridFTP client. However, tuned- helpful suggestions. This work was Kangaroo approach to data movement on the grid. guc does better than GO for large supported in part by DOE DE-AC02- In Proceedings of the 10th IEEE International Symposium on High-Performance Distributed files, though GO performance can be 06CH11357; NSF OCI-534113; and Computing (San Francisco, Aug. 7–9). IEEE Computer Society Press, Washington, D.C., 2001, 325–333. tuned further. Note, GO transfers to NIH NCRR 1 U24 RR025736-01. 20. tridgell, A. and Mackerras, P. The Rsync Algorithm a two-DTNs vs. a single-DTN are not TR-CS-96-05. Department of Computer Science, Australian National University, Canberra, 1994. substantially different, except for the References 21. wang, L., Park, K.S., Pang, R., Pai, V., and Peterson, largest transfer. We conclude that the 1. allcock, B., Bresnahan, J., Kettimuthu, R., Link, L. Reliability and security in the CoDeeN content M., Dumitrescu, C., Raicu, I., and Foster, I. The distribution network. In Proceedings of the USENIX bottleneck is not the DTNs but either Globus striped GridFTP framework and server. In Annual Technical Conference (Boston, June 27–July the network or local storage. Proceedings of the 2005 ACM/IEEE Conference on 2). USENIX Association, Berkeley, CA, 2004, 171–184. Supercomputing (Seattle, Nov. 12–18). ACM Press, 22. welch, V., Foster, I., Kesselman, C., Mulmo, O., New York, 2005. Pearlman, L., Tuecke, S., Gawor, J., Meder, S., and Conclusion 2. bell, G., Hey, T., and Szalay, A. Beyond the data deluge. Siebenlist, F. X.509 proxy certificates for dynamic Science 323, 5919 (Mar. 2009), 1297–1298. delegation. In Proceedings of the Third Annual Public Exploding data volumes are trans- 3. berriman, G.B. and Groom, S. How will astronomy Key Infrastructure R&D Workshop (Gaithersburg, archives survive the data tsunami? Commun. ACM 54, forming many researchers into data MD, Apr. 12–14), National Institute of Standards and 12 (Dec. 2011), 52–56. Technology, Gaithersburg, MD, 2004. scientists, with urgent need for more 4. chervenak, A., Schuler, R., Kesselman, C., Koranda, S. and Moe, B. Wide-area data replication for scientific capable, efficient data-management collaborations. In Proceedings of the Sixth IEEE/ACM Bryce Allen ([email protected]) is a software tools. SaaS may represent the means International Workshop on Grid Computing (Seattle, Nov. developer in the Computation Institute of the University of 13). IEEE Computer Society, Washington, D.C., 2005. Chicago and Argonne National Laboratory, Argonne, IL. by which such tools are provided cost- 5. childers, L., Liming, L., and Foster, I. Perspectives on effectively, with GO as a first step in Distributed Computing: 30 People, Four User Types, John Bresnahan ([email protected]) is a senior and the Distributed Computing User Experience, software developer in the Mathematics and Computer that direction. A hosted data-move- Technical Report ANL/MCS/CI-31. Argonne National Science Division of Argonne National Laboratory, Argonne, IL. Laboratory, Argonne, IL, 2008. ment service with intuitive interfaces, 6. cho, B. and Gupta, I., Budget-constrained bulk data Lisa C. Childers ([email protected]) is a staff automatic fault recovery, high per- transfer via Internet and shipping networks. In member of the Mathematics and Computing Science Proceedings of the Eighth ACM international conference Division of Argonne National Laboratory, Argonne, IL, and formance, and easy-to-use security, on Autonomic Computing (Karlsruhe, Germany, June of the Computation Institute of the University of Chicago. it has already (since its introduction, 14–16). ACM Press, New York, 2011, 71–80. 7. cholia, S., Skinner, D., and Boverhof, J. NEWT: A Ian T. Foster ([email protected]) is an Argonne late 2010) won enthusiastic adoption RESTful service for building high-performance Distinguished Fellow, Director of the Computation in the world of big-data science appli- computing Web applications. In Proceedings of the Institute, and the Arthur Holly Compton Distinguished 2010 Gateway Computing Environments Workshop Service Professor of Computer Science at Argonne cations. Many operators of scientific (New Orleans, Nov. 14). IEEE Computer Society Press, National Laboratory, Argonne, IL, and at the University facilities worldwide recommend GO 2010, 1–11. of Chicago. 8. cohen, B. Incentives build robustness in BitTorrent. Gopi Kandaswamy ([email protected]) is to their users. Our experiments show In Proceedings of the First International Workshop on an associate consultant in Tata Consultancy Services Economics of P2P Systems (Berkeley, CA, June 5–6, GO can achieve high performance and and former senior research systems developer in the 2003). Information Sciences Institute, Los Angeles. exceptional reliability in a variety of 9. egeland, R., Wildishb, T., and Huang, C.-H. PhEDEx settings, with low per-transfer over- data service. Journal of Physics: Conference Series Rajkumar Kettimuthu ([email protected]) is a fellow 219 (2010). in the Computation Institute and principal software head and bandwidth rarely exceeded 10. erdos, M. and Cantor, S. Shibboleth Architecture. development specialist in the Mathematics and Computer in human-tuned transfers. Internet 2, May 2, 2002; http://shibboleth.internet2. Science Division of Argonne National Laboratory, Argonne, edu/docs/draft-internet2-shibboleth-arch-v05.pdf IL, and the University of Chicago. These positive results have encour- 11. gray, J., Chong, W., Barclay, T., Szalay, A., and aged us to expand GO to address other Vandenberg, J. TeraScale SneakerNet: Using Jack Kordas ([email protected]) is a senior Inexpensive Disks for Backup, Archiving, and Data architect and developer in the Computation Institute research data-management prob- Exchange Technical Report MSR-TR 2002-54. of the University of Chicago and Argonne National Microsoft Research, Redmond, WA, 2002. Laboratory, Argonne, IL. lems. Recognizing that a notable rea- 12. hammer-Lahav, E. The OAuth 1.0 Protocol. Internet son for moving data is to share it with Engineering Task Force RFC 5849, 2010; http://tools. Michael Link ([email protected]) is a software ietf.org/html/rfc5849 developer in the Computation Institute of the University of other scientists, the GO development 13. hanushevsky, A., Trunov, A., and Cottrell, L. Peer-to- Chicago and Argonne National Laboratory, Argonne, IL. team is adding data-sharing support peer computing for secure high-performance data copying. In Proceedings of the 2001 International Stuart Martin ([email protected]) is a manager of like that provided by DropBox. To sim- Conference on Computing in High Energy and Nuclear software development in the Computation Institute of the plify the specification of sharing poli- Physics (Beijing, Sept. 3–7, 2001). University of Chicago and Argonne National Laboratory, 14. kosar, T. and Livny, M. A framework for reliable and Argonne, IL, managing software development for the cies, GO developers have integrated efficient data placement in distributed computing Globus Toolkit and portions of Globus Online. systems. Journal of Parallel and Distributed group management. In turn, these Karl Pickett ([email protected]) is a programmer in the Computing 65, 10 (Oct. 2005), 1146–1157. Computation Institute of the University of Chicago and mechanisms provide a foundation on 15. monti, H., Butt, A.R., and Vazhkudai, S.S. CATCH: A Argonne National Laboratory, Argonne, IL. cloud-based adaptive data-transfer service for HPC. which can be built a range of other ca- th In Proceedings of the 25 IEEE International Parallel Steven Tuecke ([email protected]) is deputy pabilities, notably support for collab- & Distributed Processing Symposium (Anchorage, director of the Computation Institute of the University of orative tools. Alaska, May 16–20). IEEE Computer Society, 2011, Chicago and Argonne National Laboratory, Argonne, IL, 1242–1253. leading the Globus Online project. 16. novotny, J., Tuecke, S., and Welch, V. An online Acknowledgments credential repository for the grid: MyProxy. In Proceedings of the 10th IEEE International We thank Vijay Anand, Rachana An- Symposium on High-Performance Distributed Computing (San Francisco, Aug. 7–9). IEEE Computer anthakrishnan, Joshua Boverhof, Kyle Society Press, Washington, D.C., 2001, 104–111. Chard, Ann Chervenak, Paul Davé, 17. rajasekar, A., Moore, R., Hou, C.-Y., Lee, C.A., Marciano, © 2012 ACM 0001-0782/12/02 $10.00
88 communications of the acm | february 2012 | vol. 55 | no. 2