B. Dempsey and D. Weiss

On the Performance and Scalability of a Data Mirroring Approach for I2-DSI Bert J. Dempsey, Debra Weiss University of North Carolina at Chapel Hill [email protected]

Abstract This paper presents work on scaleable design for the automated synchronization of large collections of files replicated across multiple hosts. Unlike conventional mirroring tools, our approach addresses multiple-site file synchronization by capturing file-tree update information in an output file during an initial file synchronization session. Once the update file is available, it can be transmitted over the network using parallel point-to-point file transfers or reliable multicasting and then processed at the remote sites. This paper outlines of how the above concept has been implemented as a modification to the open-source mirroring tool, rsync. It then presents performance experiments designed to characterize the server-side processing costs and network throughput requirements under realistic workloads on large storage servers. The performance experiments use the WAN testbed of the Internet2 Distributed Storage Infrastructure (I2-DSI) project, the context for this work, and the results provide guidance on the scalability of I2-DSI using the proposed data mirroring scheme.

Keywords: mirroring, distributed storage, replication, Internet2, reliable multicast, I2-DSI. and bottlenecks caused by hierarchical 1. Introduction processing of client requests, the limits Client-server applications on the Internet of caching dynamically generated server benefit from the use of content responses, and the tendency of some replication through improved access for content providers to eschew caching in widely distributed clients and balancing order to retain server-side control [1]. load across different servers and network paths. Content replication can be Mirroring source objects across server employed either in front of the server platforms is another approach to content (proxy-based caching of server replication. Among its key advantages responses) or behind the server are (1) the ability to replicate server-side (mirroring of source objects). On- functionality such as server-managed demand replication of server responses secure logins, (2) replication benefits for at proxy-based caches is widely any client-server protocol, not just HTTP deployed within the current WWW services, and (3) a basis for constructing where hierarchical or distributed manageable, deterministic control of the arrangements of cooperating caches replication process for the content handle HTTP requests and, less provider and service provider. In the commonly, those of other access current Internet, server-side replication protocols. While an important is most commonly employed for (1) the component of the current and future limited (but valuable) case of sharing Internet, proxy-based caching is tied to highly popular static file archives served specific protocols and has well-studied by FTP or HTTP or for (2) more general performance limitations. Problematic replication between servers controlled by aspects of proxy caching include delays a single organization where strong

NetStore 1999 - 1 - B. Dempsey and D. Weiss uniformity can be imposed on the server AFS. A primary advantage of our platforms. approach is that it provides a highly portable and easily modified data The context of our work is the Internet2 mirroring mechanism. Application-level Distributed Storage Initiative (I2-DSI) solutions, however, are unlikely to project [2], which is developing match the throughput achievable with replication middleware to extend source- kernel-level solutions. This paper object replication to loosely-coupled, provides empirical evidence that the file heterogeneous platforms. I2-DSI relies replication solution proposed here for I2- on a set of dedicated, geographically DSI will provide adequate performance distributed replication hosts (I2-DSI for the current I2-DSI testbed and scale servers) in the network on which user up with the number of I2-DSI replication content will be hosted. Application hosts. developers publish collections of source objects (I2-DSI content channels [3]) in The rest of the paper proceeds as conjunction with the I2-DSI interface follows. Section 2 presents the describing supported content types and motivation and design of a novel file the standard services available on the mirroring approach and the scenario for replication hosts. Current content its use within I2-DSI. In Section 3 channels are linked at [2]. As part of performance aspects of the solution are Internet2, the I2-DSI project aims to explored through controlled experiments develop an open architecture for that measure server load and the replication services to serve the 140 available network bandwidth using the Internet2 universities and, subsequently, I2-DSI project testbed. Section 4 gives the wider Internet community. additional discussion of related work, Proprietary solutions for application and Section 5 summarizes the findings hosting and replication are now of the paper. appearing in commercial efforts [4] [5]. 2 Rsync+ In this paper a design is presented for The data mirroring tool developed here, scaleable replication of large collections rsync+, is a modification to the open- of files in support of file-oriented source file-mirroring tool, rsync. Rsync replication across the I2-DSI servers.1 [7] is a widely used data mirroring tool Our design is a novel adaptation of an designed to automate synchronization of existing data mirroring tool, rsync, that file hierarchies residing on the same runs in user space and manages data machine or, alternatively, on machines replication through operating system across the Internet. Rsync has gained calls to the filesystem. Alternative popularity due to its rich feature set for approaches include block-level controlling the synchronization process, management of file replication either at and its open-source C code the middleware layer [6] or within a implementation has enabled it to be distributed filesystem such as NFS or ported to many Unix platforms and MS NT. A distinctive feature of rsync (and thus rsync+) is its use of block-level 1 I2-DSI ultimately expects to support multiple replication mechanisms in order to enable the checksumming to perform differential largest possible set of applications to benefit file update when an existing file has from its replication services.

NetStore 1999 - 2 - B. Dempsey and D. Weiss been updated, i.e., has a new disk image replication evokes a set of rsync under the same file name. sessions, one for each remote I2-DSI replication host that carries the channel. As shown, the clients for this I2-DSI Channel provider channel access the replication hosts nearest to them using resolution import mechanisms that are part of the I2-DSI replication architecture.

M The modifications developed for rsync+ rsync src/ S1::src/ are a response to two factors in the rsync src/ S2::src/ scenario in Figure 2.1 that result in rsync src/ S3::src/ inefficiency and limit scalability. First, M performs the same processing on the src/ directory for each of the three rsync sessions required to update the slave clients S3 hosts (S1, S2, and S3). This processing S1 includes checking file status information S2 and, if a file has been modified, computing checksums, and thus it can be compute-intensive for large file trees clients and/or large individual file objects. With multiple point-to-point rsync sessions, Figure 2.1: An Rsync Scenario for the processing load on M increases with Content Replication in I2-DSI the number of remote servers being In its current form, rsync synchronizes updated. Secondly, M transmits identical remote directory hierarchies during an data streams over the network in each of interactive network session involving the three master-slave rsync sessions two sites, the master site and a slave. (S1, S2, S3). Thus, for example, its use in the I2-DSI context would follow the scenario shown To enable efficient multiple-site in Figure 2.1. In the scenario shown, a synchronization, the rsync source code channel provider places new or modified was modified to create new modes of files for a file-based channel stored in operation, as shown in Figure 2.2. the directory, src/, on the master I2-DSI Options in rsync+ (the –f and –F flags) server, denoted as M. For our purposes, enable the end-to-end update process to the import mechanism here is not be decomposed into three independent specified. In the current I2-DSI actions: architecture, modifications to the source Step 1: the generation of replica objects in a channel are localized to a update information at the master master channel copy at one I2-DSI site, server, here M. The set of all other I2- Step 2: the network transport of DSI replication servers on which the update information from the channel is carried are shown within the master to the slave sites, and shaded area as S1, S2, and S3. On M Step 3: the processing of the then, a script managing channel update information at each slave

NetStore 1999 - 3 - B. Dempsey and D. Weiss

site in order to synchronize the 3 Performance Analysis slave’s files with those at the master. This section focuses on empirical results gathered to characterize the performance The key feature of rsync+ is then that it of and to provide insight into the decouples network communication (Step scalability of the data mirroring scenario 2) from filesystem update actions (Steps depicted in Figure 2.2. The data 1 and 3). This decoupling eliminates represents performance measurements redundant processing at the master site gathered from network and server and enables the use of parallel network experiments running on high- (TCP) connections or a reliable multicast performance servers within the I2-DSI transport protocol to deliver data (Step WAN testbed. In particular, the results 2) from a single master site to the quantify the balance between server multiple slave sites, resulting in a processing throughput for file scaleable approach to multiple-site file synchronization actions, server synchronization. throughput, (Step 1 and Step 3 in discussion above) and the achievable In Figure 2.2, it should be noted that the network throughput, network directory srcMaster/ that contains the throughput, (Step 2 above) when using latest updates may be located on M (as parallel TCP connections for data suggested in the Figure) or at a remote delivery. Understanding this balance is machine. In particular, srcMaster/ can crucial in identifying the current system be on the channel provider machine and bottlenecks, in planning future system rsync+ used as the import mechanism capacity, in assessing the impact that for updating src/ on M. Full details on hardware advances in server and the rsync+ implementation are in [8]. network technologies will have, and in understanding the scale at which Channel provider deployment of reliable multicast protocols becomes valuable and At M: ultimately critical for the I2-DSI system rsync+ -F srcMaster/ src/ updates of replication hosts. M The section has the three subsections: an (m)ftp updates instrumented mirroring experiment is presented that measures server throughput under rsync+, followed by data from network throughput clients S3 measurements using the well-known ttcp tool [9] and finally, a short discussion on S1 clients the scalability implications of the server S2 and network measurements. at each of S1, S2, S3: updates rsync+ -f src/ 3.1 Experiment 1: Server Throughput

Figure 2.2: Rsync+ Scenario for Content Replication in I2-DSI Methodology: In this section, performance data from a data mirroring

NetStore 1999 - 4 - B. Dempsey and D. Weiss experiment utilizing rsync+ is presented. run to synchronize separate copies of the The experiment created a mirror site of srcMaster/ and src/ directories on M. approximately 8 gigabytes (GB) of Linux-related files from a busy WWW Finally, to measure the processing that archive site (the channel provider in the would take place for the remote update language of Figure 2.2.), at each slave site (Step 3 in rsync+ metalab.unc.edu. The Linux archives at updates), we ran rsync+ again on M, this UNC MetaLab were chosen because the time with -f option using a pre-updated file archive is diverse (e.g., source code, version of each directory. In this way, program executables, documentation, each update period in the instrumented html, and graphics files) and active, mirror generated data on the rsync+ receiving several significant processing times at both the master (the contributions from developers and users –F option) and the slave (–f option) side. across the open-source Linux Mirroring data shown represents community on a typical day. The approximately one week in duration, i.e., processing costs associated with 15 updates separated by 12-hour managing the mirror’s update with intervals. Further details on the rsync+ thus represents changes experimental methodology can be found associated with a real and quite active in [8] in which the mirror was carried WWW archive with a modestly large out as described here, only using a less content base. powerful mirror host.

Seven active subdirectories under the Results: Figure 3.1 and Figure 3.2 show /pub/Linux tree (X11, apps, devel, docs, the rsync+ processing in the mirror games, system and utils) were chosen experiment. In Figure 3.1 rsync+ for the mirror experiment, and two processing times (both –f and –F modes) copies of these directories were placed for each update of the experimental on an I2-DSI server to initialize the mirror are plotted as a percentage of the mirror. To match the terminology in rsync processing time over the same Figure 2.2, the mirror machine will be data. The data shows that rsync+ –F called M here. It is an IBM enterprise- processing costs are very comparable to class storage server in the I2-DSI testbed those for the original unmodified rsync with 2GB main memory and 72 GB disk. code. The –f mode in rsync+ adds some Two copies of the Linux materials overhead, on the order of 1.5 to 2 times referred to as the srcMaster/ and src/ the unmodified rsync processing. copies to indicate their roles. The srcMaster/ was synchronized with the Figure 3.2 presents data on the directories at MetaLab twice daily using measurement of server throughput under an unmodified rsync as the import the rsync+ -F update on M to mechanism. After srcMaster/ had been synchronize the src/ with srcMaster/. updated, rsync+ -F was run on M to The dark squares shows the absolute synchronize the src/ with srcMaster/. running times of rsync+ -F processing in Each of the seven subdirectories was seconds. The updating of Linux files at updated with a separate run of rsync+ - the original MetaLab archive site was F. In addition, for a direct comparison of bursty, and several 12-hour periods rsync and rsync+ processing, rsync was resulted in no updates, though the rsync-

NetStore 1999 - 5 - B. Dempsey and D. Weiss

based mirrors must still run and traverse data sets and thus achieve throughputs in the file trees to determine that no files the high end of the measured range. have been changed. Rsync+ preserves the differential file The two other data sets in Figure 3.2 update capability in rsync. In the give two calculations of server experiment reported here, the aggregate throughput (in Mbits/sec), based on the amount of data generated by rsync+ -F amount of file update information processing represented only a 2% generated. The unnormalized throughput reduction in the total number of bytes for is defined as the sum of the size of all transmission across the network, when files updated in the mirror (added or compared with summing the bytes in all changed) divided by the running time of files added or modified over the lifetime rsync+ -F. The normalized throughput is of the mirror. At this level of defined similarly except the fixed compression, the rsync feature of overhead for rsync+ to traverse the file differential update (which can be tree is subtracted out by subtracting the disabled with a flag) seems less than minimum rsync+ -F runtime in the data useful. However, the month-long mirror (13.3 seconds) from each measured experiment reported in [8] showed that rsync+ -F runtime value. differential file update reduced the total byte count of update information by Throughput measures under both metrics almost 20%. In the I2-DSI context, the span a wide range, depending on the size decision to use differential file update of the mirror update. Over the non-zero will likely be made on a channel-by- entries, the average unnormalized channel basis since update behavior will throughput is 2.4 Mbits/s and the vary between channels. average normalized throughput is 11.4 Mbit/s. For data mirroring in I2-DSI 30 then, an important performance element is to aggregate data sets such that non- 25 null updates under rsync+ run over large 20

15

2 10 1.5 Run time as % of 1 5 rsync 0.5 0 0 0 5 10 15 20 1 3 5 7 9 runtime (sec) unnormalized tput normalized tput 11 13 Mirror updates 15

(one per 12 hours) 3.2: Rsync+ -F processing times and data throughput rsync+ -F rysnc+ -f

Figure 3.1: Rsync+ processing as a percentage of rsync processing

NetStore 1999 - 6 - B. Dempsey and D. Weiss

3.2 Experiment 2: Network the Internet2 backbone (the same as used Throughput in Experiment 1) and a local campus Data mirroring under the model in server across a regional network. The Figure 2.2 allows for transmission of the minimum speed of all network links updates file as soon as rsync+ -F between these machines is 100 processing completes. If a reliable Mbits/sec, and both are large enterprise- multicast transport is available, only a class servers. Measurements using ttcp single copy of the file need be are sensitive to socket buffer settings at transmitted into the network. While this the receiver (see [10]). The setting here is the most desirable scenario, was determined from ttcp runs between multicasting is not universally available the two machines using a range of in WANs today. Instead, in the absence values. No single value is optimal since of reliable multicast solutions, the network conditions fluctuate, but the 240 natural strategy is to transmit the file KB value is in the range of values using multiple, concurrent TCP offering the highest throughputs connections, one to each slave site. This achievable. section presents experiments measuring the throughputs attainable in current Parameter Values for high-speed WANs using concurrent TCP Experiments in connections. These results benchmark Figure 3.3 the network throughputs under the most File Size 5.45 MB likely scenario for network distribution Network Path dsi.ncni.net à between the master and slave sites in I2- (100 Mbit/s min) ils.unc.edu DSI in the near-term. Concurrent ttcp 1,2,4,8,16,24,32 Methodology: The experiments connections presented use the well-known ttcp tool Receiver socket 240 KB for measuring TCP throughput. The buffer size (KB) critical parameters for the data presented in Figure 3.3 are shown in Table 3.1. Buffer Policy Policy 1: Same-value The file size of 5.45 MB was selected Policy 2: Shared-by-n based on the average size of the updates Table 3.1: Parameters for ttcp file generated by rsync+ in the month- experiment shown in Figure 3.3 long Linux mirror reported in ([8]).

The network path for the measurement shown is between the I2-DSI server on

NetStore 1999 - 7 - B. Dempsey and D. Weiss

peak hours. The vertical axis is shown Tput, Buffer Policy 1 on a logarithmic scale. The curves Tput, Buffer Policy 2 labeled as Aggregate Throughput are Aggregate Tput, Buffer Policy 1 calculated by multiplying the average Aggregate Tput, Buffer Policy 2 throughput by the number of concurrent sessions. As shown, our data showed 100 single ttcp connections only achieved, on average, 27 Mbit/s, throughput, but when running parallel ttcp transmissions, the I2-DSI server was able to utilize up to 70 Mbit/s of the network path. The use of multiple TCP connections 10 increased the parallelism in the overall network delivery and actually achieved a better utilization of the network, though the reduced throughputs result in greater network transit times for updates.

The data suggests that using TCP to 1 emulate multicasting to at least a few 1 2 4 8 16 24 32 dozen slave sites is a feasible strategy. However, as the number of concurrent Figure 3.3: Network throughput for sessions grows, the burden of concurrent concurrent ttcp transmissions processing and data movement at the transmitter grows. In our experiments, For multiple concurrent ttcps, two starting up 48 processes running ttcps different policies for setting the socket resulted in blocked processes, possibly buffer size were used: Buffer Policy 1 related to the exhaustion of kernel assigns all concurrent ttcps the same buffers, and hence the ttcp sessions value (e.g., 240 KB), while under Buffer could not operate in parallel. While Policy 2 the concurrent ttcps “share” the further tweaking of transmission buffer size (e.g., all use 240 KB divided parameters will likely break down this by the number of concurrent ttcps). Note barrier, such tweaking may be difficult that, while our experiment uses only two to perform in operational settings. More machines and thus simulates a worst- fundamentally, the limitations of TCP- case sharing of the network path, the ttcp based delivery are evident as the number receivers discarded data upon receiving of receivers moves beyond the range of a it (the –s option) so that disk bandwidth few dozen. at the receiving machine would not be the measured bottleneck. At the 4. Related Work transmitting site, data transmissions Rsync is a recent entry in the class of were taken from a file in order to include application-level mirroring tools such as the overhead of file I/O, as would be the rdist, ftp-mirror [11], and mirror [12] case under rsync+. that automate the synchronization of file Results: Figure 3.3 shows the average hierarchies on different hosts. Key ttcp throughput as averaged over a series benefits of application-level mirroring of runs (6 per data point) taken in off-

NetStore 1999 - 8 - B. Dempsey and D. Weiss tools such as rsync+ is their high degree the hierarchy by NFS clients polling for of portability, ease and flexibility in changes at their parent. An update file modifying and tuning the replication containing commands to describe the process, and the ability to propagate a new filesystem snapshot at the parent is solution quickly. Rsync is unique in its used to avoid each child performing support of differential file update using redundant processing to discover the block-level checksumming, and our new snapshot, analogous to our single- experimental data suggests that shot processing and batch file generation differential file update provides a under rsync+ -F. This approach is significant reduction in update bytes attractive in that it leverages the using real-world data sets. However, the robustness, error resiliency, and kernel- processing costs of checksumming are level performance advantages of the high, and further investigation of non- NFS protocol. The key difference checksumming approaches would be between it and rsync+ is that the latter interesting. has the ability to exploit and readily incorporate reliable multicasting in the Network filesystems represent an network transport component. alternative approach for file sharing and distribution across distributed platforms. A recent distributed filesystem design, The widely-used commercial solutions, JetFile, [18] does explicitly incorporate AFS [13] and DFS [14], both implement reliable multicast transport. JetFile is a limited form of read-only replication explicitly designed to be a distributed that is generally statically configured for filesystem for WAN-connected Internet load balancing. Other approaches to machines, and uses multicast for replica distributed filesystems such as Coda [15] location, synchronization, and and Bayou [16] incorporate write-log management. JetFile exploits multicast management in order to maximize file for efficient multipoint distribution of availability for their distributed and/or immutable file objects. Likewise, other mobile users. The initial application researchers have proposed multicast file paradigm in I2-DSI is oriented towards a distribution in the context of enhancing publisher-subscriber model that makes rdist-like file distribution [19], push centralized update a reasonable caching [20], simple file distribution to constraint and therein the lightweight WWW servers [21], document delivery rsync+-based file distribution model of applications [22], and other applications. Figure 2.2 feasible for file-oriented As noted by the JetFile designers, channels. multicasting offers not only advantages in efficient bulk-data distribution, but An interesting case of distributing large also flexibility in organizing host file sets to widely dispersed hosts is communication that avoids the fragility described in [17]. Here a 6-level inherent to hierarchical arrangements. Network Filesystem (NFS) replication hierarchy for shared read-only data (11 Netlib [23] has been used for some time GB, 300,000 files) across a set of very now in the scientific numerical widely distributed hosts is presented as computing community to share an alternative to hierarchical rdists. repositories of freely available Changes at a master site are pulled down mathematical software and

NetStore 1999 - 9 - B. Dempsey and D. Weiss documentation. Netlib is a set of tools times as opposed to the 2 AM tests of that automates management of the the vBNS measurements, and the replication procedure across loosely- inclusion of disk I/O at the transmitting coupled servers administered by separate node. organizations. Each server may have a portion of the aggregate repository for 5. Conclusions which it controls all updates (master) We have presented a modified open- and other portions for which it mirrors source data mirroring tool, rsync+, as a remote sites (slave). Netlib sites use basis for constructing scaleable, background processes to generate an multiple-site file replication across index of each file paired with a backbone-speed networks connecting checksum over the file's contents. replication hosts. Controlled Updating remote slave filesystems uses performance experiments in the I2-DSI checksum comparisons to determine testbed with live WWW content were which files have been changed, and presented to provide insight into the network transport involves an FTP script relative match between network and generated by a Netlib process. server-side data throughput.

A key difference between the Netlib Our performance experiments with scheme and our data mirroring approach rsync+ confirm that server-side is the rsync+ emphasis on timely and processing is a significant cost in the synchronous distribution of updates to end-to-end update process during file all participating sites. By contrast, the replication. These empirical observations Netlib project emphasizes automating validate the value of the rsync+ design in the replication process in a manageable eliminating redundant server-side and robust fashion across loosely processing at a master site for multiple cooperating inter-organizational servers. mirrors. These servers are not simultaneously updated when content changes occur. As for scalability of the rsync+ design, Moreover, Netlib clients most often our performance numbers show that access the server within their TCP-only delivery to multiple remote organization, in contrast with the mirrors is feasible for small slave sets dynamic binding in I2-DSI between a (e.g., roughly less than 50). For properly client and a replication server. aggregated data sets, the server throughput measurements suggest that Performance measurements of the vBNS rsync+ can update local file hierarchies backbone using ttcp are reported daily at 10 Mbit/s or greater. Under these by researchers at [24]. These conditions, as the number of slave sites measurements of individual ttcp grows, the mirroring mechanism connections (and others) report generally becomes quickly network-bandwidth high (> 100 Mbit/s), though variable, limited (see Figure 3.3). Thus, for throughput across the vBNS. Factors in replication host sets beyond a few dozen, our measurements that result in lower reliable multicast solutions will be very throughputs include the use of a valuable in speeding system production server as the receiving performance and reducing system load. machine, runs over a range of evening

NetStore 1999 - 10 - B. Dempsey and D. Weiss

presented at InterOp '98 References Engineering Conference, Las [1] P. Rodriguez, C. Spanner, and E. Vegas. NV, May 1998. Biersack, “Web Caching [11] L. McLoughlin, “ftp-mirror” . Architectures: Hierarchical and [12] F. Leitner, “Mirror 1.0”,1996. Distributed Caching,” presented [13] A. Z. Spector and M. L. Kazar, at 4th International Web Caching “Wide Area File Service and the Workshop (WCW 99), San AFS Experimental System,” Diego, CA, 1999. Unix Review, vol. 7, March 1989. [2] M. Beck, B. J. Dempsey, and T. [14] M. L. Kazar, B. Leverett, O. Moore, “The Internet2 Anderson, and e. al, “Decorum Distributed Storage Infrastructure File System Architectural (I2-DSI) Project,”1999. Overview,” presented at 1990 [3] M. Beck and T. Moore, “The USENIX Conference, Anaheim, Internet2 Distributed Storage CA, 1990. Infrastructure Project: An [15] M. Satyanarayanan, “Scalable, Architecture for Internet Content Secure, and Highly Available Channels,” Computer Distributed File Access,” in Networking and ISDN Systems, IEEE Computer, vol. 23, 1990. vol. 30, pp. 2141-2148, 1998. [16] A. J. Demers, K. Peterson, M. [4] Akamai, Akamai Technologies, Spreitzer, D. Terry, M. Theimer, Inc., 1999. and B. Welch, “The Bayou [5] Sandpiper, Sandpiper Networks, Architecture: Support for Data Inc., 1999. Sharing among Mobile Users,” [6] Y. Chen, J. Edler, A. Goldberg, presented at Workshop on A. Gottlieb, S. Sobti, and P. Mobile Computing Systems and Yianilos, “A Prototype Applications, Santa Cruz, CA, Implementation of Archival 1994. Intermemory,” presented at 4th [17] B. Callaghan, “An NFS ACM Conference on Digital Replication Hierarchy,” Libraries (DL '99), Berkeley, presented at W3C Push CA, 1999. Technologies Workshop, [7] Rsync, 1999. Peabody, MA, 1997. [8] B. J. Dempsey and D. Weiss, [18] B. Gronvall, A. Westerlund, and “Towards An Efficient, Scalable S. Pink, “The Design of a Replication Mechanism for the Multicast-based Distributed File I2-DSI Project,” School of System,” presented at Third Information and Library Science, Symposium on Operating University of North Carolina at Systems Design and Chapel Hill TR-1999-01, April Implementation (OSDI '99), New 1999. Orleans, LA,, 1999. [9] Test TCP measurement tool, [19] J. Cooperstock and K. “TTCP archive site,”1999. Kotsopoulos, “Why use a fishing [10] G. Miller, K. Thompson, and R. line when you have a net?: An Wilder, “Performance Adaptive Multicast Data Measurements on the vBNS,”

NetStore 1999 - 11 - B. Dempsey and D. Weiss

Distribution Protocol,” presented at Usenix '96, 1996. [20] J. Touch, “The LSAM Proxy Cache: a multicast distributed virtual cache,” presented at 3rd International WWW Caching Workshop, Manchester, England, 1998. [21] J. Donnelley, “WWW Media Distribution via Hopwise Reliable Multicast,” presented at 3rd International WWW Conference, Darmstadt, Germany, 1995. [22] T. Shiroshita, O. Takahashi, and S. Shiokawa, “A large-scale contents publishing architecture based on reliable multicast,” presented at 15th Annual Conference on Computer Documentation (SIGDOC '97), 1997. [23] Netlib, 1999. [24] vBNS Performance Measurement, vBNS Engineering, 1999.

NetStore 1999 - 12 -