Experiences in Building and Operating ePOST, a Reliable Peer-to-Peer Application Alan Mislove†‡ Ansley Post†‡ Andreas Haeberlen†‡ Peter Druschel† {amislove,abpost,ahae,druschel}@rice.epostmail.org †Max Planck Institute for Software Systems, Stuhlsatzenhausweg 85, 66123 Saarbr¨ucken, Germany. ‡Rice University, 6100 Main Street, MS-132, Houston, TX 77005, USA. ABSTRACT Keywords Peer-to-peer (p2p) technology can potentially be used to build Peer-to-peer, electronic mail, reliability, decentralized sys- highly reliable applications without a single point of failure. tems However, most of the existing applications, such as file shar- ing or web caching, have only moderate reliability demands. 1. INTRODUCTION Without a challenging proving ground, it remains unclear Decentralized, cooperative systems (also known as p2p whether the full potential of p2p systems can be realized. systems) have received much attention in recent years be- To provide such a proving ground, we have designed, de- cause they promise robustness to a wide range of work- ployed and operated a p2p-based email system. We chose loads and failures, are highly scalable with respect to nodes, email because users depend on it for their daily work and users, and data, and enable cooperative sharing of otherwise therefore place high demands on the availability and reliabil- underutilized resources. The technology was first adopted ity of the service, as well as the durability, integrity, authen- for use in file sharing and content distribution systems like ticity and privacy of their email. Our system, ePOST, has KaZaA, Gnutella, eDonkey and BitTorrent [3, 20, 29, 40], been actively used by a small group of participants for over which now enjoy widespread popular use. While these sys- two years. tems have grown to a considerable scale, they have only very In this paper, we report the problems and pitfalls we en- modest reliability demands. In general, their users seem to countered in this process. We were able to address some of tolerate slow downloads, repeated failures, and even cor- them by applying known principles of system design, while rupted data, as long as they get the content eventually. others turned out to be novel and fundamental, requiring us Prototypes of more reliable p2p systems, including con- to devise new solutions. Our findings can be used to guide tent distribution [3, 10, 16, 55], information retrieval [24, 52, the design of future reliable p2p systems and provide inter- 53,56], and name resolution [41,54], have mostly focused on esting new directions for future research. availability, scalability, and resilience to overload. Coopera- tive storage systems [5,12,30] additionally require durability and consistency, but to our knowledge, these systems have Categories and Subject Descriptors not yet been developed and deployed for production use. C.2.4 [Computer-Communications networks]: Distributed Thus, it remains unclear whether the full reliability poten- Systems—Peer-to-peer applications; H.4.3 [Information Sys- tial of p2p systems can be realized in practice. tems]: Communications Applications—Electronic mail; D.4.5 In an effort to address this question, we have designed, [Operating Systems]: Reliability—Fault-tolerance; D.0 deployed and operated ePOST, a serverless email system. [Software]: General—Peer-to-peer systems We have chosen email not because it is a killer application for p2p (it isn’t). Rather, we chose it as a proving ground because it is well understood, widely used, and has strin- gent reliability requirements. Many users depend on email General Terms for their daily work and therefore place high demands on the Algorithms, Design, Measurement, Reliability, Experimen- availability of the service, as well as the durability, integrity, tation authenticity, and privacy of their email. Hence, by demon- strating that users can rely on a p2p-based email system as their primary email service, we believe that we have taken a major step towards establishing p2p technology as a viable Permission to make digital or hard copies of all or part of this work for platform for applications with high reliability demands. personal or classroom use is granted without fee provided that copies are As an additional benefit, the basic elements of an email not made or distributed for profit or commercial advantage and that copies system — event notification, durable storage of mostly im- bear this notice and the full citation on the first page. To copy otherwise, to mutable data, and some per-user mutable metadata — are republish, to post on servers or to redistribute to lists, requires prior specific characteristic of many collaborative applications. Demon- permission and/or a fee. EuroSys’06, April 18–21, 2006, Leuven, Belgium. strating that highly reliable email services can be provided Copyright 2006 ACM 1-59593-322-0/06/0004 ...$5.00. using p2p technology suggests that the technology can sup- port other collaborative applications and even enable new of streaming data, while ePOST’s focus is on availability, applications. durability and consistency. In this paper, we report on our experience in designing, End System Multicast (ESM) [10, 25, 49] is a deployed deploying, and operating ePOST. Since ePOST users ex- system used to broadcast streaming video using cooperating pect the system to perform equally well or better than a end hosts to disseminate the data. ESM, like VoIP, is used conventional server-based system, our challenge was to pro- for streaming media data, which has high requirements of vide an acceptable user experience from a system built upon timeliness but not reliability or durability. This makes the unreliable machines and networks. Surprisingly, the prob- challenges it faces very different, though no less significant, lems we encountered were generally the result of incorrect from those addressed in ePOST. assumptions made in the design of underlying component OpenHash [28, 45] is an publicly open DHT system that technologies like the structured overlay and the distributed provides a public hashtable interface allowing users to store hash table (DHT) used by ePOST. Seemingly unrelated and retrieve data. OpenHash is a public service infrastruc- user-perceivable errors and inconsistencies in ePOST oper- ture for others to build applications and therefore does not ation often led us to discover fundamental problems with emphasize data durability and consistency to the extent re- these components. quired for running ePOST. We describe in some detail the challenges we encountered CoDoNS and CoDNS [41, 44] are decentralized systems in providing users with an acceptable service. This means that seek to replace or augment the existing DNS infras- providing strong data durability, high availability, and a tructure to improve its reliability, availability, and perfor- mostly consistent view of the user’s email folders despite mance. Since DNS entries are infrequently updated, con- correlated node failures, network partitions, network anoma- sistency problems are not likely to be visible to the user. lies, and node churn. In some cases, we were able to address Coral [11,16] is a cooperative web cache, which is centrally problems by applying known principles of system design; administered on the PlanetLab [43] network and does not al- other issues turned out to be novel and fundamental, re- low untrusted nodes to join the network. Coral-CDN forms quiring us to devise new solutions. After a partial re-design a cache of data that can be found on the Internet, and as to address initial problems, ePOST has supported a small such all data contained can be recovered from the source. user base, including the authors, who have relied on the The Resilient Overlay Networks (RON) [1] project has system as their primary email service for over two years. deployed a network of nodes that can be used to increase Overall, this paper makes four contributions: (i) We re- the reliability of routing in the Internet. It augments the port on our experiences in designing and operating a co- service traditionally provided by the Internet and does not operative, serverless email service that supports real users. store data on behalf of users. (ii) We identify two problems, routing anomalies and non- transitivity, that emerged during our use of the system, and 3. BACKGROUND we describe and evaluate our solutions for them. (iii) We note some pitfalls we encountered that are of interest to de- In this section, we give a brief overview of the ePOST signers of similar systems, though we were able to address system, and we describe our current deployment. The initial them using known principles of system design. Finally, (iv) design of ePOST was previously sketched in [35]. we demonstrate that it is possible to support applications with high availability, data durability, and consistency using 3.1 Overview p2p technology. ePOST provides secure, decentralized email services to The rest of this paper is structured as follows: Section 2 its users. Each user runs the ePOST software on her lo- describes related work, and Section 3 presents background cal machine and contributes some of that machine’s CPU, on the design and deployment of ePOST. Section 4 describes storage and network bandwidth to the system. Users’ email four general lessons we distilled from our experience build- messages and metadata are stored cooperatively and redun- ing and operating ePOST, and we present and evaluate solu- dantly in a DHT backed by the participants’ disks. All data tions for several problems we had to solve. Section 5 details is signed for authenticity and encrypted for confidentiality some pitfalls we encountered, and Section 6 describes inter- before it is inserted into the cooperative storage. esting directions for future research. Section 7 presents our In order to deliver an email in ePOST, the sender inserts conclusions. the email and a special delivery request into the cooperative store. Nodes that store such a delivery request subscribe to 2. RELATED WORK a multicast group under the recipient’s identifier.